Reliable Systems

Many if not most software project leaders came up through the development ranks.  It’s generally thought of as a distinct advantage – you know the technologies you’re using, you can form your own well reasoned opinions about how hard something is, what is possible, and how long it should take.  For a long time, I felt that the best way to get results from development teams was to use my experience and knowledge to be very understanding of the challenges they faced and give them whatever time they asked for.  However, in the last few years I’ve run into several situations where I just couldn’t get them the extra time or relief from the most problematic requirements.  I predicted doom to the projects in question but instead I observed some of the best outcomes I’d ever experienced.

While the projects were successful, it bothered me that the secret sauce seemed to be a rigid adherence to schedule and delivery more than any other consideration.  This was exactly the reverse of how I wanted projects to succeed:  I wanted them to succeed because I was treating the developers how they always wanted to be, not like a stereotype from Office Space.  How could it be that better results came from ignorance of the technical details involved?

Developers Will Use All Available Time

Upon reflection, the first thing that struck me was how much an immobile deadline focused discussions and decision making. If you give a team more time, they will expand their process to consume it.  Time will get consumed by:

• Elaborate Decision Making: When you have little time, you make a choice and go with it until it appears it just can’t work.  When you have a lot of time, you sit back and look for the very best option.  That then requires defining what the best is – is it fastest, or smallest, or most scalable, or whatever.
• Development Approach: Under pressure you’ll tend to go with the proven guaranteed approach.  If you have the luxury of time you’re more likely to engage in yak shaving like investigating a new tool or approach, or writing several prototypes first before you develop the real solution.  You might even just throw caution to the wind by skipping a formal design figuring you’ll have the time to just code and test your way to a solution.

The more time a development team has, the harder it is to argue against spending it on up front luxuries.  It also can be harder to argue for long term best practices because the team has the time now to develop a solution any way they want.

Unknowns Create Boomerang Estimates

Even very experienced developers are generally terrible at estimating the duration of developing a solution.  This has been demonstrated over and over by many other parties.  The key behavior that we’ve observed is the phenomenon that from when you approach a specific development problem (like displaying a graph on a web page) until you know exactly how you’re going to solve it (and have a reason for confidence in that approach) you will tend to estimate high because in effect the only reasonable estimate is infinity.

Put another way, as long as you don’t know how you will solve a problem you don’t know for sure that it is solvable which means it will take an infinite amount of time to solve it.  Fortunately, developers are almost universally optimists so they believe they can solve anything eventually – so they’ll pull out a standard answer like three weeks or months or whatever feels like a big chunk of time to figure out the problem but not so big that it kills the project.   The reality is that until you know how you’re going to solve it, it feels like it could take forever.

Once a solution has presented itself  the development team will often find that all it will take is some cleanup and polish to be done- a very small amount of time.  What will push the team to find the answer?  We’re back to the problem of elaborate decision making when you have the luxury of time.  Finding solutions tends to not be a linear problem that will be solved with incremental development energy.  Instead, it tends to be solved by getting people together and brainstorming possible solutions until you find a few candidates and can work out what it’ll take to prove them out.  Under pressure, people tend to focus their creative energy and be more willing to compromise.   That flexibility will tend to get rid of pet requirements and developer gold-plating and focus on the most critical aspects of the problem.

What’s the alternate approach?

The key is to not let your knowledge and experience as a developer lead you to buy into the stories the team creates around what’s reasonable to get done and how long it will take.  Instead, you have to stick with the project’s goals first then the facts of the project.  The project’s goals form the objective reality of what has to be accomplished for the project to survive:  Deliver this functionality by that date, keep these people informed, solve these problems without causing those problems.

When the team runs into a wall and needs more time, instead of buying into the story of needing a lot of time, set a specific and tight goal that keeps a solid amount of time pressure on the team to solve the issue and prevent the problems above from showing up.  Ideally, find a way to give out one or two day chunks to answer incremental questions if necessary to emphasize that time is precious and has to be invested carefully.  This is where you can leverage your experience in a way that a non-developer can’t:  The team knows they can’t snow you with tech details, and you can define a specific, measurable result that can be achieved in a short period of time that they can’t argue with.  Despite this, you are bound to have to assert a few times that the time limit is the limit – solve the problem in that time.  It’s very hard because you’ve been on the other side of that conversation and it can feel like you’re the Pointy Haired Boss, but it’s fundamentally your job on the project.

What will nearly always happen is the team will surprise itself – a solution will be presented within the team that they can live with and can be done in the time they have.  It may be incomplete or have some risky shortcomings, and you’ll want to ask how long it’d take to address those.  You probably shouldn’t address them in the first round, but the team will feel better that you’ve considered through things and will buy into the outcome more if you ask.  You’ll also want to make a record of it so that the team can in the future recognize what was a predicted shortcoming vs. an accidental defect.

Do you want it solved right?

This is a question that often gets voiced within a team as a rebuttal to external time pressures and is very dangerous.  The challenge is that most non-technical people don’t get the number of ways that a problem can be solved: instead, each problem appears to have a single solution.  Take away your technical knowledge and imagine you’re the paying customer:  What’s the alternative – were you going to solve it wrong? If that’s the case, what else have you done that’s garbage?  If you took your car to a repair person and they said it’d be $500 to fix it, then when you came back they said well, if you want it fixed right it’ll actually be $1200, wouldn’t you wonder what the hell the $500 fix was?

Usually this statement is uttered in desperation when a team believes they just need more time to figure out a problem.  Nobody wants a problem solved wrong.  Skip the hyperbole and get down to action:  break down the problem into small chunks of time that can be invested for a specific measurable result, and make sure the team gets that overage time is the most precious commodity.

Side Note: This is an advantage of SCRUM in practice.  If you’re following an Agile Development practice, particularly SCRUM, this fits right in:  Focus on making each sprint deliver the user stories it was supposed to even if you have to leave some special cases for a later sprint.  The daily stand up meetings are a great place for the different team members to apply team pressure against over engineering and doomsday estimates.

Cleaning Up and Closing Out

At some point you need to close out your release and ship it. For each of the areas where you’ve had to make compromises and taken shortcuts you have to choose to either:

• Ship as Final: Decide the implementation is close enough to the intent of the end-user functional requirements that it can be the final implementation (at least until new information contradicts this decision)
• Ship as Temporary: Decide that something is better than nothing and ship the feature with limitations.
• Cut the Feature: Hold back the feature until it can be reconsidered or reimplemented.

You’re nearly always better off shipping the feature, often as a final feature pending more information because it’s very hard to gauge the true impact of each limitation.  This is particularly true of user-facing features and environments where it’s possible to evolve the software rapidly.  Inevitably once it’s in the hands of your users you’ll discover aspects of it that you didn’t think of that will require rework and you may discover that the killer feature you were sure would be the hit of the release is hardly used.  In either of these cases if you’ve invested a great deal of time in making it foolproof the team will tend to resist changing it.  It’s a natural product of the presumed relationship between effort and value.  If necessary, you might put in some temporary safeties to detect and catch the limitations you’re worried about.

The major exceptions to this approach are areas that are too dangerous to deploy if less than fully trustworthy.  For example, if your team is developing a data storage system, software deployment system, or other critical infrastructure your choices likely resolve down to making it as right as possible or holding the feature until it can be reworked.

If it turns out that the solutions that are viable within the schedule have significant limitations, you should make sure these caveats are known to the business – provided you can express them in business terms.  For example, knowing that an algorithm won’t work if your userbase doubles is probably not a significant caveat, unless you know the business plans to double in a relatively short period of time.  Every system has limits, and every software change has risks.  Business representatives don’t like to hear the same items covering the same ground repeated every time you discuss software, and it tends to make them not hear the new and important information as well as sound like you’re attempting to transfer accountability from your team to them.

Editor’s Note: This is the final article in a three article series. For a list of the entire series, see the Article Series page.

In the final article of our series we’ll look at how to create an automated build incrementally and make it a natural evolutionary process of your team, providing both immediate and ongoing value. Really, it’s no joke and it’s not going to require upending your technology or team.

Components of Build Automation

When introducing an automated build, we recommend pursing components in the following order:

1. Compile files from Source: Retrieve all of the input data from the configuration management system, and compile everything that needs to be compiled. If you’re working in a language or technology that doesn’t require compilation, just retrieve and label all of the input files. Compiling may include activities such as automatically assembling release notes from the defect tracking system or any other action necessary to create a file that is distributed. Check compiled results back into the configuration management system.
2. Assemble Build from all Files: Take all of the files needed to create your distribution including every dependency, release notes, etc. and create your distributable build. The distribution (a.k.a “build”) ultimately should be exactly one file per target platform. Copy to a central networked location either named with the unique build number or otherwise clearly identified.
3. Automated Unit Test: Using a third party framework (recommended) or your own custom framework perform automated tests designed to exercise the individual components of your system in a very detailed way. This is often easier than a real system test because the tests will tend to be more stable over time and more compartmentalized.
4. Install and Smoke Test: On a green system, perform a fresh installation and basic smoke test of the system.
5. Automated System Test: Perform a full automated test of the public surface area (that reachable by users) of the system.

Compiling Files from Source Control

The first task to take on is to automate compiling all of the files from source. This is required before the build can be centralized, and really is the cornerstone of the process. This step will require the most investment before you realize any return. We recommend taking a developer with IT administration experience and dedicate them to the task. Many IT administrators are used to automating tasks and working with installing and cleaning up software, and a solid understanding of administration can be very handy for this step. In our experience it can take as little as a few days of time to as much as a few weeks depending on the experience of the developer and the complexity of the product.

It’s important that the build process be idempotent to be valid: Regardless of where the build gets started, you should be able to restart it and have it recover, cleanup, and then work. Generally it’s best to clean up the failed build then proceed with a good build instead of trying to pick up where the previous build left off (it’s more deterministic). This approach also lets you develop the build iteratively with a failed build recovery stage added to the start of a normal build process.

The build process must ensure that it generates the right labels to meet the traceability goals of a build. The best way to do this is to generate a unique build number, label the source code, then pull the source code based on the label. Even in source code management systems that aren’t completely transactional, retrieving source code by label is going to be consistent which is the key goal. This allows the build to run at any time without requiring developers withhold from checking in or out source code due to fears of interfering with the build. For best consistency, the build process should label all of the necessary source code in as short an interval as feasible (to guard against drift between projects) and then it can pull the source code as needed during the build process.

The build should be easy to extend with new projects. This requires spending a little time considering how to externalize what projects to compile, where to get them, and where to put the output from the raw build process itself. We spent some time writing a standard build script that integrates with our product of choice which uses a single data file to tell it all of the information it needs for any one product. This lets us set up new products very quickly and amortizes the development effort of making the build process over multiple projects.

Some example products that can help you fully automate your build:

1. ANT: Free and capable. Probably the best freely available build system. Available in many flavors to match your technology (like NANT).
2. Visual Build Pro: We’ve used this on several projects and it’s what we use internally. Sports a great GUI for developing and debugging the build process and comes with build-in interfaces for pretty much anything you want to talk to on Windows, and can be easily extended. Great for folks that prefer an IDE. Very cost effective.
3. MSBuild: Visual Studio ships with an internal build environment that can be extended to handle a number of tasks and, with enough force of will can be used to do most anything you want. Debugging and extending it to perform tasks beyond basic compilation and file copying can be a challenge, and your time is probably better served using a dedicated build framework.

Centralizing the Build

Once you’ve automated your build, you can move it to a central server. To centralize the build on a common server, you need to have a mechanism that satisfies at least the following:

1. Anyone can trigger a build remotely: Anyone (subject to some basic security authorization) can trigger a build, and do so without any particularly specialized knowledge. The build system will automatically know if it’s safe to build (such as ensuring conflicting builds aren’t run at the same time). This has to be available from anywhere developers are.
2. Easy access to the status of builds remotely: It should be trivial to know if a build is underway and to know the historical success of the build process. This has to be available from anywhere developers are.
3. Works logged off and through restarts: The build process should not require the system be left logged in or require manual steps to bring online after a computer restart.
4. Runs as a unique user: The build process shouldn’t use anyone else’s identity to log into the source code management system or access other resources so it’s very clear from an audit perspective when the build did something.

Some example products that can centralize your build:

1. Cruise Control: Pretty much the standard. Free, capable, and satisfies the requirements. We use the .NET oriented version, Cruise Control .NET for our in-house build system. The user interface is fairly primitive, and the non-web client is unduly cranky, but it is a good remoting system. It isn’t a particularly good build system – you’ll want to use one of the products listed below.
2. Automated Build Studio: This commercial product for the windows platform is reasonably capable and is both a build product and a build centralization system. While it is capable, the pricing is fairly high when you compare it to Cruise Control + Visual Build Pro (see below) which is substantially as capable. This is because you really don’t need may Visual Build Pro license(s) on a typical team, but you will need a number of Automated Build Studio licenses to allow anyone to invoke and monitor a build. The centralization capability is relatively new in the product. The main reason to go this route is that one configuration IDE can give you centralization and automation, so you are trading money for time.

Challenge Your Team To Fill It Up, Then Buy A Bigger One

While it is very tempting to recycle some old developer system or server as the build server because it doesn’t feel like performance should matter, investing in a high speed build server can pay back quickly by allowing the build process to be optimized for strictness and reliability instead of performance while keeping it fast enough to preserve the development team’s attitude that builds are free. When your build process gets long (up to 1 hour) your team gets a great boost in productivity by purchasing just one system. Consider that a great new build server should top out at $5,000. If you replaced it every 18 months it’d double in performance at the same price.

When specifying a build server, you want to emphasize single processor performance and disk performance. Memory is generally not a particular issue, so invest in the fastest pair of disks you can find with a good hardware RAID controller (for the very best disk throughput) and the highest gigahertz single socket processor you can get. Build processes can very rarely take advantage of multiple cores, so anything beyond two cores isn’t going to speed up a single build, but gigahertz will. Finally, make sure it has a big network pipe to the configuration management system because it will spend a fair amount of its time pushing things in and out of it.

Automated Unit Tests

Connect your existing unit test system to the build system to automatically perform the unit tests on each build and post the results. This can generate some useful metrics that you can use to understand the quality and progress of your development:

1. Increase in Proportion to Features: As you are claiming victory adding features to your system, you should see a linear increase in unit tests. For example, if you have 100 unit tests and the current system has say 200 design features then in broad terms for every 20 design features you should see 10 unit tests. It isn’t completely accurate- it’s a guide line. However, if you see only 5 unit tests and 25 features, you know something is up
2. Indication of Design Complexity Problems: If you are seeing unit tests routinely break that previously worked, or a single unit test repeatedly break, this will tend to indicate that the design of the system (the architecture or software patterns or implementation) is unreasonably complicated for its feature goals. In the abstract it’s often hard to have team discussions in these points because it’s all a matter of tradeoffs, experience, and opinions. This will give you empirical evidence that the software is overly complicated to keep functional.
3. Indication of Performance Issues: Every unit test, when tracked over time, gives you a clear trend to understand the performance of your system. It provides a highly standardized test case where the exact same routine was run on the same hardware in the same way, and timed. If you see the duration of your unit tests change (up OR down) it’s worth investigating – it may be failing quickly or slowly internally, or you may have counter-optimized the code.

Most Automated build systems can directly run one or more unit test frameworks. The build products discussed above each can do this. If your build system can’t, you might look at one of these. You should definitely check out:

1. NUnit: Unit testing for .NET. And yes, we get the irony that the site is written in PHP. This is our unit test framework of choice, however it’s worth noting that we’d be a lot less enamored with it if it wasn’t for ReSharper’s ability to act as a dramatically better test runner.
2. JUnit: Unit testing for Java. Don’t let the nearly comical web site fool you, this test system is all meat.
3. TestComplete: A commercial product from the same folks that made Automated Build Studio. It goes way beyond unit testing (as do most commercial test products) and integrates with both Visual Studio and Automated Build Studio, so if you go with ABS you might look to use these together. It earns our honorable mention because it’s reasonably affordable, easy to use, and very approachable.
4. Hundreds of others: There are many automated testing products out there. We recommend you start with something very simple and straightforward and stay away from the large enterprise testing systems. These are really not meant for the needs of small and mid-sized development teams, particularly where your QA staff are largely development trained.

Run It Every Day

Every development project should be built every day if there’s any change made to the source it comes from. The build centralization systems mentioned above can detect if there is any change to the affected projects checked in and automatically queue a build for a fixed time of day (for a scheduled, automatic build) or a few minutes after the change is checked in. The latter is a great approach for a team that is fully adopting agile development practices: Configure it to automatically start a build after any checkiin after there haven’t been any checkiins for a few minutes. This gives rapid feedback to the team and encourages good configuration management discipline: What you check in better build and pass tests, so don’t check it in until you’re ready.

This frequent execution will ensure that build changes need to be coordinated with code changes because they’ll fail the build immediately otherwise. This quickly will instill the discipline within the development team to keep the build clean, which in the end takes up the least time. Just like it’s easier to maintain something than fix it, it will take the least time across the team to keep the build live and accurate than deal with the downstream consequences of not having your house in order.

Get There In Stages

The great thing about centralizing and automating your build process is that you can get there incrementally. You can make minor investments in time from various team members and each investment will add a little value to your automated build process which in turn will generate a return from there forward. There aren’t a lot of development practices that can be incrementally adopted in such small chunks yet produce steady returns.

The incremental approach is highly recommended because it helps conquer some objections by not requiring a major change in developer habits or a major investment in developer time up front. You can even usually get it done with such small investments in time that it doesn’t need to be a formal project or even a formal part of your project plan, if you are concerned about internal resistance to spending work on non-development activities.

Wait, What about Visual Studio Team System?

Microsoft’s Visual Studio Team System (VSTS) does provide all of the infrastructure you need to automate your build process – centralization, configuration management, testing, reporting, the whole package. If you can afford its licensing fees and the investment in time and resources it takes to set it up, it will fit the bill. VSTS is aimed at larger development teams – on the small side 20+ developers (people actually writing code). The tools and techniques discussed in this article are intended to provide value on teams down as small as two developers. Unless you have large development teams or free VSTS licenses, it’s probably not your best bet.

About Product Recommendations

This article features specific product references and recommendations. Neither the authors of this article nor eSymmetrix are affiliated with any company mentioned, nor have they received any consideration at any time from a party with an interest in these products.

Editor’s Note: This is the second article in a three article series, with a new article posted every few days.

In the last 15 years, software development tools have advanced significantly. More than language or runtime, the modern integrated development environment is an amazing productivity tool. You don’t even have to spend money to get this benefit – check out Eclipse and its ecosystem for a free and powerful tool. Combined with the extraordinary processor power of modern desktops, you can do things today that just weren’t feasible 15 years ago. In many cases, this is a great improvement, but it has come at a cost: When you can compile even a large project and debug it in tens of seconds instead of 30 minutes it tends to take you away from the good Mom and Apple Pie aspects of having a tight build process and not doing development by debugging.

Benefits of a Build System

When you have a common means of performing each build, you get a number of advantages that will pay off for your team.

1. Consistency: Without a doubt, the greatest benefit is that each build is consistent and you know that it is consistent. Both pieces are important. You will be able to dismiss with confidence questions about the accuracy of the information that indicates what exact source code was used to make each build, what defects really were fixed in each build, etc. This is particularly powerful because it is most useful when the team is under maximum stress: rapid fire builds under pressure, when there are multiple builds in play at various stages of certification, etc. It’s in these situations that you have to be able to absolutely trust that the records are accurate or you will burn a significant amount of time across your team verifying what was in each build. This is waste effort when you can tolerate it least.
2. Knowledge Containment: Each developer doesn’t need to know the details of building the entire system. Usually only one developer needs to know the packaging and release details, numbering scheme, etc. Individual developers only have to know at most what it takes to add their new project to the build process. When a new member joins the team, they can get up and running fast and start contributing without having to take on the whole beast right away. When you come back to the system six months after it shipped and need to make one change, no one needs to remember all of the intricacies of creating a full build.
3. Change Management: Whenever a new output file is added or a change needs to be made to how the application is built, the build will need to be updated. By insisting on an automated process you ensure that each change has to be fully addressed before the build can be tested. This will force issues to the surface earlier in the process when the team will tend to be under less stress instead of later when the team is under tighter deadlines.
4. Time Savings: Running a build is generally an annoying waste of time. If you take on that a build includes all of the steps we discussed in the first article in this series then even for a simple project it’s going to take a minute, but for a project of real complexity it will tend to be 10 minutes or more. More is actually better in this case because it represents more human actions that are automated into the build. If the build is happening on the developer’s system then the time savings is limited – the developer will generally be idle while the build is running. The time saved in this case is the time for a human to manually perform each step vs. the computer, perhaps 40% of the total time.

Benefits of Centralization

This part is often more controversial, but has become accepted more and more as a best practice. Many have argued that you should automate the build, but make it so each developer can run the build on their system. This was relatively straightforward in the good old days of “make”, but there are several advantages to centralizing builds for modern applications.

The big benefits of centralization are:

1. Forced Standardization: We’ve all had the experience of two computers building the same source code and not producing the same binary images. This is generally due to small differences in the software installed on each system that the build is referencing. It probably won’t hurt you, but it might – and the more third party items you reference the higher the odds it will. Instead, by building on the central server you can ensure that if there’s a new version of a common library you know exactly what one is being referenced by the build. You know that you only have to update it in that one place for it to go out, not on each developer’s system.
2. No Cheating: Let’s face it, it’s tempting – any individual developer will be tempted in the right circumstances with short cutting a build on their system, perhaps swapping one file into the output directory and just re-running the packaging step to get a new distribution. If it’s on a central system that few if any have direct access to then you know for sure each build came from the full, correct process.
3. Major Time Savings: With the build happening on a central system, the developers are free to do whatever they want while it’s running. They can continue to do extended unit testing waiting for the build to complete, or move on to entirely new functionality. The build should be designed to pull its source code immediately or otherwise ensure that it’s protected from changes checked in while it’s running to make this possible.
4. Central Objective Metrics: With a central build who’s status is published to everyone, the entire team knows if the system currently builds or doesn’t. It establishes a central point to look at to get a quick understanding of where the team is. After all, your customers don’t care if your personal code compiles, they’re buying a whole solution which requires a full build work.
5. Workstation Decoupling: With a central build process, it isn’t necessary for any individual workstation to be capable of building the entire system, they just need to be able to build their part of it. This can be very handy in several circumstances including reduced licensing costs for third party components by segmenting developers by subsystem and reducing the overhead of upgrading referenced components because individual workstation inconsistencies aren’t an issue unless the developer’s run into a defect in the course of the work they’re performing. This is particularly handy during maintenance when a developer has moved on to a new project.

Not all of these benefits will apply to every operating system or application type. In fact, it seems this discussion is most frequently had in Windows development teams. This may well be an artifact of the tendency for Windows environments to be more distributed than Unix environments with centralized builds being more the rule than the exception on Unix.

Best Practice: Continuous Integration

When you’ve created a fully automated build and then centralized it, you’re dangerously close to a best practice called Continuous Integration. Continuous Integration builds on the concept that if the entire build is automated, and it can be done on a central server, then each build is free. The only potential cost of a build is the risk that you’ll make a change while the build is running and want to start another build of the same project. If you’re coming from a world of expensive and time consuming builds, this is a revelation.

1. If builds are free, do a new build every time someone checks in a change. Now you’ll know right away if that change breaks something, and you’ll know that a build reflects the latest code changes.
2. If builds are free, let anyone trigger a build. If someone just isn’t sure if a change made it in, or you made a change to the build server, or you just want to see an updated test run, or even just want to make the lights on the server blink faster – do a build. No harm, no cost.
3. If builds are free, anything the build does is free. Seems obvious, but follow me on this one: Anything you can teach the build to do, it’s free. Why not have it run some automated tests on the software? What about uninstalling the old version from your test environment and installing the new one? How about some performance tests? Notify the development team of new builds? Move defects that are claimed resolved in the new build to the next workflow state so they get tested?

Automated Testing

We’ve found it always an uphill battle to introduce automated testing into development teams that aren’t currently doing it. Automated testing is often viewed as a post-developer action that automates what manual testers would otherwise perform but we believe this misses the point. Usually the highest benefits from automation don’t come from slavishly automating the exact process that is done manually but instead looking at what you could do if the whole process was automated. For example, no human is going to completely regression test an application for a minor change, it is simply too time consuming. It may not be feasible to perform an accurate regression test purely using the public, published surface of the product either.

Instead, look at introducing automated tests into the development process itself. Don’t get hung up on whether they are automated unit tests or system tests or just make you feel happy tests. We’re big fans of using NUnit/JUnit to get developers to start writing good, idempotent tests in parallel with their code. You can even use these when your software isn’t necessarily written in either .NET or Java, provided you can call it from .NET or Java. Most automated build systems can intrinsically talk directly to NUnit/JUnit, and if not they can call a command line to invoke the test execution capability distributed with NUnit/JUnit. This means the developers get the benefit when they are writing the code that they can prove it correct, and you get the benefit of each of these tests being run every time the system is built.

Even if you don’t get into automated unit testing, consider writing just a simple script or executable that will smoke test the application. Then, have your build process distribute the fresh package to a test system, install it, and run the smoke test application. This should be part of the success/fail criteria of the build. This will give you a base level of confidence in the entire process nose to tail, and will encourage developers to add things to the test process because they know they’ll get benefit every time the build is run.

Coming Next: Great Results through Incremental Investment

In the final article of our series we’ll look at how to create an automated build incrementally and make it a natural evolutionary process of your team, providing both immediate and ongoing value. Really, it’s no joke and it’s not going to require upending your technology or team.

Editor’s Note: This is the first article in a three article series, with a new article posted every few days.

If you haven’t experienced the difference an automated software build system can make to your entire approach to development, this article series will show you why it’s worth your time and how to get it done. Before we launch into the nuts and bolts of setting up a build automation system, lets step back and establish some common ground.

What’s A Build?

A build is the process that takes your source code and translates it into an installable product. There are some definitions that merely look at the first part (building executable files), but I prefer to look at things from a results standpoint: A process should achieve an external result, and the external result of building software is that you have a package that can be distributed and installed by users.

The critical goal is to ensure traceability from product back to the source code that created it:

1. A given version of your product must represent a unique build so you know that there’s just one “1.1.1452″ version of your product in existence.
2. Each binary file (.dll, .exe, .jar, etc.) needs to have a unique version number to ensure that there is just one “1.1.1452″ version of “MyCoolApp.exe” so you can look up the source code by that version number.
3. The source code for each binary must be labeled with the version number so you know what source code made that version.

The same rules apply to non-compiled code as well, you just tend to treat them at a higher level (e.g. a whole set of PHP files as a group instead of each individual file).

To achieve these goals, I’ve always used a few simple rules:

1. Every exchange loops through the source code control system: From computer to computer or process to process, do it by checking the output into the source code control system and getting it from there on the other side. This ensures you have a way of seeing the output of each stage.
2. Only builds leave development: When you are going to bridge from your raw development environment to any other environment – test, certification, whatever – it’s done through a full build that has its own unique tracking number. Even if you just made another build 10 minutes ago.

These rules eliminate the possibility of transient work products (e.g. binaries) getting anywhere without the tracking to back up where they came from. They also ensure that any developer that has pack rat tendencies (and most do) will have to push things from their box to the source code control system, which should be on a nice safe server that’s backed up.

Sidebar: Seriously. Your source code control system is virtually irreplaceable. It should live on server-grade hardware fed nice clean power with a UPS and regular backups. The system you select should have a strong track record of never corrupting data and you should be comfortable that your backups of it are top flight. I recommend a product that stores into a commercial-grade database because the data is just that important.

What’s In Your Build Process?

At a high level the process to achieve this traceability is going to look something like this:

1. Get the code for each project that needs to be compiled.
2. Update the version information so you get a unique version of the compiled files
3. Compile them.
4. Label the source code you compiled with the version number.
5. Package the binary files with everything else needed for the product into a distribution format.
6. Store that distribution in a central location with a version number or name indicating what version it is.

That feels very simple and straightforward, doesn’t it – just six steps. When you look closer, you’ll notice there are a lot of loops: You have to get, label, and compile the source code for every project that needs to be built. Often, these projects have to be built in a specific order to work correctly. It may not even be obvious until the code is smoke tested if they were built out of order and won’t run together as a group. You also need to do this with absolute confidence in the integrity of the process so when you find a problem on a computer and it appears to be running version “1.1.1452″ you have confidence on exactly what that means, all the way back to the source code.

Pretty much every development environment includes some form of build automation. In the old days it was “Make”. In Visual Studio it’s now MSBuild. For the most part, these tools are competent at performing the basic steps necessary to take source code and produce binaries, but they aren’t generally going to handle the other elements like labeling source code, checking in outputs, and copying the final distribution to a central location. If they can be extended to do that, it’s usually fairly high effort, and can easily get in the way of the routine work your developers need to do local builds on their development systems.

But Wait, There’s More

This is a very simplistic view of what a build looks like because it leaves out a critical step: The smoke test. It really can’t be called a build if it can’t be installed and at least fire up without laying over and dying. It’d also be nice to pull together release notes including the defects that were fixed or new features added in this build. Finally, lets notify the team that a new build exists so they can pick up where the build leaves off.

You Don’t Need an Automated Build

You can do all of this by hand indefinitely. After all, if you document the process it should be possible for a professional to correctly execute the build by hand every time, following each step.

There are three key problems with this approach:

1. Humans are fallible: A well trained professional doing an intricate task will still make a mistake around two percent of the time. That’s one in 50 opportunities: They’ll put the wrong version number on something, label the same folder twice and one not at all, not clean out the working directory first, something.
2. The potential for mistake degrades value: Because a main point of the build process is to have confidence that you can absolutely go from distribution package back to every element of source code it maintains, even the possibility that there was an error in how the build process was executed will make you doubt its integrity and therefore you won’t achieve the value you wanted.
3. It’s wasteful: Each build occupies a well trained professional’s time. If you need to do a new build at 2:00AM, you need a well trained professional to execute a possibly lengthy process accurately. This costs you resources and even worse it’s not a job any developer likes, so it costs morale.

Over time, the fact that each build is a risk and a waste will tend to unconsciously affect the decision making of the development team, making them more likely to defer a fix or change they might be able to code and unit test on their own computer but don’t feel is worth the overhead of the build.

Traditional Resistance

There are a number of reasons that are typically put forward against having a central, automated build process. The most common ones I hear are:

1. It will slow down testing and certification: Since each build that is going to be tested outside of a developer’s machine has to come from the build system, that means that even a small error found in certification will require the entire build be run before it can be tested. Why not let a developer just recompile the offending file and slip it onto the cert system to verify it?
2. It takes extra resources: Having an engineer set up and maintain the build process takes time away from development, which means my customers will get fewer features, etc.
3. It slows down change: Every time we want to add a new binary file or a dependency we will have to update the build system and possibly the build process and retest it. This will get in the way of an individual developer being able to get things done as fast as possible.
4. Single point of failure: What if the build computer fails? If it’s the only place to do a build, we’re stopped.

These objections generally spring from a few underlying problems within the development team: Developers that lack confidence and fear of change.

Developers Playing Hide the Ball

If there is a developer on your team that isn’t up to the rest of your team’s level and they’re trying to hide it, this is virtually guaranteed to bring it to everyone’s attention. They won’t be able to just slip a new file into the build or slip a fix into test without it being clear what happened.

If the time it takes to perform a build – whatever that is – is an impediment to certifying your software because you need to fix problems faster than that time, you have a more fundamental issue: Your developers are not thinking through their code before it’s included in the build. Fundamentally, it’s called Certification, not Debugging for a reason: Developers should be genuinely surprised that their code doesn’t work as expected when it leaves their hands.

If this is the case, then when a problem makes it to test it shouldn’t matter if the build takes 30 minutes or even two hours. Any development process that needs to go from the developer’s fingertips to certification in less than that time has more fundamental quality control and process issues.

If you have developers concerned that this slows down their ability to add new projects or dependencies because they have to think through how to update the build system this is really a good thing: These decisions matter by the time you want to ship a product to customers, so the earlier you can address them the lower the probability you’ll discover in certification that redistributing a particular dependency is hard or being done wrong.

Fear Of The Unknown

Most developers are not IT administrators, and all developers are humans. Human beings fundamentally don’t like change. They will actively fight change, often with very good prose. Giving up control from being able to do a local compile and take the binaries that work on their box to a central system that is opaque is uncomfortable. The very same developer that’s perfectly willing to switch to Visual Studio 2008 the second it was posted to MSDN and downloads the latest nightly build of NHibernate will come up with all sorts of creative reasons against a central, automated build because of their fear of change.

If you are following reasonable source code control rules, you really don’t need to worry about backing up an individual developer’s system: There shouldn’t be much that’s on it uniquely if it were to be lost, preferably at most a day’s work (which is within the time frame of a backup/restore loss anyway). The build system is special: As part of making it the central authority of building your distribution, it really is inconvenient to have to recreate it from scratch through reinstalling all of the software components, etc. It is likely to be slightly different than your developer’s computers (server grade hardware vs. desktops) so your normal developer image won’t work on it. Back it up as part of your normal production server backup scheme, and invest in redundant disks so it’s unlikely you’ll need those backups. This will tend to give you better build performance anyway, so it’s a double benefit.

Coming Next: Benefits of Automation and Centralization

Check back for the second article in this series focusing on the benefits for your team of automating the build process and centralizing it, including the roles and capabilities of an automated build system. From there the series will continue with how to create an automated build incrementally and make it a natural evolutionary process of your team.

Consider this simple point:

Effort ≠ Value

Think about it for a few minutes and it seems patently obvious: Just because something’s difficult doesn’t mean it has great value. For example, if I want to mail 50 letters to clients and I put an individual stamp on each one instead of using an automatic postage machine I’ve achieved the same value: I can now send these letters to each of my customers. They’ll get there just as fast, the postage is just as valid. Therefore, if it takes me 15 minutes to put the stamps on one by one vs. about 1 minute to run it through a machine I’ve spent 15 times as much effort to achieve the same value.

It works in reverse as well: Just because something has great value doesn’t mean it’s intrinsically difficult. It may be exceedingly valuable to me to get a message to a client that lives on the other side of the country, and yet it’s really easy to do: In just a few seconds with my cell phone I can reach out any time of the day, from virtually anywhere. Low effort, high value.

Obvious, and yet we ignore the implications of this every day. We naturally assume that anything worthwhile takes effort, and that anything that takes a lot of effort was worthwhile.

Good examples of low effort, high value

Cosmetic defects are a classic example of this. It isn’t unusual at all to go through a new software application and find a substantial number of cosmetic defects: Alignment issues, inconsistencies in language (is it login, logon, or user id? Do you click or press that button?), spelling or language errors and a range of items that aren’t application behavioral issues (like tab order). Developers tend to instinctively minimize these issues: They’re trivial to resolve and they don’t prevent the application from working. They aren’t anywhere near whatever hideously complicated part of the system the developer is really worried about, and they’ll take no time to get right later. They can’t be that important, so development teams tend to not talk about them or work them. Even the term “cosmetic defect” is often used as a label for trivial or low value: “that’s no big deal, it’s just a cosmetic issue. Now let’s talk about that rare crash on every other leap year if you attempt to delete a customer with no records!”. This perspective isn’t even particularly unreasonable if you’re looking at the development process from a risk management perspective: You know the issues can be cleaned up quickly and without a lot of technical risk, and if you clean them all up now you’ll still have to do a recheck of the system before release because new ones will show up.

Now look at it from the standpoint of an end user of the system. The system is a black box: They don’t see the really artful code that figures out automatically when they enter a name as Last, First or First Last or how you managed to make a really fast look-ahead search system despite the large number of records you have to work with. Instead, they’ll see what’s right in front of them: The user experience of the application itself. If they start it up and notice immediately that things aren’t lined up vertically & horizontally or there are spelling errors it will bring rise to the classic line of reasoning that if you didn’t get this right, what hope is there that the black box is right? The more you protest that this is easy to fix the worse it gets: If you couldn’t get the easy to fix simple stuff right then there’s now no way the detailed 12 step process for determining how much to bill a client is going to be right, and the user is going to have to check it all before you regain their trust.

The good news is that this direction is the easiest to avoid as a manager or team member of a software development project. Once you’ve had the above experience once or twice, you will start to get wise and do a cosmetic issue pass at strategic points in the time line – usually just a few builds before it’s going to be seen by people outside of the team. You’ll be surprised at both how many items show up each time, and how easily they clean up. Then, while you’re sweating during the big demo about whether you’re going to get a runtime error you’ll at least have the comfort that what they are seeing while they’re waiting for the next page represents the good work your team did in a way that communicates to the average user that can’t see behind the curtain.

Good examples of high effort, low value

This trap is more dangerous and harder to avoid. At many points through the development process you’ll have opportunities to chose architectures, designs, algorithms and other items that will either increase or decrease the effort it’ll take to complete the project. You might chose to not use that built-in dialog to open files and instead make your own dialog because of one annoying behavior you really want to avoid. Or decide that you want to make a better column sizing routine for the grids you display so that you can avoid either trying to cram too much on a small screen or having acres of empty space on a large one. None of these are on their own bad ideas necessarily, and that’s part of the trap: Most development processes by design tend to focus team attention on the things that are hard, high risk, or just time consuming because these have the biggest ROI for project management activities. This reinforces our built in instincts to presume that the harder the work, the greater the value.

What this ignores is that the value is essentially constant regardless of effort: Any particular feature or capability has a set value in the eyes of the user. Our goal is to realize that value with if not the minimum effort then something that appears (prior to construction) to be the minimum effort that has an acceptable risk. In its most direct form, this means that the user places the same value on a five thousand line algorithm to determine optimal column width and using a method built into a control to get it right, as long as the outcome achieves their expectation.

<tip>Corollary: Be sure what’s important to you is also important to your users before investing a lot of time. Perhaps they don’t care if there’s a bunch of empty space on their 24″ widescreen monitor as much as you do. Get evidence commensurate to the effort you think it’ll take to resolve the issue.</tip>

Understand the trap

This issue tends to manifest itself in some classic ways. One is when a developer argues passionately in favor of a complicated algorithm even in the face of peer review that casts substantial doubt on its necessity. Typically the developer caught one small aspect of the problem and has ruthlessly optimized for it, and uses that one point as the proof of why simpler approaches don’t work (”If users are constantly switching back and forth between these two displays it’s 30% faster to do it this way than what’s built in”). These items also tend to be defect prone and difficult to explain to others.

Complicating this trap are a few factors:

• There are hard problems to solve: You can’t assume every hard problem is really an overcomplicated solution. Most applications will have at least two places where there is some real trickery and engineering to get the right result each and every time. If there weren’t, your users probably wouldn’t want the application in the first place.
• There are low value problems to solve: There are hard problems that have to be solved, and some of these are even relatively low value to the customer but are still a requirement. Consider this example: The customer places relatively low value on your application not crashing when they run it. Don’t get me wrong – if it starts crashing they will be very upset, but they simply assume that it won’t crash. All joking about Microsoft aside, any application you write is virtually guaranteed to be more crash happy than Microsoft Office is. So you’re going to end up investing a lot in something users don’t really place a lot of incremental value on.
• Developers are Optimists: Developers like hard problems (after all – hard problems are valuable problems according to our instincts) and want to solve them. They will underestimate the effort going in and overstate the value of the journey. If it’s a new problem, it’s unlikely that their estimate is particularly great even if they aren’t focused on why a particular complicated approach is necessary.

Striking the Balance

How to avoid this trap? First, For these problems to get out of hand it usually requires the ability for one or two developers to go off away from the herd for long enough to cook up a complicated idea, justify the effort to themselves, and then get far enough into the swamp to be in real trouble. Depending on your specific software development approach, find ways to catch the telltale signs before developers sink enough effort into the solution to get permanently attached to it.

Second, be fully prepared to throw out an already developed solution regardless of how much code or effort it is. In other words, the decision on whether or not to back up and take another approach should be largely blind to how many lines are being thrown out as long as they are all part of the same solution. Even at this point the instinctive desire to equate effort and value will creep into the entire team’s thinking: people will look at the large block of code and assume that it must be necessary, we’re just missing the subtlety of why it is the light & the way. This is what source code control is for (you do use source code control, don’t you?). It enables you to with no fear reject a bird in hand for a simpler bird that may converge faster, have fewer issues, and ultimately be a more cost effective way of providing the value your customers are expecting. Remember that even if you’ve taken a complicated implementation through initial unit testing, there is still a substantial investment that will be made in that code over time.

A production implementation is worth many theories

This article has been talking about high effort ways of achieving value in software, and the approach shouldn’t be generalized into applying to applications simply because they are large or complicated, or even any particular solution that is large and complicated – provided it got there incrementally over time. While it’s often tempting to look at a few hundred line block of code that does just one thing and think that in this age of objects, partial template classes, interfaces, and reflection there just has to be a cute, simple implementation that’s less than half the size and complexity of the current solution there are two key issues with this thinking:

1. Large, stable code already achieved value: If the block of code is substantially stable and accepted by the customers then it has achieved its value and any effort spent on refactoring it that doesn’t also deliver more value to customers isn’t improving the value of the application
2. Refactoring introduces defects: It’s virtually guaranteed that in the process of refactoring the existing routine sufficiently to make a good dent in its complexity you’re going to introduce some new defects just due to conceptual or implementation oversight during the process. It’s generally not considered a great justification to management that you introduced defects that then require expense to clean up in an effort to avoid possible future expense maintaining code.
3. Code gets larger because it handles very subtle points: If the code organically grew over time to be a complicated routine, it probably did so because it was progressively asked to handle a number of interesting boundary cases that experience with the application proved necessary. In the minds of the users, these subtle behaviors can be some of the greatest value they place on your application – and yet they’ll never mention the feature when talking with you about it. Therefore, when you refactor it you have to preserve every little behavior, and that is often infeasible (with the exception of true defects in design of the original code – well isolated routines that can be replaced with provably equivalent code)

If it’s already in production and doesn’t have a critical flaw that the business needs addressed, leave it alone.

Clearly communicate the end-user value within your team

The best technique to avoid this problem is to make sure your development team has a tradition of discussing the end-user value of the work being done. This tradition would mean that anyone gets to clarify what the end user value of any work that’s being done is – that’s a free question never met with ridicule. It may take some practice to make sure the question is asked correctly, e.g. it’s asked in a way that gets the entire team to back up and be clear about why the work is important. Within that questioning, its important to make sure the discussion is based on what’s important to the users of the system, not the developers or other non-users. There are times to focus on maintenance or other non-user issues but with rare exception it isn’t the reason you’re writing the system.

With some practice, this will become a strong self-regulation mechanism for the team, ensuring that your discussions about design and approach are grounded in the needs of your customer. It creates a good mental yardstick for how much time to invest in a solution before going back to the customer to re-verify the requirement.

What’s your experience?

Have a good story to share? Have a critique? Post your comments or drop me a line to continue the conversation.