Reliable Systems

Let’s say you’re starting a project to create a new software system. How big does it need to scale? Realistically, either:

1. This new system fits into an existing business, possibly replacing a prior application, so you can predict with some accuracy the different aspects of scalability that apply to it.
2. It doesn’t, and you can’t.

The second scenario is the most interesting one. First off, let’s face it – your new system isn’t going to be the next Facebook, MySpace, or eBay. In short, you don’t need to worry about having your system needing to be designed front to back as a super-scalable system. This is good because the options at that level are time consuming and resource intensive.

The key question you need to understand when laying out a new software system is to what degree it needs to scale without being re-written? This scale is unlikely to be your “best case” business size, because scalability has opportunity cost. This scale should be defined as specifically as reasonable, and clearly understood and validated by both business and technical staff. This ensures that if your business grows beyond expectations that it won’t come as a surprise if you need to make even major changes to your system.

Creating facts from Air

Let’s say you’re starting to develop an application that fits into the second category above. You still need to work out what your scalability target is.

To make any decision that is better than random, you have to work out some aspects of the expected scaling of the application. In the absence of real facts to extrapolate scalability from, you need to cooperate with the business side to established presumed facts of the scalability requirements. This may sound a lot like assumptions, but they really go beyond that because these will become facts as you develop the system. As a starting point, make it clear to all involved that:

1. If the targets are low, it should be assumed you’ll have to turn away business because the system can’t scale above them.
2. If the targets are high, the system will cost more and take longer to create.

In most businesses, the second outcome is worse than the first. Why? Because the second is a price you pay up front, before the system goes into service. The first is based on an assumption: you might have to turn away business. You also might be able to realize it in time and address the issue. From a business standpoint, this is a better trade off. Finally, there’s the non-technical aspects:

1. The sooner you have a working system, the sooner the business can validate the market and start getting real data on uptake to adjust your scalability goals
2. Unless the product is a failure, you expect demand to eventually exceed the capacity of the system, it’s just a matter of when. If it does, then you should be able to afford rewriting all or part of the system. In other words, the funds to solve the problem should be available if you have the problem.

From this comes an axiom of scalability:

The system needs to be based on the lowest scale that will provide enough time and money to replace it with a new system.

Put another way, a system that is faster or more scalable than it needs to be for the business was more expensive and took longer to develop than necessary. Think of it like a race car: The ideal Indy Car would fall apart just after the judges validated it won without breaking the rules. Any stronger and that strength could have been put into something else. The time you spent making it more scalable than necessary could have added more features, fixed more defects, or gotten it out the door sooner.

Establish a Growth Curve

The growth curve needs to be sufficient to inform the developers of what decisions to make at each point. To get there, start with describing the scale from the business stand point. During design of the actual system you can keep translating this into the specific requirements for speed, storage, and capacity based on the behavior of the actual system. This will prevent you from achieving technical goals that don’t satisfy the business goals.

For most systems, you want to establish the business goals for:

1. Number of Possible Users: How many accounts will there be on the system? This is an upper bound of the number of people that could access the system if they wanted to.
2. Number of Simultaneous Users: Number of accounts that will be accessing the system at the same time. For most applications, at the same time is likely best thought of as in the same 15-30 minutes.
3. Number of Customers: For most applications delivered to businesses the number of customers (e.g. businesses) drives the scalability of some parts of the system (such as configuration and data storage) will scale based on the number of customers, not the number of accounts those customers have.
4. Data In and Out: If the system is going to have any imports and exports that aren’t user-driven (such as EDI feeds or a public API) then the number of partners (other entities that will exchange information with you) and the frequency of exchange need to be determined.

Things to not bother with:

1. Response Time: For customer interactive products, response time is dictated by what end users will tolerate and is not really going to be a business decision (aside from deciding if you’re going to produce something your customers are willing to use). For non-interactive products or back-end this may need more discussion with the business, but again – the business is going to expect you to be able to figure out what will make it a success.
2. Data Retention: Assume it all has to be kept and more indefinitely. In the end, storage is cheap and this design decision rarely costs a lot of made up front but is expensive to reverse. Data also has the amazing power to make heroes out of IT when the business starts posing questions later and you can answer them. Generate as many facts as you can now to help you out later.

These items are past the point of diminishing returns with the business. You should work them out within the development team and document them, but you shouldn’t believe that any business sign off you might get is binding or useful.

Build to the Scale

Once you’ve established your growth curves, pick your candidate architecture and translate the growth curves into system performance requirements.

Hypothetical Example: If you need to support 1000 simultaneous users for a web application, determine the dynamic web hits per second by determining how often an average user will request a dynamic page (say ever 5 seconds, which is very fast for most dynamic applications) These two numbers would give you a dynamic hits per second of (1000/5) = 200. Then add how long each page will take to calculate (make a goal of say 250ms) to get how many requests you need to be able to process at the same time: (200 * 0.250) = 50. This is the key scale point for your web application: When deployed, it must support 50 requests being processed in parallel. You’ll need to get to this point by either making it really scalable on a single server, or splitting the load over multiple servers.

One thing that should jump out of the math behind this is that anything you can do to make the calculation time of a single page drop pays big dividends: If you drop the average calculation time by half (125ms) then the number of requests in parallel drops by half (200*0.125) = 25. This in turn may well cut the number of servers you need in half, easing your maintenance and deployment cost. If you can’t do this, reduce the number of dynamic pages requested per second by either making more static pages (such as pre-rendering pages that change but don’t change frequently) or caching dynamic pages that have some predictable consistency (which really makes them static pages). This is often much trickier to do and test, so your best first option is to reduce the time for each page.

Side Point: This also highlights an easy way to accommodate guessing low on a system that’s been in service for a year or more: If you’re processor bound you can replace that hardware with current units and often pick up 30% per year it’s been since you purchased the original hardware. This won’t save you from network problems, disk storage problems, or some memory problems, but it is surprisingly handy.

As you look at each candidate architecture, look at each component and determine the critical “how much, how fast, how often” factors based on the business inputs. If you change your architecture or external interface design (the user interface or import/export capabilities) you need to re-evaluate if you’ve moved the targets as well because your design goals no longer reflect the business growth curves.

Really, to the Scale

Within your development team you will typically have two types of developers you need to watch: Those that never consider scale and those that obsessively consider scale. The former will build it however and then wait to see if there is a performance problem. The latter will try to make every system the next Amazon. Neither situation is good. Identify early people’s tendencies and work to manage them to the center. Remember that the system is only as scalable as its slowest part, and there is always a slowest part.

You can get good results by having the people that are most concerned about scalability move around on the project to different subsystems. This will tend to keep them too busy to earn the keeper of the nanosecond award on any one system (which they will do if you let them stay put and just work on one system) and will make it unlikely that more cavalier developers can hide a problem. It will also help the team learn from each other: It often isn’t worth making a specific feature as fast as possible, and it is always worth thinking about what will make a feature fast before coding it.

Finally, budget time in the development team to fix scalability issues. Regardless of how much work you put into it, once the real system is build and tested you’ll find places that are slower and less scalable than you expected. If nothing else, you need to develop an accurate model of how the system should perform in production so you can check the real world against it later. As your business grows, you need to be able to get ahead of it and understand when it is time to make the code faster, add hardware, or do something else to stay one step ahead.

Disk is Your Friend, but Beware the Network

If you’ve gone over the system from nose to tail and you’re disk bound, you’ve probably optimized that design as well as you can. Disk has gotten faster at a much slower pace than memory or processor, and being disk bound means you’re getting all the requests where they need to go in a timely manner and are able to process the inputs and outputs, so now it’s in the hands of the hardware. Unfortunately at that point there generally isn’t much more you can do: The difference in performance between server drives and the fastest drives money can buy isn’t very much.

If you’re finding that you aren’t disk bound and you aren’t processor bound then be worried. You’re either network throughput bound or you’re network latency bound. If you’re network throughput bound, you can probably fix it cost effectively with some basic engineering either in how you select what to send across the network or what you cache so you don’t need to send it across. You should try to give yourself some headroom here for growth, but faster networks can be purchased and you can generally tweak the software to mitigate this in minor updates.

Being network latency bound is a more serious issue because it often means that you are at the practical scalability limit of your application. The difference in network latency between relatively cheap hardware and the best hardware isn’t very much, and has been essentially constant for the last 10 years. You can’t buy your way out of this problem. It also is typically caused by a badly designed interface between components of the system which will need to be substantially or entirely rethought and rebuilt to address, which isn’t easy to do with a running system. If you find yourself in this situation and you aren’t sure you have met your business goals you should rethink your approach immediately. Because no amount of money on hardware can get you out of this problem, caution is the word of the day.

In software development, you’re always being asked to estimate things:  How long will the whole project take?  Just this feature?  What if we changed this feature to remove this aspect?   This is all part of the feedback cycle that is fundamental to product creation:  We have a certain amount of time and money for a given set of functionality, but if there’s something really juicy and it just takes a little more time, then maybe we adjust, or if we can get a lot of value for a little effort, perhaps we do a little mini release first.  The business decisions feed the development estimates which in turn inform new business opportunities.

Getting the feedback wrong can be disastrous: The right functionality late can kill your business; perhaps half a loaf earlier is better.  On the other hand, some things the market won’t accept half way so a full loaf it is.  The business needs to trust the information it’s getting isn’t random or capricious to make good decisions, and the development team needs to be able to provide a best guess without fear of misunderstanding.

It’s natural to pad estimates with the idea that it’s better to under promise and over deliver – so that means to guess long and come in well early.  But in the end, is that really any better?

You’re Guessing, and Possibly Lucky

We’ve been experimenting with the new Evidence Based Scheduling features of FogBugz (which we use internally for managing our software development) and one thing that it highlights quickly is that estimation isn’t good if you’re early, bad if you’re late – it’s about getting your average as close to the mark as you can.  Take a look at a graph of most of my estimates:

Ideally, the graph line would have a 1:1 slope, indicating that on average you are accurate.  Further, you want your estimates clustered pretty near to the line itself.  What you can see from my estimate curve is that I’m uneven – I tend to underestimate shorter tasks (under 1.5 hours) and overestimate longer tasks (and that’s after removing some really bad outliers…).  But notably if you draw a ruler on the 1:1 line you’ll see that I’m not even close. Don’t let the hash lines fool you – look at the numbers to see.  The thing is, the other developers in our company that are all regarded as skilled, senior developers aren’t particularly more accurate on any one estimate, and the averages work out similarly.

So what?

Why isn’t it a great thing when we beat our estimates?  There are several potential pitfalls:

Features based on Effort

We’d all like to believe in the rosy model that our customers ask for features and then we build them into the software, so if we’re done early then it’s a pure win.  It’s my experience that it’s much more like a game of Tetris:  What features we take on is dependent on how much effort we think they’ll take.  Every feature has an amount of effort above which it isn’t worth it any more.  When hashing out what makes it and what doesn’t, the effort estimate is a big factor.

If we overestimate the effort of features, then we are slanting the project management decisions away from customer-selected features in favor of the developer’s whims.  This is because some features will be estimated to take more effort than they’re worth, and a more invisible internal team dynamic.  If a project is doing well on schedule, it’s very human to take advantage of this to try out newer, riskier things, over-engineer a feature, or do other things within the team that would otherwise be successfully argued against because of their effort.  In general, the more time available, the more yak shaving the team will do.

Once a schedule is accepted, the business will tend to act on it as fact:  Customers that can’t wait for it will end up being turned away and others will be promised a schedule.  This is a necessary but painful aspect:  Developers are generally optimists and will not want to say no to a customer even though it’s generally not in the best interest of either the company or customer to rush a feature. You want the business to stick with your decisions and not pass the buck on saying no to the customer, but you also need them to trust that it’s a fair trade.

Approach based on Effort

Within the development team, decisions are made at every level on how to implement a feature with an eye towards both the feature’s estimate and the overall project’s status.  Even when not explicitly laid out, a team that believes the overall schedule is tight will feel pressured to find ways to reduce the effort on anything they do.  This means when deciding between a careful implementation that may take longer but be more scalable or easier to support the team will often opt for a more direct path to completion even if the estimate was based on the more careful approach.

If done as a conscious decision in consultation with the project’s sponsors, this can be a way of bringing the project back on track but really it’s just another way of cutting functionality to make schedule:  You’re going to cut out something you intended to deliver (say a more generalized, upgradeable framework for reporting) even though you meet the letter of the requirement in front of you (delivering a few reports).  This can lead to nasty surprises for the team down the road when your sponsor’s find out that they didn’t get what they thought they would.

Alternately, if you overestimate one feature it may have put another feature under pressure so a more expedient and risky approach was adopted for it to fit it into the schedule.  If the true effort had been known, a different decision could have been made.

Make Your Guesses A Coin Toss

In the end, being early and being late just have different ways they create problems for your development project.  Your goal when estimating is to not try to find the estimate that has the highest probability of being sufficient to get the job done but instead the estimate that is equally likely to be high as low.  In aggregate if you have enough of these items on your project (say more than 25) then you’re entire project’s estimate should also be just as likely high as low.

There is still a place for the traditional high estimate: When you move outside of the project sponsor and the development team to users that need a guarantee.  There the downside of missing a date is much worse than the impact of being early.

On your next project, try out the 50/50 approach and make it clear to both the development team and the business.  You’ll probably notice that people develop a more subtle appreciation for the fact that estimates are based on probabilities.  This can help you skip over the discussions that aren’t useful about why you are where you are and instead keep focusing on the business goals for your current situation.

Have you had the experience of looking at a situation and knowing – just knowing, that something was wrong? Perhaps it was a user interface design or a software diagram or project schedule. I’m not talking about a dispassionate concern but an emotional response – you recoiled inside and just knew. Then you had to explain to the person or team that presented the situation why this was the case. Most likely you can draw on your experience and come up with a few very convincing explanations for your gut reaction, but usually you’ll walk away unsettled. It still isn’t right, but you just couldn’t put your finger on it.

Most professionals develop an instinctive ability to size up situations within their core expertise. For example, a seasoned product manager that has worked up through the ranks of developers can often look at a schedule and get a quick feel that it can or can’t be met. Most of the time this intuition expresses itself as a strong gut reaction that you can’t explain. Malcolm Gladwell wrote the excellent book Blink talking about this phenomenon, well worth reading so that you know what to do when you have this reaction the next time.

The basic challenge is this: Even if your instinctive reaction is correct, it isn’t particularly useful until you can explain why. It’s very tempting to try to verbalize the rationalization for your reaction, but don’t. The problem is that your rational mind has no idea why you had an emotional reaction. The emotional reaction is your clue – your intellect and emotions aren’t really connected that well. They need to arrive at their conclusions independently: For you to intellectually justify your reaction you need time to perform a dispassionate intellectual process. Your instinctive emotional reaction didn’t need that time, but it can’t explain itself.

The Path Through The Woods

So if your emotional reaction is equally as reliable as your intellectual evaluation, but you can’t articulate it right away what should you do? The first thing is to develop a code phase for your team that indicates that this is your blink reaction, and not something else. This lets everyone weigh it correctly: It isn’t because you just don’t want to do it or you like another option better, it’s that there’s something instinctively wrong with it. Your team should give it equal weight to you having just expressed a cogent, real argument for there being a problem. Second, the reaction can’t be challenged – at least not directly. It’s an emotional response, so any challenge will drive the participants strait into conflict from collaboration. In our shop, we just say “I have a blink about this…” That cues everyone in.

At the same time, it’s important to recognize that your blink can and will be wrong, sometimes a great deal. It’s no more accurate than a rational analysis of the same circumstance, and that means if the circumstance is something notoriously hard to predict like an election, a project schedule, or a roll of dice then it’s not going to do better than spending some quality time in analysis. This means that when you have a blink reaction then your team should continue with the assumption that the reaction is dead on, but casually seek out the proof.

Casually Find The Evidence

Once you’ve articulated your instinctive blink reaction, have the team take the case that it’s true and then as discussions continue identify facts and data that support the reaction. Eventually, one of a few things will happen:

1. Killer supporting evidence will emerge: In the process of going through subsequent analysis, you’ll find the evidence that suddenly has the reason behind your reaction clear to the team. It’s now an intellectually reasoned response; you just got there faster instinctively.
2. You’ll realize your reaction was wrong: As time passes, your mind will keep attempting to align facts with your reaction seeking to prove or disprove it. Eventually you, or your team, will see what it was that your mind originally caught on and understand that it doesn’t apply in this case: The reaction was wrong, and now you can proceed. This was still an important exercise because you now have validation on your course of action (”we can ignore this previous best practice because it no longer applies because….”)
3. You’ll look at the problem tomorrow and get over it: Perhaps what you felt was just the normal human fear of change. So be it- now that you aren’t feeling threatened any more, your rational mind can reassert itself and look at the item more objectively and be open to new possibilities.

Each of these is a powerful collaboration result because it lets the team and the individual practice and demonstrate the characteristics of a trusting, supporting environment. The team showed the respect for the individual participant and leveraged the experience of everyone, not just those with great oratorical skills. The individual gets to articulate something they feel very deeply without being embarrassed or having to justify and then later defend something that they just can’t. Everyone is in on it – and when the instinct resolves into intellect later everyone will have better insight into the experience and thinking of the person that voiced it. This insight develops trust between the individuals that will live beyond the team.

Finally, by the team not pushing for an immediate defense and then challenging it demonstrates and reinforces that it’s a safe environment to voice ideas and opinions, and builds credibility for when a miss-step happens.

Gaining Speed

Over time as your team works together people may be able to help each other articulate their blink reactions into concrete issues to be resolved based on understanding the key emotional drivers each participant brings to the table. In our company, we have some folks that tend to have reactions over usability, others over ultimate performance (we’ve dubbed one “the keeper of the nanosecond”), and others over code simplicity. Knowing these points helps us not miss-read emotional reactions to ideas and to help each other understand what we really need to do to create the best outcome.

With practice, you’ll find your team can get to great, collaborative conclusions faster and generate the buy-in from the participants with little or no effort.

What Gets You to Blink?

Looking back, when did you have a blink reaction to something? What did you do with it? Share your story by posting a comment or dropping me a line.

Editor’s Note: This is the final article in a three article series. For a list of the entire series, see the Article Series page.

In the final article of our series we’ll look at how to create an automated build incrementally and make it a natural evolutionary process of your team, providing both immediate and ongoing value. Really, it’s no joke and it’s not going to require upending your technology or team.

Components of Build Automation

When introducing an automated build, we recommend pursing components in the following order:

1. Compile files from Source: Retrieve all of the input data from the configuration management system, and compile everything that needs to be compiled. If you’re working in a language or technology that doesn’t require compilation, just retrieve and label all of the input files. Compiling may include activities such as automatically assembling release notes from the defect tracking system or any other action necessary to create a file that is distributed. Check compiled results back into the configuration management system.
2. Assemble Build from all Files: Take all of the files needed to create your distribution including every dependency, release notes, etc. and create your distributable build. The distribution (a.k.a “build”) ultimately should be exactly one file per target platform. Copy to a central networked location either named with the unique build number or otherwise clearly identified.
3. Automated Unit Test: Using a third party framework (recommended) or your own custom framework perform automated tests designed to exercise the individual components of your system in a very detailed way. This is often easier than a real system test because the tests will tend to be more stable over time and more compartmentalized.
4. Install and Smoke Test: On a green system, perform a fresh installation and basic smoke test of the system.
5. Automated System Test: Perform a full automated test of the public surface area (that reachable by users) of the system.

Compiling Files from Source Control

The first task to take on is to automate compiling all of the files from source. This is required before the build can be centralized, and really is the cornerstone of the process. This step will require the most investment before you realize any return. We recommend taking a developer with IT administration experience and dedicate them to the task. Many IT administrators are used to automating tasks and working with installing and cleaning up software, and a solid understanding of administration can be very handy for this step. In our experience it can take as little as a few days of time to as much as a few weeks depending on the experience of the developer and the complexity of the product.

It’s important that the build process be idempotent to be valid: Regardless of where the build gets started, you should be able to restart it and have it recover, cleanup, and then work. Generally it’s best to clean up the failed build then proceed with a good build instead of trying to pick up where the previous build left off (it’s more deterministic). This approach also lets you develop the build iteratively with a failed build recovery stage added to the start of a normal build process.

The build process must ensure that it generates the right labels to meet the traceability goals of a build. The best way to do this is to generate a unique build number, label the source code, then pull the source code based on the label. Even in source code management systems that aren’t completely transactional, retrieving source code by label is going to be consistent which is the key goal. This allows the build to run at any time without requiring developers withhold from checking in or out source code due to fears of interfering with the build. For best consistency, the build process should label all of the necessary source code in as short an interval as feasible (to guard against drift between projects) and then it can pull the source code as needed during the build process.

The build should be easy to extend with new projects. This requires spending a little time considering how to externalize what projects to compile, where to get them, and where to put the output from the raw build process itself. We spent some time writing a standard build script that integrates with our product of choice which uses a single data file to tell it all of the information it needs for any one product. This lets us set up new products very quickly and amortizes the development effort of making the build process over multiple projects.

Some example products that can help you fully automate your build:

1. ANT: Free and capable. Probably the best freely available build system. Available in many flavors to match your technology (like NANT).
2. Visual Build Pro: We’ve used this on several projects and it’s what we use internally. Sports a great GUI for developing and debugging the build process and comes with build-in interfaces for pretty much anything you want to talk to on Windows, and can be easily extended. Great for folks that prefer an IDE. Very cost effective.
3. MSBuild: Visual Studio ships with an internal build environment that can be extended to handle a number of tasks and, with enough force of will can be used to do most anything you want. Debugging and extending it to perform tasks beyond basic compilation and file copying can be a challenge, and your time is probably better served using a dedicated build framework.

Centralizing the Build

Once you’ve automated your build, you can move it to a central server. To centralize the build on a common server, you need to have a mechanism that satisfies at least the following:

1. Anyone can trigger a build remotely: Anyone (subject to some basic security authorization) can trigger a build, and do so without any particularly specialized knowledge. The build system will automatically know if it’s safe to build (such as ensuring conflicting builds aren’t run at the same time). This has to be available from anywhere developers are.
2. Easy access to the status of builds remotely: It should be trivial to know if a build is underway and to know the historical success of the build process. This has to be available from anywhere developers are.
3. Works logged off and through restarts: The build process should not require the system be left logged in or require manual steps to bring online after a computer restart.
4. Runs as a unique user: The build process shouldn’t use anyone else’s identity to log into the source code management system or access other resources so it’s very clear from an audit perspective when the build did something.

Some example products that can centralize your build:

1. Cruise Control: Pretty much the standard. Free, capable, and satisfies the requirements. We use the .NET oriented version, Cruise Control .NET for our in-house build system. The user interface is fairly primitive, and the non-web client is unduly cranky, but it is a good remoting system. It isn’t a particularly good build system – you’ll want to use one of the products listed below.
2. Automated Build Studio: This commercial product for the windows platform is reasonably capable and is both a build product and a build centralization system. While it is capable, the pricing is fairly high when you compare it to Cruise Control + Visual Build Pro (see below) which is substantially as capable. This is because you really don’t need may Visual Build Pro license(s) on a typical team, but you will need a number of Automated Build Studio licenses to allow anyone to invoke and monitor a build. The centralization capability is relatively new in the product. The main reason to go this route is that one configuration IDE can give you centralization and automation, so you are trading money for time.

Challenge Your Team To Fill It Up, Then Buy A Bigger One

While it is very tempting to recycle some old developer system or server as the build server because it doesn’t feel like performance should matter, investing in a high speed build server can pay back quickly by allowing the build process to be optimized for strictness and reliability instead of performance while keeping it fast enough to preserve the development team’s attitude that builds are free. When your build process gets long (up to 1 hour) your team gets a great boost in productivity by purchasing just one system. Consider that a great new build server should top out at $5,000. If you replaced it every 18 months it’d double in performance at the same price.

When specifying a build server, you want to emphasize single processor performance and disk performance. Memory is generally not a particular issue, so invest in the fastest pair of disks you can find with a good hardware RAID controller (for the very best disk throughput) and the highest gigahertz single socket processor you can get. Build processes can very rarely take advantage of multiple cores, so anything beyond two cores isn’t going to speed up a single build, but gigahertz will. Finally, make sure it has a big network pipe to the configuration management system because it will spend a fair amount of its time pushing things in and out of it.

Automated Unit Tests

Connect your existing unit test system to the build system to automatically perform the unit tests on each build and post the results. This can generate some useful metrics that you can use to understand the quality and progress of your development:

1. Increase in Proportion to Features: As you are claiming victory adding features to your system, you should see a linear increase in unit tests. For example, if you have 100 unit tests and the current system has say 200 design features then in broad terms for every 20 design features you should see 10 unit tests. It isn’t completely accurate- it’s a guide line. However, if you see only 5 unit tests and 25 features, you know something is up
2. Indication of Design Complexity Problems: If you are seeing unit tests routinely break that previously worked, or a single unit test repeatedly break, this will tend to indicate that the design of the system (the architecture or software patterns or implementation) is unreasonably complicated for its feature goals. In the abstract it’s often hard to have team discussions in these points because it’s all a matter of tradeoffs, experience, and opinions. This will give you empirical evidence that the software is overly complicated to keep functional.
3. Indication of Performance Issues: Every unit test, when tracked over time, gives you a clear trend to understand the performance of your system. It provides a highly standardized test case where the exact same routine was run on the same hardware in the same way, and timed. If you see the duration of your unit tests change (up OR down) it’s worth investigating – it may be failing quickly or slowly internally, or you may have counter-optimized the code.

Most Automated build systems can directly run one or more unit test frameworks. The build products discussed above each can do this. If your build system can’t, you might look at one of these. You should definitely check out:

1. NUnit: Unit testing for .NET. And yes, we get the irony that the site is written in PHP. This is our unit test framework of choice, however it’s worth noting that we’d be a lot less enamored with it if it wasn’t for ReSharper’s ability to act as a dramatically better test runner.
2. JUnit: Unit testing for Java. Don’t let the nearly comical web site fool you, this test system is all meat.
3. TestComplete: A commercial product from the same folks that made Automated Build Studio. It goes way beyond unit testing (as do most commercial test products) and integrates with both Visual Studio and Automated Build Studio, so if you go with ABS you might look to use these together. It earns our honorable mention because it’s reasonably affordable, easy to use, and very approachable.
4. Hundreds of others: There are many automated testing products out there. We recommend you start with something very simple and straightforward and stay away from the large enterprise testing systems. These are really not meant for the needs of small and mid-sized development teams, particularly where your QA staff are largely development trained.

Run It Every Day

Every development project should be built every day if there’s any change made to the source it comes from. The build centralization systems mentioned above can detect if there is any change to the affected projects checked in and automatically queue a build for a fixed time of day (for a scheduled, automatic build) or a few minutes after the change is checked in. The latter is a great approach for a team that is fully adopting agile development practices: Configure it to automatically start a build after any checkiin after there haven’t been any checkiins for a few minutes. This gives rapid feedback to the team and encourages good configuration management discipline: What you check in better build and pass tests, so don’t check it in until you’re ready.

This frequent execution will ensure that build changes need to be coordinated with code changes because they’ll fail the build immediately otherwise. This quickly will instill the discipline within the development team to keep the build clean, which in the end takes up the least time. Just like it’s easier to maintain something than fix it, it will take the least time across the team to keep the build live and accurate than deal with the downstream consequences of not having your house in order.

Get There In Stages

The great thing about centralizing and automating your build process is that you can get there incrementally. You can make minor investments in time from various team members and each investment will add a little value to your automated build process which in turn will generate a return from there forward. There aren’t a lot of development practices that can be incrementally adopted in such small chunks yet produce steady returns.

The incremental approach is highly recommended because it helps conquer some objections by not requiring a major change in developer habits or a major investment in developer time up front. You can even usually get it done with such small investments in time that it doesn’t need to be a formal project or even a formal part of your project plan, if you are concerned about internal resistance to spending work on non-development activities.

Wait, What about Visual Studio Team System?

Microsoft’s Visual Studio Team System (VSTS) does provide all of the infrastructure you need to automate your build process – centralization, configuration management, testing, reporting, the whole package. If you can afford its licensing fees and the investment in time and resources it takes to set it up, it will fit the bill. VSTS is aimed at larger development teams – on the small side 20+ developers (people actually writing code). The tools and techniques discussed in this article are intended to provide value on teams down as small as two developers. Unless you have large development teams or free VSTS licenses, it’s probably not your best bet.

About Product Recommendations

This article features specific product references and recommendations. Neither the authors of this article nor eSymmetrix are affiliated with any company mentioned, nor have they received any consideration at any time from a party with an interest in these products.

Editor’s Note: This is the second article in a three article series, with a new article posted every few days.

In the last 15 years, software development tools have advanced significantly. More than language or runtime, the modern integrated development environment is an amazing productivity tool. You don’t even have to spend money to get this benefit – check out Eclipse and its ecosystem for a free and powerful tool. Combined with the extraordinary processor power of modern desktops, you can do things today that just weren’t feasible 15 years ago. In many cases, this is a great improvement, but it has come at a cost: When you can compile even a large project and debug it in tens of seconds instead of 30 minutes it tends to take you away from the good Mom and Apple Pie aspects of having a tight build process and not doing development by debugging.

Benefits of a Build System

When you have a common means of performing each build, you get a number of advantages that will pay off for your team.

1. Consistency: Without a doubt, the greatest benefit is that each build is consistent and you know that it is consistent. Both pieces are important. You will be able to dismiss with confidence questions about the accuracy of the information that indicates what exact source code was used to make each build, what defects really were fixed in each build, etc. This is particularly powerful because it is most useful when the team is under maximum stress: rapid fire builds under pressure, when there are multiple builds in play at various stages of certification, etc. It’s in these situations that you have to be able to absolutely trust that the records are accurate or you will burn a significant amount of time across your team verifying what was in each build. This is waste effort when you can tolerate it least.
2. Knowledge Containment: Each developer doesn’t need to know the details of building the entire system. Usually only one developer needs to know the packaging and release details, numbering scheme, etc. Individual developers only have to know at most what it takes to add their new project to the build process. When a new member joins the team, they can get up and running fast and start contributing without having to take on the whole beast right away. When you come back to the system six months after it shipped and need to make one change, no one needs to remember all of the intricacies of creating a full build.
3. Change Management: Whenever a new output file is added or a change needs to be made to how the application is built, the build will need to be updated. By insisting on an automated process you ensure that each change has to be fully addressed before the build can be tested. This will force issues to the surface earlier in the process when the team will tend to be under less stress instead of later when the team is under tighter deadlines.
4. Time Savings: Running a build is generally an annoying waste of time. If you take on that a build includes all of the steps we discussed in the first article in this series then even for a simple project it’s going to take a minute, but for a project of real complexity it will tend to be 10 minutes or more. More is actually better in this case because it represents more human actions that are automated into the build. If the build is happening on the developer’s system then the time savings is limited – the developer will generally be idle while the build is running. The time saved in this case is the time for a human to manually perform each step vs. the computer, perhaps 40% of the total time.

Benefits of Centralization

This part is often more controversial, but has become accepted more and more as a best practice. Many have argued that you should automate the build, but make it so each developer can run the build on their system. This was relatively straightforward in the good old days of “make”, but there are several advantages to centralizing builds for modern applications.

The big benefits of centralization are:

1. Forced Standardization: We’ve all had the experience of two computers building the same source code and not producing the same binary images. This is generally due to small differences in the software installed on each system that the build is referencing. It probably won’t hurt you, but it might – and the more third party items you reference the higher the odds it will. Instead, by building on the central server you can ensure that if there’s a new version of a common library you know exactly what one is being referenced by the build. You know that you only have to update it in that one place for it to go out, not on each developer’s system.
2. No Cheating: Let’s face it, it’s tempting – any individual developer will be tempted in the right circumstances with short cutting a build on their system, perhaps swapping one file into the output directory and just re-running the packaging step to get a new distribution. If it’s on a central system that few if any have direct access to then you know for sure each build came from the full, correct process.
3. Major Time Savings: With the build happening on a central system, the developers are free to do whatever they want while it’s running. They can continue to do extended unit testing waiting for the build to complete, or move on to entirely new functionality. The build should be designed to pull its source code immediately or otherwise ensure that it’s protected from changes checked in while it’s running to make this possible.
4. Central Objective Metrics: With a central build who’s status is published to everyone, the entire team knows if the system currently builds or doesn’t. It establishes a central point to look at to get a quick understanding of where the team is. After all, your customers don’t care if your personal code compiles, they’re buying a whole solution which requires a full build work.
5. Workstation Decoupling: With a central build process, it isn’t necessary for any individual workstation to be capable of building the entire system, they just need to be able to build their part of it. This can be very handy in several circumstances including reduced licensing costs for third party components by segmenting developers by subsystem and reducing the overhead of upgrading referenced components because individual workstation inconsistencies aren’t an issue unless the developer’s run into a defect in the course of the work they’re performing. This is particularly handy during maintenance when a developer has moved on to a new project.

Not all of these benefits will apply to every operating system or application type. In fact, it seems this discussion is most frequently had in Windows development teams. This may well be an artifact of the tendency for Windows environments to be more distributed than Unix environments with centralized builds being more the rule than the exception on Unix.

Best Practice: Continuous Integration

When you’ve created a fully automated build and then centralized it, you’re dangerously close to a best practice called Continuous Integration. Continuous Integration builds on the concept that if the entire build is automated, and it can be done on a central server, then each build is free. The only potential cost of a build is the risk that you’ll make a change while the build is running and want to start another build of the same project. If you’re coming from a world of expensive and time consuming builds, this is a revelation.

1. If builds are free, do a new build every time someone checks in a change. Now you’ll know right away if that change breaks something, and you’ll know that a build reflects the latest code changes.
2. If builds are free, let anyone trigger a build. If someone just isn’t sure if a change made it in, or you made a change to the build server, or you just want to see an updated test run, or even just want to make the lights on the server blink faster – do a build. No harm, no cost.
3. If builds are free, anything the build does is free. Seems obvious, but follow me on this one: Anything you can teach the build to do, it’s free. Why not have it run some automated tests on the software? What about uninstalling the old version from your test environment and installing the new one? How about some performance tests? Notify the development team of new builds? Move defects that are claimed resolved in the new build to the next workflow state so they get tested?

Automated Testing

We’ve found it always an uphill battle to introduce automated testing into development teams that aren’t currently doing it. Automated testing is often viewed as a post-developer action that automates what manual testers would otherwise perform but we believe this misses the point. Usually the highest benefits from automation don’t come from slavishly automating the exact process that is done manually but instead looking at what you could do if the whole process was automated. For example, no human is going to completely regression test an application for a minor change, it is simply too time consuming. It may not be feasible to perform an accurate regression test purely using the public, published surface of the product either.

Instead, look at introducing automated tests into the development process itself. Don’t get hung up on whether they are automated unit tests or system tests or just make you feel happy tests. We’re big fans of using NUnit/JUnit to get developers to start writing good, idempotent tests in parallel with their code. You can even use these when your software isn’t necessarily written in either .NET or Java, provided you can call it from .NET or Java. Most automated build systems can intrinsically talk directly to NUnit/JUnit, and if not they can call a command line to invoke the test execution capability distributed with NUnit/JUnit. This means the developers get the benefit when they are writing the code that they can prove it correct, and you get the benefit of each of these tests being run every time the system is built.

Even if you don’t get into automated unit testing, consider writing just a simple script or executable that will smoke test the application. Then, have your build process distribute the fresh package to a test system, install it, and run the smoke test application. This should be part of the success/fail criteria of the build. This will give you a base level of confidence in the entire process nose to tail, and will encourage developers to add things to the test process because they know they’ll get benefit every time the build is run.

Coming Next: Great Results through Incremental Investment

In the final article of our series we’ll look at how to create an automated build incrementally and make it a natural evolutionary process of your team, providing both immediate and ongoing value. Really, it’s no joke and it’s not going to require upending your technology or team.