Reliable Systems

It’s very popular to consider the internal users of IT services as customers, acting like IT is an in-house service provider that the rest of the company purchases services from. The goal behind this is usually a reaction to a real or imagined belief that IT isn’t being responsive to the needs and budget of the rest of the company. The thinking goes that by having IT think of the rest of the company like an outside organization would of its customers, you can ensure better accountability and buy-in. Typically, organizations that go down this road also adopt a charge-back model where the IT organization charges back all or nearly all of its costs directly to the other divisions within the company that are consuming those services.

While there are several positive aspects that can come from this approach, there are several problems that can easily be created that stem from the problem that in most cases the rest of the company really isn’t a customer in the classical sense. Why? Because they lack a true buying choice. Furthermore, it generally isn’t in a company’s overall best interest for their divisions to really be customers of their own organization.

The original motivation for taking these approach is usually to address several issues:

• Division buy-in on costs and priorities: If they are directly paying the bill, they are going to pay for what they want and not ask you for things they aren’t willing to pay for.
• Clear status and communication: The project reporting and communication model is simpler for everyone to get their head around if it’s based on something we’ve very familiar with. Each player can figure out their part.

If you model the relationship between the IT organization and the rest of the company as a service provider – customer relationship, it’s easy to miss the transitive qualities of this: if they are your customer, you are their vendor. The word Vendor casts things in a different light: If you’re a sufficiently large organization you probably have a vendor management office whose sole job is to ensure you pay the least you can for things and fosters competition between vendors. Their job is largely to keep the company from getting too cozy with any one vendor. Are you ready to be just another vendor, like the one that bids annually to supply fresh coffee or office supplies?

Benefits

There are several good things that this model will tend to create.

• Defensible functional requirements: Unreasonable requirements tend to be expensive relative to their value, and the division is more ready to discard them.
• Role Clarity: The Vendor/Customer relationship is relatively easy to understand, and each party can generally determine their role quickly. When there are disputes, there’s a natural framework for resolution.

Challenge One: Buying Choice

It isn’t a long road from treating your internal divisions as customers until they look at you as a vendor. Once they consider you just another vendor (like the one they selected to provide fresh coffee to the office, or office supplies) they’ll want the advantages that come along with being a customer. For example, it’ll seem clear to them that it should be optional to use your services. This will feel very reasonable to upper management – it’s all part of the transitive nature of IT being accountable.  If IT can’t deliver a service at the best price, why not go to another provider?

This will likely start with something that will be difficult to argue against – such as a large software development project, perhaps in a language that your in-house talent isn’t familiar with.  Now, what about hosting for that product?  If you are charging back true costs for your data center to each division, you are unlikely competitively priced with what a division could get from Rackspace or Peer1.  It isn’t necessarily that those companies are more efficient than you are at doing the same thing (indeed, if they are then you should broker your own contract with them) but instead that it isn’t an apples-to-apples comparison.

Challenge Two: Implied Requirements

Whenever an internal IT organization takes on a project, there are a number of implied requirements that affect cost and schedule. Some of these requirements are from the IT organization itself (like technology choices) and others are from the corporation (role of internal staff and contractors, project management and reporting standards, etc.). When a division looks to bid out work to an external source, these requirements are usually unstated because in many cases they aren’t requirements the division has on the solution.

Another way to look at it is that any constraint on the solution that the customer (the division in this case) doesn’t have or care about is an implied requirement and likely a competitive disadvantage when comparing internal IT costs with external costs. In broad strokes, the difference in requirements is that a division’s requirements are almost entirely about outcomes, not methods: They care about the results their users get, not how they are achieved. IT organizations often focus their requirements on how results are achieved (using this technology, in that enterprise architecture, developed with our RUP-based approved process, tracked by our PMO) and they defer to the division the functional requirements.

Local Maxima and Minima

When each division or cost center is free to chose what services they are willing to pay for, they will converge over time on only those services that are good for them. Establishing shared services is generally challenging because each party will want to ensure that everyone is paying their fair share. This is often tricky to define – should it be proportioned by feature usage? Capacity?  This often creates a “first mover disadvantage” scenario where no part of the company wants to be the first to get a new service such as a database server or SharePoint Portal because they’ll be hit with the entire cost of it unless someone else comes along.

Secondarily, upgrading services gets challenging because no drop of rain believes it is responsible for the flood: If you want to upgrade to Exchange 2007 from Exchange 2003, one division can easily say that they don’t believe it’s necessary and thus decline the costs. If you need a larger server to house SharePoint, who is going to get the bill? A game of chicken often gets created where multiple parties all want a service, but no one wants to be the first to ask and risk subsidizing everyone else.

With each cost center pushing to only pay for those things it perceives sufficient direct value to take on, they are making decisions only based on what gives them the best cost or maximum value. This isn’t likely to align with providing the overall lowest costs for the company. For example, three separate departments could easily decide to implement their own direct attach storage for disk because none of them feels they can justify the cost of a SAN, however together it would be less expensive to construct and maintain a central SAN environment with SAN backup.

There are some straightforward exceptions to this problem where shared services are generally easy to get consensus on and cost out. Typically these are raw infrastructure services such as email or file storage where there are clear units of measure that allow for proportional billing (mailboxes and gigabytes used, for example).

An Alternative: Clients, not Customers

If the Customer/Vendor model isn’t the overall best approach for a company, what alternative model can provide the benefits without the unintended consequences? How about a term that’s between User (which has accumulated a substantially negative connotation) and Customer – Client. A quick trip to the dictionary shows that a client is any person or group that is the party for which professional services are rendered, which fits reasonably enough.

As your clients, they still are entitled to a great deal, just like customers would be. As the client of the project, they:

• Determine success & failure: Your project isn’t successful just because it follows the corporate processes or works on the corporate approved IT infrastructure; those are the constraints on how you solve problems that are immaterial to your client. Success is determined by whether you achieved the goals the client created. That may mean you need to do some extra communication to make sure your client knows that their goals were met, even if that’s not in the standard process.
• Decide if it’s worth the price: In the end, the problem may just not be worth solving.  Many things can be done but the cost in time and distraction exceeds the value.

Unlike a customer, since you’re part of the same organization you can share with the client your insight into the costs and risks of the project in a way that no vendor ever could.  In the end this creates the best partnership that delivers long lasting results.

A final note

If you don’t treat your users as clients, odds are very good they will eventually get themselves a buying choice. When they do, they won’t chose you. Don’t let it come to that, it isn’t ultimately in their interest, your interest, or your company’s interest.

Many if not most software project leaders came up through the development ranks.  It’s generally thought of as a distinct advantage – you know the technologies you’re using, you can form your own well reasoned opinions about how hard something is, what is possible, and how long it should take.  For a long time, I felt that the best way to get results from development teams was to use my experience and knowledge to be very understanding of the challenges they faced and give them whatever time they asked for.  However, in the last few years I’ve run into several situations where I just couldn’t get them the extra time or relief from the most problematic requirements.  I predicted doom to the projects in question but instead I observed some of the best outcomes I’d ever experienced.

While the projects were successful, it bothered me that the secret sauce seemed to be a rigid adherence to schedule and delivery more than any other consideration.  This was exactly the reverse of how I wanted projects to succeed:  I wanted them to succeed because I was treating the developers how they always wanted to be, not like a stereotype from Office Space.  How could it be that better results came from ignorance of the technical details involved?

Developers Will Use All Available Time

Upon reflection, the first thing that struck me was how much an immobile deadline focused discussions and decision making. If you give a team more time, they will expand their process to consume it.  Time will get consumed by:

• Elaborate Decision Making: When you have little time, you make a choice and go with it until it appears it just can’t work.  When you have a lot of time, you sit back and look for the very best option.  That then requires defining what the best is – is it fastest, or smallest, or most scalable, or whatever.
• Development Approach: Under pressure you’ll tend to go with the proven guaranteed approach.  If you have the luxury of time you’re more likely to engage in yak shaving like investigating a new tool or approach, or writing several prototypes first before you develop the real solution.  You might even just throw caution to the wind by skipping a formal design figuring you’ll have the time to just code and test your way to a solution.

The more time a development team has, the harder it is to argue against spending it on up front luxuries.  It also can be harder to argue for long term best practices because the team has the time now to develop a solution any way they want.

Unknowns Create Boomerang Estimates

Even very experienced developers are generally terrible at estimating the duration of developing a solution.  This has been demonstrated over and over by many other parties.  The key behavior that we’ve observed is the phenomenon that from when you approach a specific development problem (like displaying a graph on a web page) until you know exactly how you’re going to solve it (and have a reason for confidence in that approach) you will tend to estimate high because in effect the only reasonable estimate is infinity.

Put another way, as long as you don’t know how you will solve a problem you don’t know for sure that it is solvable which means it will take an infinite amount of time to solve it.  Fortunately, developers are almost universally optimists so they believe they can solve anything eventually – so they’ll pull out a standard answer like three weeks or months or whatever feels like a big chunk of time to figure out the problem but not so big that it kills the project.   The reality is that until you know how you’re going to solve it, it feels like it could take forever.

Once a solution has presented itself  the development team will often find that all it will take is some cleanup and polish to be done- a very small amount of time.  What will push the team to find the answer?  We’re back to the problem of elaborate decision making when you have the luxury of time.  Finding solutions tends to not be a linear problem that will be solved with incremental development energy.  Instead, it tends to be solved by getting people together and brainstorming possible solutions until you find a few candidates and can work out what it’ll take to prove them out.  Under pressure, people tend to focus their creative energy and be more willing to compromise.   That flexibility will tend to get rid of pet requirements and developer gold-plating and focus on the most critical aspects of the problem.

What’s the alternate approach?

The key is to not let your knowledge and experience as a developer lead you to buy into the stories the team creates around what’s reasonable to get done and how long it will take.  Instead, you have to stick with the project’s goals first then the facts of the project.  The project’s goals form the objective reality of what has to be accomplished for the project to survive:  Deliver this functionality by that date, keep these people informed, solve these problems without causing those problems.

When the team runs into a wall and needs more time, instead of buying into the story of needing a lot of time, set a specific and tight goal that keeps a solid amount of time pressure on the team to solve the issue and prevent the problems above from showing up.  Ideally, find a way to give out one or two day chunks to answer incremental questions if necessary to emphasize that time is precious and has to be invested carefully.  This is where you can leverage your experience in a way that a non-developer can’t:  The team knows they can’t snow you with tech details, and you can define a specific, measurable result that can be achieved in a short period of time that they can’t argue with.  Despite this, you are bound to have to assert a few times that the time limit is the limit – solve the problem in that time.  It’s very hard because you’ve been on the other side of that conversation and it can feel like you’re the Pointy Haired Boss, but it’s fundamentally your job on the project.

What will nearly always happen is the team will surprise itself – a solution will be presented within the team that they can live with and can be done in the time they have.  It may be incomplete or have some risky shortcomings, and you’ll want to ask how long it’d take to address those.  You probably shouldn’t address them in the first round, but the team will feel better that you’ve considered through things and will buy into the outcome more if you ask.  You’ll also want to make a record of it so that the team can in the future recognize what was a predicted shortcoming vs. an accidental defect.

Do you want it solved right?

This is a question that often gets voiced within a team as a rebuttal to external time pressures and is very dangerous.  The challenge is that most non-technical people don’t get the number of ways that a problem can be solved: instead, each problem appears to have a single solution.  Take away your technical knowledge and imagine you’re the paying customer:  What’s the alternative – were you going to solve it wrong? If that’s the case, what else have you done that’s garbage?  If you took your car to a repair person and they said it’d be $500 to fix it, then when you came back they said well, if you want it fixed right it’ll actually be $1200, wouldn’t you wonder what the hell the $500 fix was?

Usually this statement is uttered in desperation when a team believes they just need more time to figure out a problem.  Nobody wants a problem solved wrong.  Skip the hyperbole and get down to action:  break down the problem into small chunks of time that can be invested for a specific measurable result, and make sure the team gets that overage time is the most precious commodity.

Side Note: This is an advantage of SCRUM in practice.  If you’re following an Agile Development practice, particularly SCRUM, this fits right in:  Focus on making each sprint deliver the user stories it was supposed to even if you have to leave some special cases for a later sprint.  The daily stand up meetings are a great place for the different team members to apply team pressure against over engineering and doomsday estimates.

Cleaning Up and Closing Out

At some point you need to close out your release and ship it. For each of the areas where you’ve had to make compromises and taken shortcuts you have to choose to either:

• Ship as Final: Decide the implementation is close enough to the intent of the end-user functional requirements that it can be the final implementation (at least until new information contradicts this decision)
• Ship as Temporary: Decide that something is better than nothing and ship the feature with limitations.
• Cut the Feature: Hold back the feature until it can be reconsidered or reimplemented.

You’re nearly always better off shipping the feature, often as a final feature pending more information because it’s very hard to gauge the true impact of each limitation.  This is particularly true of user-facing features and environments where it’s possible to evolve the software rapidly.  Inevitably once it’s in the hands of your users you’ll discover aspects of it that you didn’t think of that will require rework and you may discover that the killer feature you were sure would be the hit of the release is hardly used.  In either of these cases if you’ve invested a great deal of time in making it foolproof the team will tend to resist changing it.  It’s a natural product of the presumed relationship between effort and value.  If necessary, you might put in some temporary safeties to detect and catch the limitations you’re worried about.

The major exceptions to this approach are areas that are too dangerous to deploy if less than fully trustworthy.  For example, if your team is developing a data storage system, software deployment system, or other critical infrastructure your choices likely resolve down to making it as right as possible or holding the feature until it can be reworked.

If it turns out that the solutions that are viable within the schedule have significant limitations, you should make sure these caveats are known to the business – provided you can express them in business terms.  For example, knowing that an algorithm won’t work if your userbase doubles is probably not a significant caveat, unless you know the business plans to double in a relatively short period of time.  Every system has limits, and every software change has risks.  Business representatives don’t like to hear the same items covering the same ground repeated every time you discuss software, and it tends to make them not hear the new and important information as well as sound like you’re attempting to transfer accountability from your team to them.

If you spent several hours some afternoon researching on the web what technology is the best for your next project, you’d probably come to the following conclusions:  Linux is the best, or perhaps the Macintosh…  Of course everything can be written in PHP or Ruby on Rails.  If you’re feeling very stuffy, you might be old fashioned and use Java or .NET, Windows or any flavor of Unix that isn’t Linux.  For your database you should just store everything as XML files, but if you feel compelled to have a database use MySQL.  But if you’re still a slave to the 1990’s then you might decide to keep using the corporate dinosaur- Oracle or Microsoft SQL Server.  In the end, the only constant is that whatever you’ve used in the past is certainly out of fashion and certainly a slow, archaic approach to solving problems.

Nearly all teams work within a relatively closed ecosystem – the technologies and people represent only a minor subset of all technologies available today.  Even within these small groups the number of choices at every level are daunting.  Even if you’ve selected your OS, language, framework, and database – what architecture model are you going to use?  What is your data access and caching strategy?  At each turn you’ll want to pick the best option but have a flood of choices to select from.  Many people can tell you about how they led a project that used any particular technology and it worked like a champ with no drawbacks.  On the surface it makes it possible to defend just about any new technology as the way to get your next project done.

What is very hard to find is a real comparative analysis that highlights in comparable situations the results with different technologies.  It’s not a surprise this is so – such analysis is very expensive and time consuming, and few companies would try to solve the same exact problem using competing technologies because it’s not the business they’re in.

Where does this leave you?  What technology should you use on the next project?  Most likely, you’ll have the best success with an incremental improvement on the technologies you already have in your toolkit.  Why?

Infinite Solutions in Infinite Diversity

Most conversations in technology on the web are exclusive - they advocate X over Y or Y over X.  The truth is much more that X or Y can both solve the problem but do it in different ways and with different levels of effort for a team starting from scratch.  What’s much more useful is to ask why you can’t solve the problem effectively with the tools and technologies that are a natural fit for your environment.  Even if a new technology may be the easiest way to solve the specific problem you’re looking at it may not be the best choice when you consider everything that goes into creating the entire software system and maintaining it over time.

When confronted with a strong advocate for a technology shift, keep the conversation focused on the benefits of shifting away from the natural or familiar selections for your organization.

New Methods are Expensive

When you introduce new technologies or tools there is always a short term hit.  Most respected research indicates that even technologies and tools that have a substantial improvement in effectiveness are at best neutral on the first project that uses them.  The most successful technologies and tools are ones that are evolutions of things your team already knows.  The more divergent it is from that, the more time it will take to get over the learning curve, establish best practices, and generally become effective.

Known Problems vs. Unknown Problems

A common challenge when comparing a new technology against existing methods is not recognizing that while you know of all of the problems with your existing technologies, you don’t know of the problems with the new ones.  This can lead to a comparison that shows a number of critical problems with the current technology, and none on the new technology.  It isn’t that there aren’t problems with the new technology, it’s that you don’t know what they are yet. Whenever you put in a new technology, no matter how promising it is, it is going to have new, unexpected problems.

What’s worse is that your organization most likely has workarounds for every problem you’ve encountered, so they don’t really have the impact of a new problem.  Your development team may not know about them, but talk to the operations staff, support staff, and your users before you assume that a technology problem that worries you really is at the top of their list.  It may be that the big memory leak in a third-party library that has you wanting to rewrite a subsystem is conquered in production by having a script reset the service every night.

Existing Code and Libraries

It’s very easy to underestimate the value of existing libraries and practices in effectively solving problems.  When faced with a new assignment, your developers can draw from a large pool of existing, tested solutions to a range of the more mundane, plumbing aspects of the solution.  This includes storing user information, reliably working with data storage, security systems, and other functional requirements that aren’t unique to the problem at hand.  These software libraries accrue over time as developers face similar problems even in development shops that don’t place a high emphasis on modularity and reusability – as long as you have source code control and developers that aren’t paid by the line of code, they’ll naturally find ways to adapt and remold things they’ve already done to fit new needs.

When you have a major technology shift, losing the use of this common body of code will require the first project to reinvent it.  On the surface this may seem straightforward but it’s usually held up by a desire to understand exactly how to best accomplish the same common tasks in the new environment.  For example, you might have written your own security system for your previous environment which you’ll then need to either re-implement or drop in favor of a built-in capability of the new environment you’re targeting.  What’s worse is you need to make these critical decisions at the time when you have the least experience with the new environment:  Is its built in security system really sufficient for your needs?  What about logging?

Your Customers Don’t Care

With the exception of a narrow range of situations (such as developer tools), your customers really don’t care what technology you use to implement your solution.  After all, they’re buying your solution not the technology you wrote it in.  Even if the IT representative of the evaluation team in a potential customer objects because your entire solution is written in a technology they don’t like, in the end they are often overruled unless they can point to a practical implication that you can’t mitigate.  For example, you may get overruled because it’s a Unix shop and they won’t accept a solution that only runs on Windows.  Even in the most extreme cases, if you provide enough customer value it will conquer any customer technology objection.  If the prospect has no Windows servers, that translates into a finite cost for them to support a unique system in their environment.  If your value well exceeds that, then it isn’t the key challenge to crossing the chasm to that prospect.

We often hear developers discussing internals of software development and giving them the weight of user requirements.  If it isn’t visible to the customer, it isn’t a requirement. In the end, your customers don’t pay you to have a beautiful object model.  They don’t care how hard it is for you to create your product or what hoops you have to run through.  For them, it’s a cash for capabilities decision.  It may be true that doggedly sticking with an old technology will mean you can deliver fewer features with each release, or you won’t be able to run on the latest operating system but in the eyes of your customers the question is still how compelling the functionality is and whether you can run on the operating systems they use.

Ignore the Pundits

If you’re part of a shop that has a track record of producing results, be proud.  Don’t worry about what is all the rage at producing the next social networking site, focus on what is effective for you.  For projects that can afford the risk, take the opportunity to incrementally improve your technologies and methods:  Try out a new version of the development framework or new capabilities of the latest database version.  Just remember, you can always tell the pioneers:  They’re the ones lying on the ground with the arrows sticking out of their backs.  Unless you’re part of a dedicated research team, most often you’ll get the best results by waiting for the first round of adopters to figure out what did and didn’t pan out with the newest release and then benefit from that experience.  There’s no satisfaction in burning six months working out the kinks of version 1.0 just to have everything addressed in version 1.1 published a month later.

What’s Your Experience?

Have a great story about being the pioneer, working a project that was packed to the gills with the latest and greatest, only to fall on its face?  Or perhaps you found raging success completly severing your ties with the past?  Drop me a line or leave a comment about it.

Let’s say you’re starting a project to create a new software system. How big does it need to scale? Realistically, either:

1. This new system fits into an existing business, possibly replacing a prior application, so you can predict with some accuracy the different aspects of scalability that apply to it.
2. It doesn’t, and you can’t.

The second scenario is the most interesting one. First off, let’s face it – your new system isn’t going to be the next Facebook, MySpace, or eBay. In short, you don’t need to worry about having your system needing to be designed front to back as a super-scalable system. This is good because the options at that level are time consuming and resource intensive.

The key question you need to understand when laying out a new software system is to what degree it needs to scale without being re-written? This scale is unlikely to be your “best case” business size, because scalability has opportunity cost. This scale should be defined as specifically as reasonable, and clearly understood and validated by both business and technical staff. This ensures that if your business grows beyond expectations that it won’t come as a surprise if you need to make even major changes to your system.

Creating facts from Air

Let’s say you’re starting to develop an application that fits into the second category above. You still need to work out what your scalability target is.

To make any decision that is better than random, you have to work out some aspects of the expected scaling of the application. In the absence of real facts to extrapolate scalability from, you need to cooperate with the business side to established presumed facts of the scalability requirements. This may sound a lot like assumptions, but they really go beyond that because these will become facts as you develop the system. As a starting point, make it clear to all involved that:

1. If the targets are low, it should be assumed you’ll have to turn away business because the system can’t scale above them.
2. If the targets are high, the system will cost more and take longer to create.

In most businesses, the second outcome is worse than the first. Why? Because the second is a price you pay up front, before the system goes into service. The first is based on an assumption: you might have to turn away business. You also might be able to realize it in time and address the issue. From a business standpoint, this is a better trade off. Finally, there’s the non-technical aspects:

1. The sooner you have a working system, the sooner the business can validate the market and start getting real data on uptake to adjust your scalability goals
2. Unless the product is a failure, you expect demand to eventually exceed the capacity of the system, it’s just a matter of when. If it does, then you should be able to afford rewriting all or part of the system. In other words, the funds to solve the problem should be available if you have the problem.

From this comes an axiom of scalability:

The system needs to be based on the lowest scale that will provide enough time and money to replace it with a new system.

Put another way, a system that is faster or more scalable than it needs to be for the business was more expensive and took longer to develop than necessary. Think of it like a race car: The ideal Indy Car would fall apart just after the judges validated it won without breaking the rules. Any stronger and that strength could have been put into something else. The time you spent making it more scalable than necessary could have added more features, fixed more defects, or gotten it out the door sooner.

Establish a Growth Curve

The growth curve needs to be sufficient to inform the developers of what decisions to make at each point. To get there, start with describing the scale from the business stand point. During design of the actual system you can keep translating this into the specific requirements for speed, storage, and capacity based on the behavior of the actual system. This will prevent you from achieving technical goals that don’t satisfy the business goals.

For most systems, you want to establish the business goals for:

1. Number of Possible Users: How many accounts will there be on the system? This is an upper bound of the number of people that could access the system if they wanted to.
2. Number of Simultaneous Users: Number of accounts that will be accessing the system at the same time. For most applications, at the same time is likely best thought of as in the same 15-30 minutes.
3. Number of Customers: For most applications delivered to businesses the number of customers (e.g. businesses) drives the scalability of some parts of the system (such as configuration and data storage) will scale based on the number of customers, not the number of accounts those customers have.
4. Data In and Out: If the system is going to have any imports and exports that aren’t user-driven (such as EDI feeds or a public API) then the number of partners (other entities that will exchange information with you) and the frequency of exchange need to be determined.

Things to not bother with:

1. Response Time: For customer interactive products, response time is dictated by what end users will tolerate and is not really going to be a business decision (aside from deciding if you’re going to produce something your customers are willing to use). For non-interactive products or back-end this may need more discussion with the business, but again – the business is going to expect you to be able to figure out what will make it a success.
2. Data Retention: Assume it all has to be kept and more indefinitely. In the end, storage is cheap and this design decision rarely costs a lot of made up front but is expensive to reverse. Data also has the amazing power to make heroes out of IT when the business starts posing questions later and you can answer them. Generate as many facts as you can now to help you out later.

These items are past the point of diminishing returns with the business. You should work them out within the development team and document them, but you shouldn’t believe that any business sign off you might get is binding or useful.

Build to the Scale

Once you’ve established your growth curves, pick your candidate architecture and translate the growth curves into system performance requirements.

Hypothetical Example: If you need to support 1000 simultaneous users for a web application, determine the dynamic web hits per second by determining how often an average user will request a dynamic page (say ever 5 seconds, which is very fast for most dynamic applications) These two numbers would give you a dynamic hits per second of (1000/5) = 200. Then add how long each page will take to calculate (make a goal of say 250ms) to get how many requests you need to be able to process at the same time: (200 * 0.250) = 50. This is the key scale point for your web application: When deployed, it must support 50 requests being processed in parallel. You’ll need to get to this point by either making it really scalable on a single server, or splitting the load over multiple servers.

One thing that should jump out of the math behind this is that anything you can do to make the calculation time of a single page drop pays big dividends: If you drop the average calculation time by half (125ms) then the number of requests in parallel drops by half (200*0.125) = 25. This in turn may well cut the number of servers you need in half, easing your maintenance and deployment cost. If you can’t do this, reduce the number of dynamic pages requested per second by either making more static pages (such as pre-rendering pages that change but don’t change frequently) or caching dynamic pages that have some predictable consistency (which really makes them static pages). This is often much trickier to do and test, so your best first option is to reduce the time for each page.

Side Point: This also highlights an easy way to accommodate guessing low on a system that’s been in service for a year or more: If you’re processor bound you can replace that hardware with current units and often pick up 30% per year it’s been since you purchased the original hardware. This won’t save you from network problems, disk storage problems, or some memory problems, but it is surprisingly handy.

As you look at each candidate architecture, look at each component and determine the critical “how much, how fast, how often” factors based on the business inputs. If you change your architecture or external interface design (the user interface or import/export capabilities) you need to re-evaluate if you’ve moved the targets as well because your design goals no longer reflect the business growth curves.

Really, to the Scale

Within your development team you will typically have two types of developers you need to watch: Those that never consider scale and those that obsessively consider scale. The former will build it however and then wait to see if there is a performance problem. The latter will try to make every system the next Amazon. Neither situation is good. Identify early people’s tendencies and work to manage them to the center. Remember that the system is only as scalable as its slowest part, and there is always a slowest part.

You can get good results by having the people that are most concerned about scalability move around on the project to different subsystems. This will tend to keep them too busy to earn the keeper of the nanosecond award on any one system (which they will do if you let them stay put and just work on one system) and will make it unlikely that more cavalier developers can hide a problem. It will also help the team learn from each other: It often isn’t worth making a specific feature as fast as possible, and it is always worth thinking about what will make a feature fast before coding it.

Finally, budget time in the development team to fix scalability issues. Regardless of how much work you put into it, once the real system is build and tested you’ll find places that are slower and less scalable than you expected. If nothing else, you need to develop an accurate model of how the system should perform in production so you can check the real world against it later. As your business grows, you need to be able to get ahead of it and understand when it is time to make the code faster, add hardware, or do something else to stay one step ahead.

Disk is Your Friend, but Beware the Network

If you’ve gone over the system from nose to tail and you’re disk bound, you’ve probably optimized that design as well as you can. Disk has gotten faster at a much slower pace than memory or processor, and being disk bound means you’re getting all the requests where they need to go in a timely manner and are able to process the inputs and outputs, so now it’s in the hands of the hardware. Unfortunately at that point there generally isn’t much more you can do: The difference in performance between server drives and the fastest drives money can buy isn’t very much.

If you’re finding that you aren’t disk bound and you aren’t processor bound then be worried. You’re either network throughput bound or you’re network latency bound. If you’re network throughput bound, you can probably fix it cost effectively with some basic engineering either in how you select what to send across the network or what you cache so you don’t need to send it across. You should try to give yourself some headroom here for growth, but faster networks can be purchased and you can generally tweak the software to mitigate this in minor updates.

Being network latency bound is a more serious issue because it often means that you are at the practical scalability limit of your application. The difference in network latency between relatively cheap hardware and the best hardware isn’t very much, and has been essentially constant for the last 10 years. You can’t buy your way out of this problem. It also is typically caused by a badly designed interface between components of the system which will need to be substantially or entirely rethought and rebuilt to address, which isn’t easy to do with a running system. If you find yourself in this situation and you aren’t sure you have met your business goals you should rethink your approach immediately. Because no amount of money on hardware can get you out of this problem, caution is the word of the day.

In software development, you’re always being asked to estimate things:  How long will the whole project take?  Just this feature?  What if we changed this feature to remove this aspect?   This is all part of the feedback cycle that is fundamental to product creation:  We have a certain amount of time and money for a given set of functionality, but if there’s something really juicy and it just takes a little more time, then maybe we adjust, or if we can get a lot of value for a little effort, perhaps we do a little mini release first.  The business decisions feed the development estimates which in turn inform new business opportunities.

Getting the feedback wrong can be disastrous: The right functionality late can kill your business; perhaps half a loaf earlier is better.  On the other hand, some things the market won’t accept half way so a full loaf it is.  The business needs to trust the information it’s getting isn’t random or capricious to make good decisions, and the development team needs to be able to provide a best guess without fear of misunderstanding.

It’s natural to pad estimates with the idea that it’s better to under promise and over deliver – so that means to guess long and come in well early.  But in the end, is that really any better?

You’re Guessing, and Possibly Lucky

We’ve been experimenting with the new Evidence Based Scheduling features of FogBugz (which we use internally for managing our software development) and one thing that it highlights quickly is that estimation isn’t good if you’re early, bad if you’re late – it’s about getting your average as close to the mark as you can.  Take a look at a graph of most of my estimates:

Ideally, the graph line would have a 1:1 slope, indicating that on average you are accurate.  Further, you want your estimates clustered pretty near to the line itself.  What you can see from my estimate curve is that I’m uneven – I tend to underestimate shorter tasks (under 1.5 hours) and overestimate longer tasks (and that’s after removing some really bad outliers…).  But notably if you draw a ruler on the 1:1 line you’ll see that I’m not even close. Don’t let the hash lines fool you – look at the numbers to see.  The thing is, the other developers in our company that are all regarded as skilled, senior developers aren’t particularly more accurate on any one estimate, and the averages work out similarly.

So what?

Why isn’t it a great thing when we beat our estimates?  There are several potential pitfalls:

Features based on Effort

We’d all like to believe in the rosy model that our customers ask for features and then we build them into the software, so if we’re done early then it’s a pure win.  It’s my experience that it’s much more like a game of Tetris:  What features we take on is dependent on how much effort we think they’ll take.  Every feature has an amount of effort above which it isn’t worth it any more.  When hashing out what makes it and what doesn’t, the effort estimate is a big factor.

If we overestimate the effort of features, then we are slanting the project management decisions away from customer-selected features in favor of the developer’s whims.  This is because some features will be estimated to take more effort than they’re worth, and a more invisible internal team dynamic.  If a project is doing well on schedule, it’s very human to take advantage of this to try out newer, riskier things, over-engineer a feature, or do other things within the team that would otherwise be successfully argued against because of their effort.  In general, the more time available, the more yak shaving the team will do.

Once a schedule is accepted, the business will tend to act on it as fact:  Customers that can’t wait for it will end up being turned away and others will be promised a schedule.  This is a necessary but painful aspect:  Developers are generally optimists and will not want to say no to a customer even though it’s generally not in the best interest of either the company or customer to rush a feature. You want the business to stick with your decisions and not pass the buck on saying no to the customer, but you also need them to trust that it’s a fair trade.

Approach based on Effort

Within the development team, decisions are made at every level on how to implement a feature with an eye towards both the feature’s estimate and the overall project’s status.  Even when not explicitly laid out, a team that believes the overall schedule is tight will feel pressured to find ways to reduce the effort on anything they do.  This means when deciding between a careful implementation that may take longer but be more scalable or easier to support the team will often opt for a more direct path to completion even if the estimate was based on the more careful approach.

If done as a conscious decision in consultation with the project’s sponsors, this can be a way of bringing the project back on track but really it’s just another way of cutting functionality to make schedule:  You’re going to cut out something you intended to deliver (say a more generalized, upgradeable framework for reporting) even though you meet the letter of the requirement in front of you (delivering a few reports).  This can lead to nasty surprises for the team down the road when your sponsor’s find out that they didn’t get what they thought they would.

Alternately, if you overestimate one feature it may have put another feature under pressure so a more expedient and risky approach was adopted for it to fit it into the schedule.  If the true effort had been known, a different decision could have been made.

Make Your Guesses A Coin Toss

In the end, being early and being late just have different ways they create problems for your development project.  Your goal when estimating is to not try to find the estimate that has the highest probability of being sufficient to get the job done but instead the estimate that is equally likely to be high as low.  In aggregate if you have enough of these items on your project (say more than 25) then you’re entire project’s estimate should also be just as likely high as low.

There is still a place for the traditional high estimate: When you move outside of the project sponsor and the development team to users that need a guarantee.  There the downside of missing a date is much worse than the impact of being early.

On your next project, try out the 50/50 approach and make it clear to both the development team and the business.  You’ll probably notice that people develop a more subtle appreciation for the fact that estimates are based on probabilities.  This can help you skip over the discussions that aren’t useful about why you are where you are and instead keep focusing on the business goals for your current situation.