Reliable Systems

It’s very popular to consider the internal users of IT services as customers, acting like IT is an in-house service provider that the rest of the company purchases services from. The goal behind this is usually a reaction to a real or imagined belief that IT isn’t being responsive to the needs and budget of the rest of the company. The thinking goes that by having IT think of the rest of the company like an outside organization would of its customers, you can ensure better accountability and buy-in. Typically, organizations that go down this road also adopt a charge-back model where the IT organization charges back all or nearly all of its costs directly to the other divisions within the company that are consuming those services.

While there are several positive aspects that can come from this approach, there are several problems that can easily be created that stem from the problem that in most cases the rest of the company really isn’t a customer in the classical sense. Why? Because they lack a true buying choice. Furthermore, it generally isn’t in a company’s overall best interest for their divisions to really be customers of their own organization.

The original motivation for taking these approach is usually to address several issues:

• Division buy-in on costs and priorities: If they are directly paying the bill, they are going to pay for what they want and not ask you for things they aren’t willing to pay for.
• Clear status and communication: The project reporting and communication model is simpler for everyone to get their head around if it’s based on something we’ve very familiar with. Each player can figure out their part.

If you model the relationship between the IT organization and the rest of the company as a service provider – customer relationship, it’s easy to miss the transitive qualities of this: if they are your customer, you are their vendor. The word Vendor casts things in a different light: If you’re a sufficiently large organization you probably have a vendor management office whose sole job is to ensure you pay the least you can for things and fosters competition between vendors. Their job is largely to keep the company from getting too cozy with any one vendor. Are you ready to be just another vendor, like the one that bids annually to supply fresh coffee or office supplies?

Benefits

There are several good things that this model will tend to create.

• Defensible functional requirements: Unreasonable requirements tend to be expensive relative to their value, and the division is more ready to discard them.
• Role Clarity: The Vendor/Customer relationship is relatively easy to understand, and each party can generally determine their role quickly. When there are disputes, there’s a natural framework for resolution.

Challenge One: Buying Choice

It isn’t a long road from treating your internal divisions as customers until they look at you as a vendor. Once they consider you just another vendor (like the one they selected to provide fresh coffee to the office, or office supplies) they’ll want the advantages that come along with being a customer. For example, it’ll seem clear to them that it should be optional to use your services. This will feel very reasonable to upper management – it’s all part of the transitive nature of IT being accountable.  If IT can’t deliver a service at the best price, why not go to another provider?

This will likely start with something that will be difficult to argue against – such as a large software development project, perhaps in a language that your in-house talent isn’t familiar with.  Now, what about hosting for that product?  If you are charging back true costs for your data center to each division, you are unlikely competitively priced with what a division could get from Rackspace or Peer1.  It isn’t necessarily that those companies are more efficient than you are at doing the same thing (indeed, if they are then you should broker your own contract with them) but instead that it isn’t an apples-to-apples comparison.

Challenge Two: Implied Requirements

Whenever an internal IT organization takes on a project, there are a number of implied requirements that affect cost and schedule. Some of these requirements are from the IT organization itself (like technology choices) and others are from the corporation (role of internal staff and contractors, project management and reporting standards, etc.). When a division looks to bid out work to an external source, these requirements are usually unstated because in many cases they aren’t requirements the division has on the solution.

Another way to look at it is that any constraint on the solution that the customer (the division in this case) doesn’t have or care about is an implied requirement and likely a competitive disadvantage when comparing internal IT costs with external costs. In broad strokes, the difference in requirements is that a division’s requirements are almost entirely about outcomes, not methods: They care about the results their users get, not how they are achieved. IT organizations often focus their requirements on how results are achieved (using this technology, in that enterprise architecture, developed with our RUP-based approved process, tracked by our PMO) and they defer to the division the functional requirements.

Local Maxima and Minima

When each division or cost center is free to chose what services they are willing to pay for, they will converge over time on only those services that are good for them. Establishing shared services is generally challenging because each party will want to ensure that everyone is paying their fair share. This is often tricky to define – should it be proportioned by feature usage? Capacity?  This often creates a “first mover disadvantage” scenario where no part of the company wants to be the first to get a new service such as a database server or SharePoint Portal because they’ll be hit with the entire cost of it unless someone else comes along.

Secondarily, upgrading services gets challenging because no drop of rain believes it is responsible for the flood: If you want to upgrade to Exchange 2007 from Exchange 2003, one division can easily say that they don’t believe it’s necessary and thus decline the costs. If you need a larger server to house SharePoint, who is going to get the bill? A game of chicken often gets created where multiple parties all want a service, but no one wants to be the first to ask and risk subsidizing everyone else.

With each cost center pushing to only pay for those things it perceives sufficient direct value to take on, they are making decisions only based on what gives them the best cost or maximum value. This isn’t likely to align with providing the overall lowest costs for the company. For example, three separate departments could easily decide to implement their own direct attach storage for disk because none of them feels they can justify the cost of a SAN, however together it would be less expensive to construct and maintain a central SAN environment with SAN backup.

There are some straightforward exceptions to this problem where shared services are generally easy to get consensus on and cost out. Typically these are raw infrastructure services such as email or file storage where there are clear units of measure that allow for proportional billing (mailboxes and gigabytes used, for example).

An Alternative: Clients, not Customers

If the Customer/Vendor model isn’t the overall best approach for a company, what alternative model can provide the benefits without the unintended consequences? How about a term that’s between User (which has accumulated a substantially negative connotation) and Customer – Client. A quick trip to the dictionary shows that a client is any person or group that is the party for which professional services are rendered, which fits reasonably enough.

As your clients, they still are entitled to a great deal, just like customers would be. As the client of the project, they:

• Determine success & failure: Your project isn’t successful just because it follows the corporate processes or works on the corporate approved IT infrastructure; those are the constraints on how you solve problems that are immaterial to your client. Success is determined by whether you achieved the goals the client created. That may mean you need to do some extra communication to make sure your client knows that their goals were met, even if that’s not in the standard process.
• Decide if it’s worth the price: In the end, the problem may just not be worth solving.  Many things can be done but the cost in time and distraction exceeds the value.

Unlike a customer, since you’re part of the same organization you can share with the client your insight into the costs and risks of the project in a way that no vendor ever could.  In the end this creates the best partnership that delivers long lasting results.

A final note

If you don’t treat your users as clients, odds are very good they will eventually get themselves a buying choice. When they do, they won’t chose you. Don’t let it come to that, it isn’t ultimately in their interest, your interest, or your company’s interest.

Regardless of how trivial the defect is there are very real costs and risks to resolving it. Let’s say it’s as simple as a misspelling on a text label, so it’s both really easy to fix and really easy to ensure you fixed it. You still have to contend with:

1. Every Build is a Risk: Every time you package up a set of files as a build, there’s a risk of error. If your build isn’t entirely automated – and entirely means from source code through install – you run the risk of something being done wrong. More likely, the risk may be something external: Unknowingly including a newer version of a referenced library or introducing a dependency on a newer version. Either way, you need to do significant regression testing to mitigate that risk.
2. Deployment Risk: The update will need to get from your development environment to your users. Whether it’s a Software as a Service (SaaS) product that just needs to hit some web servers or packaged software deployed to thousands, your update will need to be installed for people to get any advantage of it. In most cases this will mean a special upgrade installation, notification to existing customers to come and get the upgrade, and additional support for your users.

The truth is that most defects aren’t as clear cut as a spelling error, so you will also have to contend with the possibility that no matter how well intentioned, your fix is going to cause new problems for your customers. It could be that there are advantageous side effects of the current (defective) implementation or that your fix doesn’t work on the Elbonian version of Windows XP which you didn’t discover because you did only a focused test of the fix on your key target platforms. In more elaborate cases, it could be that the loophole represented by the defect is viewed by some of your users as a feature, so fixing it makes your product less valuable. This is more likely when doing defect patching because you typically don’t have the benefit of a beta cycle and end-user involvement in considering all of the aspects of the fix.

The Last Change is to Blame

If you have the opportunity, try this experiment some time:  Announce a new version that never really happened.  Perhaps you just relabel a prior version with a new number or something else to create the placebo effect of an update.  What you will discover is:

• Surprise Fixes: Some group of your users will thank you for the new version.  It’s so much faster than the old one!  Oh, and you fixed a problem they’ve had for months.
• Surprise Defects: Unfortunately much more common than surprise fixes are the number of people that will report a problem that must have been caused by your update because it happened just after they installed it.  It could be as wide ranging as their hard drive died or Word lost its dictionary.  But they’re sure it’s your fault.
• Reinstall Rash: Some contingent of users will have problems installing the upgrade.  The problem will vary depending on how you deploy your fix, but they’ll manage to get a computer or two out of sorts over it.  Don’t think this is a Windows problem either, just look at the volume on support forums for WordPress right after a minor update.

In this case, there isn’t much you can do to minimize the problems because… you didn’t create them (after all, it’s the same software – that was the point of the test).  With the possible exception of finding better ways of deploying fixes, there just isn’t a lot you can do.  This is the minimum end-user overhead to every upgrade you make, and they’re going to make it your overhead.  The big investment you can make to minimize this is:

1. Cultivate Your Brand: If customers love you, they won’t make the leap from coincidence (two things happened at the same time) to causality.  The more they love you, the more they’ll be sure they are at fault.
2. Make Upgrades Easy: You really want to invest in ways that make updates easy.  Look at Firefox and Windows Updates for examples of really great ways to get updates out the door.   It’s easy and surprisingly trouble free, much more so than relying on users to manually know whether to uninstall the old version first, whether an update applies to them, etc.

What Are You Committed To?

It may seem cold and uncaring, but many defects just aren’t worth fixing because the downside potential of deploying the update overwhelms the likely benefit. Particularly when you are well into the next development cycle and can instead resolve the issue in the next feature release it often makes better business sense and customer satisfaction sense to leave the defect unaddressed and fold it into the future release.

If you’ve decided that the defect should wait, discuss this with the development team and your internal management and get consensus. This isn’t an easy conversation, but it’s made easier if you can show just how much effort, cost and opportunity loss there is in shipping an update for just this issue. Make sure that you leave the door open to reconsider, particularly if another issue shows up: Most of the overhead of deploying an update is essentially constant regardless of the number of issues resolved. This future potential will often push people to focus their thinking on whether this issue alone is worth all of the cost instead of talking in vague terms of commitment to quality and customer service.

Sidebar: Eliminate Build Overhead

While the overhead of creating a build and validating it is essentially a constant, that constant can be made significantly smaller with the right investments throughout the development process. The key is to automate as much of the process as possible. This broadly fits into the school of Continuous Integration or Continuous Builds, primarily because if you can’t automate the process you have no hope of doing it continuously.

• Automate Source Code to Install: Look at the process that takes raw source code and produces an install (be it a Windows Installer package, zip file, or RPM) and get humans entirely out of the loop. This can be done entirely with free software and by extending the tools you’re using already.
• Elevate Unit Testing to System Testing: Are you writing unit test libraries? If you’re using NUnit (or JUnit, or whatever) then look for ways to expose these to the build process and let the build run them every time. It doesn’t have to be this fancy – there just has to be a way of invoking tests during the build process, so this could be done with your own custom command line tool that exercises the system.

Automating the build process decreases your overhead costs during the primary development lifecycle and during maintenance. The overhead of the build is a tax on everything you produce that adds no value to your users, so focus on reducing it as much as you can. The great news is this game can be won an inch at a time: Incremental investments across your team can steadily improve your efficiency. There aren’t many other things you can do in the development process that pay off quickly and don’t require a major upfront investment.

As you gain experience with having an automated build and verification process you will find the entire team is more willing to tolerate risks because they know they have a large safety net in the automated verification process.

What Conversations Are You Having?

It’s easy to get pulled down into conversations that confuse the effort to fix the defect with the value of fixing it, or ignore the practical issues of deploying the fix or impact to other work that you can’t do because you’re pursuing the defect. Your development team will instinctively want to fix the defect – it will feel like an affront to their honor. Have the right conversations to bring everyone around to consensus on whether this one is worth it or not to what the team is trying to achieve.

As a manager and leader, your job is to generate buy-in for the decisions of the team and of the company. In the end, the worst mistake is pushing the development team where they don’t want to go. If they are determined to fix it, think hard about what the cost of letting them go ahead is. Perhaps the team can fix the defect but you don’t deploy it, deferring that cost until there’s enough value accrued to make it worth it. If they don’t think it’s worth it, perhaps it’s time for a field trip to commune with the users to understand the impact of the problem more clearly. The worst outcome would be if the team loses the passion to put in the time on all the details that have to be right to produce an outstanding product. Whatever the problem is, it isn’t going to be worth that cost.

Rarely will users identify the true underlying defect with the software: Most users know there’s a problem but can’t precisely define the true defect. Additionally, if the software was at least moderately tested before release then most defects that are visible to the end user are really multiple defects:

1. The problem the user reported.
2. The way the software handled the problem when it occurred.
3. The software design that allowed the user to get into trouble in the first place.

Typically, a user experiences a problem once the software has gone well off-track. The underlying problem began earlier than reported where it first jumped off the tracks (#1). It then snowballs until the user gets an odd message or experience sufficiently bizarre that they’re willing to report it (#2). It’s unlikely the software handles it in a pleasing and gentle way because if it did, that would mean you anticipated the problem and if that’s true, you would likely have found it in testing. You’ll want to make the software handle the problem more gracefully if something like it shows up in the future.

Finally, what was it in the overall architecture or design of the system that allowed the problem to get as far as it did without getting caught or corrected earlier (#3)? Perhaps there’s an underlying assumption that hardware is reliable or a file can’t be partially written to disk that needs to be reconsidered. This is the preventative medicine to catch all of the problems that are like the original problem. Once you understanding the basic design assumption that led to the problematic design in the first place, your team can usually see other decisions that sprung from the same thinking and look to address those before a user experiences a problem.

Side Note: Your users are already experiencing that problem too; they just haven’t reported it yet. How good are your feedback mechanisms?

It may not seem like there’s much of a risk or impact in attempting to diagnose the defect, but:

1. Diagnosis is unbounded: In most cases, determining the fundamental cause of a defect is the most time consuming part. It also defies estimation. You can time box diagnosis time to limit your exposure, but that’s not the same as being able to provide an estimate. Each defect represents the potential to throw time down a hole.
2. Workflow Impact: Your team is virtually guaranteed to be off busy on some new development or other project. They will likely have to shelve source code in a temporary state and shift back to a prior set of code to even diagnose the issue. Whatever the individual(s) involved were working on will need to stay on hold or be reassigned, complicating management and team productivity.

If your team doesn’t believe they can easily find the defect before they start looking into it, or they don’t believe it’s worth the effort, the defect is going to cost you more than time; it’ll cost you with the team. If it turns out the defect is easy to find, or while finding it the team discovers another issue they feel is more important then you’ll get lucky and face no longer term damage. More likely, the team members will add this to whatever other water they feel they’ve had to carry for you and the company. Before overruling your team, spend the time to either convince them that it’s worth it – or be convinced that it isn’t. If that doesn’t resolve the impasse, propose a time boxed approach. If you start with just a one day commitment to look into something this typically reduces resistance because you’ve eliminated the first problem (an unbounded time frame) and significantly reduced the workflow impact.

As your team looks into the problem, they will naturally come to believe that it’s both very important (because people perceive effort to equal value, so the more effort it takes the higher value it must have to justify that effort). This means they’ll tend to lose the perspective necessarily to objectively evaluate the risks and rewards of resolving the defect and deploying the fix. To help make these future conversations easier, don’t let the developers involved in the problem go too far off the reservation before revisiting the decision on whether it’s worth continuing to dig into the defect and what the potential upside to fixing it is.

Coming Next: The Resolution Perspective

Come back in a few days for the final post in this series, talking about the impact, difficulty and risks of resolving defects and deploying updates.

Your product is out in the wild, and even better – it’s in use by real users. You’ve got feedback and support structures in place and they are producing results. Now you need to take that feedback and incorporate it back into the product. To do this, you’re going to have to navigate the social dynamics of your organization around defects.

Any product, software or otherwise, has defects. Your shop may have a nice term to paper over it – incident, problem, ticket, errata, trouble report…. But let’s not paper over it – a defect is a defect. Developers also like to split hairs between feature requests and defects, but from a user standpoint it’s all defects. Everyone involved in product development will have their own way to prioritize defects, and to get the best results from your team’s time you need to be able to figure out which ones to address and how fast – and do so in a way that generates buy in from your development team, management, and customers.

There are an endless number of way to look at prioritization, but however you do it the discussions should include several perspectives:

• Impact, Difficulty, and Risk of defect to the end user.
• Impact, Difficulty, and Risk of even diagnosing the defect.
• Impact, Difficulty, and Risk of defect correction and deployment.

The End User Perspective

Your customers in general want and expect a defect-free product. Even if your average customer understands that all software has defects, intellectual understanding won’t overcome the emotional impact of running into a problem. Your users will generally start from the perspective that their problem is a defect in your software, it shouldn’t have been there in the first place, and you need to fix it immediately. Today would be nice.

It is very difficult to understand the value a customer places on fixing a particular issue from within the development team. Developers tend to grade defects based on the effort it takes to fix them and whether they produce an outright failure of the software. For example, few developers will get worked up over fixing a cosmetic defect such as a misspelling or alignment problem; If it’s that simple to fix, how valuable can it be? The exception to the natural cognitive bias that a problem must be hard to be worthy are problems that can crash the application or cause it to corrupt data. Few developers won’t see this as a deadly sin that must be resolved regardless of cost or risk.

Customers have a different perspective. They see just the surface veneer of your product and assume that it can never corrupt data or crash. Outright crashes have gotten rare enough that most users will refer to an error message as a crash. Because they have no idea what’s happening inside the black box that is your product, they will judge it solely on what they can see: Does it act like other applications, do the things they can see look clean and well crafted? Much like making a spelling error on the title page of your term paper can cause the entire work to be devalued, a small cosmetic error on the user interface can cause customers to doubt the correctness of your entire application.

Many end users will tend to discount defects that require long steps to produce or go away by restarting the application as long as they can convince themselves that they are at fault. This happens more often than you might expect – most users believe they don’t understand the rules behind the application and instead are using a rote procedure to accomplish their tasks. When something goes wrong, they will generally go back and try it again – possibly many times – before coming to the conclusion that there just might be a problem with the software and not with them. If the defect only presents occasionally they will usually write it off as their fault. Lest you think this behavior is limited to nontechnical users, this happened to NASA and resulted in a several day delay of the first Space Shuttle Flight.

With rare exception, if a defect isn’t judged as essential to fix by your customers, it’s probably not worth addressing prior to the next routine release. Every change to the software has consequences and takes effort that could go into something more important – to your team and your customers.

Coming Next: The Diagnostic Perspective

Come back in a few days for the next post in this series, talking about the impact, difficulty and risks of diagnosing and resolving defects.

Last week I was attending a training course on sales from a company I really respect – EntreQuest. One of the things I love about their courses and consulting is they aren’t shy about getting right to the fundamental (and often fundamentally hard) human basis for problems. One of the things they emphasize is that results are driven by process (including technology) which is in turn driven by mindset. If you don’t have the right mindset, you won’t achieve the results regardless of how much technology you throw at it. This is the basic justification for why the success rates of telemarketing (and other sales efforts that are all process, no mindset) are so low.

What was interesting to me in particular about this was how well it relates to conversations I typically have about reliability. Depending on where someone is in their experience curve they may talk about a particular technology, software development practice, or problem they’ve had. If they are really experienced, they go directly to either processes or culture. The very best tend to just talk about culture and mindset. This is bad news

In engineering the terms vary slightly, but I believe the principles are still completely valid: Results are driven by technology (Technology includes the processes, software, and hardware.), Technology is driven by mindset. When a mindset is held by a company, it’s called the culture. Your culture will exert a constant pressure on your technology like the current in a river: Either it will reinforce your goals or work against them.

You can make short term or localized improvements by focusing on just the results or technology, but to make a lasting change you need to be moving with the current.

Establish a Reliability Culture

Within your department or company (whatever scope you can influence), make reliability a fundamental aspect of who you are and how you solve problems. If you instill a mindset behind every discussion that your solutions will scale to a certain size, be continuously available, or other aspects of reliability, your technology choices will be imbued with this stance:

• Your development process will be designed to reduce or eliminate reliability risks. When your business partners ask for a change at the last minute, you won’t have to explain that all changes are risky.
• You won’t talk yourself into short-cutting testing. Instead, you’ll structure your development process to drive testing automation to reduce the cost of testing (allow you to run full tests more often) and ensure consistency.
• Your developers will naturally avoid low-reliability personal practices like being possessive about code, not commenting, incomplete or inconsistent error handling, and poor configuration management strategies.
• Your deployment environment will have appropriate hardware and software. You will be able to get proper monitoring tools and use hardware with sufficient redundancy and performance.
• Your business partners will be more receptive to conversations about schedules, knowing that under pressure they have to give on functionality instead of reliability.

As reliability becomes a core element of your culture, each individual will start to see the thousands of little decisions they make each day differently and unconsciously approach them from a perspective of reliability, as if they asked “what is the most reliable way to accomplish (whatever I’m doing now)”. At its best, it will shift things that happen as conflicts between people into corporate discussions – instead of your business partners feeling they have to convince you personally to add a new feature (viewing you as the roadblock) it will become how do we accommodate a business need within the context of our corporate goals for reliability. It is significantly easier to create a partnership in this scenario that has you understand their goals and them understand yours because you have a shared value and commitment to work within.

Reliability won’t always win out

Even in a reliability culture, there are very sound reasons to do certain things that entail risk. No one element of your culture is absolute, but it must always be respected and considered. For example, it could be that the system in question is an internal system that has a limited ROI. In this case it just isn’t appropriate to invest a great deal in reliability at the expense of ROI unless the system is unusable without it. Alternately, it is often appropriate during startup phases when the downside cost of a reliability problem is low (e.g. there are no or few existing users or no performance guarantee) or the mitigation cost is excessive (e.g. geodiverse hot sites).

Having reliability as a fundamental part of your mindset is still helpful in these situations because it ensures that a decision that impacts reliability is deliberately made and openly understood. As a company, you have to choose your battles and what risks you are going to mitigate. In some cases, it’s best to just run the risk and wait to see if it manifests before pouring energy into fixing it. Alternately, the risk may be scalability – if you are wildly successful, you’ll have to change your software to handle it. This is often called Technology Debt.

Taking on technology debt is often necessary when starting a product or venturing in new territory. The key is that the business and technology parties know that it’s a deliberate decision to take on that debt, instead of it being a quiet decision made just within the development team. That way if the risks turn out to become reality, the business doesn’t burn time arguing about how you got where you are and instead recognizes that it was a deliberate and well considered decision that now has a consequence that must be handled.

Reliability isn’t always Suitable

Not every company should have reliability as the defining element of its culture. It isn’t necessarily that these companies don’t want reliable results, it’s more that reliability isn’t their differentiator or important enough to be a core element of the culture. For example: Compare the Linksys and Cisco brands. Both can sell you a Wireless-G access point that on the mainline specifications are comparable: They support the same primary standards, offer comparable throughput and security features (for most people), and to many customers they would be indistinguishable.  However, Linksys tends to produce a model, make a few essential firmware updates and move on. If the unit needs to be reset periodically or a new device shows up on the scene that causes a problem with it that’s potentially OK. Customers that pay $70 instead of $700 for a wireless access point aren’t expecting the same degree of reliability. If Linksys attempted to do all of the reliability testing that Cisco does, they wouldn’t be able to hit the price points or time to market that drives their brand. Their product must be reliable enough that customers will find it suitable for the target market, but it isn’t necessary to pursue ultimate reliability.

Take a hard look around you. What level of reliability is appropriate for each area of your business? What are the reliability goals of the company? What is the prevailing culture? If you find yourself out of sync with your company’s goals on reliability, it could be that it isn’t the company that needs to shift but rather you may need to explore other options.

Change Begins at Home

The next time you’re frustrated by the results your team is achieving, don’t leap on the technology bandwagon first. Back up and look at how you might incorporate a reliability mindset into your own work as a starting point for catalyzing broader change. Have a series of conversations in your team to ensure you establish a common understanding of what your principles are – not just with reliability but other guiding principles as well. From that it will become easier to know what technologies (software, hardware, processes) will support the results you want to achieve. Start with your team and move out through your company, the results can speak for themselves.