Reliable Systems

In the past two years I’ve heard the term Technology Debt thrown around to justify a number of technology decisions.  In an effort to come up with a term that would bridge the business-technology gap, someone came up with Technology Debt to indicate that you were basically creating a future liability that would have to be paid back - rewriting a section of code or switching out a module, whatever.  Since business folks deal with assets and liabilities routinely, expressing subtle technology problems in financial terms has a lot of appeal.  The basic concept is most frequently used in relationship to software: Let’s say you defer something to a future release so you can get the release out the door.  Perhaps the feature will only work for very few customers but you don’t have time to generalize it, or the solution won’t scale as new users are added.

This is fine as far as it goes, but like any metaphor while it may be a way of explaining some aspects where two things are in common it’s very easy to overextend because in fact it isn’t a true financial liability.

Take a hypothetical example: You are supposed to add federated identity to your web application. What you want to do is create an identity broker that will allow your SaaS application to connect with Microsoft’s ADFS, OpenID, and a few variations of SAML. You believe this will get you the market reach you need, and by creating your own identity broker you can decouple your application from changes in this still evolving space, as well as support your own native security technique.

As you dig into it, you realize that this just isn’t feasible in the time you have: You need to get a solution done in two months to meet a commitment to a customer, and it turns out this customer just uses ADFS. It just so happens that your web framework can easily work with ADFS directly, so to save the schedule you drop back and just do ADFS. From a development standpoint this is a hack - you are doing something quickly you can’t extend to meet the original requirement and you’re pretty sure that you’ll need to undo this later and do it right, which will cost more than just doing it right the first time. This is Technology Debt.

Metaphors have Limited Application

The metaphor doesn’t hold for long.  First, unlike real debt there is no external requirement to pay this back.  Perhaps you never will need to support more than ADFS, or that after all of the talk customers just won’t adopt federated identity.  At the start of each release cycle, you can look at the competitive market and see what is the correct, most important work to be done.  It might be that you’ll have to make good on something you deferred, but you might not.  If you didn’t, it never was debt.  How confident are you that you can tell the difference right now?

Second, all technology has future liability. The more code you write, the more you have to maintain and support. That has a cost as well for the future. Depending on the nature of your product it could mean that you have to support questionable past API decisions or obscure and intricate features of your product. Every feature has a cost to maintain, every line of code you use or reference in a third party library has weight. Only talking about deferred development as technology debt implies that what you have right now is all asset, but it isn’t.

Rampant Misuse

The biggest challenge I’ve seen is that the metaphor is used to push development team goals on the business without having to adequately justify them.  By handing business folks something they can easily relate to, it’s easy to gloss over the underlying technical implications.

For example, say you have an application that’s currently written in Visual Basic 6.  You can justly claim that Microsoft has dropped support for it and that the day is coming when it will no longer run on the latest operating system (Microsoft has committed that it will run on Vista and Windows Server 2008, but that will likely be the end). This sounds very alarming indeed!  Naturally, the answer is to rewrite the entire application in .NET 3.5. Yes, this will take longer and cost a lot more than just adding the features you need to the existing code, but it eliminates all of the technology debt represented by that old nasty VB6 code.

Now look at it a little more objectively. Yes, you will need to eliminate that VB6 code at some point - notably when you are upgrading from Vista to whatever’s next.  When will that be?  Well, if you believe the hype that people love Windows XP and may just consider Vista in the future, you have some calendar time. As long as it’s done by then, Microsoft dropping support isn’t a real issue.  We’ve yet to work with a client who’s VB6 application didn’t work just fine on Vista, thank you very much.

Second, with rare exception every modern technology has a half-life. The notable exceptions are COBOL, C, and C++.  You can with confidence know that these languages are still going to be in active use in 15 years and that code you create now will work with minor to modest adjustment on current platforms at that time.  I would dispute if a similar claim can be made for the latest crop of languages.  A major reason for this is that modern languages are really whole environments - a programming language and an API/runtime.  While it may be theoretically possible to write a C# compiler without toting along the .NET runtime, it isn’t going to happen.  Likewise for PHP, RoR, and even Java.  Java’s framework is much more shallow which ironically will likely give it more life because it can adapt to radical changes in computing environments and there is already an intent to decouple most of the code from specific frameworks.

This means that today’s latest and greatest environment will be tomorrows VB6.  Just listen to people that jumped on .NET 1.1 discuss how they’re going to upgrade to .NET 2.0 or later, and that’s really not a particularly dramatic move.

Fundamentally, when you purchase software you’re making a bet in the vendor and community behind it. This is one place where the LAMP stack has a number of advantages because of the openness of the environment and the institutionalized tolerance of actively supporting releases for a long time. This is fundamentally enabled by their open source nature, but even if you had the source code for a commercial product it’s not likely you could do much with it; unless it’s fairly simple the burden of maintenance is likely higher than the cost of replacing it. Counteracting this are a lot of hidden costs to open source that may be difficult for a small team to absorb.

Easy Metaphors Seldom Produce Great Outcomes

Next time you’re trying to bridge the gap from technology to business, try to stay away from simple metaphors like Technology Debt. Instead, have real conversations to articulate the potential business impacts of the technical decisions and then hear from the business what they are concerned about. With a real dialog, everyone is in on why you deferred work to later and what bet is being made, and no one will be surprised when you do show up in a year to start porting that VB6 app to .NET.

Last week I was attending a training course on sales from a company I really respect - EntreQuest. One of the things I love about their courses and consulting is they aren’t shy about getting right to the fundamental (and often fundamentally hard) human basis for problems. One of the things they emphasize is that results are driven by process (including technology) which is in turn driven by mindset. If you don’t have the right mindset, you won’t achieve the results regardless of how much technology you throw at it. This is the basic justification for why the success rates of telemarketing (and other sales efforts that are all process, no mindset) are so low.

What was interesting to me in particular about this was how well it relates to conversations I typically have about reliability. Depending on where someone is in their experience curve they may talk about a particular technology, software development practice, or problem they’ve had. If they are really experienced, they go directly to either processes or culture. The very best tend to just talk about culture and mindset. This is bad news

In engineering the terms vary slightly, but I believe the principles are still completely valid: Results are driven by technology (Technology includes the processes, software, and hardware.), Technology is driven by mindset. When a mindset is held by a company, it’s called the culture. Your culture will exert a constant pressure on your technology like the current in a river: Either it will reinforce your goals or work against them.

You can make short term or localized improvements by focusing on just the results or technology, but to make a lasting change you need to be moving with the current.

Establish a Reliability Culture

Within your department or company (whatever scope you can influence), make reliability a fundamental aspect of who you are and how you solve problems. If you instill a mindset behind every discussion that your solutions will scale to a certain size, be continuously available, or other aspects of reliability, your technology choices will be imbued with this stance:

• Your development process will be designed to reduce or eliminate reliability risks. When your business partners ask for a change at the last minute, you won’t have to explain that all changes are risky.
• You won’t talk yourself into short-cutting testing. Instead, you’ll structure your development process to drive testing automation to reduce the cost of testing (allow you to run full tests more often) and ensure consistency.
• Your developers will naturally avoid low-reliability personal practices like being possessive about code, not commenting, incomplete or inconsistent error handling, and poor configuration management strategies.
• Your deployment environment will have appropriate hardware and software. You will be able to get proper monitoring tools and use hardware with sufficient redundancy and performance.
• Your business partners will be more receptive to conversations about schedules, knowing that under pressure they have to give on functionality instead of reliability.

As reliability becomes a core element of your culture, each individual will start to see the thousands of little decisions they make each day differently and unconsciously approach them from a perspective of reliability, as if they asked “what is the most reliable way to accomplish (whatever I’m doing now)”. At its best, it will shift things that happen as conflicts between people into corporate discussions - instead of your business partners feeling they have to convince you personally to add a new feature (viewing you as the roadblock) it will become how do we accommodate a business need within the context of our corporate goals for reliability. It is significantly easier to create a partnership in this scenario that has you understand their goals and them understand yours because you have a shared value and commitment to work within.

Reliability won’t always win out

Even in a reliability culture, there are very sound reasons to do certain things that entail risk. No one element of your culture is absolute, but it must always be respected and considered. For example, it could be that the system in question is an internal system that has a limited ROI. In this case it just isn’t appropriate to invest a great deal in reliability at the expense of ROI unless the system is unusable without it. Alternately, it is often appropriate during startup phases when the downside cost of a reliability problem is low (e.g. there are no or few existing users or no performance guarantee) or the mitigation cost is excessive (e.g. geodiverse hot sites).

Having reliability as a fundamental part of your mindset is still helpful in these situations because it ensures that a decision that impacts reliability is deliberately made and openly understood. As a company, you have to choose your battles and what risks you are going to mitigate. In some cases, it’s best to just run the risk and wait to see if it manifests before pouring energy into fixing it. Alternately, the risk may be scalability - if you are wildly successful, you’ll have to change your software to handle it. This is often called Technology Debt.

Taking on technology debt is often necessary when starting a product or venturing in new territory. The key is that the business and technology parties know that it’s a deliberate decision to take on that debt, instead of it being a quiet decision made just within the development team. That way if the risks turn out to become reality, the business doesn’t burn time arguing about how you got where you are and instead recognizes that it was a deliberate and well considered decision that now has a consequence that must be handled.

Reliability isn’t always Suitable

Not every company should have reliability as the defining element of its culture. It isn’t necessarily that these companies don’t want reliable results, it’s more that reliability isn’t their differentiator or important enough to be a core element of the culture. For example: Compare the Linksys and Cisco brands. Both can sell you a Wireless-G access point that on the mainline specifications are comparable: They support the same primary standards, offer comparable throughput and security features (for most people), and to many customers they would be indistinguishable.  However, Linksys tends to produce a model, make a few essential firmware updates and move on. If the unit needs to be reset periodically or a new device shows up on the scene that causes a problem with it that’s potentially OK. Customers that pay $70 instead of $700 for a wireless access point aren’t expecting the same degree of reliability. If Linksys attempted to do all of the reliability testing that Cisco does, they wouldn’t be able to hit the price points or time to market that drives their brand. Their product must be reliable enough that customers will find it suitable for the target market, but it isn’t necessary to pursue ultimate reliability.

Take a hard look around you. What level of reliability is appropriate for each area of your business? What are the reliability goals of the company? What is the prevailing culture? If you find yourself out of sync with your company’s goals on reliability, it could be that it isn’t the company that needs to shift but rather you may need to explore other options.

Change Begins at Home

The next time you’re frustrated by the results your team is achieving, don’t leap on the technology bandwagon first. Back up and look at how you might incorporate a reliability mindset into your own work as a starting point for catalyzing broader change. Have a series of conversations in your team to ensure you establish a common understanding of what your principles are - not just with reliability but other guiding principles as well. From that it will become easier to know what technologies (software, hardware, processes) will support the results you want to achieve. Start with your team and move out through your company, the results can speak for themselves.