Reliable Systems

When I was growing up I spent a lot of time with my father doing woodworking.  One lesson you pick up pretty quick when woodworking is that you have to keep the work clean at each step.

• If you take a piece of wood and don’t sand the surface smooth it won’t take a stain evenly.
• If you let glue creep out of the joint and get on the wood, the stain won’t look right in that spot.
• A piece of dust on the surface will get magnified by each layer of varnish.

Each layer depends on what’s underneath it.  Any flaw in a lower layer will tend to get magnified and distorted by layers above it.

Whenever I get involved in enterprise architecture I get reminded of this analogy because I often run into irrational exuberance that you can add a layer to an existing system and paint over the flaws underneath.  I was involved in a few projects like that early in my career:  It was too hard to talk directly to the mainframe from the web server so you put a layer in front of that.  There was already a C++ layer doing a DCE RPC gateway, but that was also too hard to program against for large use so we added a COM interface to the DCE RPC gateway.  We made some prototypes and validated the concepts and charged in full bore, only to run into big delays and teething problems near the end of the project trying to get everything polished up and suitable for production.

The problem is that at each turn you may be making the developer’s short term life easier by giving them an interface more natural to their preferred programming environment but since it just builds on the layers underneath it will end up with all of the limitations they have – and they can show up in the most surprising places.  For example, we ran into problems where certain input would cause failures which we ultimately discovered was caused by % being used as an insertion marker in a gateway library several layers underneath what we were doing – so at best it would drop the % and the following character, but at worst you’d get back a random data element if you managed to create a valid insertion name.

Layering issues are particularly problematic because they tend to be data sensitive and highly situational depending on how the various layers interact.  This means that it’s very difficult to design a comprehensive test plan:  The system can act as if it’s nondeterministic, making it infeasible to state with certainty that the various modes of the software have been demonstrated by a test plan.  At best, you can say that it worked for the exact test inputs it was given.  When you do have a problem in production, the multiple technologies in multiple layers can make it particularly hard to debug because it requires a lot of chairs around the table to hit all of the possible players.

Are you decorating?  Or covering up the problem?

Whenever you’re part of a team proposing adding a new layer over an existing system to fix its problems or adapt it to a new situation, you should be suspicious.  Is this really the right path to make the API look right?  Or are you temporarily covering over a problem?  If it’s the latter, it’ll just show through later – and now you have two problems to deal with not one.

There are good occasions to add a new layer:

• To smooth technology upgrades: When you are shifting technologies, say from COM to .NET, you may want to create a custom layer as a new standardized interoperability adapter which will let you separate the upgrade problem into phases and handle them independently.
• To support multiple technologies: Sometimes you need to support multiple types of clients – varying either by environment (say Java and .NET) or major architecture (say Client/Server and Web sites).

And a few suspect ones:

• Mitigate architecture risk: To isolate a new subsystem architecture from the main codebase. We’ve heard this one before – you want to try out something new and iffy, like say Entity Framework.  To contain the risk, you want to introduce a layer between it and the rest of the platform so if it all goes bad you can easily swap it out.
• Impedance Mismatch: You need to interact with something, but just don’t like the way it works. Perhaps it throws around ADO.NET recordsets and you prefer to work with strongly typed objects.

If you find yourself in one of the suspect scenarios, you should seriously question whether the work you’d do to create and validate a layer is really forward progress or just yak shaving.  Before you go down the path, you should seriously estimate:

• Fixing the underlying problems: If the underlying layer(s) aren’t doing what you need, what would it take to get them changed (in the technology they’re currently in) so you could work without adding another layer? That puts the responsibility where it belongs, and keeps complexity under control. Do a full estimate of this approach.
• Make a parallel layer: If you ignore the powerful aversion to creating duplicate routes to the same data, what if you created an alternate path to the underlying information. It may be that you bypass all of the layers or just some of the layers (such as down to the stored procedures that call the database). While this creates duplication, it lets each platform work in their own optimal way and allows for deterministic testing.
• Using the existing layer as it is: It’s easy to overstate the impact of reusing a known system with issues. There’s a natural tendency to not realize that you’re comparing a well understood but flawed system with an unknown solution with unknown problems. Trading known problems for unknown problems makes everyone happy at the start of a project, but creates significant project risk downstream.

Put down the shovel and back away

When you create a new layer on top of existing layers you are often digging your project into trouble, both now and downstream.  In addition to problems with each layer creating a leaky abstraction, deploying and supporting these highly layered systems is extraordinarily challenging.  It becomes prohibitively expensive to make changes in lower layers because of the high chance of unexpected side effects showing up as defects in dependent applications.  More often than not, each layer has to be held static with any changes accommodated by creating new queries or items at each layer to be served in parallel with the older methods.

Before you go ahead, be sure you look at the total lifecycle cost of that decision, including support and maintenance.   Have a good or bad experience with putting up some software wallpaper?  Let us know in the comments!

In the past two years I’ve heard the term Technology Debt thrown around to justify a number of technology decisions.  In an effort to come up with a term that would bridge the business-technology gap, someone came up with Technology Debt to indicate that you were basically creating a future liability that would have to be paid back - rewriting a section of code or switching out a module, whatever.  Since business folks deal with assets and liabilities routinely, expressing subtle technology problems in financial terms has a lot of appeal.  The basic concept is most frequently used in relationship to software: Let’s say you defer something to a future release so you can get the release out the door.  Perhaps the feature will only work for very few customers but you don’t have time to generalize it, or the solution won’t scale as new users are added.

This is fine as far as it goes, but like any metaphor while it may be a way of explaining some aspects where two things are in common it’s very easy to overextend because in fact it isn’t a true financial liability.

Take a hypothetical example: You are supposed to add federated identity to your web application. What you want to do is create an identity broker that will allow your SaaS application to connect with Microsoft’s ADFS, OpenID, and a few variations of SAML. You believe this will get you the market reach you need, and by creating your own identity broker you can decouple your application from changes in this still evolving space, as well as support your own native security technique.

As you dig into it, you realize that this just isn’t feasible in the time you have: You need to get a solution done in two months to meet a commitment to a customer, and it turns out this customer just uses ADFS. It just so happens that your web framework can easily work with ADFS directly, so to save the schedule you drop back and just do ADFS. From a development standpoint this is a hack – you are doing something quickly you can’t extend to meet the original requirement and you’re pretty sure that you’ll need to undo this later and do it right, which will cost more than just doing it right the first time. This is Technology Debt.

Metaphors have Limited Application

The metaphor doesn’t hold for long.  First, unlike real debt there is no external requirement to pay this back.  Perhaps you never will need to support more than ADFS, or that after all of the talk customers just won’t adopt federated identity.  At the start of each release cycle, you can look at the competitive market and see what is the correct, most important work to be done.  It might be that you’ll have to make good on something you deferred, but you might not.  If you didn’t, it never was debt.  How confident are you that you can tell the difference right now?

Second, all technology has future liability. The more code you write, the more you have to maintain and support. That has a cost as well for the future. Depending on the nature of your product it could mean that you have to support questionable past API decisions or obscure and intricate features of your product. Every feature has a cost to maintain, every line of code you use or reference in a third party library has weight. Only talking about deferred development as technology debt implies that what you have right now is all asset, but it isn’t.

Rampant Misuse

The biggest challenge I’ve seen is that the metaphor is used to push development team goals on the business without having to adequately justify them.  By handing business folks something they can easily relate to, it’s easy to gloss over the underlying technical implications.

For example, say you have an application that’s currently written in Visual Basic 6.  You can justly claim that Microsoft has dropped support for it and that the day is coming when it will no longer run on the latest operating system (Microsoft has committed that it will run on Vista and Windows Server 2008, but that will likely be the end). This sounds very alarming indeed!  Naturally, the answer is to rewrite the entire application in .NET 3.5. Yes, this will take longer and cost a lot more than just adding the features you need to the existing code, but it eliminates all of the technology debt represented by that old nasty VB6 code.

Now look at it a little more objectively. Yes, you will need to eliminate that VB6 code at some point – notably when you are upgrading from Vista to whatever’s next.  When will that be?  Well, if you believe the hype that people love Windows XP and may just consider Vista in the future, you have some calendar time. As long as it’s done by then, Microsoft dropping support isn’t a real issue.  We’ve yet to work with a client who’s VB6 application didn’t work just fine on Vista, thank you very much.

Second, with rare exception every modern technology has a half-life. The notable exceptions are COBOL, C, and C++.  You can with confidence know that these languages are still going to be in active use in 15 years and that code you create now will work with minor to modest adjustment on current platforms at that time.  I would dispute if a similar claim can be made for the latest crop of languages.  A major reason for this is that modern languages are really whole environments – a programming language and an API/runtime.  While it may be theoretically possible to write a C# compiler without toting along the .NET runtime, it isn’t going to happen.  Likewise for PHP, RoR, and even Java.  Java’s framework is much more shallow which ironically will likely give it more life because it can adapt to radical changes in computing environments and there is already an intent to decouple most of the code from specific frameworks.

This means that today’s latest and greatest environment will be tomorrows VB6.  Just listen to people that jumped on .NET 1.1 discuss how they’re going to upgrade to .NET 2.0 or later, and that’s really not a particularly dramatic move.

Fundamentally, when you purchase software you’re making a bet in the vendor and community behind it. This is one place where the LAMP stack has a number of advantages because of the openness of the environment and the institutionalized tolerance of actively supporting releases for a long time. This is fundamentally enabled by their open source nature, but even if you had the source code for a commercial product it’s not likely you could do much with it; unless it’s fairly simple the burden of maintenance is likely higher than the cost of replacing it. Counteracting this are a lot of hidden costs to open source that may be difficult for a small team to absorb.

Easy Metaphors Seldom Produce Great Outcomes

Next time you’re trying to bridge the gap from technology to business, try to stay away from simple metaphors like Technology Debt. Instead, have real conversations to articulate the potential business impacts of the technical decisions and then hear from the business what they are concerned about. With a real dialog, everyone is in on why you deferred work to later and what bet is being made, and no one will be surprised when you do show up in a year to start porting that VB6 app to .NET.

Last week I was attending a training course on sales from a company I really respect – EntreQuest. One of the things I love about their courses and consulting is they aren’t shy about getting right to the fundamental (and often fundamentally hard) human basis for problems. One of the things they emphasize is that results are driven by process (including technology) which is in turn driven by mindset. If you don’t have the right mindset, you won’t achieve the results regardless of how much technology you throw at it. This is the basic justification for why the success rates of telemarketing (and other sales efforts that are all process, no mindset) are so low.

What was interesting to me in particular about this was how well it relates to conversations I typically have about reliability. Depending on where someone is in their experience curve they may talk about a particular technology, software development practice, or problem they’ve had. If they are really experienced, they go directly to either processes or culture. The very best tend to just talk about culture and mindset. This is bad news

In engineering the terms vary slightly, but I believe the principles are still completely valid: Results are driven by technology (Technology includes the processes, software, and hardware.), Technology is driven by mindset. When a mindset is held by a company, it’s called the culture. Your culture will exert a constant pressure on your technology like the current in a river: Either it will reinforce your goals or work against them.

You can make short term or localized improvements by focusing on just the results or technology, but to make a lasting change you need to be moving with the current.

Establish a Reliability Culture

Within your department or company (whatever scope you can influence), make reliability a fundamental aspect of who you are and how you solve problems. If you instill a mindset behind every discussion that your solutions will scale to a certain size, be continuously available, or other aspects of reliability, your technology choices will be imbued with this stance:

• Your development process will be designed to reduce or eliminate reliability risks. When your business partners ask for a change at the last minute, you won’t have to explain that all changes are risky.
• You won’t talk yourself into short-cutting testing. Instead, you’ll structure your development process to drive testing automation to reduce the cost of testing (allow you to run full tests more often) and ensure consistency.
• Your developers will naturally avoid low-reliability personal practices like being possessive about code, not commenting, incomplete or inconsistent error handling, and poor configuration management strategies.
• Your deployment environment will have appropriate hardware and software. You will be able to get proper monitoring tools and use hardware with sufficient redundancy and performance.
• Your business partners will be more receptive to conversations about schedules, knowing that under pressure they have to give on functionality instead of reliability.

As reliability becomes a core element of your culture, each individual will start to see the thousands of little decisions they make each day differently and unconsciously approach them from a perspective of reliability, as if they asked “what is the most reliable way to accomplish (whatever I’m doing now)”. At its best, it will shift things that happen as conflicts between people into corporate discussions – instead of your business partners feeling they have to convince you personally to add a new feature (viewing you as the roadblock) it will become how do we accommodate a business need within the context of our corporate goals for reliability. It is significantly easier to create a partnership in this scenario that has you understand their goals and them understand yours because you have a shared value and commitment to work within.

Reliability won’t always win out

Even in a reliability culture, there are very sound reasons to do certain things that entail risk. No one element of your culture is absolute, but it must always be respected and considered. For example, it could be that the system in question is an internal system that has a limited ROI. In this case it just isn’t appropriate to invest a great deal in reliability at the expense of ROI unless the system is unusable without it. Alternately, it is often appropriate during startup phases when the downside cost of a reliability problem is low (e.g. there are no or few existing users or no performance guarantee) or the mitigation cost is excessive (e.g. geodiverse hot sites).

Having reliability as a fundamental part of your mindset is still helpful in these situations because it ensures that a decision that impacts reliability is deliberately made and openly understood. As a company, you have to choose your battles and what risks you are going to mitigate. In some cases, it’s best to just run the risk and wait to see if it manifests before pouring energy into fixing it. Alternately, the risk may be scalability – if you are wildly successful, you’ll have to change your software to handle it. This is often called Technology Debt.

Taking on technology debt is often necessary when starting a product or venturing in new territory. The key is that the business and technology parties know that it’s a deliberate decision to take on that debt, instead of it being a quiet decision made just within the development team. That way if the risks turn out to become reality, the business doesn’t burn time arguing about how you got where you are and instead recognizes that it was a deliberate and well considered decision that now has a consequence that must be handled.

Reliability isn’t always Suitable

Not every company should have reliability as the defining element of its culture. It isn’t necessarily that these companies don’t want reliable results, it’s more that reliability isn’t their differentiator or important enough to be a core element of the culture. For example: Compare the Linksys and Cisco brands. Both can sell you a Wireless-G access point that on the mainline specifications are comparable: They support the same primary standards, offer comparable throughput and security features (for most people), and to many customers they would be indistinguishable.  However, Linksys tends to produce a model, make a few essential firmware updates and move on. If the unit needs to be reset periodically or a new device shows up on the scene that causes a problem with it that’s potentially OK. Customers that pay $70 instead of $700 for a wireless access point aren’t expecting the same degree of reliability. If Linksys attempted to do all of the reliability testing that Cisco does, they wouldn’t be able to hit the price points or time to market that drives their brand. Their product must be reliable enough that customers will find it suitable for the target market, but it isn’t necessary to pursue ultimate reliability.

Take a hard look around you. What level of reliability is appropriate for each area of your business? What are the reliability goals of the company? What is the prevailing culture? If you find yourself out of sync with your company’s goals on reliability, it could be that it isn’t the company that needs to shift but rather you may need to explore other options.

Change Begins at Home

The next time you’re frustrated by the results your team is achieving, don’t leap on the technology bandwagon first. Back up and look at how you might incorporate a reliability mindset into your own work as a starting point for catalyzing broader change. Have a series of conversations in your team to ensure you establish a common understanding of what your principles are – not just with reliability but other guiding principles as well. From that it will become easier to know what technologies (software, hardware, processes) will support the results you want to achieve. Start with your team and move out through your company, the results can speak for themselves.