Reliable Systems

If you’ve read more than one or two articles from Reliable Systems you probably have gotten the sense that we worry a lot about how to make things just work.   It’s that quality of anything where you get what you expect and what you need every time.  It can be in an experience (like a fun drive down a country road) or a product.  As a company if you can do this over and over you create a brand people develop a strong emotional connection to:  Apple, John Deere, Starbucks…

When you want to create a product that just works, you need to get all of the details right – from packaging through to maintenance and upkeep.  It’s not one thing that’s important, it’s all the things.  We are often engaged by senior management within a client when things aren’t working, and there’s conflicting opinions on why.  Usually along the path technology is being blamed: Not enough, not the latest thing, not someone’s favorite thing, not working.  As we dig into the situation, rarely is the technology the dominant factor:  More often, it’s how the technology is being integrated with the people and processes that all have to work together.

One of the first things we have to do in these engagements is to establish the real facts on the ground:  What exactly are the problems in the system, who’s doing what with it, how many times.  It comes down to establishing metrics to make sure time and attention are paid to the parts that make the biggest difference in the outcome.  Armed with these facts in a form the business can consume it’s possible to create plans of action that deliver virtually regardless of budget.

So let’s make this easier

The biggest trick is then getting the facts you need on an ongoing basis, easily, and in a form that the business can consume.  For over a decade we’ve been building instrumentation right into the systems we’ve worked on.  We’ve created a variety of toolkits to make this easier over the years, refining them as technology and our experience has changed.

About 18 months ago we decided it was time to really invest down this path.  We believe in routinely capturing key computer metrics along with whatever logging the application can do on its own.  We won’t do a project without using a great logging system that includes a strategy for managing runtime exceptions.   Now that we’re collecting all this data, we need to have a way of managing the raw data and turning it into valuable business data.

The challenge is that businesses don’t get up in the morning and say “what our customers want us to do is have great internal tools”, so you’re nearly always doing this on the cheap:  Borrowing time from development projects internally to cobble together various free or cheap solutions.  Frankly, we got tired of having to create new solutions with each client out of the margins of each project.  So, we pooled our best thinking from all of the work we’ve done (including a previous product that we did license to our clients over the past decade called CLAS) and started creating Gibraltar.

Rock Solid from Initial Release

With Gibraltar we wanted much more than a log system.  Of course, it had to be a log system too – and a really easy to use one that could work with each of our client applications.  More than that, it had to:

• Automatically capture all of the performance metrics we wanted.
• Integrate with existing logging available on the platform, including whatever a client might already be doing (like custom in-house options)
• Be absolutely, positively, for sure safe to run in production no matter what.   That means it can’t ever use too much disk space or disk throughput or block the application.
• Not use more than 5% of the performance of the app
• Include all of the tools necessary to get data from where it was collected to the people that could get value out of it
• Include the ability to look at the detailed session data up to high level analysis:  What’s the error rate?  What’s it correlate to?  Are we doing better or worse in this version?

From this initial sketch into everything we wanted, we’ve spent 18 months including four beta periods (from 2-4 months each) to refine the vision with real customers and real scenarios.  It was essential to us that this not be just a tool for techies but be ready for use by people with a wide range of skills.  It had to be pretty and just do what you wanted, when you wanted it to.

We’ve added a lot of capabilities along the way:  It can generate print-ready reports about application reliability that can communicate with senior management, you can define all kinds of custom metrics to easily track how your application is used and by whom.  We ran a number of betas to be sure that we had hit every goal we have above.  We’re happy to report that Gibraltar is in use within large deployments of custom applications, commercial applications, and small deployments right down to our corporate web site.

This tool isn’t for everyone – Our clients are nearly all Windows shops, and if they do any custom development it’s almost invariably in .NET, so that’s what we’ve targeted.   But, if you’re interested in easily getting real data on not just infrastructure (how well the application is running) but whether or not it just works, have we got an easy path for you.  You can see a quick demo video of how it works technically at Gibraltar Software.

You also don’t have to take my word for it at all, you can hear what one of our beta users did with it, which is really a more compelling story than what we might say.

I think you’ll find that our work sweating a lot of little details, from the exact design of the API and making sure the documentation was complete to rewriting our own licensing system to be very IT Admin friendly.  If we didn’t get a detail right, we want to know.  And the great news is that we’ve just begun:  We’re obsessed with the little things, and you can bet we’ll keep listening and watching to make it better.  Of course, this is made a lot easier because we’re using Gibraltar to monitor itself, and a select group of our users is sending that information back to us so we can make sure it just works in the field for real people.

It’s easy to start your journey

If you do development for Microsoft .NET, I’d encourage you to go over and download our commercial release of Gibraltar.  You’ll get great documentation, a free agent you can use like a flight recorder “black box” in every application you create, and a trial for a tool that will make you seem wise beyond your years.  And if you pay us the ultimate honor and purchase a permanent license, I can assure you that you won’t find anyone more committed to your satisfaction than we are.

Everyone likes to be on the winning team.  We love to root for our favorite sports team, we like the car we own and the brand behind it.  So it’s no surprise that when Apple ran their I’m a Mac ads that Windows fans were in an uproar.  Now with the Laptop Hunter series the shoe’s on the other foot.   Microsoft is making a big show that Apple computers are overly expensive just for the Apple brand.  Apple fans claim that  to match a Mac, a PC has to be equipped with tons of antivirus software, a full time tech support guy, and a Witchdoctor on standby to keep it working.

Seriously people

First, Apple makes some of the finest hardware you can possibly buy.  If you compare it nose to nose with hardware that’s actually built to the same standards then it really doesn’t represent a significant price premium.  Compare a Macintosh Pro with an equivalent Dell workstation – the cost is within 5%.  It’s amazing that Apple can afford the extra engineering of an OS with that little premium.  

Second, Vista works great.   It’s running on many more systems than Mac OS is, and with volume comes a range of new problems.  The total amount of money I’ve spent on desktop antivirus software in 10 years of administering PCs?  $0.  The total number of virus problems I’ve had? 0.  My parents managed to get into trouble with one virus and Windows XP – but installing the (free) Microsoft Defender cleared it right up never to return.

As with nearly all marketing, this is a battle of perception:  Apple has done a great job of defending their brand at every turn.  This is part of their corporate ethos.  Along with a few other tenants it ensures they are a much loved but niche player:

• Only do something you can do uniquely well.
• Don’t extend into markets that might ask you to compromise your values.
• Cultivate the mystique:  Don’t show what’s behind the curtains.

Microsoft on the other hand has tenants that ensure they’ll be a volume player, but an unloved one:

• Play to win the most market share in any market you can.
• Build your ecosystem by making it easy for others to add value to it.
• Cultivate the engineers:  Provide overwhelming amounts of documentation and approaches.

The fact is, many people don’t need a top end piece of hardware like a Mac.  On the other hand, many people want a computer that’s just a tool, not a piece of art.  To them, the nearly infinite diversity and low cost of entry are essential.

I’m a People person.  I’m Good with People!

The computing needs of the average corporation and the average individual are very far apart.  To companies, computers are tools – like the desk, phone, and copier.  Very important, very powerful – tools.  They aren’t there to make you feel great or enable you to create a cool video of your vacation in France.  My partner really summed it up one day when he commented that the Mac was a really personal computer - it worked hard to create a personal connection.  

Corporations on the other hand want slow paced evolution, massive support for legacy applications and hardware (these guys are still running dot matrix printers off parallel ports) and to control costs.  They also philosophically want to have all the keys to the computers – just like they do for the buildings and offices they own.  PC’s are just end points on the large mesh that is the corporate IT network.  It’s very impersonal.

Microsoft makes a great deal of money providing businesses with the tools they need to have the computers work for them, and Apple makes a great deal of money creating computers that people love.  Either of these goals would be compromised by trying to do both.

Vista Goggles

Folks that have been in the Windows ecosystem a long time probably recognize that you could take the first year of press about Vista and substitute “Windows 2000″ and find the same article written 8 years earlier.  Vista is a surprisingly large and tricky step forwards on a number of fronts, whereas Windows XP was a visual redress of Windows 2000.  

Almost like an SAT test:

Windows 7 is to Windows Vista as

Windows XP is to Windows 2000.  

Like Windows XP, the story on Windows 7 is making virtually no architecture changes and instead just tuning for the long haul.  That’s a great thing, because there’s a lot that works very well with Vista, and now it’ll work even better with Windows 7.

The humorous thing is to read now about how people are thinking about moving to Vista once 7 ships because, well, they don’t want to move to an OS that was just released.  It’s as if Vista has been aging like a fine cheese on the shelf so the very same binary code that once was toxic is now just what the doctor ordered.  To a slight degree this is true:  Vista SP1 did address some issues that affected some people, and more importantly hardware now is dramatically faster than it was two years ago (as it will be two years from now…) so what once was aggressive is now commonplace.  The same was true of Windows 2000 when it shipped.  Requiring 64MB of RAM?  That’s just crazy talk!  Only certain video cards worked reasonably with it, and video drivers to a while to stabilize.  That sounds very familiar…

In the end, it really comes back to Perception.  Probably the biggest mistake Microsoft did was not push the OEM’s that make the computers to build machines that could responsibly run the new operating system, and be clear that meant hardware 3D video cards and plenty of memory.  And oh yeah, stop putting aftermarket firewalls, antivirus, Google Desktop, and all kinds of other things on them that are ill optimized.  At my last company we got in the habit of routinely wiping each new Dell that came in and reinstalling the OS from the Dell restore CD – because that got rid of all the noise.  It was surprising how much better that worked.  Is that Microsoft’s fault?  Not directly, but they certainly could have found a way to encourage the ecosystem to forgo some profit for usability.  But that’s just not in their corporate DNA.

With any luck, the big story for Windows 7 will be that Microsoft pushes back against their channel, even being willing to risk it by leaving Windows XP out there for folks that don’t want to play by the Windows 7 rules.  It’s hard to put up barriers when you’re a legal monopoly, so find ways to use incentives to do it right instead of punishment for doing it wrong.  And keep up the ads, because perception does matter in the long run.

Who knows, it may push Apple to get better too.  Just once when my iPod updates itself to enhance stability and performance I’d love to know what exactly was unstable or slow…

As you can tell from the timeline It’s been a while since I’ve posted anything.  It isn’t because I’ve had nothing to say – instead, I’ve been completely consumed by leading the team creating Gibraltar, a new application monitoring product for .NET teams that we’re launching.  You can download the latest version at  www.GibraltarSoftware.com.  We just published the last beta version of the product before the commercial release which is scheduled for June 1, 2009.

Bring a new product to market is really hard.  I’m sure you’ve heard that before – but however hard you think it is, it’s harder than that.  While we’re not quite across the finish line, there are a few things that have become readily apparent:

• Commercial-grade quality takes a lot to achieve.  At each turn where you might normally say “well, users just shouldn’t do that” you can’t.  Things you otherwise solved through training you can’t.  It’s the difference in construction of a commercial Amp and the receiver you bought at Best Buy.  
• Users won’t read anything.  We did a beta release where we posted in five places instructions for how to upgrade from the prior beta which required an extra step or things wouldn’t work.  We got deluged with calls about it not working from virtually every beta user; no one read any of the notes they saw, even in bold text in a yellow box in the middle of the screen.
• Marketing involvement early and often:  The feedback from our first beta version was brutal; it told us that we were going in entirely the wrong direction because the users we were building the app for weren’t going to buy anything regardless of how singing & dancing it was.  We had to step back and go a whole different direction.  That would have been far more painful if we hadn’t been early in the process.

From here on out, I’ll be contributing to a separate blog articles that are focused on .NET software development and being part of a small Independent Software Vendor (ISV).  This site will focus in more on its original goal:  IT and business strategies for reliable systems.

Regardless of how trivial the defect is there are very real costs and risks to resolving it. Let’s say it’s as simple as a misspelling on a text label, so it’s both really easy to fix and really easy to ensure you fixed it. You still have to contend with:

1. Every Build is a Risk: Every time you package up a set of files as a build, there’s a risk of error. If your build isn’t entirely automated – and entirely means from source code through install – you run the risk of something being done wrong. More likely, the risk may be something external: Unknowingly including a newer version of a referenced library or introducing a dependency on a newer version. Either way, you need to do significant regression testing to mitigate that risk.
2. Deployment Risk: The update will need to get from your development environment to your users. Whether it’s a Software as a Service (SaaS) product that just needs to hit some web servers or packaged software deployed to thousands, your update will need to be installed for people to get any advantage of it. In most cases this will mean a special upgrade installation, notification to existing customers to come and get the upgrade, and additional support for your users.

The truth is that most defects aren’t as clear cut as a spelling error, so you will also have to contend with the possibility that no matter how well intentioned, your fix is going to cause new problems for your customers. It could be that there are advantageous side effects of the current (defective) implementation or that your fix doesn’t work on the Elbonian version of Windows XP which you didn’t discover because you did only a focused test of the fix on your key target platforms. In more elaborate cases, it could be that the loophole represented by the defect is viewed by some of your users as a feature, so fixing it makes your product less valuable. This is more likely when doing defect patching because you typically don’t have the benefit of a beta cycle and end-user involvement in considering all of the aspects of the fix.

The Last Change is to Blame

If you have the opportunity, try this experiment some time:  Announce a new version that never really happened.  Perhaps you just relabel a prior version with a new number or something else to create the placebo effect of an update.  What you will discover is:

• Surprise Fixes: Some group of your users will thank you for the new version.  It’s so much faster than the old one!  Oh, and you fixed a problem they’ve had for months.
• Surprise Defects: Unfortunately much more common than surprise fixes are the number of people that will report a problem that must have been caused by your update because it happened just after they installed it.  It could be as wide ranging as their hard drive died or Word lost its dictionary.  But they’re sure it’s your fault.
• Reinstall Rash: Some contingent of users will have problems installing the upgrade.  The problem will vary depending on how you deploy your fix, but they’ll manage to get a computer or two out of sorts over it.  Don’t think this is a Windows problem either, just look at the volume on support forums for WordPress right after a minor update.

In this case, there isn’t much you can do to minimize the problems because… you didn’t create them (after all, it’s the same software – that was the point of the test).  With the possible exception of finding better ways of deploying fixes, there just isn’t a lot you can do.  This is the minimum end-user overhead to every upgrade you make, and they’re going to make it your overhead.  The big investment you can make to minimize this is:

1. Cultivate Your Brand: If customers love you, they won’t make the leap from coincidence (two things happened at the same time) to causality.  The more they love you, the more they’ll be sure they are at fault.
2. Make Upgrades Easy: You really want to invest in ways that make updates easy.  Look at Firefox and Windows Updates for examples of really great ways to get updates out the door.   It’s easy and surprisingly trouble free, much more so than relying on users to manually know whether to uninstall the old version first, whether an update applies to them, etc.

What Are You Committed To?

It may seem cold and uncaring, but many defects just aren’t worth fixing because the downside potential of deploying the update overwhelms the likely benefit. Particularly when you are well into the next development cycle and can instead resolve the issue in the next feature release it often makes better business sense and customer satisfaction sense to leave the defect unaddressed and fold it into the future release.

If you’ve decided that the defect should wait, discuss this with the development team and your internal management and get consensus. This isn’t an easy conversation, but it’s made easier if you can show just how much effort, cost and opportunity loss there is in shipping an update for just this issue. Make sure that you leave the door open to reconsider, particularly if another issue shows up: Most of the overhead of deploying an update is essentially constant regardless of the number of issues resolved. This future potential will often push people to focus their thinking on whether this issue alone is worth all of the cost instead of talking in vague terms of commitment to quality and customer service.

Sidebar: Eliminate Build Overhead

While the overhead of creating a build and validating it is essentially a constant, that constant can be made significantly smaller with the right investments throughout the development process. The key is to automate as much of the process as possible. This broadly fits into the school of Continuous Integration or Continuous Builds, primarily because if you can’t automate the process you have no hope of doing it continuously.

• Automate Source Code to Install: Look at the process that takes raw source code and produces an install (be it a Windows Installer package, zip file, or RPM) and get humans entirely out of the loop. This can be done entirely with free software and by extending the tools you’re using already.
• Elevate Unit Testing to System Testing: Are you writing unit test libraries? If you’re using NUnit (or JUnit, or whatever) then look for ways to expose these to the build process and let the build run them every time. It doesn’t have to be this fancy – there just has to be a way of invoking tests during the build process, so this could be done with your own custom command line tool that exercises the system.

Automating the build process decreases your overhead costs during the primary development lifecycle and during maintenance. The overhead of the build is a tax on everything you produce that adds no value to your users, so focus on reducing it as much as you can. The great news is this game can be won an inch at a time: Incremental investments across your team can steadily improve your efficiency. There aren’t many other things you can do in the development process that pay off quickly and don’t require a major upfront investment.

As you gain experience with having an automated build and verification process you will find the entire team is more willing to tolerate risks because they know they have a large safety net in the automated verification process.

What Conversations Are You Having?

It’s easy to get pulled down into conversations that confuse the effort to fix the defect with the value of fixing it, or ignore the practical issues of deploying the fix or impact to other work that you can’t do because you’re pursuing the defect. Your development team will instinctively want to fix the defect – it will feel like an affront to their honor. Have the right conversations to bring everyone around to consensus on whether this one is worth it or not to what the team is trying to achieve.

As a manager and leader, your job is to generate buy-in for the decisions of the team and of the company. In the end, the worst mistake is pushing the development team where they don’t want to go. If they are determined to fix it, think hard about what the cost of letting them go ahead is. Perhaps the team can fix the defect but you don’t deploy it, deferring that cost until there’s enough value accrued to make it worth it. If they don’t think it’s worth it, perhaps it’s time for a field trip to commune with the users to understand the impact of the problem more clearly. The worst outcome would be if the team loses the passion to put in the time on all the details that have to be right to produce an outstanding product. Whatever the problem is, it isn’t going to be worth that cost.

Rarely will users identify the true underlying defect with the software: Most users know there’s a problem but can’t precisely define the true defect. Additionally, if the software was at least moderately tested before release then most defects that are visible to the end user are really multiple defects:

1. The problem the user reported.
2. The way the software handled the problem when it occurred.
3. The software design that allowed the user to get into trouble in the first place.

Typically, a user experiences a problem once the software has gone well off-track. The underlying problem began earlier than reported where it first jumped off the tracks (#1). It then snowballs until the user gets an odd message or experience sufficiently bizarre that they’re willing to report it (#2). It’s unlikely the software handles it in a pleasing and gentle way because if it did, that would mean you anticipated the problem and if that’s true, you would likely have found it in testing. You’ll want to make the software handle the problem more gracefully if something like it shows up in the future.

Finally, what was it in the overall architecture or design of the system that allowed the problem to get as far as it did without getting caught or corrected earlier (#3)? Perhaps there’s an underlying assumption that hardware is reliable or a file can’t be partially written to disk that needs to be reconsidered. This is the preventative medicine to catch all of the problems that are like the original problem. Once you understanding the basic design assumption that led to the problematic design in the first place, your team can usually see other decisions that sprung from the same thinking and look to address those before a user experiences a problem.

Side Note: Your users are already experiencing that problem too; they just haven’t reported it yet. How good are your feedback mechanisms?

It may not seem like there’s much of a risk or impact in attempting to diagnose the defect, but:

1. Diagnosis is unbounded: In most cases, determining the fundamental cause of a defect is the most time consuming part. It also defies estimation. You can time box diagnosis time to limit your exposure, but that’s not the same as being able to provide an estimate. Each defect represents the potential to throw time down a hole.
2. Workflow Impact: Your team is virtually guaranteed to be off busy on some new development or other project. They will likely have to shelve source code in a temporary state and shift back to a prior set of code to even diagnose the issue. Whatever the individual(s) involved were working on will need to stay on hold or be reassigned, complicating management and team productivity.

If your team doesn’t believe they can easily find the defect before they start looking into it, or they don’t believe it’s worth the effort, the defect is going to cost you more than time; it’ll cost you with the team. If it turns out the defect is easy to find, or while finding it the team discovers another issue they feel is more important then you’ll get lucky and face no longer term damage. More likely, the team members will add this to whatever other water they feel they’ve had to carry for you and the company. Before overruling your team, spend the time to either convince them that it’s worth it – or be convinced that it isn’t. If that doesn’t resolve the impasse, propose a time boxed approach. If you start with just a one day commitment to look into something this typically reduces resistance because you’ve eliminated the first problem (an unbounded time frame) and significantly reduced the workflow impact.

As your team looks into the problem, they will naturally come to believe that it’s both very important (because people perceive effort to equal value, so the more effort it takes the higher value it must have to justify that effort). This means they’ll tend to lose the perspective necessarily to objectively evaluate the risks and rewards of resolving the defect and deploying the fix. To help make these future conversations easier, don’t let the developers involved in the problem go too far off the reservation before revisiting the decision on whether it’s worth continuing to dig into the defect and what the potential upside to fixing it is.

Coming Next: The Resolution Perspective

Come back in a few days for the final post in this series, talking about the impact, difficulty and risks of resolving defects and deploying updates.