Reliable Systems

In software development, you’re always being asked to estimate things:  How long will the whole project take?  Just this feature?  What if we changed this feature to remove this aspect?   This is all part of the feedback cycle that is fundamental to product creation:  We have a certain amount of time and money for a given set of functionality, but if there’s something really juicy and it just takes a little more time, then maybe we adjust, or if we can get a lot of value for a little effort, perhaps we do a little mini release first.  The business decisions feed the development estimates which in turn inform new business opportunities.

Getting the feedback wrong can be disastrous: The right functionality late can kill your business; perhaps half a loaf earlier is better.  On the other hand, some things the market won’t accept half way so a full loaf it is.  The business needs to trust the information it’s getting isn’t random or capricious to make good decisions, and the development team needs to be able to provide a best guess without fear of misunderstanding.

It’s natural to pad estimates with the idea that it’s better to under promise and over deliver – so that means to guess long and come in well early.  But in the end, is that really any better?

You’re Guessing, and Possibly Lucky

We’ve been experimenting with the new Evidence Based Scheduling features of FogBugz (which we use internally for managing our software development) and one thing that it highlights quickly is that estimation isn’t good if you’re early, bad if you’re late – it’s about getting your average as close to the mark as you can.  Take a look at a graph of most of my estimates:

Ideally, the graph line would have a 1:1 slope, indicating that on average you are accurate.  Further, you want your estimates clustered pretty near to the line itself.  What you can see from my estimate curve is that I’m uneven – I tend to underestimate shorter tasks (under 1.5 hours) and overestimate longer tasks (and that’s after removing some really bad outliers…).  But notably if you draw a ruler on the 1:1 line you’ll see that I’m not even close. Don’t let the hash lines fool you – look at the numbers to see.  The thing is, the other developers in our company that are all regarded as skilled, senior developers aren’t particularly more accurate on any one estimate, and the averages work out similarly.

So what?

Why isn’t it a great thing when we beat our estimates?  There are several potential pitfalls:

Features based on Effort

We’d all like to believe in the rosy model that our customers ask for features and then we build them into the software, so if we’re done early then it’s a pure win.  It’s my experience that it’s much more like a game of Tetris:  What features we take on is dependent on how much effort we think they’ll take.  Every feature has an amount of effort above which it isn’t worth it any more.  When hashing out what makes it and what doesn’t, the effort estimate is a big factor.

If we overestimate the effort of features, then we are slanting the project management decisions away from customer-selected features in favor of the developer’s whims.  This is because some features will be estimated to take more effort than they’re worth, and a more invisible internal team dynamic.  If a project is doing well on schedule, it’s very human to take advantage of this to try out newer, riskier things, over-engineer a feature, or do other things within the team that would otherwise be successfully argued against because of their effort.  In general, the more time available, the more yak shaving the team will do.

Once a schedule is accepted, the business will tend to act on it as fact:  Customers that can’t wait for it will end up being turned away and others will be promised a schedule.  This is a necessary but painful aspect:  Developers are generally optimists and will not want to say no to a customer even though it’s generally not in the best interest of either the company or customer to rush a feature. You want the business to stick with your decisions and not pass the buck on saying no to the customer, but you also need them to trust that it’s a fair trade.

Approach based on Effort

Within the development team, decisions are made at every level on how to implement a feature with an eye towards both the feature’s estimate and the overall project’s status.  Even when not explicitly laid out, a team that believes the overall schedule is tight will feel pressured to find ways to reduce the effort on anything they do.  This means when deciding between a careful implementation that may take longer but be more scalable or easier to support the team will often opt for a more direct path to completion even if the estimate was based on the more careful approach.

If done as a conscious decision in consultation with the project’s sponsors, this can be a way of bringing the project back on track but really it’s just another way of cutting functionality to make schedule:  You’re going to cut out something you intended to deliver (say a more generalized, upgradeable framework for reporting) even though you meet the letter of the requirement in front of you (delivering a few reports).  This can lead to nasty surprises for the team down the road when your sponsor’s find out that they didn’t get what they thought they would.

Alternately, if you overestimate one feature it may have put another feature under pressure so a more expedient and risky approach was adopted for it to fit it into the schedule.  If the true effort had been known, a different decision could have been made.

Make Your Guesses A Coin Toss

In the end, being early and being late just have different ways they create problems for your development project.  Your goal when estimating is to not try to find the estimate that has the highest probability of being sufficient to get the job done but instead the estimate that is equally likely to be high as low.  In aggregate if you have enough of these items on your project (say more than 25) then you’re entire project’s estimate should also be just as likely high as low.

There is still a place for the traditional high estimate: When you move outside of the project sponsor and the development team to users that need a guarantee.  There the downside of missing a date is much worse than the impact of being early.

On your next project, try out the 50/50 approach and make it clear to both the development team and the business.  You’ll probably notice that people develop a more subtle appreciation for the fact that estimates are based on probabilities.  This can help you skip over the discussions that aren’t useful about why you are where you are and instead keep focusing on the business goals for your current situation.

Back in the day, corporate remote access meant modem pools that people dialed into from wherever they were. Even then it was like watching a feature film on your IPod; you got a sense of the action but it was ultimately as much frustrating as useful.

Things change. Over the past eight years broadband in some form has become available in most cities across the nation. This bandwidth has made dedicated remote access a thing of the past. Now you can provide remote access to your employees over your Internet connection. Traditionally, IPSec has been the technology of choice to provide a virtual private networking solution for your employees but over the past two years there’s been a new game in town – SSL VPNs.

if you are using IPSec for your mobile users, you owe it to them and you to check out one of the SSL VPN options at your disposal. We’ve used IPSec VPNs for network to network access reliably, but they’ve always been tough to support for mobile users. Offhand, there isn’t any specific reason this should be true, but it is. For mobile users, we seem to consistently run into a few problems:

• Installation: The success rate for an average user being able to install an IPSec client and get the VPN tunnels to work, even with phone support, was around 15%. Most of the time the user had to bring in the computer or we had to send a tech on site.
• Compatibility: Different physical network technologies – notably DSL – run into performance problems with IPSec in many configurations, requiring adjustments on the client, routers, or other things that you just can’t expect end users to understand.
• Portability: IPSec is very easy to block on a network. In fact, it took some time for most network routers to be compatible with IPSec. Now try to get it to work at 8 PM over a wireless network in a hotel in Buffalo.

In contrast, a few years ago at the urging of Watchguard (our resident firewall vendor) we tried out their SSL VPN product, which was basically a version of the Citrix Access Gateway SSL VPN solution running on a Watchguard hardware appliance. Out of the box it worked – every time, and even faster than IPSec. We had resisted the option because we preferred standards-based solutions, and this sounded like yet another proprietary security technology. We used a demonstration appliance for a month but the feedback from our users was so strong we purchased a unit after a few weeks. Upon reflection, there really is a good bit of sense to why it works so well:

• SSL is Simple, IPSec is complicated: SSL is a single TCP/IP socket with a relatively straightforward, self-configuring, and invisible to intervening appliances.
• SSL is essential, IPSec is a threat: No one can afford to block SSL on their network without basically admitting to not having a network at all. It’s very expensive to proxy by decrypting and re-encrypting, so few companies do it. On the other hand, many networks view with suspicion the goal of establishing an encrypted connection out of their network, so blocking IPSec may sound like a good idea.

With the SSL VPN solution we had about an 85% end-user self install rate without support, and a 100% rate of not requiring a tech to go on site. Even better, the reviews from end users was that it was fast to connect, easy to use, and performance was good. Because it was so easy to get set up, many more users started connecting from home in the evenings or in bad weather to get work done. The net cost? While your firewall probably offers an IPSec client for free, you can expect to pay a few thousand for a dedicated SSL VPN appliance and depending on licensing $50-$200 per concurrent additional user after the first five or so. For a company with say 100 users that might have at most 20 concurrent users the cost is in the order of $4,000 to $6,000.

Making the Business Case

Jumping from “free” to $6,000 may seem questionable until you look at it from the value standpoint: A service that was expensive to setup and of questionable reliability became cheap to set up and rock solid. In other words, this is the real cost to provide this service. An unreliable solution isn’t a business solution. If it’s more than your business is willing to pay, wait a little while – the cost has come down by half in the last two years, and some vendors (like Watchguard in their Fireware Pro product) are offering it alongside their free IPSec VPN option.

Right now, the power is out at my home. That doesn’t happen often – in fact, it’s been almost two years since we lost power long enough for my UPS to shut down my home network. Normally this would be a small inconvenience, but I still host a few things for my wife out of my house which are now down. The largest of these is a fairly popular forum for an author she likes, but there are other sites as well.

Why am I still hosting these at home? Really there’s no reason – I’ve shifted hosting for my personal services out to other providers, and our company services are also hosted by hosting companies. I just haven’t moved her stuff out of my house.

We talk with a lot of small and medium sized businesses that are still hosting all of their own services internally for pretty much the same reasons – they originally had them in house when they were much smaller and the market was different, and haven’t considered what it would mean to have those computers live somewhere else. It’s time for a change.

Why It’s time to Use the Cloud

You should look at all of your important business services – things that your business can’t operate without – and work out a plan to no longer host those items in your facility. As a first step, just consider what it means to provide the same applications and services, but have the computers not live within your company. The main goals for moving these services out are:

1. Business Agility: When you use a hosting company it’s easier to change capacity as your needs change, even to bring services up temporarily as a trial run and then shut them down if they don’t pan out. This makes it easy to experiment with new software technology without the traditional problems of hosting getting in the way.
2. Low Cost Reliability: If you want those services available, the cost to outfit a room to provide redundant cooling and power for a single rack of equipment is easily $50,000. To host one rack of equipment in a basic Tier-2 data center can cost around $1,500 to $3000 a month, which includes power and Internet. At that rate, how quickly will you get an ROI on your facility investment?
3. Improved Focus: Getting this equipment out of your shop improves your focus on the things you really need to be spending time on: Projects for the business and end-user support. The rest of it is overhead.
4. Access from Anywhere: When you set up your services so they can live in the cloud and be used from your office, it’s easy to make those same services available to employees from home and from laptops. Not as second class citizens but with all of the ranks and privileges of being in the office. This helps you leverage employee talent wherever it is. It’s also easier to set up rock-solid extranet access for customers and suppliers.

When you start looking at each thing you provide as a service, you might also find that some of them – like Microsoft Exchange – really aren’t worth hosting yourself at all even in a data center, and it’d be ultimately in your best interest to outsource it entirely to a hosted Exchange provider. There are number that can do this very effectively. While the cost may seem high based on what it cost you to purchase your initial Exchange licenses, when you look at the real cash costs for Exchange over two to three years they are very cost effective.

Once you’ve taken the step of taking an existing service and outsourced it entirely, you might even consider a Software as a Service offering for some of your core services (such as a hosted CRM). This is the most aggressive mode of outsourcing and does create a set of unique risks and opportunities.

But I can’t See It

Two common objections we hear from IT administrators about moving services out of their shop, even if it’s just relocating servers into a data center. is that it will make it hard for them to get upgrades when necessary because the business won’t be able to see & feel the new equipment. Out of sight, out of mind as the saying goes. The second main objection is that the IT administrators want to be able to do a laying of hands on the equipment to maintain it. There’s a comfort factor in knowing you can walk into a room and flip the power switch or move a drive or just bask in the warm glow of blinking lights.

Here’s the good news: Both of these reasons are not only suspect in their own right, but are preventing your shop from getting to the next level in IT’s relationship with the business.

First, even though vendors do a good job of making server hardware look serious and fun, in the end it’s just a business appliance: It either is good enough to deliver for the business or it isn’t. With rare exception, there is no extra business value for it to look good, new, or cool. If you find that you need to show the business physical servers to explain your costs, you’re missing out on the critical opportunity to establish a real partnership between business and IT. You need to be sure you’re spending when it’s time to spend and saving when it’s time to save, and have discussions in the language the business would use for any other service it would acquire.

Second, If your IT administration patterns and practices require routinely touching your physical infrastructure then you need to re-examine them. It generally means you either have equipment that is no longer up to the task or that you’re not doing enough automation of IT tasks. If you have trouble-prone hardware, it’s time to either fix the fundamental issue or ditch the hardware. Ironically, this type of problem is often easier in a hosted environment because it generally isn’t your problem: it’s the hosting company’s.

Automation is essential because humans are the most error-prone part of any standard process. Your routine IT administration time shouldn’t be going to consistent tasks – they should be automated, leaving your time for user support and other business value-add services. That’s right – even in your shop with your existing staff you can find more time to spend on projects instead of support events by automating recurring tasks.

Some Things Still Stay

There are some things that should be on site for performance reasons. Regardless of how big your Internet connection is, you’re going to want basic file and printer sharing services to be local. Depending on the size of your site, you’ll probably also want a directory server for whatever your directory system is (e.g. Microsoft Active Directory). Even here the central services help: If you have a reasonable Internet connection, you can have your local file server back itself up to the data center by using one of a few distributed backup systems (such as Microsoft’s Data Protection Manager or a third-party option like NSI Software’s Double-Take). This eliminates the time and attention that local disk backups require.

Perhaps not Now, but Soon – and For the Rest of Your Life

It may not be appropriate to move a number of your services outside yet; If you have only one business site, light access by employees externally, and aren’t expecting that to change then you can host most things yourself. A number of the considerations still apply – but you might just use an external facility for your public web presence and for backing up your essential data for business continuity.

Even if you don’t do much now, you should find some opportunity to put a service outside so you and your company can gain experience at working with external hosting providers and you’ll stay current on the capabilities and costs so that as new business requirements evolve you’re ready to take care of them. You’ll be in a better position to advise your company on when to move things out of the shop, and as you do you’ll discover that instead of focusing your time and talent inward at the routine operations of infrastructure you’ll have time for those projects that really make a difference to your business.

How Has the Cloud Delivered For You?

Have a story about what has and hasn’t worked with hosting? Drop me a line or post a comment to share it.

I’ve been a big fan of multiple screens on any computer I use – going back to my first job where I put an older portrait display on my Macintosh IIcx. In that day (1991) it was one of the most amazing features of the Macintosh – you could just pop in extra cards and connect screens, and boom – have a wildly shaped desktop with different color capabilities and purposes.

Ever since in every shop I visit I push for two screens to be the default configuration of any computer. Large screens are great, but there are real practical limits where a larger screen doesn’t do much for productivity, but a second screen will. The best part is that two modest sized screens are invariably cheaper than the same number of pixels in a large screen due to manufacturing cost.

Bring On the Workspace

The primary advantage of two or more screens is they are a very cost effective way of increasing the amount of information your users can see at one time. For between $200 and $400, you can add a 19″ LCD flat panel to a system. With that, users can do several tasks much easier:

1. Comparisons: When you want to compare two things, say two Word documents or spreadsheets, it’s generally ideal to have twice as much desktop area as you normally use for just one. This allows you to interact with each the way you’re used to without having to adjust to navigation and toolbars being in the wrong place because you’re cramming things into a size you don’t normally use (and the document may not be oriented for.
2. Active and Background tasks: A great use of multiple screens is to have one for your active working task and another for background tasks you are monitoring. For example, put Outlook on a secondary screen and your main task (say development, writing an article, or analyzing a spreadsheet) on your main screen. In this configuration, it’s important that your main screen is directly in front of you and your secondary screen is off to the side where it won’t be distracting.

Why Not Just Bigger?

Why not just get one larger monitor? There are a few advantages to multiple displays.

1. Maximize Behavior: When you maximize a window, it goes to the dimensions of the monitor it’s on. This makes it really easy to move windows from screen to screen quickly because you just need to maximize it then drag it, it will size to the full extent of the target window. Get to love maximize – if you can see the background, you’re not displaying as much information as you could.
2. Too big is too big: You should be able to see the entire extent of your screen without moving your head and really without moving your eyes a lot. The entire screen should be in your peripheral vision. With very large screens, you’ll tend to find that you don’t use all the screen because some is too far to the left or right for comfort. This is particularly an issue with the latest very large widescreen monitors (over 24 inches diagonally). If you find that you just can’t use the entire screen, then that monitor is just too big, you really should have another.
3. Optimize for Purpose: Instead of accepting just one generally good shape and set of capabilities you can optimize different screens for different uses. When using two screens, I like to have my primary be a 24″ widescreen and my secondary be a 20″ normal aspect ratio. This fits well with my normal set of activities where I want some extra width for my primary activities but I find that I like the normal ratio for my background task screen where I leave Outlook and put things like online help windows, web sites I’m monitoring, etc. I worked with a quality assurance manager who had three screens with the center one in portrait orientation which fit her work very well – she would have the application being tested on one screen, the QA tracking application on another, and her email or other background window on a third.

Common Objections

The most common objection, particularly from people that aren’t used to working with multiple screens, is that it’s too expensive. This is usually because the cost is viewed as a percentage of increase to new computer cost. From that standpoint it looks significant – if your shop tries to keep individual computers in the $1500 range (which is very easy in the current market for a typical desktop) this will increase the cost by around $250, closing in on 20%! Why, that would require that you increase your budget for desktop PC purchases by 20%, who’s going to pay for that?

This objection completely ignores both the total cost of ownership of a computer and, more importantly, the value of the computer as a tool to the user. A second screen adds very little to the TCO over one screen (LCD monitors are low support cost items) so if you looked at the entire cost of the PC over its life – say $5000 – this cost is much more reasonable.

A second objection is that users will not use it for productivity gains but to instead couple non-work activities with work time or in general just not be able to make effective use of the space.  While users may become very successful on their own with multiple screens, some quick orientation on how to make the best use multiple screens will pay off with users changing how they work with applications to take advantage of the new capability.  If you don’t do this, many will pick it up on their own as they watch their peers.

At the last company I worked at before eSymmetrix, I finally achieved the goal of everyone – even the receptionist – having two screens.  No more was it a privilege for the elite or the engineers; the benefits apply to any information worker so it was done everywhere.

Have you had the experience of looking at a situation and knowing – just knowing, that something was wrong? Perhaps it was a user interface design or a software diagram or project schedule. I’m not talking about a dispassionate concern but an emotional response – you recoiled inside and just knew. Then you had to explain to the person or team that presented the situation why this was the case. Most likely you can draw on your experience and come up with a few very convincing explanations for your gut reaction, but usually you’ll walk away unsettled. It still isn’t right, but you just couldn’t put your finger on it.

Most professionals develop an instinctive ability to size up situations within their core expertise. For example, a seasoned product manager that has worked up through the ranks of developers can often look at a schedule and get a quick feel that it can or can’t be met. Most of the time this intuition expresses itself as a strong gut reaction that you can’t explain. Malcolm Gladwell wrote the excellent book Blink talking about this phenomenon, well worth reading so that you know what to do when you have this reaction the next time.

The basic challenge is this: Even if your instinctive reaction is correct, it isn’t particularly useful until you can explain why. It’s very tempting to try to verbalize the rationalization for your reaction, but don’t. The problem is that your rational mind has no idea why you had an emotional reaction. The emotional reaction is your clue – your intellect and emotions aren’t really connected that well. They need to arrive at their conclusions independently: For you to intellectually justify your reaction you need time to perform a dispassionate intellectual process. Your instinctive emotional reaction didn’t need that time, but it can’t explain itself.

The Path Through The Woods

So if your emotional reaction is equally as reliable as your intellectual evaluation, but you can’t articulate it right away what should you do? The first thing is to develop a code phase for your team that indicates that this is your blink reaction, and not something else. This lets everyone weigh it correctly: It isn’t because you just don’t want to do it or you like another option better, it’s that there’s something instinctively wrong with it. Your team should give it equal weight to you having just expressed a cogent, real argument for there being a problem. Second, the reaction can’t be challenged – at least not directly. It’s an emotional response, so any challenge will drive the participants strait into conflict from collaboration. In our shop, we just say “I have a blink about this…” That cues everyone in.

At the same time, it’s important to recognize that your blink can and will be wrong, sometimes a great deal. It’s no more accurate than a rational analysis of the same circumstance, and that means if the circumstance is something notoriously hard to predict like an election, a project schedule, or a roll of dice then it’s not going to do better than spending some quality time in analysis. This means that when you have a blink reaction then your team should continue with the assumption that the reaction is dead on, but casually seek out the proof.

Casually Find The Evidence

Once you’ve articulated your instinctive blink reaction, have the team take the case that it’s true and then as discussions continue identify facts and data that support the reaction. Eventually, one of a few things will happen:

1. Killer supporting evidence will emerge: In the process of going through subsequent analysis, you’ll find the evidence that suddenly has the reason behind your reaction clear to the team. It’s now an intellectually reasoned response; you just got there faster instinctively.
2. You’ll realize your reaction was wrong: As time passes, your mind will keep attempting to align facts with your reaction seeking to prove or disprove it. Eventually you, or your team, will see what it was that your mind originally caught on and understand that it doesn’t apply in this case: The reaction was wrong, and now you can proceed. This was still an important exercise because you now have validation on your course of action (”we can ignore this previous best practice because it no longer applies because….”)
3. You’ll look at the problem tomorrow and get over it: Perhaps what you felt was just the normal human fear of change. So be it- now that you aren’t feeling threatened any more, your rational mind can reassert itself and look at the item more objectively and be open to new possibilities.

Each of these is a powerful collaboration result because it lets the team and the individual practice and demonstrate the characteristics of a trusting, supporting environment. The team showed the respect for the individual participant and leveraged the experience of everyone, not just those with great oratorical skills. The individual gets to articulate something they feel very deeply without being embarrassed or having to justify and then later defend something that they just can’t. Everyone is in on it – and when the instinct resolves into intellect later everyone will have better insight into the experience and thinking of the person that voiced it. This insight develops trust between the individuals that will live beyond the team.

Finally, by the team not pushing for an immediate defense and then challenging it demonstrates and reinforces that it’s a safe environment to voice ideas and opinions, and builds credibility for when a miss-step happens.

Gaining Speed

Over time as your team works together people may be able to help each other articulate their blink reactions into concrete issues to be resolved based on understanding the key emotional drivers each participant brings to the table. In our company, we have some folks that tend to have reactions over usability, others over ultimate performance (we’ve dubbed one “the keeper of the nanosecond”), and others over code simplicity. Knowing these points helps us not miss-read emotional reactions to ideas and to help each other understand what we really need to do to create the best outcome.

With practice, you’ll find your team can get to great, collaborative conclusions faster and generate the buy-in from the participants with little or no effort.

What Gets You to Blink?

Looking back, when did you have a blink reaction to something? What did you do with it? Share your story by posting a comment or dropping me a line.