Reliable Systems

If you are responsible for a product, you need to actively seek out negative feedback from the best friends of your product. Sure, these folks probably give you a lot of positive feedback, and while that shouldn’t be ignored, it isn’t likely to help you move forward. If you are in a corporate setting developing products for the company this is doubly important. Even the most maligned project or team has its sponsors and friends. Find them, and work with them long enough to make sure they understand you’re able to handle negative feedback and that you really want it. Act on that feedback, and make sure they know where you got it from to encourage more feedback.

If you aren’t getting useful feedback on your product for where you fall short, you should consider why. It isn’t because your product is perfect, it’s going to be because either:

• Lack of Opportunity: You aren’t providing any straightforward ways of getting feedback from the users to you. Users have many things competing for their time, so you need to lower barriers in the way of them providing you feedback when they experience it.
• Perceived Resistance: If the user believes their feedback won’t be received or even worse if they’ll be called out to argue and defend it, they won’t provided.

Lack of Opportunity

What are you really doing to make it easy to get feedback from users and prospects at the time they conceive it? You want to have the barrier in time and difficulty as low as feasible, but you also need to be sure you collect enough information that your feedback is worth it. If the barrier is too low and you don’t collect much you won’t be able to make sense out of it, and the user won’t seek out a more qualified way. However, it’s better to risk that than have the user experience a moment of frustration or insight and not know where to send that information to you.

• Integrate Feedback: Whatever the application is, make it possible to send you feedback from within it. Web application, client server, whatever. Preferably not via email, but directly to a web service or other clean, hidden means. When you integrate it, you can more easily capture where the user was and some other state information to make the most sense out of what they say.
• On Your Product Web Site: Make it really easy for people that are evaluating, window shopping, getting support for, or otherwise interacting with you over the web to get commentary to you.
• When Interacting with Prospects: One of the hardest, and most useful, things to do if you are part of the sales process is to find out why you didn’t get a customer. Find ways to ask the prospect in a sincere way why they didn’t go with your product. They may not be truthful in their response, and you should make no effort to reverse their decision, but you will set yourself apart from your competition if they get that you are sincere about understanding where you fell short.

Bottom line: The higher the barriers to providing you feedback, the less good feedback you’ll get. Make it high enough and you’ll only get irate feedback that isn’t actionable.

Perceived Resistance

There is a general pact you have to agree to when you ask for feedback: You’ll accept whatever you’re told and not argue with it. You can sometimes ask for clarification, but this has to be done with the greatest care to neither create a burden on the person you’re interviewing or the impression you’re arguing with their feedback. If you don’t preserve this pact, you won’t get subsequent feedback from this individual or even people they talk with.

When you’re tempted to argue with them, remember that whatever they wrote is true for them. If, for example, they say that something wasn’t documented but you can point to the prose, then consider why they felt it wasn’t documented: Was it unclear? Could they not find the documentation? Did the terminology change between what they saw and what was documented, making them think they weren’t connected? Whatever they wrote was true for them in that moment, so look for all the ways that it could be true.

A handy technique is to take the case that what they are saying is true, and then prove that it is true. You can do this verbally with your team: “Take the case that users believe we didn’t document this. How could they come to that conclusion?” The small language change of “Take the case” can shift people’s willingness to have a good discussion because it doesn’t violate their believe of what is true, it just asks them to momentarily suspend it.

Still No Results?

If you aren’t getting negative feedback, it isn’t because there’s no negative feedback to be had. If your product is in use at all, people will have opinions about how it could be better. If you aren’t hearing about it, then there has to be a reason they don’t believe they can give it to you.

If feedback isn’t viewed as anonymous (either because it appears traceable or because your company is small enough that the user thinks you’ll be able to figure them out) and you or your team have a reputation of defending your product, then you’ve created an emotional barrier for them to overcome. You need to go out of your way to break this down.

Last Christmas holiday I took off more time than normal and had a great opportunity to spend time with my brothers and sisters and their families. One of the exciting aspects of having your siblings marry is they bring in to the family people with a very different perspective than you grew up with.  Now, my immediate family is entirely Windows (although go back 10 years and they were mostly Apple, interestingly). My sister and her fiancé (who both live and work in academia) are strong Apple proponents. They have Apple laptops that talk to their Apple wireless router and their iPods and iPhones, etc.

At one point, my sister’s fiancé noticed that my new laptop was running Vista and missed no opportunity to comment on all of the problems with Vista.  Eventually I took the bait and asked how he was handling all of the problems with OS X 10.5. His reaction was that there weren’t any.  When I asked for more specifics, pulling up some reference articles from the net, he downplayed the problems. Along the way he said “well, they did break X Windows. If I’m using many X programs, I have to reboot perhaps a few times a day, but I can handle that.” I think there are few phrases that typify better how we develop an emotional attraction to brands and products that goes well beyond our reasoning.

It’s OK to Be Emotional

Everyone I’ve met has a soft spot for at least a few products and brands. These things hold a special place where we actively want them to succeed and be awesome. Some companies and products, for whatever reason manage to cultivate this feeling in more people: Probably the gold standard is Apple and perhaps Starbucks. When you relate to a brand this way, you tend to give it the benefit of the doubt and even defend it from anything that might be seen as detracting from it. You will tend to keep this relationship with a brand even if you aren’t a current customer, because this is an emotional attachment, not a rationale evaluation.

For myself, some of the brands I relate to in this way are (in no particular order):  Ford, Volvo, Microsoft, Dell, John Deere, Yamaha (Audio equipment)…  There are also brands that I have the opposite reaction to: I tend to be highly suspicious of their products from the start: Apple, IBM, Oracle, Honda, Toyota… This emotional tie-in with brands is a well known phenomenon that is heavily exploited by product marketing and ultimately is what creates value in a brand itself.

One aspect of how you react to products and brands that you love is that you tend to not want to brook any criticism of the brand or products and have an emotional first reaction before you’ll listen to rationale concerns.  Think about your favorite computer or car brand:  If you hear someone else say something negative about that brand (”Those Honda cars, they’re just underpowered and can’t get moving.”) how do you react?  It’s the intensity of that reaction that lets you know how you relate to that brand.

Conversely, most people have purchased a product against your brand preference. It may be because what you want is out of stock and you can’t wait, or because there is one killer feature you just can’t live without, or some other consideration that manages to overpower your brand loyalty.  When you do, have you had the experience that you are quick to criticize even the smallest detail of the product? It’s natural - you want to reinforce with reasoning your original opinion that was created out of emotion. You’re even more likely to post a blog article or seek out another way to get your critique to the vendor than you would be if it was your preferred brand.

Help the Brands You Love

If you really care about a brand, you should do your best to provide the strongest, most critical feedback for it. In fact, when reviewing survey results or customer feedback within a company it’s the negative feedback that has value. When working within a company, I’m not particularly interested in people that give marks above 80%; that feedback isn’t giving me anything I can work on. What’s interesting is the low marks. What’s really useful are low marks with explanation so I can understand where the person was coming from.

If you want to help the brands you love, look for each opportunity to not give them the benefit of the doubt but instead practice tough love - provide specific, reasoned feedback on what the product needs to do better. Find ways to get it to the people within that brand that can make a difference. In my experience, the folks inside those companies hunger for it, and it has an impact much greater than you expect because these companies are made of people, and their decisions are colored by what evidence they can find to back up what’s being spoken.

Are You Really Extremely Satisfied?

The next time you get a form that asks you to rate your experience from say 1-5 or 1-10, be really honest. While you know that they are looking for everything to be in the top bracket, can you really say that you had an exception or extraordinary experience in every category? While that may do well for some internal marketing statistics report, it isn’t really useful to the product engineers behind the product.

When I bought my last car, I was really honest: I had high expectations of the dealership, and they met them - that meant when it asked did we meet or exceed your expectations, they.. met them. I got a call from a regional marketing rep later wanting to know why I had given them low marks. Two things happened from this: I was able to explain to them that I had high expectations and why (the dealership had done very well in the past for me) and personally comment on a few problem areas that really weren’t the dealership’s fault (they didn’t have access to some information to help me that they should have). This helped the brand in a way I’m sure me just checking the 10 box wouldn’t have.

A popular technique in project management is to create and maintain a risks list. This list is supposed to track the things that could cause the project to fail to meet its objectives and ensure that each one gets a mitigation strategy. You generally prioritize the risks by looking at the product of impact and probability: How bad would it be if the risk came to pass, and how likely is that to happen.

Risk mitigation is generally done in three flavors:

1. Make adjustments now to mitigate the risk preemptively.
2. Have a plan ready to go in case the risk materializes.
3. Do nothing and wait and see.

Most project managers (particularly the Project Monitor variety) are highly reluctant to take the last approach: Do nothing, wait and see. They want to have a mitigation strategy for every risk, preferably one that involves taking action now so that the risk is fully mitigated and can be removed from the list. This is counter productive to the project because:

• In the short term it will have the team spend energy and time on mitigating risks of ever decreasing probability, on tasks that are pure overhead in the project.
• In the long term it teaches team members to not mention risks because any risk mentioned will have to be mitigated. This is a strong disincentive to open risk discussion, much like obsessing about defects can cause people to not log defects.

If you want to get value out of a risks list, it has to be treated like a “hold harmless” list: The team has to believe that management won’t hold any risks against them, that they won’t have to mitigate any particular risk on the list, and that intelligent decisions will be made on what risks need to be drilled into. One way to help this is for the project manager to have a strong preference to have each risk either be categorized as Do Nothing or at most mitigate when it happens. Have the team accountable for pushing risks to a higher status by their own consensus. This can remove the fear on the part of team members that identifying a risk and speaking it aloud will cause extra work for the team to debate it and have to mitigate it. Instead, focus your team’s risk list as an opportunity for each participant to identify risks and then suggest mitigation for a risk if they so choose.

Can Your Project Create a Black Hole?

There will be some risks that have a reasonable probability and a serious impact that may require more serious intervention, however there is another item that comes into play at this point. Many high impact risks also should have no reasonable mitigation strategy. If the impact is that the project is sure to fail or even the company is sure to fail, unless you believe you can mitigate it with a modest amount of work, you should probably just let it go. This may seem crazy up front - these are your most serious risks yet I’m suggesting that you treat them as less important than others farther down the list. Quoting World Radio Switzerland (and also from The New York Times):

A lawsuit has been filed against Geneva’s CERN research facility over claims its experiments could destroy the planet. Two scientists in Hawaii say the European Organization for Nuclear Research might create black holes that could swallow the earth. The Large Hadron Collider is the world’s biggest particle accelerator and will essentially crash beams of protons together to create what the universe was like right after the Big Bang. It will do so in a massive underground ring stretching underneath the Swiss and French border and the scientists claim CERN has not fully tested the LHC.

From a project management standpoint, this is essentially the definition of a risk with infinite impact: What could be greater than unstoppable destruction of the Earth? This is a Terminal Risk.The practical reality is that if the result of a risk is that your project will fail, and it can’t be mitigated in a way that still has your project succeed, then you have already failed - if it happens, there’s no solution.

In business, these risks happen all of the time: There is a finite risk that the government will change laws that will ruin the economics of what you do. A competitor could come out with a game changing product that makes you effectively the last buggy whip manufacturer. These are all terminal risks: If they happen, you’re dead. Therefore, spend no time on them at all. Any amount of time you worry about them will not help your project succeed and will only demoralize the team. Additionally, once your team gets focused on a terminal risk, they will start identifying other similar terminal risks, and there’s no running out of them. It is a spiral that doesn’t produce any gain to your project or team.

Instead, focus your energy on the middle set: risks that are reasonable to mitigate, have a fair shot at coming to pass, and would have a real impact to the project. Those are the ones that you’ll have difficulty explaining why you didn’t do anything about them in a post-project review, and your team will likely feel better about the project and the process in response.

I am a lucky, lucky guy. Among the many reasons I believe that is a gift for timing. Last week I installed a new Microsoft Windows Home Server (WHS) for our house. It replaced an aging server that just wasn’t cost effective to keep adding storage to. For our home needs of photos, music, and videos it just didn’t seem to make sense to buy high performance storage and a new HP Windows Home Server seemed a perfect fit. It also offered a number of features that looked pretty good - like full automatic backup of the various desktops in our house and easy restore.

Amazingly enough, I got to test that feature out just a week after putting the server in place - the drive in my wife’s machine tanked out of the blue. Dell shipped us out a new drive that arrived the next day, and I was looking forward to being able to pop in the Microsoft-provided full recovery CD, restore the machine, and have her on her way without any incident. It might be one of my few towering triumphs of home technology.

And Microsoft put some real work into this: They provide the recovery CD, you boot from it and it asks for one password and connects to the server to see what backups are available to be restored. It even prompts you to restore the backups for the local machine which it must be figuring out by MAC address or something (since the drive was absolutely brand new, so nothing to go off there). So far I was pretty impressed.

Then I ran into it, the usability issue I knew had to be there. It refused to run a restore to the new drive because, well, it was new - it wasn’t formatted, so it didn’t know what to do. A dialog popped up informing me that I might want to run disk administrator, which it provides a nice button to start up.

Here’s the thing

I know exactly what to do with disk administrator; I can administer a Windows 2003 server which is basically what WHS is under the covers. I’ve been using Windows NT since, well - it was just NT. So this didn’t throw me personally, but Microsoft is really hoping - and billing - that WHS is a server for the rest of the world - for say my parents to use, and they could use one. The problem is that for all of the work they did to get it this far (and it must have been a lot of work - the recovery CD booted up a special running copy of Vista and went right into a recovery wizard that was, over all, pretty smart) they then dropped the ball completely by not recognizing that a blank drive means… Time for a full system restore. If not, at least do what the OS installation has been doing since… NT 3.1 like prompting “would you like recovery to format the drive?” or something that people will understand.

If my parents had gotten to that point, they would have stopped and had to call me, and that’s pretty much game over from a usability standpoint. I don’t even know if I could easily walk them through how to use disk administrator from scratch to partition and format a drive without seeing the screen.

Close Microsoft, very close. But the home user market really wants choices in plain language to tell them what to do, particularly when recovering from something as traumatic as a complete system failure. Great usability isn’t cheap, but it’s worth it. Here’s the worst part: I’m sure that the WHS team at Microsoft, which by all appearances is pretty smart and had to make a lot of hard choices, had a meeting where they probably debated this point, and taking the time to smooth out this wrinkle reliably lost out. Possibly because it wouldn’t work correctly if the home system was running Turkish windows and had SCSI RAID or some other such boundary case. Too bad. Apple would have taken the time to make it work - an easier task with a lot less hardware variation to support, but making it work right when it’s hard is exactly what people pay for. Do it often enough and you get a reputation for excellence.

My wife’s system was back up and running in no time, overall from her perspective it rocked: Her machine died a rapid, painful death and was back up exactly where she was quickly, having lost only a few hours work. For me, it was another in a line of recent experiences with Windows where it was so good in so many ways that the few cigarette burns and scratches in the finish really stand out.

A Plea to Microsoft

Stop living up to what your detractors say about you. You spend a fortune on getting things right and I know enough to really respect just how hard it is to accomplish what you do. The Microsoft folks I’ve worked with are passionate and take a lot of pride in what they do. Check out the Windows Home Server Team Blog - these folks love what they do. But the Microsoft brand is not known for just working and for being easy enough you can recommend it to your Mom. You can fix this.

As part of the product set that my company develops we are being very aggressive about automated unit testing and validation. Our product is used primarily by companies that operate custom developed software systems to monitor and ensure they have reliable systems, and accordingly they expect it to do no harm and essentially be lean, mean, and bulletproof. One of our set of unit tests was designed to ferret out problems by varying numeric inputs by their smallest increment and checking the output for discontinuities. This technique had proven surprisingly effective at discovering a range of existing defects in everything from the Intel and AMD processors through the .NET runtime. When we would find an issue, we’d verify it and then put a workaround in our calculation routines so that our users wouldn’t have to worry about them.

Recently we were introducing a new set of 64-bit specific tests and ran into a challenge: It wasn’t possible to exhaustively validate the routines. Our automated build system can run around 500,000 tests per second, meaning it takes about 2.5 hours to process all possible 32-bit values. But, to cover all 64-bit values we’d need to run that 2.5 hour test about 4 billion times. This works out to be well over a million years, give or take a rounding error of all recorded history. We intuitively knew the test would take a while, and had bandied about a range of possible times, but in the end we failed to comprehend that 64-bits is the square of 32-bits.

That’s A Big Twinkie

Think about it. In your life you often have to deal with linear approximation and even multiplicative approximation, but never exponential. About the biggest effect we can intuitively get our head around is an order of magnitude - 10 times a value. If you see one car, you can visualize 10, possibly then mentally extend that to 100 (two orders of magnitude) by mentally creating ten rows of ten cars. Now go one more order of magnitude to 1000 by say visualizing a 10 story parking garage with each floor having 100 cars. By going an order of magnitude at a time, you can get through around three before it becomes meaningless. That is roughly the difference between one megabyte and one gigabyte. This is a ratio we’ve developed some familiarity with: Memory has (until very recently) generally been thought of in megabytes and hard drives in gigabytes. Only in the past year or two has it become commonplace to talk about memory in gigabytes and hard disk space in terabytes, each around 1000 times greater than their predecessor.

But what we’re talking about with the difference between 32 and 64 bits isn’t about orders of magnitude. A 64-bit value can contain over 4 billion sets of 4 billion values each - nearly 100 orders of magnitude greater. In traditional Big-O notation for understanding algorithmic complexity you drop out purely additive or multiplicative affects as inconsequential and focus on exponential effects. Despite having been both trained in this and working around complex software for nearly two decades, with the incredible increase of processing power in the last 10 years it has become very easy to forget that things can still take real time to calculate, sometimes nearly infinite time.

The Last of My Lifetime?

We have also become conditioned that every doubling in bit width is only a temporary reprieve: In my career we’ve gone from 8 to 16 bit, 16 to 32, and now are starting 32 to 64. It’s natural to project forward and assume that we’ll similarly need to move to 128 bit within a decade, but these doublings in bit widths are exponential increases in size, and the step from 32 to 64 is vast in the amount of area it opens up. When Windows NT shipped with 32 bit support, it was possible to buy a computer with 1 GB of RAM or one quarter the entire 32 bit address space. That would require approximately one billion gigabytes of memory to equal for 64 bit. Just to purchase that would require around 25 billion dollars today, and would nearly consume the entire world production of DRAM for one year. It isn’t the same game anymore, we aren’t going to see 128-bit memory spaces any time soon, at least not for the traditional reasons we’ve upped bit width. To a certain degree that’s a shame - Having gone through the other transitions before which were the cause of years of trauma to hardware and software developers alike I’ve been amazed at how our entire .NET product just worked the first time on 64-bit. That’s progress.