Reliable Systems

Your product is out in the wild, and even better – it’s in use by real users. You’ve got feedback and support structures in place and they are producing results. Now you need to take that feedback and incorporate it back into the product. To do this, you’re going to have to navigate the social dynamics of your organization around defects.

Any product, software or otherwise, has defects. Your shop may have a nice term to paper over it – incident, problem, ticket, errata, trouble report…. But let’s not paper over it – a defect is a defect. Developers also like to split hairs between feature requests and defects, but from a user standpoint it’s all defects. Everyone involved in product development will have their own way to prioritize defects, and to get the best results from your team’s time you need to be able to figure out which ones to address and how fast – and do so in a way that generates buy in from your development team, management, and customers.

There are an endless number of way to look at prioritization, but however you do it the discussions should include several perspectives:

• Impact, Difficulty, and Risk of defect to the end user.
• Impact, Difficulty, and Risk of even diagnosing the defect.
• Impact, Difficulty, and Risk of defect correction and deployment.

The End User Perspective

Your customers in general want and expect a defect-free product. Even if your average customer understands that all software has defects, intellectual understanding won’t overcome the emotional impact of running into a problem. Your users will generally start from the perspective that their problem is a defect in your software, it shouldn’t have been there in the first place, and you need to fix it immediately. Today would be nice.

It is very difficult to understand the value a customer places on fixing a particular issue from within the development team. Developers tend to grade defects based on the effort it takes to fix them and whether they produce an outright failure of the software. For example, few developers will get worked up over fixing a cosmetic defect such as a misspelling or alignment problem; If it’s that simple to fix, how valuable can it be? The exception to the natural cognitive bias that a problem must be hard to be worthy are problems that can crash the application or cause it to corrupt data. Few developers won’t see this as a deadly sin that must be resolved regardless of cost or risk.

Customers have a different perspective. They see just the surface veneer of your product and assume that it can never corrupt data or crash. Outright crashes have gotten rare enough that most users will refer to an error message as a crash. Because they have no idea what’s happening inside the black box that is your product, they will judge it solely on what they can see: Does it act like other applications, do the things they can see look clean and well crafted? Much like making a spelling error on the title page of your term paper can cause the entire work to be devalued, a small cosmetic error on the user interface can cause customers to doubt the correctness of your entire application.

Many end users will tend to discount defects that require long steps to produce or go away by restarting the application as long as they can convince themselves that they are at fault. This happens more often than you might expect – most users believe they don’t understand the rules behind the application and instead are using a rote procedure to accomplish their tasks. When something goes wrong, they will generally go back and try it again – possibly many times – before coming to the conclusion that there just might be a problem with the software and not with them. If the defect only presents occasionally they will usually write it off as their fault. Lest you think this behavior is limited to nontechnical users, this happened to NASA and resulted in a several day delay of the first Space Shuttle Flight.

With rare exception, if a defect isn’t judged as essential to fix by your customers, it’s probably not worth addressing prior to the next routine release. Every change to the software has consequences and takes effort that could go into something more important – to your team and your customers.

Coming Next: The Diagnostic Perspective

Come back in a few days for the next post in this series, talking about the impact, difficulty and risks of diagnosing and resolving defects.

So you’ve done it. You believe that feedback from your users is the ultimate input into what will ensure your product succeeds, and you have a range of mechanisms both passive and active to collect it. You routinely analyze that feedback to understand what it means, and you incorporate it into your development process.

As you move down this road, you’ll inevitably have an experience where you got negative feedback on a feature that you felt was very well thought out. You might make a change to help that feature be even better, only to discover you are getting the same rate or even more negative feedback than before.

Herein lies a problem that can easily derail your team taking feedback seriously: You are always going to have some people that don’t like any given feature of your system, or provide feedback too fast when they don’t first figure something out, only to work it out later and be fine with what you have.

When evaluating user feedback, you need to know that some of the user community will not like any given aspect of your product. What you’re listening for isn’t the individual comments – the notes played by just one instrument – but the overall feeling of the music created by all of the comments.

User interface design and, in a broader sense, all product design is a function of tradeoffs:

• Discoverability: If you make every feature clearly visible, it will take up too much space or confuse beginners. If you put one item in more than one place because it’s important some users will think a different feature should get the same or better treatment based on how they use the product.
• Performance: No matter how fast your application is, there will be a user with a computer slower than that who will complain that the application is almost, but not quite, fast enough to be usable. Or they will use it in increasingly silly ways until performance suffers.

Music isn’t Statistics

Listening to the Symphony is not an exercise in statistics. It’s not as simple as counting how many instruments are playing a line or what they are. The individual notes played by the musicians combine in very complicated ways to create the net effect you experience listening to the piece.

Likewise, your feedback as a whole shouldn’t be simply lined up and counted. Who the feedback is coming from is important, not just how many. For example, is the user an early adopter or a mainstream user? Are they part of your past market or where you are expanding? If your product has crossed the chasm, the early adopters are ready to move on and frankly aren’t where you need to focus your attention. On the other hand, if your product is young then the early adopters are everything and mainstream users that don’t get it may not be reachable even if you addressed their concerns because they just aren’t ready to be in your market.

Depending on how you gather the information, it may be relatively easy or may require looking at each piece and gauging where the submitter is in the technology adoption cycle relative to your product. While you are in transition from early adopter to mainstream use, the feedback of the early adopters will still overwhelm that of your new market statistically, so it’s important to have a method of giving weight to feedback by user market instead of just as a whole.

Solos are Rare

Occasionally you’re going to hear a solo comment that you know is right on – the user clearly explained the problem, it’s a legitimate intended use of the product, and you can conceive a way to change your product to handle it. Solo comments aren’t frequent. More often you’re going to get a lot of little comments about areas of the system.

One place where solo comments tend to happen is with modern social media (Blogs, Wikis, forums, etc.). The Internet gives essentially anyone a personal soapbox to broadcast their opinions, and modern search and indexing tools are uncanny in their ability to pull out these solos and amplify them across both time and distance. Get in the habit of connecting with good search tools so you’re in the audience. There is a great deal of good will generated when you reach out quickly to connect with either a strong positive or negative commentary; your contribution to the chorus will be found as easily and most readers will listen to the whole piece instead of just the solo.

Listen to Trends

When listening to a great symphony, you’ll hear many of the same basic musical themes repeated in each movement with slight variations. As you pay more attention to the music, it’s the variations that stand out in each movement. The variations could be tempo, volume, timbre, notes… many small details that extend or change the previously introduced musical phrase.

When evaluating the bulk of the feedback you receive look for what’s changing over time, particularly between releases where you attempted to address user concerns or introduce new features. Is the overall amount of negative feedback going up? If you’re hearing more dissonance, even if it isn’t localized in one area, you need to look for general issues that could be increasing user confusion. You want to interpret the trends in the context of where you are and where you’re going. At a higher level than simply noticing the motifs in the piece is understanding those motifs within the broader context of the overall mood and emotional impact of the piece.

A change in feedback may have nothing to do with what you did to your product at all but instead a change in user expectations. For example, they may be trending to a new browser that doesn’t work as well with your product or they might be getting concerned about identity theft or natural disasters or some other real-world change that users are coupling with an aspect of your application.

One reason I love to be involved in the sales process is to hear the questions being asked by prospects. I find that about every six months the dominant (top 5) questions change with something dropping out because it’s just assumed handled and replaced by the hot new concern. If you can catch the tip of these changes (for example, when customers start asking about federated identity) then you may be able to make product changes in time to catch the wave (integrate with federated identity managers your customers are likely to want) or know exactly why you’re going to let it pass you by.

Silence is Music Too

One quick way to tell casual classical concert goers from regulars is to notice if they applaud based on silence instead of when the piece is over. If the conductor’s arms are still up, the silence is part of the music.

Next to negative feedback, silence is the most important. If you are hearing nothing about a particular area or feature of the system, that’s feedback – either people aren’t using it, it isn’t important, or they can’t figure it out enough to even express their thoughts. If you have metrics about application usage (and if you don’t, you should – passive metrics are much harder to argue with than fuzzy user comments) and know a feature is being used but you aren’t getting anything back then the conductors arms are up – you need to listen carefully.

If your metrics tell you that a feature isn’t being used, this is a problem on its own: Most likely users can’t find the feature when they need it, or they don’t have confidence that it’s better than the alternative approach they have conceived to get the result they want.

On the web, if you notice a shift in browser or operating system usage be particularly wary of silence from a particular community: If you notice a falloff in Safari users, you should probably recheck that your application looks and works well in Safari. It’s easy to forget that each of the major browsers has a dominant platform its designers are coming from (Windows, X Window, MacOS) and that colors how they create standard widgets like combo boxes, lists, etc. in addition to the general problems of creating a well behaved web application in each of the major browsers.

Listen to the Music Today

Regardless of where you are in process of integrating feedback into your product development, there is feedback to be found buried in logs, CRM systems, the Internet, and other sources. Get creative, pull some together, and start analyzing the music. You’ll learn unexpected things that will invigorate your team.

Have a story to share? Please send me a note or post a comment below.

If you are responsible for a product, you need to actively seek out negative feedback from the best friends of your product. Sure, these folks probably give you a lot of positive feedback, and while that shouldn’t be ignored, it isn’t likely to help you move forward. If you are in a corporate setting developing products for the company this is doubly important. Even the most maligned project or team has its sponsors and friends. Find them, and work with them long enough to make sure they understand you’re able to handle negative feedback and that you really want it. Act on that feedback, and make sure they know where you got it from to encourage more feedback.

If you aren’t getting useful feedback on your product for where you fall short, you should consider why. It isn’t because your product is perfect, it’s going to be because either:

• Lack of Opportunity: You aren’t providing any straightforward ways of getting feedback from the users to you. Users have many things competing for their time, so you need to lower barriers in the way of them providing you feedback when they experience it.
• Perceived Resistance: If the user believes their feedback won’t be received or even worse if they’ll be called out to argue and defend it, they won’t provided.

Lack of Opportunity

What are you really doing to make it easy to get feedback from users and prospects at the time they conceive it? You want to have the barrier in time and difficulty as low as feasible, but you also need to be sure you collect enough information that your feedback is worth it. If the barrier is too low and you don’t collect much you won’t be able to make sense out of it, and the user won’t seek out a more qualified way. However, it’s better to risk that than have the user experience a moment of frustration or insight and not know where to send that information to you.

• Integrate Feedback: Whatever the application is, make it possible to send you feedback from within it. Web application, client server, whatever. Preferably not via email, but directly to a web service or other clean, hidden means. When you integrate it, you can more easily capture where the user was and some other state information to make the most sense out of what they say.
• On Your Product Web Site: Make it really easy for people that are evaluating, window shopping, getting support for, or otherwise interacting with you over the web to get commentary to you.
• When Interacting with Prospects: One of the hardest, and most useful, things to do if you are part of the sales process is to find out why you didn’t get a customer. Find ways to ask the prospect in a sincere way why they didn’t go with your product. They may not be truthful in their response, and you should make no effort to reverse their decision, but you will set yourself apart from your competition if they get that you are sincere about understanding where you fell short.

Bottom line: The higher the barriers to providing you feedback, the less good feedback you’ll get. Make it high enough and you’ll only get irate feedback that isn’t actionable.

Perceived Resistance

There is a general pact you have to agree to when you ask for feedback: You’ll accept whatever you’re told and not argue with it. You can sometimes ask for clarification, but this has to be done with the greatest care to neither create a burden on the person you’re interviewing or the impression you’re arguing with their feedback. If you don’t preserve this pact, you won’t get subsequent feedback from this individual or even people they talk with.

When you’re tempted to argue with them, remember that whatever they wrote is true for them. If, for example, they say that something wasn’t documented but you can point to the prose, then consider why they felt it wasn’t documented: Was it unclear? Could they not find the documentation? Did the terminology change between what they saw and what was documented, making them think they weren’t connected? Whatever they wrote was true for them in that moment, so look for all the ways that it could be true.

A handy technique is to take the case that what they are saying is true, and then prove that it is true. You can do this verbally with your team: “Take the case that users believe we didn’t document this. How could they come to that conclusion?” The small language change of “Take the case” can shift people’s willingness to have a good discussion because it doesn’t violate their believe of what is true, it just asks them to momentarily suspend it.

Still No Results?

If you aren’t getting negative feedback, it isn’t because there’s no negative feedback to be had. If your product is in use at all, people will have opinions about how it could be better. If you aren’t hearing about it, then there has to be a reason they don’t believe they can give it to you.

If feedback isn’t viewed as anonymous (either because it appears traceable or because your company is small enough that the user thinks you’ll be able to figure them out) and you or your team have a reputation of defending your product, then you’ve created an emotional barrier for them to overcome. You need to go out of your way to break this down.

Last Christmas holiday I took off more time than normal and had a great opportunity to spend time with my brothers and sisters and their families. One of the exciting aspects of having your siblings marry is they bring in to the family people with a very different perspective than you grew up with.  Now, my immediate family is entirely Windows (although go back 10 years and they were mostly Apple, interestingly). My sister and her fiancé (who both live and work in academia) are strong Apple proponents. They have Apple laptops that talk to their Apple wireless router and their iPods and iPhones, etc.

At one point, my sister’s fiancé noticed that my new laptop was running Vista and missed no opportunity to comment on all of the problems with Vista.  Eventually I took the bait and asked how he was handling all of the problems with OS X 10.5. His reaction was that there weren’t any.  When I asked for more specifics, pulling up some reference articles from the net, he downplayed the problems. Along the way he said “well, they did break X Windows. If I’m using many X programs, I have to reboot perhaps a few times a day, but I can handle that.” I think there are few phrases that typify better how we develop an emotional attraction to brands and products that goes well beyond our reasoning.

It’s OK to Be Emotional

Everyone I’ve met has a soft spot for at least a few products and brands. These things hold a special place where we actively want them to succeed and be awesome. Some companies and products, for whatever reason manage to cultivate this feeling in more people: Probably the gold standard is Apple and perhaps Starbucks. When you relate to a brand this way, you tend to give it the benefit of the doubt and even defend it from anything that might be seen as detracting from it. You will tend to keep this relationship with a brand even if you aren’t a current customer, because this is an emotional attachment, not a rationale evaluation.

For myself, some of the brands I relate to in this way are (in no particular order):  Ford, Volvo, Microsoft, Dell, John Deere, Yamaha (Audio equipment)…  There are also brands that I have the opposite reaction to: I tend to be highly suspicious of their products from the start: Apple, IBM, Oracle, Honda, Toyota… This emotional tie-in with brands is a well known phenomenon that is heavily exploited by product marketing and ultimately is what creates value in a brand itself.

One aspect of how you react to products and brands that you love is that you tend to not want to brook any criticism of the brand or products and have an emotional first reaction before you’ll listen to rationale concerns.  Think about your favorite computer or car brand:  If you hear someone else say something negative about that brand (“Those Honda cars, they’re just underpowered and can’t get moving.”) how do you react?  It’s the intensity of that reaction that lets you know how you relate to that brand.

Conversely, most people have purchased a product against your brand preference. It may be because what you want is out of stock and you can’t wait, or because there is one killer feature you just can’t live without, or some other consideration that manages to overpower your brand loyalty.  When you do, have you had the experience that you are quick to criticize even the smallest detail of the product? It’s natural – you want to reinforce with reasoning your original opinion that was created out of emotion. You’re even more likely to post a blog article or seek out another way to get your critique to the vendor than you would be if it was your preferred brand.

Help the Brands You Love

If you really care about a brand, you should do your best to provide the strongest, most critical feedback for it. In fact, when reviewing survey results or customer feedback within a company it’s the negative feedback that has value. When working within a company, I’m not particularly interested in people that give marks above 80%; that feedback isn’t giving me anything I can work on. What’s interesting is the low marks. What’s really useful are low marks with explanation so I can understand where the person was coming from.

If you want to help the brands you love, look for each opportunity to not give them the benefit of the doubt but instead practice tough love – provide specific, reasoned feedback on what the product needs to do better. Find ways to get it to the people within that brand that can make a difference. In my experience, the folks inside those companies hunger for it, and it has an impact much greater than you expect because these companies are made of people, and their decisions are colored by what evidence they can find to back up what’s being spoken.

Are You Really Extremely Satisfied?

The next time you get a form that asks you to rate your experience from say 1-5 or 1-10, be really honest. While you know that they are looking for everything to be in the top bracket, can you really say that you had an exception or extraordinary experience in every category? While that may do well for some internal marketing statistics report, it isn’t really useful to the product engineers behind the product.

When I bought my last car, I was really honest: I had high expectations of the dealership, and they met them – that meant when it asked did we meet or exceed your expectations, they.. met them. I got a call from a regional marketing rep later wanting to know why I had given them low marks. Two things happened from this: I was able to explain to them that I had high expectations and why (the dealership had done very well in the past for me) and personally comment on a few problem areas that really weren’t the dealership’s fault (they didn’t have access to some information to help me that they should have). This helped the brand in a way I’m sure me just checking the 10 box wouldn’t have.