Reliable Systems

So you’ve done it. You believe that feedback from your users is the ultimate input into what will ensure your product succeeds, and you have a range of mechanisms both passive and active to collect it. You routinely analyze that feedback to understand what it means, and you incorporate it into your development process.

As you move down this road, you’ll inevitably have an experience where you got negative feedback on a feature that you felt was very well thought out. You might make a change to help that feature be even better, only to discover you are getting the same rate or even more negative feedback than before.

Herein lies a problem that can easily derail your team taking feedback seriously: You are always going to have some people that don’t like any given feature of your system, or provide feedback too fast when they don’t first figure something out, only to work it out later and be fine with what you have.

When evaluating user feedback, you need to know that some of the user community will not like any given aspect of your product. What you’re listening for isn’t the individual comments – the notes played by just one instrument – but the overall feeling of the music created by all of the comments.

User interface design and, in a broader sense, all product design is a function of tradeoffs:

• Discoverability: If you make every feature clearly visible, it will take up too much space or confuse beginners. If you put one item in more than one place because it’s important some users will think a different feature should get the same or better treatment based on how they use the product.
• Performance: No matter how fast your application is, there will be a user with a computer slower than that who will complain that the application is almost, but not quite, fast enough to be usable. Or they will use it in increasingly silly ways until performance suffers.

Music isn’t Statistics

Listening to the Symphony is not an exercise in statistics. It’s not as simple as counting how many instruments are playing a line or what they are. The individual notes played by the musicians combine in very complicated ways to create the net effect you experience listening to the piece.

Likewise, your feedback as a whole shouldn’t be simply lined up and counted. Who the feedback is coming from is important, not just how many. For example, is the user an early adopter or a mainstream user? Are they part of your past market or where you are expanding? If your product has crossed the chasm, the early adopters are ready to move on and frankly aren’t where you need to focus your attention. On the other hand, if your product is young then the early adopters are everything and mainstream users that don’t get it may not be reachable even if you addressed their concerns because they just aren’t ready to be in your market.

Depending on how you gather the information, it may be relatively easy or may require looking at each piece and gauging where the submitter is in the technology adoption cycle relative to your product. While you are in transition from early adopter to mainstream use, the feedback of the early adopters will still overwhelm that of your new market statistically, so it’s important to have a method of giving weight to feedback by user market instead of just as a whole.

Solos are Rare

Occasionally you’re going to hear a solo comment that you know is right on – the user clearly explained the problem, it’s a legitimate intended use of the product, and you can conceive a way to change your product to handle it. Solo comments aren’t frequent. More often you’re going to get a lot of little comments about areas of the system.

One place where solo comments tend to happen is with modern social media (Blogs, Wikis, forums, etc.). The Internet gives essentially anyone a personal soapbox to broadcast their opinions, and modern search and indexing tools are uncanny in their ability to pull out these solos and amplify them across both time and distance. Get in the habit of connecting with good search tools so you’re in the audience. There is a great deal of good will generated when you reach out quickly to connect with either a strong positive or negative commentary; your contribution to the chorus will be found as easily and most readers will listen to the whole piece instead of just the solo.

Listen to Trends

When listening to a great symphony, you’ll hear many of the same basic musical themes repeated in each movement with slight variations. As you pay more attention to the music, it’s the variations that stand out in each movement. The variations could be tempo, volume, timbre, notes… many small details that extend or change the previously introduced musical phrase.

When evaluating the bulk of the feedback you receive look for what’s changing over time, particularly between releases where you attempted to address user concerns or introduce new features. Is the overall amount of negative feedback going up? If you’re hearing more dissonance, even if it isn’t localized in one area, you need to look for general issues that could be increasing user confusion. You want to interpret the trends in the context of where you are and where you’re going. At a higher level than simply noticing the motifs in the piece is understanding those motifs within the broader context of the overall mood and emotional impact of the piece.

A change in feedback may have nothing to do with what you did to your product at all but instead a change in user expectations. For example, they may be trending to a new browser that doesn’t work as well with your product or they might be getting concerned about identity theft or natural disasters or some other real-world change that users are coupling with an aspect of your application.

One reason I love to be involved in the sales process is to hear the questions being asked by prospects. I find that about every six months the dominant (top 5) questions change with something dropping out because it’s just assumed handled and replaced by the hot new concern. If you can catch the tip of these changes (for example, when customers start asking about federated identity) then you may be able to make product changes in time to catch the wave (integrate with federated identity managers your customers are likely to want) or know exactly why you’re going to let it pass you by.

Silence is Music Too

One quick way to tell casual classical concert goers from regulars is to notice if they applaud based on silence instead of when the piece is over. If the conductor’s arms are still up, the silence is part of the music.

Next to negative feedback, silence is the most important. If you are hearing nothing about a particular area or feature of the system, that’s feedback – either people aren’t using it, it isn’t important, or they can’t figure it out enough to even express their thoughts. If you have metrics about application usage (and if you don’t, you should – passive metrics are much harder to argue with than fuzzy user comments) and know a feature is being used but you aren’t getting anything back then the conductors arms are up – you need to listen carefully.

If your metrics tell you that a feature isn’t being used, this is a problem on its own: Most likely users can’t find the feature when they need it, or they don’t have confidence that it’s better than the alternative approach they have conceived to get the result they want.

On the web, if you notice a shift in browser or operating system usage be particularly wary of silence from a particular community: If you notice a falloff in Safari users, you should probably recheck that your application looks and works well in Safari. It’s easy to forget that each of the major browsers has a dominant platform its designers are coming from (Windows, X Window, MacOS) and that colors how they create standard widgets like combo boxes, lists, etc. in addition to the general problems of creating a well behaved web application in each of the major browsers.

Listen to the Music Today

Regardless of where you are in process of integrating feedback into your product development, there is feedback to be found buried in logs, CRM systems, the Internet, and other sources. Get creative, pull some together, and start analyzing the music. You’ll learn unexpected things that will invigorate your team.

Have a story to share? Please send me a note or post a comment below.