Archive for Software Development
Listen to the Symphony
Posted by: | CommentsSo you’ve done it. You believe that feedback from your users is the ultimate input into what will ensure your product succeeds, and you have a range of mechanisms both passive and active to collect it. You routinely analyze that feedback to understand what it means, and you incorporate it into your development process.
As you move down this road, you’ll inevitably have an experience where you got negative feedback on a feature that you felt was very well thought out. You might make a change to help that feature be even better, only to discover you are getting the same rate or even more negative feedback than before.
Herein lies a problem that can easily derail your team taking feedback seriously: You are always going to have some people that don’t like any given feature of your system, or provide feedback too fast when they don’t first figure something out, only to work it out later and be fine with what you have.
When evaluating user feedback, you need to know that some of the user community will not like any given aspect of your product. What you’re listening for isn’t the individual comments – the notes played by just one instrument – but the overall feeling of the music created by all of the comments.
User interface design and, in a broader sense, all product design is a function of tradeoffs:
- Discoverability: If you make every feature clearly visible, it will take up too much space or confuse beginners. If you put one item in more than one place because it’s important some users will think a different feature should get the same or better treatment based on how they use the product.
- Performance: No matter how fast your application is, there will be a user with a computer slower than that who will complain that the application is almost, but not quite, fast enough to be usable. Or they will use it in increasingly silly ways until performance suffers.
Music isn’t Statistics
Listening to the Symphony is not an exercise in statistics. It’s not as simple as counting how many instruments are playing a line or what they are. The individual notes played by the musicians combine in very complicated ways to create the net effect you experience listening to the piece.
Likewise, your feedback as a whole shouldn’t be simply lined up and counted. Who the feedback is coming from is important, not just how many. For example, is the user an early adopter or a mainstream user? Are they part of your past market or where you are expanding? If your product has crossed the chasm, the early adopters are ready to move on and frankly aren’t where you need to focus your attention. On the other hand, if your product is young then the early adopters are everything and mainstream users that don’t get it may not be reachable even if you addressed their concerns because they just aren’t ready to be in your market.
Depending on how you gather the information, it may be relatively easy or may require looking at each piece and gauging where the submitter is in the technology adoption cycle relative to your product. While you are in transition from early adopter to mainstream use, the feedback of the early adopters will still overwhelm that of your new market statistically, so it’s important to have a method of giving weight to feedback by user market instead of just as a whole.
Solos are Rare
Occasionally you’re going to hear a solo comment that you know is right on – the user clearly explained the problem, it’s a legitimate intended use of the product, and you can conceive a way to change your product to handle it. Solo comments aren’t frequent. More often you’re going to get a lot of little comments about areas of the system.
One place where solo comments tend to happen is with modern social media (Blogs, Wikis, forums, etc.). The Internet gives essentially anyone a personal soapbox to broadcast their opinions, and modern search and indexing tools are uncanny in their ability to pull out these solos and amplify them across both time and distance. Get in the habit of connecting with good search tools so you’re in the audience. There is a great deal of good will generated when you reach out quickly to connect with either a strong positive or negative commentary; your contribution to the chorus will be found as easily and most readers will listen to the whole piece instead of just the solo.
Listen to Trends
When listening to a great symphony, you’ll hear many of the same basic musical themes repeated in each movement with slight variations. As you pay more attention to the music, it’s the variations that stand out in each movement. The variations could be tempo, volume, timbre, notes… many small details that extend or change the previously introduced musical phrase.
When evaluating the bulk of the feedback you receive look for what’s changing over time, particularly between releases where you attempted to address user concerns or introduce new features. Is the overall amount of negative feedback going up? If you’re hearing more dissonance, even if it isn’t localized in one area, you need to look for general issues that could be increasing user confusion. You want to interpret the trends in the context of where you are and where you’re going. At a higher level than simply noticing the motifs in the piece is understanding those motifs within the broader context of the overall mood and emotional impact of the piece.
A change in feedback may have nothing to do with what you did to your product at all but instead a change in user expectations. For example, they may be trending to a new browser that doesn’t work as well with your product or they might be getting concerned about identity theft or natural disasters or some other real-world change that users are coupling with an aspect of your application.
One reason I love to be involved in the sales process is to hear the questions being asked by prospects. I find that about every six months the dominant (top 5) questions change with something dropping out because it’s just assumed handled and replaced by the hot new concern. If you can catch the tip of these changes (for example, when customers start asking about federated identity) then you may be able to make product changes in time to catch the wave (integrate with federated identity managers your customers are likely to want) or know exactly why you’re going to let it pass you by.
Silence is Music Too
One quick way to tell casual classical concert goers from regulars is to notice if they applaud based on silence instead of when the piece is over. If the conductor’s arms are still up, the silence is part of the music.
Next to negative feedback, silence is the most important. If you are hearing nothing about a particular area or feature of the system, that’s feedback – either people aren’t using it, it isn’t important, or they can’t figure it out enough to even express their thoughts. If you have metrics about application usage (and if you don’t, you should – passive metrics are much harder to argue with than fuzzy user comments) and know a feature is being used but you aren’t getting anything back then the conductors arms are up – you need to listen carefully.
If your metrics tell you that a feature isn’t being used, this is a problem on its own: Most likely users can’t find the feature when they need it, or they don’t have confidence that it’s better than the alternative approach they have conceived to get the result they want.
On the web, if you notice a shift in browser or operating system usage be particularly wary of silence from a particular community: If you notice a falloff in Safari users, you should probably recheck that your application looks and works well in Safari. It’s easy to forget that each of the major browsers has a dominant platform its designers are coming from (Windows, X Window, MacOS) and that colors how they create standard widgets like combo boxes, lists, etc. in addition to the general problems of creating a well behaved web application in each of the major browsers.
Listen to the Music Today
Regardless of where you are in process of integrating feedback into your product development, there is feedback to be found buried in logs, CRM systems, the Internet, and other sources. Get creative, pull some together, and start analyzing the music. You’ll learn unexpected things that will invigorate your team.
Have a story to share? Please send me a note or post a comment below.
If Not Your Friends, Who?
Posted by: | CommentsIf you are responsible for a product, you need to actively seek out negative feedback from the best friends of your product. Sure, these folks probably give you a lot of positive feedback, and while that shouldn’t be ignored, it isn’t likely to help you move forward. If you are in a corporate setting developing products for the company this is doubly important. Even the most maligned project or team has its sponsors and friends. Find them, and work with them long enough to make sure they understand you’re able to handle negative feedback and that you really want it. Act on that feedback, and make sure they know where you got it from to encourage more feedback.
If you aren’t getting useful feedback on your product for where you fall short, you should consider why. It isn’t because your product is perfect, it’s going to be because either:
- Lack of Opportunity: You aren’t providing any straightforward ways of getting feedback from the users to you. Users have many things competing for their time, so you need to lower barriers in the way of them providing you feedback when they experience it.
- Perceived Resistance: If the user believes their feedback won’t be received or even worse if they’ll be called out to argue and defend it, they won’t provided.
Lack of Opportunity
What are you really doing to make it easy to get feedback from users and prospects at the time they conceive it? You want to have the barrier in time and difficulty as low as feasible, but you also need to be sure you collect enough information that your feedback is worth it. If the barrier is too low and you don’t collect much you won’t be able to make sense out of it, and the user won’t seek out a more qualified way. However, it’s better to risk that than have the user experience a moment of frustration or insight and not know where to send that information to you.
- Integrate Feedback: Whatever the application is, make it possible to send you feedback from within it. Web application, client server, whatever. Preferably not via email, but directly to a web service or other clean, hidden means. When you integrate it, you can more easily capture where the user was and some other state information to make the most sense out of what they say.
- On Your Product Web Site: Make it really easy for people that are evaluating, window shopping, getting support for, or otherwise interacting with you over the web to get commentary to you.
- When Interacting with Prospects: One of the hardest, and most useful, things to do if you are part of the sales process is to find out why you didn’t get a customer. Find ways to ask the prospect in a sincere way why they didn’t go with your product. They may not be truthful in their response, and you should make no effort to reverse their decision, but you will set yourself apart from your competition if they get that you are sincere about understanding where you fell short.
Bottom line: The higher the barriers to providing you feedback, the less good feedback you’ll get. Make it high enough and you’ll only get irate feedback that isn’t actionable.
Perceived Resistance
There is a general pact you have to agree to when you ask for feedback: You’ll accept whatever you’re told and not argue with it. You can sometimes ask for clarification, but this has to be done with the greatest care to neither create a burden on the person you’re interviewing or the impression you’re arguing with their feedback. If you don’t preserve this pact, you won’t get subsequent feedback from this individual or even people they talk with.
When you’re tempted to argue with them, remember that whatever they wrote is true for them. If, for example, they say that something wasn’t documented but you can point to the prose, then consider why they felt it wasn’t documented: Was it unclear? Could they not find the documentation? Did the terminology change between what they saw and what was documented, making them think they weren’t connected? Whatever they wrote was true for them in that moment, so look for all the ways that it could be true.
A handy technique is to take the case that what they are saying is true, and then prove that it is true. You can do this verbally with your team: “Take the case that users believe we didn’t document this. How could they come to that conclusion?” The small language change of “Take the case” can shift people’s willingness to have a good discussion because it doesn’t violate their believe of what is true, it just asks them to momentarily suspend it.
Still No Results?
If you aren’t getting negative feedback, it isn’t because there’s no negative feedback to be had. If your product is in use at all, people will have opinions about how it could be better. If you aren’t hearing about it, then there has to be a reason they don’t believe they can give it to you.
If feedback isn’t viewed as anonymous (either because it appears traceable or because your company is small enough that the user thinks you’ll be able to figure them out) and you or your team have a reputation of defending your product, then you’ve created an emotional barrier for them to overcome. You need to go out of your way to break this down.
Your Project is a Black Hole
Posted by: | CommentsA popular technique in project management is to create and maintain a risks list. This list is supposed to track the things that could cause the project to fail to meet its objectives and ensure that each one gets a mitigation strategy. You generally prioritize the risks by looking at the product of impact and probability: How bad would it be if the risk came to pass, and how likely is that to happen.
Risk mitigation is generally done in three flavors:
- Make adjustments now to mitigate the risk preemptively.
- Have a plan ready to go in case the risk materializes.
- Do nothing and wait and see.
Most project managers (particularly the Project Monitor variety) are highly reluctant to take the last approach: Do nothing, wait and see. They want to have a mitigation strategy for every risk, preferably one that involves taking action now so that the risk is fully mitigated and can be removed from the list. This is counter productive to the project because:
- In the short term it will have the team spend energy and time on mitigating risks of ever decreasing probability, on tasks that are pure overhead in the project.
- In the long term it teaches team members to not mention risks because any risk mentioned will have to be mitigated. This is a strong disincentive to open risk discussion, much like obsessing about defects can cause people to not log defects.
If you want to get value out of a risks list, it has to be treated like a “hold harmless” list: The team has to believe that management won’t hold any risks against them, that they won’t have to mitigate any particular risk on the list, and that intelligent decisions will be made on what risks need to be drilled into. One way to help this is for the project manager to have a strong preference to have each risk either be categorized as Do Nothing or at most mitigate when it happens. Have the team accountable for pushing risks to a higher status by their own consensus. This can remove the fear on the part of team members that identifying a risk and speaking it aloud will cause extra work for the team to debate it and have to mitigate it. Instead, focus your team’s risk list as an opportunity for each participant to identify risks and then suggest mitigation for a risk if they so choose.
Can Your Project Create a Black Hole?
There will be some risks that have a reasonable probability and a serious impact that may require more serious intervention, however there is another item that comes into play at this point. Many high impact risks also should have no reasonable mitigation strategy. If the impact is that the project is sure to fail or even the company is sure to fail, unless you believe you can mitigate it with a modest amount of work, you should probably just let it go. This may seem crazy up front – these are your most serious risks yet I’m suggesting that you treat them as less important than others farther down the list. Quoting World Radio Switzerland (and also from The New York Times):
A lawsuit has been filed against Geneva’s CERN research facility over claims its experiments could destroy the planet. Two scientists in Hawaii say the European Organization for Nuclear Research might create black holes that could swallow the earth. The Large Hadron Collider is the world’s biggest particle accelerator and will essentially crash beams of protons together to create what the universe was like right after the Big Bang. It will do so in a massive underground ring stretching underneath the Swiss and French border and the scientists claim CERN has not fully tested the LHC.
From a project management standpoint, this is essentially the definition of a risk with infinite impact: What could be greater than unstoppable destruction of the Earth? This is a Terminal Risk.The practical reality is that if the result of a risk is that your project will fail, and it can’t be mitigated in a way that still has your project succeed, then you have already failed – if it happens, there’s no solution.
In business, these risks happen all of the time: There is a finite risk that the government will change laws that will ruin the economics of what you do. A competitor could come out with a game changing product that makes you effectively the last buggy whip manufacturer. These are all terminal risks: If they happen, you’re dead. Therefore, spend no time on them at all. Any amount of time you worry about them will not help your project succeed and will only demoralize the team. Additionally, once your team gets focused on a terminal risk, they will start identifying other similar terminal risks, and there’s no running out of them. It is a spiral that doesn’t produce any gain to your project or team.
Instead, focus your energy on the middle set: risks that are reasonable to mitigate, have a fair shot at coming to pass, and would have a real impact to the project. Those are the ones that you’ll have difficulty explaining why you didn’t do anything about them in a post-project review, and your team will likely feel better about the project and the process in response.
64-Bit is Big. Inconceivably Big.
Posted by: | CommentsAs part of the product set that my company develops we are being very aggressive about automated unit testing and validation. Our product is used primarily by companies that operate custom developed software systems to monitor and ensure they have reliable systems, and accordingly they expect it to do no harm and essentially be lean, mean, and bulletproof. One of our set of unit tests was designed to ferret out problems by varying numeric inputs by their smallest increment and checking the output for discontinuities. This technique had proven surprisingly effective at discovering a range of existing defects in everything from the Intel and AMD processors through the .NET runtime. When we would find an issue, we’d verify it and then put a workaround in our calculation routines so that our users wouldn’t have to worry about them.
Recently we were introducing a new set of 64-bit specific tests and ran into a challenge: It wasn’t possible to exhaustively validate the routines. Our automated build system can run around 500,000 tests per second, meaning it takes about 2.5 hours to process all possible 32-bit values. But, to cover all 64-bit values we’d need to run that 2.5 hour test about 4 billion times. This works out to be well over a million years, give or take a rounding error of all recorded history. We intuitively knew the test would take a while, and had bandied about a range of possible times, but in the end we failed to comprehend that 64-bits is the square of 32-bits.
That’s A Big Twinkie
Think about it. In your life you often have to deal with linear approximation and even multiplicative approximation, but never exponential. About the biggest effect we can intuitively get our head around is an order of magnitude – 10 times a value. If you see one car, you can visualize 10, possibly then mentally extend that to 100 (two orders of magnitude) by mentally creating ten rows of ten cars. Now go one more order of magnitude to 1000 by say visualizing a 10 story parking garage with each floor having 100 cars. By going an order of magnitude at a time, you can get through around three before it becomes meaningless. That is roughly the difference between one megabyte and one gigabyte. This is a ratio we’ve developed some familiarity with: Memory has (until very recently) generally been thought of in megabytes and hard drives in gigabytes. Only in the past year or two has it become commonplace to talk about memory in gigabytes and hard disk space in terabytes, each around 1000 times greater than their predecessor.
But what we’re talking about with the difference between 32 and 64 bits isn’t about orders of magnitude. A 64-bit value can contain over 4 billion sets of 4 billion values each – nearly 100 orders of magnitude greater. In traditional Big-O notation for understanding algorithmic complexity you drop out purely additive or multiplicative affects as inconsequential and focus on exponential effects. Despite having been both trained in this and working around complex software for nearly two decades, with the incredible increase of processing power in the last 10 years it has become very easy to forget that things can still take real time to calculate, sometimes nearly infinite time.
The Last of My Lifetime?
We have also become conditioned that every doubling in bit width is only a temporary reprieve: In my career we’ve gone from 8 to 16 bit, 16 to 32, and now are starting 32 to 64. It’s natural to project forward and assume that we’ll similarly need to move to 128 bit within a decade, but these doublings in bit widths are exponential increases in size, and the step from 32 to 64 is vast in the amount of area it opens up. When Windows NT shipped with 32 bit support, it was possible to buy a computer with 1 GB of RAM or one quarter the entire 32 bit address space. That would require approximately one billion gigabytes of memory to equal for 64 bit. Just to purchase that would require around 25 billion dollars today, and would nearly consume the entire world production of DRAM for one year. It isn’t the same game anymore, we aren’t going to see 128-bit memory spaces any time soon, at least not for the traditional reasons we’ve upped bit width. To a certain degree that’s a shame – Having gone through the other transitions before which were the cause of years of trauma to hardware and software developers alike I’ve been amazed at how our entire .NET product just worked the first time on 64-bit. That’s progress.
This Article is Late
Posted by: | CommentsThat’s right – I set a schedule of publishing an article every Monday and Thursday around 1 AM. If I’m ahead in getting articles ready, I queue them up for the future. But the queue has been empty this week, so this article is late. Now, I’m a creative guy so I have all kinds of reasons excuses for why I didn’t have an article ready either Monday morning or yesterday morning, but that doesn’t change the fact that I didn’t have it ready, so there wasn’t one, so this article is late.
It’s easy to treat being late but with a good reason as being the same as being on time. But they aren’t – and you damage your reputation for reliability if you act like they are.
I used to have an employee that worked for me who couldn’t get to work on time. Now, we had flex time so the definition of “on time” was really broad, but one aspect of it was some rough degree of consistency: If you were going to come in around 9:00, let’s make it average within an hour of that over the course of a month. When he was late, he’d come into my office and I got treated to “Well, I wasn’t late – there was traffic on the Beltway” or “I wasn’t late – I had to take the children to day care because my wife’s car didn’t work this morning” or whatever.
The problem is that why he was late didn’t change that he was late. I’m a really empathetic guy: intent and context matter when you move from what happened to what are we going to do about it. I liked the guy, I wasn’t going to fire him or even do any official HR action because he couldn’t get in on time. I was much more concerned that he didn’t really take ownership of being late than anything else. After all:
- We live in a major metropolitan area. When is the beltway interstate not busy between 8 and 10 AM?
- There are other ways to get your kids to day care. They may not be cheap or reasonable, but they do exist.
The bottom line is that when he looked at what was happening in his life, he chose to be late rather than take another option that would have resulted in getting to work on time. The real problem was that he didn’t see it was really a choice; not that he was powerless in the face of the circumstances of his life but instead that he was choosing to be late. In fact, it may well have been the right choice – I certainly wouldn’t ask him to go to extraordinary or expensive lengths to avoid being late.
Own Your Results
Most software projects I’ve worked on have been late. In fact, I’d say nearly all if you pulled out the letter of the law – did we deliver every feature to every user on or before the delivery date. This is hardly news to anyone that works with software development projects: Most studies treat these projects as “on time” if they deliver 80% of the functionality within 80% of schedule or some other similar tolerance.
As IT professionals, we’ve so internalized that software projects run long, that it’s easy to not see all of the decisions that we made as a development team to be late. It’s a lot easier to assume that we are powerless in the face of the circumstances of our project. But we make choices all of the time that drive being late, and in many cases it may be that the alternative aren’t reasonable (they would be very costly, politically unacceptable, or business unacceptable) but they do exist, you are not powerless to the unpredictability of software. It starts with accepting the development schedule in the first place.
The next time you’re late with something – personally or professionally – find out how powerful it is to just say clearly and emphatically We Were Late, and then move quickly to what are we going to do about it. Arguing about what happened (whether you are or aren’t late) isn’t going to get you to a good answer about what to do about it. In the same message where you deliver that you’re late, make it clear what the impact to others is: What it means to the company, your customers, generally anyone that depends on you.
This may feel like professional suicide, but I’d offer up that the more your organization tends to eat those that admit their mistakes, the more powerful it is to take this approach. As a friend used to remind me, they can’t kill you, they can only fire you. Think about it: If they did, are they the organization you want to work for? Really?
Instead, you’re most likely to find that it defuses the situation: People are ready to confront you with all of the evidence of how late you are and what it means to them. If you’re on the same side of the conversation they are, how much energy can there be in the confrontation?
If you have the reputation as being the first to own up when you can’t do what you said you would, you will have a reputation for reliability even when you aren’t.
