Reliable Systems

It’s very popular to consider the internal users of IT services as customers, acting like IT is an in-house service provider that the rest of the company purchases services from. The goal behind this is usually a reaction to a real or imagined belief that IT isn’t being responsive to the needs and budget of the rest of the company. The thinking goes that by having IT think of the rest of the company like an outside organization would of its customers, you can ensure better accountability and buy-in. Typically, organizations that go down this road also adopt a charge-back model where the IT organization charges back all or nearly all of its costs directly to the other divisions within the company that are consuming those services.

While there are several positive aspects that can come from this approach, there are several problems that can easily be created that stem from the problem that in most cases the rest of the company really isn’t a customer in the classical sense. Why? Because they lack a true buying choice. Furthermore, it generally isn’t in a company’s overall best interest for their divisions to really be customers of their own organization.

The original motivation for taking these approach is usually to address several issues:

• Division buy-in on costs and priorities: If they are directly paying the bill, they are going to pay for what they want and not ask you for things they aren’t willing to pay for.
• Clear status and communication: The project reporting and communication model is simpler for everyone to get their head around if it’s based on something we’ve very familiar with. Each player can figure out their part.

If you model the relationship between the IT organization and the rest of the company as a service provider – customer relationship, it’s easy to miss the transitive qualities of this: if they are your customer, you are their vendor. The word Vendor casts things in a different light: If you’re a sufficiently large organization you probably have a vendor management office whose sole job is to ensure you pay the least you can for things and fosters competition between vendors. Their job is largely to keep the company from getting too cozy with any one vendor. Are you ready to be just another vendor, like the one that bids annually to supply fresh coffee or office supplies?

Benefits

There are several good things that this model will tend to create.

• Defensible functional requirements: Unreasonable requirements tend to be expensive relative to their value, and the division is more ready to discard them.
• Role Clarity: The Vendor/Customer relationship is relatively easy to understand, and each party can generally determine their role quickly. When there are disputes, there’s a natural framework for resolution.

Challenge One: Buying Choice

It isn’t a long road from treating your internal divisions as customers until they look at you as a vendor. Once they consider you just another vendor (like the one they selected to provide fresh coffee to the office, or office supplies) they’ll want the advantages that come along with being a customer. For example, it’ll seem clear to them that it should be optional to use your services. This will feel very reasonable to upper management – it’s all part of the transitive nature of IT being accountable.  If IT can’t deliver a service at the best price, why not go to another provider?

This will likely start with something that will be difficult to argue against – such as a large software development project, perhaps in a language that your in-house talent isn’t familiar with.  Now, what about hosting for that product?  If you are charging back true costs for your data center to each division, you are unlikely competitively priced with what a division could get from Rackspace or Peer1.  It isn’t necessarily that those companies are more efficient than you are at doing the same thing (indeed, if they are then you should broker your own contract with them) but instead that it isn’t an apples-to-apples comparison.

Challenge Two: Implied Requirements

Whenever an internal IT organization takes on a project, there are a number of implied requirements that affect cost and schedule. Some of these requirements are from the IT organization itself (like technology choices) and others are from the corporation (role of internal staff and contractors, project management and reporting standards, etc.). When a division looks to bid out work to an external source, these requirements are usually unstated because in many cases they aren’t requirements the division has on the solution.

Another way to look at it is that any constraint on the solution that the customer (the division in this case) doesn’t have or care about is an implied requirement and likely a competitive disadvantage when comparing internal IT costs with external costs. In broad strokes, the difference in requirements is that a division’s requirements are almost entirely about outcomes, not methods: They care about the results their users get, not how they are achieved. IT organizations often focus their requirements on how results are achieved (using this technology, in that enterprise architecture, developed with our RUP-based approved process, tracked by our PMO) and they defer to the division the functional requirements.

Local Maxima and Minima

When each division or cost center is free to chose what services they are willing to pay for, they will converge over time on only those services that are good for them. Establishing shared services is generally challenging because each party will want to ensure that everyone is paying their fair share. This is often tricky to define – should it be proportioned by feature usage? Capacity?  This often creates a “first mover disadvantage” scenario where no part of the company wants to be the first to get a new service such as a database server or SharePoint Portal because they’ll be hit with the entire cost of it unless someone else comes along.

Secondarily, upgrading services gets challenging because no drop of rain believes it is responsible for the flood: If you want to upgrade to Exchange 2007 from Exchange 2003, one division can easily say that they don’t believe it’s necessary and thus decline the costs. If you need a larger server to house SharePoint, who is going to get the bill? A game of chicken often gets created where multiple parties all want a service, but no one wants to be the first to ask and risk subsidizing everyone else.

With each cost center pushing to only pay for those things it perceives sufficient direct value to take on, they are making decisions only based on what gives them the best cost or maximum value. This isn’t likely to align with providing the overall lowest costs for the company. For example, three separate departments could easily decide to implement their own direct attach storage for disk because none of them feels they can justify the cost of a SAN, however together it would be less expensive to construct and maintain a central SAN environment with SAN backup.

There are some straightforward exceptions to this problem where shared services are generally easy to get consensus on and cost out. Typically these are raw infrastructure services such as email or file storage where there are clear units of measure that allow for proportional billing (mailboxes and gigabytes used, for example).

An Alternative: Clients, not Customers

If the Customer/Vendor model isn’t the overall best approach for a company, what alternative model can provide the benefits without the unintended consequences? How about a term that’s between User (which has accumulated a substantially negative connotation) and Customer – Client. A quick trip to the dictionary shows that a client is any person or group that is the party for which professional services are rendered, which fits reasonably enough.

As your clients, they still are entitled to a great deal, just like customers would be. As the client of the project, they:

• Determine success & failure: Your project isn’t successful just because it follows the corporate processes or works on the corporate approved IT infrastructure; those are the constraints on how you solve problems that are immaterial to your client. Success is determined by whether you achieved the goals the client created. That may mean you need to do some extra communication to make sure your client knows that their goals were met, even if that’s not in the standard process.
• Decide if it’s worth the price: In the end, the problem may just not be worth solving.  Many things can be done but the cost in time and distraction exceeds the value.

Unlike a customer, since you’re part of the same organization you can share with the client your insight into the costs and risks of the project in a way that no vendor ever could.  In the end this creates the best partnership that delivers long lasting results.

A final note

If you don’t treat your users as clients, odds are very good they will eventually get themselves a buying choice. When they do, they won’t chose you. Don’t let it come to that, it isn’t ultimately in their interest, your interest, or your company’s interest.

Many if not most software project leaders came up through the development ranks.  It’s generally thought of as a distinct advantage – you know the technologies you’re using, you can form your own well reasoned opinions about how hard something is, what is possible, and how long it should take.  For a long time, I felt that the best way to get results from development teams was to use my experience and knowledge to be very understanding of the challenges they faced and give them whatever time they asked for.  However, in the last few years I’ve run into several situations where I just couldn’t get them the extra time or relief from the most problematic requirements.  I predicted doom to the projects in question but instead I observed some of the best outcomes I’d ever experienced.

While the projects were successful, it bothered me that the secret sauce seemed to be a rigid adherence to schedule and delivery more than any other consideration.  This was exactly the reverse of how I wanted projects to succeed:  I wanted them to succeed because I was treating the developers how they always wanted to be, not like a stereotype from Office Space.  How could it be that better results came from ignorance of the technical details involved?

Developers Will Use All Available Time

Upon reflection, the first thing that struck me was how much an immobile deadline focused discussions and decision making. If you give a team more time, they will expand their process to consume it.  Time will get consumed by:

• Elaborate Decision Making: When you have little time, you make a choice and go with it until it appears it just can’t work.  When you have a lot of time, you sit back and look for the very best option.  That then requires defining what the best is – is it fastest, or smallest, or most scalable, or whatever.
• Development Approach: Under pressure you’ll tend to go with the proven guaranteed approach.  If you have the luxury of time you’re more likely to engage in yak shaving like investigating a new tool or approach, or writing several prototypes first before you develop the real solution.  You might even just throw caution to the wind by skipping a formal design figuring you’ll have the time to just code and test your way to a solution.

The more time a development team has, the harder it is to argue against spending it on up front luxuries.  It also can be harder to argue for long term best practices because the team has the time now to develop a solution any way they want.

Unknowns Create Boomerang Estimates

Even very experienced developers are generally terrible at estimating the duration of developing a solution.  This has been demonstrated over and over by many other parties.  The key behavior that we’ve observed is the phenomenon that from when you approach a specific development problem (like displaying a graph on a web page) until you know exactly how you’re going to solve it (and have a reason for confidence in that approach) you will tend to estimate high because in effect the only reasonable estimate is infinity.

Put another way, as long as you don’t know how you will solve a problem you don’t know for sure that it is solvable which means it will take an infinite amount of time to solve it.  Fortunately, developers are almost universally optimists so they believe they can solve anything eventually – so they’ll pull out a standard answer like three weeks or months or whatever feels like a big chunk of time to figure out the problem but not so big that it kills the project.   The reality is that until you know how you’re going to solve it, it feels like it could take forever.

Once a solution has presented itself  the development team will often find that all it will take is some cleanup and polish to be done- a very small amount of time.  What will push the team to find the answer?  We’re back to the problem of elaborate decision making when you have the luxury of time.  Finding solutions tends to not be a linear problem that will be solved with incremental development energy.  Instead, it tends to be solved by getting people together and brainstorming possible solutions until you find a few candidates and can work out what it’ll take to prove them out.  Under pressure, people tend to focus their creative energy and be more willing to compromise.   That flexibility will tend to get rid of pet requirements and developer gold-plating and focus on the most critical aspects of the problem.

What’s the alternate approach?

The key is to not let your knowledge and experience as a developer lead you to buy into the stories the team creates around what’s reasonable to get done and how long it will take.  Instead, you have to stick with the project’s goals first then the facts of the project.  The project’s goals form the objective reality of what has to be accomplished for the project to survive:  Deliver this functionality by that date, keep these people informed, solve these problems without causing those problems.

When the team runs into a wall and needs more time, instead of buying into the story of needing a lot of time, set a specific and tight goal that keeps a solid amount of time pressure on the team to solve the issue and prevent the problems above from showing up.  Ideally, find a way to give out one or two day chunks to answer incremental questions if necessary to emphasize that time is precious and has to be invested carefully.  This is where you can leverage your experience in a way that a non-developer can’t:  The team knows they can’t snow you with tech details, and you can define a specific, measurable result that can be achieved in a short period of time that they can’t argue with.  Despite this, you are bound to have to assert a few times that the time limit is the limit – solve the problem in that time.  It’s very hard because you’ve been on the other side of that conversation and it can feel like you’re the Pointy Haired Boss, but it’s fundamentally your job on the project.

What will nearly always happen is the team will surprise itself – a solution will be presented within the team that they can live with and can be done in the time they have.  It may be incomplete or have some risky shortcomings, and you’ll want to ask how long it’d take to address those.  You probably shouldn’t address them in the first round, but the team will feel better that you’ve considered through things and will buy into the outcome more if you ask.  You’ll also want to make a record of it so that the team can in the future recognize what was a predicted shortcoming vs. an accidental defect.

Do you want it solved right?

This is a question that often gets voiced within a team as a rebuttal to external time pressures and is very dangerous.  The challenge is that most non-technical people don’t get the number of ways that a problem can be solved: instead, each problem appears to have a single solution.  Take away your technical knowledge and imagine you’re the paying customer:  What’s the alternative – were you going to solve it wrong? If that’s the case, what else have you done that’s garbage?  If you took your car to a repair person and they said it’d be $500 to fix it, then when you came back they said well, if you want it fixed right it’ll actually be $1200, wouldn’t you wonder what the hell the $500 fix was?

Usually this statement is uttered in desperation when a team believes they just need more time to figure out a problem.  Nobody wants a problem solved wrong.  Skip the hyperbole and get down to action:  break down the problem into small chunks of time that can be invested for a specific measurable result, and make sure the team gets that overage time is the most precious commodity.

Side Note: This is an advantage of SCRUM in practice.  If you’re following an Agile Development practice, particularly SCRUM, this fits right in:  Focus on making each sprint deliver the user stories it was supposed to even if you have to leave some special cases for a later sprint.  The daily stand up meetings are a great place for the different team members to apply team pressure against over engineering and doomsday estimates.

Cleaning Up and Closing Out

At some point you need to close out your release and ship it. For each of the areas where you’ve had to make compromises and taken shortcuts you have to choose to either:

• Ship as Final: Decide the implementation is close enough to the intent of the end-user functional requirements that it can be the final implementation (at least until new information contradicts this decision)
• Ship as Temporary: Decide that something is better than nothing and ship the feature with limitations.
• Cut the Feature: Hold back the feature until it can be reconsidered or reimplemented.

You’re nearly always better off shipping the feature, often as a final feature pending more information because it’s very hard to gauge the true impact of each limitation.  This is particularly true of user-facing features and environments where it’s possible to evolve the software rapidly.  Inevitably once it’s in the hands of your users you’ll discover aspects of it that you didn’t think of that will require rework and you may discover that the killer feature you were sure would be the hit of the release is hardly used.  In either of these cases if you’ve invested a great deal of time in making it foolproof the team will tend to resist changing it.  It’s a natural product of the presumed relationship between effort and value.  If necessary, you might put in some temporary safeties to detect and catch the limitations you’re worried about.

The major exceptions to this approach are areas that are too dangerous to deploy if less than fully trustworthy.  For example, if your team is developing a data storage system, software deployment system, or other critical infrastructure your choices likely resolve down to making it as right as possible or holding the feature until it can be reworked.

If it turns out that the solutions that are viable within the schedule have significant limitations, you should make sure these caveats are known to the business – provided you can express them in business terms.  For example, knowing that an algorithm won’t work if your userbase doubles is probably not a significant caveat, unless you know the business plans to double in a relatively short period of time.  Every system has limits, and every software change has risks.  Business representatives don’t like to hear the same items covering the same ground repeated every time you discuss software, and it tends to make them not hear the new and important information as well as sound like you’re attempting to transfer accountability from your team to them.

If you are responsible for a product, you need to actively seek out negative feedback from the best friends of your product. Sure, these folks probably give you a lot of positive feedback, and while that shouldn’t be ignored, it isn’t likely to help you move forward. If you are in a corporate setting developing products for the company this is doubly important. Even the most maligned project or team has its sponsors and friends. Find them, and work with them long enough to make sure they understand you’re able to handle negative feedback and that you really want it. Act on that feedback, and make sure they know where you got it from to encourage more feedback.

If you aren’t getting useful feedback on your product for where you fall short, you should consider why. It isn’t because your product is perfect, it’s going to be because either:

• Lack of Opportunity: You aren’t providing any straightforward ways of getting feedback from the users to you. Users have many things competing for their time, so you need to lower barriers in the way of them providing you feedback when they experience it.
• Perceived Resistance: If the user believes their feedback won’t be received or even worse if they’ll be called out to argue and defend it, they won’t provided.

Lack of Opportunity

What are you really doing to make it easy to get feedback from users and prospects at the time they conceive it? You want to have the barrier in time and difficulty as low as feasible, but you also need to be sure you collect enough information that your feedback is worth it. If the barrier is too low and you don’t collect much you won’t be able to make sense out of it, and the user won’t seek out a more qualified way. However, it’s better to risk that than have the user experience a moment of frustration or insight and not know where to send that information to you.

• Integrate Feedback: Whatever the application is, make it possible to send you feedback from within it. Web application, client server, whatever. Preferably not via email, but directly to a web service or other clean, hidden means. When you integrate it, you can more easily capture where the user was and some other state information to make the most sense out of what they say.
• On Your Product Web Site: Make it really easy for people that are evaluating, window shopping, getting support for, or otherwise interacting with you over the web to get commentary to you.
• When Interacting with Prospects: One of the hardest, and most useful, things to do if you are part of the sales process is to find out why you didn’t get a customer. Find ways to ask the prospect in a sincere way why they didn’t go with your product. They may not be truthful in their response, and you should make no effort to reverse their decision, but you will set yourself apart from your competition if they get that you are sincere about understanding where you fell short.

Bottom line: The higher the barriers to providing you feedback, the less good feedback you’ll get. Make it high enough and you’ll only get irate feedback that isn’t actionable.

Perceived Resistance

There is a general pact you have to agree to when you ask for feedback: You’ll accept whatever you’re told and not argue with it. You can sometimes ask for clarification, but this has to be done with the greatest care to neither create a burden on the person you’re interviewing or the impression you’re arguing with their feedback. If you don’t preserve this pact, you won’t get subsequent feedback from this individual or even people they talk with.

When you’re tempted to argue with them, remember that whatever they wrote is true for them. If, for example, they say that something wasn’t documented but you can point to the prose, then consider why they felt it wasn’t documented: Was it unclear? Could they not find the documentation? Did the terminology change between what they saw and what was documented, making them think they weren’t connected? Whatever they wrote was true for them in that moment, so look for all the ways that it could be true.

A handy technique is to take the case that what they are saying is true, and then prove that it is true. You can do this verbally with your team: “Take the case that users believe we didn’t document this. How could they come to that conclusion?” The small language change of “Take the case” can shift people’s willingness to have a good discussion because it doesn’t violate their believe of what is true, it just asks them to momentarily suspend it.

Still No Results?

If you aren’t getting negative feedback, it isn’t because there’s no negative feedback to be had. If your product is in use at all, people will have opinions about how it could be better. If you aren’t hearing about it, then there has to be a reason they don’t believe they can give it to you.

If feedback isn’t viewed as anonymous (either because it appears traceable or because your company is small enough that the user thinks you’ll be able to figure them out) and you or your team have a reputation of defending your product, then you’ve created an emotional barrier for them to overcome. You need to go out of your way to break this down.

That’s right – I set a schedule of publishing an article every Monday and Thursday around 1 AM. If I’m ahead in getting articles ready, I queue them up for the future. But the queue has been empty this week, so this article is late. Now, I’m a creative guy so I have all kinds of reasons excuses for why I didn’t have an article ready either Monday morning or yesterday morning, but that doesn’t change the fact that I didn’t have it ready, so there wasn’t one, so this article is late.

It’s easy to treat being late but with a good reason as being the same as being on time. But they aren’t – and you damage your reputation for reliability if you act like they are.

I used to have an employee that worked for me who couldn’t get to work on time. Now, we had flex time so the definition of “on time” was really broad, but one aspect of it was some rough degree of consistency: If you were going to come in around 9:00, let’s make it average within an hour of that over the course of a month. When he was late, he’d come into my office and I got treated to “Well, I wasn’t late – there was traffic on the Beltway” or “I wasn’t late – I had to take the children to day care because my wife’s car didn’t work this morning” or whatever.

The problem is that why he was late didn’t change that he was late. I’m a really empathetic guy: intent and context matter when you move from what happened to what are we going to do about it. I liked the guy, I wasn’t going to fire him or even do any official HR action because he couldn’t get in on time. I was much more concerned that he didn’t really take ownership of being late than anything else. After all:

• We live in a major metropolitan area. When is the beltway interstate not busy between 8 and 10 AM?
• There are other ways to get your kids to day care. They may not be cheap or reasonable, but they do exist.

The bottom line is that when he looked at what was happening in his life, he chose to be late rather than take another option that would have resulted in getting to work on time. The real problem was that he didn’t see it was really a choice; not that he was powerless in the face of the circumstances of his life but instead that he was choosing to be late. In fact, it may well have been the right choice – I certainly wouldn’t ask him to go to extraordinary or expensive lengths to avoid being late.

Own Your Results

Most software projects I’ve worked on have been late. In fact, I’d say nearly all if you pulled out the letter of the law – did we deliver every feature to every user on or before the delivery date. This is hardly news to anyone that works with software development projects: Most studies treat these projects as “on time” if they deliver 80% of the functionality within 80% of schedule or some other similar tolerance.

As IT professionals, we’ve so internalized that software projects run long, that it’s easy to not see all of the decisions that we made as a development team to be late. It’s a lot easier to assume that we are powerless in the face of the circumstances of our project. But we make choices all of the time that drive being late, and in many cases it may be that the alternative aren’t reasonable (they would be very costly, politically unacceptable, or business unacceptable) but they do exist, you are not powerless to the unpredictability of software. It starts with accepting the development schedule in the first place.

The next time you’re late with something – personally or professionally – find out how powerful it is to just say clearly and emphatically We Were Late, and then move quickly to what are we going to do about it. Arguing about what happened (whether you are or aren’t late) isn’t going to get you to a good answer about what to do about it. In the same message where you deliver that you’re late, make it clear what the impact to others is: What it means to the company, your customers, generally anyone that depends on you.

This may feel like professional suicide, but I’d offer up that the more your organization tends to eat those that admit their mistakes, the more powerful it is to take this approach. As a friend used to remind me, they can’t kill you, they can only fire you. Think about it: If they did, are they the organization you want to work for? Really?

Instead, you’re most likely to find that it defuses the situation: People are ready to confront you with all of the evidence of how late you are and what it means to them. If you’re on the same side of the conversation they are, how much energy can there be in the confrontation?

If you have the reputation as being the first to own up when you can’t do what you said you would, you will have a reputation for reliability even when you aren’t.

I grew up as the third son in our family. When my oldest brother was a newly minted driver, like every new driver he was a little rough. And like any younger siblings, my other brother and I were kind and gentle in our commentary about it. This led him to declaring his first driving rule: No comments on his driving when he was driving. He was in command, and that was it.

One day soon thereafter, he was backing the car out of the garage with my other brother and I in it. Now, he didn’t normally park on the right side of the garage – that was where my dad’s car went. But by whatever fluke, there the big Ford station wagon was – on the right side of the garage. When backing out, you have to start turning right away because the driveway isn’t straight. In fact, you have to start turning left, meaning the front of the car goes to the right. Normally this was not a problem since there was plenty of clearance. But when starting on the right side of the garage, on the right is… the garage. As he started backing out, my brother and I quietly sat there, watched the side of the garage come right up and *bang* *crunch* we hit it. In the “after action review” that followed, my brother exclaimed “Why didn’t you tell me I was going to hit the garage?” You can imagine our response – “Because you had said never comment on your driving.”

At the time, I was smug in my righteousness. We had done exactly what he’d asked, we weren’t the driver and the driver is responsible for the car, so it was a big ‘not-my-problem’.

We were dead wrong.We were in a position to have prevented the problem, and we should have spoken up. Ever since, we’ve had the rule in our family that when riding in a car. The rule is to speak up if you see a problem without fear that the driver will be upset. The potential consequences of not calling a problem to the driver’s attention are too great.

How Do You Play the Blame Game?

The same story often plays out in the aftermath of a technology problem. Hang around a software development team long enough and you’re bound to hear a developer complain “Why didn’t QA find that defect? They should have found it before it shipped.” The difference between an experienced, healthy team and an amateur team is whether the developer is just venting or actually believes they are justified.

We often have a strong desire to try to reduce accountability for avoiding issues to a single party:

• QA is responsible for finding all defects in the software before it is released.
• IT Operations is responsible for keeping all of the servers running.
• The receptionist is responsible for ensuring we don’t run out of coffee.

Before looking at the contentious examples, look at just the last one. Say that you noticed that you pulled the next-to-last box of K-cups out of the supply cabinet. You’re not out of coffee yet – there are 24 individual servings in the box you pulled, one more box on the shelf. In most small companies, that’s at least a day’s worth of coffee. Would you tell the receptionist that you need more coffee? Or just assume that it will be taken care of? Say you then run out of coffee two days later, and everyone has to run out to Starbucks to feed their habit. Would you feel at all responsible for not speaking up when the problem was still avoidable?

You probably would have spoken up – the receptionist is a nice person, it’s an easy enough thing to do and you like your coffee.

Now look at the other two scenarios. The only real difference between them and running out of coffee is that these two will tend to be political and possibly even contractual. While you’d likely also speak up if you saw a defect in your company’s product before it was released or if you saw that a server was just about out of disk space, you wouldn’t want to accept any accountability after the fact if things went bad.

Here’s the elephant in the room: Your customers don’t care who was accountable for avoiding a problem. They care that the problem happened. They pay you for something that works (and has to work according to their definition of what works means). Anything else is just internal noise. If you want to drive your business forward – and really, if you don’t, you need to look to work somewhere else – this needs to be your motivator.

Formal vs. Practical Accountability

What if, instead of looking at issues as someone else’s problem, you followed these two principles?

• If you are in a position to prevent a problem, you are accountable for preventing it.
• If you are responsible for ensuring a problem doesn’t happen, you need to stay in a position to prevent it.

This means that many different people and groups may each be 100% accountable for a problem, because the most useful way to look at accountability is based on the ability and responsibility for preventing the problem. Why the most useful? Because the problem happened, that’s a matter of fact. While recriminations, blame, and shame may be cathartic or fun, they aren’t useful because they don’t further the goals of the team or the company. Put simply, your customers don’t care who’s at fault within your organization, just that you get the seriousness of the problem and you’re making it right. When debriefing your team, the ideal outcome is that everyone in the room sees how they could have prevented the problem, and takes on that they should have prevented the problem. From that, you then work into who was in the best place to prevent it – who could have seen it first, and addressed it while it was cheapest to address. You want to have everyone walk out with a balanced perspective of how they could have prevented it and how to identify when you’re in the best spot to prevent it.

A natural concern with this approach, particularly if it’s new to your organization, is that after action reviews are often a game of musical chairs – while there’s a superficial impression of honesty and openness, the true goal is to not be left without a chair when the music stops. Far from a well-calculated political move, this is really an emotional and ego driven outcome. No one likes admitting they are wrong, and with practice people get very skilled at justifying their emotional responses with pseudo-intellectual reasoning – it is called rationalization.

The next time you’re in this situation, try being the first party to speak up about what you could have done to avoid the problem, and make sure you communicate sincere regret you didn’t catch it. If you are completely open in this – sticking just to what you could have done without any back handedness (that’s right – you can’t say “I couldn’t cover up his incompetence.” That doesn’t count.) you’ll be amazed at how quickly the mood in the room changes. Very quickly others will jump in with what they could have done. You’ve created an environment where people can speak the real fears that are on their mind without posturing.

Once you’ve established this environment, you need to be active in maintaining it. If someone jumps into the attack, speak up and redirect the conversation. This is true particularly if the attack isn’t directed at you. Keep listening to have the conversation stay in even tones and that each party is either talking about what their area could have done or is constructively helping the overall conversation.

Eventually, there will be a fundamentally sticky conversation about which party was in the best position to avoid the problem. At this point it’s going to come down to culture – if your culture is one that learns from mistakes, it will be a clear and short conversation. Depending on how strong the duck-and-cover instinct is in your shop, it can be very painful. In the end – speak up if your team is the one that should have the spotlight. Fear of accountability is often overstated. In practice, managers know that in the end they need people that will be accountable for what happened, and the experience can still be positive in the long term. Great managers actively hunt out people that are quick to learn from their mistakes and own them.