Archive for Software Development
Three Levels of Developer Zen
Posted by: | CommentsOne of the first things I’m listening for when interviewing developers is for how they relate to problems with authoring code, namely how do they approach a defect. Depending on how much experience they have and how much they’ve thought about it, I’ve found developers go through three levels of Zen when considering problems:
It’s Not the Compiler
Similar to the classic discussion that Select isn’t Broken, every developer when first starting out runs into something they are just sure is the compiler’s fault. The code is right, the compiler just doesn’t agree. It may be tempting to even use #pragma or the equivalent in the language to get rid of the warning or error.
In the end, it isn’t the compiler, it’s your code. Try explaining carefully and thoroughly to another developer why the code is right and the compiler is wrong, and most of the time in the middle of your explanation it’ll hit you what the real problem is, and that the compiler was right (if perhaps obtuse in how it is reporting the problem). This is an excellent trick to breaking through cognitive bias: Explain to someone else exactly why the world is flat, and in so doing your mind will often make the creative or fresh jump to what the real problem is.
It’s In Your Code
It’s one thing to trust the compiler – compilers have generally matured over decades, and even newer languages like C# and Java have very low defect rates of compiling because even if the syntax is new, the basic lexical parsing is very close to C++ and the long standing family of engineering behind it.
If the compiler is the most trustworthy code, what’s the least trustworthy? The code you’re writing right now. Everything else is more tested and more reviewed. In general, the probability you’re going to run into a defect in an operating system library or framework is relatively low, low enough compared to the probability of defects in your own code as to be statistically insignificant.
Even in areas of high churn, like evolving web standards and CSS for Internet Explorer, if the output isn’t right the defect is virtually guaranteed to be in your code.
It Doesn’t Matter Where It Is – You Have To Fix It
The ultimate evolution of this thinking is practical: It doesn’t matter where the defect is, you have to fix it – and you have to fix it in your code (or at least code you control). In the end, you’re writing software to deliver features for someone, and you’re only successful if it works. They really aren’t going to care whether you can’t make headway because of a problem in your code, an underlying library, or the operating system: They all produce the same effective result of the feature not working. So in the end, it isn’t a question of who’s right but rather what you’re going to do to resolve it.
There is some danger in this last point. There is a fine line between pragmatically realizing you have to make it work one way or another and just hacking away at a problem you don’t understand under the assumption that you’ve found an underlying defect you need to work around. Before you assume this is the case, you should prove not just to your satisfaction you’ve found an honest defect in another library but seek some independent evidence of this: A knowledgebase article, defect report from the library owner, or at least corroboration of your peers is a good place to start. Developers that charge straight to attempting to work around a perceived defect in a library without verification are really back at the first level.
Listen To Where You Are
When I’m talking with developers, I can usually find out where they are by listening for key phrases, such as “it works on my box.” If you hear that and it’s offered in anything other than a sheepish or self deprecating tone, this developer probably has a way to go. Listen to your own language – what do you say when you run into something? If you find yourself getting into a cognitive bias death spiral, that’s the time to get up, take a break, and come back to it. Repeating my favorite trick: Go find a talented friend to explain just how right your thinking is. In my experience, that exercise invariably has me figure out where I’m off track even if the other party can’t follow what I’m saying.
Editors Note: Sometimes you let an article spend too long cooking before you publish it. Jeff Atwood came at this from a very similar angle on Coding Horror while this was in review. Check out his article as well.
Project Leader, Manager, or Monitor?
Posted by: | CommentsAs I work with different clients doing software development, I’ve been struck by how different the Project Managers are. From what I have seen, the people in the role of Project Manager tend to fit into one of three archetypes:
- Project Leaders: Command the technical respect of the development team and the business respect of the client. Generally could write the software if necessary, but it isn’t an appropriate use of their time. Nonetheless, they frequently review code and occasionally contribute something because they just can’t help themselves.
- Project Managers: Typically have experience in either the client’s business or general software development, but augment that with specific training in project management tools and principles. Don’t contribute or review code, but actively participate in functionality and technical discussions.
- Project Monitors: Are trained in project management tools and principles, possibly deeply up to and including being a Certified Project Management Professional®. Wouldn’t dream of contributing or reviewing code, can’t actively participate in technical discussions, and can only lightly participate in functional discussions.
Here’s the question: What type of project manager do you need to have the best outcome of a software development project? For the sake of discussion, let’s say the best outcome is the one that produces:
- The most capability, closest to schedule, with the fewest defects discovered in the first 90 days after release.
- The clearest understanding by the participants and internal observers of the status of development during the project.
The first item is the real challenge, because it revolves around the fundamental software development triangle of quality, functionality, and schedule/budget. The problem is that only one of them is really a variable – functionality. Few people will view a project as being successful if quality is significantly adjusted or if the project is late. An experienced team can optimize the effort put into quality assurance, but the largest swings in project effort come from dropping or changing functionality to either save time or eliminate defects. A feature not implemented is a feature that doesn’t have to be tested and proven.
Adjusting functionality to save time seems very simple on the surface – remove new features from the list and the development time that was estimated can be applied to other aspects of the project. The challenge is – how much time will you really get back if you drop a particular feature? Rarely are features so isolated that this is an easy question to confidently answer. Secondly, what is the probability the feature will stay dropped? What may not seem essential to the development team may be the cornerstone reason your sponsor is funding the project.
To make effective decisions about what features can be cut or reduced, you need to be able to:
- Clearly understand the true incremental benefit in time and quality to the project of the change.
- Clearly understand the ability of the project’s customer/stakeholders to accept the project as being successful even if that feature isn’t available.
This is parked squarely at the intersection of technical and business expertise. If you don’t know enough about the customer, the feature won’t stay cut – it’ll come back at the request of the people funding the project, and usually do so at a much less convenient time. If you don’t know enough technically, you will tend to over-credit the project schedule for each feature removed. Developers are optimists: They will tend to underestimate how much work it will take to deliver on features in the product, and overestimate the initial savings of removing a feature, particularly if it’s problematic to implement.
Pick Your Battles
Project Leaders – those that have the technical and business skill necessary to intuitively understand what it means to add or remove a feature – are most likely to produce the best outcome on a software development project. They still need to know the techniques and tools of project management – otherwise you won’t have the clarity around the project necessary for people to view it as being successful – but they need to come at it with their business and technical hats in equal measure.
Project leaders are in very short supply, so what projects need true project leaders and which can still be successful with project managers or even monitors? Think of the difference between flying a plane and driving a train. A train is on tracks – you only control one variable (speed). Planes have many more variables – not just traveling in three dimensions but also complexities of many more external interactions. A Project Monitor could drive a train. You need a Project Manager for plane, and your Project Leaders? They’re for the fighter jet on a combat mission.
Monitors can track what’s being done, Managers can contribute to how it’s being done, but a Leader understands deeply why it’s being done and can change up the game when needed to win.
Key Times To Pull Out the Big Guns
Projects that break new ground: When the team will need clear guidance on what’s crucial to deliver and what it means to deliver. A leader can achieve more than the others not just because of the other attributes already described but also because the team will accept tactical management better because of the leader’s technical credentials.
Teams with strong personality conflicts: In this scenario you need both peacemaker and mutual respect, so strong technical and business credentials are important to get each party to stop talking and start listening, then to carry through on the project together.
Projects with no obvious variables: While it’s somewhat of a hyperbole that a project can’t tolerate a change in schedule, quality, or functionality it does sometimes come to pass. In particular, some projects have hard deadlines that are externally imposed and are non-negotiable – like compliance with a government regulation or natural business cycle (the holiday season). These projects need to emphasize creativity in approach to achieve results within tight constraints.
Distributed project participation: When the project is very distributed or requires significant interdisciplinary involvement for success it requires a wider range of business and technical skills to bridge all of the small gaps between mindsets. This often requires a fine listening for what people are repeating back to detect deviations from course while they’re still small enough to fix.
What Are You Looking For?
When looking for new project “leaders”, most corporations emphasize project management skills and techniques – IPMA Level B, PMI Certification, and heavy experience with Microsoft Project or some enterprise project management software package. Not surprisingly, this tends to attract project monitors, not leaders. The next time you’re lucky enough to participate in the selection process of a project manager or leader in your company, think about the hard skills you can’t easily teach them first.
Technology is not Scalable
Posted by: | CommentsI was watching Start-Up Junkies the other day which is following a group of people attempting to get Earth Class Mail off the ground. On one recent episode, the main focus was converting their system from being PHP-based to Microsoft .NET.
As you watch the episode, you hear two different reasons given for this transition. The CEO advances the issue that they started with PHP because they needed to get something done quick, but now they need to switch to ASP.NET to make it scale. They always knew they’d have to do it, but now they’re against the wall because they have been invited to demo at a Microsoft conference. The lead engineering staff and operations manager advance a different point: We are about to be demoing in Europe in Microsoft’s booth at a major convention that’s key to our growth, we need to be using Microsoft’s technology for this to happen.
Of these two reasons, which do you think is the better reason for their technology choice:
- Convert to ASP.NET to scale up because PHP can’t scale.
- Convert to ASP.NET because we’re getting marketing and sales assistance from Microsoft, which we’ll only get on their platform.
If you picked #1, two things are likely true: First, you haven’t worked with enough technology to know that the choice between PHP and ASP.NET for scalability is pretty far down on the list of “things that control how much we can scale”. Second, you’re probably not balancing business and technology interests effectively.
Go and check out the technology portfolio used at the most scalable web sites in the world – say the top ten super scalable systems, systems that are going to be at least two orders of magnitude greater than anything you’re likely to create (and this isn’t a negative thing; it’s a liberating thing). Notice anything in particular? You don’t tend to see either Microsoft ASP.NET or J2EE in the web infrastructure. In fact, you tend to see a lot of… PHP.
There are a few key reasons that the super scalable sites like these solutions:
- Open source means they have the source: You can bet that these sites aren’t able to use anything off the shelf. Their needs so outstrip the normal system that it isn’t reasonable that any off-the-shelf framework is going to fit their needs entirely. In fact, if it did that framework is likely seriously over-engineered for the majority of its user base.
- Licensing costs add up: If you’re a small shop, licensing costs are highly unlikely to be a significant percentage of your total cost of goods for a technology product; bandwidth, hosting, and above all people are the big numbers. If you’re Google, you don’t want to pay even $10 in licensing per server. This is similar to a large manufacturer worry about saving $.05 on a bolt; small incremental costs still add up.
- Scalability is their first concern: More important than ease of development, cool debuggers, third party component libraries, or anything else. If it can’t meet their scale, it isn’t even a potential solution. Perhaps more importantly, they have to have both the experience and human belief that it will scale. If you’re one of these sites, there is no way a vendor has tested their solution at your scale – you’ll be the first. If you’re going to be the first, you want to have a simple solution that you can adjust and correct.
If you’re honest, your decision matrix isn’t the same as this. It’s highly unlikely you’ll create the next MySpace, even if you are successful. While the principles of scalability are constant, the importance of scalability vs. other constraints changes. More likely, you need to base your technology choices on a mix of:
- What resources do you have? If you already have a staff of people experienced at technology X, they are likely to produce more results in any moderate interval of time (say one to three months) with this technology than any new one. If you have a large body of existing code in technology X, this is a big accelerator to your project.
- What resources can you get? When picking a technology, buy the community, not the product. If you can take a number of pieces off the shelf, particularly for things you aren’t attempting to innovate (such as security, content management, grid controls, reporting..) it will accelerate your product curve. Conversely, if you can’t get great people that want to work with a technology, it really doesn’t matter how great the technology is.
- What religion is your market? Many markets have a non-rational product selection bias. For example, if you want to sell your product primarily to Macintosh users, you probably shouldn’t use ASP.NET. It isn’t that it should make a difference to how the product works for them, but as a group Macintosh users tend to put “Not Microsoft” on their evaluation lists. Similarly Linux users. Conversely, there are several products that are defined as “just like X, but in ASP.NET!” If your market typically has a technology selection criteria that isn’t based on business or practical fundamentals, it’s best to respect it, otherwise you’ll have to focus additional energy during your sales and marketing efforts to overcome what your market will perceive as a natural disadvantage. The coolest technology, developed quickly and cheaply, is no good if your target customer won’t even invite you to the dance.
Back to Earth Class Mail. Could they scale using PHP? Absolutely, others have. Should they switch to ASP.NET? Probably – they wanted to leverage the marketing advantage of Microsoft. I suspect if IBM was the big animal in the space they wanted and a deal could have been made it would have been WebSphere instead of ASP.NET. Each of these technologies can scale, or not scale, depending on how they are used.
No drop of rain believes it’s responsible for the flood
Posted by: | CommentsI grew up as the third son in our family. When my oldest brother was a newly minted driver, like every new driver he was a little rough. And like any younger siblings, my other brother and I were kind and gentle in our commentary about it. This led him to declaring his first driving rule: No comments on his driving when he was driving. He was in command, and that was it.
One day soon thereafter, he was backing the car out of the garage with my other brother and I in it. Now, he didn’t normally park on the right side of the garage – that was where my dad’s car went. But by whatever fluke, there the big Ford station wagon was – on the right side of the garage. When backing out, you have to start turning right away because the driveway isn’t straight. In fact, you have to start turning left, meaning the front of the car goes to the right. Normally this was not a problem since there was plenty of clearance. But when starting on the right side of the garage, on the right is… the garage. As he started backing out, my brother and I quietly sat there, watched the side of the garage come right up and *bang* *crunch* we hit it. In the “after action review” that followed, my brother exclaimed “Why didn’t you tell me I was going to hit the garage?” You can imagine our response – “Because you had said never comment on your driving.”
At the time, I was smug in my righteousness. We had done exactly what he’d asked, we weren’t the driver and the driver is responsible for the car, so it was a big ‘not-my-problem’.
We were dead wrong.We were in a position to have prevented the problem, and we should have spoken up. Ever since, we’ve had the rule in our family that when riding in a car. The rule is to speak up if you see a problem without fear that the driver will be upset. The potential consequences of not calling a problem to the driver’s attention are too great.
How Do You Play the Blame Game?
The same story often plays out in the aftermath of a technology problem. Hang around a software development team long enough and you’re bound to hear a developer complain “Why didn’t QA find that defect? They should have found it before it shipped.” The difference between an experienced, healthy team and an amateur team is whether the developer is just venting or actually believes they are justified.
We often have a strong desire to try to reduce accountability for avoiding issues to a single party:
- QA is responsible for finding all defects in the software before it is released.
- IT Operations is responsible for keeping all of the servers running.
- The receptionist is responsible for ensuring we don’t run out of coffee.
Before looking at the contentious examples, look at just the last one. Say that you noticed that you pulled the next-to-last box of K-cups out of the supply cabinet. You’re not out of coffee yet – there are 24 individual servings in the box you pulled, one more box on the shelf. In most small companies, that’s at least a day’s worth of coffee. Would you tell the receptionist that you need more coffee? Or just assume that it will be taken care of? Say you then run out of coffee two days later, and everyone has to run out to Starbucks to feed their habit. Would you feel at all responsible for not speaking up when the problem was still avoidable?
You probably would have spoken up – the receptionist is a nice person, it’s an easy enough thing to do and you like your coffee.
Now look at the other two scenarios. The only real difference between them and running out of coffee is that these two will tend to be political and possibly even contractual. While you’d likely also speak up if you saw a defect in your company’s product before it was released or if you saw that a server was just about out of disk space, you wouldn’t want to accept any accountability after the fact if things went bad.
Here’s the elephant in the room: Your customers don’t care who was accountable for avoiding a problem. They care that the problem happened. They pay you for something that works (and has to work according to their definition of what works means). Anything else is just internal noise. If you want to drive your business forward – and really, if you don’t, you need to look to work somewhere else – this needs to be your motivator.
Formal vs. Practical Accountability
What if, instead of looking at issues as someone else’s problem, you followed these two principles?
- If you are in a position to prevent a problem, you are accountable for preventing it.
- If you are responsible for ensuring a problem doesn’t happen, you need to stay in a position to prevent it.
This means that many different people and groups may each be 100% accountable for a problem, because the most useful way to look at accountability is based on the ability and responsibility for preventing the problem. Why the most useful? Because the problem happened, that’s a matter of fact. While recriminations, blame, and shame may be cathartic or fun, they aren’t useful because they don’t further the goals of the team or the company. Put simply, your customers don’t care who’s at fault within your organization, just that you get the seriousness of the problem and you’re making it right. When debriefing your team, the ideal outcome is that everyone in the room sees how they could have prevented the problem, and takes on that they should have prevented the problem. From that, you then work into who was in the best place to prevent it – who could have seen it first, and addressed it while it was cheapest to address. You want to have everyone walk out with a balanced perspective of how they could have prevented it and how to identify when you’re in the best spot to prevent it.
A natural concern with this approach, particularly if it’s new to your organization, is that after action reviews are often a game of musical chairs – while there’s a superficial impression of honesty and openness, the true goal is to not be left without a chair when the music stops. Far from a well-calculated political move, this is really an emotional and ego driven outcome. No one likes admitting they are wrong, and with practice people get very skilled at justifying their emotional responses with pseudo-intellectual reasoning – it is called rationalization.
The next time you’re in this situation, try being the first party to speak up about what you could have done to avoid the problem, and make sure you communicate sincere regret you didn’t catch it. If you are completely open in this – sticking just to what you could have done without any back handedness (that’s right – you can’t say “I couldn’t cover up his incompetence.” That doesn’t count.) you’ll be amazed at how quickly the mood in the room changes. Very quickly others will jump in with what they could have done. You’ve created an environment where people can speak the real fears that are on their mind without posturing.
Once you’ve established this environment, you need to be active in maintaining it. If someone jumps into the attack, speak up and redirect the conversation. This is true particularly if the attack isn’t directed at you. Keep listening to have the conversation stay in even tones and that each party is either talking about what their area could have done or is constructively helping the overall conversation.
Eventually, there will be a fundamentally sticky conversation about which party was in the best position to avoid the problem. At this point it’s going to come down to culture – if your culture is one that learns from mistakes, it will be a clear and short conversation. Depending on how strong the duck-and-cover instinct is in your shop, it can be very painful. In the end – speak up if your team is the one that should have the spotlight. Fear of accountability is often overstated. In practice, managers know that in the end they need people that will be accountable for what happened, and the experience can still be positive in the long term. Great managers actively hunt out people that are quick to learn from their mistakes and own them.
Reliability is a Mindset
Posted by: | CommentsLast week I was attending a training course on sales from a company I really respect – EntreQuest. One of the things I love about their courses and consulting is they aren’t shy about getting right to the fundamental (and often fundamentally hard) human basis for problems. One of the things they emphasize is that results are driven by process (including technology) which is in turn driven by mindset. If you don’t have the right mindset, you won’t achieve the results regardless of how much technology you throw at it. This is the basic justification for why the success rates of telemarketing (and other sales efforts that are all process, no mindset) are so low.
What was interesting to me in particular about this was how well it relates to conversations I typically have about reliability. Depending on where someone is in their experience curve they may talk about a particular technology, software development practice, or problem they’ve had. If they are really experienced, they go directly to either processes or culture. The very best tend to just talk about culture and mindset. This is bad news
In engineering the terms vary slightly, but I believe the principles are still completely valid: Results are driven by technology (Technology includes the processes, software, and hardware.), Technology is driven by mindset. When a mindset is held by a company, it’s called the culture. Your culture will exert a constant pressure on your technology like the current in a river: Either it will reinforce your goals or work against them.
You can make short term or localized improvements by focusing on just the results or technology, but to make a lasting change you need to be moving with the current.
Establish a Reliability Culture
Within your department or company (whatever scope you can influence), make reliability a fundamental aspect of who you are and how you solve problems. If you instill a mindset behind every discussion that your solutions will scale to a certain size, be continuously available, or other aspects of reliability, your technology choices will be imbued with this stance:
- Your development process will be designed to reduce or eliminate reliability risks. When your business partners ask for a change at the last minute, you won’t have to explain that all changes are risky.
- You won’t talk yourself into short-cutting testing. Instead, you’ll structure your development process to drive testing automation to reduce the cost of testing (allow you to run full tests more often) and ensure consistency.
- Your developers will naturally avoid low-reliability personal practices like being possessive about code, not commenting, incomplete or inconsistent error handling, and poor configuration management strategies.
- Your deployment environment will have appropriate hardware and software. You will be able to get proper monitoring tools and use hardware with sufficient redundancy and performance.
- Your business partners will be more receptive to conversations about schedules, knowing that under pressure they have to give on functionality instead of reliability.
As reliability becomes a core element of your culture, each individual will start to see the thousands of little decisions they make each day differently and unconsciously approach them from a perspective of reliability, as if they asked “what is the most reliable way to accomplish (whatever I’m doing now)”. At its best, it will shift things that happen as conflicts between people into corporate discussions – instead of your business partners feeling they have to convince you personally to add a new feature (viewing you as the roadblock) it will become how do we accommodate a business need within the context of our corporate goals for reliability. It is significantly easier to create a partnership in this scenario that has you understand their goals and them understand yours because you have a shared value and commitment to work within.
Reliability won’t always win out
Even in a reliability culture, there are very sound reasons to do certain things that entail risk. No one element of your culture is absolute, but it must always be respected and considered. For example, it could be that the system in question is an internal system that has a limited ROI. In this case it just isn’t appropriate to invest a great deal in reliability at the expense of ROI unless the system is unusable without it. Alternately, it is often appropriate during startup phases when the downside cost of a reliability problem is low (e.g. there are no or few existing users or no performance guarantee) or the mitigation cost is excessive (e.g. geodiverse hot sites).
Having reliability as a fundamental part of your mindset is still helpful in these situations because it ensures that a decision that impacts reliability is deliberately made and openly understood. As a company, you have to choose your battles and what risks you are going to mitigate. In some cases, it’s best to just run the risk and wait to see if it manifests before pouring energy into fixing it. Alternately, the risk may be scalability – if you are wildly successful, you’ll have to change your software to handle it. This is often called Technology Debt.
Taking on technology debt is often necessary when starting a product or venturing in new territory. The key is that the business and technology parties know that it’s a deliberate decision to take on that debt, instead of it being a quiet decision made just within the development team. That way if the risks turn out to become reality, the business doesn’t burn time arguing about how you got where you are and instead recognizes that it was a deliberate and well considered decision that now has a consequence that must be handled.
Reliability isn’t always Suitable
Not every company should have reliability as the defining element of its culture. It isn’t necessarily that these companies don’t want reliable results, it’s more that reliability isn’t their differentiator or important enough to be a core element of the culture. For example: Compare the Linksys and Cisco brands. Both can sell you a Wireless-G access point that on the mainline specifications are comparable: They support the same primary standards, offer comparable throughput and security features (for most people), and to many customers they would be indistinguishable. However, Linksys tends to produce a model, make a few essential firmware updates and move on. If the unit needs to be reset periodically or a new device shows up on the scene that causes a problem with it that’s potentially OK. Customers that pay $70 instead of $700 for a wireless access point aren’t expecting the same degree of reliability. If Linksys attempted to do all of the reliability testing that Cisco does, they wouldn’t be able to hit the price points or time to market that drives their brand. Their product must be reliable enough that customers will find it suitable for the target market, but it isn’t necessary to pursue ultimate reliability.
Take a hard look around you. What level of reliability is appropriate for each area of your business? What are the reliability goals of the company? What is the prevailing culture? If you find yourself out of sync with your company’s goals on reliability, it could be that it isn’t the company that needs to shift but rather you may need to explore other options.
Change Begins at Home
The next time you’re frustrated by the results your team is achieving, don’t leap on the technology bandwagon first. Back up and look at how you might incorporate a reliability mindset into your own work as a starting point for catalyzing broader change. Have a series of conversations in your team to ensure you establish a common understanding of what your principles are – not just with reliability but other guiding principles as well. From that it will become easier to know what technologies (software, hardware, processes) will support the results you want to achieve. Start with your team and move out through your company, the results can speak for themselves.
