Reliable Systems

It is very tempting to be one of the herd of gazelles in technology.  Every time there’s  a sense of a shift in the wind, everyone starts to run in a new direction.  For the past year I’ve been reading about how it’s all going to be laptop computers from here on out.  In fact, not even full fledged laptops, but netbooks - computers with small screens and small keyboard who’s main distinguishing characteristic is that they’re less of a computer than anything else around.

If all this sounds a little off kilter from reality, perhaps a few hard numbers would help:

Quoting Computer World, who asked “Do Business Desktop PCs have a future?”:

While desktop PCs account for the bulk of personal computers sold to enterprises, the gap in laptop sales to enterprises is closing. Of 168 million PCs sold worldwide to professional organizations in 2008, about 95 million were desktops and 73 million were laptops. That’s compared to 94.6 million desktops and 47.3 million laptops that shipped in 2006.

Now, as with any statistics there’s two ways to look at these numbers:

1. Laptops have grown tremendously in their total percentage of the market, and that growth rate has them on track to take over the world.
2. The majority of the growth in computer sales is coming in the form of laptops.

The gazelles are taking the first road.  And why not?  People love to assume the disruptive is true, it’s a lot more interesting.  Before you charge down that road, consider what seems likely.  There are a few problems with the first conclusion:

• Two data points don’t make a pattern: If you follow the trend back farther, the sales of PC desktops has held up consistently, but laptop sales go up and down.  This would seem to indicate that the most likely interpretations of the data are that either the overall market is expanding (for example by people having two systems) or that this is a momentary, periodic surge in laptop purchases.
• Past large growth rarely projects forward: Just because there was a large growth in one year (either in absolute or percentage turns) doesn’t mean it will repeat at all.  It’s just as likely that the next year pattern will be flat or even retreat.

So before you see the first twitch and assume it signals a migration of the whole herd, step back and think through the underlying facts.  Is this really the first sign of a monumental shift?  Or just another twitch of the needle?  Then look at your own situation.

Now, we have a few laptops, but we have more hard core desktops - the laptops are used for on the road presentations or working at Starbucks for fun.  Of course, we’re developers so we’re in the category of users that are always excluded from the norm.  But what’s not to love about a desktop?  For the same money they will always be faster and more capable than a laptop because they don’t have the burden of being small or extra power efficient.  Even if you buy into the idea that everything will be run through the web so computers are just glorified terminals…  Something still has to compose all of those web pages and make it all come together, and web apps can burn a surprising amount of processor and RAM locally.

In the end, I think we’re seeing a lot of folks buying second computers or getting additional laptops for other uses that complement their primary work computer experience.  Additionally, there are folks in emerging markets that need what laptops offer (self-contained, reliable power) more than performance but this reflects an increase in the overall market, not a shift in the existing market.

When I was growing up I spent a lot of time with my father doing woodworking.  One lesson you pick up pretty quick when woodworking is that you have to keep the work clean at each step.

• If you take a piece of wood and don’t sand the surface smooth it won’t take a stain evenly.
• If you let glue creep out of the joint and get on the wood, the stain won’t look right in that spot.
• A piece of dust on the surface will get magnified by each layer of varnish.

Each layer depends on what’s underneath it.  Any flaw in a lower layer will tend to get magnified and distorted by layers above it.

Whenever I get involved in enterprise architecture I get reminded of this analogy because I often run into irrational exuberance that you can add a layer to an existing system and paint over the flaws underneath.  I was involved in a few projects like that early in my career:  It was too hard to talk directly to the mainframe from the web server so you put a layer in front of that.  There was already a C++ layer doing a DCE RPC gateway, but that was also too hard to program against for large use so we added a COM interface to the DCE RPC gateway.  We made some prototypes and validated the concepts and charged in full bore, only to run into big delays and teething problems near the end of the project trying to get everything polished up and suitable for production.

The problem is that at each turn you may be making the developer’s short term life easier by giving them an interface more natural to their preferred programming environment but since it just builds on the layers underneath it will end up with all of the limitations they have - and they can show up in the most surprising places.  For example, we ran into problems where certain input would cause failures which we ultimately discovered was caused by % being used as an insertion marker in a gateway library several layers underneath what we were doing - so at best it would drop the % and the following character, but at worst you’d get back a random data element if you managed to create a valid insertion name.

Layering issues are particularly problematic because they tend to be data sensitive and highly situational depending on how the various layers interact.  This means that it’s very difficult to design a comprehensive test plan:  The system can act as if it’s nondeterministic, making it infeasible to state with certainty that the various modes of the software have been demonstrated by a test plan.  At best, you can say that it worked for the exact test inputs it was given.  When you do have a problem in production, the multiple technologies in multiple layers can make it particularly hard to debug because it requires a lot of chairs around the table to hit all of the possible players.

Are you decorating?  Or covering up the problem?

Whenever you’re part of a team proposing adding a new layer over an existing system to fix its problems or adapt it to a new situation, you should be suspicious.  Is this really the right path to make the API look right?  Or are you temporarily covering over a problem?  If it’s the latter, it’ll just show through later - and now you have two problems to deal with not one.

There are good occasions to add a new layer:

• To smooth technology upgrades: When you are shifting technologies, say from COM to .NET, you may want to create a custom layer as a new standardized interoperability adapter which will let you separate the upgrade problem into phases and handle them independently.
• To support multiple technologies: Sometimes you need to support multiple types of clients - varying either by environment (say Java and .NET) or major architecture (say Client/Server and Web sites).

And a few suspect ones:

• Mitigate architecture risk: To isolate a new subsystem architecture from the main codebase. We’ve heard this one before - you want to try out something new and iffy, like say Entity Framework.  To contain the risk, you want to introduce a layer between it and the rest of the platform so if it all goes bad you can easily swap it out.
• Impedance Mismatch: You need to interact with something, but just don’t like the way it works. Perhaps it throws around ADO.NET recordsets and you prefer to work with strongly typed objects.

If you find yourself in one of the suspect scenarios, you should seriously question whether the work you’d do to create and validate a layer is really forward progress or just yak shaving.  Before you go down the path, you should seriously estimate:

• Fixing the underlying problems: If the underlying layer(s) aren’t doing what you need, what would it take to get them changed (in the technology they’re currently in) so you could work without adding another layer? That puts the responsibility where it belongs, and keeps complexity under control. Do a full estimate of this approach.
• Make a parallel layer: If you ignore the powerful aversion to creating duplicate routes to the same data, what if you created an alternate path to the underlying information. It may be that you bypass all of the layers or just some of the layers (such as down to the stored procedures that call the database). While this creates duplication, it lets each platform work in their own optimal way and allows for deterministic testing.
• Using the existing layer as it is: It’s easy to overstate the impact of reusing a known system with issues. There’s a natural tendency to not realize that you’re comparing a well understood but flawed system with an unknown solution with unknown problems. Trading known problems for unknown problems makes everyone happy at the start of a project, but creates significant project risk downstream.

Put down the shovel and back away

When you create a new layer on top of existing layers you are often digging your project into trouble, both now and downstream.  In addition to problems with each layer creating a leaky abstraction, deploying and supporting these highly layered systems is extraordinarily challenging.  It becomes prohibitively expensive to make changes in lower layers because of the high chance of unexpected side effects showing up as defects in dependent applications.  More often than not, each layer has to be held static with any changes accommodated by creating new queries or items at each layer to be served in parallel with the older methods.

Before you go ahead, be sure you look at the total lifecycle cost of that decision, including support and maintenance.   Have a good or bad experience with putting up some software wallpaper?  Let us know in the comments!

Everyone likes to be on the winning team.  We love to root for our favorite sports team, we like the car we own and the brand behind it.  So it’s no surprise that when Apple ran their I’m a Mac ads that Windows fans were in an uproar.  Now with the Laptop Hunter series the shoe’s on the other foot.   Microsoft is making a big show that Apple computers are overly expensive just for the Apple brand.  Apple fans claim that  to match a Mac, a PC has to be equipped with tons of antivirus software, a full time tech support guy, and a Witchdoctor on standby to keep it working.

Seriously people

First, Apple makes some of the finest hardware you can possibly buy.  If you compare it nose to nose with hardware that’s actually built to the same standards then it really doesn’t represent a significant price premium.  Compare a Macintosh Pro with an equivalent Dell workstation - the cost is within 5%.  It’s amazing that Apple can afford the extra engineering of an OS with that little premium.  

Second, Vista works great.   It’s running on many more systems than Mac OS is, and with volume comes a range of new problems.  The total amount of money I’ve spent on desktop antivirus software in 10 years of administering PCs?  $0.  The total number of virus problems I’ve had? 0.  My parents managed to get into trouble with one virus and Windows XP - but installing the (free) Microsoft Defender cleared it right up never to return.

As with nearly all marketing, this is a battle of perception:  Apple has done a great job of defending their brand at every turn.  This is part of their corporate ethos.  Along with a few other tenants it ensures they are a much loved but niche player:

• Only do something you can do uniquely well.
• Don’t extend into markets that might ask you to compromise your values.
• Cultivate the mystique:  Don’t show what’s behind the curtains.

Microsoft on the other hand has tenants that ensure they’ll be a volume player, but an unloved one:

• Play to win the most market share in any market you can.
• Build your ecosystem by making it easy for others to add value to it.
• Cultivate the engineers:  Provide overwhelming amounts of documentation and approaches.

The fact is, many people don’t need a top end piece of hardware like a Mac.  On the other hand, many people want a computer that’s just a tool, not a piece of art.  To them, the nearly infinite diversity and low cost of entry are essential.

I’m a People person.  I’m Good with People!

The computing needs of the average corporation and the average individual are very far apart.  To companies, computers are tools - like the desk, phone, and copier.  Very important, very powerful - tools.  They aren’t there to make you feel great or enable you to create a cool video of your vacation in France.  My partner really summed it up one day when he commented that the Mac was a really personal computer - it worked hard to create a personal connection.  

Corporations on the other hand want slow paced evolution, massive support for legacy applications and hardware (these guys are still running dot matrix printers off parallel ports) and to control costs.  They also philosophically want to have all the keys to the computers - just like they do for the buildings and offices they own.  PC’s are just end points on the large mesh that is the corporate IT network.  It’s very impersonal.

Microsoft makes a great deal of money providing businesses with the tools they need to have the computers work for them, and Apple makes a great deal of money creating computers that people love.  Either of these goals would be compromised by trying to do both.

Vista Goggles

Folks that have been in the Windows ecosystem a long time probably recognize that you could take the first year of press about Vista and substitute “Windows 2000″ and find the same article written 8 years earlier.  Vista is a surprisingly large and tricky step forwards on a number of fronts, whereas Windows XP was a visual redress of Windows 2000.  

Almost like an SAT test:

Windows 7 is to Windows Vista as

Windows XP is to Windows 2000.  

Like Windows XP, the story on Windows 7 is making virtually no architecture changes and instead just tuning for the long haul.  That’s a great thing, because there’s a lot that works very well with Vista, and now it’ll work even better with Windows 7.

The humorous thing is to read now about how people are thinking about moving to Vista once 7 ships because, well, they don’t want to move to an OS that was just released.  It’s as if Vista has been aging like a fine cheese on the shelf so the very same binary code that once was toxic is now just what the doctor ordered.  To a slight degree this is true:  Vista SP1 did address some issues that affected some people, and more importantly hardware now is dramatically faster than it was two years ago (as it will be two years from now…) so what once was aggressive is now commonplace.  The same was true of Windows 2000 when it shipped.  Requiring 64MB of RAM?  That’s just crazy talk!  Only certain video cards worked reasonably with it, and video drivers to a while to stabilize.  That sounds very familiar…

In the end, it really comes back to Perception.  Probably the biggest mistake Microsoft did was not push the OEM’s that make the computers to build machines that could responsibly run the new operating system, and be clear that meant hardware 3D video cards and plenty of memory.  And oh yeah, stop putting aftermarket firewalls, antivirus, Google Desktop, and all kinds of other things on them that are ill optimized.  At my last company we got in the habit of routinely wiping each new Dell that came in and reinstalling the OS from the Dell restore CD - because that got rid of all the noise.  It was surprising how much better that worked.  Is that Microsoft’s fault?  Not directly, but they certainly could have found a way to encourage the ecosystem to forgo some profit for usability.  But that’s just not in their corporate DNA.

With any luck, the big story for Windows 7 will be that Microsoft pushes back against their channel, even being willing to risk it by leaving Windows XP out there for folks that don’t want to play by the Windows 7 rules.  It’s hard to put up barriers when you’re a legal monopoly, so find ways to use incentives to do it right instead of punishment for doing it wrong.  And keep up the ads, because perception does matter in the long run.

Who knows, it may push Apple to get better too.  Just once when my iPod updates itself to enhance stability and performance I’d love to know what exactly was unstable or slow…

Since leaving my last company and getting into the wild as a consultant, I’ve been amazed by the divergence between the conventional wisdom prevailing on the Internet and what I see actually happening on the ground with clients.

The prevailing wisdom appears to be:

• Google is the world’s best technology company.  Anything they solve they are the best at, and if you aren’t doing it their way you’re a dinosaur.
• Client / Server is dead.  All new applications will be web applications, most likely delivered as a service.

We visit a lot of companies, both prospects and clients.  Here’s what we’ve actually seen over the past two years:

• Outside of search, Google doesn’t have their act together.  Pretty much everything else is an academic experiment.  There’s nothing wrong with experiments, but there’s a big distance from there to running your business.
• Business are run on Client / Server.  They’re still creating new apps this way - they may use newer technologies like WPF to do it, but if it is core to making the business work, it’s usually not a web app.

With a name like Google, it’s got to be good!

Our own internal experience mirrors this.  We use Exchange and Outlook for email so I can’t say a lot directly about Gmail, however it’s clear from Google’s reactions to recent outages that their perspective doesn’t fit the enterprise because it doesn’t take into account the premium companies place on predictability and communication.  Predictability meaning that things are consistent - you get a consistent experience so your users can get their jobs done.  Businesses are change averse for good reason:  Users will adapt even to crazy problems and discover the patterns that work to get their jobs done.  When the patterns keep changing, they get frustrated.  On the communication front, businesses prize feedback on knowing the true scope of a problem and how long it may take to get it resolved.  

SalesForce learned this a few years ago and in response introduced Trust.Salesforce.Com, which went a long way to getting companies what they wanted to be comfortable with an outsourced critical solution.   If you operate a SaaS, you would do well to model after this.   Now, it isn’t necessarily bad that Google doesn’t do anything like this for Gmail - it just means it isn’t a solution for a large set of businesses.  

The thing we use the most internally from Google is Google Analytics.  It’s very pretty and easy to use, however we’ve noticed a lot of “what’s broken today?” experiences with it, enough that we can’t recommend it to anyone.  Two of the sites we monitor appear to be chronically under counted by Google Analytics, and we can’t figure out why.  And like most things Google - you’re on self support.  Now, there are paid options however unlike email we haven’t been able to find a strong paid competitor that is actively competing with Google.  It feels like most have left the field of battle, or are exorbitantly expensive (and aimed at large enterprises).

After much early on talk about how Google Docs was going to make Office obsolete, it simply hasn’t come to pass - and Microsoft continues to sell a lot of copies of Office.  It turns out that making a great word processor and spreadsheet is a very hard problem to do through the web.  Now, you might take the perspective that Microsoft’s announcement that they are going to offer lightweight web versions of Office 14 applications as being an admission that the old model is bankrupt, but it really points to an increase in reach:  Reaching many users that wouldn’t have been purchasing the product before.  Casual home users that wouldn’t purchase a “real” copy of office may find what they want in Google Docs, and would also be happy with a lightweight feature set of Office.

In Google’s defense, the products are worth what you’re paying for them:  Free.  But, you have to ask yourself:  If it wasn’t for the Google brand, would you give them the time of day?  For search, absolutely.  Finding an address and getting a street view?  Bring it on.  But don’t feel bad for depending on traditional software next time you want to buy a copy of Office, Photoshop, or use Outlook for your email.

It’s all SaaS These Days

Make no mistake, web applications, SaaS and Cloud Computing are all here to stay.  However, that doesn’t mean that there isn’t a lot going on in Client \ Server as well.  The key question to consider is how many applications you would have made, but done in another technology are being built as a web app instead?  There certainly are some, but for the most part web applications are creating entirely new spaces and solving problems that weren’t being solved before instead of replacing entrenched problems.  

Take document management:  On the surface this feels like a problem that should go entirely web and not look back because the web is very good when the readers to editors ratio is very high.  That said, the big document management companies still have very robust, integrated traditional offerings as well as their web portals.  You do have people using web document management solutions (like SharePoint) that never had document management before, which is a case of expanding the size of the market not replacing an alternative option.

When you are running your business, you have a set of requirements that are often best suited by a traditional client application:

• Users need advanced capabilities:  Once you stray out of the basics, it is invariably harder to provide features through web technologies than traditional technologies.  Tools are improving and making it better, but the cost per feature is lower in a modern client development environment than through a browser.  This is particularly true since you can put a browser in your app to enable things that it does really well - like show HTML content - you just let it do and keep the tricky stuff - like that big set of coordinated data entry fields - in your traditional app.
• Users are doing a lot:  One thing that is easy to lose perspective on is that when you run a client/server application every computer is bringing power to the party.  You often don’t need that strong a central server because the real action is happening out on the clients, and every new client that logs in is bringing their own muscle with them.  In the web, it’s all on the server. Not only that, but you’re rendering things in a less efficient way due to the stateless nature and limited protocols available for data exchange.
• Users need access to diverse data:  With web technologies it’s straightforward to manage information once you’ve brought it into the cloud, but it’s tricky to provide end users with fast casual access to a range of data in a range of formats.  In many businesses data is coming in pieces from many sources and being assembled to produce a coherent output.  This isn’t easy to do in any environment, and it’s one thing that the modern PC operating system, particularly Windows, has gotten very good at.  You can double click a file and almost always get it to open into a good viewer.  You can preview files, drag and drop from one program into another or a file into a program.  All of this is intuitive and fast for users that aren’t fitting into a pre-packaged user scenario.  

There are counter balancing effects:

• Transient user community:  The more far flung your people are, the more work it is to keep them up to date.  The more transient that user community is, the higher a barrier installation is.  This is the leading reason why you want to make something a web app:  It just isn’t worth the deployment effort to do it any other way.
• Diverse user community:  If you want to service Linux, Windows, and the Mac then web technologies are the lowest common denominator, it’s just the way it is.

What we’re seeing in the field fits into this:  People are creating a lot of new, lightweight web apps to solve point problems they probably wouldn’t have solved through technology before.  But they’re also still heavily investing in traditional applications.  

As a development team, it’s easy to get caught up thinking that Effort is the same as Value - just because something was a lot of work it must have a commensurate value.  The fact is, that there’s just no evidence that effort and value are correlated.  On the web, if you want to create a great looking and functioning generalized tool you’re signing up for a lot of effort.  And the value may be there - it could be that you’re going to reach a whole set of new users that otherwise wouldn’t use an application at all.  On the other hand, it could be that users perceive no extra value for it being in the web so all the extra work you put into creating the graphics, testing in five browsers, establishing identity, and the dozen other things you wouldn’t have needed to worry about if you were just running on the desktop netted you nothing.

So if you’re a business wondering how to approach that next application you need, don’t be afraid to get one that isn’t all wrapped up in Web 2.0 goodness.  In the end, it’s all about making the solution work for you and your users.

Many if not most software project leaders came up through the development ranks.  It’s generally thought of as a distinct advantage - you know the technologies you’re using, you can form your own well reasoned opinions about how hard something is, what is possible, and how long it should take.  For a long time, I felt that the best way to get results from development teams was to use my experience and knowledge to be very understanding of the challenges they faced and give them whatever time they asked for.  However, in the last few years I’ve run into several situations where I just couldn’t get them the extra time or relief from the most problematic requirements.  I predicted doom to the projects in question but instead I observed some of the best outcomes I’d ever experienced.

While the projects were successful, it bothered me that the secret sauce seemed to be a rigid adherence to schedule and delivery more than any other consideration.  This was exactly the reverse of how I wanted projects to succeed:  I wanted them to succeed because I was treating the developers how they always wanted to be, not like a stereotype from Office Space.  How could it be that better results came from ignorance of the technical details involved?

Developers Will Use All Available Time

Upon reflection, the first thing that struck me was how much an immobile deadline focused discussions and decision making. If you give a team more time, they will expand their process to consume it.  Time will get consumed by:

• Elaborate Decision Making: When you have little time, you make a choice and go with it until it appears it just can’t work.  When you have a lot of time, you sit back and look for the very best option.  That then requires defining what the best is - is it fastest, or smallest, or most scalable, or whatever.
• Development Approach: Under pressure you’ll tend to go with the proven guaranteed approach.  If you have the luxury of time you’re more likely to engage in yak shaving like investigating a new tool or approach, or writing several prototypes first before you develop the real solution.  You might even just throw caution to the wind by skipping a formal design figuring you’ll have the time to just code and test your way to a solution.

The more time a development team has, the harder it is to argue against spending it on up front luxuries.  It also can be harder to argue for long term best practices because the team has the time now to develop a solution any way they want.

Unknowns Create Boomerang Estimates

Even very experienced developers are generally terrible at estimating the duration of developing a solution.  This has been demonstrated over and over by many other parties.  The key behavior that we’ve observed is the phenomenon that from when you approach a specific development problem (like displaying a graph on a web page) until you know exactly how you’re going to solve it (and have a reason for confidence in that approach) you will tend to estimate high because in effect the only reasonable estimate is infinity.

Put another way, as long as you don’t know how you will solve a problem you don’t know for sure that it is solvable which means it will take an infinite amount of time to solve it.  Fortunately, developers are almost universally optimists so they believe they can solve anything eventually - so they’ll pull out a standard answer like three weeks or months or whatever feels like a big chunk of time to figure out the problem but not so big that it kills the project.   The reality is that until you know how you’re going to solve it, it feels like it could take forever.

Once a solution has presented itself  the development team will often find that all it will take is some cleanup and polish to be done- a very small amount of time.  What will push the team to find the answer?  We’re back to the problem of elaborate decision making when you have the luxury of time.  Finding solutions tends to not be a linear problem that will be solved with incremental development energy.  Instead, it tends to be solved by getting people together and brainstorming possible solutions until you find a few candidates and can work out what it’ll take to prove them out.  Under pressure, people tend to focus their creative energy and be more willing to compromise.   That flexibility will tend to get rid of pet requirements and developer gold-plating and focus on the most critical aspects of the problem.

What’s the alternate approach?

The key is to not let your knowledge and experience as a developer lead you to buy into the stories the team creates around what’s reasonable to get done and how long it will take.  Instead, you have to stick with the project’s goals first then the facts of the project.  The project’s goals form the objective reality of what has to be accomplished for the project to survive:  Deliver this functionality by that date, keep these people informed, solve these problems without causing those problems.

When the team runs into a wall and needs more time, instead of buying into the story of needing a lot of time, set a specific and tight goal that keeps a solid amount of time pressure on the team to solve the issue and prevent the problems above from showing up.  Ideally, find a way to give out one or two day chunks to answer incremental questions if necessary to emphasize that time is precious and has to be invested carefully.  This is where you can leverage your experience in a way that a non-developer can’t:  The team knows they can’t snow you with tech details, and you can define a specific, measurable result that can be achieved in a short period of time that they can’t argue with.  Despite this, you are bound to have to assert a few times that the time limit is the limit - solve the problem in that time.  It’s very hard because you’ve been on the other side of that conversation and it can feel like you’re the Pointy Haired Boss, but it’s fundamentally your job on the project.

What will nearly always happen is the team will surprise itself - a solution will be presented within the team that they can live with and can be done in the time they have.  It may be incomplete or have some risky shortcomings, and you’ll want to ask how long it’d take to address those.  You probably shouldn’t address them in the first round, but the team will feel better that you’ve considered through things and will buy into the outcome more if you ask.  You’ll also want to make a record of it so that the team can in the future recognize what was a predicted shortcoming vs. an accidental defect.

Do you want it solved right?

This is a question that often gets voiced within a team as a rebuttal to external time pressures and is very dangerous.  The challenge is that most non-technical people don’t get the number of ways that a problem can be solved: instead, each problem appears to have a single solution.  Take away your technical knowledge and imagine you’re the paying customer:  What’s the alternative - were you going to solve it wrong? If that’s the case, what else have you done that’s garbage?  If you took your car to a repair person and they said it’d be $500 to fix it, then when you came back they said well, if you want it fixed right it’ll actually be $1200, wouldn’t you wonder what the hell the $500 fix was?

Usually this statement is uttered in desperation when a team believes they just need more time to figure out a problem.  Nobody wants a problem solved wrong.  Skip the hyperbole and get down to action:  break down the problem into small chunks of time that can be invested for a specific measurable result, and make sure the team gets that overage time is the most precious commodity.

Side Note: This is an advantage of SCRUM in practice.  If you’re following an Agile Development practice, particularly SCRUM, this fits right in:  Focus on making each sprint deliver the user stories it was supposed to even if you have to leave some special cases for a later sprint.  The daily stand up meetings are a great place for the different team members to apply team pressure against over engineering and doomsday estimates.

Cleaning Up and Closing Out

At some point you need to close out your release and ship it. For each of the areas where you’ve had to make compromises and taken shortcuts you have to choose to either:

• Ship as Final: Decide the implementation is close enough to the intent of the end-user functional requirements that it can be the final implementation (at least until new information contradicts this decision)
• Ship as Temporary: Decide that something is better than nothing and ship the feature with limitations.
• Cut the Feature: Hold back the feature until it can be reconsidered or reimplemented.

You’re nearly always better off shipping the feature, often as a final feature pending more information because it’s very hard to gauge the true impact of each limitation.  This is particularly true of user-facing features and environments where it’s possible to evolve the software rapidly.  Inevitably once it’s in the hands of your users you’ll discover aspects of it that you didn’t think of that will require rework and you may discover that the killer feature you were sure would be the hit of the release is hardly used.  In either of these cases if you’ve invested a great deal of time in making it foolproof the team will tend to resist changing it.  It’s a natural product of the presumed relationship between effort and value.  If necessary, you might put in some temporary safeties to detect and catch the limitations you’re worried about.

The major exceptions to this approach are areas that are too dangerous to deploy if less than fully trustworthy.  For example, if your team is developing a data storage system, software deployment system, or other critical infrastructure your choices likely resolve down to making it as right as possible or holding the feature until it can be reworked.

If it turns out that the solutions that are viable within the schedule have significant limitations, you should make sure these caveats are known to the business - provided you can express them in business terms.  For example, knowing that an algorithm won’t work if your userbase doubles is probably not a significant caveat, unless you know the business plans to double in a relatively short period of time.  Every system has limits, and every software change has risks.  Business representatives don’t like to hear the same items covering the same ground repeated every time you discuss software, and it tends to make them not hear the new and important information as well as sound like you’re attempting to transfer accountability from your team to them.