When you want to create a product that just works, you need to get all of the details right – from packaging through to maintenance and upkeep.  It’s not one thing that’s important, it’s all the things.  We are often engaged by senior management within a client when things aren’t working, and there’s conflicting opinions on why.  Usually along the path technology is being blamed: Not enough, not the latest thing, not someone’s favorite thing, not working.  As we dig into the situation, rarely is the technology the dominant factor:  More often, it’s how the technology is being integrated with the people and processes that all have to work together.

One of the first things we have to do in these engagements is to establish the real facts on the ground:  What exactly are the problems in the system, who’s doing what with it, how many times.  It comes down to establishing metrics to make sure time and attention are paid to the parts that make the biggest difference in the outcome.  Armed with these facts in a form the business can consume it’s possible to create plans of action that deliver virtually regardless of budget.

So let’s make this easier

The biggest trick is then getting the facts you need on an ongoing basis, easily, and in a form that the business can consume.  For over a decade we’ve been building instrumentation right into the systems we’ve worked on.  We’ve created a variety of toolkits to make this easier over the years, refining them as technology and our experience has changed.

About 18 months ago we decided it was time to really invest down this path.  We believe in routinely capturing key computer metrics along with whatever logging the application can do on its own.  We won’t do a project without using a great logging system that includes a strategy for managing runtime exceptions.   Now that we’re collecting all this data, we need to have a way of managing the raw data and turning it into valuable business data.

The challenge is that businesses don’t get up in the morning and say “what our customers want us to do is have great internal tools”, so you’re nearly always doing this on the cheap:  Borrowing time from development projects internally to cobble together various free or cheap solutions.  Frankly, we got tired of having to create new solutions with each client out of the margins of each project.  So, we pooled our best thinking from all of the work we’ve done (including a previous product that we did license to our clients over the past decade called CLAS) and started creating Gibraltar.

Rock Solid from Initial Release

With Gibraltar we wanted much more than a log system.  Of course, it had to be a log system too – and a really easy to use one that could work with each of our client applications.  More than that, it had to:

• Automatically capture all of the performance metrics we wanted.
• Integrate with existing logging available on the platform, including whatever a client might already be doing (like custom in-house options)
• Be absolutely, positively, for sure safe to run in production no matter what.   That means it can’t ever use too much disk space or disk throughput or block the application.
• Not use more than 5% of the performance of the app
• Include all of the tools necessary to get data from where it was collected to the people that could get value out of it
• Include the ability to look at the detailed session data up to high level analysis:  What’s the error rate?  What’s it correlate to?  Are we doing better or worse in this version?

From this initial sketch into everything we wanted, we’ve spent 18 months including four beta periods (from 2-4 months each) to refine the vision with real customers and real scenarios.  It was essential to us that this not be just a tool for techies but be ready for use by people with a wide range of skills.  It had to be pretty and just do what you wanted, when you wanted it to.

We’ve added a lot of capabilities along the way:  It can generate print-ready reports about application reliability that can communicate with senior management, you can define all kinds of custom metrics to easily track how your application is used and by whom.  We ran a number of betas to be sure that we had hit every goal we have above.  We’re happy to report that Gibraltar is in use within large deployments of custom applications, commercial applications, and small deployments right down to our corporate web site.

This tool isn’t for everyone – Our clients are nearly all Windows shops, and if they do any custom development it’s almost invariably in .NET, so that’s what we’ve targeted.   But, if you’re interested in easily getting real data on not just infrastructure (how well the application is running) but whether or not it just works, have we got an easy path for you.  You can see a quick demo video of how it works technically at Gibraltar Software.

You also don’t have to take my word for it at all, you can hear what one of our beta users did with it, which is really a more compelling story than what we might say.

I think you’ll find that our work sweating a lot of little details, from the exact design of the API and making sure the documentation was complete to rewriting our own licensing system to be very IT Admin friendly.  If we didn’t get a detail right, we want to know.  And the great news is that we’ve just begun:  We’re obsessed with the little things, and you can bet we’ll keep listening and watching to make it better.  Of course, this is made a lot easier because we’re using Gibraltar to monitor itself, and a select group of our users is sending that information back to us so we can make sure it just works in the field for real people.

It’s easy to start your journey

If you do development for Microsoft .NET, I’d encourage you to go over and download our commercial release of Gibraltar.  You’ll get great documentation, a free agent you can use like a flight recorder “black box” in every application you create, and a trial for a tool that will make you seem wise beyond your years.  And if you pay us the ultimate honor and purchase a permanent license, I can assure you that you won’t find anyone more committed to your satisfaction than we are.

• If you take a piece of wood and don’t sand the surface smooth it won’t take a stain evenly.
• If you let glue creep out of the joint and get on the wood, the stain won’t look right in that spot.
• A piece of dust on the surface will get magnified by each layer of varnish.

Each layer depends on what’s underneath it.  Any flaw in a lower layer will tend to get magnified and distorted by layers above it.

Whenever I get involved in enterprise architecture I get reminded of this analogy because I often run into irrational exuberance that you can add a layer to an existing system and paint over the flaws underneath.  I was involved in a few projects like that early in my career:  It was too hard to talk directly to the mainframe from the web server so you put a layer in front of that.  There was already a C++ layer doing a DCE RPC gateway, but that was also too hard to program against for large use so we added a COM interface to the DCE RPC gateway.  We made some prototypes and validated the concepts and charged in full bore, only to run into big delays and teething problems near the end of the project trying to get everything polished up and suitable for production.

The problem is that at each turn you may be making the developer’s short term life easier by giving them an interface more natural to their preferred programming environment but since it just builds on the layers underneath it will end up with all of the limitations they have – and they can show up in the most surprising places.  For example, we ran into problems where certain input would cause failures which we ultimately discovered was caused by % being used as an insertion marker in a gateway library several layers underneath what we were doing – so at best it would drop the % and the following character, but at worst you’d get back a random data element if you managed to create a valid insertion name.

Layering issues are particularly problematic because they tend to be data sensitive and highly situational depending on how the various layers interact.  This means that it’s very difficult to design a comprehensive test plan:  The system can act as if it’s nondeterministic, making it infeasible to state with certainty that the various modes of the software have been demonstrated by a test plan.  At best, you can say that it worked for the exact test inputs it was given.  When you do have a problem in production, the multiple technologies in multiple layers can make it particularly hard to debug because it requires a lot of chairs around the table to hit all of the possible players.

Are you decorating?  Or covering up the problem?

Whenever you’re part of a team proposing adding a new layer over an existing system to fix its problems or adapt it to a new situation, you should be suspicious.  Is this really the right path to make the API look right?  Or are you temporarily covering over a problem?  If it’s the latter, it’ll just show through later – and now you have two problems to deal with not one.

There are good occasions to add a new layer:

• To smooth technology upgrades: When you are shifting technologies, say from COM to .NET, you may want to create a custom layer as a new standardized interoperability adapter which will let you separate the upgrade problem into phases and handle them independently.
• To support multiple technologies: Sometimes you need to support multiple types of clients – varying either by environment (say Java and .NET) or major architecture (say Client/Server and Web sites).

And a few suspect ones:

• Mitigate architecture risk: To isolate a new subsystem architecture from the main codebase. We’ve heard this one before – you want to try out something new and iffy, like say Entity Framework.  To contain the risk, you want to introduce a layer between it and the rest of the platform so if it all goes bad you can easily swap it out.
• Impedance Mismatch: You need to interact with something, but just don’t like the way it works. Perhaps it throws around ADO.NET recordsets and you prefer to work with strongly typed objects.

If you find yourself in one of the suspect scenarios, you should seriously question whether the work you’d do to create and validate a layer is really forward progress or just yak shaving.  Before you go down the path, you should seriously estimate:

• Fixing the underlying problems: If the underlying layer(s) aren’t doing what you need, what would it take to get them changed (in the technology they’re currently in) so you could work without adding another layer? That puts the responsibility where it belongs, and keeps complexity under control. Do a full estimate of this approach.
• Make a parallel layer: If you ignore the powerful aversion to creating duplicate routes to the same data, what if you created an alternate path to the underlying information. It may be that you bypass all of the layers or just some of the layers (such as down to the stored procedures that call the database). While this creates duplication, it lets each platform work in their own optimal way and allows for deterministic testing.
• Using the existing layer as it is: It’s easy to overstate the impact of reusing a known system with issues. There’s a natural tendency to not realize that you’re comparing a well understood but flawed system with an unknown solution with unknown problems. Trading known problems for unknown problems makes everyone happy at the start of a project, but creates significant project risk downstream.

Put down the shovel and back away

When you create a new layer on top of existing layers you are often digging your project into trouble, both now and downstream.  In addition to problems with each layer creating a leaky abstraction, deploying and supporting these highly layered systems is extraordinarily challenging.  It becomes prohibitively expensive to make changes in lower layers because of the high chance of unexpected side effects showing up as defects in dependent applications.  More often than not, each layer has to be held static with any changes accommodated by creating new queries or items at each layer to be served in parallel with the older methods.

Before you go ahead, be sure you look at the total lifecycle cost of that decision, including support and maintenance.   Have a good or bad experience with putting up some software wallpaper?  Let us know in the comments!

I know it’s very uncool to admit – IE is only available on Windows, isn’t the most standards compliant, it has a bad reputation for security and there aren’t 1000 cool addins for it. And oh yeah, it’s from Microsoft – aren’t they evil?

But here’s the thing: From day one, it was the practical choice. Not because it was distributed with Windows, but because it worked. Worked as in it was easy to keep upgraded (thank Windows Update for that) it could do anything we needed it to (and yes, that used ActiveX) and it supported integrated authentication – so users could just point their web browser at our company sites and not get any login prompts, just get access to all the resources they needed.

Not only that, but IE was really forgiving. The thing that many people miss is that at the end of the day it isn’t about being standards compliant per se, it’s about the web browser just working. Put another way, while you want to develop using something really strict, when it comes time to hand it over to your users do what I meant, not what I said is best. Frankly, I was amazed at how tolerant IE was of HTML errors. In the end, do you think that users would say Thank you for not doing what I meant because it would have meant breaking the rule that you can’t wrap a div with an href? Most users I know would just rather it displayed the page.

Is the pace of IE development slower than Firefox or other browsers? Well, yes – but again, so what: businesses really don’t like change, change costs money. Change means retesting applications, and for what? It has to give them something to justify that cost. It isn’t like IE won’t remain a viable tool for browsing the Internet for some time. Remember, these are the same companies that are still (justifiably) writing applications in VB6, an environment that its creator has been trying hard to kill for 8 years.

So it makes a lot of sense that when it came to IE 8, Microsoft focused on what IE users really wanted. It isn’t standards compliance for its own sake, it was:

1. Make it as fast as possible for how people browse the web today.
2. Fix as many of the quirks in standards interpretation as feasible that make it easy to develop good sites for IE.
3. Do what you can to make it hard to have a rash of new security problems.
4. Don’t break any site that works today.

If you want a good feel for just how hard this problem is, Joel Spolsky did a great (but long) writeup.

So I’m writing this in IE 8 – and it is just what Microsoft promised, it hits all of the points above and is certainly a solid improvement over IE 7. Microsoft made a great call in focusing not on Javascript performance but looking at the totality of what affects the apparent browsing performance for users and addressing that. It feels nice and fast, and I haven’t encountered anything that’s broken.

But it’s no longer my default browser. While I’ve been trying to love it, I just can’t get there. I switched over to Chrome when it went into release and haven’t looked back. On the surface of it, this is a bit crazy:

• Many sites don’t work quite right with Chrome. I’ve gotten halfway through a shopping cart with Chrome and had the buttons not work to go next. Things don’t align right… Virtually every web app we use at eSymmetrix had to be patched to work with Chrome. Even WordPress seems to work better with IE than Chrome in some ways.
• Google is getting evil. They’re doing things Microsoft never could get away with. For example, every Google thing installs its own Google Updater service. It doesn’t ask if it can, I can’t find any way to get rid of them through uninstall.. they’re just there. I have no idea if they’ve ever updated Chrome because they’ve never asked. You can be sure that if Microsoft this they’d be in court faster than you can say Slashdot.
• It’s not really stable. About every other day I get the admittedly cute Aw Snap! page where Chrome just isn’t quite happy.

But it doesn’t matter, I still dramatically prefer it to the other three browsers on my computer. Why?

• It’s fast.
• I love the automatic 9 recent site dashboard with preview. I love how it handles browsing history.
• The document inspector is great for web development, so I use it all the time when developing web pages and style sheets.
• It’s the future.

It’s the last point that’s really got my interest. It’s the future. Why? Because while it really doesn’t matter right now, the approach Chrome uses for Javascript is going to rewrite the face of the web.

That’s just Crazy Talk

Right now Chrome has around 1.5% of the total browser market share. That’s nothing – about what Opera and Netscape have put together, much less than Safari at 8%. On the other hand, IE 6 and IE 7 each have the market share of all the rest put together (more or less). So statistically, Chrome is completely irrelevant – and it’s had a lot of time to get to that point in market share. IE 8 cot to that point of market share pretty much as soon as it was released.

So what justifies it being the future? Because the V8 Javascript compiler is a game changer. Not on its own, it’s a piece of the puzzle. Four things have to come together:

1. Universal high bandwidth: Enough to move say 3mb in 5 seconds reliably.
2. A compiler that converts Javascript into machine code within 50% of the performance of compiled Java or .NET.
3. A standard Javascript library that can function as a GUI Toolkit on par with .NET WinForms or Cocoa.
4. A visual Integrated Development Environment (IDE) that can bring it all together with a good debugger for browser and server.

We already have the first one. It takes about 6 mbps throughput – cable modem speed – to move 3MB of data in 5 seconds. Chrome brings in the second one (and actually bests that by some margin). Now we need the third and fourth.

Up until now the third item has been held back by the general problem that Javascript was slow enough that the overhead of having a common API (full of things you didn’t need for any single situation) was infeasible. Writing something broadly reusable is about an order of magnitude harder than writing it to handle a specific scenario, and ends up doing a number of things you don’t need to do because someone might need it. That overhead isn’t acceptable with the historical performance of Javascript, but if it’s compiled down to native code then it becomes a small, inconsequential optimization.

So now that there’s an environment that can run it, we need a general UI toolkit and the IDE to develop with so we’re putting our time and attention into creating features for our users, not how to make a menu that dynamically expands and highlights. The IDE needs to provide an end user experience like developing for WinForms or WPF in Visual Studio – clean, easy, visual, without surprises.

Microsoft was right – today it isn’t about Javascript performance. But if we’re lucky, tomorrow will be – and we’ll be able to develop much better, stronger applications for the web without resorting to Flash, Silverlight, or very expensive development and testing cycles.

So with my natural preference for hard core engineering I’d like to report that the best software comes from a group of driven software engineers. Technically, that may be true – a big group of engineers can make a very technically sophisticated product. But, really great products? Well, that requires a lot more than just technical excellence.

I think this is the backstory behind Vista’s successes and failures. We’ve been using Vista is our corporate OS since January of 2008, not long after it was widely available. It’s worked very well for us – even better since SP1. But again, we’re engineers: half of our systems are 64 bit, and we use high end hardware so we were very good candidates.

A Whole Lotta Polish

Last weekend I installed Windows 7. Now, even though I generally love new toys I haven’t been chomping at the bit to try out Windows 7 because Vista is working great for me, and we’ve had a lot of deadlines I didn’t want to risk. But, with the release of build 7100 last week, I couldn’t resist.

What’s the big difference between Windows 7 and Windows Vista? Polish. A whole lotta non-engineering polish. I was using the media center capabilities last night and noticing all of the little things that are completely irrelevant from an engineering / functional standpoint. These same things make all the difference in how you perceive the quality of the product and, more importantly the quality of the experience in using the product.

Is Build 7100 without issues? No – there are some optimization issues that I’ve run into, but they’re likely known already within Microsoft and they have months to refine them. The big picture is that the risky, time consuming design details are all there. I haven’t even turned off UAC yet, and I couldn’t live with that under Vista for more than two hours.

Now, it may be that if you’re creating the next version of SQL Server that this fundamentally human element of intuitive adjustment and polish isn’t as necessary. SQL Server could be all about hard core specifications, tests, and optimization. That’s reasonable when the human to product interface is either through a standard you can’t affect (e.g. T-SQL) or is confined to highly technical specialists.

Goes to Eleven

When you’re creating an application, you aren’t going to find the polish by reading a functional specification. You also aren’t going to get it just by using any particular development methodology – Agile, Waterfall, whatever. What you have to be willing to do is go beyond the written functional and system specification and look carefully at each aspect of the human – computer interface in your product.

This dedication requires a few things:

1. Access to a User Experience (UX) / Human Computer Interface (HCI) specialist. These folks are experts not for facts and figures or things you can read in a book but their experience and practiced eye that lets them pick out the key details that make all the difference.
2. Dedication to making it better: At each turn, and in very difficult moments, you’re going to have to repeatedly look at what you have and what you’ve done and say OK, how do we make this better. Take the case that we can leap beyond this, what would that look like?

Done right, this experience can be tortuous to engineers because it’s about iterating through hard to quantify, experimentally determined states without objective metrics to guide your process. You will see the results of your work – but as the sound of distant thunder as your users either rave more and more for what you’ve done or just accept meekly what you give them. Engineers are used to tweaking a knob and seeing the needle move in a quick, quantifiable way.

If you want to get a sense of what happens when people think deeply about how to create software that interacts well with people, read the Microsoft document on how to write an error dialog for Vista. This is 28 pages on how to do a good error message and why. Warnings? another 12 pages. Even if you’re a hard core engineer, some of the Vista User Experience Guidelines is a great read to understand why it takes many iterations and at least equal measure of instinct and intellect.

Fighting the Good Fight

The challenge with pushing for breakthroughs in the user experience with your product is that it doesn’t fit well into traditional engineering problem solving techniques. That may be why some of the most successful organizations at it have a strong command & control personality (like Apple) that emphasizing an individual making an intuitive judgment to decide what’s best. Trying to apply traditional engineering approaches will generally stifle and drive away the very talent that excels at solving these problems. Just ask Google. Their well respected expert on design and usability quit this year, saying:

I’m thankful for the opportunity I had to work at Google. I learned more than I thought I would…. But I won’t miss a design philosophy that lives or dies strictly by the sword of data.

The full text is an interesting read. Probably the most poignant example was testing what shade of blue should be used in a specific scenario. This is a good example of trusting your judgment, but don’t try to explain it. It’s a fundamentally human, intuitive leap and you might be able to rationalize it, but that doesn’t mean you can really explain it.

The best part is that if Microsoft is finally getting the message that it isn’t enough to just complete on business and engineering requirements but instead you have to battle for the hearts and minds of the people that use products it’s only good for everyone. Just like Linux has pushed Microsoft to be faster at evolving Windows (and creating more low cost licensing options), this may push players that are known for great design to have to up their game as well. I can’t wait.

The prevailing wisdom appears to be:

• Google is the world’s best technology company.  Anything they solve they are the best at, and if you aren’t doing it their way you’re a dinosaur.
• Client / Server is dead.  All new applications will be web applications, most likely delivered as a service.

We visit a lot of companies, both prospects and clients.  Here’s what we’ve actually seen over the past two years:

• Outside of search, Google doesn’t have their act together.  Pretty much everything else is an academic experiment.  There’s nothing wrong with experiments, but there’s a big distance from there to running your business.
• Business are run on Client / Server.  They’re still creating new apps this way – they may use newer technologies like WPF to do it, but if it is core to making the business work, it’s usually not a web app.

With a name like Google, it’s got to be good!

Our own internal experience mirrors this.  We use Exchange and Outlook for email so I can’t say a lot directly about Gmail, however it’s clear from Google’s reactions to recent outages that their perspective doesn’t fit the enterprise because it doesn’t take into account the premium companies place on predictability and communication.  Predictability meaning that things are consistent – you get a consistent experience so your users can get their jobs done.  Businesses are change averse for good reason:  Users will adapt even to crazy problems and discover the patterns that work to get their jobs done.  When the patterns keep changing, they get frustrated.  On the communication front, businesses prize feedback on knowing the true scope of a problem and how long it may take to get it resolved.  

SalesForce learned this a few years ago and in response introduced Trust.Salesforce.Com, which went a long way to getting companies what they wanted to be comfortable with an outsourced critical solution.   If you operate a SaaS, you would do well to model after this.   Now, it isn’t necessarily bad that Google doesn’t do anything like this for Gmail – it just means it isn’t a solution for a large set of businesses.  

The thing we use the most internally from Google is Google Analytics.  It’s very pretty and easy to use, however we’ve noticed a lot of “what’s broken today?” experiences with it, enough that we can’t recommend it to anyone.  Two of the sites we monitor appear to be chronically under counted by Google Analytics, and we can’t figure out why.  And like most things Google – you’re on self support.  Now, there are paid options however unlike email we haven’t been able to find a strong paid competitor that is actively competing with Google.  It feels like most have left the field of battle, or are exorbitantly expensive (and aimed at large enterprises).

After much early on talk about how Google Docs was going to make Office obsolete, it simply hasn’t come to pass – and Microsoft continues to sell a lot of copies of Office.  It turns out that making a great word processor and spreadsheet is a very hard problem to do through the web.  Now, you might take the perspective that Microsoft’s announcement that they are going to offer lightweight web versions of Office 14 applications as being an admission that the old model is bankrupt, but it really points to an increase in reach:  Reaching many users that wouldn’t have been purchasing the product before.  Casual home users that wouldn’t purchase a “real” copy of office may find what they want in Google Docs, and would also be happy with a lightweight feature set of Office.

In Google’s defense, the products are worth what you’re paying for them:  Free.  But, you have to ask yourself:  If it wasn’t for the Google brand, would you give them the time of day?  For search, absolutely.  Finding an address and getting a street view?  Bring it on.  But don’t feel bad for depending on traditional software next time you want to buy a copy of Office, Photoshop, or use Outlook for your email.

It’s all SaaS These Days

Make no mistake, web applications, SaaS and Cloud Computing are all here to stay.  However, that doesn’t mean that there isn’t a lot going on in Client \ Server as well.  The key question to consider is how many applications you would have made, but done in another technology are being built as a web app instead?  There certainly are some, but for the most part web applications are creating entirely new spaces and solving problems that weren’t being solved before instead of replacing entrenched problems.  

Take document management:  On the surface this feels like a problem that should go entirely web and not look back because the web is very good when the readers to editors ratio is very high.  That said, the big document management companies still have very robust, integrated traditional offerings as well as their web portals.  You do have people using web document management solutions (like SharePoint) that never had document management before, which is a case of expanding the size of the market not replacing an alternative option.

When you are running your business, you have a set of requirements that are often best suited by a traditional client application:

• Users need advanced capabilities:  Once you stray out of the basics, it is invariably harder to provide features through web technologies than traditional technologies.  Tools are improving and making it better, but the cost per feature is lower in a modern client development environment than through a browser.  This is particularly true since you can put a browser in your app to enable things that it does really well – like show HTML content – you just let it do and keep the tricky stuff – like that big set of coordinated data entry fields – in your traditional app.
• Users are doing a lot:  One thing that is easy to lose perspective on is that when you run a client/server application every computer is bringing power to the party.  You often don’t need that strong a central server because the real action is happening out on the clients, and every new client that logs in is bringing their own muscle with them.  In the web, it’s all on the server. Not only that, but you’re rendering things in a less efficient way due to the stateless nature and limited protocols available for data exchange.
• Users need access to diverse data:  With web technologies it’s straightforward to manage information once you’ve brought it into the cloud, but it’s tricky to provide end users with fast casual access to a range of data in a range of formats.  In many businesses data is coming in pieces from many sources and being assembled to produce a coherent output.  This isn’t easy to do in any environment, and it’s one thing that the modern PC operating system, particularly Windows, has gotten very good at.  You can double click a file and almost always get it to open into a good viewer.  You can preview files, drag and drop from one program into another or a file into a program.  All of this is intuitive and fast for users that aren’t fitting into a pre-packaged user scenario.  

There are counter balancing effects:

• Transient user community:  The more far flung your people are, the more work it is to keep them up to date.  The more transient that user community is, the higher a barrier installation is.  This is the leading reason why you want to make something a web app:  It just isn’t worth the deployment effort to do it any other way.
• Diverse user community:  If you want to service Linux, Windows, and the Mac then web technologies are the lowest common denominator, it’s just the way it is.

What we’re seeing in the field fits into this:  People are creating a lot of new, lightweight web apps to solve point problems they probably wouldn’t have solved through technology before.  But they’re also still heavily investing in traditional applications.  

As a development team, it’s easy to get caught up thinking that Effort is the same as Value – just because something was a lot of work it must have a commensurate value.  The fact is, that there’s just no evidence that effort and value are correlated.  On the web, if you want to create a great looking and functioning generalized tool you’re signing up for a lot of effort.  And the value may be there – it could be that you’re going to reach a whole set of new users that otherwise wouldn’t use an application at all.  On the other hand, it could be that users perceive no extra value for it being in the web so all the extra work you put into creating the graphics, testing in five browsers, establishing identity, and the dozen other things you wouldn’t have needed to worry about if you were just running on the desktop netted you nothing.

So if you’re a business wondering how to approach that next application you need, don’t be afraid to get one that isn’t all wrapped up in Web 2.0 goodness.  In the end, it’s all about making the solution work for you and your users.

Bring a new product to market is really hard.  I’m sure you’ve heard that before – but however hard you think it is, it’s harder than that.  While we’re not quite across the finish line, there are a few things that have become readily apparent:

• Commercial-grade quality takes a lot to achieve.  At each turn where you might normally say “well, users just shouldn’t do that” you can’t.  Things you otherwise solved through training you can’t.  It’s the difference in construction of a commercial Amp and the receiver you bought at Best Buy.  
• Users won’t read anything.  We did a beta release where we posted in five places instructions for how to upgrade from the prior beta which required an extra step or things wouldn’t work.  We got deluged with calls about it not working from virtually every beta user; no one read any of the notes they saw, even in bold text in a yellow box in the middle of the screen.
• Marketing involvement early and often:  The feedback from our first beta version was brutal; it told us that we were going in entirely the wrong direction because the users we were building the app for weren’t going to buy anything regardless of how singing & dancing it was.  We had to step back and go a whole different direction.  That would have been far more painful if we hadn’t been early in the process.

From here on out, I’ll be contributing to a separate blog articles that are focused on .NET software development and being part of a small Independent Software Vendor (ISV).  This site will focus in more on its original goal:  IT and business strategies for reliable systems.

While the projects were successful, it bothered me that the secret sauce seemed to be a rigid adherence to schedule and delivery more than any other consideration.  This was exactly the reverse of how I wanted projects to succeed:  I wanted them to succeed because I was treating the developers how they always wanted to be, not like a stereotype from Office Space.  How could it be that better results came from ignorance of the technical details involved?

Developers Will Use All Available Time

Upon reflection, the first thing that struck me was how much an immobile deadline focused discussions and decision making. If you give a team more time, they will expand their process to consume it.  Time will get consumed by:

• Elaborate Decision Making: When you have little time, you make a choice and go with it until it appears it just can’t work.  When you have a lot of time, you sit back and look for the very best option.  That then requires defining what the best is – is it fastest, or smallest, or most scalable, or whatever.
• Development Approach: Under pressure you’ll tend to go with the proven guaranteed approach.  If you have the luxury of time you’re more likely to engage in yak shaving like investigating a new tool or approach, or writing several prototypes first before you develop the real solution.  You might even just throw caution to the wind by skipping a formal design figuring you’ll have the time to just code and test your way to a solution.

The more time a development team has, the harder it is to argue against spending it on up front luxuries.  It also can be harder to argue for long term best practices because the team has the time now to develop a solution any way they want.

Unknowns Create Boomerang Estimates

Even very experienced developers are generally terrible at estimating the duration of developing a solution.  This has been demonstrated over and over by many other parties.  The key behavior that we’ve observed is the phenomenon that from when you approach a specific development problem (like displaying a graph on a web page) until you know exactly how you’re going to solve it (and have a reason for confidence in that approach) you will tend to estimate high because in effect the only reasonable estimate is infinity.

Put another way, as long as you don’t know how you will solve a problem you don’t know for sure that it is solvable which means it will take an infinite amount of time to solve it.  Fortunately, developers are almost universally optimists so they believe they can solve anything eventually – so they’ll pull out a standard answer like three weeks or months or whatever feels like a big chunk of time to figure out the problem but not so big that it kills the project.   The reality is that until you know how you’re going to solve it, it feels like it could take forever.

Once a solution has presented itself  the development team will often find that all it will take is some cleanup and polish to be done- a very small amount of time.  What will push the team to find the answer?  We’re back to the problem of elaborate decision making when you have the luxury of time.  Finding solutions tends to not be a linear problem that will be solved with incremental development energy.  Instead, it tends to be solved by getting people together and brainstorming possible solutions until you find a few candidates and can work out what it’ll take to prove them out.  Under pressure, people tend to focus their creative energy and be more willing to compromise.   That flexibility will tend to get rid of pet requirements and developer gold-plating and focus on the most critical aspects of the problem.

What’s the alternate approach?

The key is to not let your knowledge and experience as a developer lead you to buy into the stories the team creates around what’s reasonable to get done and how long it will take.  Instead, you have to stick with the project’s goals first then the facts of the project.  The project’s goals form the objective reality of what has to be accomplished for the project to survive:  Deliver this functionality by that date, keep these people informed, solve these problems without causing those problems.

When the team runs into a wall and needs more time, instead of buying into the story of needing a lot of time, set a specific and tight goal that keeps a solid amount of time pressure on the team to solve the issue and prevent the problems above from showing up.  Ideally, find a way to give out one or two day chunks to answer incremental questions if necessary to emphasize that time is precious and has to be invested carefully.  This is where you can leverage your experience in a way that a non-developer can’t:  The team knows they can’t snow you with tech details, and you can define a specific, measurable result that can be achieved in a short period of time that they can’t argue with.  Despite this, you are bound to have to assert a few times that the time limit is the limit – solve the problem in that time.  It’s very hard because you’ve been on the other side of that conversation and it can feel like you’re the Pointy Haired Boss, but it’s fundamentally your job on the project.

What will nearly always happen is the team will surprise itself – a solution will be presented within the team that they can live with and can be done in the time they have.  It may be incomplete or have some risky shortcomings, and you’ll want to ask how long it’d take to address those.  You probably shouldn’t address them in the first round, but the team will feel better that you’ve considered through things and will buy into the outcome more if you ask.  You’ll also want to make a record of it so that the team can in the future recognize what was a predicted shortcoming vs. an accidental defect.

Do you want it solved right?

This is a question that often gets voiced within a team as a rebuttal to external time pressures and is very dangerous.  The challenge is that most non-technical people don’t get the number of ways that a problem can be solved: instead, each problem appears to have a single solution.  Take away your technical knowledge and imagine you’re the paying customer:  What’s the alternative – were you going to solve it wrong? If that’s the case, what else have you done that’s garbage?  If you took your car to a repair person and they said it’d be $500 to fix it, then when you came back they said well, if you want it fixed right it’ll actually be $1200, wouldn’t you wonder what the hell the $500 fix was?

Usually this statement is uttered in desperation when a team believes they just need more time to figure out a problem.  Nobody wants a problem solved wrong.  Skip the hyperbole and get down to action:  break down the problem into small chunks of time that can be invested for a specific measurable result, and make sure the team gets that overage time is the most precious commodity.

Side Note: This is an advantage of SCRUM in practice.  If you’re following an Agile Development practice, particularly SCRUM, this fits right in:  Focus on making each sprint deliver the user stories it was supposed to even if you have to leave some special cases for a later sprint.  The daily stand up meetings are a great place for the different team members to apply team pressure against over engineering and doomsday estimates.

Cleaning Up and Closing Out

At some point you need to close out your release and ship it. For each of the areas where you’ve had to make compromises and taken shortcuts you have to choose to either:

• Ship as Final: Decide the implementation is close enough to the intent of the end-user functional requirements that it can be the final implementation (at least until new information contradicts this decision)
• Ship as Temporary: Decide that something is better than nothing and ship the feature with limitations.
• Cut the Feature: Hold back the feature until it can be reconsidered or reimplemented.

You’re nearly always better off shipping the feature, often as a final feature pending more information because it’s very hard to gauge the true impact of each limitation.  This is particularly true of user-facing features and environments where it’s possible to evolve the software rapidly.  Inevitably once it’s in the hands of your users you’ll discover aspects of it that you didn’t think of that will require rework and you may discover that the killer feature you were sure would be the hit of the release is hardly used.  In either of these cases if you’ve invested a great deal of time in making it foolproof the team will tend to resist changing it.  It’s a natural product of the presumed relationship between effort and value.  If necessary, you might put in some temporary safeties to detect and catch the limitations you’re worried about.

The major exceptions to this approach are areas that are too dangerous to deploy if less than fully trustworthy.  For example, if your team is developing a data storage system, software deployment system, or other critical infrastructure your choices likely resolve down to making it as right as possible or holding the feature until it can be reworked.

If it turns out that the solutions that are viable within the schedule have significant limitations, you should make sure these caveats are known to the business – provided you can express them in business terms.  For example, knowing that an algorithm won’t work if your userbase doubles is probably not a significant caveat, unless you know the business plans to double in a relatively short period of time.  Every system has limits, and every software change has risks.  Business representatives don’t like to hear the same items covering the same ground repeated every time you discuss software, and it tends to make them not hear the new and important information as well as sound like you’re attempting to transfer accountability from your team to them.

In the past several months the three partners of our firm have been scattered to handle all of the challenges of our growing business, but with the completion of a major engagement with a client we had the opportunity to spend some time back together to focus on more long range concerns.  We had intended to spend the time working on product and marketing strategy for our upcoming commercial release of Gibraltar, but instead on a whim decided to do something we called the Code Monkey Challenge.

The goal was to make something complete and useful within a very short period of time.  Complete for us meant we were going to create a software component, test it, package it, and publish it to a new web site with descriptive content all in 48 hours.  We didn’t even have a hosting company set up beforehand.  To be useful it had to be usable by people outside of our company as delivered, and fulfill a real need, at least for a set of users.

We started at 9:00 AM with the goal of publishing the final version to a new web site by 6:00 PM the following day.  Our initial approach was to have one hour sprints of work divided between each team member, then meet and review and set up the next sprint.  In the end we used three hour sprints, delivering to each other through our source code system at the end of each sprint.  We made our goal – you can see the results at www.gibraltarsoftware.com.  Most importantly, we achieved our result of bringing together our team members.

Update:  The small Logger we wrote as part of the Code Monkey Challenge has since been rewritten and incorporated into the Gibraltar Agent.

We eliminated any distraction we could so that we could apply as much of the 48 hours as possible to the project.  We worked well into the evenings and had others provide some support to make sure we had food, coffee, and no outside distractions.

The intense time pressure gave us a good tool to eliminate most conflict: there simply wasn’t time for much philosophy on how to approach the different technical aspects.  The lack of time removed ambiguity that otherwise would be the source of conflict.  There wasn’t much time for yak shaving either – we had to produce results at the end of each sprint.  Still the first few sprints were spent mostly in exploration and validation of the approaches with our first rough cut of each element done by the end of the first day.  The second day was then about refining and documenting.

While we did skip over some aspects that we would normally do on a larger scale project, we did include most elements of our preferred development process.  We did coordinated sprints of the team separated by an after action review to adjust our plan.  We distributed code and results between the team through our source code control system, we wrote automated unit tests to verify key aspects of the system, and did peer design reviews.  If anything, the lack of time made us less likely to bypass most parts of the process because we knew we wouldn’t have time to dig our way out of any problems we created.

The real goal of the exercise was to bring us back together as a leadership team and establish more of the shared experiences that define interpersonal relationships.  It’s helpful to bridge the gaps that easily develop between architects and developers, developers and designers, engineering and marketing.  We all had to come together to achieve our result because it was clear that we’d all fail if we couldn’t.  Most business projects are sufficiently fuzzy that it isn’t nearly as clear what the cost of not working together is, and politics can overwhelm cooperation.  It’s an experience I’d recommend in any software team, particularly if you’re in a company that can mix skills and responsibilities.

The next time your shop is done with a project, or when you need a break consider doing your own Code Monkey Challenge.  It only takes two days.  Here’s the rules:

• Produce a Real Product: Scope out a product that is really useful to a specific audience.  Sure, you’re giving it away for free, but it still needs to stand up to scrutiny.
• Marketing Material Too: What good is a product if no one knows about it or understand how or why to use it?  Even though you’re giving it away you want people to be able to understand and take advantage of your effort.
• Help and Usage Information: Like a real product, it has to be usable without you sitting over the customer’s shoulders.  Depending on what it is, you need to tell them how to install it, program with it, and provide guidance on recommended usage scenarios.
• In Little Time: Extra time is the enemy – it will reduce the pressure to work together and encourage unnecessary bells and whistles while removing the catalyst that gets everyone to work together.
• Publish to the World: If you’re  a large company, perhaps it’s just the company.  Wherever it is, it has to feel like real stakes to the team:  Your results will be on display.

To set it up, be sure that you can eliminate anything that would distract the team – they can work into the evenings, have food brought in, whatever.  The closer it is to a total immersion experience the better.  It improves the sense of camerarderie developed within the team and ensures the most creative energy is directed to the project. 

If you try it out, or have another software team building story, please drop me a line and tell me how it went, or leave a comment below.

This is fine as far as it goes, but like any metaphor while it may be a way of explaining some aspects where two things are in common it’s very easy to overextend because in fact it isn’t a true financial liability.

Take a hypothetical example: You are supposed to add federated identity to your web application. What you want to do is create an identity broker that will allow your SaaS application to connect with Microsoft’s ADFS, OpenID, and a few variations of SAML. You believe this will get you the market reach you need, and by creating your own identity broker you can decouple your application from changes in this still evolving space, as well as support your own native security technique.

As you dig into it, you realize that this just isn’t feasible in the time you have: You need to get a solution done in two months to meet a commitment to a customer, and it turns out this customer just uses ADFS. It just so happens that your web framework can easily work with ADFS directly, so to save the schedule you drop back and just do ADFS. From a development standpoint this is a hack – you are doing something quickly you can’t extend to meet the original requirement and you’re pretty sure that you’ll need to undo this later and do it right, which will cost more than just doing it right the first time. This is Technology Debt.

Metaphors have Limited Application

The metaphor doesn’t hold for long.  First, unlike real debt there is no external requirement to pay this back.  Perhaps you never will need to support more than ADFS, or that after all of the talk customers just won’t adopt federated identity.  At the start of each release cycle, you can look at the competitive market and see what is the correct, most important work to be done.  It might be that you’ll have to make good on something you deferred, but you might not.  If you didn’t, it never was debt.  How confident are you that you can tell the difference right now?

Second, all technology has future liability. The more code you write, the more you have to maintain and support. That has a cost as well for the future. Depending on the nature of your product it could mean that you have to support questionable past API decisions or obscure and intricate features of your product. Every feature has a cost to maintain, every line of code you use or reference in a third party library has weight. Only talking about deferred development as technology debt implies that what you have right now is all asset, but it isn’t.

Rampant Misuse

The biggest challenge I’ve seen is that the metaphor is used to push development team goals on the business without having to adequately justify them.  By handing business folks something they can easily relate to, it’s easy to gloss over the underlying technical implications.

For example, say you have an application that’s currently written in Visual Basic 6.  You can justly claim that Microsoft has dropped support for it and that the day is coming when it will no longer run on the latest operating system (Microsoft has committed that it will run on Vista and Windows Server 2008, but that will likely be the end). This sounds very alarming indeed!  Naturally, the answer is to rewrite the entire application in .NET 3.5. Yes, this will take longer and cost a lot more than just adding the features you need to the existing code, but it eliminates all of the technology debt represented by that old nasty VB6 code.

Now look at it a little more objectively. Yes, you will need to eliminate that VB6 code at some point – notably when you are upgrading from Vista to whatever’s next.  When will that be?  Well, if you believe the hype that people love Windows XP and may just consider Vista in the future, you have some calendar time. As long as it’s done by then, Microsoft dropping support isn’t a real issue.  We’ve yet to work with a client who’s VB6 application didn’t work just fine on Vista, thank you very much.

Second, with rare exception every modern technology has a half-life. The notable exceptions are COBOL, C, and C++.  You can with confidence know that these languages are still going to be in active use in 15 years and that code you create now will work with minor to modest adjustment on current platforms at that time.  I would dispute if a similar claim can be made for the latest crop of languages.  A major reason for this is that modern languages are really whole environments – a programming language and an API/runtime.  While it may be theoretically possible to write a C# compiler without toting along the .NET runtime, it isn’t going to happen.  Likewise for PHP, RoR, and even Java.  Java’s framework is much more shallow which ironically will likely give it more life because it can adapt to radical changes in computing environments and there is already an intent to decouple most of the code from specific frameworks.

This means that today’s latest and greatest environment will be tomorrows VB6.  Just listen to people that jumped on .NET 1.1 discuss how they’re going to upgrade to .NET 2.0 or later, and that’s really not a particularly dramatic move.

Fundamentally, when you purchase software you’re making a bet in the vendor and community behind it. This is one place where the LAMP stack has a number of advantages because of the openness of the environment and the institutionalized tolerance of actively supporting releases for a long time. This is fundamentally enabled by their open source nature, but even if you had the source code for a commercial product it’s not likely you could do much with it; unless it’s fairly simple the burden of maintenance is likely higher than the cost of replacing it. Counteracting this are a lot of hidden costs to open source that may be difficult for a small team to absorb.

Easy Metaphors Seldom Produce Great Outcomes

Next time you’re trying to bridge the gap from technology to business, try to stay away from simple metaphors like Technology Debt. Instead, have real conversations to articulate the potential business impacts of the technical decisions and then hear from the business what they are concerned about. With a real dialog, everyone is in on why you deferred work to later and what bet is being made, and no one will be surprised when you do show up in a year to start porting that VB6 app to .NET.

Nearly all teams work within a relatively closed ecosystem – the technologies and people represent only a minor subset of all technologies available today.  Even within these small groups the number of choices at every level are daunting.  Even if you’ve selected your OS, language, framework, and database – what architecture model are you going to use?  What is your data access and caching strategy?  At each turn you’ll want to pick the best option but have a flood of choices to select from.  Many people can tell you about how they led a project that used any particular technology and it worked like a champ with no drawbacks.  On the surface it makes it possible to defend just about any new technology as the way to get your next project done.

What is very hard to find is a real comparative analysis that highlights in comparable situations the results with different technologies.  It’s not a surprise this is so – such analysis is very expensive and time consuming, and few companies would try to solve the same exact problem using competing technologies because it’s not the business they’re in.

Where does this leave you?  What technology should you use on the next project?  Most likely, you’ll have the best success with an incremental improvement on the technologies you already have in your toolkit.  Why?

Infinite Solutions in Infinite Diversity

Most conversations in technology on the web are exclusive - they advocate X over Y or Y over X.  The truth is much more that X or Y can both solve the problem but do it in different ways and with different levels of effort for a team starting from scratch.  What’s much more useful is to ask why you can’t solve the problem effectively with the tools and technologies that are a natural fit for your environment.  Even if a new technology may be the easiest way to solve the specific problem you’re looking at it may not be the best choice when you consider everything that goes into creating the entire software system and maintaining it over time.

When confronted with a strong advocate for a technology shift, keep the conversation focused on the benefits of shifting away from the natural or familiar selections for your organization.

New Methods are Expensive

When you introduce new technologies or tools there is always a short term hit.  Most respected research indicates that even technologies and tools that have a substantial improvement in effectiveness are at best neutral on the first project that uses them.  The most successful technologies and tools are ones that are evolutions of things your team already knows.  The more divergent it is from that, the more time it will take to get over the learning curve, establish best practices, and generally become effective.

Known Problems vs. Unknown Problems

A common challenge when comparing a new technology against existing methods is not recognizing that while you know of all of the problems with your existing technologies, you don’t know of the problems with the new ones.  This can lead to a comparison that shows a number of critical problems with the current technology, and none on the new technology.  It isn’t that there aren’t problems with the new technology, it’s that you don’t know what they are yet. Whenever you put in a new technology, no matter how promising it is, it is going to have new, unexpected problems.

What’s worse is that your organization most likely has workarounds for every problem you’ve encountered, so they don’t really have the impact of a new problem.  Your development team may not know about them, but talk to the operations staff, support staff, and your users before you assume that a technology problem that worries you really is at the top of their list.  It may be that the big memory leak in a third-party library that has you wanting to rewrite a subsystem is conquered in production by having a script reset the service every night.

Existing Code and Libraries

It’s very easy to underestimate the value of existing libraries and practices in effectively solving problems.  When faced with a new assignment, your developers can draw from a large pool of existing, tested solutions to a range of the more mundane, plumbing aspects of the solution.  This includes storing user information, reliably working with data storage, security systems, and other functional requirements that aren’t unique to the problem at hand.  These software libraries accrue over time as developers face similar problems even in development shops that don’t place a high emphasis on modularity and reusability – as long as you have source code control and developers that aren’t paid by the line of code, they’ll naturally find ways to adapt and remold things they’ve already done to fit new needs.

When you have a major technology shift, losing the use of this common body of code will require the first project to reinvent it.  On the surface this may seem straightforward but it’s usually held up by a desire to understand exactly how to best accomplish the same common tasks in the new environment.  For example, you might have written your own security system for your previous environment which you’ll then need to either re-implement or drop in favor of a built-in capability of the new environment you’re targeting.  What’s worse is you need to make these critical decisions at the time when you have the least experience with the new environment:  Is its built in security system really sufficient for your needs?  What about logging?

Your Customers Don’t Care

With the exception of a narrow range of situations (such as developer tools), your customers really don’t care what technology you use to implement your solution.  After all, they’re buying your solution not the technology you wrote it in.  Even if the IT representative of the evaluation team in a potential customer objects because your entire solution is written in a technology they don’t like, in the end they are often overruled unless they can point to a practical implication that you can’t mitigate.  For example, you may get overruled because it’s a Unix shop and they won’t accept a solution that only runs on Windows.  Even in the most extreme cases, if you provide enough customer value it will conquer any customer technology objection.  If the prospect has no Windows servers, that translates into a finite cost for them to support a unique system in their environment.  If your value well exceeds that, then it isn’t the key challenge to crossing the chasm to that prospect.

We often hear developers discussing internals of software development and giving them the weight of user requirements.  If it isn’t visible to the customer, it isn’t a requirement. In the end, your customers don’t pay you to have a beautiful object model.  They don’t care how hard it is for you to create your product or what hoops you have to run through.  For them, it’s a cash for capabilities decision.  It may be true that doggedly sticking with an old technology will mean you can deliver fewer features with each release, or you won’t be able to run on the latest operating system but in the eyes of your customers the question is still how compelling the functionality is and whether you can run on the operating systems they use.

Ignore the Pundits

If you’re part of a shop that has a track record of producing results, be proud.  Don’t worry about what is all the rage at producing the next social networking site, focus on what is effective for you.  For projects that can afford the risk, take the opportunity to incrementally improve your technologies and methods:  Try out a new version of the development framework or new capabilities of the latest database version.  Just remember, you can always tell the pioneers:  They’re the ones lying on the ground with the arrows sticking out of their backs.  Unless you’re part of a dedicated research team, most often you’ll get the best results by waiting for the first round of adopters to figure out what did and didn’t pan out with the newest release and then benefit from that experience.  There’s no satisfaction in burning six months working out the kinks of version 1.0 just to have everything addressed in version 1.1 published a month later.

What’s Your Experience?

Have a great story about being the pioneer, working a project that was packed to the gills with the latest and greatest, only to fall on its face?  Or perhaps you found raging success completly severing your ties with the past?  Drop me a line or leave a comment about it.