Archive for Infrastructure
Walking the Walk – Gibraltar Moves You Down the Path
Posted by: | Comments
If you’ve read more than one or two articles from Reliable Systems you probably have gotten the sense that we worry a lot about how to make things just work. It’s that quality of anything where you get what you expect and what you need every time. It can be in an experience (like a fun drive down a country road) or a product. As a company if you can do this over and over you create a brand people develop a strong emotional connection to: Apple, John Deere, Starbucks…
When you want to create a product that just works, you need to get all of the details right – from packaging through to maintenance and upkeep. It’s not one thing that’s important, it’s all the things. We are often engaged by senior management within a client when things aren’t working, and there’s conflicting opinions on why. Usually along the path technology is being blamed: Not enough, not the latest thing, not someone’s favorite thing, not working. As we dig into the situation, rarely is the technology the dominant factor: More often, it’s how the technology is being integrated with the people and processes that all have to work together.
One of the first things we have to do in these engagements is to establish the real facts on the ground: What exactly are the problems in the system, who’s doing what with it, how many times. It comes down to establishing metrics to make sure time and attention are paid to the parts that make the biggest difference in the outcome. Armed with these facts in a form the business can consume it’s possible to create plans of action that deliver virtually regardless of budget.
So let’s make this easier
The biggest trick is then getting the facts you need on an ongoing basis, easily, and in a form that the business can consume. For over a decade we’ve been building instrumentation right into the systems we’ve worked on. We’ve created a variety of toolkits to make this easier over the years, refining them as technology and our experience has changed.
About 18 months ago we decided it was time to really invest down this path. We believe in routinely capturing key computer metrics along with whatever logging the application can do on its own. We won’t do a project without using a great logging system that includes a strategy for managing runtime exceptions. Now that we’re collecting all this data, we need to have a way of managing the raw data and turning it into valuable business data.
The challenge is that businesses don’t get up in the morning and say “what our customers want us to do is have great internal tools”, so you’re nearly always doing this on the cheap: Borrowing time from development projects internally to cobble together various free or cheap solutions. Frankly, we got tired of having to create new solutions with each client out of the margins of each project. So, we pooled our best thinking from all of the work we’ve done (including a previous product that we did license to our clients over the past decade called CLAS) and started creating Gibraltar.
Rock Solid from Initial Release
With Gibraltar we wanted much more than a log system. Of course, it had to be a log system too – and a really easy to use one that could work with each of our client applications. More than that, it had to:
- Automatically capture all of the performance metrics we wanted.
- Integrate with existing logging available on the platform, including whatever a client might already be doing (like custom in-house options)
- Be absolutely, positively, for sure safe to run in production no matter what. That means it can’t ever use too much disk space or disk throughput or block the application.
- Not use more than 5% of the performance of the app
- Include all of the tools necessary to get data from where it was collected to the people that could get value out of it
- Include the ability to look at the detailed session data up to high level analysis: What’s the error rate? What’s it correlate to? Are we doing better or worse in this version?
From this initial sketch into everything we wanted, we’ve spent 18 months including four beta periods (from 2-4 months each) to refine the vision with real customers and real scenarios. It was essential to us that this not be just a tool for techies but be ready for use by people with a wide range of skills. It had to be pretty and just do what you wanted, when you wanted it to.
We’ve added a lot of capabilities along the way: It can generate print-ready reports about application reliability that can communicate with senior management, you can define all kinds of custom metrics to easily track how your application is used and by whom. We ran a number of betas to be sure that we had hit every goal we have above. We’re happy to report that Gibraltar is in use within large deployments of custom applications, commercial applications, and small deployments right down to our corporate web site.
This tool isn’t for everyone – Our clients are nearly all Windows shops, and if they do any custom development it’s almost invariably in .NET, so that’s what we’ve targeted. But, if you’re interested in easily getting real data on not just infrastructure (how well the application is running) but whether or not it just works, have we got an easy path for you. You can see a quick demo video of how it works technically at Gibraltar Software.
You also don’t have to take my word for it at all, you can hear what one of our beta users did with it, which is really a more compelling story than what we might say.
I think you’ll find that our work sweating a lot of little details, from the exact design of the API and making sure the documentation was complete to rewriting our own licensing system to be very IT Admin friendly. If we didn’t get a detail right, we want to know. And the great news is that we’ve just begun: We’re obsessed with the little things, and you can bet we’ll keep listening and watching to make it better. Of course, this is made a lot easier because we’re using Gibraltar to monitor itself, and a select group of our users is sending that information back to us so we can make sure it just works in the field for real people.
It’s easy to start your journey
If you do development for Microsoft .NET, I’d encourage you to go over and download our commercial release of Gibraltar. You’ll get great documentation, a free agent you can use like a flight recorder “black box” in every application you create, and a trial for a tool that will make you seem wise beyond your years. And if you pay us the ultimate honor and purchase a permanent license, I can assure you that you won’t find anyone more committed to your satisfaction than we are.
Watch the Gazelles Turn
Posted by: | CommentsIt is very tempting to be one of the herd of gazelles in technology. Every time there’s a sense of a shift in the wind, everyone starts to run in a new direction. For the past year I’ve been reading about how it’s all going to be laptop computers from here on out. In fact, not even full fledged laptops, but netbooks – computers with small screens and small keyboard who’s main distinguishing characteristic is that they’re less of a computer than anything else around.
If all this sounds a little off kilter from reality, perhaps a few hard numbers would help:
Quoting Computer World, who asked “Do Business Desktop PCs have a future?”:
While desktop PCs account for the bulk of personal computers sold to enterprises, the gap in laptop sales to enterprises is closing. Of 168 million PCs sold worldwide to professional organizations in 2008, about 95 million were desktops and 73 million were laptops. That’s compared to 94.6 million desktops and 47.3 million laptops that shipped in 2006.
Now, as with any statistics there’s two ways to look at these numbers:
- Laptops have grown tremendously in their total percentage of the market, and that growth rate has them on track to take over the world.
- The majority of the growth in computer sales is coming in the form of laptops.
The gazelles are taking the first road. And why not? People love to assume the disruptive is true, it’s a lot more interesting. Before you charge down that road, consider what seems likely. There are a few problems with the first conclusion:
- Two data points don’t make a pattern: If you follow the trend back farther, the sales of PC desktops has held up consistently, but laptop sales go up and down. This would seem to indicate that the most likely interpretations of the data are that either the overall market is expanding (for example by people having two systems) or that this is a momentary, periodic surge in laptop purchases.
- Past large growth rarely projects forward: Just because there was a large growth in one year (either in absolute or percentage turns) doesn’t mean it will repeat at all. It’s just as likely that the next year pattern will be flat or even retreat.
So before you see the first twitch and assume it signals a migration of the whole herd, step back and think through the underlying facts. Is this really the first sign of a monumental shift? Or just another twitch of the needle? Then look at your own situation.
Now, we have a few laptops, but we have more hard core desktops – the laptops are used for on the road presentations or working at Starbucks for fun. Of course, we’re developers so we’re in the category of users that are always excluded from the norm. But what’s not to love about a desktop? For the same money they will always be faster and more capable than a laptop because they don’t have the burden of being small or extra power efficient. Even if you buy into the idea that everything will be run through the web so computers are just glorified terminals… Something still has to compose all of those web pages and make it all come together, and web apps can burn a surprising amount of processor and RAM locally.
In the end, I think we’re seeing a lot of folks buying second computers or getting additional laptops for other uses that complement their primary work computer experience. Additionally, there are folks in emerging markets that need what laptops offer (self-contained, reliable power) more than performance but this reflects an increase in the overall market, not a shift in the existing market.
Careful with that thing – it’s running Vista!
Posted by: | Comments
Everyone likes to be on the winning team. We love to root for our favorite sports team, we like the car we own and the brand behind it. So it’s no surprise that when Apple ran their I’m a Mac ads that Windows fans were in an uproar. Now with the Laptop Hunter series the shoe’s on the other foot. Microsoft is making a big show that Apple computers are overly expensive just for the Apple brand. Apple fans claim that to match a Mac, a PC has to be equipped with tons of antivirus software, a full time tech support guy, and a Witchdoctor on standby to keep it working.
Seriously people
First, Apple makes some of the finest hardware you can possibly buy. If you compare it nose to nose with hardware that’s actually built to the same standards then it really doesn’t represent a significant price premium. Compare a Macintosh Pro with an equivalent Dell workstation – the cost is within 5%. It’s amazing that Apple can afford the extra engineering of an OS with that little premium.
Second, Vista works great. It’s running on many more systems than Mac OS is, and with volume comes a range of new problems. The total amount of money I’ve spent on desktop antivirus software in 10 years of administering PCs? $0. The total number of virus problems I’ve had? 0. My parents managed to get into trouble with one virus and Windows XP – but installing the (free) Microsoft Defender cleared it right up never to return.
As with nearly all marketing, this is a battle of perception: Apple has done a great job of defending their brand at every turn. This is part of their corporate ethos. Along with a few other tenants it ensures they are a much loved but niche player:
- Only do something you can do uniquely well.
- Don’t extend into markets that might ask you to compromise your values.
- Cultivate the mystique: Don’t show what’s behind the curtains.
Microsoft on the other hand has tenants that ensure they’ll be a volume player, but an unloved one:
- Play to win the most market share in any market you can.
- Build your ecosystem by making it easy for others to add value to it.
- Cultivate the engineers: Provide overwhelming amounts of documentation and approaches.
The fact is, many people don’t need a top end piece of hardware like a Mac. On the other hand, many people want a computer that’s just a tool, not a piece of art. To them, the nearly infinite diversity and low cost of entry are essential.
I’m a People person. I’m Good with People!
The computing needs of the average corporation and the average individual are very far apart. To companies, computers are tools – like the desk, phone, and copier. Very important, very powerful – tools. They aren’t there to make you feel great or enable you to create a cool video of your vacation in France. My partner really summed it up one day when he commented that the Mac was a really personal computer - it worked hard to create a personal connection.
Corporations on the other hand want slow paced evolution, massive support for legacy applications and hardware (these guys are still running dot matrix printers off parallel ports) and to control costs. They also philosophically want to have all the keys to the computers – just like they do for the buildings and offices they own. PC’s are just end points on the large mesh that is the corporate IT network. It’s very impersonal.
Microsoft makes a great deal of money providing businesses with the tools they need to have the computers work for them, and Apple makes a great deal of money creating computers that people love. Either of these goals would be compromised by trying to do both.
Vista Goggles
Folks that have been in the Windows ecosystem a long time probably recognize that you could take the first year of press about Vista and substitute “Windows 2000″ and find the same article written 8 years earlier. Vista is a surprisingly large and tricky step forwards on a number of fronts, whereas Windows XP was a visual redress of Windows 2000.
Almost like an SAT test:
Windows 7 is to Windows Vista as
Windows XP is to Windows 2000.
Like Windows XP, the story on Windows 7 is making virtually no architecture changes and instead just tuning for the long haul. That’s a great thing, because there’s a lot that works very well with Vista, and now it’ll work even better with Windows 7.
The humorous thing is to read now about how people are thinking about moving to Vista once 7 ships because, well, they don’t want to move to an OS that was just released. It’s as if Vista has been aging like a fine cheese on the shelf so the very same binary code that once was toxic is now just what the doctor ordered. To a slight degree this is true: Vista SP1 did address some issues that affected some people, and more importantly hardware now is dramatically faster than it was two years ago (as it will be two years from now…) so what once was aggressive is now commonplace. The same was true of Windows 2000 when it shipped. Requiring 64MB of RAM? That’s just crazy talk! Only certain video cards worked reasonably with it, and video drivers to a while to stabilize. That sounds very familiar…
In the end, it really comes back to Perception. Probably the biggest mistake Microsoft did was not push the OEM’s that make the computers to build machines that could responsibly run the new operating system, and be clear that meant hardware 3D video cards and plenty of memory. And oh yeah, stop putting aftermarket firewalls, antivirus, Google Desktop, and all kinds of other things on them that are ill optimized. At my last company we got in the habit of routinely wiping each new Dell that came in and reinstalling the OS from the Dell restore CD – because that got rid of all the noise. It was surprising how much better that worked. Is that Microsoft’s fault? Not directly, but they certainly could have found a way to encourage the ecosystem to forgo some profit for usability. But that’s just not in their corporate DNA.
With any luck, the big story for Windows 7 will be that Microsoft pushes back against their channel, even being willing to risk it by leaving Windows XP out there for folks that don’t want to play by the Windows 7 rules. It’s hard to put up barriers when you’re a legal monopoly, so find ways to use incentives to do it right instead of punishment for doing it wrong. And keep up the ads, because perception does matter in the long run.
Who knows, it may push Apple to get better too. Just once when my iPod updates itself to enhance stability and performance I’d love to know what exactly was unstable or slow…
Remote Access for Everyone
Posted by: | CommentsBack in the day, corporate remote access meant modem pools that people dialed into from wherever they were. Even then it was like watching a feature film on your IPod; you got a sense of the action but it was ultimately as much frustrating as useful.
Things change. Over the past eight years broadband in some form has become available in most cities across the nation. This bandwidth has made dedicated remote access a thing of the past. Now you can provide remote access to your employees over your Internet connection. Traditionally, IPSec has been the technology of choice to provide a virtual private networking solution for your employees but over the past two years there’s been a new game in town – SSL VPNs.
if you are using IPSec for your mobile users, you owe it to them and you to check out one of the SSL VPN options at your disposal. We’ve used IPSec VPNs for network to network access reliably, but they’ve always been tough to support for mobile users. Offhand, there isn’t any specific reason this should be true, but it is. For mobile users, we seem to consistently run into a few problems:
- Installation: The success rate for an average user being able to install an IPSec client and get the VPN tunnels to work, even with phone support, was around 15%. Most of the time the user had to bring in the computer or we had to send a tech on site.
- Compatibility: Different physical network technologies – notably DSL – run into performance problems with IPSec in many configurations, requiring adjustments on the client, routers, or other things that you just can’t expect end users to understand.
- Portability: IPSec is very easy to block on a network. In fact, it took some time for most network routers to be compatible with IPSec. Now try to get it to work at 8 PM over a wireless network in a hotel in Buffalo.
In contrast, a few years ago at the urging of Watchguard (our resident firewall vendor) we tried out their SSL VPN product, which was basically a version of the Citrix Access Gateway SSL VPN solution running on a Watchguard hardware appliance. Out of the box it worked – every time, and even faster than IPSec. We had resisted the option because we preferred standards-based solutions, and this sounded like yet another proprietary security technology. We used a demonstration appliance for a month but the feedback from our users was so strong we purchased a unit after a few weeks. Upon reflection, there really is a good bit of sense to why it works so well:
- SSL is Simple, IPSec is complicated: SSL is a single TCP/IP socket with a relatively straightforward, self-configuring, and invisible to intervening appliances.
- SSL is essential, IPSec is a threat: No one can afford to block SSL on their network without basically admitting to not having a network at all. It’s very expensive to proxy by decrypting and re-encrypting, so few companies do it. On the other hand, many networks view with suspicion the goal of establishing an encrypted connection out of their network, so blocking IPSec may sound like a good idea.
With the SSL VPN solution we had about an 85% end-user self install rate without support, and a 100% rate of not requiring a tech to go on site. Even better, the reviews from end users was that it was fast to connect, easy to use, and performance was good. Because it was so easy to get set up, many more users started connecting from home in the evenings or in bad weather to get work done. The net cost? While your firewall probably offers an IPSec client for free, you can expect to pay a few thousand for a dedicated SSL VPN appliance and depending on licensing $50-$200 per concurrent additional user after the first five or so. For a company with say 100 users that might have at most 20 concurrent users the cost is in the order of $4,000 to $6,000.
Making the Business Case
Jumping from “free” to $6,000 may seem questionable until you look at it from the value standpoint: A service that was expensive to setup and of questionable reliability became cheap to set up and rock solid. In other words, this is the real cost to provide this service. An unreliable solution isn’t a business solution. If it’s more than your business is willing to pay, wait a little while – the cost has come down by half in the last two years, and some vendors (like Watchguard in their Fireware Pro product) are offering it alongside their free IPSec VPN option.
So Why are You Still Hosting?
Posted by: | CommentsRight now, the power is out at my home. That doesn’t happen often – in fact, it’s been almost two years since we lost power long enough for my UPS to shut down my home network. Normally this would be a small inconvenience, but I still host a few things for my wife out of my house which are now down. The largest of these is a fairly popular forum for an author she likes, but there are other sites as well.
Why am I still hosting these at home? Really there’s no reason – I’ve shifted hosting for my personal services out to other providers, and our company services are also hosted by hosting companies. I just haven’t moved her stuff out of my house.
We talk with a lot of small and medium sized businesses that are still hosting all of their own services internally for pretty much the same reasons – they originally had them in house when they were much smaller and the market was different, and haven’t considered what it would mean to have those computers live somewhere else. It’s time for a change.
Why It’s time to Use the Cloud
You should look at all of your important business services – things that your business can’t operate without – and work out a plan to no longer host those items in your facility. As a first step, just consider what it means to provide the same applications and services, but have the computers not live within your company. The main goals for moving these services out are:
- Business Agility: When you use a hosting company it’s easier to change capacity as your needs change, even to bring services up temporarily as a trial run and then shut them down if they don’t pan out. This makes it easy to experiment with new software technology without the traditional problems of hosting getting in the way.
- Low Cost Reliability: If you want those services available, the cost to outfit a room to provide redundant cooling and power for a single rack of equipment is easily $50,000. To host one rack of equipment in a basic Tier-2 data center can cost around $1,500 to $3000 a month, which includes power and Internet. At that rate, how quickly will you get an ROI on your facility investment?
- Improved Focus: Getting this equipment out of your shop improves your focus on the things you really need to be spending time on: Projects for the business and end-user support. The rest of it is overhead.
- Access from Anywhere: When you set up your services so they can live in the cloud and be used from your office, it’s easy to make those same services available to employees from home and from laptops. Not as second class citizens but with all of the ranks and privileges of being in the office. This helps you leverage employee talent wherever it is. It’s also easier to set up rock-solid extranet access for customers and suppliers.
When you start looking at each thing you provide as a service, you might also find that some of them – like Microsoft Exchange – really aren’t worth hosting yourself at all even in a data center, and it’d be ultimately in your best interest to outsource it entirely to a hosted Exchange provider. There are number that can do this very effectively. While the cost may seem high based on what it cost you to purchase your initial Exchange licenses, when you look at the real cash costs for Exchange over two to three years they are very cost effective.
Once you’ve taken the step of taking an existing service and outsourced it entirely, you might even consider a Software as a Service offering for some of your core services (such as a hosted CRM). This is the most aggressive mode of outsourcing and does create a set of unique risks and opportunities.
But I can’t See It
Two common objections we hear from IT administrators about moving services out of their shop, even if it’s just relocating servers into a data center. is that it will make it hard for them to get upgrades when necessary because the business won’t be able to see & feel the new equipment. Out of sight, out of mind as the saying goes. The second main objection is that the IT administrators want to be able to do a laying of hands on the equipment to maintain it. There’s a comfort factor in knowing you can walk into a room and flip the power switch or move a drive or just bask in the warm glow of blinking lights.
Here’s the good news: Both of these reasons are not only suspect in their own right, but are preventing your shop from getting to the next level in IT’s relationship with the business.
First, even though vendors do a good job of making server hardware look serious and fun, in the end it’s just a business appliance: It either is good enough to deliver for the business or it isn’t. With rare exception, there is no extra business value for it to look good, new, or cool. If you find that you need to show the business physical servers to explain your costs, you’re missing out on the critical opportunity to establish a real partnership between business and IT. You need to be sure you’re spending when it’s time to spend and saving when it’s time to save, and have discussions in the language the business would use for any other service it would acquire.
Second, If your IT administration patterns and practices require routinely touching your physical infrastructure then you need to re-examine them. It generally means you either have equipment that is no longer up to the task or that you’re not doing enough automation of IT tasks. If you have trouble-prone hardware, it’s time to either fix the fundamental issue or ditch the hardware. Ironically, this type of problem is often easier in a hosted environment because it generally isn’t your problem: it’s the hosting company’s.
Automation is essential because humans are the most error-prone part of any standard process. Your routine IT administration time shouldn’t be going to consistent tasks – they should be automated, leaving your time for user support and other business value-add services. That’s right – even in your shop with your existing staff you can find more time to spend on projects instead of support events by automating recurring tasks.
Some Things Still Stay
There are some things that should be on site for performance reasons. Regardless of how big your Internet connection is, you’re going to want basic file and printer sharing services to be local. Depending on the size of your site, you’ll probably also want a directory server for whatever your directory system is (e.g. Microsoft Active Directory). Even here the central services help: If you have a reasonable Internet connection, you can have your local file server back itself up to the data center by using one of a few distributed backup systems (such as Microsoft’s Data Protection Manager or a third-party option like NSI Software’s Double-Take). This eliminates the time and attention that local disk backups require.
Perhaps not Now, but Soon – and For the Rest of Your Life
It may not be appropriate to move a number of your services outside yet; If you have only one business site, light access by employees externally, and aren’t expecting that to change then you can host most things yourself. A number of the considerations still apply – but you might just use an external facility for your public web presence and for backing up your essential data for business continuity.
Even if you don’t do much now, you should find some opportunity to put a service outside so you and your company can gain experience at working with external hosting providers and you’ll stay current on the capabilities and costs so that as new business requirements evolve you’re ready to take care of them. You’ll be in a better position to advise your company on when to move things out of the shop, and as you do you’ll discover that instead of focusing your time and talent inward at the routine operations of infrastructure you’ll have time for those projects that really make a difference to your business.
How Has the Cloud Delivered For You?
Have a story about what has and hasn’t worked with hosting? Drop me a line or post a comment to share it.
