Reliable Systems

The price of memory has continued to go down over the past several years, making it very cost effective to solve many memory utilization problems by purchasing more memory instead of optimizing your application. Consider that if your application runs in 500 MB, it will cost you more to have the developers reduce that footprint down to 256MB than the memory you save. Now, this won’t be an option in the case of a COTS application you’re shipping to customer’s desktops, but it can be very effective in web servers.

Your developers probably already know this - they gave up bothering about optimizing memory until they run into a real problem, it’s all about writing more features faster now, right? Everything’s grand up to a point… the inherit limitations of 32-bit memory space. Most people are familiar with the issue that you basically can’t put more than 4GB of memory in a server under 32-bit (because who’s going to invest in programming for AWE now that 64 bit operating systems are readily available?) and that means each process can only get 4GB of well, even if it’s a 32-bit process on a 64-bit OS. This is where the problem comes in.

In practice, if you see your 32-bit process using more than about 1.5GB of memory then you probably are running it out of RAM. Why? It has to do with two issues: Memory fragmentation and CPU design.

A bit of CPU design and NT history…

Way back in the day, Microsoft decided that Windows NT was going to support multiple processor architectures. It originally shipped on three processor types - DEC Alpha, MIPS, and Intel 386. One more was added for NT 4- PowerPC. These processors all differed in a number of respects, and the NT design team had to make some concessions to commonality. One such concession was how to split up the 32-bit process. Each of these processors reserves some of the address space for instructions and some for data, called the code space and the data space respectively. This is fundamentally part of the design of modern processors for performance and process safety reasons. When the designers of the various chips looked at the problem, most decided that you’d never need more than 1GB for instructions for 3GB of data. The one odd man out was the MIPS - it was designed around an equal sharing of 2GB for each, and it couldn’t switch modes. Accordingly, Windows NT was designed to accommodate the same model, and to make it easier to port code between the various processor types it was decided that every architecture would use the same model.

Practical process limits for 32-bit

Out of the box, a 32-bit process is configured with 2GB for instructions and 2GB for data. Why then should you be worried if you see utilization higher than 1.5GB? Two main reasons:

1. Nearly all of that utilization will be data space, so it counts against 2GB instead of 4GB. I mean really - how large are all of the binaries in your application? That would be the very greatest possible utilization of application space.
2. Things have to fit into memory in contiguous chunks, and can’t be moved once they’re created. If the runtime can’t find a big enough space to meet your request, you’re out of memory.

In practice, it’s the second issue - memory fragmentation - that is going to kill you. In production it’s a big problem for two reasons: First, it’ll cause bizarre low level problems that will report in interesting ways because when it runs out of memory, who knows if it can get enough memory free to nicely report the problem intelligently. The problem tends to be worse in highly abstracted environments (like .NET) because there will be a lot more distance between your code and the raw memory of the system. For example - if you’re using relatively small objects by the time it runs out of enough contiguous memory to create them, will it have enough to create that nice exception object with a stack trace?

Another reason memory fragmentation is very problematic is that it will often take time to show up. For example, your application may be just peachy keen when you run your unit tests against the whole production data set because even though it allocates lots of objects and uses up memory, everything is nice and contiguous. As time goes on and objects are freed and created, you’ll tend to get pockets of memory that get progressively smaller. This particularly happens if your application is casual (and nearly all are) about how it allocates objects that are going to hang around and objects that are very short lived. In short, it is very hard to predict what will finally start causing it to fail to allocate objects. It will also start with a few failures and then progressively escalate as the process continues to be used.

Microsoft has your back with IIS

For web applications, there are some safety features built into IIS that can really help out, and this is why they’re there. You can configure it to automatically reset the process after a certain number of requests, certain number of requests and most importantly maximum used memory. You’ll want to set this to a value less than 1.5GB. It will then nicely halt the request pipeline, restart your process, and resume processing. This will cause a notable delay in processing, but that’s much better than the alternative.

COM+ has similar capabilities.

Services are for serious developers

Sure, anyone can whip one out now with the wizard in Visual Studio, but if you really have your development chops together then you can write a service that will run reliably without the coddling safety net of IIS and COM+. The point isn’t getting your service to run or even execute its functional tests - it’s that it can operate for months at a time doing all of the work it should without leaking memory or fragmenting RAM to the point that the process starts experiencing problems.

If you aren’t sure quite how to get that right, you can use COM+ to cheat a bit - it can similarly automatically reset itself under high memory utilization. This is often going to be a lot easier than the cost of finding a minor memory leak that adds up over time. I’m not advocating that it’s OK to leak memory, just that you may need to pick your battles.

Memory leaks and garbage collectors

If you’ve been reading through the above and thinking smugly I use .NET/Java/VB6/Whatever and it’s garbage collected, we never leak memory! Well, I’ve got news for you: First, you can leak memory if your fancy runtime leaks memory. It makes no difference whether technically it was your code or the runtime in response to your code, if your memory utilization goes up with each request then somebody’s leaking something. Second, while it isn’t fashionable to refer to it as a memory leak if you create an object structure that isn’t cleaned up when you intend it to be, that’s a memory leak too. This was pretty easy in VB6 and somewhat in Java, somewhat harder in .NET because it detects multiple step circular references that are the only reason objects are still around, but the bottom line is that if you aren’t clear in your data structures you can consume more memory with every request, and that’s effectively a memory leak.

Finally, less memory is still better

Keep in mind it takes times to free memory and allocate memory. If the OS has paged parts of your memory to disk (which it will do even if it has free memory to save time when under memory stress) it has to release that as part of freeing up your process. This all hits disk, and disk is much much slower than RAM. Bottom line - controlled, reasonable memory access is the best for ultimate performance.

What’s your experience?

How do you make the design decisions to balance memory usage with caching and other requirements? Post your comments or drop me a line to continue the conversation.