A Messy Transition: Practical Problems With 32bit Addressing In Windows
by Ryan Smith on July 12, 2007 12:00 PM EST- Posted in
- Software
As we stated earlier, the inspiration behind this article is Supreme Commander, which was crashing during gameplay due to what we discovered to be was the game exhausting its user space. Without taking extreme and convoluted measures, most applications can do little besides (gracefully) crash when they've exhausted their user space. Server applications, which on the whole tend to be the king of memory hogs in the first place, tend to have some sort of compensation mechanism, but for more generalized applications like games this is not the case.
In this case, Supreme Commander was crashing late in to big games on a system that otherwise is completely stable. No amount of testing could come up with a problem outside of Supreme Commander, and most telltale of all was that it was only a problem with large maps with large number of players with high quality settings; turning anything down made the crashing stop. Although not clearly a memory error it greatly reduces the list of possible problems to a few things. The entire process, we suspect, will be similar for everyone hitting the 2GB barrier: the affected application is crashing under heavy load. Thankfully the problem is possible to debug, albeit with moderate difficulty.
With all of that said, I did not come up with the diagnosis or the solution myself and while the problem in retrospect is quite obvious, at the time I did not even take the 2GB barrier in to consideration. As such, credit for the diagnosis and solution belongs to MadBoris of the Gas Powered Games forums who identified Supreme Commander as hitting the 2GB barrier and publishing a solution for it, which we will cover here.
Getting to the root of the problem requires additional tools beyond what comes with Windows XP or Vista. The task manger, the logical tool for this task, doesn't track the information we need to know. At best, the task manager can list how much data is in the various memory pools altogether, however this is not the metric we're looking for. What we actually need to know is how much of the application's share of the virtual address space has been allocated (the fundamental problem after all is when we run out of unused virtual address space to allocate) which due to a multitude of reasons can be much greater than the amount of memory in use by an application. For finding this we'll need Sysinternals' Process Explorer.
The above screenshot was taken as this paragraph was written, showcasing several different memory values for the running processes. The WS Private (Working Set Private) column lists how much physical memory the application is using, and this is the same column listed by the Windows Task Manager for memory usage. The Private Bytes column is the total amount of memory the application is using. Last but not least however is the Virtual Size column, which finally lists the amount of virtual address space allocated, the metric we're after.
We'll quickly focus on Microsoft Word as an example of the discrepancy between physical memory usage and private size. While Word is only using 17MB of physical memory, its virtual size is 170MB, 10 times the physical memory usage and 9 times the total memory usage. This is more or less a worse case scenario but it also points out just how misleading any memory usage - physical or total - is when trying to diagnose this kind of problem. Applications or games with high memory usage tend to not have nearly as a severe spread, but as we can see it's possible for an application to hit the 2GB barrier well before it needs 2GB of physical memory.
Getting to the meat of things however, the process readout for Supreme Commander basically says everything that needs to be said. In order to prove an exaggerated point, we started a 6 player game with 5 AI units on a maximum size map (81km x 81km) and set the game speed to maximum with maximum quality settings, while patching Supreme Commander and adjusting our system configuration to break the 2GB barrier. This actually isn't very playable for a variety of reasons (we'll overtax the CPU quickly due to all the AI units, slowing the game down dramatically) but represents not only what can happen in bad situations, but also is something that happens even in more manageable situations on smaller maps later in to the game.
Just at the start of the game, our virtual size was already in excess of 1.6GB, dangerously close to the 2GB barrier. 16 minutes in to the game, we have already hit a virtual size of 2.2GB, meaning had we not broken the 2GB barrier the game would have already crashed. At this point our total memory usage (private bytes) is only at 1.7GB, with 1.4GB of that in physical memory, nearly maximizing the RAM usage of our 2GB system. Our spread between memory usage and virtual size is .5GB, showcasing how deceptive memory usage is in identifying 2GB barrier issues.
Finally at 22 minutes in to the game, the game crashes as the virtual size has reached the 2.6GB barrier we have reconfigured this system for. Perhaps the most troubling thing at this point is that Supreme Commander is not aware that it ran out of user space in its virtual address pool, as we are kicked out of the game with a generic error message. Unfortunately Windows Vista reverted to a non-accelerated desktop at this point, preventing us from capturing a screenshot of the exact memory readouts or the error message.
Facebook
Twitter
RSS
69 Comments
View All Comments
instant - Saturday, July 21, 2007 - link
Could the writer please detail what, exactly, is wrong with XP-64bit edition?Having run it since it was released and never had a problem with it, I would be interested in knowing what problems I should have experienced.
BUL - Tuesday, July 17, 2007 - link
A few things I've found about /3GB in XP...1) In terms of "consumer" OS's, XP Pro is the only one to support it. (XP Home and W2K Pro DO NOT!) Vista has a different method, outlined in the article.
2) If you change boot.ini, make sure you add a "3GB" boot option to your operating systems, not just change the one entry to /3GB. If a new driver gets installed that doesn't like /3GB, you could be left without the ability to change boot.ini back. (I can't find any info on whether /3GB affects Safe Mode, however safe mode may run in "3GB" mode!) So, unless you have the ability to WRITE to NTFS partitions outside of Windows, you may be stuck if you don't. See http://www.gehrytechnologies.com/catia/catia/catia...">http://www.gehrytechnologies.com/catia/catia/catia... for how to do it...
Now, personally, I feel that a limitation like memory shows the "men from the boys" in terms of programmers. Windows and "endless memory" has allowed for bloatware for far too long and allowed for sloppy, inefficient programming... Just think back how many games & other programs could run on an XT with 640KB of RAM because the programmers wrote highly efficient code. (Lotus 1-2-3 for DOS was written primarily in assembler; Quattro Pro could open even larger spreadsheets than 1-2-3 because it utilized pseudo-virtual memory).
Now we've come full-circle into the world of Windows, and some of these programs now have to become more efficient. That's what programmers are paid to do...
Anorax - Monday, July 16, 2007 - link
On some of the larger maps in AW2 the game will occasional crash to desktop and leave an error window saying it has exceeded the 2GB memory space. Not that helpful but at least it tells you why it crashed.the Chase - Sunday, July 15, 2007 - link
Thank you so much for this article. I had a post awhile back in the forums asking for help on why in Vista I couldn't even run PR for bf2 unless I turned down the texture setting to Medium.Never figured it out though people here did offer some good ideas for fixes. After reading this article I knew this is what was going on. I searched on the Net and found out how to use Visual Studio C++ to modify the Bf2 executable. Problem solved! (I have the 64 bit version of Vista).
And to think I was ready(thinkeng about anyway) going to buy another 2 GB's of memory to try and cure the problem. Thanks again.
P.S. I also did the same fix for XP pro 64 and now PR runs much smoother than before in that OS also.
spookware - Friday, July 13, 2007 - link
Just wanted to mention that the description of how windows uses memory is correct for XP and previous versions but it is not quite how windows vista works.Specifically, vista does not automatically take the upper 2GB of adress space for the kernel any more. It instead grows the kernel adress space downard on demand. Allthough the associated switches (/3G) have the same prupose in vista what they are setting is actually the upper maximum for the kernel adress space.
Magumi - Friday, July 13, 2007 - link
I guess this means that if I want to buy a computer to last me at least three or four years, I need to get 64bit Vista and 8GB of RAM and hope that drivers and incompatibilities get resolved soon.Greyhead - Friday, July 13, 2007 - link
Ryan, great article. Some other posters mentioned the use of the /PAE on servers. I recently discovered that MS recommends NOT using the /3GB switch in conjunction with /PAE. I followed their advice on a large SQL server we have and saw immediate positive results. The problem with /PAE /3GB combination is that when the OS is limited to 1 GB, there is not enough room for the size of the heap needed to support the /PAE option! This can be viewed using performance monitor - selecting "memory" options and then viewing total PTEs available (Page Table Entries). There are MS articles that describe the minimum PTEs needed, and before I changed our server it was way under the minimum. We had stupid errors on the server - blue screens, "not enough memory" errors when transferring files to another machine. Once the change was made, these problems disappeared. 2003 server has a more tunable /3GB switch using the /USERVA switch. There are technet articles which provide guidance on it's usage.Keep up the great work -
-bill
redpriest_ - Friday, July 13, 2007 - link
binary is illegal.Kougar - Friday, July 13, 2007 - link
The moment I saw the title I was thinking of Supreme Commander. I particularly enjoy the insanely huge 8-player games, but it only took about 40 minutes before these would crash... oddly enough they did not always crash either in some situations, making the crashing appear random and only confusing my attempts to troubleshoot this game. It would have been quite appreciated that this issue would have at least been mentioned in the game's readme file, if nothing else.Having (very luckily) stumbled across MadBorris's thread I made the modifications to XP and SC has been running since. I have not run into any instability or issues with XP configured with the /3gb switch for what it is worth. Am I wrong in that users with video cards featuring smaller onboard memory sizes will have an "advantage" with this problem? There is a large difference between a 320mb GTS and 740MB GTX, or heaven forbid a 1GB HD 2900 card? And while on the subject does a dual-GPU configuration (and therefore dual VGA memory) make things even worse?
Ryan Smith - Friday, July 13, 2007 - link
Yes to the first question. To the second question I believe that's also a yes, but I don't have a system configured to test that theory.