A Messy Transition: Practical Problems With 32bit Addressing In Windows
by Ryan Smith on July 12, 2007 12:00 PM EST- Posted in
- Software
As it turns out, it's possible and actually quite easy to move the 2GB barrier by increasing the size of the user space, but at the cost of reducing the size of the kernel space. Under Windows XP, this is the fabled "/3gb" switch for boot.ini, and for Windows Vista it's the "IncreaseUserVa" option in BCDedit. By using these options applications can use more than 2GB of virtual address space (generally up to 3GB), and ideally this would be the end of the article.
Unfortunately this is not the case as there are problems on both the application and kernel side of things. On the application side, a common poor programming practice has been to always assume that an application will only be dealing with 2GB of user space; code that makes this assumption will likely error if more than 2GB of user space is actually available. This is avoidable by following proper programming practices, but as a safety precaution even with additional virtual address space allocated to user space Windows still defaults to limiting an application to 2GB. Only finally, if an application indicates to Windows that it is capable of handling more than 2GB, via the "/LARGEADDRESSAWARE" flag, may it have access to any space above 2GB.
As for the kernel, having had up to half of its space taken away must now find a way to live in a smaller space. The (in)ability of any specific system/Windows configuration to deal with this is why the 3gb switch is considered dangerous, seldom recommended, and just generally a bad idea. The biggest culprit here is drivers that run in kernel space. Like applications, they may assume that there's an entire 2GB of address space to work with, except unlike applications this space gets smaller instead of bigger.
Windows' own memory needs can also cause problems with the reduced kernel space. As we mentioned before, space is required for the kernel to do a multitude of things, if a lot of space is required - video cards with a lot of memory are a particular offender here - then everything needing space may not fit in the kernel space. Because there are no strong safeguards against these conditions it may cause a failure to boot or system instability, especially if the culprit is a driver that is well enough behaved to boot. Many modern drivers from hardware vendors that deal with enterprise-level hardware are capable of handling this, many more consumer hardware drivers are not. Stability concerns are the number one reason that breaking the 2GB barrier on a 32bit version of Windows is not recommended.
There is also a second concern however: performance. While an individual application may benefit from more user space in which to work, the kernel now has less space to cache data (as non-obvious as this may seem given all the addresses are virtual) and this can in theory hurt performance. This condition is something we will take a look at in a bit.
Facebook
Twitter
RSS
69 Comments
View All Comments
miahallen - Saturday, July 14, 2007 - link
What I found extremely interesting is that you had problems modifying your boot.ini file for a lsrger than 2.6GB app space. I have an almost identical configuration, and have modded almost all my games headers, and my boot.ini is set to 3GB app space. When I modded the boot.ini, I was unaware of the possible problems, but since it didn't cause any, I've been perfectly happy with it.A8N-SLI Delux
Opty 165 @ 2.5GHz
2GB DDR400
8800GTX (768MB VRAM)
Vista x86
I've modded the headers for:
STALKER
C&C3
CoH
Dark Messiah M&M
DiRT
FSX
Silent Hunter 4
TDU
Thanks Ryan, for the great article!
MadBoris - Sunday, July 15, 2007 - link
This should be made clear to people.
If an application is not Large Address Aware there is usually very good reason the developers did not include it that way. There maybe things you break in the game or cause stability problems by adding it.
So people should not be just adding this to all their games, especially considering many games don't come close to the 2GB ceiling in the first place, so their is no benefit only potential negatives.
ChristopherO - Monday, July 16, 2007 - link
This is true... One shouldn't go setting the binary flag because they *can*.I will however confirm that C&C3 and FSX run into the 2GB barrier. FSX is fine once it is patched.
However, C&C3 is so poorly written that it can easily pass 2GB and run into the revised 3GB barrier on any multiplayer map with the maximum number of AI and/or opponents. Detail settings as low as the "medium" preset do nothing to alleviate this problem.
Unfortunately C&C apparently has absolutely no code to purge and re-use memory as I've been in huge games, on the downward slope in terms of remaning opponents/units, and the app still hits a revised 3GB memory space and crashes.
As a developer myself, memory management is *not* that complicated. It takes some forethought, but the EA people apparently never bothered. This is a huge letdown since I've been a C&C fan since the first game came out my first year of college.
miahallen - Saturday, July 14, 2007 - link
I have to add one more thing about my system, may be important.....most of my games I play at 2048x1536, as opposed to you test rig at 1600x1200. That and you were using Vista Ultimate, I'm using Home Premium. Do you think this is what is effecting the results? If so, Home Premium (or even Home Basic) is the x86 OS of choice for gamers, not Ultimate!miahallen - Sunday, July 15, 2007 - link
I was hoping you would comment on this Ryan, do you think the difference in our boot.ini mod was due to the version of Vista we run? Any thoughts?sheh - Thursday, July 12, 2007 - link
...and disturbingly it doesn't set off any sort of multiplayer cheat detection in the game in spite of the fact that we have modified the executable in a very visible way.The game probably checks the code (and maybe data) section(s) in memory, and not the actual EXE file (makes sense considering you can use memory patchers). The header might not be important for cheat prevention.
MadBoris - Thursday, July 12, 2007 - link
Yeah, I remember thinking that the first day I did it, actually in those days it was Securom protected which I was actually more suprised about bypassing. But seriously the exe header should not be something that cheat protection should look for. I can only say glad it didn't or crashing would be a constant problem. ;)
atlr - Thursday, July 12, 2007 - link
I thoroughly enjoyed this article. Good job.atlr - Thursday, July 12, 2007 - link
A lot of good comments have helped edit and tune this article too. (meh, not this one though) Yayyy, community.JetBlack69 - Thursday, July 12, 2007 - link
From a programming perspective, what is a programmer to do? I assume this can happen in a program when "new" returns NULL because the program is out of memory, but what can a program do gracefully?I assume it could just display an error message, but if it were during a game, how could it handle it gracefully and not crash or give an error message? Would it lower the texture detail or remove unneeded objects on the screen?