Server Buying Decisions: Memory
by Johan De Gelas on December 19, 2013 10:00 AM EST- Posted in
- Enterprise
- Memory
- IT Computing
- Cloud Computing
- server
Measuring Stream Throughput
Before we start with the real world tests, it is good to perform a few low level benchmarks. First, we measured the bandwidth in Linux. The binary was compiled with the Open64 compiler 5.0 (Opencc). It is a multi-threaded, OpenMP based, 64-bit binary. The following compiler switches were used:
-Ofast -mp -ipa
The results are expressed in GB per second. Note that we also tested with gcc 4.8.1 and compiler options
-O3 –fopenmp –static
Results were consistently 20 to 30% lower with gcc. So we feel our choice for Open64 is appropriate: everybody can reproduce our results (Open64 is freely available) and as the binary is capable of reaching higher speeds, it is easier to spot speed differences between DIMMs. We equipped the HP with a Sandy Bridge EP Xeon (Xeon 2690) and an Ivy Bridge EP Xeon (Xeon 2680v2). Note that the Stream benchmark is not limited by the CPUs at all. All tests were done with 32 or 40 threads.
The extra buffering inside the LR-DIMMs has very little impact on the effective bandwidth. RDIMMs deliver only 3% more bandwidth at 1866 MHz, 1DPC. This bandwidth gap is 0 when we run the same test on our "Sandy Bridge EP" Xeon.
At 3DPC, there is no bandwidth gap at all. Both DIMMs are running at the same speed. Also note that the newer Xeon outperforms the older one by 8 to 33% in this test.
Facebook
Twitter
RSS
27 Comments
View All Comments
JohanAnandtech - Friday, December 20, 2013 - link
First of all, if your workload is read intensive, more RAM will almost always be much faster than any flash cache. Secondly, it greatly depends on your storage vendor whether adding more flash can be done at "dramatically lower cost". The tier-one vendors still charge an arm and a leg for flash cache, while the server vendors are working at much more competitive prices. I would say that in general it is cheaper and more efficient to optimize RAM caching versus optimizing your storage (unless your are write limited).blaktron - Friday, December 20, 2013 - link
Not only are you correct, but significantly so. Enterprise flash storage at decent densities is more costly PER GIG than DDR3. Not only that, but you need the 'cadillac' model SANs to support more than 2 SSDs. Not to mention fabric management is a lot more resource intensive and more prone to error.Right now, the best bet (like always) to get performance is to stuff your servers with memory and distribute your workload. Because its poor network architecture that creates bottlenecks in any environment where you need to stuff more than 256GB of RAM into a single box.
hoboville - Friday, December 20, 2013 - link
Another thing about HPC is that, as long as a processor has: enough RAM to do its dataset on the CPU/GPU before it needs more data, the quantity of RAM is enough. Saving on RAM can let you buy more nodes, which gives you more performance capacity.markhahn - Saturday, January 4, 2014 - link
headline should have been: if you're serving static content, your main goal is to maximize ram per node. not exactly a shocker eh? in the real world, at least the HPC corner of it, 1G/core is pretty common, and 32G/core is absurd. hence, udimms are actually a good choice sometimes.mr map - Monday, January 20, 2014 - link
Very interesting article, Johan!I would very much like to know what specific memory model (brand, model number) you are referring to regarding the 32GB LRDIMM—1866 option.
I have searched at no avail.
Johan? / Anyone?
Thank you in advance!
/ Tomas
Gasaraki88 - Thursday, January 30, 2014 - link
A great article as always.ShirleyBurnell - Tuesday, November 5, 2019 - link
I don't know why people are still going after server hardware. I mean it's the 21st century. Now everything is on cloud. Where you have the ability to scale your server anytime you want to. I mean the hosting provider companies like: AWS, DigitalOcean, Vultr hosting https://www.cloudways.com/en/vultr-hosting.php, etc. has made it very easy to rent your server.