Introduction

TCO and ROI have been abused repeatedly by sales representatives, in the hope of getting you to swallow the sometimes outrageously high pricing on their quotation for a trendy new technology. However, server virtualization is one of the few ICT technologies that really lives up to its hype. The cost savings are real and the TCO is great, as long as you obey a few basic rules like not installing bloatware or extremely large memory limited databases. There is more.

Server consolidation is superb for the IT professional who is also a hardware enthusiast (and thus reads it.anandtech.com ?). Hardware purchases used to be motivated by the fact that the equipment was written off or because the maintenance contract was at the end of its life. Can you even think of a more boring reason to buy new hardware? The timeframe between the beginning of the 21st century and the start of commercially viable virtualization solutions was the timeframe where the bean counters ruled the datacenter. Few people were interested in hearing how much faster the newest servers were, as in most cases the extra processing power would go to waste 95% of the time anyway.

Now with virtualization, we hardware nerds are back with a vengeance. Every drop of performance you wring out of your servers translates into potentially higher consolidation ratios (more VMs per physical machine) or better response time per VM. More VMs per machine means immediate short- and long-term cost savings, and better performance per VM means happier users. Yes, performance matters once again and system administrators are seen as key persons, vital to accomplishing the business goals. But how do you know what hardware you should buy for virtualization? There are only two consolidation benchmarks out there: Intel's vConsolidate and VMware's VMmark. Both are cumbersome to set up and both are based on industry benchmarks (SPECJbb2005) that are only somewhat or even hardly representative of real-world applications. The result is that VMmark, despite the fact that it is a valuable benchmark, has turned into yet another OEM benchmark(et)ing tool. The only goal of the OEMs seems to be to produce scores as high as possible; that is understandable from their point of view, but not very useful for the IT professional. Without an analysis of where the extra performance comes from, the scores give a quick first impression but nothing more.

Yes, this article is long overdue, but the Sizing Servers Lab proudly presents the AnandTech readers with our newest virtualization benchmark, vApus Mark I, which uses real-world applications in a Windows Server Consolidation scenario. Our goal is certainly not to replace nor to discredit VMmark, but rather to give you another data point -- an independent second opinion based on solid benchmarking. Combining our own testing with what we find on the VMmark page, we will be able to understand the virtualization performance landscape a bit better. Before we dive into the results, let's discuss the reasoning behind some of the choices we made.

The Virtualization Benchmarking Chaos
POST A COMMENT

66 Comments

View All Comments

  • JohanAnandtech - Friday, May 22, 2009 - link

    Most of the time, the number of sessions on TS are limited by the amount of memory. Can you give some insight in what you are running inside a session? If it is light on CPU or I/O resources, sizing will be based on the amount of memory per session only. Reply
  • dragunover - Thursday, May 21, 2009 - link

    would be interesting if this was done on desktop CPU's with price / performance ratios Reply
  • jmke - Thursday, May 21, 2009 - link

    nope, that would not be interesting at all. You don't want desktop motherboards, RAM or CPUs in your server room;
    nor do you run ESX at home. So there's no point to test performance of desktop CPUs.
    Reply
  • simtex - Thursday, May 21, 2009 - link

    Why so harsh, virtualization will eventually become a part of desktops users everyday life.

    Imagine, tabbing between different virtualization, like you do in your browser. You might have a secure virtualization for your webapplications, a fast virtualization for your games. Another for streaming music and maybe capturing television. All on one computer, which you seldom have to reboot because everything runs virtualized.
    Reply
  • Azsen - Monday, May 25, 2009 - link

    Why would you run all those applications on your desktop in VMs? Surely they would just be separate application processes running under the one OS. Reply
  • flipmode - Thursday, May 21, 2009 - link

    Speaking from the perspective of how the article can be the most valuable, it is definitely better off to stick to true server hardware for the time being.

    For desktop users, it is a curiosity that "may eventually" impart some useful data. The tests are immediately valuable for servers and for current server hardware. They are merely of academic curiosity for desktop users on hardware that will be outdated by the time virtualization truly becomes a mainstream scenario on the desktop.

    And I do not think he was being harsh, I think he was just being as brief as possible.
    Reply

Log in

Don't have an account? Sign up now