Archive 1997–2024

Real-world virtualization benchmarking: the best server CPUs compared

vApus Mark I vs. VMmark

By now, it should be clear that vApus Mark I is not meant to replace VMmark or VConsolidate. The largest difference is that VMmark for example tries to mimic the "average" virtualized datacenter, while vApus Mark I clearly focuses on the heavier service oriented applications. vApus Mark I focuses on a smaller part of the market, while the creators of VMmark have invested a lot of thought into getting a nice mix of the typical applications found in a datacenter. We have listed the most important differences below.

vApus Mark I compared to VMmark
  vApus Mark I VMmark
Goal Virtualization benchmarking across Guest OS, Hypervisor, and Hardware Measuring what the best hardware is for ESX
Reproducible by third parties No; for now it's only available to AnandTech and Sizing Server Lab Yes
Modeling "Harder to virtualize", "heavy duty" applications A balanced mix of virtualized applications in the "typical" datacenter
VMs Large "heavy duty" VMs; 4GB with 4 VCPUs Small VMs 0.5-2GB, 1-2 VCPUs
Market coverage Small but important part of the market Large part of datacenter market
Relevance to the real-world Uses real-world applications Uses industry standard benchmarks

The advantages of vApus Mark I are the fact that we use real-world applications and test them as if they are loaded by real people. The advantages of VMmark are that it is available to everyone and it has a mix of applications that is closer to what is found in the majority of datacenters. vApus Mark I focuses more on heavy duty applications.

There's one small difference between the existing benchmarks like VMmark and VConsolidate and our "vApus Mark I" virtual test. VMmark and VConsolidate add additional groups of VMs (called tiles or CSUs) until the benchmark score does not increase anymore, or until all the system processors are fully utilized. Our virtualization benchmark tries to get close to 100% CPU load much quicker. This is a result of the fact that our VMs require relatively large amounts of memory: each VM needs 4GB. If we used a throttled load such as VMmark or VConsolidate, we would require massive amounts of memory to measure servers with 16 cores and more. Six VMs that make up a tile in VMmark take only 5GB, while our four VMs require 16GB. Our current monitoring shows that this benchmark could run in 10-11GB, and thanks to VMware's shared memory technique probably less than 9GB. With four VMs we can test up to 12 physical CPUs, or 16 logical CPUs (8 Physical + 8 SMT). We need eight VMs (or two "tiles") to fully stress 16 to 24 physical cores.

Originally published at https://www.anandtech.com/show/2770 (Wayback Machine).