LRDIMMs, RDIMMs, and Supermicro's Latest Twin
TCO
We did a quick cost calculation based on the following assumptions:
- Your applications are memory limited not CPU limited
- A fully equipped system without memory costs about $8000
- You pay 12 cents per kWh
- The real yearly (and not only in the winter) PUE is about two, or if you don't own the datacenter, you pay twice the electricity costs (24 cents per kWh; datacenter clients often pay higher fees for energy costs)
We did a 3-year TCO calculation, without the management costs. Managing half as many physical servers reduces management costs, but it won't be much. Most of the administration costs depends on how many VMs you have to administer, which is the same in both cases.
| Cost | 32GB LRDIMMs | 16GB RDIMMs |
|---|---|---|
| Number of servers | 1 (2 nodes) | 2 (4 nodes) |
| Price per server | 8000 | 8000 |
| Max RAM per server | 1024GB | 512GB |
| Price per DIMM | 1100 (32GB) | 290 (16GB) |
| Number of DIMMs | 32 | 64 |
| Energy per server (kW) | 0.4 | 0.4 |
| Server cost | 8000 | 16000 |
| RAM investment | 35200 | 18560 |
| Energy costs (3 years) | 2523 | 5046 |
| TCO w/o management costs | 45723 | 39606 |
With the current prices, LRDIMMs cannot justify the cost by hardware alone. But the difference is not that large anymore (about 15%). On a 3-year basis, administering half as many servers could definitely reduce the administration costs, but it is very hard to calculate. It could amount to several thousand dollars though, depending on the local labor costs.
If you are paying a software license per server (Ansys comes to mind) and you are memory starved, 32GB LRDIMMs are very attractive. In all other cases, 16GB RDIMMs are most likely the best choice. Unfortunately, VMware decided to make their larger customers pay per vRAM used, so investing in the largest memory available does not reduce licensing costs at first sight when you are running an ESXi cluster. The situation is entirely different with Microsoft's Hyper-V, KVM and Citrix Xen however. This customer unfriendly licensing could convince quite a few people to look the other way. In that case, LRDIMMs allow you to significantly reduce the response time even when the CPU load is high. Our virtualization benchmarking scenario was among the worst to show off the benefits of larger memory capacities and we still noticed up to 2.5 lower response times and 15% lower power consumption.
Back to the hardware. The iMB buffer technology has very little latency impact, so you should not worry about that. The end conclusion is that LRDIMM is a mature technology that offers 50 to 100% more memory capacity or slightly higher bandwidth, but currently at a cost that is about 3 to 4 times higher. If your application is memory limited and you are paying hefty licenses per server, you should definitely consider it. Also, LRDIMM prices are falling. As we type this, industry sources tell us that the price of 32GB LRDIMMs will soon be around $800. If you are in the market for a new high capacity server, keep an eye on the DIMM pricing.
And what about the Supermicro SuperServers? Well, the Twins are typically not considered for virtualization purposes, but now that the latest ones come with 16 slots and 10Gb Ethernet, they become a worthy cost, space, and energy efficient alternative. Our Twin worked flawlessly with the latest ESXi 5, although with 8 DIMM slots (16 DIMMs total), it is not meant to be a virtualization server. We have good hopes that the SuperServer 6027TR-D70RF+ (32 DIMMs total) will be a very affordable alternative to the tier oone OEM servers out there.
Best of all, most of the Twins can be found on VMware's and other Hypervisors's Hardware Compatibility Lists (HCLs). You can definitely save a lot of money as not only the servers themselves are very attractively priced, but you only also benefit from the fact that you do not have to pay a "vendor specific" premium for high capacity memory.
Originally published at https://www.anandtech.com/show/6068/lrdimms-rdimms-supermicros-latest-twin (Wayback Machine).