Archive 1997–2024

The Next Generation Open Compute Hardware: Tried and Tested

The Latest and Greatest: Leopard

Leopard, the latest update to the Windmill motherboard, is equipped with the Intel C226 chipset to support up to two E5-2600v3 Haswell Xeons.

One processor mode is fully supported, in which the CPU can access all RAM onboard. Increased thermal margins (mainly because of upping the chassis height to 2 OU in Winterfell), bigger CPU heatsinks, and better airflow guidance allow the system to receive CPUs with a maximum TDP of 145 Watt, which means you can insert every Xeon except for E5-2687W v3 (160W TDP). Only eight DIMM channels are connected per CPU, but DDR4 allows for a maximum capacity of 128GB per DIMM resulting in a theoretical maximum of 2TB RAM, which Facebook reckons is plenty for years to come. New in this generation is that you can now plug NVDIMM modules (persistent flash storage on a DIMM form factor), which Facebook is testing to see if it can replace PCIe-based add-in cards.

Besides the generational CPU update, other major changes include the removal of the onboard external PCIe connector, support for a mezzanine card with dual QSFP receptacles, a TPM header, the addiction of an mSATA/M.2 slot for SATA/NVMe based storage, and 8 more PCIe lanes routed to the riser card slot for a total of 24. The SAS connector has been removed, as Leopard will not be used as a head node for Knox.


Leopards, with the optional debug board (power/reset buttons and serial-to-USB) plugged in

A big addition to the board is a baseboard management controller (BMC). A simple headless Aspeed AST1250 controller provides traditional Out Of Band IPMI access to query sensor and FRU data, control system power and provide Serial Over Lan. But Facebook taught it some new tricks: to aid bare-metal debugging, it keeps 256 post codes in buffer, offers 128kB of serial console output, and you are able to remotely dump MSR data, which is done automatically after the IERR/MCERR signal is active.

A rather unique feature of the BMC is that it allows you to update the CPLD, VR, BMC and UEFI firmware (basically all the firmware present on the motherboard) remotely, a feature also fully validated by all suppliers of the mentioned components. Another feature that's been added is average power reporting, the BMC keeps a buffer of 600 power measurements, and permits you to query the buffer for a specific interval via IPMI. To improve the accuracy of the power sensor data, factory determined (non)-linear compensations are applied to the measured power usage. Lastly, another unique feature that stems from better rack-level integration is the ability to throttle CPU power usage when power demand in the power zone exceeds capacity – for instance when a PSU dies. When the load increases to the PSU capacity, it executes a quick temporary drop to 1 Volt. This triggers an 'Under Voltage' condition in the servers which in turns activates the Fast Proc Hot signal on the CPUs, causing them to clock down for a certain amount of time and thus decreasing PSU load, allowing it to remain active instead of shutting down.

Originally published at https://www.anandtech.com/show/9138/open-compute-hardware-tried-and-tested (Wayback Machine).