Supermicro's Twin: Two nodes and 16 cores in 1U
- Posted in
- IT Computing
Supermicro 6015T-T/6015T-INF Server
Two models exist : the 6015T-INF and the 6015T-T. The only difference is that the former has an additional InfiniBand connector. Supermicro sent us the 6015T-INF. The Superserver 6015T-INF consists of two Supermicro X7DBT-INF dual processor server boards. Each board is based on Intel's 5000P chipset and support two Intel 53xx/51xx/50xx series processors. Let's take a look at the scheme below.

The Intel 5000p chipset has two I/O units, one with two x8 PCIe lanes and a second one with an x8 + x4 PCIe lane configuration. The first PCIe x8 connection of the first I/O unit is used to attach the 20 Gbit/s InfiniBand connection to the Northbridge (MCH). PCIe x8 offers 2 GB/s full duplex, which roughly matches the peak data bandwidth of the InfiniBand connection. InfiniBand is a point-to-point bidirectional serial interconnect, but that probably doesn't say much. To simplify we could describe it as a technology similar to PCI Express, but one that can also be used externally as a network technology like Ethernet - with the difference that this switched networking technology:
Latency sensitive cluster applications that exchange a lot of (synchronizing) data between the nodes are a prime example of where one should use InfiniBand. A good example is Fluent, a Computational Fluid Dynamics (CFD) package. This article, written by Gilad Shainer, a senior technical marketing manager at Mellanox, shows how using low latency InfiniBand offers much better scaling over multiple nodes than Gigabit Ethernet.

As you can see, with more than 32 cores (or 16 dual core Woodcrest CPUs), Gigabit Ethernet severely limits the performance scaling.
The second PCIe x8 is used as a PCIe slot. PCI-X is not an option due to the limited space available on each X7DB motherboard. So that is one of the limitations of the Twin: there is only one PCIe expansion slot available, which can contain a 2U height card. The PCIe x8 of the second I/O port offers Intel I/O Acceleration Technology (IOAT) to the dual Gigabit Controller. TCP/IP processing is only a small part of the CPU load that is a result of intensive networking activity. Especially with small data blocks, handling the interrupts and signals consumes many more cycles than pure TCP/IP processing.

The X7DBT has eight 240-pin DIMM sockets that can support up to 32 GB of ECC FBD (Fully Buffered DIMM) DDR2-667/533 SDRAM and support memory spares.

All memory modules used to populate the system should be the same size, type and speed.

Each board has four SATA connectors, but only two of them are connected to the storage backplane. That means that in reality, you are limited to two SATA disks per node. This is not a real limitation as this server is not intended to be a fileserver or database server. For a front end or HPC cluster node, two SATA disks in RAID-1 should be sufficient.

As you can see, expansion is limited to one half height PCIe card. A result of fitting two nodes in a 1U server.
Two models exist : the 6015T-INF and the 6015T-T. The only difference is that the former has an additional InfiniBand connector. Supermicro sent us the 6015T-INF. The Superserver 6015T-INF consists of two Supermicro X7DBT-INF dual processor server boards. Each board is based on Intel's 5000P chipset and support two Intel 53xx/51xx/50xx series processors. Let's take a look at the scheme below.

The Intel 5000p chipset has two I/O units, one with two x8 PCIe lanes and a second one with an x8 + x4 PCIe lane configuration. The first PCIe x8 connection of the first I/O unit is used to attach the 20 Gbit/s InfiniBand connection to the Northbridge (MCH). PCIe x8 offers 2 GB/s full duplex, which roughly matches the peak data bandwidth of the InfiniBand connection. InfiniBand is a point-to-point bidirectional serial interconnect, but that probably doesn't say much. To simplify we could describe it as a technology similar to PCI Express, but one that can also be used externally as a network technology like Ethernet - with the difference that this switched networking technology:
- offers more than 10 times the bandwidth of Gigabit Ethernet (1500 MB/s vs. 125 MB/s)
- 15 to 40 times lower latency than Gigabit Ethernet (2.25 µs vs. 100 µs)
- at a ten times lower CPU load (5% versus 50%)
Latency sensitive cluster applications that exchange a lot of (synchronizing) data between the nodes are a prime example of where one should use InfiniBand. A good example is Fluent, a Computational Fluid Dynamics (CFD) package. This article, written by Gilad Shainer, a senior technical marketing manager at Mellanox, shows how using low latency InfiniBand offers much better scaling over multiple nodes than Gigabit Ethernet.

As you can see, with more than 32 cores (or 16 dual core Woodcrest CPUs), Gigabit Ethernet severely limits the performance scaling.
The second PCIe x8 is used as a PCIe slot. PCI-X is not an option due to the limited space available on each X7DB motherboard. So that is one of the limitations of the Twin: there is only one PCIe expansion slot available, which can contain a 2U height card. The PCIe x8 of the second I/O port offers Intel I/O Acceleration Technology (IOAT) to the dual Gigabit Controller. TCP/IP processing is only a small part of the CPU load that is a result of intensive networking activity. Especially with small data blocks, handling the interrupts and signals consumes many more cycles than pure TCP/IP processing.

The X7DBT has eight 240-pin DIMM sockets that can support up to 32 GB of ECC FBD (Fully Buffered DIMM) DDR2-667/533 SDRAM and support memory spares.

All memory modules used to populate the system should be the same size, type and speed.

Each board has four SATA connectors, but only two of them are connected to the storage backplane. That means that in reality, you are limited to two SATA disks per node. This is not a real limitation as this server is not intended to be a fileserver or database server. For a front end or HPC cluster node, two SATA disks in RAID-1 should be sufficient.

As you can see, expansion is limited to one half height PCIe card. A result of fitting two nodes in a 1U server.
Originally published at https://www.anandtech.com/show/2240 (Wayback Machine).