Archive 1997–2024

Supermicro's Twin: Two nodes and 16 cores in 1U

APUS: a new way of benchmarking

Using industry benchmarks and full disclosure of testing methods is important, as it allows other people to verify your numbers. Therefore we have included a benchmark that everyone can verify (3DS Max architecture rendering). However, testing with real heavy duty applications is so much more interesting. These are the applications that are running in the datacenters, not SPECjbb or TPC. People really care about the performance, as when the applications start to crawl, it affects their work and business. Therefore, let me introduce you to our completely new way of benchmarking with our unique software, called APUS.

As many of our loyal readers know, quite a few of the IT-based articles are based on a close collaboration with the server lab of the College University of West-Flanders. While the academic server research is beyond the scope of this article, the main advantage is that we get the opportunity to work with IT firms which develop rather interesting server applications.

Of course testing these applications is rather complex: how can you recreate the most intensive parts of the real world use of an application and get at the same time a very repeatable and reliable real world benchmark? The answer lies in software developed at the same university called APUS. APUS (or Application Unique Stress-testing) allows us to use the logs of real world applications and turn them into a repeatable benchmark. The application is able to read the logs of almost every popular relational Database (DB2, Oracle, Sybase, MySQL, SQL server, PostGreSQL...) and web applications out there.

We are also working on "special protocols" so we can also use this benchmarking method for other socket applications. A highly tuned threading system allows us to simulate a few thousand users (and more) on one dual core portable. Complex setups with tens of clients are not necessary at all.


There is more: thanks to the hard work of Leandro, Ben, Brecht, and Dieter, APUS is much more than the typical stress tests you can find all over the web. It integrates:
  • the monitoring of many important client parameters
  • and server parameters
  • performance measurements
  • and is able to capture the corresponding power consumption in real time, making use of the EXTECH 380801
Integrating client monitoring with performance measurements allows us to detect when our client CPU or network is becoming the bottleneck. Server monitoring, possible on both Linux and Windows platforms, shows us what has been happening on the server during a particular benchmark session.

You may expect us to test with several real world applications in the future, but for now, let me introduce you to our first "as real world as it can get" application: MCS eFMS.

MCS eFMS software suite

One of the very interesting and more processing intensive applications that we encountered was developed by MCS. The MCS Enterprise Facility Management Solutions (MCS eFMS) is a state-of-the-art Facility Management Information System (FMIS). It includes applications such as space management (buildings), asset management (furniture, IT equipment, etc.), helpdesk, cable management, maintenance, meeting room reservations, financial management, reporting, and many more. MCS eFMS stores all information in a central Oracle database.

MCS eFMS integrates space management, meeting room reservations and much more
Click to enlarge

What makes the application interesting to us as IT researchers is the integration of three key technologies:
  • A web-based front end (IIS + PHP)
  • Integrated CAD drawings
  • Gets its information from a rather complex, ERP-like Oracle database.
The application allows users to view unfoldable location trees that display all buildings, floors, rooms, etc. including detailed floor maps. It can also provide an overview of all available meeting rooms and bookings. In practice, MCS eFMS is one of the most demanding web applications we have encountered so far.

It uses the following software:
  • Microsoft IIS 6.0 (Windows 2003 Server Standard Edition R2)
  • PHP 4.4.0
  • FastCGI
  • Oracle 9.2
The next version of MCS eFMS works with PHP 5 and Oracle 10. MCS eFMS is used daily by large international companies such as Siemens, Ernst & Young, ISS, and PricewaterhouseCoopers, which makes testing this application even more attractive. We used the specially developed APUS (Application Unique Stress-testing) software, developed by our own lab to analyze the logs we got from MCS and turn these logs into a demanding stress test. The result is a benchmark which closely models the way users access MCS web servers around the world.

The client, database server, and web server are connected via the same D-link Gigabit Switch. All cable connections worked at 1 Gbit/s full duplex


The only difference is that the hundreds of simulated users access the web server over one Gigabit Ethernet connection, while in the real world people access the MCS web applications over internal LANs as well over different WAN connections. As some of the pages easily take 400 to 800 ms (and higher under heavy load!) between receiving and sending the request, the few milliseconds that a good internet connection adds will not be significant.

MCS received a detailed and accurate model of how the web server and the clients react under different loads, large user groups or heavy pages, which enables them to optimize the MCS eFMS suite even more. Let us see the most interesting results of that report.

Originally published at https://www.anandtech.com/show/2240 (Wayback Machine).