Futuremark: of Benchmarks and Gaming
- Posted in
- Software
What's in a benchmark?
At their core, modern 3D games consist of a graphics engine designed to spit out polygons onto your screen, and the internal workings of these engines can vary wildly from game to game. To discount the 3DMark graphics engines as "synthetic" is missing the mark; they are no more synthetic than any other gaming engine. The problem of course is that while other gaming engines are actually used in — at the very least — one game, 3DMark exists as its own entity.
What do performance benchmarks using The Witcher, Unreal Tournament 3, Crysis, or any other title really tell us? They tell us how specific hardware happens to perform in that title; no more and no less. Sure, gaming engines are often used in multiple titles, but once the developers are done modding and tweaking things, there's no guarantee that performance of the engine in one title will be the same as what we find in another title. Consider Gears of War, Bioshock, and Unreal Tournament 3 as an example; generally speaking, performance is similar but rarely identical.
![]() |
![]() |
![]() |
Things are further complicated when you consider that a benchmark of one specific area of a game/game engine doesn't tell you much about how hardware might perform in other areas. Take Oblivion as an example: performance in towns, dungeons, outdoor areas, underwater, and the gates can and does differ; if you want a truly comprehensive look at Oblivion performance, you would need to benchmark representative selections of all those environments. That's precisely what we did when Oblivion first launched, and we found that some while the GPU was clearly the bottleneck in outdoor environments, other areas demanded more of the CPU.
But is that really necessary? Most people are really only concerned with worst-case performance; you might be able to get 80 FPS or more inside a town, but if you drop to 15 FPS in outdoor environments it is unlikely you will be happy with the overall experience. We can substitute any other title in place of Oblivion — Crysis for example has some seriously taxing levels towards the end of the game, where frame rates might be less than one fourth of what you experience at the beginning of the game. The net result is that any benchmark is only a snapshot of performance.
When we look at performance in a variety of benchmarks, we do so in order that we might gain an overall impression of how hardware actually performs. History helps explain why we do this. Which is the faster architecture: Pentium 4 or Athlon XP? The answer: it depends on the application. Which GPU architecture is better: G70 or R500? Again, it depends what game you're playing. It makes our job easy when we get hardware that is clearly faster in virtually every situation, but that rarely happens. Driver and compiler optimizations can even help turn a mutt into a purebred — sort of like DNA tampering.
Originally published at https://www.anandtech.com/show/2430 (Wayback Machine).


