The 2017 Benchmark Suite

For our review, we are implementing our fresh CPU testing benchmark suite, using new scripts developed specifically for this testing. This means that with a fresh OS install, we can configure the OS to be more consistent, install the new benchmarks, maintain version consistency without random updates and start running the tests in under 5 minutes. After that it's a one button press to start an 8-10hr test (with a high-performance core) with nearly 100 relevant data points in the benchmarks given below for CPUs, followed by our CPU gaming tests which run for 4-5 hours for each of the GPUs used. The CPU tests cover a wide range of segments, some of which will be familiar but some of the tests are new to benchmarking in general, but still highly relevant for the markets they come from.

Our new CPU tests go through six main areas. We cover the Web (we've got an un-updateable version of Chrome 56), general system tests (opening tricky PDFs, emulation, brain simulation, AI, 2D image to 3D model conversion), rendering (ray tracing, modeling), encoding (compression, AES, h264 and HEVC), office based tests (PCMark and others), and our legacy tests, throwbacks from another generation of bad code but interesting to compare.

All of our benchmark results can also be found in our benchmark engine, Bench.

A side note on OS preparation. As we're using Windows 10, there's a large opportunity for something to come in and disrupt our testing. So our default strategy is multiple: disable the ability to update as much as possible, disable Windows Defender, uninstall OneDrive, disable Cortana as much as possible, implement the high performance mode in the power options, and disable the internal platform clock which can drift away from being accurate if the base frequency drifts (and thus the timing ends up inaccurate).

Web Tests on Chrome 56

Sunspider 1.0.2
Mozilla Kraken 1.1
Google Octane 2.0

System Tests

PDF Opening
3DPM v2.1
Dolphin v5.0
DigiCortex v1.20
Agisoft PhotoScan v1.0

Rendering Tests

Corona 1.3
Blender 2.78
LuxMark v3.1 CPU C++
LuxMark v3.1 CPU OpenCL
POV-Ray 3.7.1b4
Cinebench R15 ST
Cinebench R15 MT

Encoding Tests

7-Zip 9.2
WinRAR 5.40
AES Encoding (TrueCrypt 7.2)
HandBrake v1.0.2 x264 LQ
HandBrake v1.0.2 x264-HQ
HandBrake v1.0.2 HEVC-4K

Office / Professional

Chromium Compile (v56)

Legacy Tests

3DPM v1 ST / MT
x264 HD 3 Pass 1, Pass 2
Cinebench R11.5 ST / MT
Cinebench R10 ST / MT

CPU Gaming Tests

For our new set of GPU tests, we wanted to think big. There are a lot of users in the ecosystem that prioritize gaming above all else, especially when it comes to choosing the correct CPU. If there's a chance to save $50 and get a better graphics card for no loss in performance, then this is the route that gamers would prefer to tread. The angle here though is tough - lots of games have different requirements and cause different stresses on a system, with various graphics cards having different reactions to the code flow of a game. Then users also have different resolutions and different perceptions of what feels 'normal'. This all amounts to more degrees of freedom than we could hope to test in a lifetime, only for the data to become irrelevant in a few months when a new game or new GPU comes into the mix. Just for good measure, let us add in DirectX 12 titles that make it easier to use more CPU cores in a game to enhance fidelity.

Our original list of nine games planned in February quickly became six, due to the lack of professional-grade controls on Ubisoft titles. If you want to see For Honor, Steep or Ghost Recon: Wildlands benchmarked on AnandTech, please point Ubisoft Annecy or Ubisoft Montreal in my direction. While these games have in-game benchmarks worth using, unfortunately they do not provide enough frame-by-frame detail to the end user, despite using it internally to produce the data the user eventually sees (and it typically ends up obfuscated by another layer as well). I would instead perhaps choose to automate these benchmarks via inputs, however the extremely variable loading time is a strong barrier to this.

So we have the following benchmarks as part of our 4/2 script, automated to the point of a one-button run and out pops the results four hours later, per GPU. Also listed are the resolutions and settings used.

  • Civilization 6 (1080p Ultra, 4K Ultra)
  • Ashes of the Singularity: Escalation* (1080p Extreme, 4K Extreme)
  • Shadow of Mordor (1080p Ultra, 4K Ultra)
  • Rise of the Tomb Raider #1 - GeoValley (1080p High, 4K Medium)
  • Rise of the Tomb Raider #2 - Prophets (1080p High, 4K Medium)
  • Rise of the Tomb Raider #3 - Mountain (1080p High, 4K Medium)
  • Rocket League (1080p Ultra, 4K Ultra)
  • Grand Theft Auto V (1080p Very High, 4K High)

For each of the GPUs in our testing, these games (at each resolution/setting combination) are run four times each, with outliers discarded. Average frame rates, 99th percentiles and 'Time Under x FPS' data is sorted, and the raw data is archived.

The four GPUs we've managed to obtain for these tests are:

  • MSI GTX 1080 Gaming X 8G
  • ASUS GTX 1060 Strix 6G
  • Sapphire Nitro R9 Fury 4GB
  • Sapphire Nitro RX 480 8GB

In our testing script, we save a couple of special things for the GTX 1080 here. The following tests are also added:

  • Civilization 6 (8K Ultra, 16K Lowest)

This benchmark, with a little coercion, are able to be run beyond the specifications of the monitor being used, allowing for 'future' testing of GPUs at 8K and 16K with some amusing results. We are only running these tests on the GTX 1080, because there's no point watching a slideshow more than once.

Test Bed and Setup Benchmarking Performance: CPU System Tests
Comments Locked


View All Comments

  • WoWFishmonger - Thursday, August 17, 2017 - link

    I thought it was "The proof is in the PUDDING"
    All this time I've been eating pudding, looking for proof..... explains why I haven't found any yet. :|

    Nice write up, its good to see that even if people won't use this new mode, they do have the choice to enable it.

    Nothing wrong with choice IMO.
  • Ian Cutress - Thursday, August 17, 2017 - link

    Heh, wow. That's a bad typo. Fixed, thanks :)
  • edzieba - Thursday, August 17, 2017 - link

    "I thought it was "The proof is in the PUDDING""

    The phrase is: "the proof of the pudding is in the eating".
  • boozed - Thursday, August 17, 2017 - link

  • Alexvrb - Saturday, August 19, 2017 - link

    I've always heard "the proof is in the pudding". The shorter version's meaning is still pretty apparent. Plus it rolls off the tongue better, so to speak. Mmmm..... pudding.
  • sprockincat - Thursday, August 17, 2017 - link

    While we're on the topic, I think it's "Game Mode as originally envisioned by AMD."
  • NikosD - Thursday, August 17, 2017 - link

    So, you read my comment regarding your mistake at the first TR review of assuming a 16C/16T CPU after enabling Game Mode instead of a 8C/16T and you corrected that in your new review.

    Now, you only have to repeat your tests with DDR4-3200 and select a different, more "workstation" kind of benchmarks in order to test monsters of 32 threads and not PDF opening of course !

    Mercy !
  • Aisalem - Thursday, August 17, 2017 - link

    For the average person reading most of tech sites the more workstation benchmarks doesn't really makes sense.

    What I would like to see if you can enable game mode and disable SMT. That will leave 1950X with 8 cores available to the system which still should be enough for gaming but might present even better results.
  • Zstream - Thursday, August 17, 2017 - link

    For the love of all things... no one buys TR to just play games, or open .PDF's.
  • Gothmoth - Thursday, August 17, 2017 - link

    well noobs do.

    but i think websites like anandtech should know better.. but well anand is gone and.
    the new generation is obviously no adequat replacement.

Log in

Don't have an account? Sign up now