Test Driving Futuremark's PCMark 10 Benchmark

Futuremark is a well-respected provider of computer benchmark applications. Their PCMark and 3DMark benchmarks have been around for almost 20 years, and provide a good indication of the system performance for various workloads. Today, Futuremark is launching PCMark 10, their seventh major update to the PCMark series of benchmarks first launched in 2002. PCMark 10 builds upon the PCMark 8 platform, adds a few workloads and streamlines the rest in order to present a vendor-neutral, complete, and easy-to-use benchmark for home and office environments. Futuremark provided us with a preview copy of the benchmark, and we took it for a test drive.

Introduction and Evaluation Setup

Futuremark's first unified benchmark for evaluation of system performance was PCMark 2002, released in March 2002. It was followed by PCMark 04 in late 2003, and PCMark 05 (with support for Windows Vista). PCMark Vantage was introduced in 2007 for evaluation of systems running Windows 7, and PCMark 7 in 2011. PCMark 8 was launched in 2013, with full support for Windows 8 / 8.1. Even though Windows 10 is not an officially supported OS, we have had no issues in processing the PCMark 8 workloads on machines running that OS. PCMark 10 brings official support for Windows 10.

PCMark 10, like BAPCo's SYSmark 2014 SE, caters mainly to system vendors who want to bid for government contracts / tenders for high-volume PC purchases from businesses and enterprises. The PCMark scores help the vendors in advertising a performance level for their systems. From the other perspective, it can also help businesses to mandate a minimum performance level for systems that they hope to purchase. PCMark 10 measures system performance using workloads based on real-world applications and activities reflective of the usage of modern-day PCs in an office environment.

PCMark 10 comes in three different flavours - the Basic edition is free to download, while the Advanced edition costs $30 (suitable for home consumers). The Professional edition is the one suitable for business use, and it comes with a host of features that make the benchmark easy to use in an automated manner. Pricing for the professional edition is not public.

The PCMark 10 benchmark can be processed in one of three built-in modes - default, express or extended. These built-in modes correspond to benchmark definitions (in XML) that are supplied along with the installation. User-generated XML files conforming to the correct format can be created for custom benchmarks. The 'Express' benchmark is a subset of the default one, which, in itself, is a subset of the 'Extended' benchmark. The express benchmark and focuses on typical office workloads such as web browsing and selected productivity tasks. The 'Extended' tests add demanding graphics and physics test to provide an idea of the CPU as well as the GPU performance.

Each of these workloads is composed of sub-tests, as shown below.

The application start-up workload involves measuring the time taken to launch a number of programs of varying complexity such as the Chromium web browser, the Firefox web browser, LibreOffice Writer word processor, and the GIMP image manipulation program. Web browsing tests involve navigation of a social media site, online shopping, map navigation, video playback (H.264 and VP9 at 1080p and 4K resolutions), and a static web page, using both Firefox and Chromium. The video conferencing workload deals with both low-quality one-on-one conferencing (720p30 H.264 encode and decode, as well as face detection running on an available OpenCL device or CPU), as well as high-quality group conferencing (1080p30 H.264 encode, and 3x 720p30 H.264 decode, as well as face detection). Productivity tests involve word processing (using LibreOffice Writer) and spreadsheet editing (using LibreOffice Calc). Digital content creation tests involve photo editing (using ImageMagick, capable of OpenCL acceleration) with various filters, video editing (using ffmpeg, capable of OpenCL acceleration) for downscaling, sharpening and deshaking, and rendering and visualization. The last workload uses the OpenGL 4.3 code path from 3DMark's Sling Shot workload, and a custom POV-Ray workload. The gaming tests are directly from the 3DMark Fire Strike workload.

Setting up the Evaluation Flow

We processed PCMark 10 on a number of systems that we had readily configured. The details of the systems are presented in the next section. The first step was to create an installation script. Though PCMark 10 offers an uninstall option, I have found it better to create a system image prior to installation of such benchmarks. This provides us with an option to perform 'System Image Recovery' to go back to the prior version. Towards this, I created the following Powershell script, assuming that a USB drive with the volume label 'PCImageBackups' is connected to the system.

​$BackupVolume = Get-Volume -FileSystemLabel PCImageBackups

​$BackupDrive = $BackupVolume.DriveLetter + ":"
Write-Host "`nUsing ${BackupDrive} for backing up..."
wbAdmin start backup -backupTarget:$BackupDrive "-include:C:,D:" -allCritical -quiet

Write-Host "`nInstalling PCMark 10..."
cmd /c start /wait .\pcmark10-setup.exe /quiet /silent
& 'C:\Program Files\Futuremark\PCMark 10\PCMark10Cmd.exe' --register=<PCMark10 Pro Key>

After executing the above script (resident in the same folder as pcmark10-setup.exe), the following run script was then copied over to a local folder on the machine, and the system was restarted after dismounting all attached USB drives.

​$CWD = (Get-Item -Path ".\" -Verbose).FullName

​& 'C:\Program Files\Futuremark\PCMark 10\PCMark10Cmd.exe' --definition=pcm10_extended.pcmdef --out=$CWD\bmark.pcmark10-result --export-pdf=$CWD\bmark.extended.pdf --systeminfo on --systeminfomonitor on

Processing the above script generated the benchmark results in the same folder as that of the script. The results file could be opened in the PCMark 10 application to observe the monitoring data and low-level workload scores, while the PDF held a portable copy of the high-level results and system information.

BAPCo SYSmark 2014 SE - A Comparison

BAPCo's SYSmark 2014 SE was released recently, and it caters to the same market segment as the Futuremark PCMark 10. Having set up an evaluation flow using SYSmark 2014 SE recently, it was interesting to try out PCMark 10 and evaluate its approach to system benchmarking. The SYSmark 2014 SE flow is considerably more complicated, as it uses commercial versions of programs such as Adobe Acrobat. This creates issues with clean uninstallation of the benchmark (the system image creation scripting was done as a solution for that initially). Fortunately, PCMark 10 doesn't use any commercial software as part of its workloads. SYSmark 2014 SE tries to ensure that all OS settings are consistent across different benchmark runs as long as a particular profile is loaded. Unfortunately, PCMark 10 has a set of default internal settings, but really doesn't care about the OS settings. In the course of our evaluation of PCMark 10, we found that scores could vary widely with different OS settings, making it the prerogative of the benchmark processor to ensure consistency while evaluating different systems. We document that, as well as additional selected sensitivity analysis, in the next section.

​