Introducing the 2012 Mobile Benchmark Matrix
by Jarred Walton on January 6, 2012 8:02 PM EST2012: Meet Our New Mobile Benchmark Suite
Testing computer hardware can be a difficult process. On the one hand there’s a desire for more information and benchmarks, and on the other hand there’s a desire for timely reviews. Our goal at AnandTech has always been to deliver the most comprehensive reviews possible, and while we strive to timeliness there are occasions where additional testing or questions may delay a review. Ultimately, there’s a balancing act that needs to be maintained, and over time we periodically refresh our review suite and testing methodologies.
With 2012 now here, we’re launching a new suite of benchmarks for our laptop reviews. We'll also have the results from our first laptop using the new tests, courtesy of ASUS' G74SX. Some of the tests have already been in use for a while and others are brand new. In order to provide a single location with a list of our benchmarks and testing procedures, we have put together this short overview. We plan on using the following test suite throughout 2012, and while it’s possible we will add some benchmarks, we don’t have any plans to stop using any of the following at least for the next year.
General Performance Tests
Starting with our general tests, all of these have been in use for several months at least, with many tests dating back to 2010 and earlier. We’ll continue to use the full PCMark 7 suite, PCMark Vantage (x64), Graysky’s x264 HD encoding test, Cinebench 11.5, 3DMark06, 3DMark Vantage (Entry-Level and Performance defaults), and 3DMark 11. We’ll also continue with our battery life tests (now with Internet Explorer 9 in place of IE8) and LCD tests. So for most areas, our test suite remains largely unchanged—we’re finally dropping Cinebench 10, but that’s about it.
As we’ll mention in the conclusion, we’re willing to add some additional general performance benchmarks if there are any specific requests. One of the difficult things to quantify with modern PCs is how fast they are in the things most people do on a regular basis. Part of the problem is that most PCs from the past three or four years are all “fast enough” for generic tasks like surfing the web—if you’re actually reading the content of web pages rather than just repeatedly loading a complex page, I’m not sure most users would notice the difference between a 2GHz Core 2 Duo or Athlon X2 laptop and a quad-core i7-2760QM. This is why battery life is such an important element, as where many wouldn’t notice the difference between a web page loading in two seconds and a web page loading in one second, they’re far more likely to notice two hours of battery life versus four or eight hours. Anyway, let us know if you have other mobile benchmarks you’d like us to consider.
With that out of the way, we’ll save the next page for the major changes: our updated gaming suite.
Facebook
Twitter
RSS
48 Comments
View All Comments
bennyg - Friday, January 6, 2012 - link
When I read the headline... nice fancy name there for just a reshuffle of what you already do. And there's enough of that going around already isn't there...I skim through AT reviews despite superior production quality.
I read notebookcheck reviews in full despite frequent editing and translation errors because (aside from much preferring monolithic-single-page-rendering...) they consider so many more aspects to the product in front of them.
They also take the effort to disassemble and put up a lot more pictures of the unit and thorough screen (incl viewing angle) assessment. If I google a model for reviews, I want to read new stuff (not most of the same stuff as every other review) and NBC gets my browser time most of the time for that reason.
And please please please incorporate some of the methodology from http://techreport.com/articles.x/21516 rather than just simplistic FPS measures, especially for the SLI/Xfire setups...
JarredWalton - Friday, January 6, 2012 - link
Fancy headline for a "reshuffle"? How about, I thought it would be useful to open up discussion to see if there's anything we missed that people would like to see us include, along with a detailed list of the benchmark settings we're using for games (so that I don't have to try and put this all into each laptop review)? And you do realize that this is an article specific to laptop testing, so we're not going to go into more detail on gaming performance and we rarely test SLI/CF setups, right? As for the pictures, I'm not sure what more you'd want from us. We do everything you mention -- screen angle shots, pictures from lots of angles, etc.Anyway, as I note in the article, we're open for suggestions on what you'd like to see added that isn't already there. Notebookcheck has a rundown of each laptop that's pretty much just regurgitating the spec sheet, so I think we're covered there. We run a standardized set of benchmarks that includes more detailed graphs, though perhaps some would prefer the NBC approach (e.g. just show the scores from the laptop being tested with a "heat map" below showing the spread and frequency of other scores)?
Consider the scope of the review and by all means let us know which aspects of laptop reviews you'd like us to cover more. About the only major test that NBC runs that we don't have is CrystalDisk, but I'm not sure how useful that really is. SSDs are much faster than HDDs, and the differences between HDDs are largely meaningless by comparison. I generally figure anyone after fast storage for a laptop will be looking to upgrade to an SSD regardless, and if that's the case they'll be reading our SSD reviews after determing which laptop they want. But let me throw this out there:
Are there others that would like us to run one of these "quick and dirty" storage benchmarks on the laptops we test? Is the PCMark 7 Storage score insufficient in what it reports? I'm not going to add a test because of one request for it, but if a lot of you would like some additional tests let me know.
kedesh83 - Saturday, January 7, 2012 - link
You would think most laptop gamers would be either playing World of warcraft, or Starcraft 2. I'm not being a fanboy or anything but why would they not include those in the games list? Most of the kids i see playing games on my campus are playing Blizzard titles.JarredWalton - Saturday, January 7, 2012 - link
They're older games and DX9 titles as well. If the gaming suite we test runs sufficiently fast on a laptop, I can pretty much guarantee SC2 and WoW will run.ananduser - Saturday, January 7, 2012 - link
When testing the battery please include a moderate amount of FLASH only sites the likes of Tag Heuer or famous car brands flash minisites. Flash is an important part of the web and would make your tests more realistic.signorRosso - Saturday, January 7, 2012 - link
In hardware or software?10-bit is mentioned at the bottom of this AT article page...
http://www.anandtech.com/show/4380/discrete-htpc-g...
signorRosso - Saturday, January 7, 2012 - link
Disregard the previous comment entirely!Paedric - Saturday, January 7, 2012 - link
I don't know if it is possible to do it, or if it is useful, but what about testing the performance of virtual machines?Also, you said you now use IE9 instead of IE8, is there a significant difference in battery life between Chrome/Firefox/IE9/Opera?
gero9mo - Saturday, January 7, 2012 - link
" I’m not sure most users would notice the difference between a 2GHz Core 2 Duo or Athlon X2 laptop and a quad-core i7-2760QM. This is why battery life is such an important element, as where many wouldn’t notice the difference between a web page loading in two seconds and a web page loading in one second, they’re far more likely to notice two hours of battery life versus four or eight hours. "I can honestly tell you that one second feels for me like a lifetime in computing. If i where to load 100 pages during a surfing-session, if i can call it that (my English ain't the best), I would most certainly prefer those pages to load in 1 second instead of two. And, in the werry moment you get to a bit more complex page, you are without a doubt gona notice a difference between lets say a Intel Sandy Bridge based CPU versus any AMD CPU. Most users also do other stuff than just surf a web-page. They also extract zip,rar and other files, and even here youre gona notice a difference between Athlon X2 and a i7-2760QM. And if youre seeling computers, be shure to look two to three years forward in time. I would prefer to sit with a i5-2310M versus any Athlon X2 laptop.
So even if most people won't notice a huge difference, a second here and a secongd there still counts.
PreacherEddie - Saturday, January 7, 2012 - link
On the third page, third paragraph from the bottom: "We are still early enough in 2011...", I think should be "2012", unless you also have developed that time travel machine.