Introducing the 2012 Mobile Benchmark Matrix
by Jarred Walton on January 6, 2012 8:02 PM EST2012: Meet Our New Mobile Benchmark Suite
Testing computer hardware can be a difficult process. On the one hand there’s a desire for more information and benchmarks, and on the other hand there’s a desire for timely reviews. Our goal at AnandTech has always been to deliver the most comprehensive reviews possible, and while we strive to timeliness there are occasions where additional testing or questions may delay a review. Ultimately, there’s a balancing act that needs to be maintained, and over time we periodically refresh our review suite and testing methodologies.
With 2012 now here, we’re launching a new suite of benchmarks for our laptop reviews. We'll also have the results from our first laptop using the new tests, courtesy of ASUS' G74SX. Some of the tests have already been in use for a while and others are brand new. In order to provide a single location with a list of our benchmarks and testing procedures, we have put together this short overview. We plan on using the following test suite throughout 2012, and while it’s possible we will add some benchmarks, we don’t have any plans to stop using any of the following at least for the next year.
General Performance Tests
Starting with our general tests, all of these have been in use for several months at least, with many tests dating back to 2010 and earlier. We’ll continue to use the full PCMark 7 suite, PCMark Vantage (x64), Graysky’s x264 HD encoding test, Cinebench 11.5, 3DMark06, 3DMark Vantage (Entry-Level and Performance defaults), and 3DMark 11. We’ll also continue with our battery life tests (now with Internet Explorer 9 in place of IE8) and LCD tests. So for most areas, our test suite remains largely unchanged—we’re finally dropping Cinebench 10, but that’s about it.
As we’ll mention in the conclusion, we’re willing to add some additional general performance benchmarks if there are any specific requests. One of the difficult things to quantify with modern PCs is how fast they are in the things most people do on a regular basis. Part of the problem is that most PCs from the past three or four years are all “fast enough” for generic tasks like surfing the web—if you’re actually reading the content of web pages rather than just repeatedly loading a complex page, I’m not sure most users would notice the difference between a 2GHz Core 2 Duo or Athlon X2 laptop and a quad-core i7-2760QM. This is why battery life is such an important element, as where many wouldn’t notice the difference between a web page loading in two seconds and a web page loading in one second, they’re far more likely to notice two hours of battery life versus four or eight hours. Anyway, let us know if you have other mobile benchmarks you’d like us to consider.
With that out of the way, we’ll save the next page for the major changes: our updated gaming suite.
Facebook
Twitter
RSS
48 Comments
View All Comments
Ryan Smith - Saturday, January 7, 2012 - link
Brink's really not any better. It's a multiplayer game that no one plays (and I say that as an owner), and because it's a MP game there's no practical way to structure a repeatable benchmark.We would have definitely liked to include an OpenGL game, but even I have to admit that OpenGL just isn't very important right now. Maybe Doom 4 or Prey 2 will change that, but with id not licensing Tech 5, OpenGL is quickly becoming an API that's only applicable to id games.
jjj - Friday, January 6, 2012 - link
"as where many wouldn’t notice the difference between a web page loading in two seconds and a web page loading in one second"That's not all that true,there are some actual numbers from Google and Amazon about how page loading time relates to sales/searches:
-Amazon : every 100 ms increase in load time of Amazon.com decreased sales by 1%
-Google: a change in a 10-result page loading in 0.4 seconds to a 30-result page loading in 0.9 seconds decreased traffic and ad revenues by 20%.
In the end it might be a more usefull test than some synthetic tests already on the list.
Intel low power CPUs choke when both GPU and CPU are active and maybe the 17W Trinity SKUs will too.Maybe there should be a test that reflects that,besides the gaming ones since it's a rather important piece of info.
Finally,the most important test that is missing is WIFI perf,it is a mobile device after all.
Conficio - Friday, January 6, 2012 - link
It's so funny that Google knows about the numbers and does even provide tools to measure the load times and make suggestions how to improve. However when I run the performance analysis against one of my websites, some of the suggestions to improve are for the Google Analytics, and Google Adsense scripts (like not allowing caching of the script, or only allowing for less than a week, or scripts loaded from redirections, etc.).I also see many web pages waiting for serves that load google AdSense or analytics.
Is it just me, or should Google start to eat it's own dog food?
JarredWalton - Friday, January 6, 2012 - link
What would you like us to test with WiFi? Maximum throughput? Part of that will be determined by the chipset, a larger part more than likely will come from the choice of router, and the rest is noise and interference between runs. I do make a point of listing the WiFi chipset, which tells you about 90% of what you need to know.(Hint: 1x1:1 MIMO solutions are the bottom of the barrel and come in most laptops; the 2x2:2 solutions are okay, and if you have a 5GHz router that can really help remove interference. I've only tested about three laptops with 3x3:3 MIMO over the years, and sadly my current router can't even support the maximum throughput.)
jjj - Saturday, January 7, 2012 - link
You guys are already testing WIFI for phones and tablets,it's easy to just apply the same methodology.I do have a hard time understanding why do it there and not here and this internet thing is something that tends to be used a lot.I guess testing notebooks started by doing the same thing as on desktops and it didn't seemed obvious that this should be tested. Listing the part used helps only folks that know what it means and you get what you expect only if the OEM does things right. I don't have a laptop example but look at Asus Transformer Prime and it's WIFI and GPS problems, can you be sure that there aren't a bunch of laptops that offer a lot less than they should? Poor WIFI perf could also be a deal breaker for many,if they knew about it and maybe,just maybe,some manufacturers would pay more attention to it.
JarredWalton - Saturday, January 7, 2012 - link
For smartphones, you're looking at 1x1:1 2.4GHz solutions with a very small device that can easily run into connectivity problems if, for instance, the casing is all aluminum. For laptops, it's usually not much of a concern (e.g. they're plenty large to route wires in a sensible fashion for the antennae). WiFi speeds usually aren't a concern unless you're transferring large files. If you're doing that, then you're going to typically be at the limits of the chipset, not the laptop, and you'd be far better off with Ethernet regardless.Anyway, there are other problems with trying to test laptop wireless speeds. One is that we have at least two different test locations, so that necessitates getting identical hardware on both sides -- router for sure, and testing location won't be identical. Even with the right hardware, outside interference (from neighboring networks) is a potential problem.
The better solution IMO is a separate review of wireless chipsets. I tried to do that with the Killer Wireless-N review a while back, and Brian is working on a more complete roundup of wireless chipsets. Outside of that review, I'll see what Anand thinks about getting us equipped with the necessary hardware to test wireless.
jjj - Saturday, January 7, 2012 - link
maybe this helps http://www.smallnetbuilder.com/wireless/wireless-f...Anyway design matters here too and ofc it impacts range too not only throughput.For using Ethernet,you don't always have it,maybe you use your laptop mostly as a desktop replacement but in the end that's not it's main purose for many.
The differences are there and i don't know how many folks wouldn't care at least about range.
jalexoid - Friday, January 6, 2012 - link
Please, please, please add OpenGL benchmarks for professional users. OpenCL would be quite good also, specifically data transfer between the main mem to graphics memory.JarredWalton - Saturday, January 7, 2012 - link
We generally run SpecViewPerf on workstation GPUs (Quadro FX), but is there really a desire to see those tests run on consumer graphics as well? Even a basic Quadro FX will generally outperform the fastest consumer cards in professional OpenGL testing, simply because of the drivers/firmware.For OpenCL, Ryan tests that on GPUs to some extent, but I'm not sure how many people are seriously looking at OpenCL performance on laptops. Would you suggest using the same tests as Ryan is using, or do you have some specific OpenCL benchmark you'd like us to run?
ltcommanderdata - Saturday, January 7, 2012 - link
PowerDirector 10 uses OpenCL to render it's video effects. Video editing seems like a good use case that most users can relate to rather than say fluid simulation or Monte Carlo calculations.PowerDirector 10 also supports AMD APP Acceleration (which is OpenCL I suppose), nVidia CUDA, and Intel QuickSync for final encoding so could be useful to compare each platform's ideal accelerated encoding method.
The upcoming WinZip 16.5 is supposed to be OpenCL accelerated for compression, decompression, and encryption making another benchmark with a use case that is applicable to most users.