Testing Methodology

If you've been keeping up with our case reviews, our testing methodology for the fans here is going to seem relatively similar in some ways. Our test system may seem a bit unusual in more than a few ways, but stick with me and I'll explain why I put it together and tested it the way I did.

The processor, with its healthy voltage boost and overclock, throws a pretty substantial amount of heat at our cooling system. Testing with an i7-2700K at stock speeds would defeat the purpose; Intel's own stock cooler can handle that, we want to "separate the men from the boys" so to speak.

Fan and Radiator Testing Configuration
CPU Intel Core i7-2700K overclocked to 4.4GHz @ 1.4V
Motherboard Zotac Z77-ITX WiFi
Graphics Intel HD 3000 IGP
Memory 2x4GB Crucial Ballistix Sport Low Profile DDR3-1600
Drives Kingston SSDNow V+ 100 64GB SSD
Power Supply Rosewill Hive 650W 80 Plus Bronze Modular
Enclosure BitFenix Shinobi XL Window

I needed a case that could produce adequate airflow, handle all of the different cooling systems without much trouble, and did not include any sound dampening features. You might be surprised at just how difficult that was to find, but BitFenix came to the rescue and sent over a Shinobi XL. BitFenix's enclosure didn't get the best review when I tested it, but it's actually ideal for this testbed. I removed every case fan but the front intake, which I ran at 5V to prevent it from affecting acoustics while still providing adequate airflow.

Since a dedicated GPU wasn't needed, one wasn't used. This prevents a graphics card from generating additional heat or noise or deflecting airflow.

Thermal and acoustic test cycles were done the same way as our case reviews. First, the system is left powered and idle for fifteen minutes. At this point the sound level is tested, room ambient temperature is recorded, and idle temperatures are recorded. Then eight threads of small FFTs in Prime95 are run for fifteen minutes, and load temperatures are recorded.

Each cooler was tested using its available presets; the PWM-controlled coolers were tested between 30% and 100% using motherboard control as well as at a constant 100%.

Thank You!

Before moving on, we'd like to thank the following vendors for providing us with the hardware used in our roundup.

  • Thank you to iBuyPower for providing us with the Intel Core i7-2700K.
  • Thank you to Zotac for providing us with the Z77-ITX WiFi motherboard.
  • Thank you to Kingston for providing us with the SSDNow V+ 100 SSD.
  • Thank you to Crucial for providing us with the Ballistix Sport Low Profile DDR3.
  • Thank you to Rosewill for providing us with the Hive 650W 80 Plus Bronze PSU.
  • Thank you to BitFenix for providing us with the Shinobi XL Window enclosure.
Ease of Installation Performance Results
Comments Locked

63 Comments

View All Comments

  • Touche - Thursday, March 14, 2013 - link

    I agree. The test setup is bad and not representative of most real life configurations. So bad that the whole review is pointless.
  • BrokenCrayons - Thursday, March 14, 2013 - link

    I disagree. The coolers are measured under similar conditions and their performance in degress over ambient are listed. While seeing them in a real-world scenario might give you measurements that are more in line with a fully populated case, the variances introduced by the additional, unnecessary hardware might make it more difficult to get consistent, comparable results. I'm more concerned about relative performance and how coolers stack against one another and Anandtech's testing methods appear to be the best way to obtain those results without unreasonable expectations about controlled environment facilities.

    -BC
  • lever_age - Thursday, March 14, 2013 - link

    Consistency and simplicity are good, but a procedure with a significant inherent bias is not (especially if this detail is not discussed by the reviewer... when it is not, it frankly seems like shortsightedness or incompetence, if not something worse). I rarely complain about testing procedures on most reputable review sites, but something is obviously flawed here. The results, which are out of line with most others', seem to indicate problems as well.

    There is nothing much that similar about how the air coolers and CLCs are being tested. The air coolers are being tested inside a relatively restricted box. The side-blowing towers are blowing towards a metal wall with some puncture holes from where the fan should be (also note the offset between holes and cooler, exacerbated by using a miniITX motherboard with socket unusually low). The CLCs are being tested with radiators and fans actually up against the openings in the chassis.

    Actually, for that matter, it would be good to know exactly the orientation and placement of the radiators and fans for how the CLCs were tested.

    Anyway, if you want a simplified setup, something more fair would be to just use an open-air test bench or no chassis at all.
  • Touche - Friday, March 15, 2013 - link

    I was about to reply to BrokenCrayons, but you summed it up pretty well.
  • BrokenCrayons - Friday, March 15, 2013 - link

    An open air or open frame case might be a reasonable option. I hadn't really thought much about it and that's not a bad point. However, the net results -- the coolers in question all being tested in the same container -- would be the same. Honesty, Dustin could have stuffed all the hardware in a Tupperware container or a shoebox that used to have a pair of Nine & Co. Drama Print Peep-Toe Pumps in them and, as long as they were all under the same conditions, the result would be just as valid. Cooler effectiveness is a relative scale thing anyway since the conditions in which each person will use them are going to be unique to them. So many factors (room location, case exposure to sunlight, time of day, relative humidity, where you put your US Robotics ISA 28.8 data/fax modem, the number of Quantum BigFoot hard drives you have, etc.) impact a processor's operating tempratures that the testing methodology is trivial to the process of judging coolers as long as it remains consistent.

    Atop that, Intel ships perfectly usable coolers with their boxed processors. While I enjoy the reviews, if I want a faster CPU, I'll invest the extra $100 on a next-model-up CPU instead of a pointlessly chromed-out cooler or I'll wait until a faster CPU is brought to market or I'll just do something else with my computer. Obviously that's a personal preference and others might feel differently, but in the grand scheme of things, only a few people can be bothered to even care about the difference between a CLC, an aftermarket air cooler, and something that came out of the box or was included in it when the OEM put it together.
  • lever_age - Friday, March 15, 2013 - link

    With regards to Intel's usable boxed cooler that's good enough for most people, I quite agree. However, they're not good enough for people who want to overclock significantly or want lower noise or temperatures or whatever; though the class of DIY builders who care about these things is small, they are relatively highly represented among AnandTech readers. Anyway, regardless of review subject material, if something's posted on AnandTech, it should be insightful and accurate, right?

    As for the other paragraph, I think you're misunderstanding an important point here, maybe because nobody really stated it explicitly.

    If you test different cooling designs in different test setups, the relative rankings will be different. That's why the test configuration is an important consideration. I'm not so concerned by the absolute numbers reported. Dustin used a test setup that very much does not represent what most people would be using, and this setup favors the CLCs over the air coolers. Someone not realizing this will read the results and think that CLCs have better performance relative to the air coolers than what you'd get in the real world in most case configurations. It's highly misleading.

    Or at least, that's what I believe based on reading others' test results and based on my intuition on the subject. If Dustin could prove me wrong and alleviate these concerns by retesting just a couple coolers on a different setup, I would be very appreciative and would promptly tuck the tail between the legs and run off.
  • lever_age - Friday, March 15, 2013 - link

    (continuing... sorry, no edit feature exists I think)

    As an example of how test setup would affect results, consider two test setups:
    A -- ATX-style case with strong front-to-back airflow, closed-off side panel with no mesh
    B -- open-air test bench

    Down-blowing CPU coolers would put up a better fight against side-blowing tower coolers in B than in A. Hopefully it is intuitive why.
  • inmytaxi@gmail.com - Saturday, March 16, 2013 - link

    Then why'd you read the review?
  • Tetracycloide - Sunday, March 17, 2013 - link

    This:

    "the testing methodology is trivial to the process of judging coolers as long as it remains consistent."

    Is only true as long as the coolers being tested are fundamentally similar. What you're saying is akin to claiming it's ok to test a custom water loop against an air tower without hooking the water loop up to the CPU block because "the conditions are the same." How is it a fair comparison if you're denying one solution's access to the medium by which it cools?
  • Azethoth - Tuesday, October 22, 2013 - link

    No, you are missing the issue. Yes, you can see relative performance, but because of the test flaw you can only compare the air coolers to each other (accurately) and the water coolers only to each other but you have to keep in mind that their fans bias even their results yielding advantage to the CLC with the most airflow added by its radiator fans.

    To make it fair, you can possibly move the radiator outside the case and maybe run it in a hotbox that matches the test case interior.

    Still, I think the only fair comparison is to add fans to the air coolers to match the radiator coolers as mentioned above. So you test CLC + radiator taking up vent space vs air cooler + extra fan taking up that same vent space.

    So in practice I would expect this test to show that CLC are amazing and top performers. We see this. In fact the results made me look at the test method to see if the CLC were even allowed to reach equilibrium or not they seemed so skewed.

    So good test for comparing only CLC to each other, or only AC to each other. Sucks for determining the true champions though because AC got shafted.

Log in

Don't have an account? Sign up now