Performance Consistency

We've been looking at performance consistency since the Intel SSD DC S3700 review in late 2012 and it has become one of the cornerstones of our SSD reviews. Back in the days many SSD vendors were only focusing on high peak performance, which unfortunately came at the cost of sustained performance. In other words, the drives would push high IOPS in certain synthetic scenarios to provide nice marketing numbers, but as soon as you pushed the drive for more than a few minutes you could easily run into hiccups caused by poor performance consistency. 

Once we started exploring IO consistency, nearly all SSD manufacturers made a move to improve consistency and for the 2015 suite, I haven't made any significant changes to the methodology we use to test IO consistency. The biggest change is the move from VDBench to Iometer 1.1.0 as the benchmarking software and I've also extended the test from 2000 seconds to a full hour to ensure that all drives hit steady-state during the test.

For better readability, I now provide bar graphs with the first one being an average IOPS of the last 400 seconds and the second graph displaying the IOPS divided by standard deviation during the same period. Average IOPS provides a quick look into overall performance, but it can easily hide bad consistency, so looking at standard deviation is necessary for a complete look into consistency.

I'm still providing the same scatter graphs too, of course. However, I decided to dump the logarithmic graphs and go linear-only since logarithmic graphs aren't as accurate and can be hard to interpret for those who aren't familiar with them. I provide two graphs: one that includes the whole duration of the test and another that focuses on the last 400 seconds of the test to get a better scope into steady-state performance. These results are for all drives 240GB and up.

Steady-State 4KB Random Write Performance

Steady-state performance is fairly good, although given that the SX930 employes 12% over-provisioning it's expected to perform better than drives with only 7%. 

Steady-State 4KB Random Write Consistency

Consistency, on the other hand, isn't that good. The good news is that the JMF670H is more consistent than the SM2246EN in the BX100, but there's a long way to achieve Samsung-level consistency. 

ADATA XPG SX930
Default
25% Over-Provisioning

As the SX930 isn't very consistent, we saw performance dropping below 1,000 IOPS. It seems that consistency is one of the areas that truly separate Samsung and Marvell drives from the rest because JMicron, Silicon Motion and Phison based drives all have trouble sustaining steady performance. Basically, the baseline performance is about 1,000 IOPS for all controllers with frequent peaks of +5,000 IOPS, whereas Samsung and Marvell drives have very little variation in performance. I suspect the controller design itself has something to do with this because Samsung and Marvell controllers are all multi-core, but JMicron and Silicon Motion rely on single-core designs for higher cost efficiency. 

ADATA XPG SX930
Default
25% Over-Provisioning
Introduction, The Drives & The Test AnandTech Storage Bench - The Destroyer
Comments Locked

67 Comments

View All Comments

  • bill.rookard - Thursday, July 16, 2015 - link

    It is more of a power-user scenario. I have a pretty fast desktop (i5-4670k, 16GB ram, Gigabyte Gaming-5 mobo) paired with an older (!) Samsung 830 128Gb boot drive. The computer, once you get past the POST, boots in seconds. Your average user would never notice a faster boot, since it's so bloody quick already.
  • DIYEyal - Monday, July 20, 2015 - link

    SSDs are far more than just fast boot.
  • frenchy_2001 - Thursday, July 16, 2015 - link

    As Anand used to repeat in most of its SSD tests, there is very little difference for typical client usage between SSDs. There are order of magnitudes between SSDs and HDDs (particularly useful for random access latencies), but once that jump is made, there is very little difference for client usage.
    Just check the "Storage Bench Light Latency" graphs. This is the ranking for typical client usage and most SSDs are in the same range.

    Only enterprises and power users will really benefit from PCIe/NVMe storage. For the rest of us, any modern SSD is good enough.
  • Stochastic - Friday, July 17, 2015 - link

    To me this raises the question of what's next in client computing. Any half-decent PC today with an SSD already excels at most consumer tasks--web browsing, office applications, light photo editing, 1080p video, etc. Compute intensive tasks such as computer vision and natural language processing are being pushed to the cloud. Gamers have faster GPUs and next-generation displays (OLED, 4K, wide-gamut, adaptive V-sync) to look forward to, but aside from that there's nothing else in the horizon that gets me excited. Maybe VR is the next big thing?
  • Kristian Vättö - Friday, July 17, 2015 - link

    Unified memory architecture (i.e. no more memory and storage, just one type that serves as both) is really the next big thing. We'll likely see something in that front in the early 2020s.
  • AnnonymousCoward - Saturday, July 18, 2015 - link

    Are you saying a non-volatile technology will replace DRAM in 5 years? How could that possibly happen?
  • Kristian Vättö - Saturday, July 18, 2015 - link

    Not replace right away, of course, but in five years time the next generation non-volatile memory (ReRAM, NRAM, MRAM or whatever it ends up being) should start to be in meaningful production volume. Once that happens it will slowly start to replace DRAM and NAND, although that transition may easily take a decade, but some enterprise applications will probably take advantage of a unified memory architecture rather quickly.
  • AnnonymousCoward - Tuesday, July 21, 2015 - link

    Huh. I haven't heard of all 3 of those. But without even knowing about them, I can say in general it will be damn hard to beat DRAM performance and density/cost. Maybe in 2030.
  • ATC9001 - Thursday, July 16, 2015 - link

    Is there ever going to be a drive that challenges the EVO for price and performance? It competes with higher end drives, but is nearly the cheapest available...
  • Refuge - Thursday, July 16, 2015 - link

    Honestly idk...

    Samsung is very well set up in the SSD space right now, they have their own in house controller that is a very solid performer, and they are completely vertically integrated with some of the best fabs around.

Log in

Don't have an account? Sign up now