Testbed Travails

Regular readers of our direct-attached storage reviews might have noticed that we upgrade our DAS testbed approximately every couple of years. It is important to keep the testbeds consistent across reviews of different storage devices so that the effect of the CPU, DRAM bandwidth, etc. remain the same across different runs of the benchmarks on different devices.

Our most recent update to the DAS testbed was in early 2019 - a move to the Hades Canyon NUC triggered by the inconsistent performance of the Alpine Ridge port in our Skylake DAS testbed. This inconsistency started showing up after we attempted a Thunderbolt firmware upgrade to enable eGFX functionality on the machine. Prior to the Skylake-based testbed, we were using a Haswell-based system. We opted against building a fresh DAS testbed with either the TRX40 or Z490 boards, as the imminent arrival of USB4 meant that we could again be forced to do an upgrade. A testbed change not only involves preparing a new machine - it also involves benchmarking older drives again (to the extent possible). We wanted to put this off as much as possible.

The Hades Canyon NUC, without PCIe expansion cards handling capability, was ruled out as a recipient of the Yottamaster C5. The initial plan was to use the Ghost Canyon NUC for this purpose after removing the discrete GPU. As we soon discovered, the PSU of the Ghost Canyon NUC doesn't come with SATA power cables, and an adapter cable wasn't handy. We moved on to our next option, the Skylake-based DAS testbed.


Yottamaster C5 in the GIGABYTE Z170X-UD5 TH ATX Motherboard

The installation of the Yottamaster C5 in the GIGABYTE Z170X-UD5 TH ATX board was uneventful - no drivers to install, as the ASMedia ASM3242 in the C5 uses Microsoft's XHCI drivers built into Windows 10 (May'20 update). As a first step, we took the SanDisk Extreme Portable SSD v2 (SuperSpeed USB 10Gbps) for which we already had recent benchmark numbers, and processed it with our test suite using the C5's Type-C port. The benchmarks with the ASM3242 host delivered better results than what was obtained with the Alpine Ridge port of our regular testbed - but this was to be expected, given the ASMedia chipsets at either end of the chain. After being satisfied with the shaping up of the updated testbed, we connected the SanDisk Extreme PRO Portable SSD v2 to the C5's port. Unfortunately, the drive kept connecting and disconnecting frequently (YouTube video link for screen capture). Sometimes, it stayed up for long enough to process a couple of iterations of one of the CrystalDiskMark workloads before disappearing (as shown in the screenshot below).

Initially, the suspicion was on the Plugable USBC-TKEY in the middle of the chain (kept in place for power measurement), but the behavior was the same with the direct connection too. The WD_BLACK P50 also exhibited the same problems. Based on the online reviews, this problem doesn't seem to be isolated to the Yottamaster C5 - ASM3242 cards from other vendors also appear to have similar issues.

Ruling out the Skylake-based testbed for the evaluation, we decided to attempt the installation of the card on our Haswell-based testbed. In this system, we no longer had the disconnection issue. Our test suite managed to run to completion on all the drives that we wanted to test.


Testing in Progress on the 'Best-Performing' USB 3.2 Gen 2x2 Testbed - (Core i7-4790 / Asus Z97-PRO Wi-Fi ac ATX / Corsair Air 540)

We did observe one hiccup in the set of tests - while processing the CrystalDiskMark 4K random reads and writes with 16 threads and a queue depth of 32, the system completely froze up for a good 30-60s before recovering (the effect can be seen in the CrystalDiskMark power consumption graphs in a later section). Our internal SSDs review editor, Billy, was able to reproduce the same with a Haswell-based system (using Core i7-4790K) at his end while using an Intel USB 3.0 port and a SuperSpeed 10Gbps enclosure using the JMicron JMS583 chipset. The problem was not reproducible with internal drives. Our inference is that the combination of high queue depth and thread count creates way too much driver overhead that the Haswell-based systems find difficult to handle.

As a final resort, we shifted back to the current DAS testbed, the Hades Canyon NUC. Taking the eGFX route, we connected the PowerColor Gaming Station to the Thunderbolt 3 port after removing its internal daughtercard responsible for all of its I/O ports. The PowerColor Gaming Station unofficially supports a SATA drive, which meant that its PSU has a spare SATA power cable. Using this, it was a breeze to get the Yottamaster C5 up and running in the eGPU enclosure.

Our test suite was processed on the WD_BLACK P50 and the SanDisk Extreme PRO Portable SSD v2 using multiple testbed configurations detailed above. We also processed the SanDisk Extreme Portable SSD v2 (SuperSpeed USB 10Gbps device) using the same ports for comparison purposes. The two SuperSpeed USB 20Gbps drives were also processed with our regular testbed to provide an idea of their performance when connected to regular Gen 2 (SuperSpeed USB 10Gbps) ports.

AnandTech DAS Testbed Configurations for USB 3.2 Gen 2x2 Testing
Configuration Suffix in Graphs Notes
Asus Z97-PRO Wi-Fi ac ATX
Core i7-4790
Corsair Vengeance Pro CMY32GX3M4A2133C11 DDR3-2133
32 GB (4x 8GB) @ 11-11-11-27
Seagate 600 Pro 400 GB
Yottamaster C5 USB 3.2 Gen 2x2 Expansion Card
Corsair AX760i 760 W
Corsair Air 540
[ASM3242] N/A
Intel NUC8i7HVK
Core i7-8809G
Crucial Technology Ballistix DDR4-2400 SODIMM
32GB (2x 16GB) @ 16-16-16-39
Intel Optane SSD 800p SSDPEK1W120GA
Intel SSD 545s SSDSCKKW512G8
PowerColor Gaming Station
Yottamaster C5 USB 3.2 Gen 2x2 Expansion Card
[ASM3242 via JHL6540] N/A
Intel NUC8i7HVK
Core i7-8809G
Crucial Technology Ballistix DDR4-2400 SODIMM
32GB (2x 16GB) @ 16-16-16-39
Intel Optane SSD 800p SSDPEK1W120GA
Intel SSD 545s SSDSCKKW512G8
[JHL6540] Alpine Ridge
Thunderbolt 3 port
used in
USB 3.1 Gen 2 mode
GIGABYTE Z170X-UD5 TH ATX
Core i5-6600K
G.Skill Ripjaws 4 F4-2133C15-8GRR DDR4-2133
32 GB ( 4x 8GB) @ 15-15-15-35
Samsung SM951 MZVPV256 NVMe 256 GB
Yottamaster C5 USB 3.2 Gen 2x2 Expansion Card
Cooler Master V750 750 W
Cooler Master HAF XB EVO
[ASM3242 Skylake] SanDisk
Extreme Portable SSD v2 only

The table above lists all the configurations that were tested, along with notes on the implications of the suffix seen in the graphs in the following sections.

Introduction Device Features and Characteristics
POST A COMMENT

81 Comments

View All Comments

  • henkhilti - Monday, October 5, 2020 - link

    Why do boot, office, PCMark10 runs.
    DAS is only about copying data from internal to external move the DAS to other system and do the opposite (copy data from external to internal).
    Users don't have ram drives or use robocopy (you already have the synthetic benchmarks).

    Just use fast internal drives (that people actually buy) and start dragging files/folders with File Explorer like humans do :)
    Reply
  • repoman27 - Monday, October 5, 2020 - link

    I boot from USB external media all the time. It's a pretty common part of many workflows. So is running VM's that reside on external media. Or working with large media files that live on external media dedicated to a particular project / client. Reply
  • hubick - Tuesday, October 6, 2020 - link

    I was bike commuting from home to work with my SSD (Samsung X5 Thunderbolt 3) in my pocket and booting Linux off it at either end. Easier than trying to sync data, plus I only had to maintain the, like, 1000 development tools I need in once place. Reply
  • drajitshnew - Monday, October 5, 2020 - link

    I have been reading Anandtech for 16 years now and this is the most outright confusing article that I have read for the reasons listed
    1. I disagree with the testbed choice-- IT IS NOT A SYSTEM,IT IS A HACK . I do not think that it has realworld applicability. Could you try a couple of Ryzen system. Failing that just stop giving 16 threads q32 results. I cannot think of a case where a 1 TB, bus powered DAS would be used in a realworld use requiring 16*32 random reads. Can you?
    2. I agree with @danneely @spunjji @stormyparis, USB 3.2 is a mess. Even rtfm fails. I would request your reviewers to add the supported speeds to each system, board and flagship mobile review. With 8k video (and limited storage) it is not unthinkable to use one of these devices to transfer files.
    Reply
  • ganeshts - Monday, October 5, 2020 - link

    The article is a description of our attempts to make use of USB 3.2 Gen 2x2 as a regular consumer. Rarely do people go out on a limb and make a new PC for something that could be achieved with an expansion card. That is the reason for the piece being described as 'ecosystem review', rather than a review of just the WD_BLACK P50 and the SanDisk Extreme PRO v2.

    The testbed choice was an attempt to use the three 'direct-attached storage testbeds' we have used since starting the reviews set back in 2014. It did end up as a 'hack', but that serves the purpose of this particular review well. If you see many more Gen 2x2 reviews using the same Haswell testbed, then your complaint is justified.

    AnandTech's editors do not operate out of a central location. Most (including me) are freelancers spread out throughout the world. As far as testing out a couple of Ryzen systems goes - all the Ryzen PCs with me are SFF machines without a PCIe expansion slot. I am loath to building a new testbed at *this point in time* because USB4 is just around the corner. It is better to build a testbed that can serve the purpose well for at least 2 to 3 years.

    As for bus-powered DAS and the 4KQ32T16 workload - if you don't think it is suitable, feel free to disregard the numbers. The limited applicability of the workload is exactly the reason we felt it was OK to present the results from the other workloads on the same machine.
    Reply
  • supdawgwtfd - Monday, October 5, 2020 - link

    What editors? Reply
  • drajitshnew - Tuesday, October 6, 2020 - link

    Thank you for addressing my concerns. In light of this I have 2 fresh ones
    1. As you also agree the host system is a hack-- it should not become the regular testbed. I do not bother with T16q32 results in any review, because I do not have any workload that can generate such a workload.
    2. Please again I would like like that all devices with USB 3xyz are labelled with speeds and power output supported.
    Reply
  • PaulHoule - Monday, October 5, 2020 - link

    The issues they run into here make me think of the practical problems I have with USB.

    I have a few recent Windows laptops that have USB 3 Type A and Type C ports.

    In the old days you were supposed to be able to plug a hub into a host then plug a hub into a hub and do it again and have it work. The spec said you could do it and you really could.

    In the USB3 spec I don't see anything promised as to what kind of configurations are supported and I find I can't take it for granted that I plug my PC into a 4-way hub and can then plug my monitor into the hub and then plug my keyboard and mouse into the monitor.

    Some configurations work but then I plug in the RealSense camera and my mouse stops working; or maybe the SD card reader connects and disconnects all the time and I am always hearing the notification tone for that.

    The hard way I learned "at most four way hubs" and "never plug a hub into a hub" and I've finally settled in on something that works but plug in a USB hard drive and I pray that the filesystem doesn't get corrupted.

    Let's see an expose on that!
    Reply
  • eastcoast_pete - Monday, October 5, 2020 - link

    I know this will read old-fashioned, but I really wish that one of these USB flavors would allow true "serial" use, i.e. the ability to daisy-chain USB devices directly in the plug. With most "ultraportables" now down to 1-2 ports, USB charging is often made absurd by the then-lost connectivity. If one could simply plug another device into the back of the male USB plug, that issue would be moot. Is there such a solution? I will gladly stay on 3.1 or 3.2 if that feature would be enabled. Reply
  • repoman27 - Monday, October 5, 2020 - link

    In theory, they all do. You just need to embed a USB hub to create that topology. It costs money, adds complexity, and consumes power, but plenty of dongles and some chargers already do this. Reply

Log in

Don't have an account? Sign up now