This morning the PCI Special Interest Group (PCI-SIG) is releasing the much-awaited final (1.0) specification for PCI Express 6.0. The next generation of the ubiquitous bus is once again doubling the data rate of a PCIe lane, bringing it to 8GB/second in each direction – and far, far higher for multi-lane configurations. With the final version of the specification now sorted and approved, the group expects the first commercial hardware to hit the market in 12-18 months, which in practice means it should start showing up in servers in 2023.

First announced in the summer of 2019, PCI Express 6.0 is, as the name implies, the immediate follow-up to the current-generation PCIe 5.0 specification. Having made it their goal to keep doubling PCIe bandwidth roughly every 3 years, the PCI-SIG almost immediately set about work on PCIe 6.0 once the 5.0 specification was completed, looking at ways to once again double the bandwidth of PCIe. The product of those development efforts is the new PCIe 6.0 spec, and while the group has missed their original goal of a late 2021 release by mere weeks, today they are announcing that the specification has been finalized and is being released to the group’s members.

As always, the creation of an even faster version of PCIe technology has been driven by the insatiable bandwidth needs of the industry. The amount of data being moved by graphics cards, accelerators, network cards, SSDs, and other PCIe devices only continues to increase, and thus so must bus speeds to keep these devices fed. As with past versions of the standard, the immediate demand for the faster specification comes from server operators, whom are already regularly using large amounts of high-speed hardware. But in due time the technology should filter down to consumer devices (i.e. PCs) as well.

By doubling the speed of a PCIe link, PCIe 6.0 is an across-the-board doubling of bandwidth rates. X1 links move from 4GB/second/direction to 8GB/second/direction, and that scales all the way up to 128GB/second/direction for a full x16 link. For devices that are already suturing a link of a given width, the extra bandwidth represents a significant increase in bus limits; meanwhile for devices that aren’t yet saturating a link, PCIe 6.0 offers an opportunity to reduce the width of a link, maintaining the same bandwidth while bringing down hardware costs.

PCI Express Bandwidth
(Full Duplex: GB/second/direction)
Slot Width PCIe 1.0
(2003)
PCIe 2.0
(2007)
PCIe 3.0
(2010)
PCIe 4.0
(2017)
PCIe 5.0
(2019)
PCIe 6.0
(2022)
x1 0.25GB/sec 0.5GB/sec ~1GB/sec ~2GB/sec ~4GB/sec 8GB/sec
x2 0.5GB/sec 1GB/sec ~2GB/sec ~4GB/sec ~8GB/sec 16GB/sec
x4 1GB/sec 2GB/sec ~4GB/sec ~8GB/sec ~16GB/sec 32GB/sec
x8 2GB/sec 4GB/sec ~8GB/sec ~16GB/sec ~32GB/sec 64GB/sec
x16 4GB/sec 8GB/sec ~16GB/sec ~32GB/sec ~64GB/sec 128GB/sec

PCI Express was first launched in 2003, and today’s 6.0 release essentially marks the third major revision of the technology. Whereas PCIe 4.0 and 5.0 were “merely” extensions to earlier signaling methods – specifically, continuing to use PCIe 3.0’s 128b/130b signaling with NRZ – PCIe 6.0 undertakes a more significant overhaul, arguably the largest in the history of the standard.

In order to pull of another bandwidth doubling, the PCI-SIG has upended the signaling technology entirely, moving from the Non-Return-to-Zero (NRZ) tech used since the beginning, and to Pulse-Amplitude Modulation 4 (PAM4).

As we wrote at the time that development on PCIe 6.0 was first announced:

At a very high level, what PAM4 does versus NRZ is to take a page from the MLC NAND playbook, and double the number of electrical states a single cell (or in this case, transmission) will hold. Rather than traditional 0/1 high/low signaling, PAM4 uses 4 signal levels, so that a signal can encode for four possible two-bit patterns: 00/01/10/11. This allows PAM4 to carry twice as much data as NRZ without having to double the transmission bandwidth, which for PCIe 6.0 would have resulted in a frequency around 30GHz(!).
 
PAM4 itself is not a new technology, but up until now it’s been the domain of ultra-high-end networking standards like 200G Ethernet, where the amount of space available for more physical channels is even more limited. As a result, the industry already has a few years of experience working with the signaling standard, and with their own bandwidth needs continuing to grow, the PCI-SIG has decided to bring it inside the chassis by basing the next generation of PCIe upon it.

The tradeoff for using PAM4 is of course cost. Even with its greater bandwidth per Hz, PAM4 currently costs more to implement at pretty much every level, from the PHY to the physical layer. Which is why it hasn’t taken the world by storm, and why NRZ continues to be used elsewhere. The sheer mass deployment scale of PCIe will of course help a lot here – economies of scale still count for a lot – but it will be interesting to see where things stand in a few years once PCIe 6.0 is in the middle of ramping up.

Meanwhile, not unlike the MLC NAND in my earlier analogy, because of the additional signal states a PAM4 signal itself is more fragile than a NRZ signal. And this means that along with PAM4, for the first time in PCIe’s history the standard is also getting Forward Error Correction (FEC). Living up to its name, Forward Error Correction is a means of correcting signal errors in a link by supplying a constant stream of error correction data, and it’s already commonly used in situations where data integrity is critical and there’s no time for a retransmission (such as DisplayPort 1.4 w/DSC). While FEC hasn’t been necessary for PCIe until now, PAM4’s fragility is going to change that. The inclusion of FEC shouldn’t make a noticeable difference to end-users, but for the PCI-SIG it’s another design requirement to contend with. In particular, the group needs to make sure that their FEC implementation is low-latency while still being appropriately robust, as PCIe users won’t want a significant increase in PCIe’s latency.

It’s worth noting that FEC is also being paired with Cyclic Redundancy Checking (CRC) as a final layer of defense against bit errors. Packets that, even after FEC still fail a CRC – and thus are still corrupt – will trigger a full retransmission of the packet.

The upshot of the switch to PAM4 then is that by increasing the amount of data transmitted without increasing the frequency, the signal loss requirements won’t go up. PCIe 6.0 will have the same 36dB loss as PCIe 5.0, meaning that while trace lengths aren’t officially defined by the standard, a PCIe 6.0 link should be able to reach just as far as a PCIe 5.0 link. Which, coming from PCIe 5.0, is no doubt a relief to vendors and engineers alike.

Alongside PAM4 and FEC, the final major technological addition to PCIe 6.0 is its FLow control unIT (FLIT) encoding method. Not to be confused with PAM4, which is at the physical layer, FLIT encoding is employed at the logical level to break up data into fixed-size packets. It’s by moving the logical layer to fixed size packets that PCIe 6.0 is able to implement FEC and other error correction methods, as these methods require said fixed-size packets. FLIT encoding itself is not a new technology, but like PAM4, is essentially being borrowed from the realm of high-speed networking, where it’s already used. And, according to the PCI-SIG, it’s one of the most important pieces of the specification, as it’s the key piece to enabling (continued) low-latency operation of PCIe with FEC, as well as allowing for very minimal overhead. All told, PCI-SIG considers PCIe 6.0 encoding to be a 1b/1b encoding method, as there’s no overhead in the data encoding itself (there is however overhead in the form of additional FEC/CRC packets).

As it’s more of an enabling piece than a feature of the specification, FLIT encoding should be fairly invisible to users. However, it’s important to note that the PCI-SIG considered it important/useful enough that FLIT encoding is also being backported in a sense to lower link rates; once FLIT is enabled on a link, a link will remain in FLIT mode at all times, even if the link rate is negotiated down. So, for example, if a PCIe 6.0 graphics card were to drop from a 64 GT/s (PCIe 6.0) rate to a 2.5GT/s (PCIe 1.x) rate to save power at idle, the link itself will still be operating in FLIT mode, rather than going back to a full PCIe 1.x style link. This both simplifies the design of the spec (not having to renegotiate connections beyond the link rate) and allows all link rates to benefit from the low latency and low overhead of FLIT.

As always, PCIe 6.0 is backwards compatible with earlier specifications; so older devices will work in newer hosts, and newer devices will work in older hosts. As well, the current forms of connectors remain supported, including the ubiquitous PCIe card edge connector. So while support for the specification will need to be built into newer generations of devices, it should be a relatively straightforward transition, just like previous generations of the technology.

Unfortunately, the PCI-SIG hasn’t been able to give us much in the way of guidance on what this means for implementations, particularly in consumer systems – the group just makes the standard, it’s up to hardware vendors to implement it. Because the switch to PAM4 means that the amount of signal loss for a given trace length hasn’t gone up, conceptually, placing PCIe 6.0 slots should be about as flexible as placing PCIe 5.0 slots. That said, we’re going to have to wait and see what AMD and Intel devise over the next few years. Being able to do something, and being able to do it on a consumer hardware budget are not always the same thing.

Wrapping things up, with the PCIe 6.0 specification finally completed, the PCI-SIG tells us that, based on previous adoption timelines, we should start seeing PCIe 6.0 compliant hardware hit the market in 12-18 months. In practice this means that we should see the first server gear next year, and then perhaps another year or two for consumer gear.

Source: PCI-SIG

POST A COMMENT

77 Comments

View All Comments

  • back2future - Tuesday, January 11, 2022 - link

    this further: ~1TB/s/pin would be PCIe12 (~2040, on 3yrs cadence), being one pin on peripheral component interface express, same bandwidth like todays registers/top level cache (L1) ? Reply
  • Kamen Rider Blade - Tuesday, January 11, 2022 - link

    What if they swapped to a 4-7 year cadence?

    PCIe 3.x -> 4.0 was a 7 year gap.

    Asking them to innovate every 3 years is pushing it.

    I think every 4 years, like the olympics is a more reasonable ask with every now and then having a 7 year gap.
    Reply
  • mode_13h - Wednesday, January 12, 2022 - link

    It seems to me there are two factors driving the cadence. First, there needs to be a demand for more bandwidth, which seems (recently) to have been supplied by ever-faster network links, SSDs, and deep learning. I doubt those will slow much, anytime soon.

    However, then you have the technology development that's needed to deliver new speeds in a cost-effective way. And that seems harder to predict. The PCI SIG can only move forward with a new spec, once the enabling technologies are proven.
    Reply
  • back2future - Wednesday, January 12, 2022 - link

    guess 2020 combined bandwidth of global internet infrastructure capacity was ~75TBps, what are ~10 PCIe12 x16, half duplex mode, network cards (or 1280 PCIe5 x16)?

    (PAM-16 was copied from above link: each position 2/4/8/16 level modulated pulse, seem to be a clock rate (count) reduction by half for each modulation increase and 16bit words?)
    Reply
  • back2future - Wednesday, January 12, 2022 - link

    more or less PCI-SIG's roadmap towards what's possible in todays silicon and copper (within a cpu and nm scale), but with doubling the bandwidth 'til then, would be necessary for a peripherals interface _ Whoo, let engineers surprise us. (L1 cache is todays highest bw bus speeds, maybe 256bit bus width, enabling ~1-xTB/s bus speed, 1 or 2 pins then would enable that for peripherals, like 1pin PCIe6 replaces a 16x PCIe2.x bus)
    not saying it's not possible, but maybe needs light, graphene, quantum theory
    Reply
  • back2future - Wednesday, January 12, 2022 - link

    correction: lane instead of pin ( like one lane (x1) = 4 differential pairs for electrical connection ) Reply
  • back2future - Wednesday, January 12, 2022 - link

    correction: lane instead of pin ( like one lane (x1) = 2 differential pairs for electrical connection ) Reply
  • back2future - Wednesday, January 12, 2022 - link

    What's the reason PCI-SIG doesn't show a renewed roadmap (this 5/1/2022) including PCIe7?
    (maybe a doubling bandwidth claim on an average 3 years pace is only valid up to PCIe6 within their planings, but that would not sound like PCI-SIG?)
    Reply
  • back2future - Wednesday, January 12, 2022 - link

    Statista.com
    Revenue of the computer hardware market worldwide 2012-2025, by segment
    "In 2020, the global computer hardware market generated a total revenue of over 298 billion U.S. dollars. The two segments that generated the majority of the revenue were laptop sales, with around 123 billion U.S. dollars of revenue and tablets with around 57 billion U.S. dollars in revenue. The Statista Consumer Market Outlook estimates that a peak revenue in 2021 will be followed by a period of decline until revenue across all segments will again increase in 2025."

    various
    mobile equipment probably not the market for PCIe6 volume (still ~118bnUSD left)
    seen numbers for global mainboard market ~13bnUSD for 2021 (annual growth ~1.4%)
    at the moment PCIe seems being a ~20-25bnUSD global market (expected growth to ~48bnUSD ~2026, annual growth ~4-5%, with ~40% origin from NorthAmerica, most from switches (~9bnUSD share) and storage (been ~7bnUSD 2020) )
    (2020: dominant PCIe3 was ~8.6bnUSD, dominant geographical region APAC ~7.9bnUSD before North America, Europe)

    Statista.com
    global IT device spending 2021 ~802bn USD (forecast 2022 ~821bnUSD)

    Global IT spending forecast 2012-2022, by segment
    "The global information technology (IT) spending on devices, including PCs, tablets, mobile phones, printers, as well as data center systems, enterprise software, and communications services came to 3.87 trillion U.S. dollars in 2020 and is expected to increase by approximately 9.5 percent to around 4.24 trillion U.S. dollars in 2021. This is likely due to an increase in demand for technological devices for remotely working employees, which has surged since the outbreak of the COVID-19 pandemic."
    datacenter systems ~200bnUSD
    enterprise software ~680bnUSD
    devices ~821bnUSD
    IT services ~1294bnUSD
    Communication services ~1482bnUSD

    maybe not (especially) for PCIe, but customers for consumer hardware are a reliable source for production volume for industrial revenue (maybe therefore PCI-SIG won't increase a spreading between standards and availability between datacenters and consumer desktop/workstation and mobile equipment too much)
    gap between PCIe3 and PCIe4 - lower investment funding and technical delays from supporters?
    there was a time with lowest spendings for global IT around 2015-2016,
    first PCIe4 hardware announces started 2016 (PCIe5 ~Q4/2019-Q2020 with controller and cpu integration) and PCIe retimer market got visible ~2016
    Reply
  • mode_13h - Thursday, January 13, 2022 - link

    > The Statista Consumer Market Outlook estimates that a peak revenue in 2021
    > will be followed by a period of decline

    I'm not sure about that. There's a lot of pent-up demand that's waiting for prices to come down & availability to increase. It's a bold move to be making predictions in these times, but I understand that's their whole business.

    > maybe therefore PCI-SIG won't increase a spreading between standards and availability
    > between datacenters and consumer desktop/workstation and mobile equipment too much

    This makes no sense to me. They're going to try to keep pace with datacenter needs, or risk losing that market almost entirely. PCI SIG members who need faster interconnects for future datacenter products are going to try to push it into that role, and if they meet too much resistance, they'll go elsewhere.
    Reply

Log in

Don't have an account? Sign up now