PCI Express Roadmap Is More Than Speed Bumps

Not content with just cranking-up the bus speed, the PCI-SIG® is driving new protocol extensions that should increase the usage and adoption of PCI Express®. In addition to increasing clock rate by more than three times, the PCIe® roadmap addresses virtualization, latency and power management. These enhancements, which increase the coupling between I/O and compute subsystems, allow data from peripherals to be processed faster. Furthermore, PCIe is becoming more capable to serve other high-performance system needs, such as links between CPUs and special-function accelerators and backplane applications.

During the Intel® Developer Forum in September, Intel Fellow Ajay Bhatt provided a chronology of PCI Express, as shown in the figure. This year PCIe 2.0 and two virtualization specifications were approved, and early details of PCIe 3.0 extensions were released.

PCIe 2.0 Ready for the Holidays

It’s a great time to be a PC gamer, as systems supporting PCIe 2.0 are starting to hit retail shelves. Ever since PCIe 1.0 began replacing AGP back in 2003, high-end graphics cards have been early adopters of the latest PCIe technology. Dell and ASUS launched workstation and highperformance desktop PC motherboards with x16 slots of PCIe 2.0 supported by the Intel® X38 Express chipset. Video enthusiasts can add graphics cards based on the NVIDIA GeForce 8800 GT graphics processing unit (GPU) or the ATI Radeon HD 3800 Series of GPUs.

The most notable PCIe 2.0 enhancement is the doubling of the transfer rate, from 2.5 giga-transfers per second (GT/s) to 5 GT/s, over PCI 1.x. There are other new features including dynamic link speed management and improved control and alert services.

Virtualization Reduces Cost and Complexity

Seemingly every industry publication is espousing the benefits from virtualization in data centers. Virtualization deployed on server clusters – pools of independent servers working together as a single system to provide high availability of services – facilitates load-balancing and reallocating resources to services in high demand. The PCI-SIG is working on bus enhancements that address virtualization and other I/O demands of high-performance infrastructure, such as storage and networking.

Seemingly every industry publication is espousing the benefits from virtualization in data centers. Virtualization deployed on server clusters – pools of independent servers working together as a single system to provide high availability of services – facilitates load-balancing and reallocating resources to services in high demand. The PCI-SIG is working on bus enhancements that address virtualization and other I/O demands of high-performance infrastructure, such as storage and networking.

The PCI-SIG developed several specifications that provide two levels of I/O virtualization (IOV). First, the Address Translation Services (ATS) specification provides a set of transactions for PCI Express components to exchange and use translated addresses in support of I/O virtualization. Second, Single Root IOV allows multiple operating systems running simultaneously within a single computer – single root topography – to natively share PCIe devices.

Currently under review, the Multi-Root IOV specification extends virtualization support to multiple root topographies, such as blade servers. This will simplify the sharing of I/O devices between software applications and server boards. Applications will be able to access I/O and storage devices throughout the network. Network and storage adapters can reside on switches, not on every server blade, which reduces component count and system complexity. Today, most blades with root complexes have their own network adapters, which adds cost and redundancy as peripherals and ports are proliferated across the network infrastructure. In the future, shared network adapters should simplify I/O load balancing and bandwidth management within a virtualized environment.

PCIe 3.0 on the Drawing Board

In August, the PCI-SIG released some details on the next generation PCIe architecture, PCIe 3.0, including its 8 GT/s bit rate and backwards compatibility to prior generations. Completion of the specification is expected late 2009, targeting products for 2010 and beyond. The PCIe 3.0 specification is expected to make significant improvements in throughput, latency and power management as well as incorporate the virtualization enhancements discussed previously. Vendors backing PCIe see opportunities to develop higher performance accelerators that speedup specific tasks such as video, encryption, XML and data mining functions.

After thorough technical analysis, the PCI-SIG approved 8 GT/s for the bit rate, ending debate on a 10 GT/s version. The slower speed offers cost and implementation advantages while simplifying the task of ensuring backward compatibility. Components can be manufactured in mainstream silicon process technology and deployed with existing low-cost materials, while maintaining full mechanical compatibility and imposing negligible changes to the PCIe protocol stack.

One challenge to preserving backward compatibility will be the transition away from 8b/10b encoding to a scrambling technique, where a known binary polynomial is applied to the data stream. The 8b/10b is a code that maintains DCbalance on differential signal lines. On average, two bits are added for every eight data bits, providing enough bus state changes to prevent common-mode voltage shifts and allow clock recovery. Scrambling introduces more DC wander than 8b/10b; therefore, the receiver (Rx) circuit must either tolerate the DC wander as a reduction in signal margin or implement a DC wander correction capability. The choice for the scrambling polynomial is currently under study.

By adding non-data bits to the data stream, 8b/10b encoding imposes a 20 percent overhead on the raw bit rate. By transitioning to scrambling, PCIe 3.0 supports twice the throughput of PCIe 2.0 even though the bit rate increases by just 60 percent, from 5 GT/s to 8 GT/s. This is illustrated in the preceding figure, where PCIe 2.0 and 3.0 achieve approximately 16 GB/s and 32 GB/s for a x16 link, respectively.

The increase in throughput will simplify board designs by reducing lane count. For example, applications deploying 10 giga-bit Ethernet typically require four PCIe 2.0 lanes; this can be reduced to two lanes with PCIe 3.0 in the future. This fanout improvement will ease the design of dense switching boards.

Latency, Software and Power Enhancements

PCIe 3.0 developers are seeking a tighter coupling between I/O and compute sub-systems. They are investigating multiple protocol extensions and enhancements that help server CPUs access priority I/O data faster. There will be mechanisms that provide “data re-use hints” that improve the caching of reusable data in system memory, thereby reducing data latency. Another enhancement supports transaction attributes and hints that optimize the ordering of transactions within the root complex and the memory subsystem. Also under consideration are Pause and Resume operations that control the interrupts of low priority transmissions and allow higher priority transaction to take precedence.

As PCIe evolves beyond standard I/O interconnect to hardware accelerators, the specification requires enhancements to help developers maintain memory coherency within a system. Hardware accelerators, unlike most I/O adapters, have their own local memory that must remain coherent with other memory sub-systems on the network. PCIe 3.0 is investigating software model enhancements, such as atomic read-modify-write mechanisms, that prevent network elements from accessing stale or corrupt data. These mechanisms help avoid having a CPU access data while a special-function accelerator is working on it.

Power management features will be added to support dynamic performance/ power operation modes. System software will be able to dynamically adjust the power consumption of endpoints in accordance to I/O throughput requirements.

PCI Express In Backplane Applications?

Today, Ethernet is dominant in the backplane space. Its large installed base, supported by a large software investment, has proven to be very dependable. Yet, some developers are asking whether PCI Express may make inroads as bus speeds increase to 10 gigabit. Like Ethernet, PCI Express is becoming ubiquitous with many chipsets supporting it natively. It also leverages years of legacy software dating back to the 1990’s.

PCI Express offers some advantages over Ethernet. It is far more scalable, with options to utilize links with difference lane sizes, x1 all the way up to x32. PCI express also has lower latency and overhead (packet header) than Ethernet, making it faster and more efficient. These differences are magnified when packet sizes are small, as in control plane applications. PCI Express also has several QoS features, such as flow control and guaranteed error-free packets and delivery, whereas Ethernet is less proactive and requires receivers to notify the sender of dropped packets.

During the transition from one to ten gigabits, some developers utilizing smaller payloads with heavy processing requirements may give PCI Express consideration for the backplane. Applications such as communications, military and high-end medical equipment may benefit from greater scalability, lower overhead, lower latency and the cost efficiency of PCI Express.

PCI Express technology has moved well beyond graphics applications, expanding into communications, embedded systems and home entertainment. And with the addition of I/O virtualization and system level enhancements, those who saw PCIe as a bridging or transitional technology may have a change of heart. The combination of a large installed base, extensive ecosystem and a long term view may broaden the adoption of PCIe into emerging applications and new usage models.

Craig Szydlowski is a writer specializing in business and technology. He has over 20 years of engineering and marketing experience with embedded and communications systems at Intel, IBM and Siemens.

.