Resolve signal integrity issues in cloud computing platforms
While the cloud promises to reduce enterprise network infrastructure and operating costs, the remote execution of applications makes factors such as latency, performance, and reliability critical considerations in the design and deployment of cloud computing platforms. A key factor determining server performance, cloud or not, is maintaining a bit error rate (BER) on the order of 1.E-12 for the overall system.
Given that a single bit error can necessitate the resending of an entire packet, real-time data performance drops sharply as the BER increases.
Data travels over numerous high-speed interfaces as it passes through the cloud, and poor signal integrity over any of these interfaces is a leading cause for undesirable BER degradation. Thus, as data rates continue to increase, assuring proper signal integrity through the signal channel becomes critical. However, the long trace distances inherent in data center equipment make maintaining signal integrity challenging.
For example, consider a typical server chipset integrating a PCI Express Gen 3 controller with a maximum channel loss spec of 20 dB. At Gen 3's signaling rate of 8 Gbps, maximum trace length, given FR4 PCB losses, typically equates to approximately 18", less with connections and vias. Server motherboards have a deep form factor, however, and include many other sources of attenuation, including multiple connectors and vias that reduce signal integrity and the length over which traces can be driven reliably (see Figure 1). It is also not uncommon for server manufacturers to include mid-plane and daughtercard connectors in the channel path to support different product line options. When this connector is not populated, it has a jumper inserted instead, and the resulting signal path, with all of its sources of loss, is likely to push beyond the specified limits of the interface.
Signal integrity also plays an important role in the server switch fabric. Switches must pass high-speed, 10 Gbps Ethernet signals across the backplane and multiple connectors. Some traces within the switch fabric may run quite long, exceeding the chipset or endpoint’s maximum drive capability. A similar situation arises with storage applications using 12 Gbps SAS.
Restoring signal integrity
Jitter, and the attenuation it results in, is one of the leading causes of signal loss in server applications. There are two basic types of jitter: random and deterministic. Random jitter arises mainly from the clock source as any jitter in the clock will be reflected in every signal associated with that clock. Deterministic jitter is generated primarily by a channel’s characteristics and builds cumulatively as more connectors, vias, and trace lengths are added to the channel.
There are several ways to address signal loss, either by reducing losses or actively restoring signal integrity. Designers can reduce losses by limiting the sources of jitter and attenuation in the system. Two common methods used to achieve this are using a board material that has lower losses than FR4 and by adjusting layout guidelines to use wider traces.
More exotic board materials, while better preserving signal integrity, are more expensive than FR4. In addition, they unnecessarily increase system cost across the entire motherboard, not just for those signals that need added margin. Widening traces also has limited application, given that the motherboard, although large, is already crowded with components.
Alternatively, designers can address signal loss by restoring and boosting signal integrity using active components like redrivers and retimers in the signal channel that condition a signal to compensate for expected losses. This is achieved by applying pre-emphasis to signals on the transmit side to match the loss characteristics of the channel. Similarly, on the receive side, signal integrity is recaptured through equalization to open and restore the signal eye. The result is added signal margin above the standard specification for the interface across PCB traces, connectors, and even cables (see Figure 2).
A redriver, or repeater as it is also known as, amplifies and equalizes a signal at the physical layer. The name repeater comes from the way these components can be placed in the channel to boost and restore signal integrity on very long signal paths. Alternatively, a retimer conditions signals at the protocol level, processing signals at both the PHY and MAC layers. As a result, both clock and data are retimed before the signal is passed on.
Whether to use a redriver or retimer depends upon the application. In general, retimers can provide longer drive distance but, because they process the signal at the protocol level, they cost more and add latency on the order of hundreds of ns. Retimers also require a clock reference, introduce additional traces associated with the clock reference, and consume more power.
In contrast, a redriver comes in a small package that can be placed in the signal path without overtly disrupting the overall board layout. In addition, high-frequency signals can be driven at least double the distance of a signal without signal conditioning. This is more than sufficient for most applications where designers need to extend signal reach to simplify layout and avoid complex techniques to manage signal losses. Because redrivers are transparent to in-band signaling, they can be used in applications utilizing link training sequences. Finally, the low latency of a redriver (<1 ns) is important for high-speed applications such as servers and switches.
An important facet of maintaining reliability and performance of high frequency signals that is often overlooked is to ensure the integrity of the signal at its origin (see Figure 3). Specifically, if the signal clock source has high jitter, this will reduce the quality of the signal output by the interface controller. As a result, part of the signal margin is immediately lost when the signal is generated. By minimizing clock jitter, signal integrity can be maximized from source to destination.
Signal conditioning provides a cost-effective way to maintain signal integrity in server, switching, and other applications utilizing high-frequency signals like PCI Express Gen 3, 10G Ethernet, and SAS. By being able to double the length a signal can be driven while still meeting interface BER specs, designers can guarantee system performance and reliability. In addition, system cost can be minimized since signal conditioning only needs to be applied to those signals that require increased signal integrity.