Exceeding the Speed Limit
Exceeding the Speed Limit
Challenges and solutions to the support of serial 10
Gigabit Ethernet over backplane interconnects
Adam Healey
Chair, IEEE P802.3ap Task Force
Consulting Member of Technical Staff, Agere Systems
March 10, 2006
Before we begin…
Per IEEE-SA Standards Board Operations Manual, January 2005:
• At lectures, symposia, seminars, or educational courses, an individual
presenting information on IEEE standards shall make it clear that his or
her views should be considered the personal views of that individual
rather than the formal position, explanation, or interpretation of the IEEE.
Note:
• Currently, IEEE P802.3ap is under review per the IEEE 802.3 Working
Group ballot process and is subject to change.
• The work in this presentation is per IEEE P802.3ap Draft 2.1.
2
March 10, 2006 (r1.0)
Agenda
Introduction to Backplane Ethernet
Backplane Architecture and Reference Model
10GBASE-KR
Forward Error Correction for 10GBASE-KR
Closing Remarks
3
March 10, 2006 (r1.0)
Backplane Ethernet Profile
“Ethernet in a box”
Target Applications
• Wireline and wireless access equipment
• Blade servers
• Enterprise switching
Leverages field proven Ethernet controller and switching IP
Reduces solution cost and complexity
• Leverages Ethernet economies of scale
• Flattens the backplane fabric eliminating encapsulation protocols and
associated processing
• Standard enables multi-vendor interoperability and facilitates use of
COTS devices
4
March 10, 2006 (r1.0)
Specification Methodology
Supply a device specification and not a backplane specification
Be application agnostic (do no specify the backplane connector)
Supply informative recommendations to assist backplane designers in
identifying backplane channels that are interoperable with “Backplane
Ethernet compliant” devices
5
March 10, 2006 (r1.0)
Agenda
Introduction to Backplane Ethernet
Backplane Architecture and Reference Model
10GBASE-KR
Forward Error Correction for 10GBASE-KR
Closing Remarks
6
March 10, 2006 (r1.0)
Backplane Architecture and Topology
via
Blade may be a generic µP and
memory complex with an I/O-specific
mezzanine card
Backplane
connector
H
Pin-out
example
AC-coupling
capacitor (at
receiver)
A B BG C D DG E F FG G H HG
Hub Slot X
Channel (N+1)
“stub”
Hub Slot X
Channel N
N2
Port 2
Port 3
Port 0
Port 1
TX +/-
RX +/-
B
7
March 10, 2006 (r1.0)
N1
TX +/-
RX +/-
Reference Model
The transmitter and receiver blocks include all off-chip components
associated with the respective block
• Example, external AC-coupling capacitors, if required, are included in the
receiver block
Channel consists of line card and backplane traces plus associated
backplane and mezzanine connectors
TP1
TP4
package
Transmitter
8
March 10, 2006 (r1.0)
package
Channel
Receiver
Backplane Ethernet Objectives
Support up to 1 m differential traces, including two connectors, on
improved FR-4 printed circuit boards
Support a BER of 10-12 or better
N1
(mm)
N2
(mm)
B
(mm)
H
(mm)
Total
(mm)
No.
Connectors
AC / DC
Coupling
Example (dual-star)1
0
102
244
102
447
2
AC
Full mesh2
0
127
533
127
787
2
AC
76
102
533
127
838
3
AC
2 to 3 chassis/rack4
0
152
559
305
1016
2
AC or DC
5 to 8 chassis/rack4
0
127
432
229
787
2
AC or DC
Description
AdvancedTCA
Blade Server
Proposed worst-case3
Switch / Router
1
9
kundu_01_0504, 2 PICMG 3.0 R2.0, March 18, 2005, 3 koenen_01_0504, 4 goergen_01_0304
March 10, 2006 (r1.0)
Fitted Attenuation
Defined to be the least-mean squares line fit to the insertion loss data
within the frequency range f1 and f2
• The values of f1 and f2 are dependent on the PHY type of interest
Fitted attenuation is compared to a limit based on a 1 m differential
trace (W = 6 mil) on an improved FR-4 printed circuit board
0
5
5
10
10
15
20
1500
2000
2500
3000 3500 4000
Frequency [MHz]
4500
Example #1
10
10
15
20
March 10, 2006 (r1.0)
5000
5500
6000
30
1000
15
20
25
25
25
30
1000
5
Insertion Loss [dB]
0
Insertion Loss [dB]
Insertion Loss [dB]
In effect, constrains the dielectric loss-length product of the channel
1500
2000
2500
3000 3500 4000
Frequency [MHz]
4500
Example #2
5000
5500
6000
30
1000
1500
2000
2500
3000 3500 4000
Frequency [MHz]
4500
Example #3
5000
5500
6000
Insertion Loss Limits
Discourages encroachment of stub-related resonances
into critical frequencies for the PHY type of interest
0
0
10
10
10
20
20
20
30
40
50
Insertion Loss [dB]
0
Insertion Loss [dB]
Insertion Loss [dB]
Based on the fitted attenuation limit with allowances for
passband ripple and stub-related resonances
30
40
50
30
40
50
60
60
60
70
70
70
80
0
5000
10000
Frequency [MHz]
Example #1
11
March 10, 2006 (r1.0)
15000
80
0
5000
10000
Frequency [MHz]
Example #2
15000
80
0
5000
10000
Frequency [MHz]
Example #3
15000
Insertion Loss Deviation
Defined to the be the difference between the channel
insertion loss and the fitted attenuation in the frequency
range f1 to f2
4
4
3
3
3
2
2
2
1
0
-1
-2
1500
2000
2500
3000 3500 4000
Frequency [MHz]
4500
Example #1
12
1
0
-1
-2
-3
-3
-4
1000
Insertion Loss Deviation [dB]
4
Insertion Loss Deviation [dB]
Insertion Loss Deviation [dB]
In effect, constrains passband ripple due to impedance
mismatches in the transmission path
March 10, 2006 (r1.0)
5000
5500
6000
-4
1000
1
0
-1
-2
-3
1500
2000
2500
3000 3500 4000
Frequency [MHz]
4500
Example #2
5000
5500
6000
-4
1000
1500
2000
2500
3000 3500 4000
Frequency [MHz]
4500
Example #3
5000
5500
6000
Crosstalk
Total crosstalk is defined to the be the power sum of the individual
aggressors as measured at TP4
Includes near-end crosstalk (NEXT) and far-end crosstalk (FEXT)
Assumes the aggressors and victim are asynchronous and (or)
uncorrelated
0
0
10
10
10
20
20
20
30
40
50
Crosstalk Loss [dB]
0
Crosstalk Loss [dB]
Crosstalk Loss [dB]
Assumes the aggressors and victim are driven by identical sources
30
40
50
30
40
50
60
60
60
70
70
70
80
13
0
5000
10000
15000
80
0
5000
10000
15000
80
0
5000
10000
Frequency [MHz]
Frequency [MHz]
Frequency [MHz]
Example #1
Example #2
Example #3
March 10, 2006 (r1.0)
15000
Insertion Loss to Crosstalk Ratio (ICR)
Defined to be the least-mean squares line fit to the difference of the
total crosstalk loss and the insertion loss (both in dB) in the frequency
range fa to fb
Limits the total crosstalk seen at TP4
50
50
45
45
45
40
40
40
35
30
25
20
15
10
5
0
100
35
30
25
20
15
10
5
1000
Frequency [MHz]
Example #1
14
Insertion Loss to Crosstalk Ratio [dB]
50
Insertion Loss to Crosstalk Ratio [dB]
Insertion Loss to Crosstalk Ratio [dB]
Lower loss links are allowed higher crosstalk and vice versa
March 10, 2006 (r1.0)
10000
0
100
35
30
25
20
15
10
5
1000
Frequency [MHz]
Example #2
10000
0
100
1000
Frequency [MHz]
Example #3
10000
Differential Skew
T1
vi
2
vo
T2
−
vi
2
The skew is the magnitude of
the difference between the two
delays, |T1−T2|
1
0.9
The differential skew from TP1
to TP4 is recommended to be
less than 24 ps (10GBASE-KR)
0.8
Gain (W/W)
0.7
0.6
0.5
0.4
2
Vo (ω )
1
= (1 + cos(ω (T2 − T1 )))
Vi (ω )
2
0.3
0.2
0.1
0
15
To isolate the impact of skew,
render the <p> and <n> sides
of the differential pair as delay
T1 and T2 respectively
0
0.25
0.5
0.75
Frequency x Differential Skew
March 10, 2006 (r1.0)
1
Environmental Variation
Channel recommendations are to be satisfied over the full
range of system operating conditions
0
Insertion loss (dB)
20oC, 20% RH
5
40oC, 20% RH
10
20oC, 85% RH
60oC, 20% RH
40oC, 85% RH
60oC, 85% RH
15
20
25
Insertion loss variation due to
temperature and humidity
30
35
40
16
March 10, 2006 (r1.0)
0
1000
2000
3000
4000
Frequency (MHz)
5000
6000
Impact of Device Terminations
1
ZS
2
1
S11
S12
S 21
S 22
2
vi
Transmitter
ΓS
ΓL
1
ZL
vo
Receiver
2
0
2
Insertion loss (dB)
5
vo
vi
10
(1 + ΓL )(1 − ΓS )
vo S 21
=
vi
2 1 − ΓS S11 − ΓL S 22 − ΓS ΓL S12 S 21 + ΓS ΓL S11S 22
15
20
S 21
25
30
17
0
1000
2000
3000
4000 5000 6000
Frequency (MHz)
March 10, 2006 (r1.0)
7000
8000
9000 10000
Agenda
Introduction to Backplane Ethernet
Backplane Architecture and Reference Model
10GBASE-KR
Forward Error Correction for 10GBASE-KR
Closing Remarks
18
March 10, 2006 (r1.0)
10GBASE-KR Architecture
10 Gb/s connection between two
media access controllers over an
electrical backplane
HIGHER LAYERS
Low-overhead 64B/66B encoding
MAC CONTROL (OPTIONAL)
Statistical transition density and
DC balance (scrambled)
Signaling speed is 10.3125 Gbaud
Optional Forward Error Correction
Transmitter equalizer programmed
by the link partner during start-up
LLC
MAC
RECONCILIATION
XGMII *
64B/66B PCS
FEC*
PMA
10GBASE-KR
PHY
PMD
AN
Adaptive receiver equalization
MEDIUM
19
March 10, 2006 (r1.0)
* Optional
Transmit Equalizer Signal Shaping
c1
c0 c−1
c1
V pk
c0 c−1
V pst
Vss
c1
c0 c−1
0
V pre
c1
c0
c−1
− V pk
T
T
V pre = −c1 − c0 + c−1
Vss = c1 + c0 + c−1
V pst = −c1 + c0 + c−1
V pk = c1 + c0 + c−1
Assumes
Assumesthat
thatthe
theimplementation
implementationisisaathree-tap
three-taptransversal
transversalfilter
filterwith
withunit
unitdelay.
delay.
20
March 10, 2006 (r1.0)
Receive Equalizer
Requirements developed under the assumption of a 5-tap
decision feedback equalizer
However, the implementation of this architecture is not
necessarily required for compliance
Instead, the receiver in question must exhibit an expected
level of performance as established via the interference
tolerance test
The interference tolerance test examines the ability of the
receiver to equalize a high-loss channel in the presence of
horizontal (jitter) and vertical (noise) interference
21
March 10, 2006 (r1.0)
Interference Tolerance Testing
Transmitter signal shaping
controlled by adaptation
algorithm in the DUT
Pattern
generator
Transmitter
control
Equalizer
Frequencydependent
attenuator
Interference
injection
Device
under test
Test channel
Clock
source
Modulation input
Frequency
synthesizer
Interference
generator
Introduces static timing
offset and jitter
22
March 10, 2006 (r1.0)
Additive sinusoid swept in
frequency and amplitude
Insertion loss emulates
worst-case attenuation
10GBASE-KR Link Model
At
F (z )
2.5
t − kT
Π
T
2
Ht ( f )
1.5
1
Amplitude (au)
xk
Slicer input eye
0.5
0
-0.5
-1
-1.5
-2
Transmitter output eye
H0 ( f )
0.5
-2.5
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
Time Offset (baud)
0.2
0.3
0.4
0.5
0.4
0.3
εk
Amplitude [au]
0.2
0.1
0
ts + k
-0.1
-0.2
Hr ( f )
-0.3
T
M
C (z )
x̂k
↓M
-0.4
-0.5
-0.5
23
0
0.5
Time Offset [UI]
1
March 10, 2006 (r1.0)
1.5
Gr ( f )
D(z )
10GBASE-KR Performance Simulations
σ1
Slicer Input Eye: Test Case D
Crosstalk-limited cases
2
1.5
A1
Normalized Amplitude
1
0.5
0
-0.5
A0
-1
-1.5
-2
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
Time Offset [UI]
0.2
SNR =
Loss-limited case
Stub-limited case
Required SNR for 1E-12
Bit Error Ratio (BER)
0.3
0.4
0.5
σ0
A1 − A0
σ1 + σ 0
10GBASE-KR Start-Up Protocol
Optimizes transmitter FIR
Automatic power control
• Receiver may steer the transmitter output voltage to the minimum
level required for acceptable performance
• May also mitigate crosstalk
Optimize receiver equalizer
• Joint adaptation of transmitter and receiver yields superior solution
to independent adaptation
Accomplished through the exchange of fixed-length
training frames
25
March 10, 2006 (r1.0)
Training Frame Structure
Octets
4
16
Coefficient update
16
Status report
512
Control channel is differential Manchester encoded
at one-quarter of the 10GBASE-KR signaling speed
(one octet per control bit)
Two cycles of a PRBS-11 pattern (seeded with all
ones) padded with two zeros
Training pattern
15
0
UG
Reserved
Update gain
11 = 1X
10 = 2X
01 = 4X
00 = 8X
26
0xFFFF0000 (does not appear in control channel or
training pattern)
Frame marker
March 10, 2006 (r1.0)
c(1) c(0) c(−1)
Coefficient update
11 = reserved
10 = increment
01 = decrement
00 = hold
15
RR
0
Reserved
Receiver Ready
1 = Training complete
0 = Training in progress
c(1) c(0) c(−1)
Coefficient status
11 = maximum
10 = minimum
01 = updated
00 = not_updated
Timing Diagram
ReceiverRdy = 0
ReceiverRdy = 1
Device A
Training Frames
Training Frames
Equalizer Training Period
IDLE and DATA
wait_timer
Auto-Negotiation
Equalizer Training Period
Device B
Training Frames
ReceiverRdy = 0
27
March 10, 2006 (r1.0)
wait_timer
Training Frames
ReceiverRdy = 1
IDLE and DATA
Agenda
Introduction to Backplane Ethernet
Backplane Architecture and Reference Model
10GBASE-KR
Forward Error Correction for 10GBASE-KR
Closing Remarks
28
March 10, 2006 (r1.0)
Forward Error Correction for 10GBASE-KR
FEC improves 10GBASE-KR link
performance at the expense of
added complexity and latency
Not required to meet objectives
Burst error correction especially
suitable for DFE-based receiver
architecture
• Counter-acts error propagation
Signaling speed is not altered by
FEC
HIGHER LAYERS
LLC
MAC CONTROL (OPTIONAL)
MAC
RECONCILIATION
XGMII *
64B/66B PCS
FEC*
PMA
PMD
AN
MEDIUM
29
March 10, 2006 (r1.0)
10GBASE-KR
PHY
FEC Frame Structure
Transcode 66B blocks into FEC frame (32 blocks/frame)
• T-bit is the second bit of the 66B sync header (0 = control, 1 =
data)
Append 32 parity bits (total frame length is 2112 bits)
T
64-bit
payload
1
T
64-bit payload
2
T
64-bit payload word 3
Transmission
order
isword
left-to-right
and word
top-to-bottom
T0
64-bit payload word 0
1
T4
64-bit payload word 4
T5
64-bit payload word 5
T6
64-bit payload word 6
T7
64-bit payload word 7
T8
64-bit payload word 8
T9
64-bit payload word 9
T10
64-bit payload word 10
T11
64-bit payload word 11
T12
64-bit payload word 12
T13
64-bit payload word 13
T14
64-bit payload word 14
T15
64-bit payload word 15
T16
64-bit payload word 16
T17
64-bit payload word 17
T18
64-bit payload word 18
T19
64-bit payload word 19
T20
64-bit payload word 20
T21
64-bit payload word 21
T22
64-bit payload word 22
T23
64-bit payload word 23
T24
64-bit payload word 24
T25
64-bit payload word 25
T26
64-bit payload word 26
T27
64-bit payload word 27
T28
64-bit payload word 28
T29
64-bit payload word 29
T30
64-bit payload word 30
T31
64-bit payload word 31
32 parity bits
30
March 10, 2006 (r1.0)
2
3
FEC Encoder and Decoder
Parity bits computed using a shortened cyclic code (2112,
2080)
Frame is then scrambled
Approximately 2 dB coding gain
Guaranteed error correction for bursts up to 11 bits in
length
Complexity estimated to be 14K gates plus a 64 x 33 dual
port RAM at 312.5 MHz
Latency estimated to be 200 ns
31
March 10, 2006 (r1.0)
Agenda
Introduction to Backplane Ethernet
Backplane Architecture and Reference Model
10GBASE-KR
Forward Error Correction for 10GBASE-KR
Closing Remarks
32
March 10, 2006 (r1.0)
Observations
Backplane channels are not as predictable or easily
classified as cabling channels
Backplane channel length is a misleading metric
• Significant passband ripple, stub-related resonances, or crosstalk
coupling may make short links unworkable
• With proper material selection and design practices, 1 m links are
feasible
Backplane behavior may differ significantly from what is
measured in the lab
• Ideal terminations are replaced with actual devices
• Temperature, humidity, and resin tolerances can cause variability
in backplanes in the field – margin must be allocated
33
March 10, 2006 (r1.0)
Conclusions
IEEE P802.3ap defines a comprehensive Physical Layer
suite to support Ethernet as a backplane fabric
• Addresses the absence of an industry standard for Backplane
Ethernet
• Provides the industry a roadmap to 10 Gb/s serial transmission
over an electrical backplane
Still IEEE P802.3ap is only a piece of an Ethernet fabric
solution
• Improvements to congestion management are necessary to make
Ethernet a viable storage or IPC solution
• Improvements to congestion management enhance networking
and transport performance
34
March 10, 2006 (r1.0)
Thank you!
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising