Cisco | N7K-C7009 | Fabric Modules

Fabric Modules
Cisco Nexus 7000 Switch
Architecture
BRKARC-3470
BRKARC-3470
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
2
Session Goal
• To provide a thorough understanding of the Cisco Nexus™ 7000 switching
architecture, supervisor, fabric, and I/O module design, packet flows, and
key forwarding engine functions
• This session will examine only the latest additions to the Nexus 7000
platform
• This session will not examine NX-OS software
architecture or other Nexus platform architectures
3
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
3
What is Nexus 7000?
Data-center class Ethernet switch designed to deliver high-availability,
system scale, usability, investment protection
Supervisor Engines
I/O Modules
Chassis
Fabrics
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
4
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
5
Nexus 7000 Chassis Family
Nexus 7010
NX-OS 4.1(2) and later
Nexus 7018
25RU
21RU
Front
N7K-C7010
Rear
Front
N7K-C7018
Rear
NX-OS 5.2(1) and later
NX-OS 6.1(2) and later
Nexus 7004
Nexus 7009
14RU
7RU
Front
Front
Presentation_ID
BRKARC-3470
N7K-C7009
N7K-C7004
Rear
Rear
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
6
Supported in NX-OS release 6.1(2) and later
Nexus 7004 Chassis
• 4 slot chassis – 2 payload slots, 2
supervisor slots
• No fabric modules – I/O modules
connect back-to-back
• Side-to-back airflow
• 3 X 3000W power supplies (AC or DC)
• All FRUs accessed from chassis front
• Supports Sup2 / 2E only
• Supports M1L, M2, F2, F2E modules
‒ No support for M1 non-L, F1 modules
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
7
Key Chassis Components
• Common components:
‒ Supervisor Engines
‒ I/O Modules
‒ Power Supplies (except 7004)
• Chassis-specific components:
‒ Fabric Modules
‒ Fan Trays
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
8
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
9
Supervisor Engine 2 / 2E
• Next generation supervisors providing control plane and management functions
Supervisor Engine 2
Supervisor Engine 2E
Base performance
High performance
One quad-core 2.1GHz CPU with 12GB DRAM
Two quad-core 2.1GHz CPU with 32GB DRAM
• Second-generation dedicated central arbiter ASIC
‒ Controls access to fabric bandwidth via dedicated arbitration path to I/O modules
• Interfaces with I/O modules via 1G switched EOBC
N7K-SUP2/N7K-SUP2E
ID LED
Status
LEDs
Presentation_ID
BRKARC-3470
Console Port
Management
Ethernet
USB Ports
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Reset Button
Cisco Public
10
Nexus 7000 I/O Module Families
M Series and F Series
• M Series – L2/L3/L4 with large forwarding tables and rich feature set
N7K-M108X2-12L
N7K-M224XP-23L
N7K-M206FQ-23L
N7K-M148GT-11L
N7K-M148GS-11L
N7K-M132XP-12L
N7K-M202CF-22L
• F Series – High performance, low latency, low power with streamlined
feature set
N7K-F248XP-25E
N7K-F248XP-25
N7K-F132XP-15
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
11
Supported in NX-OS release 6.1(1) and later
24-Port 10G M2 I/O Module
N7K-M224XP-23L
• 24-port 10G with SFP+ transceivers
• 240G full-duplex fabric connectivity
• Two integrated forwarding engines (120Mpps)
‒ Support for “XL” forwarding tables (licensed feature)
• Distributed L3 multicast replication
N7K-M224XP-23L
• 802.1AE LinkSec on all ports
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
12
24-Port 10G M2 I/O Module Architecture
N7K-M224XP-23L
EOBC
To Fabric Modules
To Central Arbiters
Arbitration
Aggregator
…
Fabric 2 ASIC
LC
CPU
VOQs
VOQs
Forwarding
Engine
Forwarding
Engine
Replication
Engine
VOQs
Replication
Engine
Replication
Engine
Replication
Engine
12 X 10G MAC / LinkSec
1
2
3
4
5
VOQs
6
7
8
9
12 X 10G MAC / LinkSec
10 11 12
13 14 15 16 17 18 19 20 21 22 23 24
Front Panel Ports
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
13
Supported in NX-OS release 6.1(1) and later
6-Port 40G M2 I/O Module
N7K-M206FQ-23L
• 6-port 40G with QSFP+ transceivers
‒ Option to breakout to 4X10G interfaces per 40G port*
• 240G full-duplex fabric connectivity
• Two integrated forwarding engines (120Mpps)
• Support for “XL” forwarding tables (licensed feature)
N7K-M206FQ-23L
• Distributed L3 multicast replication
• 802.1AE LinkSec on all ports
* Roadmap feature
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
15
6-Port 40G M2 I/O Module Architecture
N7K-M206FQ-23L
EOBC
To Fabric Modules
To Central Arbiters
Arbitration
Aggregator
…
Fabric 2 ASIC
LC
CPU
VOQs
VOQs
Forwarding
Engine
Forwarding
Engine
Replication
Engine
VOQs
VOQs
Replication
Engine
Replication
Engine
Replication
Engine
3 X 40G MAC / LinkSec
1
2
3 X 40G MAC / LinkSec
3
4
5
6
Front Panel Ports
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
16
40G Transceivers – QSFP+
• 40GBASE-SR4 supported in 6.1(1)
‒ 12-fibre MPO/MTP connector
QSFP-40G-SR4
‒ 100m over OM3 MMF, 150m over OM4 MMF
• Other form-factors TBA
Interior of ribbon fibre cable
MPO Optical Connector
40G MPO interface
Presentation_ID
BRKARC-3470
(one row of 12 fibres)
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
40G 12-strand ribbon fibre
(4 middle fibres unused)
Cisco Public
17
Supported in NX-OS release 6.1(1) and later
2-Port 100G M2 I/O Module
N7K-M202CF-22L
• 2-port 100G with CFP transceivers
‒ Option to breakout to 2X40G or 10X10G interfaces per 100G port*
• 200G full-duplex fabric connectivity
• Two integrated forwarding engines (120Mpps)
• Support for “XL” forwarding tables (licensed feature)
• Distributed L3 multicast replication
N7K-M202CF-22L
• 802.1AE LinkSec on all ports
* Roadmap feature
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
18
2-Port 100G M2 I/O Module Architecture
N7K-M202CF-22L
EOBC
To Fabric Modules
To Central Arbiters
Arbitration
Aggregator
…
Fabric 2 ASIC
LC
CPU
VOQs
VOQs
Forwarding
Engine
Forwarding
Engine
Replication
Engine
VOQs
VOQs
Replication
Engine
Replication
Engine
Replication
Engine
1 X 100G MAC / LinkSec
1 X 100G MAC / LinkSec
1
2
Front Panel Ports
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
19
100G Module Transceivers – 40G and 100G CFP
• 100GBASE-LR4 supported from 6.1(1)
‒ SC connector
‒ 10km over SMF
• Other form-factors on roadmap
CFP-100G-LR4
• 40GBASE-SR4 supported from 6.1(2)
‒ 12-fibre MPO/MTP connector
‒ 100m over MMF
• 40GBASE-LR4 supported from 6.1(2)
CFP-40G-SR4
CFP-40G-LR4
‒ SC connector
‒ 10km over SMF
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
20
40G and 100G Flow Limits – Internal versus “On the Wire”
Internal to Nexus 7000 System
On the Wire (40G)
40G Port
Ingress Modules
1 packet
n
…
4
3
2
1
64 bits
10G 10G
40G
40G
100G
1 VQI 1 VQI
4 VQIs
4 VQIs
10 VQIs
Egress Interfaces
64/66B Encoding
Spines
Spines
Spines
Spines
Fabrics
Destination
VQIs
Tx 1
5
1
Tx 2
6
2
Tx 3
…
3
Tx 4
4
66 bits
• Packets split into 66-bit “code words”
• Each VQI sustains 10-12G traffic flow
• Four code words transmitted in parallel, one on each
physical Tx fibre
• Single-flow limit is ~10G
• No per-flow limit imposed – splitting occurs at physical layer
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
21
Supported in NX-OS release 6.0(1) and later
48-Port 1G/10G F2 I/O Module
N7K-F248XP-25
• 48-port 1G/10G with SFP/SFP+ transceivers
• 480G full-duplex fabric connectivity
• System-on-chip (SoC)* forwarding engine design
‒ 12 independent SoC ASICs
• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS)
• Supports Nexus 2000 (FEX) connections
N7K-F248XP-25
• FabricPath-capable
• FCoE-capable
* sometimes called “switch-on-chip”
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
22
Supported in NX-OS release 6.1(2) and later
48-Port 1G/10G F2E I/O Modules (Fibre and Copper)
N7K-F248XP-25E / N7K-F248XT-25E
N7K-F248XP-25E
• Enhanced version of original F2 I/O module
• Fibre and copper version
• 480G full-duplex fabric connectivity
• Same basic SoC architecture as original F2 with
some additional functionality
N7K-F248XT-25E
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
23
What’s Different in F2E?
• Interoperability with M1/M2, in Layer 2 mode*
‒ Proxy routing for inter-VLAN/L3 traffic
• LinkSec support*
‒ Fibre version: 8 ports
‒ Copper version: 48 ports
• Energy Efficient Ethernet (EEE) capability on F2E copper version
• FabricPath learning enhancements
‒ No learning on broadcast frames
* Roadmap feature
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
24
Energy Efficient Ethernet (IEEE 802.3az)
• IEEE standard for reducing power consumption during idle periods
• Auto-negotiated at Layer 1, like speed and duplex
• Introduces Low Power Idle (LPI) mode for Ethernet ports
‒ Systems on both ends of link save power in LPI mode
‒ Transparent to upper layer protocols
Enter LPI mode
Wake up
Wake up
EEE-capable port
Idles threshold crossed
PHY
EEE Buffer
Enter LPI mode
EEE-capable partner
Auto-negotiate EEE
Normal traffic flow
10G MAC
PHY
10G MAC
Signal LPI mode
Non-idle character
detected
Buffer packets
Presentation_ID
BRKARC-3470
Send wake signal
Count idles
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
25
48-Port 1G/10G F2 / F2E I/O Module Architecture
N7K-F248XP-25 / N7K-F248XP-25E / N7K-F248XT-25
To Central Arbiters
To Fabric Modules
EOBC
Arbitration
Aggregator
LC
CPU
…
Fabric 2
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
4 X 10G
SoC
1
5
9
13
17
21
25
29
33
37
41
45
3
2
4
7
6
8
11
10
12
14
15
16
18
19
20
22
23
24
26
27
28
30
31
32
34
35
36
Front Panel Ports
39
38
40
42
43
44
46
47
48
LinkSec-capable (F2E fibre only)
LinkSec-capable (F2E copper only)
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
26
F2-Only VDC
Communication between F2-only
VDC and M1/M2/F1 VDC must be
through external connection
F2 module
• F2/F2E modules do not interoperate with
other Nexus 7000 modules*
F2E module
F2-only
VDC
F2E module
• Must deploy in an “F2 only” VDC
• Can be default VDC, or any other VDC
‒ Use the limit-resource module-type f2 VDC
configuration command
M2 module
• System with only F2 modules and empty
configuration boots with F2-only default VDC
automatically
M1 module
M2 module
M1/M2/F1
VDC
F1 module
M1/M2/F1 modules can exist in same
chassis as F2/F2E modules, but not
in the same VDC
* F2E will interoperate in Layer 2 mode with M1/M2 in a future software release
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
28
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
29
M-Series Forwarding Engine Hardware
• Hardware forwarding engine(s) integrated on
every I/O module
• OTV
• IGMP snooping
• 60Mpps per forwarding engine Layer 2
bridging with hardware MAC learning
• RACL/VACL/PACL
• 60Mpps per forwarding engine Layer 3 IPv4
and 30Mpps Layer 3 IPv6 unicast
• Policy-based routing (PBR)
• Layer 3 IPv4 and IPv6 multicast support (SM,
SSM, bidir)
• QoS remarking and policing policies
• Unicast RPF check and IP source guard
• Ingress and egress NetFlow (full and sampled)
• MPLS
Presentation_ID
BRKARC-3470
Hardware Table
M-Series Modules
without Scale License
M-Series Modules
with Scale License
FIB TCAM
128K
900K
Classification TCAM (ACL/QoS)
64K
128K
MAC Address Table
128K
128K
NetFlow Table
512K
512K
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
30
M-Series Forwarding Engine Architecture
FE Daughter
Card
 Ingress NetFlow
collection
Ingress Pipeline
 Ingress policing
 FIB TCAM and
adjacency table
lookups for
Layer 3
forwarding
 ECMP hashing
 Multicast RPF
check
Egress Pipeline
 Ingress ACL
and QoS
classification
Layer 3
Engine
 Unicast RPF
check
 Egress NetFlow
collection
 Egress policing
 Egress ACL and QoS classification
 Ingress MAC
table lookups
 IGMP snooping
lookups
 IGMP snooping
redirection
Layer 2
Engine
Packet Headers from
I/O Module Replication Engine
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
 Egress MAC
lookups
 IGMP snooping
lookups
Final lookup result to
I/O Module Replication Engine
Cisco Public
31
F2/F2E Forwarding Engine Hardware
• Each SoC forwarding engine services 4 frontpanel 10G ports (12 SoCs per module)
• RACL/VACL/PACL
• 60Mpps per SoC Layer 2 bridging with
hardware MAC learning
• Policy-based routing (PBR)
• 60Mpps per forwarding engine Layer 3 IPv4/
IPv6 unicast
• FabricPath forwarding
• QoS remarking and policing policies
• Unicast RPF check and IP source guard
• Ingress sampled NetFlow
• Layer 3 IPv4 and IPv6 multicast support (SM,
SSM)
• FCoE
• IGMP snooping
Hardware Table
Per F2 SoC
Per F2 Module
MAC Address Table
16K
256K*
FIB TCAM
32K IPv4/16K IPv6 32K IPv4/16K IPv6
Classification TCAM (ACL/QoS)
16K
192K*
* Assumes specific configuration to scale SoC resources
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
32
F2/F2E Forwarding Engine
To/From Central
Arbiter
To Fabric
Ingress and egress
forwarding decisions
(L2/L3 lookups,
ACL/QoS, etc.)
Forwarding
tables
From Fabric
4 X 10G
SoC
Decision Engine
L2 Lookup (post-L3)
Virtual output
queues
Ingress
Buffer (VOQ)
MAC
Table
FIB/ADJ
Egress
Buffer
Layer 3 Lookup
QoS / ACL
CL
Egress fabric
receive buffer
L2 Lookup (pre-L3)
Ingress Parser
“Skid buffer” –
Accommodates pause
reaction time
Egress Parser
Pause Latency Buffer
1G and 10G capable
interface MAC
1G/10G MAC
Four front-panel
interfaces per
ASIC
Presentation_ID
BRKARC-3470
Port A
1G/10G
1G/10G MAC
Port B
1G/10G
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Port C
1G/10G
Port D
1G/10G
Cisco Public
33
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
34
Crossbar Switch Fabric Modules
N7K-C7018-FAB-1/FAB-2
• Provide interconnection of I/O modules in Nexus 7009 / 7010 / 7018 chassis
• Each installed fabric increases available per-payload slot bandwidth
• Two fabric generations available – Fabric 1 and Fabric 2
Fabric Module
Supported Chassis
Supported
I/O Modules
Per-fabric module
bandwidth
Total bandwidth with
5 fabric modules
Fabric 1
7010 / 7018
All
46Gbps per slot
230Gbps per slot
Fabric 2
7009 / 7010 / 7018
All
110Gbps per slot
550Gbps per slot
• Different I/O modules leverage different amount of fabric bandwidth
• Access to fabric bandwidth controlled using QoS-aware central arbitration with VOQ
N7K-C7009-FAB-2
Presentation_ID
BRKARC-3470
N7K-C7010-FAB-1/FAB-2
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
35
Multistage Crossbar
Nexus 7000 implements 3-stage crossbar switch fabric
• Stages 1 and 3 on I/O modules
• Stage 2 on fabric modules
Fabric Modules
2nd stage
1
Fabric ASIC
2
Fabric ASIC
3
Fabric ASIC
4
Fabric ASIC
5
Fabric ASIC
2 x 23Gbps (Fab1) –or–
2 x 55Gbps (Fab2)
per slot, per fabric module
Up to 230Gbps (Fab1) –or–
Up to 550Gbps (Fab2)
per I/O module with 5 fabric modules
1st stage
Presentation_ID
BRKARC-3470
Fabric ASIC
Fabric ASIC
Ingress
Module
Egress
Module
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
3rd stage
Cisco Public
36
I/O Module Capacity – Fabric 1
Fabric 1 Modules
230Gbps
46Gbps
184Gbps
138Gbps
92Gbps
per slot bandwidth
Local Fabric 2
(480G)
46Gbps/slot
Fabric 1
ASICs
46Gbps/slot
Fabric 1
ASICs
46Gbps/slot
Fabric 1
ASICs
46Gbps/slot
Fabric 1
ASICs
46Gbps/slot
Fabric 1
ASICs
One fabric
• Any port can pass traffic to any
other port in system
Two fabrics
• 80G M1 module has full bandwidth
Local Fabric 2
(240G)
1
2
3
Five fabrics
• 240G M2 module limited to 230G
per slot
• 480G F2/F2E module limited to
230G per slot
Presentation_ID
BRKARC-3470
Local Fabric 1
(80G)
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
4
5
37
I/O Module Capacity – Fabric 2
Fab2 does NOT make Fab1-based
modules faster!!
550Gbps
110Gbps
440Gbps
220Gbps
330Gbps
Fabric channels run at
lowest common speed
Fabric 2 Modules
110Gbps/slot
Fabric 2
ASICs
Local Fabric 2
(480G)
per slot bandwidth
Fabric 2
ASICs
One fabric
• Any port can pass traffic to any
other port in system
Two fabrics
Local Fabric 2
(240G)
Fabric 2
ASICs
1
2
3
• 80G M1 module has full
bandwidth
Three fabrics
• 240G M2 module has maximum
bandwidth
Fabric 2
ASICs
Local Fabric 1
(80G)
Five fabrics
Fabric 2
ASICs
• 480G F2 module has maximum
bandwidth
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
4
5
38
What About Nexus 7004?
• Nexus 7004 has no fabric modules
• I/O modules have local fabric with 10 available fabric channels
‒ I/O modules connect “back-to-back” via 8 fabric channels
‒ Two fabric channels “borrowed” to connect supervisor engines
• Available inter-module bandwidth dependent on installed module types
M1 Modules in Nexus 7004
Sup Module 1 Crossbar
ASIC
F2/F2E/M2 Modules in Nexus 7004
Crossbar Sup Module 2
ASIC
Sup Module 1 Crossbar
ASIC
Crossbar Sup Module 2
ASIC
2 * 23G
fabric channels
M1 Module 3
Fabric 1
Crossbar
ASIC
Fabric 1
Crossbar
ASIC
M1 Module 4
8 * 23G local fabric channels
interconnect I/O modules (184G)
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
2 * 55G
fabric channels
F2/F2E/M2
Module 3
Fabric 2
Crossbar
ASIC
Fabric 2
Crossbar
ASIC
F2/F2E/M2
Module 4
8 * 55G local fabric channels
interconnect I/O modules (440G)
Cisco Public
39
Arbitration, VOQ and Crossbar Fabric
• Arbitration, VOQ, and fabric combine to provide all necessary
infrastructure for packet transport inside switch
• Central arbitration – Controls scheduling of traffic into fabric based on
fairness, priority, and bandwidth availability at egress ports
• Virtual Output Queues (VOQs) – Provide buffering and queuing for
ingress-buffered switch architecture
• Crossbar fabric – Provides dedicated, high-bandwidth interconnects
between ingress and egress I/O modules
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
40
Central Arbitration
• Access to fabric for unicast traffic controlled using central arbitration
‒ Ensures fair access to available bandwidth on each egress port
‒ Can provide no-drop service for some traffic classes
• Arbiter ASIC on Supervisor Engine provides central arbitration via
dedicated arbitration path to every module
• Arbitration performed on per-destination, per-priority basis
‒ Ensures high priority traffic takes precedence over low priority traffic
• For multidestination traffic, no central arbitration
‒ Ingress broadcast, multicast, unknown unicast frames sent unarbitrated
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
41
Virtual Output Queues (VOQs)
• VOQs at ingress to fabric provide buffering and queuing for egress
destinations reached through the fabric
• Queuing of traffic entering fabric based on destination port (VQI) and
packet priority
‒ Four levels of priority per destination
• VOQs prevent congested egress ports from blocking ingress traffic
destined to other ports
‒ Provide independent scheduling for individual egress destinations
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
42
48-port 1G/10G F2/F2E I/O Module
VOQ Destinations (VQIs)
• Each egress interface has one or more
associated “Virtual Queuing Indexes” (VQIs) or
“VOQ Destinations”
One VQI
24-port 10G M2 I/O Module
• Each VQI has four priority levels / classes
• For 1G / 10G interfaces:
One VQI
‒ One VQI for each 1G or 10G port
• For 40G interfaces:
6-port 40G M2 I/O Module
‒ Four VQIs for each 40G port –or–
‒ One VQI for each 10G breakout port
• For 100G interfaces:
Four VQIs
‒ Ten VQIs for each 100G port –or–
‒ Four VQIs for each 40G breakout port –or–
2-port 100G M2 I/O Module
‒ One VQI for each 10G breakout port
Ten VQIs
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
43
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
44
Buffering, Queuing, and Scheduling
• Buffering – storing packets in memory
‒ Needed to absorb bursts, manage congestion
• Queuing – buffering packets according to traffic class
‒ Provides dedicated buffer for packets of different priority
• Scheduling – controlling the order of transmission of buffered packets
‒ Ensures preferential treatment for packets of higher priority and fair treatment for packets of
equal priority
• Nexus 7000 uses queuing policies and network-QoS policies to define buffering,
queuing, and scheduling behaviour
• Default queuing and network-QoS policies always in effect in absence of any user
configuration
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
45
I/O Module Buffering Models
• Buffering model varies by I/O module family
‒ M-series modules: hybrid model combining ingress VOQ-buffered architecture
with egress port-buffered architecture
‒ F-series modules: pure ingress VOQ-buffered architecture
• All configuration through Modular QoS CLI (MQC)
‒ Queuing parameters applied using class-maps/policy-maps/service-policies
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
46
Hybrid Ingress/Egress Buffered Model
M-Series I/O Modules
Ingress port buffer – Manages congestion in ingress forwarding/replication engines only
Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) over fabric
Egress VOQ buffer – Receives frames from fabric; also buffers multidestination frames
Egress port buffer – Manages congestion at egress interface
Ingress
port buffer
Ingress
VOQ buffer
Egress
VOQ buffer
Ingress Module
Ingress Module
Crossbar
Fabric
Egress
port buffer
Egress Module
Ingress Module
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
47
Ingress Buffered Model
F-Series I/O Modules
Ingress “skid” buffer – Absorbs packets in flight after external flow control asserted
Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) over fabric
Egress VOQ buffer – Receives frames from fabric; also buffers multidestination frames
Ingress
skid buffer
Ingress
VOQ buffer
Egress
VOQ buffer
Ingress Module
Ingress Module
Crossbar
Fabric
Egress Module
Ingress Module
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
49
Distributed Buffer Pool
• Ingress-buffered architecture implements large, distributed buffer pool to absorb congestion
• Absorbs congestion at all ingress ports contributing to congestion, leveraging all per-port ingress
buffer
• Excess traffic does not consume fabric bandwidth, only to be dropped at egress port
8:1 Ingress:Egress
2:1 Ingress:Egress
Ingress
VOQ buffer
Available buffer
for congestion
management:
1
Available buffer
for congestion
management:
2
3
1
4
Fabric
Fabric
2
5
Egress
6
Egress
7
Ingress
8
Ingress
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
51
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
52
Layer 2 Forwarding
• Layer 2 forwarding – traffic steering based on destination MAC address
• Hardware MAC learning
‒ CPU not directly involved in learning
• Forwarding engine(s) on each module have copy of MAC table
‒ New learns communicated to other forwarding engines via hardware “flood to
fabric” mechanism
‒ Software process ensures continuous MAC table sync
• Spanning tree (PVRST or MST), Virtual Port Channel (VPC), or
FabricPath ensures loop-free Layer 2 topology
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
53
Hardware Layer 2 Forwarding Process
MAC table lookup drives Layer 2 forwarding
• Source MAC and destination MAC lookups performed for each frame,
based on {VLAN,MAC} pairs
• Source MAC lookup drives new learns and refreshes aging timers
• Destination MAC lookup dictates outgoing switchport
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
54
HDR
= Packet Headers
M2 L2 Packet Flow – 10G
= Packet Data
DATA

Credit grant
for fabric
access
CTRL
= Internal Signalling
Supervisor Engine

Return
credit
to pool
Central Arbiter

Fabric Module 1
Fabric Module 2
Fabric Module 3
Fabric ASIC
Fabric ASIC
Fabric ASIC

Receive from
fabric
 Return buffer
credit
VOQ arbitration
and queuing
Module 1

Module 2
ACL/QoS/
NetFlow
lookups

Fabric 2 ASIC

Forwarding
Engine


L2 SMAC/ DMAC
lookups and
hash result
VOQs
Layer 3
Engine


Replication
Engine
Return result
12 X 10G MAC / LinkSec

Hash-based
uplink selection
Submit packet
headers for
lookup
LinkSec decryption
VOQs
Replication
Engine
Layer 2
Engine


VOQs
Layer 3
Engine
Replication
Engine
Static RE uplink
selection
Fabric 2 ASIC
Forwarding
Engine
VOQs
Layer 2
Engine

Round-robin
transmit to fabric
Round-robin
transmit to VOQ
Egress
port QoS
Replication
Engine
12 X 10G MAC / LinkSec
Ingress port QoS

Presentation_ID
BRKARC-3470
Receive
packet
from wire
e1/1
© 2012
2013 Cisco and/or its affiliates. All rights reserved.


Transmit
packet on
Cisco wire
Public

Static
downlink
selection
LinkSec
encryption
e2/2
55
10G M2 Module Ingress Path
To Fabric Modules
Fabric 0
VOQ→Fabric
Round Robin
VOQ 0
VOQ 1
VOQ 2
VOQ 3
Forwarding Engine
Hash Result
RE→VOQ
Replication
Engine 0
Replication
Engine 1
Replication
Engine 2
Replication
Engine 3
Static Mapping
Port ASIC→RE
Presentation_ID
BRKARC-3470
10G Port ASIC 0
10G Port ASIC 1
Ports 1-12
Ports 13-24
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
56
Replication Engine Selection on Ingress
– 10G M2 Module
• Front-panel ports statically mapped to replication engine uplinks
Replication
Engine 0
Port 0
Replication
Engine 1
Port 1
Port 0
Replication
Engine 2
Port 1
Port 0
Port 1
Port ASIC 0
1
Presentation_ID
BRKARC-3470
2
3
4
5
6
7
Replication
Engine 3
Port 0
Port 1
Port ASIC 1
8
9
10 11
12
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
13 14 15 16 17 18 19 20 21 22 23 24
Cisco Public
57
10G M2 Module Egress Path
From Fabric Modules
Fabric 0
Fabric→VOQ
Static Mapping +
Round Robin
VOQ 0
VOQ 1
VOQ 2
VOQ 3
VOQ→RE
Replication
Engine 0
Static Mapping
Replication
Engine 1
Replication
Engine 2
Replication
Engine 3
RE→Port ASIC
10G Port ASIC 0
Ports 1-12
Presentation_ID
BRKARC-3470
Static Mapping
10G Port ASIC 1
Ports 13-24
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
58
10G M2 Module Egress VQI Mapping
Fabric 0
Odd
Ports
Even
Ports
Odd
Ports
RR
RR
VOQ 0
RR
VOQ 1
1,5,9
3,7,11
2,6,10
Replication
Engine 0
Port 0
4,8,12
13,17,21
Replication
Engine 1
Port 1
Port 0
Presentation_ID
BRKARC-3470
2
3
4
5
6
7
RR
VOQ 2
VOQ 3
15,19,23
14,18,22
Replication
Engine 2
Port 1
Port 0
16,20,24
Replication
Engine 3
Port 1
Port 0
Port 1
Port ASIC 1
Port ASIC 0
1
Even
Ports
8
9
10 11
12
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
13 14 15 16 17 18 19 20 21 22 23 24
Cisco Public
59
HDR
= Packet Headers
M2 L2 Packet Flow – 40G/100G
= Packet Data
DATA

Credit grant
for fabric
access
CTRL
= Internal Signalling
Supervisor Engine

Return
credit
to pool
Central Arbiter

Fabric Module 1
Fabric Module 2
Fabric Module 3
Fabric ASIC
Fabric ASIC
Fabric ASIC

Receive from
fabric
 Return buffer
credit
VOQ arbitration
and queuing
Module 1

Module 2
ACL/QoS/
NetFlow
lookups

Fabric 2 ASIC

Forwarding
Engine


L2 SMAC/
DMAC lookups
and hash result
Layer 3
Engine
Return result
Layer 2
Engine

VOQs


Replication
Engine
Hash-based
uplink selection
Layer 3
Engine
Submit packet
headers for
lookup
Layer 2
Engine
Replication
Engine
3 X 40G or 1 X 100G MAC / LinkSec


LinkSec decryption
Fabric 2 ASIC
Forwarding
Engine
VOQs
Hash-based uplink
selection

Round-robin
transmit to fabric
Round-robin
transmit to VOQ
Egress
port QoS
VOQs
VOQs
Replication
Engine
Replication
Engine
3 X 40G or 1 X 100G MAC / LinkSec
Ingress port QoS

Presentation_ID
BRKARC-3470
Receive
packet
from wire
e1/1
© 2012
2013 Cisco and/or its affiliates. All rights reserved.


Transmit
packet on
Cisco wire
Public

Static RE
downlink
selection
LinkSec
encryption
e2/2
60
40G / 100G M2 Module Ingress Path
To Fabric Modules
Fabric 0
VOQ→Fabric
Round Robin
VOQ 0
VOQ 1
VOQ 2
VOQ 3
Forwarding Engine
Hash Result
RE→VOQ
Replication
Engine 0
Replication
Engine 1
Replication
Engine 2
Replication
Engine 3
Port ASIC
Hash Result
Port ASIC→RE
Presentation_ID
BRKARC-3470
40G / 100G Port ASIC 0
40G / 100G Port ASIC 1
Ports 1-3 / Port 1
Ports 4-6 / Port 2
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
61
Replication Engine Selection on Ingress
– 40G / 100G M2 Module
• Hash Result generated by Port ASIC selects replication engine uplink
• Hash input uses Layer 3 + Layer 4 information
Replication
Engine 0
Port 0
Replication
Engine 1
Port 1
Port 0
Port 1
Replication
Engine 2
Port 0
Replication
Engine 3
Port 1
Port ASIC 0
Port 0
Port ASIC 1
Hash function
1
2
3
Different Flows
Presentation_ID
BRKARC-3470
Port 1
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Hash function
4
5
6
Different Flows
Cisco Public
62
40G / 100G M2 Module Egress Path
From Fabric Modules
Fabric 0
Fabric→VOQ
Static Mapping +
Round Robin
VOQ 0
VOQ 1
VOQ 2
VOQ 3
VOQ→RE
Replication
Engine 0
Static Mapping
Replication
Engine 1
Replication
Engine 2
Replication
Engine 3
RE→Port ASIC
Static Mapping
40G / 100G Port ASIC 0
Ports 1-3 / Port 1
Presentation_ID
BRKARC-3470
40G / 100G Port ASIC 1
Ports 4-6 / Port 2
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
63
40G / 100G M2 Module Egress VQI Mapping
Fabric 0
50% of
VQIs
100G example
50% of
VQIs
50% of
VQIs
RR
RR
VOQ 0
a,b,c
RR
VOQ 1
d,e
f,g,h
Replication
Engine 0
Port 0
I,j
a,b,c
Replication
Engine 1
Port 1
Port 0
VOQ 3
d,e,f
g,h,i
Replication
Engine 2
Port 1
Port 0
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
j,k,l
Replication
Engine 3
Port 1
Port 0
Port 1
Port ASIC 1
4
1
40G example
RR
VOQ 2
Port ASIC 0
Presentation_ID
BRKARC-3470
50% of
VQIs
5
Cisco Public
6
64
HDR
DATA
= Packet Headers
= Packet Data
CTRL
= Internal Signalling
F2 / F2E L2 Packet Flow

Supervisor Engine
Credit grant for
fabric access

Return
credit
to pool
Central Arbiter
Fabric Module 1
Fabric Module 2
Fabric Module 3
Fabric Module 4
Fabric Module 5
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC

Transmit
to fabric



VOQ arbitration

Fabric ASIC
Submit packet headers for lookup

VOQ DE

Receive from fabric
Return buffer credit
Fabric ASIC
Ingress L2 SMAC/ DMAC
lookups, ACL/QoS lookups
VOQ
Return result
SoC

Presentation_ID
BRKARC-3470
Ingress
port QoS
(VOQ)

e1/1
SoC
Module 1
Receive
packet
from wire
© 2012
2013 Cisco and/or its affiliates. All rights reserved.

Transmit
packet
on wire
Module 2

e2/1
Cisco Public
Egress port QoS
(Scheduling)
65
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
66
IP Forwarding
• Nexus 7000 decouples control plane and data plane
• Forwarding tables built on control plane using routing protocols or static
configuration
‒OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing
• Tables downloaded to forwarding engine hardware for data plane
forwarding
‒FIB TCAM contains IP prefixes
‒Adjacency table contains next-hop information
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
67
Hardware IP Forwarding Process
• FIB TCAM lookup based on destination prefix (longest-match)
• FIB “hit” returns adjacency, adjacency contains rewrite information (nexthop)
• Pipelined forwarding engine architecture also performs ACL, QoS, and
NetFlow lookups, affecting final forwarding result
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
68
IPv4 FIB TCAM Lookup (M1/M2)
Generate TCAM lookup key
(destination IP address)
Generate
Lookup Key
10.1.1.10
HIT!
Forwarding Engine
Flow
Data
Index, # next-hops
10.1.1.3
Index, # next-hops
10.1.1.4
Index, # next-hops
10.10.0.10
Index, # next-hops
10.10.0.100
Index, # next-hops
10.10.0.33
Index, # next-hops
10.1.1.xx
10.1.2.xx
Index, # next-hops
10.1.3.xx
Index, # next-hops
10.10.100.xx
Index, # next-hops
10.1.1.xx
Index, # next-hops
10.100.1.xx
Index, # next-hops
10.10.0.xx
Index, # next-hops
FIB TCAM
Presentation_ID
BRKARC-3470
Compare
lookup
key
10.1.1.2
10.100.1.xx
Ingress
unicast IPv4
packet header
Hit in FIB
Index,
returns
result#
in FIB DRAM
next-hops
Next-hop 1 (IF, MAC)
Load-Sharing
Hash
Next-hop 2 (IF, MAC)
Next-hop 3 (IF, MAC)
Offset
mod
Return
lookup
result
Next-hop 4 (IF, MAC)
# nexthops
Next-hop 5 (IF, MAC)
Next-hop 6 (IF, MAC)
Adj Index
Adjacency
index
identifies ADJ
block to use
Next-hop 7 (IF, MAC)
Modulo
function selects
exact next hop
entry to use
FIB DRAM
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Result
Adjacency Table
Cisco Public
69
IPv4 FIB TCAM Lookup (F2 / F2E)
Generate TCAM lookup key
(destination IP address)
Generate
Lookup Key
10.1.1.10
HIT!
Ingress
unicast IPv4
packet header
Compare
lookup
key
Forwarding Block
Flow
Data
10.1.1.2
Index, # next-hops
10.1.1.3
Index, # next-hops
10.1.1.4
Adj Index
Next-hop 1 (IF, MAC)
Adj Index
Next-hop 2 (IF, MAC)
Index, # next-hops
Adj Index
Next-hop 3 (IF, MAC)
10.10.0.10
Index, # next-hops
Adj Index
Next-hop 4 (IF, MAC)
10.10.0.100
Index, # next-hops
Adj Index
Next-hop 5 (IF, MAC)
10.10.0.33
Index, # next-hops
Adj Index
Next-hop 6 (IF, MAC)
10.1.1.xx
10.1.2.xx
Index, # next-hops
Adj Index
Next-hop 7 (IF, MAC)
10.1.3.xx
Index, # next-hops
10.10.100.xx
Index, # next-hops
10.1.1.xx
Index, # next-hops
10.100.1.xx
Index, # next-hops
10.10.0.xx
Index, # next-hops
10.100.1.xx
FIB TCAM
Presentation_ID
BRKARC-3470
Use of Load-Sharing Table decouples
prefix entries and adjacency entries
Load-Sharing
Hash
Offset
mod
Return
lookup
result
Result
Hit in FIB
Index,
returns result
in FIB DRAM
# next-hops
# nexthops
LS Index
Load-sharing
table index
identifies
block to use
FIB DRAM
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Modulo
function
selects
which LS
entry to use
Adj Index
Next-hop 8 (IF, MAC)
Adj Index
Next-hop 9 (IF, MAC) Adjacency
Adj Index
Next-hop 10 (IF,
Adj Index
Next-hop 11 (IF,
Adj Index
Next-hop 12 (IF, MAC)
entry
MAC) contains
next-hop
MAC)information
Adj Index
Adj Index
Load-Sharing Table
Cisco Public
Adjacency Table
70
ECMP Load Sharing
• Up to 16 hardware load-sharing paths per prefix
10.10.0.0/16
• Use maximum-paths command in routing protocols to
control number of load-sharing paths
• Load-sharing is per-IP flow
A
• Configure load-sharing hash options with global
ip load-sharing command:
B
‒ Source and Destination IP addresses
‒ Source and Destination IP addresses plus L4 ports (default)
‒ Destination IP address and L4 port
• Additional randomised number added to hash
prevents polarisation
10.10.0.0/16
via Rtr-A
via Rtr-B
‒ Automatically generated or user configurable value
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
71
HDR
= Packet Headers
M2 L3 Packet Flow
= Packet Data
DATA

Credit grant
for fabric
access
CTRL
= Internal Signalling
Supervisor Engine

Return
credit
to pool
Central Arbiter



Fabric Module 1
Fabric Module 2
Fabric Module 3
Fabric ASIC
Fabric ASIC
Fabric ASIC

Module 1
Module 2
L3 FIB/ADJ lookup
Ingress and egress
ACL/QoS/NetFlow
lookups

Fabric 2 ASIC
L2 ingress and
egress SMAC/
DMAC lookups
Forwarding
Engine
Layer 3
Engine
Return result
Layer 2
Engine

Round-robin
transmit to fabric
VOQs

Round-robin
transmit to VOQ

Replication
Engine
Hash-based
uplink selection
Submit packet
headers for
lookup
LinkSec decryption
VOQs
Replication
Engine
Layer 2
Engine


VOQs
Layer 3
Engine
Replication
Engine
10G / 40G / 100G MAC / LinkSec
Fabric 2 ASIC
Forwarding
Engine
VOQs
Static or Hash-based
uplink selection

Receive from
fabric
 Return buffer
credit
VOQ arbitration
and queuing



Egress
port QoS
Replication
Engine
10G / 40G / 100G MAC / LinkSec
Ingress port QoS

Presentation_ID
BRKARC-3470
Receive
packet
from wire
e1/1
© 2012
2013 Cisco and/or its affiliates. All rights reserved.


Transmit
packet on
Cisco wire
Public

Static RE
downlink
selection
LinkSec
encryption
e2/2
72
HDR
DATA
= Packet Headers
= Packet Data
CTRL
= Internal Signalling
F2 / F2E L3 Packet Flow

Supervisor Engine
Credit grant for
fabric access

Return
credit
to pool
Central Arbiter
Fabric Module 1
Fabric Module 2
Fabric Module 3
Fabric Module 4
Fabric Module 5
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC
Fabric ASIC


VOQ arbitration
Transmit
to fabric

Fabric ASIC
Submit packet headers for lookup
Fabric ASIC



Receive from fabric
Egress port QoS
Return buffer credit

VOQ FE

Return result
SoC

Presentation_ID
BRKARC-3470
Ingress
port QoS
(VOQ)
Module 1

e1/1
L2 ingress and egress SMAC/
DMAC lookups
 L3 FIB/ADJ lookup
 Ingress and egress ACL/QoS
lookups
Receive
packet
from wire
© 2012
2013 Cisco and/or its affiliates. All rights reserved.

Transmit
packet
on wire
VOQ
SoC
Module 2
e2/1
Cisco Public
73
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
74
What is Classification?
• Matching packets
‒ Layer 2, Layer 3, and/or Layer 4 information
• Used to decide whether to apply a particular policy to a packet
‒ Enforce security, QoS, or other policies
• Some examples:
‒ Match TCP/UDP source/destination port numbers to enforce security policy
‒ Match destination IP addresses to apply policy-based routing (PBR)
‒ Match 5-tuple to apply marking policy
‒ Match protocol-type to apply Control Plane Policing (CoPP)
‒ etc.
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
75
Security ACL
CL TCAM Lookup – ACL
Packet header:
SIP: 10.1.1.1
DIP: 10.2.2.2
Protocol: TCP
SPORT: 33992
DPORT: 80
Generate TCAM
lookup key
Generate
Lookup Key
SIP | DIP | Pr | SP | DP
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
Compare lookup
key to CL TCAM
entries
Comparisons
(X = “Mask”)
Forwarding Engine
xxxxxxx| 10.1.2.100
| 10.2.2.2 | |xx
| xxx
xxxxxxx
xx| xxx
| xxx
| xxx
Permit
xxxxxxx | 10.1.68.44 | xx | xxx | xxx
Deny
xxxxxxx | 10.33.2.25 | xx | xxx | xxx
Deny
xxxxxxx
xxxxxxx| |tcp
tcp| xxx
| xxx
22
xxxxxxx | xxxxxxx
| |80
Permit
xxxxxxx | xxxxxxx | tcp | xxx | 23
Deny
xxxxxxx | xxxxxxx | udp | xxx | 514
Deny
HIT! xxxxxxx | xxxxxxx | tcp | xxx | 80
SIP
|
DIP
Results
Permit
xxxxxxx | xxxxxxx | udp | xxx | 161
Permit
Result
Result affects
final packet
handling
| Pr | SP | DP
CL TCAM
Presentation_ID
BRKARC-3470
ip access-list example
permit ip any host 10.1.2.100
deny ip any host 10.1.68.44
deny ip any host 10.33.2.25
permit tcp any any eq 22
deny tcp any any eq 23
deny udp any any eq 514
permit tcp any any eq 80
permit udp any any eq 161
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Hit in CL TCAM
returns result in
CL SRAM
CL SRAM
Cisco Public
Return
lookup
result
76
CL TCAM Lookup – QoS
Packet header:
SIP: 10.1.1.1
DIP: 10.2.2.2
Protocol: TCP
SPORT: 33992
DPORT: 80
QoS Classification ACLs
ip access-list police
permit ip any 10.3.3.0/24
permit ip any 10.4.12.0/24
ip access-list remark-dscp-32
permit udp 10.1.1.0/24 any
ip access-list remark-dscp-40
permit tcp 10.1.1.0/24 any
ip access-list remark-prec-3
permit tcp any 10.5.5.0/24 eq 23
Generate TCAM
lookup key
Generate
Lookup Key
SIP | DIP | Pr | SP | DP
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
Forwarding Engine
Compare
lookup key
Comparisons
(X = “Mask”)
xxxxxxx | 10.2.2.xx
10.3.3.xx | xx | xxx | xxx
Policer ID 1
xxxxxxx | 10.4.12.xx | xx | xxx | xxx
Policer ID 1
10.1.1.xx
xxx |xxx
xxx
10.1.1.xx || xxxxxxx
xxxxxxx || udp
tcp || xxx|
Remark DSCP 32
HIT! 10.1.1.xx | xxxxxxx | tcp | xxx | xxx
Remark DSCP 40
xxxxxxx | 10.5.5.xx| tcp | xxx | 23
Remark IP Prec 3
SIP
|
DIP
| Pr | SP | DP
CL TCAM
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Hit in CL TCAM
returns result in
CL SRAM
CL SRAM
Cisco Public
Results
Result
Result affects
final packet
handling
Return
lookup
result
77
Atomic Policy Programming
• Avoids packet loss during policy updates
• Enabled by default
• Atomic programming process:
‒ Program new policy in free/available CL TCAM entries
‒ Enable new policy by swapping the ACL label on interface
‒ Free CL TCAM resources used by previous policy
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
78
Atomic Policy Programming Cont.
• To support atomic programming, software reserves 50% of available TCAM
• If insufficient resources available, system returns an error and no modifications
made in hardware
‒ Failed to complete Verification: Tcam will be over used, please turn
off atomic update
• Disable with no platform access-list update atomic
‒ Disabling may be necessary for very large ACL configurations
‒ Atomic programming attempted but not mandatory
• User can disable atomic programming and perform update non-atomically
(assuming ACL fits in CL TCAM)
‒ “Default” ACL result (deny by default) returned for duration of reprogramming
‒ Use [no] hardware access-list update default-result permit to control default result
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
79
Classification Configuration Sessions
Two ways to configure ACL/QoS policies:
• Normal configuration mode (config terminal)
‒ Configuration applied immediately line by line
‒ Recommended only for small ACL/QoS configurations, or non-data-plane ACL configuration
• Session config mode (config session)
‒ Configuration only applied after commit command issued
‒ Recommended for large ACL/QoS configurations
• Config session mode also provides verify facility to “dry-run” the configuration
against available system resources
‒ No change to existing hardware configuration after verification (regardless of verification result)
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
80
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
81
NetFlow on Nexus 7000
• NetFlow collects flow data for packets traversing the switch
• Each module maintains independent NetFlow table
Presentation_ID
BRKARC-3470
M1 / M2
F2 / F2E
Per-interface NetFlow
Yes
Yes
NetFlow direction
Ingress/Egress
Ingress only
Full NetFlow
Yes
No
Sampled NetFlow
Yes
Yes
Bridged NetFlow
Yes
Yes
Hardware Cache
Yes
No
Software Cache
No
Yes
Hardware Cache Size
512K entries per
forwarding engine
N/A
NDE (v5/v9)
Yes
Yes
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
82
Full vs. Sampled NetFlow
• NetFlow collects full or sampled flow data
• Full NetFlow: Accounts for every packet of every flow on interface
‒ Available on M-Series modules only
‒ Flow data collection up to capacity of hardware NetFlow table
• Sampled NetFlow: Accounts for M in N packets on interface
‒ Available on both M-Series (ingress/egress) and F2/F2E (ingress only)
‒ M-Series: Flow data collection up to capacity of hardware NetFlow table
‒ F2/F2E: Flow data collection for up to ~1000pps per module
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
83
Sampled NetFlow Details
• Random packet-based sampling
• M:N sampling: Out of N consecutive packets, select M consecutive
packets and account only for those flows
• On M-Series, sampled packets create hardware NetFlow table entry
• On F2/F2E, sampled packets sent to LC CPU via module inband
‒ Rate limited to ~1000pps per module
• Software multiplies configured sampler rate by 100 on F2/F2E modules
‒ Example: when using 1 out-of 100 sampler on F2/F2E interface, sampled rate
becomes 1:10000
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
84
NetFlow on M1/M2 Modules
To NetFlow Collector
Generate NetFlow v5
or v9 export packets
M2 Module
Fabric
ASIC
via Supervisor
Inband
Supervisor
Engine
LC
NetFlow
CPU
Table
Aged Flows
Forwarding
Engine
Hardware
Flow Creation
VOQs
M2 Module
Main
CPU
via mgmt0
Switched
EOBC
Mgmt
Enet
LC
NetFlow
CPU
Table
Aged Flows
Forwarding
Engine
M1 Module
LC
CPU
To NetFlow Collector
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Hardware
Flow Creation
NetFlow
Table
Aged Flows
Forwarding
Engine
Cisco Public
Hardware
Flow Creation
85
Sampled NetFlow on F2/F2E Modules
To NetFlow Collector
Age flows and
generate NetFlow v5
or v9 export packets
Fabric
ASIC
NetFlow
Cache
LC
CPU
Supervisor
Engine
via Module
Inband
Sampled
Packets
F2E Module
LC
CPU
Switched
EOBC
via Module
Inband
Sampled
Packets
SoC
Decision
Engine
Data Flow
F2E Module
DRAM
NetFlow
Cache
Mgmt
Enet
Aged
Flows
LC
CPU
To NetFlow Collector
Presentation_ID
BRKARC-3470
Data Flow
NetFlow
Cache
Aged
Flows
via mgmt0
SoC
Decision
Engine
DRAM
VOQs
Main
CPU
F2 Module
DRAM
Aged
Flows
via Supervisor
Inband
Populate cache based
on received samples
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
via Module
Inband
Sampled
Packets
SoC
Decision
Engine
Cisco Public
Data Flow
86
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• IP Forwarding
• Classification
• NetFlow
• Conclusion
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
87
Nexus 7000 Architecture Summary
Multiple chassis
designs with
density and airflow
options
Control plane
protocols, system and
network management
Supervisor Engines
I/O Modules
Chassis
Fabrics
Variety of front-panel interface and
transceiver types with hardware-based
forwarding and services, including
unicast/multicast, bridging/routing, ACL/QoS
classification, and NetFlow statistics
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
High-bandwidth fabric to
interconnect I/O modules and
provide investment protection
Cisco Public
88
Conclusion
• You should now have a thorough understanding of
the Nexus 7000 switching architecture, I/O module
design, packet flows, and key forwarding engine
functions…
• Any questions?
Presentation_ID
BRKARC-3470
© 2012
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
89
89
Q&A
Complete Your Online Session
Evaluation
Give us your feedback and receive
a Cisco Live 2013 Polo Shirt!
Complete your Overall Event Survey and 5
Session Evaluations.
 Directly from your mobile device on the
Cisco Live Mobile App
 By visiting the Cisco Live Mobile Site
www.ciscoliveaustralia.com/mobile
 Visit any Cisco Live Internet Station located
throughout the venue
Polo Shirts can be collected in the World of
Solutions on Friday 8 March 12:00pm-2:00pm
BRKARC-3470
Don’t forget to activate your
Cisco Live 365 account for
access to all session material,
communities, and on-demand and live activities throughout
the year. Log into your Cisco Live portal and click the
"Enter Cisco Live 365" button.
www.ciscoliveaustralia.com/portal/login.ww
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
92
BRKARC-3470
© 2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising