TF-Storage – FCoE: An Overview

TF-Storage – FCoE: An Overview
Andy Vallely
andyv@cisco.com
June 9th 2011
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
1
LAN
SAN A
SAN B
LAN
16 Servers
8 * 1GbE 4 * FC
1 Server
8 * 1GbE 2 * FC
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
2
LAN
© 2010 Cisco and/or its affiliates. All rights reserved.
SAN A
SAN B
LAN
SAN A
SAN B
Cisco Confidential
3
Mid 1980’s
Mid 1990’s
Early 2000’s
Late 2000’s
10Mb
100Mb
1Gb
10Gb
UTP Cat 3
UTP Cat 5
Connector
(Media)
SFP+ CU*
copper
X2 CX4
copper
SFP+ USR
MMF, ultra short reach
SFP+ SR
MMF,short reach
RJ45 10GBASE-T
copper
© 2010 Cisco and/or its affiliates. All rights reserved.
* Terminated cable
UTP Cat 5
MMF, SMF
UTP Cat6a
MMF, SMF
TwinAx, CX4
Distance
Power
(each side)
Twinax
<10m
~ 0.1W
~ 0.1µs
SFF 8431**
Twinax
15m
4W
~ 0.1µs
IEEE 802.3ak
MM OM2
MM OM3
10m
100m
1W
~0
MM OM2
MM OM3
82m
300m
1W
~0
IEEE 802.3ae
Cat6
Cat6a/7
Cat6a/7
55m
100m
30m
~ 6W***
~ 6W***
~ 4W***
2.5µs
2.5µs
1.5µs
IEEE 802.3an
Cable
** Draft 3.0, not final
Transceiver
Latency (link)
Standard
none
Cisco Confidential
*** As of 2008; expected to decrease over time
4
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
5
•  iSCSI
No guarantee of No-Packet-Drop
No native FC Support
No native FC tools or services
Heave TCP/IP process – requires TOE
•  Infiniband
Skills gap
Cost for new network
10Gb les efficient than 10GbE due to encoding
•  Ethernet
Pervasive Ethernet
Requires additional development (Loss-Less Ethernet)
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
6
•  From a Fibre Channel standpoint it s
FC connectivity over a new type of cable called… an Ethernet cloud
•  From an Ethernet standpoints it s
Yet another ULP (Upper Layer Protocol) to be transported, but… a challenging one!
•  And technically…
FCoE is an extension of Fibre Channel
onto a Lossless Ethernet fabric
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
7
•  10Gbps Ethernet
•  Lossless Ethernet
Matches the lossless behavior guaranteed in FC by B2B credits
•  Ethernet jumbo frames
Max FC frame payload = 2112 bytes
Normal ethernet frame, ethertype = FCoE
FCS
EOF
FC Payload
CRC
FC
Header
FCoE
Header
Ethernet
Header
Same as a physical FC frame
Control information: version, ordered sets (SOF, EOF)
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
8
Ethernet
Header
FCoE
Header
12 bytes (MAC addresses) +
4 bytes (802.1Q tag)
16 bytes
FC
Header
FC Payload
24 bytes
Tota
l: 2
Up to 2112 bytes
180
byte
s
4 bytes
1 byte (EOF) + 3 bytes (padding)
CRC
EOF
FCS
© 2010 Cisco and/or its affiliates. All rights reserved.
4 bytes
Cisco Confidential
9
Destination MAC Address
Source MAC Address
IEEE 802.1Q Tag
ET = FCoE
Ver
Reserved
Reserved
Reserved
Reserved
SOF
Encapsulated FC Frame
(Including FC-CRC)
EOF
Reserved
FCS
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
10
FCoE is Fibre Channel at the host and switch level
Easy to
Understand
Same
Operational Model
Completely based
on the FC model
Same host-to-switch and switch-to-switch
behavior of FC
Same Techniques of
Traffic Management
E.g., in order delivery or
FSPF load balancing
Same Management
and Security Models
WWNs, FC-IDs, hard/soft
zoning, DNS, RSCN
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
11
•  ENode (FCoE Node): a Fibre Channel HBA implemented within an Ethernet NIC
aka CNA (Converged Network Adapter)
FC Node
© 2010 Cisco and/or its affiliates. All rights reserved.
FCoE
FCoE
Enet
port
Enet
port
Cisco Confidential
12
•  FCF (Fibre Channel Forwarder), the forwarding entity inside an FCoE switch
FC
port
FCoE Switch
FC
port
FCF
FCoE
FC
port
FC
port
Ethernet Bridge
Eth
port
© 2010 Cisco and/or its affiliates. All rights reserved.
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Cisco Confidential
13
LAN
CNA
Link
© 2010 Cisco and/or its affiliates. All rights reserved.
PCIe
Fibre Channel
Drivers
Ethernet
Operating System
Fibre Channel
Ethernet Drivers
Link
Fibre Channel
Ethernet
PCIe
10GbEE
10GbEE
HBA
HBA
10GbE
10GbE
Link
HBA
PCIe
Ethernet Drivers
Fibre Channel
Drivers
Operating System
Cisco Confidential
14
•  Standard drivers
•  Same management
•  Operating System sees:
Dual port 10 Gigabit Ethernet adapter
Dual Port 4 Gbps Fibre Channel HBAs
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
15
SAN A
SAN B
10GE
Backbone
VF_Ports
VN_Ports
10GE
4/8 Gbps FC
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
16
•  Direct attached
FCoE storage
•  Unified FCoE network environment
Enhanced
Ethernet Fabric
FCoE
Storage
•  Preserve SAN architecture for high
availability
SAN A and SAN B
FCoE
Enhanced Ethernet and FCoE
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
17
•  VE_Ports and VF_Ports always use MAC addresses derived from the switch pool
•  VN_Ports may use two types of MAC addresses:
SPMA (Server Provided MAC Addresses)
FPMA (Fabric Provided MAC Addresses)
•  MAC Addresses are negotiated in FIP
•  Initial deployment will use FPMA only
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
18
•  A dedicated MAC address for each FC-ID
Assigned by the FC fabrics
Consistent with the FC model
OUIs with U/L = 1 (Local addressing), called FC-MAPs
Multiple FC-MAPs may be supported (one per FC fabric)
24 bits
FC-MAP
(ex 02-12-34)
MAC
address
© 2010 Cisco and/or its affiliates. All rights reserved.
FC-MAP
(ex 02-12-34)
FC-ID
7.8.9
24 bits
FC-ID
7.8.9
48 bits
Cisco Confidential
19
ENode
FCoE Switch
VLAN
Discovery
FCF
Discovery
VLAN
Discovery
Solicita
tio
Advertisement
n
FCF
Discovery
FLOGI/FDISC
FLOGI/FDISC Accept
FC Command
FC Command
responses
© 2010 Cisco and/or its affiliates. All rights reserved.
FIP:
FCoE
Initialization
Protocol
FCOE
Protocol
Cisco Confidential
20
•  FCoE frames have:
MAC addresses (hop-by-hop)
FC addresses (end-to-end)
FC Storage
FCID 7.1.1
FC Domain 3
MAC A
FC Domain 7
FC Fabric
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
© 2010 Cisco and/or its affiliates. All rights reserved.
FC Fabric
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
Ethernet
Fabric
Dest. = MAC B
Srce. = MAC A
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FCID 1.1.1
MAC C
FC Domain 1
MAC B
Ethernet
Fabric
Dest. = MAC C
Srce. = MAC B
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FCoE
Frame
Cisco Confidential
21
Feature/
Standard
Benefit
Lossless Ethernet
IEEE 802.1Qbb (PFC)
Enable multiple traffic types to share a common Ethernet link
without interfering with each other
Class of Service Based
Bandwidth Management
IEEE 802.1Qaz
Enable consistent management of Quality of Service at the
network level by providing consistent scheduling
Congestion Management IEEE
802.1Qau
End-to-end congestion management for
L2 network
Data Center Bridging Exchange,
DCBX
Management protocol for Enhanced Ethernet capabilities
L2 Multipath for
Unicast and Multicast
Increase bandwidth, multiple active paths.
No spanning tree
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
22
•  PAUSE functionality per Ethernet priority
IEEE 802.1Q defines 8 priorities
Traffic classes are mapped to different priorities:
no traffic interference
Storage traffic may be paused while IP traffic is being forwarded
Or, vice versa
Requires independent resources per priority (buffers)
•  High level of industry support
Cisco distributed proposal
Standard Track in IEEE 802.1Qbb
16
Ethertype = IEEE 802.1Q
3
Priority
1
CFI
12 bits
VLAN ID
IEEE 802.1Q tag
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
23
Priority based
Bandwidth Management
Priority based Flow Control
Transmit Queues
Ethernet Link
Receive Buffers
Zero
Zero
One
One
Two
Two
Three
STOP
PAUSE
Three
Four
Four
Five
Five
Six
Six
Seven
Seven
•  Enables lossless behavior
for each class of service
•  PAUSE sent per priority
when buffers limit exceeded
© 2010 Cisco and/or its affiliates. All rights reserved.
Offered Load
Realized Load
3G/s
3G/s
2G/s
3G/s
3G/s
3G/s
20%
30%
3G/s
HPC Traffic
3G/s
2G/s
3G/s
Storage Traffic
3G/s
3G/s
3G/s
LAN Traffic
4G/s
5G/s
t1
t2
50%
3G/s
t1
4G/s
t2
6G/s
t3
t3
•  Enables intelligent sharing of
bandwidth between traffic classes
control of bandwidth
•  802.1Qaz Enhanced Transmission
Cisco Confidential
24
•  IEEE 802.1Qau provides end-to-end congestion management for L2 Ethernet networks
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
25
•  Port types
VF_port / VN_port
VE_port
F_port trunking / channeling
•  Device types
Fibre Channel Forwarder (FCF)
FIP snooping bridge
•  Switching modes
FC switching
NPV (NPIV gateway)
FIP snooping
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
26
•  FCoE frames have:
MAC addresses (hop-by-hop)
FC addresses (end-to-end)
FC Storage
VE_port
FC link
FCID 7.1.1
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
© 2010 Cisco and/or its affiliates. All rights reserved.
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
VF_port
Ethernet link
FC Fabric
FC Domain 7
VE_port
FC Domain 3
MAC A
Dest. = MAC B
Srce. = MAC A
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
VN_port
Ethernet link
FC Domain 1
MAC B
FCID 1.1.1
MAC C
Dest. = MAC C
Srce. = MAC B
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FCoE
Frame
Cisco Confidential
27
FC Storage
VF_port
FC link
FCID 7.1.1
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
© 2010 Cisco and/or its affiliates. All rights reserved.
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
VF_port
Ethernet link
FC Fabric
FC Domain 7
VNP_port
FC Domain 1
MAC A
Dest. = MAC B
Srce. = MAC A
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
VN_port
Ethernet link
No domain
MAC B
FCID 1.1.1
MAC C
Dest. = MAC C
Srce. = MAC B
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FCoE
Frame
Cisco Confidential
28
FC Storage
VF_port
FC link
FCID 7.1.1
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
© 2010 Cisco and/or its affiliates. All rights reserved.
Ethernet link
FC Fabric
FC Domain 7
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
VN_port
FC Domain 1
MAC A
Dest. = MAC C
Srce. = MAC A
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
Ethernet link
No domain
MAC B
FCID 1.1.1
MAC C
Dest. = MAC C
Srce. = MAC A
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FCoE
Frame
Cisco Confidential
29
•  Intermediate switches in the ethernet cloud
All are FIP snooping bridges
FC Storage
VE_port
FCID 7.1.1
FC link
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
© 2010 Cisco and/or its affiliates. All rights reserved.
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
VF_port
Ethernet
Fabric
FC Fabric
FC Domain 7
VE_port
FC Domain 3
MAC A
Dest. = MAC B
Srce. = MAC A
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
VN_port
Ethernet
Fabric
FC Domain 1
MAC B
FCID 1.1.1
MAC C
Dest. = MAC C
Srce. = MAC B
D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FCoE
Frame
Cisco Confidential
30
•  All links are forwarding
•  IS-IS computes the forwarding tables
L2MP
Backbone
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
31
Nexus
LAN
Distinct Management
and Control Planes
Single Data Plane
vPC
Virtual Port-channels
Cisco Nexus 5000
Consolidated I/O
Servers to Access Layer
Data Center Ethernet / FCoE Links
© 2010 Cisco and/or its affiliates. All rights reserved.
MDS
SAN A
SAN B
Preserves Existing
Dual SAN
Environments
10 GbE
Fibre Channel
10 GbE FCoE
Cisco Confidential
32
Nexus
MDS
LAN
Distinct Management
and Control Planes
Single Data Plan
SAN A
SAN B
VPC
Preserves Existing
Dual SAN
Environments
Virtual Port-channels
Cisco Nexus 5000
VPC
Active/Active
Consolidated I/O
Servers to Access Layer
Data Centre Ethernet / FCoE Links
© 2010 Cisco and/or its affiliates. All rights reserved.
10 GbE
Fibre Channel
10 GbE FCoE
Cisco Confidential
33
Thank you.
L2MP and FCoE
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
35
•  All links are forwarding
•  IS-IS computes the forwarding tables
This will be represented in the
following slides as a cloud for
simplicity
L2MP
Backbone
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
36
SAN-A
SAN A
SAN B
SAN-B
L2MP
Backbone
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
37
•  IS-IS provides L2MP for non-FCoE
•  Only access switches may be an exit point toward FC
Exit points use 4/8 GFC
•  Access switches may be FCFs
Domain Number = Number of access switches
Eight in the example (four per each fabric)
FSPF provides L2MP for FCoE
•  In alternative, access switches may be NPIV proxy
No additional domains are required
•  Easy to implement, but not really a unified fabric
•  SAN-A and SAN-B have no common Data Plane or Control Plane
•  SAN Admin and Network Admin roles are clearly separated
•  An FCoE host may need to cross the legacy Ethernet Cloud to reach an FCoE Storage
Array
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
38
SAN A
SAN B
These links are now
10GE FCoE
L2MP
Backbone
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
39
Differences with #1 in orange
•  IS-IS provides L2MP for non-FCoE
•  Only access switches may be an exit point toward FC
Exit points use 10 GE FCoE
•  Access switches may be FCFs
Domain Number = Number of access switches
Eight in the example (four per each fabric)
FSPF provides L2MP for FCoE
•  In alternative, Access switches may be NPIV proxy
No additional domains are required
•  SAN-A and SAN-B have no common Data Plane or Control Plane
•  SAN Admin and Network Admin roles are clearly separated
•  An FCoE host may reach a native FCoE Storage Array without going through legacy FC
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
40
SAN A
SAN B
All the L2MP switches
are also FCFs
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
41
Differences with #1a in orange
•  IS-IS provides L2MP for non-FCoE
•  Any switch may be an exit point toward FC
They may use 4/8 GFC or 10 GE FCoE
•  All switches are FCFs
Domain Number = Number of access and core switches
Twelve in the example (six per each fabric)
FSPF provides L2MP for FCoE
•  SAN-A and SAN-B have no common Data Plane or Control Plane
•  SAN Admin and Network Admin roles are clearly separated
•  An FCoE host may reach a native FCoE Storage Array without going through legacy FC
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
42
•  Let’s focus on an access switch of SAN-B
These 4 links can be used for L2MP of nonFCoE traffic
These 2 links can be used for L2MP of FCoE
traffic
These 2 links are more prone to
congestion
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
43
•  Share the switches, dedicate the links
These 4 links can be used for L2MP of nonFCoE traffic
These 2 red links are 10GE links dedicated to
L2MP of FCoE traffic
No link is more prone to congestion
This also provides better traffic isolation than VSANs/VLANs:
different ports, different protocols!
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
44
SAN A
SAN B
FCFs
A mix of FCFs and NPIV Proxy
NPIV
Proxies
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
45
Differences with #2 in orange
•  IS-IS provides L2MP for non-FCoE
•  Core switches may be an exit point toward FC
They may use 4/8 GFC or 10 GE FCoE
•  Core switches are FCFs and Access switches are NPIV proxy
One additional FC domain per core switch is required
4 in the example (2 per each fabric)
•  SAN-A and SAN-B have no common Data Plane or Control Plane
•  SAN Admin and Network Admin roles are clearly separated
•  An FCoE host may reach a native FCoE Storage Array without going through legacy FC
© 2010 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential
46
Thank you.
Download PDF
Similar pages