6–Configuring RoCE
Preparing the Adapter
Preparing the Adapter
The following steps describe how to enable DCBX, and specify the RoCE priority using the HII management application. For information about the HII application,
see Chapter 5, Preboot Adapter Configuration
.
1.
In the Main Configuration Page, select Data Center Bridging (DCB)
Settings, and then click Finish.
2.
In the Data Center Bridging (DCB) Settings window, click the DCBX
Protocol drop-down. The 45000 Series Adapter supports both CEE and
IEEE protocols. This value should match the corresponding value on the
DCB switch. In this example, select CEE.
3.
Click the RoCE Priority field, and type a priority value. This value should match the corresponding value on the DCB switch. In this example, type 4.
4.
Click Back.
5.
When prompted, click Yes to save the changes. Changes will not take effect unit after a system reset.
Preparing the Ethernet Switch
This section describes how to configure a Cisco
®
Nexus
®
6000 Ethernet Switch and a Dell
®
Z9100 Ethernet switch for RoCE.
Cisco Nexus 6000 Ethernet Switch
Dell Z9100 Ethernet Switch
Arista 7060X Switch
Cisco Nexus 6000 Ethernet Switch
Configuring the Cisco Nexus 6000 Ethernet Switch for RoCE comprises configuring class maps, configuring policy maps, applying the policy, and assigning a VLAN ID to the switch port.
1.
Open a config terminal session.
Switch# config terminal switch(config)#
2.
Configure quality of service (QoS) class maps, setting the RoCE priority to match the adapter (4).
switch(config)# class-map type qos class-roce switch(config)# match cos 4
44 BC0154501-00 F
6–Configuring RoCE
Preparing the Ethernet Switch
3.
Configure queuing class maps.
switch(config)# class-map type queuing class-roce switch(config)# match qos-group 3
4.
Configure network QoS class maps.
switch(config)# class-map type network-qos class-ro switch(config)# match qos-group 3
5.
Configure QoS policy maps.
switch(config)# policy-map type qos roce switch(config)# class type qos class-roc switch(config)# set qos-group 3
6.
Configure queuing policy maps to assign network bandwidth. In this example, use a value of 50 percent.
switch(config)# policy-map type queuing roce switch(config)# class type queuing class-roce switch(config)# bandwidth percent 50
7.
Configure network QoS policy maps to set priority flow control for no drop traffic class.
switch(config)# policy-map type network-qos roce switch(config)# class type network-qos class-roce switch(config)# pause no-drop
8.
Apply the new policy at the system level.
switch(config)# system qos switch(config)# service-policy type qos input roce switch(config)# service-policy type queuing output roce switch(config)# service-policy type queuing input roce switch(config)# service-policy type network-qos roce
9.
Assign a VLAN ID to the switch port to match the VLAN ID assigned to the adapter (5).
switch(config)# interface ethernet x/x switch(config)# switchport mode trunk switch(config)# switchport trunk allowed vlan 1,5
45 BC0154501-00 F
6–Configuring RoCE
Preparing the Ethernet Switch
Dell Z9100 Ethernet Switch
Configuring the Dell Z9100 Ethernet Switch for RoCE comprises configuring a
DCB map for RoCE, configuring priority-based flow control (PFC) and enhanced transmission selection (ETS), verifying the DCB map, applying the DCB map to the port, verifying PFC and ETS on the port, specifying the DCB protocol, and assigning a VLAN ID to the switch port.
1.
Create a DCB map.
Dell# configure
Dell(conf)# dcb-map roce
Dell(conf-dcbmap-roce)#
2.
Configure two ETS traffic classes in the DCB map with 50 percent bandwidth assigned for RoCE (group 1).
Dell(conf-dcbmap-roce)# priority-group 0 bandwidth 50 pfc off
Dell(conf-dcbmap-roce)# priority-group 1 bandwidth 50 pfc on
3.
Configure the PFC priority to match the adapter priority (4).
Dell(conf-dcbmap-roce)# priority-pgid 0 0 0 0 1 0 0 0
4.
Verify the DCB map configuration. Priority group
Dell(conf-dcbmap-roce)# do show qos dcb-map roce
-----------------------
State :Complete
PfcMode :ON
--------------------
PG:0 TSA:ETS BW:40 PFC:OFF
Priorities:0 1 2 3 5 6 7
PG:1 TSA:ETS BW:60 PFC:ON
Priorities:4
5.
Apply the DCB map to the port.
Dell(conf)# interface twentyFiveGigE 1/8/1
Dell(conf-if-tf-1/8/1)# dcb-map roce
46 BC0154501-00 F
6–Configuring RoCE
Preparing the Ethernet Switch
6.
Verify the ETS and PFC configuration on the port.
Dell(conf-if-tf-1/8/1)# do show interfaces twentyFiveGigE 1/8/1 ets summary
Interface twentyFiveGigE 1/8/1
Max Supported TC is 4
Number of Traffic Classes is 8
Admin mode is on
Admin Parameters :
------------------
Admin is enabled
PG-grp Priority# BW-% BW-COMMITTED BW-PEAK TSA
% Rate(Mbps) Burst(KB) Rate(Mbps) Burst(KB)
------------------------------------------------------------------------
0 0,1,2,3,5,6,7 40 - - - - ETS
1 4 60 - - - - ETS
2 - - - - - -
3 - - - - - -
Dell(Conf)# do show interfaces twentyFiveGigE 1/5/1 pfc detail
Interface twentyFiveGigE 1/5/1
Admin mode is on
Admin is enabled, Priority list is 4
Remote is enabled, Priority list is 4
Remote Willing Status is enabled
Local is enabled, Priority list is 4
Oper status is init
PFC DCBX Oper status is Up
State Machine Type is Feature
TLV Tx Status is enabled
PFC Link Delay 65535 pause quntams
Application Priority TLV Parameters :
--------------------------------------
FCOE TLV Tx Status is disabled
ISCSI TLV Tx Status is enabled
Local FCOE PriorityMap is 0x0
Local ISCSI PriorityMap is 0x10
Remote ISCSI PriorityMap is 0x10
66 Input TLV pkts, 99 Output TLV pkts, 0 Error pkts, 0 Pause Tx pkts, 0 Pause Rx pkts
66 Input Appln Priority TLV pkts, 99 Output Appln Priority TLV pkts, 0 Error Appln
Priority TLV Pkts
7.
Configure the DCBX protocol (CEE in this example).
Dell(conf)# interface twentyFiveGigE 1/8/1
Dell(conf-if-tf-1/8/1)# protocol lldp
Dell(conf-if-tf-1/8/1-lldp)# dcbx version cee
47 BC0154501-00 F
6–Configuring RoCE
Preparing the Ethernet Switch
8.
Assign a VLAN ID to the switch port to match the VLAN ID assigned to the adapter (5).
Dell(conf)# interface vlan 5
Dell(conf-if-vl-5)# tagged twentyFiveGigE 1/8/1
Arista 7060X Switch
Configuring RoCE APP TLVs for both RoCE v1 and RoCE v2
The following is an example on configuring RoCE APP TLV for both RoCE v1 and and RoCE v2:
Arista-7060X-EIT(config)#dcbx application ether 0x8915 priority 4
Arista-7060X-EIT(config)#dcbx application udp 4791 priority 4
Mapping Priority to Traffic Class
In the example below, Priority 4 is mapped to traffic class 1, and the rest of the priorities are mapped to traffic class 0:
Arista-7060X-EIT(config)#dcbx ets qos map cos 0 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 1 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 2 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 3 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 4 traffic-class 1
Arista-7060X-EIT(config)#dcbx ets qos map cos 5 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 6 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 7 traffic-class 0
Setting up ETS
In example below, traffic class 0 is given bandwidth 5%, and traffic class 1 is given bandwidth 95%:
Arista-7060X-EIT(config)#dcbx ets traffic-class 0 bandwidth 5
Arista-7060X-EIT(config)#dcbx ets traffic-class 1 bandwidth 95
Configuring Interface
The same VLAN ID needs to be assigned to the Server Adapter Ports. The configuration below needs to be done on all the switch interfaces where the
Server Adapter Ports are connected. DCBX Protocol mode is set to CEE. PFC is enabled for Priority 4.
Arista-7060X-EIT(config)#interface ethernet 26/1
Arista-7060X-EIT(config-if-Et26/1)#dcbx mode cee
Arista-7060X-EIT(config-if-Et26/1)# priority-flow-control mode on
Arista-7060X-EIT(config-if-Et26/1)# priority-flow-control priority 4 no-drop
Arista-7060X-EIT(config-if-Et26/1)# switchport mode trunk
Arista-7060X-EIT(config-if-Et26/1)# switchport trunk allowed vlan 1,5
48 BC0154501-00 F
