Cavium FastLinQ 41000 Network Adapter User’s Guide

The FastLinQ 41000 series network adapters offer a range of features designed to improve network performance and efficiency. The adapters support a variety of protocols including iSCSI, FCoE, RoCE, and iWARP. They also offer features such as SR-IOV, VMMQ, and VXLAN offload. The adapter is compatible with a range of operating systems, including Windows, Linux, and VMware.
PDF
Document
User’s Guide
Converged Network Adapters and
Intelligent Ethernet Adapters
FastLinQ 41000 Series
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
Document Revision History
Revision 01, February 7, 2017
Revision A, June 26, 2017
Changes
Sections Affected
Updated the document title.
Front title page
For the supported products:
“Supported Products” on page xvi
 Changed the section title from “Overview” to
“Supported Products”.
 Added 10GB Intelligent Ethernet Adapter
QL41112HLCU-CK/SP/BK to the list of supported adapters.
 Removed the note referencing pre-GA availability of adapters.
Added a list of documents applicable to this product.
“Related Materials” on page xvii
Updated the features bullets with new features in
this release.
“Features” on page 1
For the adapter management tools:
“Adapter Management” on page 3
 Added a cross-reference to the new Related
Materials section.
 Added a note stating the requirement of RPC
agents for QCS CLI, QCC GUI, and QCC PowerKit.
 Updated the QLogic QConvergeConsole GUI
description.
 Added QConvergeConsole PowerKit as a tool.
In Table 2-1, corrected the requirements in the
PCIe row. In the note following Table 2-2, updated
the downloads information.
“System Requirements” on page 7
In the To install the adapter procedure, updated
Step 6 and Step 8.
“Installing the Adapter” on page 9
In the section introduction, replaced RoCE with
RDMA, and added iWARP. In Table 3-1, updated
the description of the qed driver.
“Installing Linux Driver Software” on page 11
Renamed the section title and references within
the section from RoCE to RDMA.
“Installing the Linux Drivers Without RDMA” on
page 13
Changed the references from RoCE to RDMA.
“Removing the Linux Drivers” on page 13
ii
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
Renamed the section title and references within
the section from RoCE to RDMA. Added a new
step, “To build and install the libqedr user space
library...”
“Installing the Linux Drivers with RDMA” on
page 17
In Table 3-2, changed the parameter
gro_enable to gro_disable.
“Linux Driver Optional Parameters” on page 18
Changed the section title from “Linux Driver
Parameter Defaults” to “Linux Driver Operation
Defaults” and changed “parameter” to “operation”
in the section and in Table 3-3.
“Linux Driver Operation Defaults” on page 18
Changed the title from “ESXi Driver Packages by
Release” to “ESXi Driver Packages,” and removed
the Version column.
“VMware Drivers and Driver Packages” on
page 25
Added adapter properties to Table 5-1.
“Getting Started” on page 34
Updated the To configure the device-level
parameters procedure and screen shots. Updated
the parameters in Table 5-2.
“Configuring Device-level Parameters” on page 38
Updated the To configure the port-level parameters procedure and screen shots.
“Configuring Port-level Parameters” on page 39
In the introduction to Table 6-1, added iWARP. In
Table 6-1, updated the title, table column headings
and the data for Linux, Ubuntu, and CentOS.
“Supported Operating Systems and OFED” on
page 53
Corrected the section title (was “Planning for
FCoE”).
“Planning for RoCE” on page 54
Moved the subsection, “RoCE v2 Configuration for
Linux” to the end of the section.
“Configuring RoCE on the Adapter for Linux” on
page 62
In the To verify RoCE configuration on Linux
procedure, updated step 2.
“Verifying the RoCE Configuration on Linux” on
page 67
Added a new chapter for configuring Internet wide
area RDMA protocol (iWARP).
Chapter 7 iWARP Configuration
Added a new chapter for configuring iSCSI.
Chapter 8 iSCSI Configuration
Added a new chapter for configuring Fibre Channel over Ethernet (FCoE).
Chapter 9 FCoE Configuration
Added a new chapter for configuring single root
input/output virtualization (SR-IOV).
Chapter 10 SR-IOV Configuration
Updated the chapter title from “iSCSI Extensions
for RDMA.”
Chapter 11 iSER Configuration
iii
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
Updated the list of OSs on which iSER is supported in inbox OFED.
“Before You Begin” on page 155
In step 1, updated “OFED-3.18/3.81-1 GA” to
“OFED 3.18-2 GA/3.18-3 GA”.
“Configuring iSER for RHEL” on page 155
Removed two bnx2x legacy driver issues.
“Linux-specific Issues” on page 213
Added a new section in the troubleshooting chapter.
“Collecting Debug Data” on page 215
Updated Table B-1 with new cables and optical
solutions and added a link to the Web for the new
QLogic FastLinQ 41000/45000 Series Interoperability Matrix for the latest support list.
“Tested Cables and Optical Modules” on page 218
Added a new section in the cables and optical
modules appendix.
“Tested Switches” on page 222
Added a new appendix listing information about
feature constraints implemented in the current
release.
Appendix C Feature Constraints
Added terms and definitions to the glossary.
Glossary
iv
AH0054601-00 A
Table of Contents
Preface
Supported Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
What Is in This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Related Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
License Agreements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Technical Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Downloading Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contact Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Knowledge Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Warranty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Laser Safety—FDA Notice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Agency Certification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
EMI and EMC Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . .
KCC: Class A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Product Safety Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
xvi
xvi
xvi
xvii
xviii
xx
xx
xx
xxi
xxi
xxii
xxii
xxii
xxii
xxii
xxiii
xxiii
xxiv
Product Overview
Functional Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Adapter Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
QLogic Control Suite CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
QLogic QConvergeConsole GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
QLogic QConvergeConsole vCenter Plug-in. . . . . . . . . . . . . . . . . . . .
QConvergeConsole PowerKit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FastLinQ ESXCLI VMware Plug-in . . . . . . . . . . . . . . . . . . . . . . . . . . .
Adapter Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Physical Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Standards Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
v
1
1
3
4
4
4
5
5
5
5
5
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
2
Hardware Installation
System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Preinstallation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
Driver Installation
Installing Linux Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing the Linux Drivers Without RDMA . . . . . . . . . . . . . . . . . . . . .
Removing the Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing Linux Drivers Using the src RPM Package . . . . . . . . .
Installing Linux Drivers Using the kmp/kmod RPM Package . . .
Installing Ubuntu Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing Linux Drivers Using the TAR File. . . . . . . . . . . . . . . . .
Installing the Linux Drivers with RDMA . . . . . . . . . . . . . . . . . . . . . . . .
Linux Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Linux Driver Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing Windows Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Removing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Managing Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Setting Power Management Options. . . . . . . . . . . . . . . . . . . . . . . . . .
Installing Drivers for Windows Nano Server . . . . . . . . . . . . . . . . . . . .
Creating a Nano ISO Image, Injecting Drivers, and Updating the
Multiboot/Flash Image on a Nano Server . . . . . . . . . . . . . . . . . . . . .
Installing VMware Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
VMware Drivers and Driver Packages. . . . . . . . . . . . . . . . . . . . . . . . .
Installing the VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
VMware Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . .
VMware Driver Parameter Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . .
Removing the VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FCoE Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
7
8
9
9
11
13
13
15
16
16
16
17
18
18
19
19
19
19
20
20
22
22
23
24
25
26
27
29
30
30
30
Firmware Upgrade Utility
Image Verification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Upgrading Adapter Firmware on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Upgrading Adapter Firmware on Windows. . . . . . . . . . . . . . . . . . . . . . . . . .
Upgrading Adapter Firmware on Windows Nano . . . . . . . . . . . . . . . . . . . . .
vi
31
32
32
33
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
5
Adapter Preboot Configuration
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Displaying Firmware Image Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring Device-level Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring Port-level Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring FCoE Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
RoCE Configuration
Supported Operating Systems and OFED . . . . . . . . . . . . . . . . . . . . . . . . . .
Planning for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Preparing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Preparing the Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring the Cisco Nexus 6000 Ethernet Switch . . . . . . . . . . . . . .
Configuring the Dell Z9100 Ethernet Switch . . . . . . . . . . . . . . . . . . . .
Configuring the Arista 7060X Ethernet Switch . . . . . . . . . . . . . . . . . .
Configuring RoCE on the Adapter for Windows Server . . . . . . . . . . . . . . . .
Configuring RoCE on the Adapter for Linux . . . . . . . . . . . . . . . . . . . . . . . . .
RoCE Configuration for RHEL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
RoCE Configuration for SLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
RoCE Configuration for Ubuntu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Verifying the RoCE Configuration on Linux . . . . . . . . . . . . . . . . . . . . .
VLAN Interfaces and GID Index Values . . . . . . . . . . . . . . . . . . . . . . .
RoCE v2 Configuration for Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Identifying RoCE v2 GID Index or Address . . . . . . . . . . . . . . . .
Verifying RoCE v1 or v2 GID Index and Address from sys
and class Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Verifying RoCE v1 or v2 Functionality Through perftest
Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
34
38
38
39
43
44
47
53
54
55
55
55
57
59
60
62
63
64
64
67
69
70
70
71
72
iWARP Configuration
Configuring iWARP on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring iWARP on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Detecting the Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supported iWARP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Running Perftest for iWARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring NFS-RDMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vii
76
81
81
81
82
82
83
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
8
iSCSI Configuration
iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI Boot Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Selecting the Preferred iSCSI Boot Mode . . . . . . . . . . . . . . . . .
Configuring the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring iSCSI Boot Parameters . . . . . . . . . . . . . . . . . . . . . .
Adapter UEFI Boot Mode Configuration . . . . . . . . . . . . . . . . . . . . . . .
Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Static iSCSI Boot Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Dynamic iSCSI Boot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . .
Enabling CHAP Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring the DHCP Server to Support iSCSI Boot . . . . . . . . . . . . . . . . .
DHCP iSCSI Boot Configurations for IPv4 . . . . . . . . . . . . . . . . . . . . .
DHCP Option 17, Root Path . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DHCP Option 43, Vendor-specific Information . . . . . . . . . . . . . .
Configuring the DHCP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring DHCP iSCSI Boot for IPv6 . . . . . . . . . . . . . . . . . . . . . . . .
DHCPv6 Option 16, Vendor Class Option . . . . . . . . . . . . . . . . .
DHCPv6 Option 17, Vendor-Specific Information . . . . . . . . . . . .
Configuring VLANs for iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring iSCSI Boot from SAN for SLES 12 . . . . . . . . . . . . . . . . . . . . . .
iSCSI Offload in Windows Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing QLogic Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installing the Microsoft iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring Microsoft Initiator to Use QLogic’s iSCSI Offload. . . . . . .
iSCSI Offload FAQs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows Server 2012, 2012 R2, and 2016 iSCSI Boot Installation . .
iSCSI Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI Offload in Linux Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Differences from bnx2i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring qedi.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Verifying iSCSI Interfaces in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Open-iSCSI and Boot from SAN Considerations . . . . . . . . . . . . . . . . . . . . .
9
85
86
86
86
87
88
90
91
97
99
100
100
100
101
102
102
102
102
103
104
107
108
108
108
114
115
116
116
116
117
117
120
FCoE Configuration
FCoE Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Preparing System BIOS for FCoE Build and Boot. . . . . . . . . . . . . . . .
Specifying the BIOS Boot Protocol . . . . . . . . . . . . . . . . . . . . . . .
Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . .
viii
123
124
124
124
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
Windows FCoE Boot from SAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows Server 2012, 2012 R2, and 2016 FCoE Boot
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring FCoE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FCoE Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Injecting (Slipstreaming) Adapter Drivers into Windows Image Files . . . . . .
Configuring Linux FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Differences Between qedf and bnx2fc . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring qedf.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Verifying FCoE Devices in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Boot from SAN Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
136
143
150
iSER Configuration
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring iSER for RHEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring iSER for SLES 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Using iSER with iWARP on RHEL and SLES . . . . . . . . . . . . . . . . . . . . . . .
Configuring iSER for Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring LIO as Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring the Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Optimizing Linux Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring CPUs to Maximum Performance Mode . . . . . . . . . . . . . .
Configuring Kernel sysctl Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring IRQ Affinity Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring Block Device Staging . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
129
130
130
131
132
133
133
134
135
SR-IOV Configuration
Configuring SR-IOV on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring SR-IOV on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring SR-IOV on VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
129
155
155
159
160
161
161
167
168
168
169
169
169
Windows Server 2016
Configuring RoCE Interfaces with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . .
Creating a Hyper-V Virtual Switch with an RDMA Virtual NIC. . . . . . .
Adding a VLAN ID to Host Virtual NIC. . . . . . . . . . . . . . . . . . . . . . . . .
Verifying If RoCE is Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Adding Host Virtual NICs (Virtual Ports) . . . . . . . . . . . . . . . . . . . . . . .
Mapping the SMB Drive and Running RoCE Traffic . . . . . . . . . . . . . .
RoCE over Switch Embedded Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Creating a Hyper-V Virtual Switch with SET and RDMA Virtual NICs .
Enabling RDMA on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
ix
170
171
172
173
173
174
175
176
176
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
RoCE over Switch Embedded Teaming (continued)
Assigning a VLAN ID on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Running RDMA Traffic on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring QoS for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring QoS by Disabling DCBX on the Adapter . . . . . . . . . . . . .
Configuring QoS by Enabling DCBX on the Adapter. . . . . . . . . . . . . .
Configuring VMMQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Enabling VMMQ on the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Setting the VMMQ Max QPs Default and Non-Default VPort . . . . . . .
Creating a Virtual Machine Switch with or Without SR-IOV . . . . . . . .
Enabling VMMQ on the Virtual Machine Switch . . . . . . . . . . . . . . . . .
Getting the Virtual Machine Switch Capability . . . . . . . . . . . . . . . . . . .
Creating a VM and Enabling VMMQ on VMNetworkadapters in
the VM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Default and Maximum VMMQ Virtual NIC . . . . . . . . . . . . . . . . . . . . . .
Enabling and Disabling VMMQ on a Management NIC . . . . . . . . . . .
Monitoring Traffic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Enabling VXLAN Offload on the Adapter. . . . . . . . . . . . . . . . . . . . . . .
Deploying a Software Defined Network. . . . . . . . . . . . . . . . . . . . . . . .
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring the Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Deploying a Hyper-Converged System . . . . . . . . . . . . . . . . . . . . . . . .
Deploying the Operating System . . . . . . . . . . . . . . . . . . . . . . . .
Configuring the Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuring Storage Spaces Direct. . . . . . . . . . . . . . . . . . . . . . .
Deploying and Managing a Nano Server . . . . . . . . . . . . . . . . . . . . . . . . . . .
Roles and Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Deploying a Nano Server on a Physical Server . . . . . . . . . . . . . . . . .
Deploying a Nano Server in a Virtual Machine . . . . . . . . . . . . . . . . . .
Managing a Nano Server Remotely . . . . . . . . . . . . . . . . . . . . . . . . . .
Managing a Nano Server with Windows PowerShell Remoting .
Adding the Nano Server to a List of Trusted Hosts . . . . . . . . . .
Starting the Remote Windows PowerShell Session . . . . . . . . . .
Managing QLogic Adapters on a Windows Nano Server . . . . . . . . . .
RoCE Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
x
176
177
177
177
182
185
186
186
187
189
189
190
191
191
191
192
192
192
193
193
194
194
194
196
199
199
201
203
205
205
205
206
206
206
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
13
Troubleshooting
Troubleshooting Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Verifying that Current Drivers Are Loaded . . . . . . . . . . . . . . . . . . . . . . . . . .
Verifying Drivers in Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Verifying Drivers in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Verifying Drivers in VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Testing Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Testing Network Connectivity for Windows . . . . . . . . . . . . . . . . . . . . .
Testing Network Connectivity for Linux . . . . . . . . . . . . . . . . . . . . . . . .
Microsoft Virtualization with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Linux-specific Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Miscellaneous Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Troubleshooting Windows FCoE and iSCSI Boot from SAN . . . . . . . . . . . .
Collecting Debug Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A
Adapter LEDS
B
Cables and Optical Modules
Supported Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Tested Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C
210
211
211
211
212
212
212
213
213
213
213
214
215
217
218
222
Feature Constraints
Glossary
Index
xi
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
List of Figures
Figure
Page
3-1
Setting Advanced Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
3-2
Power Management Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
5-1
System Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
5-2
System Setup: Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
5-3
Main Configuration Page, Setting Default Partitioning Mode . . . . . . . . . . . . . . . . . .
36
5-4
Main Configuration Page, Setting NPAR Partitioning Mode. . . . . . . . . . . . . . . . . . .
36
5-5
Firmware Information Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
5-6
Device Level Configuration Page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
5-7
Port Level Configuration Page: Setting Link Speed . . . . . . . . . . . . . . . . . . . . . . . . .
41
5-8
Port Level Configuration Page: Setting Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . .
42
5-9
FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
5-10 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
5-11 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
46
5-12 iSCSI Initiator Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
46
5-13 iSCSI First Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
5-14 iSCSI Second Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
5-15 Partitions Configuration Page (No FCoE Offload or iSCSI Offload). . . . . . . . . . . . .
48
5-16 Partitions Configuration Page (with FCoE Offload and iSCSI Offload) . . . . . . . . . .
48
5-17 Global Bandwidth Allocation Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49
5-18 Partition 1 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
5-19 Partition 2 Configuration: FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51
5-20 Partition 3 Configuration: iSCSI Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51
5-21 Partition 4 Configuration: Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
6-1
Configuring RoCE Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
61
6-2
Switch Settings, Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
73
6-3
Switch Settings, Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
74
6-4
Configuring RDMA_CM Applications: Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
74
6-5
Configuring RDMA_CM Applications: Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
75
7-1
System Setup for iWARP: NIC Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
77
7-2
Windows PowerShell Command: Get-NetAdapterRdma . . . . . . . . . . . . . . . . . . . . .
78
7-3
Windows PowerShell Command: Get-NetOffloadGlobalSetting . . . . . . . . . . . . . . .
78
7-4
Perfmon: Add Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
79
7-5
Perfmon: Verifying iWARP Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
80
8-1
Systems Utilities at Boot Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
88
8-2
Configuration Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
88
8-3
Selecting Port Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
89
8-4
Port Level Configuration, Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
90
8-5
Selecting iSCSI Boot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
91
8-6
Selecting General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
92
8-7
iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
92
8-8
Selecting iSCSI Initiator Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
93
8-9
iSCSI Initiator Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
94
8-10 iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
94
xii
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
8-11
8-12
8-13
8-14
8-15
8-16
8-17
8-18
8-19
8-20
8-21
8-22
8-23
8-24
8-25
8-26
9-1
9-2
9-3
9-4
9-5
9-6
9-7
9-8
10-1
10-2
10-3
10-4
10-5
10-6
10-7
10-8
10-9
10-10
10-11
10-12
10-13
10-14
10-15
11-1
11-2
11-3
11-4
11-5
12-1
iSCSI First Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI Second Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Saving iSCSI Changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI Initiator Configuration, VLAN ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Configuration: Setting Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Configuration: Setting Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Configuration: Setting DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Configuration: Setting DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI Initiator Properties, Configuration Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI Initiator Node Name Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSCSI Initiator—Discover Target Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Target Portal IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Selecting the Initiator IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Connecting to the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Connect To Target Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Utilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Configuration, Port Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Port Level Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Boot Mode in Port Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FCoE Offload Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Selecting General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Setup for SR-IOV: Integrated Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Setup for SR-IOV: Device Level Configuration . . . . . . . . . . . . . . . . . . . . . .
Adapter Properties, Advanced: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . .
Virtual Switch Manager: Enabling SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Settings for VM: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Device Manager: VM with QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: Get-NetadapterSriovVf . . . . . . . . . . . . . . . . . . . .
System Setup: Processor Settings for SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System Setup for SR-IOV: Integrated Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Editing the grub.conf File for SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Command Output for sriov_numvfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Command Output for ip link show Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
RHEL68 Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Add New Virtual Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
VMware Host Edit Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
RDMA Ping Successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iSER Portal Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Iface Transport Confirmed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Checking for New iSCSI Device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
LIO Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Enabling RDMA in Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xiii
95
96
97
99
104
104
105
105
107
109
109
110
111
112
113
114
124
125
125
126
127
127
128
129
137
137
138
139
141
142
143
144
145
146
147
148
149
150
153
156
157
158
158
160
171
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
12-2
12-3
12-4
12-5
12-6
12-7
12-8
12-9
12-10
12-11
12-12
12-13
12-14
12-15
12-16
12-17
12-18
12-19
12-20
12-21
12-22
12-23
13-1
Hyper-V Virtual Ethernet Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: Get-VMNetworkAdapter . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: Get-NetAdapterRdma . . . . . . . . . . . . . . . . . . . . .
Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Performance Monitor Shows RoCE Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: New-VMSwitch . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: Get-NetAdapter. . . . . . . . . . . . . . . . . . . . . . . . . .
Advanced Properties: Enable QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Advanced Properties: Enabling QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Advanced Properties: Enabling Virtual Switch RSS. . . . . . . . . . . . . . . . . . . . . . . . .
Advanced Properties: Setting VMMQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Virtual Switch Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: Get-VMSwitch. . . . . . . . . . . . . . . . . . . . . . . . . . .
Advanced Properties: Enabling VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Example Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: Get-NetAdapter. . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: Get-NetAdapterRdma . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: New-Item . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: New-SMBShare . . . . . . . . . . . . . . . . . . . . . . . . .
Windows PowerShell Command: Get-NetAdapterStatistics . . . . . . . . . . . . . . . . . .
Windows Setup Error Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xiv
172
172
173
174
175
176
176
178
179
183
184
186
187
188
189
192
193
207
207
208
208
209
214
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
List of Tables
Table
2-1
2-2
3-1
3-2
3-3
3-4
3-5
3-6
3-7
3-8
3-9
3-10
5-1
5-2
6-1
6-2
8-1
8-2
8-3
8-4
12-1
13-1
A-1
B-1
B-2
Host Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Minimum Host Operating System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . .
QLogic 41000 Series Adapters Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
qede Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
ESXi Driver Packages by Release. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
VMware Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
VMware Driver Parameter Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
QLogic 41000 Series Adapter VMware FCoE Driver . . . . . . . . . . . . . . . . . . . . . . . .
QLogic 41000 Series Adapter iSCSI Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Device-level Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
OS Support for RoCE v1/v2, iWARP, and OFED . . . . . . . . . . . . . . . . . . . . . . . . . . .
Advanced Properties for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DHCP Option 17 Parameter Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DHCP Option 43 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
DHCP Option 17 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Roles and Features of Nano Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Collecting Debug Data Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Adapter Port Link and Activity LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Switches Tested for Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xv
Page
7
8
11
18
18
23
25
25
27
29
30
30
37
39
53
60
87
100
101
103
199
215
216
218
222
AH0054601-00 A
Preface
This preface lists the supported products, specifies the intended audience,
explains the typographic conventions used in this guide, lists related documents,
provides technical support and contact information, and describes legal notices.
Supported Products
This user’s guide describes the following FastLinQ products:

10Gb Intelligent Ethernet Adapter: QL41112HLCU-CK/SP/BK

10Gb Intelligent Ethernet Adapter: QL41112HLRJ-CK/SP/BK

10Gb Converged Network Adapter: QL41162HLRJ-CK/SP/BK

10/25Gb Intelligent Ethernet Adapter: QL41212HLCU-CK/SP/BK

10/25Gb Converged Network Adapter: QL41262HLCU-CK/SP/BK
Intended Audience
This guide is intended for system administrators and other technical staff
members responsible for configuring and managing adapters installed on servers
in Windows, Linux&reg;, or VMware&reg; environments.
What Is in This Guide
Following this preface, the remainder of this guide is organized into the following
chapters and appendices:

Chapter 1 Product Overview provides a product functional description, a list
of features, adapter management tool descriptions, and the adapter
specifications.

Chapter 2 Hardware Installation describes how to install the adapter,
including the list of system requirements and a preinstallation checklist.

Chapter 3 Driver Installation describes the installation of the adapter drivers
on Windows, Linux, and VMware.

Chapter 4 Firmware Upgrade Utility describes the use of the utility to
upgrade adapter firmware and boot code.
xvi
AH0054601-00 A
Preface
Related Materials

Chapter 5 Adapter Preboot Configuration describes the preboot adapter
configuration tasks using the Human Infrastructure Interface (HII)
application.

Chapter 6 RoCE Configuration describes how to configure the adapter, the
Ethernet switch, and the host to use RDMA over converged Ethernet
(RoCE).

Chapter 7 iWARP Configuration provides procedures for configuring Internet
wide area RDMA protocol (iWARP) on Windows and Linux systems.

Chapter 8 iSCSI Configuration describes iSCSI boot, iSCSI crash dump,
and iSCSI offload for Windows and Linux.

Chapter 9 FCoE Configuration describes Fibre Channel over Ethernet
(FCoE) boot from SAN and booting from SAN after installation.

Chapter 10 SR-IOV Configuration provides procedures for configuring single
root input/output virtualization (SR-IOV) on Windows, Linux, and VMware
systems.

Chapter 11 iSER Configuration describes how to configure iSCSI
Extensions for RDMA (iSER) for Linux RHEL and SLES.

Chapter 12 Windows Server 2016 describes the Windows Server 2016
features.

Chapter 13 Troubleshooting describes a variety of troubleshooting methods
and resources.

Appendix A Adapter LEDS lists the adapter LEDs and their significance.

Appendix B Cables and Optical Modules lists the cables and optical
modules that the 41000 Series Adapters support.

Appendix C Feature Constraints provides information about feature
constraints implemented in the current release.
At the end of this guide is a glossary of terms and an index to help you quickly
locate the information you need.
Related Materials
For additional information, refer to the following documents that are available on
the Downloads and Documentation page of the QLogic Web site:
http://driverdownloads.qlogic.com

Installation Guide—QConvergeConsole GUI (part number SN0051105-00)
contains detailed information on how to install and use the
QConvergeConsole GUI management tool.
xvii
AH0054601-00 A
Preface
Documentation Conventions

Help System—QConvergeConsole GUI help topics, available while using
the QCC GUI.

User's Guide—QLogic Control Suite CLI (part number BC0054511-00)
contains detailed information on how to install, start, and use QLogic Control
Suite CLI.

User's Guide—PowerShell (part number BC0054518-00) contains detailed
information on how to install QConvergeConsole FastLinQ PowerKit to
manage the QLogic FastLinQ adapters on the system.

User’s Guide—QConvergeConsole Plug-ins for vSphere (part number
SN0054677-00) provides details for using the two plug-ins to extend the
capabilities of VMware vCenter Server and the vSphere Web Client.

User’s Guide—FastLinQ ESXCLI VMware Plug-in (part number
BC0151101-00) describes the plug-in that extends the capabilities of the
ESX&reg; CLI to manage QLogic 3400, 8400, and 45000 Series Adapters
installed in VMware ESX/ESXi hosts.
Documentation Conventions
This guide uses the following documentation conventions:

NOTE

CAUTION
without an alert symbol indicates the presence of a hazard
that could cause damage to equipment or loss of data.

CAUTION with an alert symbol indicates the presence of a hazard that
could cause minor or moderate injury.

provides additional information.
!
!
WARNING indicates the presence of a hazard that could cause serious
injury or death.


Text in blue font indicates a hyperlink (jump) to a figure, table, or section in
this guide, and links to Web sites are shown in underlined blue. For
example:

Table 9-2 lists problems related to the user interface and remote agent.

See “Installation Checklist” on page 6.

For more information, visit www.qlogic.com.
Text in bold font indicates user interface elements such as a menu items,
buttons, check boxes, or column headings. For example:

Click the Start button, point to Programs, point to Accessories, and
then click Command Prompt.
xviii
AH0054601-00 A
Preface
Documentation Conventions




Under Notification Options, select the Warning Alarms check box.
Text in Courier font indicates a file name, directory path, or command line
text. For example:

To return to the root directory from anywhere in the file structure:
Type cd /root and press ENTER.

Issue the following command: sh ./install.bin
Key names and key strokes are indicated with UPPERCASE:

Press CTRL+P.

Press the UP ARROW key.
Text in italics indicates terms, emphasis, variables, or document titles. For
example:

For a complete listing of license agreements, refer to the QLogic
Software End User License Agreement.

What are shortcut keys?

To enter the date, type mm/dd/yyyy (where mm is the month, dd is the
day, and yyyy is the year).

Topic titles between quotation marks identify related topics either within this
manual or in the online help, which is also referred to as the help system
throughout this document.

Command line interface (CLI) command syntax conventions include the
following:

Plain text indicates items that you must type as shown. For example:


qaucli -pr nic -ei
&lt; &gt; (angle brackets) indicate a variable whose value you must
specify. For example:

&lt;serial_number&gt;
NOTE
For CLI commands only, variable names are always indicated
using angle brackets instead of italics.

[ ] (square brackets) indicate an optional parameter. For example:

[&lt;file_name&gt;] means specify a file name, or omit it to select
the default file name.

| (vertical bar) indicates mutually exclusive options; select one option
only. For example:
xix
AH0054601-00 A
Preface
License Agreements



on|off

1|2|3|4
... (ellipsis) indicates that the preceding item may be repeated. For
example:

x... means one or more instances of x.

[x...] means zero or more instances of x.
( ) (parentheses) and { } (braces) are used to avoid logical
ambiguity. For example:

a|b c is ambiguous
{(a|b) c} means a or b, followed by c
{a|(b c)} means either a, or b c
License Agreements
Refer to the QLogic Software End User License Agreement for a complete listing
of all license agreements affecting this product.
Technical Support
Customers should contact their authorized maintenance provider for technical
support of their QLogic products. QLogic-direct customers may contact QLogic
Technical Support; others will be redirected to their authorized maintenance
provider. Visit the QLogic support Web site listed in Contact Information for the
latest firmware and software updates.
For details about available service plans, or for information about renewing and
extending your service, visit the Service Program Web page:
www.qlogic.com/Support/Pages/ServicePrograms.aspx
Downloading Updates
The QLogic Web site provides periodic updates to product firmware, software,
and documentation.
To download firmware, software, and documentation:
1.
Go to the QLogic Downloads and Documentation page:
driverdownloads.qlogic.com
2.
Type the QLogic model name in the search box.
3.
In the search results list, locate and select the firmware, software, or
documentation for your product.
xx
AH0054601-00 A
Preface
Technical Support
4.
View the product details Web page to ensure that you have the correct
firmware, software, or documentation. For additional information, click
Read Me and Release Notes under Support Files.
5.
Click Download Now.
6.
Save the file to your computer.
7.
If you have downloaded firmware, software, drivers, or boot code, follow the
installation instructions in the Read Me file.
Instead of typing a model name in the search box, you can perform a guided
search as follows:
1.
Click the product type tab: Adapters, Switches, Routers, or ASICs.
2.
Click the corresponding button to search by model or operating system.
3.
Click an item in each selection column to define the search, and then click
Go.
4.
Locate the firmware, software, or document you need, and then click the
item’s name or icon to download or open the item.
Training
QLogic Global Training maintains a Web site at www.qlogictraining.com offering
online and instructor-led training for all QLogic products. In addition, sales and
technical professionals may obtain Associate and Specialist-level certifications to
qualify for additional benefits from QLogic.
Contact Information
QLogic Technical Support for products under warranty is available during local
standard working hours excluding QLogic Observed Holidays. For customers with
extended service, consult your plan for available hours. For Support phone
numbers, see the Contact Support link:
support.qlogic.com
Support Headquarters
QLogic Corporation
12701 Whitewater Drive
Minnetonka, MN 55343 USA
QLogic Web Site
www.qlogic.com
Technical Support Web Site
support.qlogic.com
Technical Support E-mail
[email protected]
Technical Training E-mail
[email protected]
xxi
AH0054601-00 A
Preface
Legal Notices
Knowledge Database
The QLogic knowledge database is an extensive collection of QLogic product
information that you can search for specific solutions. QLogic is constantly adding
to the collection of information in the database to provide answers to your most
urgent questions. Access the database from the QLogic Support Center:
support.qlogic.com
Legal Notices
Legal notices covered in this section include warranty, laser safety (FDA notice),
agency certification, and product safety compliance.
Warranty
For warranty details, please check the QLogic Web site:
www.qlogic.com/Support/Pages/Warranty.aspx
Laser Safety—FDA Notice
This product complies with DHHS Rules 21CFR Chapter I, Subchapter J. This
product has been designed and manufactured according to IEC60825-1 on the
safety label of laser product.
CLASS I LASER
Class 1
Laser Product
Caution—Class 1 laser radiation when open
Do not view directly with optical instruments
Appareil laser
de classe 1
Attention—Radiation laser de classe 1
Ne pas regarder directement avec des instruments optiques
Produkt der
Laser Klasse 1
Vorsicht—Laserstrahlung der Klasse 1 bei ge&ouml;ffneter Abdeckung
Direktes Ansehen mit optischen Instrumenten vermeiden
Luokan 1 Laserlaite Varoitus—Luokan 1 lasers&auml;teily&auml;, kun laite on auki
&Auml;l&auml; katso suoraan laitteeseen k&auml;ytt&auml;m&auml;ll&auml; optisia instrumenttej
Agency Certification
The following sections summarize the EMC and EMI test specifications performed
on the 41000 Series Adapters.
xxii
AH0054601-00 A
Preface
Legal Notices
EMI and EMC Requirements
FCC Part 15 compliance: Class A
FCC compliance information statement: This device complies with Part 15 of
the FCC Rules. Operation is subject to the following two conditions: (1) this device
may not cause harmful interference, and (2) this device must accept any
interference received, including interference that may cause undesired operation.
ICES-003 Compliance: Class A
This Class A digital apparatus complies with Canadian ICES-003. Cet appareil
num&eacute;riqu&eacute; de la classe A est conform&eacute; &agrave; la norme NMB-003 du Canada.
CE Mark 2004/108/EC EMC Directive Compliance:
EN55022:2010 Class A1:2007/CISPR22:2006: Class A
EN55024:2010
EN61000-3-2: Harmonic Current Emission
EN61000-3-3: Voltage Fluctuation and Flicker
Immunity Standards
EN61000-4-2: ESD
EN61000-4-3: RF Electro Magnetic Field
EN61000-4-4: Fast Transient/Burst
EN61000-4-5: Fast Surge Common/ Differential
EN61000-4-6: RF Conducted Susceptibility
EN61000-4-8: Power Frequency Magnetic Field
EN61000-4-11: Voltage Dips and Interrupt
VCCI: 2010-04 Class A
AS/NZS CISPR22: Class A
KCC: Class A
Korea RRA Class A Certified
Product Name/Model: Converged Network Adapters and
Intelligent Ethernet Adapters
Certification holder: QLogic Corporation
Manufactured date: Refer to date code listed on product
Manufacturer/Country of origin: QLogic Corporation/USA
A class equipment
(Business purpose
info/telecommunications
equipment)
As this equipment has undergone EMC registration for business purpose, the seller and/or the buyer is asked to beware
of this point and in case a wrongful sale or purchase has
been made, it is asked that a change to household use be
made.
xxiii
AH0054601-00 A
Preface
Legal Notices
Korean Language Format—Class A
Product Safety Compliance
UL, cUL product safety:
UL60950-1 (2nd Edition), 2007-03-3-27
UL CSA C22.2 60950-1-07 (2nd Edition)
Use only with listed ITE or equivalent.
Complies with 21 CFR 1040.10 and 1040.11.
2006/95/EC low voltage directive:
TUV EN60950-1:2006+A11+A1+A12
IEC60950-1 2nd Edition (2005) CB
CB Certified to IEC 60950-1 2nd Edition
xxiv
AH0054601-00 A
1
Product Overview
This chapter provides the following information for the 41000 Series Adapters:

Functional Description

Features

Adapter Management

Adapter Specifications
Functional Description
The QLogic FastLinQ 41000 Series Adapters include 10 and 25Gb Converged
Network Adapters and Intelligent Ethernet Adapters that are designed to perform
accelerated data networking for server systems. The 41000 Series Adapter
includes a 10/25Gb Ethernet MAC with full-duplex capability.
Using the operating system’s teaming feature, you can split your network into
virtual LANs (VLANs), as well as group multiple network adapters together into
teams to provide network load balancing and fault tolerance. For more information
about teaming, see your operating system documentation.
Features
The 41000 Series Adapters provide the following features. Some features may
not be available on all adapters:

NIC partitioning (NPAR)

FCoE offload

iSCSI offload

Universal RDMA: Remote direct memory access over converged Ethernet,
versions 1 and 2 (RoCE v1/v2) and Internet Wide Area Protocol (iWARP):

iSCSI Extensions for RDMA (iSER)

NVMe over Fabrics (NVMe-oF) (For details, download the Deployment
Guide: Non-volatile Memory Express over Fabrics with Universal
Remote Direct Memory Access, part number BC0054519-00)

NFS over RDMA (NFSoRDMA)
1
AH0054601-00 A
1–Product Overview
Features




1
Data center bridging (DCB):

Enhanced transmission selection (ETS; IEEE 802.1Qaz)

Priority-based flow control (PFC; IEEE 802.1Qbb)

Data center bridging eXchange protocol (DCBX; CEE version 1.01,
IEEE)
Single-chip solution:

10/25Gb MAC

SerDes interface for direct attach copper (DAC) transceiver connection

RJ-45 10GBASE-T interface for CAT cable

PCI Express 3.0 x8

Zero copy capable hardware
Performance features:

TCP, IP, UDP checksum offloads

TCP segmentation offload (TSO)

Large segment offload (LSO)

Generic segment offload (GSO)

HW based Generic Receive Offload (HW-GRO)

Large receive offload (LRO)

Receive segment coalescing (RSC)

Microsoft&reg; dynamic virtual machine queue (VMQ), and Linux
multiqueue

Adaptive interrupts

Transmit/receive side scaling (TSS/RSS)

Stateless offloads for Network Virtualization using Generic Routing
Encapsulation (NVGRE) and virtual LAN (VXLAN) L2/L3 GRE
tunneled traffic1
Manageability:

System management bus (SMB) controller

Advanced Configuration and Power Interface (ACPI) 1.1a compliant
(multiple power modes)

Network controller-sideband interface (NC-SI) support

The management applications listed in Adapter Management
This feature requires OS or Hypervisor support to use the offloads.
2
AH0054601-00 A
1–Product Overview
Adapter Management

Advanced network features:

Jumbo frames (up to 9,600 bytes). The OS and the link partner must
support jumbo frames.

Virtual LANs (VLAN)

Flow control (IEEE Std 802.3x)

Logical link control (IEEE Std 802.2)

High-speed on-chip reduced instruction set computer (RISC) processor

Integrated 96KB frame buffer memory (not applicable to all models)

1,024 classification filters (not applicable to all models)

Support for multicast addresses through 128-bit hashing hardware function

Serial flash NVRAM memory

PCI Power Management Interface (v1.1)

64-bit base address register (BAR) support

EM64T processor support

iSCSI and FCoE boot support2
Adapter Management
The following applications are available to manage 41000 Series Adapters:

QLogic Control Suite CLI

QLogic QConvergeConsole GUI

QLogic QConvergeConsole vCenter Plug-in

QConvergeConsole PowerKit

FastLinQ ESXCLI VMware Plug-in
NOTE
QCS CLI, QCC GUI, and QCC PowerKit also require installation of their
specific RPC agent.
For document part numbers and download instructions, see “Related Materials”
on page xvii.
2
Hardware support limit of SR-IOV VFs varies. The limit may be lower on some OS environments;
refer to the appropriate section for your OS.
3
AH0054601-00 A
1–Product Overview
Adapter Management
QLogic Control Suite CLI
QLogic Control Suite (QCS) CLI is a console application that you can run from a
Windows command prompt or a Linux terminal console. Use QCS CLI to manage
QLogic FastLinQ 3400/8400/41000/45000 Series Adapters and any QLogic
adapter based on 57xx/57xxx controllers on both local and remote computer
systems. For information about installing and using QCS CLI, see the User’s
Guide—QLogic Control Suite CLI.
NOTE
Although QLogic Control Suite CLI does not run on a Windows Nano Server
shell, you can install the QLNXRemote agent on a Nano Server system and
manage it remotely through QCS CLI using the addhost (Add Host) and
removehost (Remove Host) commands. For QCS CLI to connect to the
remote QLNXRemote Nano agent, you must disable or correctly configure
the firewall on the Nano system. For instructions on installing the
QLNXRemote agent on a Nano Server, refer to the QConvergeConsole
Windows Agent Installers Readme located inside the QCS CLI for Windows
download available on the QLogic Downloads and Documentation page:
www.qlogic.com
QLogic QConvergeConsole GUI
The QConvergeConsole (QCC) GUI is a Web-based management tool for
configuring and managing QLogic Fibre Channel Adapters, Converged Network
Adapters, and Intelligent Ethernet Adapters. You can use QCC GUI on Windows
and Linux platforms to manage QLogic adapters on both local and remote
computer systems. QCC GUI is dependent upon additional software (a
management agent) for the adapter. The management agent is installed
separately from QCC GUI. QCC GUI cannot communicate with the hardware until
the management agent has been installed.
For information about installing QCC GUI, see the Installation
Guide—QConvergeConsole GUI. For information about using the QCC GUI, see
the online help.
QLogic QConvergeConsole vCenter Plug-in
The QConvergeConsole vCenter&reg; Plug-in is a Web-based management tool that
is integrated into the VMware vCenter Server for configuring and managing
QLogic Fibre Channel adapters, Converged Network Adapters, and Intelligent
Ethernet Adapters in a virtual environment. You can use the vCenter Plug-in
installed in the VMware vSphere clients to manage QLogic adapters. For
information about installing and using the vCenter Plug-in, see the User’s
Guide—QConvergeConsole Plug-ins for vSphere.
4
AH0054601-00 A
1–Product Overview
Adapter Specifications
QConvergeConsole PowerKit
The QConvergeConsole PowerKit lets you manage QLogic FastLinQ
3400/8400/41000/45000 Series Adapters on the system using QLogic cmdlets in
the Windows PowerShell&reg; application. Windows PowerShell is a
Microsoft-developed scriptable language for performing task automation and
configuration management both locally and remotely. Windows PowerShell is
based on the .NET framework and includes a command-line shell and a
GUI-integrated scripting environment (ISE) that allows you to create scripts
without having to type all the commands in the command line. This feature allows
you to streamline and automate repetitive and monotonous Windows and Linux
server jobs through scripts by linking multiple instructions together. In addition to
being a powerful scripting tool, the QLogic PowerKit comes with a selection of
preconfigured cmdlets (scripts that perform a single function) to monitor and
manage your QLogic FastLinQ adapters. For information about installing and
using the QConvergeConsole PowerKit, refer to the User's Guide—PowerShell.
FastLinQ ESXCLI VMware Plug-in
The FastLinQ ESXCLI plug-in extends the capabilities of the ESX&reg; CLI to manage
QLogic FastLinQ 3400/8400/41000/45000 Series Adapters installed in VMware
ESX/ESXi hosts. For information about installing and using the ESXCLI plug-in,
see the User’s Guide—FastLinQ ESXCLI VMware Plug-in.
Adapter Specifications
The 41000 Series Adapter specifications include the adapter’s physical
characteristics and standards-compliance references.
Physical Characteristics
The 41000 Series Adapters are standard PCI Express&reg; cards and ship with either
a full-height or a low-profile bracket for use in a standard PCIe&reg; slot.
Standards Specifications
Supported standards specifications include:

PCI Express Base Specification, rev. 3.0

PCI Express Card Electromechanical Specification, rev. 3.0

PCI Bus Power Management Interface Specification, rev. 1.2

IEEE Specifications:

802.3-2012 IEEE Standard for Ethernet (flow control)

802.1q (VLAN)

802.1AX (Link Aggregation)
5
AH0054601-00 A
1–Product Overview
Adapter Specifications

802.1ad (QinQ)

802.1p (Priority Encoding)

1588-2002 PTPv1 (Precision Time Protocol)

1588-2008 PTPv2

IEEE 802.3az Energy Efficient Ethernet (EEE)

IPv4 (RFQ 791)

IPv6 (RFC 2460)
6
AH0054601-00 A
2
Hardware Installation
This chapter provides the following hardware installation information:

System Requirements

Safety Precautions

Preinstallation Checklist

Installing the Adapter
System Requirements
Before you install a QLogic 41000 Series Adapter, verify that your system meets
the hardware and operating system requirements shown in Table 2-1 and
Table 2-2. For a complete list of supported operating systems, visit the QLogic
Downloads and Documentation page:
driverdownloads.qlogic.com
Table 2-1. Host Hardware Requirements
Hardware
Requirement
Architecture
IA-32 or EMT64 that meets operating system requirements
PCIe
PCIe Gen2 x8 (2x10G NIC)
PCIe Gen3 x8 (2x25G NIC)
Full dual-port 25Gb bandwidth is supported on PCIe Gen3 x8 or
faster slots.
Memory
8GB RAM (minimum)
Cables and Optical
Modules
The 41000 Series Adapters have been tested for interoperability
with a variety of 10G and 25G cables and optical modules. See
“Tested Cables and Optical Modules” on page 218.
7
AH0054601-00 A
2–Hardware Installation
Safety Precautions
Table 2-2. Minimum Host Operating System Requirements
Operating
System
Requirement
Windows Server
2012, 2012 R2, 2016 (including Nano)
Linux
RHEL 6.6, 7.0, and higher
SLES 11 SP4, 12, and higher
CentOS&reg; 7.0 and higher
Ubuntu&reg; 14.04 LTS, 16.04 LTS
VMware
ESXi 6.0 U2 and later for 25G adapters
ESXi 5.5.x, 6.0.x and later for 40G adapters
ESXi 6.5 and later for 100G adapters
NOTE
Table 2-2 denotes minimum host OS requirements. For a complete list of
supported operating systems, visit the QLogic Downloads and
Documentation page:
driverdownloads.qlogic.com
Safety Precautions
!
WARNING
The adapter is being installed in a system that operates with voltages that
can be lethal. Before you open the case of your system, observe the
following precautions to protect yourself and to prevent damage to the
system components.
 Remove any metallic objects or jewelry from your hands and wrists.
 Make sure to use only insulated or nonconducting tools.
 Verify that the system is powered OFF and is unplugged before you
touch internal components.
 Install or remove adapters in a static-free environment. The use of a
properly grounded wrist strap or other personal antistatic devices and an
antistatic mat is strongly recommended.
8
AH0054601-00 A
2–Hardware Installation
Preinstallation Checklist
Preinstallation Checklist
Before installing the adapter, complete the following:
1.
Verify that your system meets the hardware and software requirements
listed under “System Requirements” on page 7.
2.
Verify that your system is using the latest BIOS.
NOTE
If you acquired the adapter software from the QLogic Web site
(driverdownloads.qlogic.com), verify the path to the adapter driver
files.
3.
If your system is active, shut it down.
4.
When system shutdown is complete, turn off the power and unplug the
power cord.
5.
Remove the adapter from its shipping package and place it on an anti-static
surface.
6.
Check the adapter for visible signs of damage, particularly on the edge
connector. Never attempt to install a damaged adapter.
Installing the Adapter
The following instructions apply to installing the QLogic 41000 Series Adapters in
most systems. For details about performing these tasks, refer to the manuals that
were supplied with your system.
To install the adapter:
1.
Review “Safety Precautions” on page 8 and “Preinstallation Checklist” on
page 9. Before you install the adapter, ensure that the system power is OFF,
the power cord is unplugged from the power outlet, and that you are
following proper electrical grounding procedures.
2.
Open the system case, and select the slot that matches the adapter size,
which can be PCIe Gen 2 x8 or PCIe Gen 3 x8. A lesser width adapter can
be seated into a greater width slot (x8 in an x16), but a greater width adapter
cannot be seated into a lesser width slot (x8 in an x4). If you do not know
how to identify a PCIe slot, refer to your system documentation.
3.
Remove the blank cover-plate from the slot that you selected.
4.
Align the adapter connector edge with the PCI Express connector slot in the
system.
9
AH0054601-00 A
2–Hardware Installation
Installing the Adapter
5.
Applying even pressure at both corners of the card, push the adapter card
into the slot until it is firmly seated. When the adapter is properly seated, the
adapter port connectors are aligned with the slot opening, and the adapter
faceplate is flush against the system chassis.
CAUTION
Do not use excessive force when seating the card, as this may
damage the system or the adapter. If you have difficulty seating the
adapter, remove it, realign it, and try again.
6.
Secure the adapter with the adapter clip or screw as recommended by the
system vendor.
7.
Close the system case and disconnect any personal anti-static devices.
8.
Connect to the Ethernet network using your preferred and supported cabling
option (see “Tested Cables and Optical Modules” on page 218).
10
AH0054601-00 A
3
Driver Installation
This chapter provides the following information about driver installation:

Installing Linux Driver Software

Installing Windows Driver Software

Installing VMware Driver Software
Installing Linux Driver Software
This section describes how to install Linux drivers with or without RDMA and
iWARP. It also describes the Linux driver optional parameters, default values,
messages, and statistics.

Installing the Linux Drivers Without RDMA

Installing the Linux Drivers with RDMA

Linux Driver Optional Parameters

Linux Driver Operation Defaults

Linux Driver Messages

Statistics
The 41000 Series Adapter Linux drivers and supporting documentation are
available on the QLogic Downloads and Documentation page:
driverdownloads.qlogic.com
Table 3-1 describes the 41000 Series Adapter Linux drivers.
Table 3-1. QLogic 41000 Series Adapters Linux Drivers
Linux
Driver
qed
Description
The qed core driver module directly controls the firmware, handles interrupts, and provides the low-level API for the protocol specific driver set. The qed interfaces with the
qede, qedr, qedi, and qedf drivers. The Linux core module manages all PCI device
resources (registers, host interface queues, and so on). The qed core module requires
Linux kernel version 2.6.32 or later. Testing was concentrated on the x86_64 architecture.
11
AH0054601-00 A
3–Driver Installation
Installing Linux Driver Software
Table 3-1. QLogic 41000 Series Adapters Linux Drivers (Continued)
Linux
Driver
Description
qede
Linux Ethernet driver for the 41000 Series Adapter. This driver directly controls the hardware and is responsible for sending and receiving Ethernet packets on behalf of the Linux
host networking stack. This driver also receives and processes device interrupts on
behalf of itself (for L2 networking). The qede driver requires Linux kernel version 2.6.32 or
later. Testing was concentrated on the x86_64 architecture.
qedr
Linux RDMA over converged Ethernet (RoCE) driver. This driver works in the OpenFabrics Enterprise Distribution (OFED™) environment in conjunction with the qed core module and the qede Ethernet driver. RDMA user space applications also require that the
libqedr user library is installed on the server.
qedi
Linux iSCSI-Offload driver for the 41000 Series Adapters. This driver works with the Open
iSCSI library.
qedf
Linux FCoE-Offload driver for the 41000 Series Adapters. This driver works with Open
FCoE library.
The Linux drivers can be installed using a source Red Hat&reg; Package Manager
(RPM) package or a kmod RPM package. The RHEL RPM packages are as
follows:

qlgc-fastlinq-&lt;version&gt;.&lt;OS&gt;.src.rpm

qlgc-fastlinq-kmp-default-&lt;version&gt;.&lt;arch&gt;.rpm
The SLES source and kmp RPM packages are as follows:

qlgc-fastlinq-&lt;version&gt;.&lt;OS&gt;.src.rpm

qlgc-fastlinq-kmp-default-&lt;version&gt;.&lt;OS&gt;.&lt;arch&gt;.rpm
The Ubuntu Debian package is as follows:

qlgc-fastlinq-dkms-&lt;version&gt;_all.deb
The following kernel module (kmod) RPM installs Linux drivers on SLES hosts
running the Xen Hypervisor:

qlgc-fastlinq-kmp-xen-&lt;version&gt;.&lt;OS&gt;.&lt;arch&gt;.rpm
The following source RPM installs the RDMA library code on RHEL and SLES
hosts:

qlgc-libqedr-&lt;version&gt;.&lt;OS&gt;.&lt;arch&gt;.src.rpm
12
AH0054601-00 A
3–Driver Installation
Installing Linux Driver Software
The following source code TAR BZip2 (BZ2) compressed file installs Linux drivers
on RHEL and SLES hosts:

fastlinq-&lt;version&gt;.tar.bz2
NOTE
For network installations through NFS, FTP, or HTTP (using a network boot
disk), a driver disk that contains the qede driver may be needed. Linux boot
drivers can be compiled by modifying the makefile and the make
environment.
Installing the Linux Drivers Without RDMA
NOTE
When using the following procedures in a CentOS environment, follow the
instructions for RHEL. When performing these tasks in an Ubuntu
environment, see the Install.txt document in the Ubuntu package that
is available for your adapter on the QLogic Downloads and Documentation
page:
driverdownloads.qlogic.com
To install the Linux drivers without RDMA:
1.
Download the 41000 Series Adapter Linux drivers from QLogic:
driverdownloads.qlogic.com
2.
Remove the existing Linux drivers, as described in “Removing the Linux
Drivers” on page 13.
3.
Install the new Linux drivers using one of the following methods:

Installing Linux Drivers Using the src RPM Package

Installing Linux Drivers Using the kmp/kmod RPM Package

Installing Linux Drivers Using the TAR File
Removing the Linux Drivers
There are two procedures for removing Linux drivers: one for a non-RDMA
environment and another for an RDMA environment. Choose the procedure that
matches your environment.
13
AH0054601-00 A
3–Driver Installation
Installing Linux Driver Software
To remove Linux drivers in a non-RDMA environment, unload and remove
the drivers:
Follow the procedure that relates to the original installation method and the OS.

If the Linux drivers were installed using an RPM package, issue the following
commands:
rmmod qede
rmmod qed
depmod -a
rpm -e qlgc-fastlinq-kmp-default-&lt;version&gt;.&lt;arch&gt;

If the Linux drivers were installed using a TAR file, issue the following
commands:
rmmod qede
rmmod qed
depmod -a

For RHEL and CentOS:
cd /lib/modules/&lt;version&gt;/extra/qlgc-fastlinq
rm -rf qed.ko qede.ko qedr.ko

For SLES:
cd /lib/modules/&lt;version&gt;/updates/qlgc-fastlinq
rm -rf qed.ko qede.ko qedr.ko

For Ubuntu, issue the following command:
dpkg -r fastlinq-dkms
To remove Linux drivers in an RDMA environment:
1.
Unload and remove the Linux drivers.
modprobe -r qedr
modprobe -r qede
modprobe -r qed
depmod -a
2.
Remove the driver module files:

If the drivers were installed using an RPM package, issue the following
command:
rpm -e qlgc-fastlinq-kmp-default-&lt;version&gt;.&lt;arch&gt;
14
AH0054601-00 A
3–Driver Installation
Installing Linux Driver Software

If the drivers were installed using a TAR file, issue the following
commands for your operating system:
For RHEL and CentOS:
cd /lib/modules/&lt;version&gt;/extra/qlgc-fastlinq
rm -rf qed.ko qede.ko qedr.ko
For SLES:
cd /lib/modules/&lt;version&gt;/updates/qlgc-fastlinq
rm -rf qed.ko qede.ko qedr.ko
Installing Linux Drivers Using the src RPM Package
To install Linux drivers using the src RPM package:
1.
Issue the following at a command prompt:
rpm -ivh qlgc-fastlinq-&lt;version&gt;.src.rpm
2.
Change the directory to the RPM path and build the binary RPM for the
kernel.
For RHEL and CentOS:
cd /root/rpmbuild
rpmbuild -bb SPECS/fastlinq-&lt;version&gt;.spec
For SLES:
cd /usr/src/packages
rpmbuild -bb SPECS/fastlinq-&lt;version&gt;.spec
3.
Install the newly compiled RPM:
rpm -ivh RPMS/&lt;arch&gt;/qlgc-fastlinq-&lt;version&gt;.&lt;arch&gt;.rpm
NOTE
The --force option may be needed on some Linux distributions if
conflicts are reported.
The drivers will be installed in the following paths.
For SLES:
/lib/modules/&lt;version&gt;/updates/qlgc-fastlinq
For RHEL and CentOS:
/lib/modules/&lt;version&gt;/extra/qlgc-fastlinq
15
AH0054601-00 A
3–Driver Installation
Installing Linux Driver Software
4.
Turn on all ethX interfaces as follows:
ifconfig &lt;ethX&gt; up
5.
For SLES, use YaST to configure the Ethernet interfaces to automatically
start at boot by setting a static IP address or enabling DHCP on the
interface.
Installing Linux Drivers Using the kmp/kmod RPM Package
To install kmod RPM package:
1.
Issue the following command at a command prompt:
rpm -ivh qlgc-fastlinq-&lt;version&gt;.&lt;arch&gt;.rpm
2.
Reload the driver:
modprobe -r qede
modprobe qede
Installing Ubuntu Linux Drivers
To install Ubuntu Linux drivers, issue the following command:
dpkg -i fastlinq-dkms_&lt;version&gt;_all.deb
Installing Linux Drivers Using the TAR File
To install Linux drivers using the TAR file:
1.
Create a directory and extract the TAR files to the directory:
tar xjvf fastlinq-&lt;version&gt;.tar.bz2
2.
Change to the recently created directory, and then install the drivers:
cd fastlinq-&lt;version&gt;
make clean; make install
The qed and qede drivers will be installed in the following paths.
For SLES:
/lib/modules/&lt;version&gt;/updates/qlgc-fastlinq
For RHEL and CentOS:
/lib/modules/&lt;version&gt;/extra/qlgc-fastlinq
3.
Test the drivers by loading them (unload the existing drivers first, if
necessary):
rmmod qede
rmmod qed
16
AH0054601-00 A
3–Driver Installation
Installing Linux Driver Software
modprobe qed
modprobe qede
Installing the Linux Drivers with RDMA
For information on iWARP, see Chapter 7 iWARP Configuration.
NOTE
When using the following procedures in a CentOS environment, follow the
instructions for RHEL. When performing these tasks in an Ubuntu
environment, see the Install.txt document in the Ubuntu package that
is available for your adapter on the QLogic Downloads and Documentation
page:
driverdownloads.qlogic.com
To install Linux drivers in an inbox OFED environment:
1.
Download the 41000 Series Adapter Linux drivers from QLogic:
driverdownloads.qlogic.com
2.
Configure RoCE on the adapter, as described in “Configuring RoCE on the
Adapter for Linux” on page 62.
3.
Remove existing Linux drivers, as described in “Removing the Linux Drivers”
on page 13.
4.
Install the new Linux drivers using one of the following methods:
5.

Installing Linux Drivers Using the kmp/kmod RPM Package

Installing Linux Drivers Using the TAR File
Install libqedr libraries to work with RDMA user space applications. The
libqedr RPM is available only for inbox OFED. You must select which RDMA
(RoCEv1/v2 or iWARP) is used in UEFI until concurrent RoCE+iWARP
capability is supported in the firmware). None is enabled by default. Issue
the following command:
rpm –ivh qlgc-libqedr-&lt;version&gt;.&lt;arch&gt;.rpm
6.
To build and install the libqedr user space library, issue the following
command:
'make libqedr_install'
7.
Test the drivers by loading them as follows:
modprobe qed
modprobe qede
17
AH0054601-00 A
3–Driver Installation
Installing Linux Driver Software
modprobe qedr
Linux Driver Optional Parameters
Table 3-2 describes the optional parameters for the qede driver.
Table 3-2. qede Driver Optional Parameters
Parameter
Description
debug
Controls driver verbosity level similar to ethtool -s &lt;dev&gt; msglvl.
int_mode
Controls interrupt mode other than MSI-X.
gro_enable
Enables or disables the hardware generic receive offload (GRO) feature.
This feature is similar to the kernel's software GRO, but is only performed
by the device hardware.
err_flags_override
A bitmap for disabling or forcing the actions taken in case of a hardware
error:
 bit #31 – An enable bit for this bitmask
 bit #0 – Prevent hardware attentions from being reasserted
 bit #1 – Collect debug data
 bit #2 – Trigger a recovery process
 bit #3 – Call WARN to get a call trace of the flow that led to the error
Linux Driver Operation Defaults
Table 3-3 lists the qed and qede Linux driver operation defaults.
Table 3-3. Linux Driver Operation Defaults
Operation
qed Driver Default
qede Driver Default
Speed
Auto-negotiation with speed
advertised
Auto-negotiation with speed
advertised
MSI/MSI-X
Enabled
Enabled
Flow Control
—
Auto-negotiation with RX and
TX advertised
MTU
—
1500 (range is 46–9600)
Rx Ring Size
—
1000
Tx Ring Size
—
4078 (range is 128–8191)
Coalesce Rx Microseconds
—
24 (range is 0–255)
18
AH0054601-00 A
3–Driver Installation
Installing Windows Driver Software
Table 3-3. Linux Driver Operation Defaults (Continued)
Operation
qed Driver Default
qede Driver Default
Coalesce Tx Microseconds
—
48
TSO
—
Enabled
Linux Driver Messages
To set the Linux driver message detail level, issue one of the following commands:

ethtool -s &lt;interface&gt; msglvl &lt;value&gt;

modprobe qede debug=&lt;value&gt;
Where &lt;value&gt; represents bits 0–15, which are standard Linux networking
values, and bits 16 and greater are driver-specific.
Statistics
To view detailed statistics and configuration information, use the ethtool utility. See
the ethtool man page for more information.
Installing Windows Driver Software
For information on iWARP, see Chapter 7 iWARP Configuration.

Installing the Windows Drivers

Removing the Windows Drivers

Managing Adapter Properties

Setting Power Management Options

Installing Drivers for Windows Nano Server

Creating a Nano ISO Image, Injecting Drivers, and Updating the
Multiboot/Flash Image on a Nano Server
Installing the Windows Drivers
NOTE
 Note that there is no other separate procedure to install RoCE-supported
drivers in Windows.
 For information on building the Windows Nano virtual hard disk, go to:
https://technet.microsoft.com/en-us/windows-server-docs/compute/
nano-server/getting-started-with-nano-server
19
AH0054601-00 A
3–Driver Installation
Installing Windows Driver Software
To install the Windows drivers:
1.
Download the Windows device drivers for the 41000 Series Adapter from
the QLogic Downloads page:
driverdownloads.qlogic.com
2.
Launch the downloaded QLogic FastLinQ&reg; Driver Install package
(setup.exe).
3.
Complete the InstallShield Wizard:
4.
a.
In the Welcome dialog box, click Next.
b.
Follow the wizard instructions, accepting the terms of the license
agreement.
c.
Click Install to start the installation.
d.
When the installation is complete, click Finish.
Verify that the Windows drivers have been installed:
a.
Click Start and then click Control Panel.
b.
In the Control Panel, click Programs, and then click Programs and
Features.
c.
In the installed programs list, locate QLogic FastLinQ Driver Installer.
Removing the Windows Drivers
To remove the Windows drivers:
1.
In the Control Panel, click Programs, and then click Programs and
Features.
2.
In the list of programs, select QLogic FastLinQ Driver Installer, and then
click Uninstall.
3.
Follow the instructions to remove the drivers.
Managing Adapter Properties
To view or change the 41000 Series Adapter properties:
1.
In the Control Panel, click Device Manager.
2.
On the properties of the selected adapter, click the Advanced tab.
20
AH0054601-00 A
3–Driver Installation
Installing Windows Driver Software
3.
On the Advanced page (Figure 3-1), select an item under Property and then
change the Value for that item as needed.
Figure 3-1. Setting Advanced Adapter Properties
21
AH0054601-00 A
3–Driver Installation
Installing Windows Driver Software
Setting Power Management Options
You can set power management options to allow the operating system to turn off
the controller to save power or to allow the controller to wake up the computer. If
the device is busy (servicing a call, for example), the operating system will not
shut down the device. The operating system attempts to shut down every possible
device only when the computer attempts to go into hibernation. To have the
controller remain on at all times, do not select the Allow the computer to turn off
the device to save power check box (Figure 3-2).
Figure 3-2. Power Management Options
NOTE
 The Power Management page is available only for servers that support
power management.
 Do not select Allow the computer to turn off the device to save
power for any adapter that is a member of a team.
Installing Drivers for Windows Nano Server
NOTE
To obtain the individual driver files available prior to starting Step 1 of this
procedure, you must extract the drivers from the Windows Driver Installer.
The installer is available on the QLogic Downloads page:
driverdownloads.qlogic.com
To install the drivers for Windows Nano Server:
1.
Add and install the driver package by issuing the following command:
pnputil.exe -i -a a:\usbcam\USBCAM.INF
22
AH0054601-00 A
3–Driver Installation
Installing Windows Driver Software
Table 3-4 lists some of the Windows drivers.
Table 3-4. Windows Drivers
Windows Driver
QeVBD
2.
Description
Core driver
QeND
Ethernet networking driver
QeOIS
iSCSI-Offload driver
QeFCoE
FCoE-Offload driver
QxDiag
Diagnostics driver
Delete the oem0.inf package by issuing the following command:

Option 1: pnputil.exe -d oem0.inf

Option 2: pnputil.exe -f -d oem0.inf
NOTE
The oem0.inf may not always be the device file. Locate the
corresponding oemx.inf file first.
Creating a Nano ISO Image, Injecting Drivers, and Updating
the Multiboot/Flash Image on a Nano Server
To create a Nano ISO image, inject drivers, and update the MBI/Flash Image
on Nano:
1.
Create a Nano Server image.
NOTE
For more information on how to create a Nano Server image, refer to
the Microsoft documentation at:
https://technet.microsoft.com/en-us/windows-server-docs/compute/
nano-server/getting-started-with-nano-server.
2.
To inject the QLogic drivers into the Nano image:
a.
Download the Windows Driver Installer from the QLogic Downloads
page and extract the relevant drivers.
driverdownloads.qlogic.com
23
AH0054601-00 A
3–Driver Installation
Installing VMware Driver Software
b.
Place the extracted individual files in a temporary folder.
NOTE
You will use the QLogic drivers files during Nano Server image
creation. For more information on a specific command, refer to the
Injecting Drivers section in the Microsoft link noted in Step 1.
3.
To use Microsoft's pnputil tool to upgrade or install the QLogic drivers:
a.
Copy the QLogic driver files to the Nano Server.
b.
After creating a Nano Server image, mount it, and copy over the files.
c.
Navigate to the QLogic driver directory, and issue the following
command:
pnputil /add-driver c:\drivers\*.inf /install
NOTE
The QLogic drivers were copied to the c:\drivers folder in the
preceding command.
4.
To run the Firmware Upgrade Utility in a Nano Server environment:
a.
Copy the utility into the Nano Server.
b.
After creating Nano Server image, mount it, and copy the utility.
NOTE
For more information on connecting remotely to a Nano Server,
refer to the Using Windows PowerShell section in the Microsoft
link noted in Step 1.
c.
After connected remotely, navigate to the utility directory, and then run
the utility.
Installing VMware Driver Software
This section describes the qedentv VMware ESXi driver for the 41000 Series
Adapters:

VMware Drivers and Driver Packages

Installing the VMware Driver

VMware Driver Optional Parameters
24
AH0054601-00 A
3–Driver Installation
Installing VMware Driver Software

VMware Driver Parameter Defaults

Removing the VMware Driver

FCoE Support

iSCSI Support
VMware Drivers and Driver Packages
Table 3-5 lists the VMware ESXi drivers for the protocols.
Table 3-5. VMware Drivers
VMware Drivers
a
Description
qedentv
Native networking driver
qedrntv
Native RDMA-Offload (RoCEv1 and RoCEv2) driver a
qedf
Native FCoE-Offload driver
qedil
Legacy iSCSI-Offload driver
The certified RoCE driver is not included in this release. The uncertified driver may be
available as an early preview.
The ESXi drivers are included as individual driver packages and are not bundled
together. Table 3-6 lists the ESXi versions and applicable driver versions.
Table 3-6. ESXi Driver Packages by Release
ESXi Release
ESXi 6.5
ESXi 6.0u3
Protocol
Driver Name
NIC
qedentv
FCoE
qedf
iSCSI
qedil
RoCE
qedrntv
NIC
qedentv
FCoE
qedf
iSCSI
qedil
Install individual drivers using either:

Standard ESXi package installation commands (see Installing the VMware
Driver)
25
AH0054601-00 A
3–Driver Installation
Installing VMware Driver Software

Procedures in the individual driver Read Me files

Procedures in the following VMware KB article:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&amp;
cmd=displayKC&amp;externalId=2137853
QLogic recommends that you install the NIC driver first, followed by the storage
drivers.
Installing the VMware Driver
You can use the driver ZIP file to install a new driver or update an existing driver.
Be sure to install the entire driver set from the same driver ZIP file. Mixing drivers
from different ZIP files will cause problems.
To install the VMware driver:
1.
Download the VMware driver for the 41000 Series Adapter from the VMware
support page:
www.vmware.com/support.html
2.
Power up the ESX host, and then log into an account with administrator
authority.
3.
Unzip the driver ZIP file, and then extract the .vib file.
4.
Use the Linux scp utility to copy a .vib file from a local system into the
/tmp directory on an ESX server with IP address 10.10.10.10. For example,
issue the following command:
#scp qedentv-1.0.3.11-1OEM.550.0.0.1331820.x86_64.vib [email protected]:/tmp
You can place the file anywhere that is accessible to the ESX console shell.
NOTE
If you do not have a Linux machine, you can uses the vSphere
datastore file browser to upload the files to the server.
5.
Place the host in maintenance mode by issuing the following command:
#esxcli --maintenance-mode
26
AH0054601-00 A
3–Driver Installation
Installing VMware Driver Software
6.
Select one of the following installation options:

Option 1: Install the .vib directly on an ESX server using either the
CLI or the VMware Update Manager (VUM):

To install the .vib file using the CLI, issue the following
command. Be sure to specify the full .vib file path.
# esxcli software vib install -v
/tmp/qedentv-1.0.3.11-1OEM.550.0.0.1331820.x86_64.vib

To install the .vib file using the VUM, see the knowledge base
article here:
Updating an ESXi/ESX host using VMware vCenter Update
Manager 4.x and 5.x (1019545)

Option 2: Install all of the individual VIBs at one time by issuing the
following command:
# esxcli software vib install –d
/tmp/qedentv-bundle-2.0.3.zip
To upgrade an existing driver:
Follow the steps for a new installation, except replace the command in the
preceding Option 1 with the following:
#esxcli software vib update -v
/tmp/qedentv-1.0.3.11-1OEM.550.0.0.1331820.x86_64.vib
VMware Driver Optional Parameters
Table 3-7 describes the optional parameters that can be supplied as command
line arguments to the esxcfg-module command.
Table 3-7. VMware Driver Optional Parameters
Parameter
hw_vlan
Description
Globally enables (1) or disables (0) hardware VLAN insertion and removal.
Disable this parameter when the upper layer needs to send or receive fully
formed packets. hw_vlan=1 is the default.
27
AH0054601-00 A
3–Driver Installation
Installing VMware Driver Software
Table 3-7. VMware Driver Optional Parameters (Continued)
Parameter
num_queues
Description
Specifies the number of TX/RX queue pairs. num_queues can be 1–11 or
one of the following:
 –1 allows the driver to determine the optimal number of queue pairs
(default).
 0 uses the default queue.
You can specify multiple values delimited by commas for multiport or multifunction configurations.
multi_rx_filters
Specifies the number of RX filters per RX queue, excluding the default
queue. multi_rx_filters can be 1–4 or one of the following values:
 –1 uses the default number of RX filters per queue.
 0 disables RX filters.
disable_tpa
Enables (0) or disables (1) the TPA (LRO) feature. disable_tpa=0 is the
default.
max_vfs
Specifies the number of virtual functions (VFs) per physical function (PF).
max_vfs can be 0 (disabled) or 64 VFs on a single port (enabled). The
64 VF maximum support for ESXi is an OS resource allocation constraint.
RSS
Specifies the number of receive side scaling queues used by the host or virtual extensible LAN (VXLAN) tunneled traffic for a PF. RSS can be 2, 3, 4,
or one of the following values:
 –1 uses the default number of queues.
 0 or 1 disables RSS queues.
You can specify multiple values delimited by commas for multiport or multifunction configurations.
debug
Specifies the level of data that the driver records in the vmkernel log file.
debug can have the following values, shown in increasing amounts of data:
 0x80000000 indicates Notice level.
 0x40000000 indicates Information level (includes the Notice level).
 0x3FFFFFFF indicates Verbose level for all driver submodules (includes
the Information and Notice levels).
auto_fw_reset
Enables (1) or disables (0) the driver automatic firmware recovery capability.
When this parameter is enabled, the driver attempts to recover from events
such as transmit timeouts, firmware asserts, and adapter parity errors. The
default is auto_fw_reset=1.
vxlan_filter_en
Enables (1) or disables (0) the VXLAN filtering based on the outer MAC, the
inner MAC, and the VXLAN network (VNI), directly matching traffic to a specific queue. The default is vxlan_filter_en=1. You can specify multiple
values delimited by commas for multiport or multifunction configurations.
28
AH0054601-00 A
3–Driver Installation
Installing VMware Driver Software
Table 3-7. VMware Driver Optional Parameters (Continued)
Parameter
Description
enable_vxlan_offld
Enables (1) or disables (0) the VXLAN tunneled traffic checksum offload and
TCP segmentation offload (TSO) capability. The default is
enable_vxlan_offld=1. You can specify multiple values delimited by
commas for multiport or multifunction configurations.
VMware Driver Parameter Defaults
Table 3-8 lists the VMware driver parameter default values.
Table 3-8. VMware Driver Parameter Defaults
Parameter
Default
Speed
Autonegotiation with all speeds advertised. The speed
parameter must be the same on all ports. If autonegotiation is enabled on the device, all of the device
ports will use autonegotiation.
Flow Control
Autonegotiation with RX and TX advertised
MTU
1,500 (range 46–9,600)
Rx Ring Size
8,192 (range 128–8,192)
Tx Ring Size
8,192 (range 128–8,192)
MSI-X
Enabled
Transmit Send Offload
(TSO)
Enabled
Large Receive Offload
(LRO)
Enabled
RSS
Numbered value
HW VLAN
Enabled
Number of Queues
Numbered value
Wake on LAN (WoL)
Disabled
29
AH0054601-00 A
3–Driver Installation
iSCSI Support
Removing the VMware Driver
To remove the .vib file (qedentv), issue the following command:
# esxcli software vib remove --vibname qedentv
To remove the driver, issue the following command:
# vmkload_mod -u qedentv
FCoE Support
Table 3-9 describes the driver included in the VMware software package to
support QLogic FCoE converged network interface controllers (C-NICs). The
FCoE and DCB feature set is supported on VMware ESXi 5.0 and later.
Table 3-9. QLogic 41000 Series Adapter VMware FCoE Driver
Driver
qedf
Description
The QLogic VMware FCoE driver is a kernel-mode driver that provides a translation layer between the VMware SCSI stack and the
QLogic FCoE firmware and hardware.
iSCSI Support
Table 3-10 describes the iSCSI driver.
Table 3-10. QLogic 41000 Series Adapter iSCSI Driver
Driver
qedil
Description
The qedil driver is the QLogic VMware iSCSI HBA driver. Similar to
qedf, qedil is a kernel mode driver that provides a translation layer
between the VMware SCSI stack and the QLogic iSCSI firmware
and hardware. qedil leverages the services provided by the VMware
iscsid infrastructure for session management and IP services.
30
AH0054601-00 A
4
Firmware Upgrade Utility
QLogic provides scripts to automate the adapter firmware and boot code upgrade
process for Windows and Linux systems. Each script identifies all 41000 Series
Adapters and upgrades all firmware components. To upgrade adapter firmware on
VMware systems, see the User’s Guide—FastLinQ ESXCLI VMware Plug-in or
the User’s Guide—QConvergeConsole Plug-ins for vSphere.
CAUTION
Upgrading the adapter firmware may require several minutes. To avoid
damaging your adapters, do not exit, reboot, or power cycle the system
during the upgrade.
Each script executes its respective utility: lnxfwnx2 for Linux and winfwnx2 for
Windows. Both utilities are console applications that can be called from other
processes or used as a command line tool with parameters. For more information
about these utilities, their command syntax, and parameters, see their respective
ReadMe files.
This chapter includes the following firmware upgrade information:

Image Verification

Upgrading Adapter Firmware on Linux

Upgrading Adapter Firmware on Windows

Upgrading Adapter Firmware on Windows Nano
Image Verification
During firmware updates, the image is verified by comparing a digital signature to
a computed signature on the image hash. During manufacturing, the hash value is
computed on the management firmware (MFW) and sent for a signature at the
Signature Authority (SA). The SA signs it using its private key, and then appends
the signature to the MFW.
31
AH0054601-00 A
4–Firmware Upgrade Utility
Upgrading Adapter Firmware on Linux
When the MFW requires verification, the hash is calculated on the MFW. The
hash is compared with the signature after running the opposite algorithm using the
equivalent public-key.


Signature = Digital-Signature-Algorithm (Private-Key, HASH)
Verified if (HASH == Digital-Signature-Algorithm (Public-Key, Signature)
Upgrading Adapter Firmware on Linux
To upgrade adapter firmware on a Linux system:
1.
Install the qed and qede drivers, and confirm that they are running.
2.
Download the Firmware Upgrade Utility for Linux from QLogic:
driverdownloads.qlogic.com
3.
Unzip the Firmware Upgrade Utility on the system where the adapter is
installed. All files should reside in the same folder.
4.
Confirm that the adapter port link status is up (ifconfig ethx up).
5.
Run the Linux upgrade utility script by issuing the following command:
./LnxQlgcUpg.sh
The flash update script should display information to the console indicating
that it is proceeding. If the utility is unable to identify the installed adapter,
ensure that the system is running the latest available driver versions.
Upgrading Adapter Firmware on Windows
To upgrade adapter firmware on a Windows system:
1.
Install the Window eVBD or qeVBD (as applicable) driver.
2.
Download the Firmware Upgrade Utility for Windows from QLogic:
driverdownloads.qlogic.com
3.
Unzip the Firmware Upgrade Utility on the system where the adapter is
installed. All files should reside in the same folder.
4.
Run the Windows upgrade utility script by issuing the following command:
C:\Windows_FWupg\AMD64\WinQlgcUpg.bat
32
AH0054601-00 A
4–Firmware Upgrade Utility
Upgrading Adapter Firmware on Windows Nano
Upgrading Adapter Firmware on Windows Nano
To upgrade adapter firmware on a Windows Nano system:
1.
Install the Windows eVBD or qeVBD (as applicable) driver.
2.
Download the Firmware Upgrade Utility for Windows from QLogic:
driverdownloads.qlogic.com
3.
Unzip the Firmware Upgrade Utility on the system where the adapter is
installed. All files should reside in the same folder.
4.
Copy the Firmware Upgrade Utility files onto the Windows Nano Server
system.
5.
Remote into Windows Nano Server by PowerShell.
6.
Run the Windows upgrade utility script by issuing the following command:
C:\Windows_FWupg\AMD64\WinQlqcUpg.bat
33
AH0054601-00 A
5
Adapter Preboot
Configuration
During the host boot process, you have the opportunity to pause and perform
adapter management tasks using the Human Infrastructure Interface (HII)
application. These tasks include the following:

Displaying Firmware Image Properties

Configuring Device-level Parameters

Configuring Port-level Parameters

Configuring FCoE Boot

Configuring iSCSI Boot

Configuring Partitions
NOTE
The HII screen shots in this chapter are representative and may not match
the screens that you see on your system.
Getting Started
To start the HII application:
1.
Open the System Setup window for your platform. For information about
launching the System Setup, consult the user guide for your system.
2.
In the System Setup window (Figure 5-1), select Device Settings, and then
press ENTER.
34
AH0054601-00 A
5–Adapter Preboot Configuration
Getting Started
Figure 5-1. System Setup
3.
In the Device Settings window (Figure 5-2), select the 41000 Series Adapter
port that you want to configure, and then press ENTER.
Figure 5-2. System Setup: Device Settings
4.
The Main Configuration Page presents the adapter management options:

If you are not using NPAR, set Partitioning Mode to Default, as
shown in Figure 5-3.
35
AH0054601-00 A
5–Adapter Preboot Configuration
Getting Started
Figure 5-3. Main Configuration Page, Setting Default Partitioning Mode

Setting Partitioning Mode to NPAR adds the Partitions
Configuration option to the Main Configuration Page, as shown in
Figure 5-4.
Figure 5-4. Main Configuration Page, Setting NPAR Partitioning Mode
36
AH0054601-00 A
5–Adapter Preboot Configuration
Getting Started
In Figure 5-3 and Figure 5-4, the Main Configuration Page shows the following:






Firmware Image Properties
Device Level Configuration
Port Level Configuration
iSCSI Configuration (if iSCSI remote boot is available)
FCoE Configuration (if FCoE boot from SAN is available)
Partitions Configuration (if NPAR is selected as the Partitioning Mode)
In addition, the Main Configuration Page presents the adapter properties listed in
Table 5-1.
Table 5-1. Adapter Properties
Adapter Property
Description
Partitioning Mode
Default or NPAR
Device Name
Factory-assigned device name
Chip Type
ASIC version
PCI Device ID
Unique vendor-specific PCI device ID
PCI Address
PCI device address in bus-device function format
Link Status
External link status
Permanent MAC Address
Manufacturer-assigned permanent device MAC address
Virtual MAC Address
User-defined device MAC address
iSCSI MAC Address
Manufacturer-assigned permanent device iSCSI Offload
MAC address
iSCSI Virtual MAC
Address
User-defined device iSCSI Offload MAC address
FCoE MAC Address
Manufacturer-assigned permanent device FCoE Offload
MAC address
FCoE Virtual MAC
Address
User-defined device FCoE Offload MAC address
FCoE WWPN
Manufacturer-assigned permanent device FCoE Offload
WWPN (world wide port name)
FCoE Virtual WWPN
User-defined device FCoE Offload WWPN
FCoE WWNN
Manufacturer-assigned permanent device FCoE Offload
WWNN (world wide node name)
FCoE Virtual WWNN
User-defined device FCoE Offload WWNN
37
AH0054601-00 A
5–Adapter Preboot Configuration
Displaying Firmware Image Properties
Displaying Firmware Image Properties
To view the properties for the firmware image, select Firmware Image Properties
on the Main Configuration Page, and then press ENTER. The Firmware
Information page (Figure 5-5) specifies the following view-only data:

Family Firmware Version is the multiboot image version, which comprises
several firmware component images.

MFW Version is the management firmware version.

UEFI Driver Version is the unified extensible firmware interface (UEFI) or
extensible firmware interface (EFI) driver version.
Figure 5-5. Firmware Information Window
Configuring Device-level Parameters
NOTE
The iSCSI physical functions (PFs) are enumerated when the iSCSI Offload
feature is enabled. The FCoE physical functions (PFs) are enumerated
when the FCoE Offload feature is enabled. Not all adapter models support
iSCSI Offload and FCoE Offload.
Device-level configuration includes the following parameters:



SR-IOV
MFW Crash Dump Feature
UEFI Driver Debug Level
To configure device-level parameters:
1.
On the Main Configuration Page, select Device Level Configuration (see
Figure 5-3 on page 36), and then press ENTER.
2.
Under Device Level Configuration, select values for the device-level
parameters shown in Figure 5-6.
38
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Port-level Parameters
Figure 5-6. Device Level Configuration Page
3.
Click Back.
4.
When prompted, click Yes to save the changes. Changes will take effect
after a system reset.
Table 5-2 describes the adapter device-level parameters.
Table 5-2. Device-level Parameters
Parameter
Description
SR-IOV
Enables (Enabled) or disables (Disabled) SR-IOV.
MFW Crash Dump Feature
Enables (Enabled) or disables (Disabled) the collecting of
adapter-specific running state information when specific system
crashes occur.
UEFI Driver Debug Level
Sets the UEFI runtime debug messages that are saved to the qdbg log
file, similar to the Linux dmesg logs. Possible values are 0 to
0xFFFFFFFFF.
Configuring Port-level Parameters
Port-level configuration comprises the following parameters:










Link Speed
Boot Mode
DCBX Protocol
RoCE Priority
iSCSI Offload
FCoE Offload
PXE VLAN Mode
PXE VLAN ID
Link Up Delay
RDMA Protocol Support
39
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Port-level Parameters
To configure the port-level parameters:
1.
On the Main Configuration Page, select Port Level Configuration, and then
press ENTER.
2.
For Link Speed (Figure 5-7), select one of the following values, where the
last specified value becomes the default until a different value is selected:




Auto Negotiated
10 Gbps
25 Gbps
SmartAN (see description below)
NOTE
For a 10G SFP+ and 10GBASE-T (RJ45) interfaced 41000 Series
Adapter, available values are Auto Negotiated, 1Gbps, and 10Gbps.
The FastLinQ SmartAN™ Link Speed option sets the port to use Smart
Auto Negotiation. SmartAN is the default factory setting on the 41000 Series
Adapters. SmartAN provides the ability to automatically set up a link
between a 25G adapter and 10G switch over DAC and optics media without
user intervention or manual setup. The IEEE standards do not provide a
standards-based method to auto-negotiate between a 10G switch and a
25G adapter or between DACs and optics. Cavium’s SmartAN provides an
automatic and convenient method to detect the switch and to determine and
set the link speed, FEC types, media type and length, and so on. Although
SmartAN is most useful for connectivity with a 10G switch and for
determining the type of transceiver that is inserted in the adapter cage, it is
also useful for a wider set of connectivity use cases. The SmartAN link
speed is enabled on the 41000 Series Adapters, but you can select a
different link speed value in the HII console.
40
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Port-level Parameters
Figure 5-7. Port Level Configuration Page: Setting Link Speed
3.
If you selected a Link Speed of 10 Gbps or 25 Gbps in Step 2, the FEC
Mode control appears as shown in Figure 5-7. Select one of the following
fixed-speed FEC Mode values:

None for no FEC.

Fire Code enables FEC (BASE-R-FEC) on 10Gb and 25Gb adapters.

Reed-Solomon Code enables Reed Solomon FEC on 25Gb adapters.
This setting is required for optical connections.
NOTE
The FEC Mode control is not visible if either of the following are true:
 In Step 2, you selected a Link Speed of SmartAN, Auto
Negotiated, or 1 Gbps.
 You have a 10GBASE-T (RJ45) interfaced 41000 Series Adapter.
41
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Port-level Parameters
4.
For Boot Mode, select values for the port-level parameters as shown in
Figure 5-8.
Select from the following Boot Mode options:

Set Boot Mode to PXE to enable PXE boot.

Set Boot Mode to FCoE to enable FCoE boot from SAN over the
hardware offload pathway.

Set Boot Mode to iSCSI (SW) to enable iSCSI remote boot over the
software pathway.

Set Boot Mode to iSCSI (HW) to enable iSCSI remote boot over the
hardware offload pathway.

Set Boot Mode to Disabled to prevent this port from being used as a
remote boot source.
Figure 5-8. Port Level Configuration Page: Setting Boot Mode
5.
For DCBX Protocol support, select one of the following options:

Dynamic automatically determines the DCBX type currently in use on
the attached switch:

IEEE uses only IEEE DCBX protocol.

CEE uses only CEE DCBX protocol.

Disabled disables DCBX on that port.
6.
For RoCE Priority, select a value from 0 to 7. Depending upon the RoCE
configuration, this value must match RoCE traffic class priority value of the
attached switch.
7.
For iSCSI Offload, select either Enabled or Disabled. This setting is only
visible on 41000 Series Adapters that are enabled with iSCSI Offload.
8.
For FCoE Offload, select either Enabled or Disabled. This setting is only
visible on 41000 Series Adapters that are enabled with FCoE Offload.
42
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring FCoE Boot
9.
For PXE VLAN Mode, select either Enabled or Disabled.
10.
For PXE VLAN ID, select a VLAN tag value from 1 to 4094. This value must
match the network settings of the attached switch. This control appears only
when PXE VLAN Mode is Enabled. PXE VLAN ID applies to both PXE and
iSCSI remote boots.
11.
For Link Up Delay, select a value from 0 to 30 seconds. The delay value
specifies how long a remote boot should wait for the switch to enable the
port, such as when the switch port is using spanning tree protocol loop
detection.
12.
For RDMA Protocol Support, select either None, RoCE, or iWARP.
13.
Press the ESC key (Back).
14.
When prompted, select Yes to save the changes. Changes take effect after
a system reset.
Configuring FCoE Boot
FCoE general parameters include the following:

FIP VLAN ID is usually set to 0, but if the FIP VLAN ID is known beforehand,
you can set the value here. If a non-zero value is used, FIP VLAN discovery
is not performed.

Fabric Login Retry Count

Target Login Retry Count options include:

Connect (1–8) is Enabled or Disabled.

WWPN (1–8) shows the WWPN in use (all zeros if the target is offline).
A (+) indicates that the specified target is online, while a (-) indicates
that the specified target is offline.

Boot LUN (1–8) shows the boot LUN (0–255) that is used by the target
array.
To configure the FCoE boot configuration parameters:
1.
On the Main Configuration Page, select FCoE Boot Configuration Menu,
and then select one of the following options:

FCoE General Parameters (Figure 5-9)

FCoE Target Configuration (Figure 5-10)
2.
Press ENTER.
3.
Choose values for the FCoE General or FCoE Target Configuration
parameters.
43
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring iSCSI Boot
Figure 5-9. FCoE General Parameters
Figure 5-10. FCoE Target Configuration
4.
Click Back.
5.
When prompted, click Yes to save the changes. Changes take effect after a
system reset.
Configuring iSCSI Boot
To configure the iSCSI boot configuration parameters:
1.
On the Main Configuration Page, select iSCSI Boot Configuration Menu,
and then select one of the following options:

iSCSI General Configuration
44
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring iSCSI Boot



iSCSI Initiator Configuration
iSCSI First Target Configuration
iSCSI Second Target Configuration
2.
Press ENTER.
3.
Choose values for the appropriate iSCSI configuration parameters:

iSCSI General Configuration (Figure 5-11 on page 46)










iSCSI Initiator Configuration (Figure 5-12 on page 46)










IPv4 Address
IPv4 Subnet Mask
IPv4 Default Gateway
IPv4 Primary DNS
IPv4 Secondary DNS
VLAN ID
iSCSI Name
CHAP ID
CHAP Secret
iSCSI First Target Configuration (Figure 5-13 on page 47)








TCP/IP Parameters Via DHCP
iSCSI Parameters Via DHCP
CHAP Authentication
CHAP Mutual Authentication
IP Version
ARP Redirect
DHCP Request Timeout
Target Login Timeout
DHCP Vendor ID
Connect
IPv4 Address
TCP Port
Boot LUN
iSCSI Name
CHAP ID
CHAP Secret
iSCSI Second Target Configuration (Figure 5-14 on page 47)







Connect
IPv4 Address
TCP Port
Boot LUN
iSCSI Name
CHAP ID
CHAP Secret
45
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring iSCSI Boot
4.
Click Back.
5.
When prompted, click Yes to save the changes. Changes take effect after a
system reset.
Figure 5-11. iSCSI General Configuration
Figure 5-12. iSCSI Initiator Configuration
46
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Partitions
Figure 5-13. iSCSI First Target Configuration
Figure 5-14. iSCSI Second Target Configuration
Configuring Partitions
You can configure bandwidth ranges for each partition on a 25Gb adapter.
To configure the maximum and minimum bandwidth allocations:
1.
On the Main Configuration Page, select Partitions Configuration, and then
press ENTER.
2.
On the Partitions Configuration page, select Global Bandwidth Allocation.
Figure 5-15 shows the page when FCoE Offload and iSCSI Offload are
disabled.
47
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Partitions
Figure 5-15. Partitions Configuration Page (No FCoE Offload or iSCSI Offload)
Figure 5-16 shows the page when NPAR mode is enabled with FCoE
Offload and iSCSI Offload enabled.
Figure 5-16. Partitions Configuration Page (with FCoE Offload and iSCSI Offload)
3.
On the Global Bandwidth Allocation page (Figure 5-17), click each partition
minimum and maximum TX bandwidth field for which you want to allocate
bandwidth. There are eight partitions per port in dual-port mode.
48
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Partitions
Figure 5-17. Global Bandwidth Allocation Page

Partition n Minimum TX Bandwidth is the minimum transmit
bandwidth of the selected partition expressed as a percentage of the
maximum physical port link speed. Values can be 0–100. When DCBX
ETS mode is enabled, the per-traffic class DCBX ETS minimum
bandwidth value is used simultaneously with the per-partition minimum
TX bandwidth value.The total of the minimum TX bandwidth values of
all partitions on a single port must equal 100 or be all zeros.

Partition n Maximum TX Bandwidth is the maximum transmit
bandwidth of the selected partition expressed as a percentage of the
maximum physical port link speed. Values can be 1–100. The
per-partition maximum TX bandwidth value applies regardless of the
DCBX ETS mode setting. If the maximum TX bandwidth for a partition
is set to 0, the effective bandwidth is one percent.
Type a value in each selected field, and then click Back.
4.
When prompted, click Yes to save the changes. Changes will take effect
after a system reset.
5.
To examine a specific partition configuration, on the Partitions Configuration
page (Figure 5-15 on page 48), select Partition n Configuration.
49
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Partitions
For example, selecting Partition 1 Configuration, opens the Partition 1
Configuration page (Figure 5-18), which shows the following Partition 1
parameters:






Personality
NIC Mode
PCI Device ID
PCI (bus) Address
Permanent MAC Address
Virtual MAC Address
Partition 1 is always present and cannot be disabled.
Figure 5-18. Partition 1 Configuration
If FCoE Offload is present, the Partition 2 Configuration shows the
Personality as FCoE (Figure 5-19) and the following additional parameters:










NIC Mode (Disabled)
FCoE Offload Mode (Enabled)
FCoE (FIP) MAC Address
Virtual FIP MAC Address
World Wide Port Name
Virtual World Wide Port Name
World Wide Node Name
Virtual World Wide Node Name
PCI Device ID
PCI Address
50
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Partitions
Figure 5-19. Partition 2 Configuration: FCoE Offload
If iSCSI Offload is present, the Partition 3 Configuration shows the
Personality as iSCSI (Figure 5-20) and the following additional parameters:






NIC Mode (Disabled)
iSCSI Offload Mode (Enabled)
iSCSI MAC Address
Virtual iSCSI Offload MAC Address
PCI Device ID
PCI Address
Figure 5-20. Partition 3 Configuration: iSCSI Offload
51
AH0054601-00 A
5–Adapter Preboot Configuration
Configuring Partitions
The remaining partitions, including the previous (if not offload-enabled), are
Ethernet. Opening a partition 2 or greater Ethernet partition shows the
Personality as Ethernet (Figure 5-21) and the following additional
parameters:

NIC Mode (Enabled or Disabled)—When disabled, the partition is
hidden such that it does not show up to the OS if fewer than the
maximum quantity of partitions (or PCI PFs) are detected.

PCI Device ID

PCI Address

Permanent MAC Address

Virtual MAC Address
Figure 5-21. Partition 4 Configuration: Ethernet
52
AH0054601-00 A
6
RoCE Configuration
This chapter describes RDMA over converged Ethernet (RoCE v1 and v2)
configuration on the 41000 Series Adapter, the Ethernet switch, and the Windows
or Linux host, including:

Supported Operating Systems and OFED

Planning for RoCE

Preparing the Adapter

Preparing the Ethernet Switch

Configuring RoCE on the Adapter for Windows Server

Configuring RoCE on the Adapter for Linux
NOTE
Some RoCE features may not be fully enabled in the current release. For
details, refer to Appendix C Feature Constraints.
Supported Operating Systems and OFED
Table 6-1 shows the operating system support for RoCE v1 and v2, iWARP, and
OFED.
Table 6-1. OS Support for RoCE v1/v2, iWARP, and OFED
RoCE v1
RoCE v2
Inbox OFED
OFED 3.18-3
GA
Windows Server 2012 R2 a
Yes
Yes
No
No
Windows Server 2016 a
Yes
Yes
No
No
RHEL 6.8
Yes
No
Yes a
Yes a
RHEL 6.9
Yes
No
Yes a
No
RHEL 7.2
Yes
No
Yes a
Yes a
Operating System
53
AH0054601-00 A
6–RoCE Configuration
Planning for RoCE
Table 6-1. OS Support for RoCE v1/v2, iWARP, and OFED (Continued)
RoCE v1
RoCE v2
Inbox OFED
OFED 3.18-3
GA
RHEL 7.3
Yes
Yes
Yes a
No
SLES 11 SP4
Yes
No
Yes a
Yes a
SLES 12 SP1
Yes
No
Yes
Yes
SLES 12 SP2
Yes
Yes
Yes a
No
CentOS 7.3
Yes
Yes
Yes a
No
Ubuntu 14.04 LTS
Yes
No
Yes a
No
Ubuntu 16.04 LTS
Yes
No
Yes a
No
VMware ESXi 6.5 b
Yes
Yes
–
–
Operating System
a
b
iWARP-supported OSs.
Subject to OS support availability.
Planning for RoCE
As you prepare to implement RoCE, consider the following limitations:

If you are using the inbox OFED, the operating system should be the same
on the server and client systems. Some of the applications may work
between different operating systems, but there is no guarantee. This is an
OFED limitation.

For OFED applications (most often perftest applications), server and client
applications should use the same options and values. Problems can arise if
the operating system and the perftest application have different versions. To
confirm the perftest version, issue the following command:
# ib_send_bw --version

Building libqedr in inbox OFED requires installing libibverbs-devel.

Running user space applications in inbox OFED requires installing the
InfiniBand&reg; Support group, by yum groupinstall “InfiniBand Support” that
contains libibcm, libibverbs, and more.

OFED and RDMA applications that depend on libibverbs also require the
QLogic RDMA user space library, libqedr. Install libqedr using the libqedr
RPM or source packages.
54
AH0054601-00 A
6–RoCE Configuration
Preparing the Adapter

RoCE supports only little endian.

RoCE does not work over a VF in an SR-IOV environment.
Preparing the Adapter
Follow these steps to enable DCBX and specify the RoCE priority using the HII
management application. For information about the HII application, see Chapter 5
Adapter Preboot Configuration.
1.
In the Main Configuration Page, select Data Center Bridging (DCB)
Settings, and then click Finish.
2.
In the Data Center Bridging (DCB) Settings window, click the DCBX
Protocol option. The 41000 Series Adapter supports both CEE and IEEE
protocols. This value should match the corresponding value on the DCB
switch. In this example, select CEE.
3.
In the RoCE Priority box, type a priority value. This value should match the
corresponding value on the DCB switch. In this example, type 5. Typically, 0
is used for the default lossy traffic class, 3 is used for the FCoE traffic class,
and 4 is used for lossless iSCSI-TLV over DCB traffic class.
4.
Click Back.
5.
When prompted, click Yes to save the changes. Changes will not take effect
until after a system reset.
For Windows, you can configure DCBX using the HII or QoS method.The
configuration shown in this section is through HII. For QoS, refer to
“Configuring QoS for RoCE” on page 177“.
Preparing the Ethernet Switch
This section describes how to configure a Cisco&reg; Nexus&reg; 6000 Ethernet Switch, a
Dell&reg; Z9100 Ethernet Switch, and an Arista™ 7060X Switch for RoCE.

Configuring the Cisco Nexus 6000 Ethernet Switch

Configuring the Dell Z9100 Ethernet Switch

Configuring the Arista 7060X Ethernet Switch
Configuring the Cisco Nexus 6000 Ethernet Switch
Configuring the Cisco Nexus 6000 Ethernet Switch for RoCE comprises
configuring class maps, configuring policy maps, applying the policy, and
assigning a VLAN ID to the switch port.
55
AH0054601-00 A
6–RoCE Configuration
Preparing the Ethernet Switch
To configure the Cisco switch:
1.
Open a config terminal session.
Switch# config terminal
switch(config)#
2.
Configure quality of service (QoS) class map and set the RoCE priority to
match the adapter (5).
switch(config)# class-map type qos class-roce
switch(config)# match cos 5
3.
Configure queuing class maps.
switch(config)# class-map type queuing class-roce
switch(config)# match qos-group 3
4.
Configure network QoS class maps.
switch(config)# class-map type network-qos class-roce
switch(config)# match qos-group 3
5.
Configure QoS policy maps.
switch(config)# policy-map type qos roce
switch(config)# class type qos class-roce
switch(config)# set qos-group 3
6.
Configure queuing policy maps to assign network bandwidth. In this
example, use a value of 50 percent.
switch(config)# policy-map type queuing roce
switch(config)# class type queuing class-roce
switch(config)# bandwidth percent 50
7.
Configure network QoS policy maps to set priority flow control for no-drop
traffic class.
switch(config)# policy-map type network-qos roce
switch(config)# class type network-qos class-roce
switch(config)# pause no-drop
8.
Apply the new policy at the system level.
switch(config)# system qos
switch(config)# service-policy type qos input roce
switch(config)# service-policy type queuing output roce
switch(config)# service-policy type queuing input roce
switch(config)# service-policy type network-qos roce
56
AH0054601-00 A
6–RoCE Configuration
Preparing the Ethernet Switch
9.
Assign a VLAN ID to the switch port to match the VLAN ID assigned to the
adapter (5).
switch(config)# interface ethernet x/x
switch(config)# switchport mode trunk
switch(config)# switchport trunk allowed vlan 1,5
Configuring the Dell Z9100 Ethernet Switch
Configuring the Dell Z9100 Ethernet Switch for RoCE comprises configuring a
DCB map for RoCE, configuring priority-based flow control (PFC) and enhanced
transmission selection (ETS), verifying the DCB map, applying the DCB map to
the port, verifying PFC and ETS on the port, specifying the DCB protocol, and
assigning a VLAN ID to the switch port.
To configure the Dell switch:
1.
Create a DCB map.
Dell# configure
Dell(conf)# dcb-map roce
Dell(conf-dcbmap-roce)#
2.
Configure two ETS traffic classes in the DCB map with 50 percent
bandwidth assigned for RoCE (group 1).
Dell(conf-dcbmap-roce)# priority-group 0 bandwidth 50 pfc off
Dell(conf-dcbmap-roce)# priority-group 1 bandwidth 50 pfc on
3.
Configure the PFC priority to match the adapter priority (5).
Dell(conf-dcbmap-roce)# priority-pgid 0 0 0 0 0 1 0 0
4.
Verify the DCB map configuration priority group.
Dell(conf-dcbmap-roce)# do show qos dcb-map roce
----------------------State
:Complete
PfcMode :ON
-------------------PG:0 TSA:ETS
BW:40
PFC:OFF
Priorities:0 1 2 3 4 6 7
PG:1 TSA:ETS
BW:60
PFC:ON
Priorities:5
57
AH0054601-00 A
6–RoCE Configuration
Preparing the Ethernet Switch
5.
Apply the DCB map to the port.
Dell(conf)# interface twentyFiveGigE 1/8/1
Dell(conf-if-tf-1/8/1)# dcb-map roce
6.
Verify the ETS and PFC configuration on the port. The following examples
show summarized interface information for ETS and detailed interface
information for PFC.
Dell(conf-if-tf-1/8/1)# do show interfaces twentyFiveGigE 1/8/1 ets summary
Interface twentyFiveGigE 1/8/1
Max Supported TC is 4
Number of Traffic Classes is 8
Admin mode is on
Admin Parameters :
-----------------Admin is enabled
PG-grp
Priority#
BW-%
BW-COMMITTED
BW-PEAK TSA
%
Rate(Mbps) Burst(KB) Rate(Mbps) Burst(KB)
-----------------------------------------------------------------------0
0,1,2,3,4,6,7
40
ETS
1
5
60
ETS
2
3
Dell(Conf)# do show interfaces twentyFiveGigE 1/8/1 pfc detail
Interface twentyFiveGigE 1/8/1
Admin mode is on
Admin is enabled, Priority list is 5
Remote is enabled, Priority list is 5
Remote Willing Status is enabled
Local is enabled, Priority list is 5
Oper status is init
PFC DCBX Oper status is Up
State Machine Type is Feature
TLV Tx Status is enabled
PFC Link Delay 65535 pause quntams
Application Priority TLV Parameters :
-------------------------------------FCOE TLV Tx Status is disabled
ISCSI TLV Tx Status is enabled
Local FCOE PriorityMap is 0x0
Local ISCSI PriorityMap is 0x20
Remote ISCSI PriorityMap is 0x20
66 Input TLV pkts, 99 Output TLV pkts, 0 Error pkts, 0 Pause Tx pkts, 0 Pause Rx pkts
58
AH0054601-00 A
6–RoCE Configuration
Preparing the Ethernet Switch
66 Input Appln Priority TLV pkts, 99 Output Appln Priority TLV pkts, 0 Error Appln
Priority TLV Pkts
7.
Configure the DCBX protocol (CEE in this example).
Dell(conf)# interface twentyFiveGigE 1/8/1
Dell(conf-if-tf-1/8/1)# protocol lldp
Dell(conf-if-tf-1/8/1-lldp)# dcbx version cee
8.
Assign a VLAN ID to the switch port to match the VLAN ID assigned to the
adapter (5).
Dell(conf)# interface vlan 5
Dell(conf-if-vl-5)# tagged twentyFiveGigE 1/8/1
Configuring the Arista 7060X Ethernet Switch
The following examples show Arista 7060X Switch configuration.
Configuring RoCE APP TLVs for both RoCE v1 and RoCE v2
The following is an example on configuring RoCE APP TLV for both RoCE v1 and
RoCE v2:
Arista-7060X-EIT(config)#dcbx application ether 0x8915 priority 5
Arista-7060X-EIT(config)#dcbx application udp 4791 priority 5
Mapping Priority to Traffic Class
In the following example, Priority 5 is mapped to traffic class 1, and the rest of the
priorities are mapped to traffic class 0:
Arista-7060X-EIT(config)#dcbx ets qos map cos 0 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 1 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 2 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 3 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 4 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 5 traffic-class 1
Arista-7060X-EIT(config)#dcbx ets qos map cos 6 traffic-class 0
Arista-7060X-EIT(config)#dcbx ets qos map cos 7 traffic-class 0
Setting Up ETS
In the following example, traffic class 0 is configured with 5 percent bandwidth,
and traffic class 1 is configured with 95 percent bandwidth:
Arista-7060X-EIT(config)#dcbx ets traffic-class 0 bandwidth 5
Arista-7060X-EIT(config)#dcbx ets traffic-class 1 bandwidth 95
59
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Windows Server
Configuring the Interface
The same VLAN ID must be assigned to the server adapter ports. Perform the
following configuration on all of the switch interfaces where the server adapter
ports are connected. DCBX The protocol mode is set to CEE and the PFC is
enabled for Priority 5.
Arista-7060X-EIT(config)#interface ethernet 26/1
Arista-7060X-EIT(config-if-Et26/1)#dcbx mode cee
Arista-7060X-EIT(config-if-Et26/1)# priority-flow-control mode on
Arista-7060X-EIT(config-if-Et26/1)# priority-flow-control priority
5 no-drop
Arista-7060X-EIT(config-if-Et26/1)# switchport mode trunk
Arista-7060X-EIT(config-if-Et26/1)# switchport trunk allowed vlan
1,5
Configuring RoCE on the Adapter for Windows
Server
Configuring RoCE on the adapter for Windows Server comprises enabling RoCE
on the adapter and verifying the Network Direct MTU size.
To configure RoCE on a Windows Server host:
1.
2.
Enable RoCE on the adapter.
a.
Open the Windows Device Manager, and then open the 41000 Series
Adapters NDIS Miniport Properties.
b.
On the QLogic FastLinQ Adapter Properties, click the Advanced tab.
On the Advanced page, configure the properties listed in Table 6-2 by
selecting each item under Property and choosing an appropriate Value for
that item. Then click OK.
Table 6-2. Advanced Properties for RoCE
Property
Value or Description
Network Direct Functionality
Enabled
Network Direct Mtu Size
The network direct MTU size must be less than the
jumbo packet size.
RDMA Mode
RoCE v1 or RoCE v2
VLAN ID
Assign any VLAN ID to the interface. The value must
be the same as is assigned on the switch.
60
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Windows Server
Figure 6-1 shows an example of configuring a property value.
Figure 6-1. Configuring RoCE Properties
3.
Verify that RoCE is enabled on the adapter using Windows PowerShell. The
Get-NetAdapterRdma command lists the adapters that support
RDMA—both ports are enabled.
NOTE
If you are configuring RoCE over Hyper-V, do not assign a VLAN ID to
the physical interface.
PS C:\Users\Administrator&gt; Get-NetAdapterRdma
Name
InterfaceDescription
Enabled
-----
--------------------
-------
SLOT 4 3 Port 1
QLogic FastLinQ QL45212...
True
SLOT 4 3 Port 2
QLogic FastLinQ QL45212...
True
61
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
4.
Verify that RoCE is enabled on the host operating system using PowerShell.
The Get-NetOffloadGlobalSetting command shows NetworkDirect
is enabled.
PS C:\Users\Administrators&gt; Get-NetOffloadGlobalSetting
5.
ReceiveSideScaling
: Enabled
ReceiveSegmentCoalescing
: Enabled
Chimney
: Disabled
TaskOffload
: Enabled
NetworkDirect
: Enabled
NetworkDirectAcrossIPSubnets
: Blocked
PacketCoalescingFilter
: Disabled
Connect a server message block (SMB) drive, run RoCE traffic, and verify
the results.
To set up and connect to an SMB drive, view the information available online
from Microsoft:
https://technet.microsoft.com/en-us/library/hh831795(v=ws.11).aspx
6.
By default, Microsoft's SMB Direct establishes two RDMA connections per
port, which provides good performance, including line rate at a higher block
size (for example, 64KB). To optimize performance, you can change the
quantity of RDMA connections per RDMA interface to four (or greater).
To increase the quantity of RDMA connections to four (or more), issue the
following command in PowerShell:
PS C:\Users\Administrator&gt; Set-ItemProperty -Path &quot;HKLM:\SYSTEM\CurrentControlSet\
Services\LanmanWorkstation\Parameters&quot; ConnectionCountPerRdmaNetworkInterface -Type
DWORD -Value 4 –Force
Configuring RoCE on the Adapter for Linux
This section describes the RoCE configuration procedure for RHEL and SLES. It
also describes how to verify the RoCE configuration and provides some guidance
about using group IDs (GIDs) with VLAN interfaces.

RoCE Configuration for RHEL

RoCE Configuration for SLES

RoCE Configuration for Ubuntu

Verifying the RoCE Configuration on Linux

VLAN Interfaces and GID Index Values

RoCE v2 Configuration for Linux
62
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
RoCE Configuration for RHEL
To configure RoCE on the adapter, the Open Fabrics Enterprise Distribution
(OFED) must be installed and configured on the RHEL host.
To prepare inbox OFED for RHEL:
1.
While installing or upgrading the operating system, select the InfiniBand&reg;
and OFED support packages.
2.
Install the following RPMs from the RHEL ISO image:
libibverbs-devel-x.x.x.x86_64.rpm
(required for libqedr library)
perftest-x.x.x.x86_64.rpm
(required for InfiniBand bandwidth and latency applications)
or, using Yum, install the inbox OFED:
yum groupinstall &quot;Infiniband Support&quot;
yum install perftest
yum install tcl tcl-devel tk zlib-devel libibverbs
libibverbs-devel
NOTE
During installation, if you already selected the previously mentioned
packages, you need not reinstall them. The inbox OFED and support
packages may vary depending on the operating system version.
3.
Install the new Linux drivers as described in “Installing the Linux Drivers with
RDMA” on page 17.
63
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
RoCE Configuration for SLES
To configure RoCE on the adapter for an SLES host, OFED must be installed and
configured on the SLES host.
To install inbox OFED for SLES Linux:
1.
While installing or upgrading the operating system, select the InfiniBand
support packages.
2.
Install the following RPMs from the corresponding SLES SDK kit image:
libibverbs-devel-x.x.x.x86_64.rpm
(required for libqedr installation)
perftest-x.x.x.x86_64.rpm
(required for bandwidth and latency applications)
3.
Install the Linux drivers, as described in “Installing the Linux Drivers with
RDMA” on page 17.
RoCE Configuration for Ubuntu
To configure RoCE on the adapter for an Ubuntu host, RDMA must be installed
and configured on the Ubuntu host.
To configure and set up RoCE for Ubuntu 14.04.5/16.04.1 Linux:
1.
When you begin installing the Ubuntu server, verify if the basic packages,
modules, and tools are available for Ethernet and RDMA. Log in as a ROOT,
and install all required packages.
a.
Install the basic packages required for Ubuntu:
# apt-get install -f build-essential pkg-config vlan
automake autoconf dkms git
b.
Install the following RDMA packages required for Ubuntu:
# apt-get install –f libibverbs* librdma* libibcm.*
libibmad.* libibumad*
c.
Install RDMA user space tools and libraries required for Ubuntu:
# apt-get install -f libtool ibutils ibverbs-utils
rdmacm-utils infiniband-diags perftest librdmacm-dev
libibverbs-dev numactl libnuma-dev libnl-3-200
libnl-route-3-200 libnl-route-3-dev libnl-utils
64
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
NOTE
You might see a few packages already installed because of a
dependency, but make sure all of the preceding packages are installed
(dpkg --get-selections | grep &lt;pkg_name&gt;), and follow the
package installation method prescribed by Ubuntu.
2.
If the file /etc/udev/rules.d/40-rdma.rules does not exist, create it
with the following content:
KERNEL==&quot;umad*&quot;, NAME=&quot;infiniband/%k&quot;
KERNEL==&quot;issm*&quot;, NAME=&quot;infiniband/%k&quot;
KERNEL==&quot;ucm*&quot;, NAME=&quot;infiniband/%k&quot;, MODE=&quot;0666&quot;
KERNEL==&quot;uverbs*&quot;, NAME=&quot;infiniband/%k&quot;, MODE=&quot;0666&quot;
KERNEL==&quot;ucma&quot;, NAME=&quot;infiniband/%k&quot;, MODE=&quot;0666&quot;
KERNEL==&quot;rdma_cm&quot;, NAME=&quot;infiniband/%k&quot;, MODE=&quot;0666&quot;
3.
Edit the /etc/security/limits.conf file to increase the size of memory,
which can be locked by a non-root user. Add the following lines, and then log
out:
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
4.
Log into the system again, or verify after reboot. Then issue the following
command:
#ulimit -l
You should get the output as unlimited.
5.
Reboot the system.
6.
To allow the device to be recognized as an InfiniBand device that can be
used by OFED, install the FastLinQ package by issuing the following
commands:
# cd fastlinq-X.X.X.X
# make clean
# make install
65
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
7.
Install the libqedr libraries to work with RDMA user space applications using
one of the following command options:

Option 1:
# cd fastlinq-X.X.X.X
# make libqedr_install

Option 2:
# cd fastlinq-X.X.X.X/libqedr-X.X.X.X/
# ./configure --prefix=/usr --libdir=${exec_prefix}/lib
--sysconfdir=/etc
# make install
8.
Before loading the QLogic Ethernet and RDMA drivers, uninstall the existing
out-of-box or inbox drivers by issuing the following commands:
# modprobe –r qede
# depmod –a
# modprobe -v qedr
9.
Load the RDMA modules by issuing the following commands. You must
perform this step whenever you reboot the system.
# modprobe rdma_cm
# modprobe ib_uverbs
# modprobe rdma_ucm
# modprobe ib_ucm
# modprobe ib_umad
10.
To list RoCE devices, issue the ibv_devinfo command:
# ibv_devinfo
11.
Assign IP addresses to the Ethernet interfaces:

To assign the static IP address, edit the /etc/network/interfaces
file with the following entries:
auto eth0
iface eth0 inet static
address 192.168.10.5
netmask 255.255.255.0
gateway 192.168.10.254
66
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux

To assign the DHCP IP address, edit the /etc/network/interfaces
file with the following entries:
auto eth0
iface eth0 inet dhcp
Verifying the RoCE Configuration on Linux
After installing OFED, installing the Linux driver, and loading the RoCE drivers,
verify that the RoCE devices were detected on all Linux operating systems.
To verify RoCE configuration on Linux:
1.
Stop firewall tables using service/systemctl commands.
2.
For RHEL only: If the RDMA service is installed (yum install rdma), verify
that the RDMA service has started.
NOTE
For RHEL 6.x and SLES 11 SP4, you must start RDMA service after
reboot. For RHEL 7.x and SLES 12 SPX and later, RDMA service
starts itself after reboot. For Ubuntu, there are no RDMA service, so
the user must load each RDMA module.
On RHEL or CentOS: Use the service rdma status command to start
service:

If RDMA has not started, issue the following command:
# service rdma start

If RDMA does not start, issue either of the following alternative
commands:
# /etc/init.d/rdma start
or
# systemctl start rdma.service
3.
Verify that the RoCE devices were detected by examining the dmesg logs:
# dmesg|grep qedr
[87910.988411] qedr: discovered and registered 2 RoCE funcs
67
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
4.
Verify that all of the modules have been loaded. For example:
# lsmod|grep qedr
5.
qedr
89871
0
qede
96670
1 qedr
qed
2075255
ib_core
88311 16 qedr, rdma_cm, ib_cm,
ib_sa,iw_cm,xprtrdma,ib_mad,ib_srp,
ib_ucm,ib_iser,ib_srpt,ib_umad,
ib_uverbs,rdma_ucm,ib_ipoib,ib_isert
2 qede,qedr
Configure the IP address and enable the port using a configuration method
such as ifconfig:
# ifconfig ethX 192.168.10.10/24 up
6.
Issue the ibv_devinfo command. For each PCI function, you should see a
separate hca_id, as shown in the following example:
root@captain:~# ibv_devinfo
hca_id: qedr0
transport:
InfiniBand (0)
fw_ver:
8.3.9.0
node_guid:
020e:1eff:fe50:c7c0
sys_image_guid:
020e:1eff:fe50:c7c0
vendor_id:
0x1077
vendor_part_id:
5684
hw_ver:
0x0
phys_port_cnt:
1
port:
1
state:
7.
PORT_ACTIVE (1)
max_mtu:
4096 (5)
active_mtu:
1024 (3)
sm_lid:
0
port_lid:
0
port_lmc:
0x00
link_layer:
Ethernet
Verify the L2 and RoCE connectivity between all servers: one server acts as
a server, another acts as a client.

Verify the L2 connection using a simple ping command.
68
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux

Verify the RoCE connection by performing an RDMA ping on the
server or client:
On the server, issue the following command:
ibv_rc_pingpong -d &lt;ib-dev&gt; -g 0
On the client, issue the following command:
ibv_rc_pingpong -d &lt;ib-dev&gt; -g 0 &lt;server L2 IP address&gt;
The following are examples of successful ping pong tests on the server and the
client.
Server Ping:
root@captain:~# ibv_rc_pingpong -d qedr0 -g 0
local address: LID 0x0000, QPN 0xff0000, PSN 0xb3e07e, GID fe80::20e:1eff:fe50:c7c0
remote address: LID 0x0000, QPN 0xff0000, PSN 0x934d28, GID fe80::20e:1eff:fe50:c570
8192000 bytes in 0.05 seconds = 1436.97 Mbit/sec
1000 iters in 0.05 seconds = 45.61 usec/iter
Client Ping:
root@lambodar:~# ibv_rc_pingpong -d qedr0 -g 0 192.168.10.165
local address: LID 0x0000, QPN 0xff0000, PSN 0x934d28, GID fe80::20e:1eff:fe50:c570
remote address: LID 0x0000, QPN 0xff0000, PSN 0xb3e07e, GID fe80::20e:1eff:fe50:c7c0
8192000 bytes in 0.02 seconds = 4211.28 Mbit/sec
1000 iters in 0.02 seconds = 15.56 usec/iter

To display RoCE statistics, issue the following commands, where X is the
device number:
&gt; mount -t debugfs nodev /sys/kernel/debug
&gt; cat /sys/kernel/debug/qedr/qedrX/stats
VLAN Interfaces and GID Index Values
If you are using VLAN interfaces on both the server and the client, you must also
configure the same VLAN ID on the switch. If you are running traffic through a
switch, the InfiniBand applications must use the correct GID value, which is based
on the VLAN ID and VLAN IP address.
Based on the following results, the GID value (-x 4 / -x 5) should be used for any
perftest applications.
# ibv_devinfo -d qedr0 -v|grep GID
GID[ 0]:
fe80:0000:0000:0000:020e:1eff:fe50:c5b0
GID[ 1]:
0000:0000:0000:0000:0000:ffff:c0a8:0103
GID[ 2]:
2001:0db1:0000:0000:020e:1eff:fe50:c5b0
GID[ 3]:
2001:0db2:0000:0000:020e:1eff:fe50:c5b0
GID[ 4]:
0000:0000:0000:0000:0000:ffff:c0a8:0b03 IP address for VLAN interface
69
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
GID[
5]:
fe80:0000:0000:0000:020e:1e00:0350:c5b0
VLAN ID 3
NOTE
The default GID value is zero (0) for back-to-back or pause settings. For
server/switch configurations, you must identify the proper GID value. If you
are using a switch, refer to the corresponding switch configuration
documents for the proper settings.
RoCE v2 Configuration for Linux
To verify RoCE v2 functionality, you must use RoCE v2 supported kernels.
To configure RoCE v2 for Linux:
1.
Ensure that you are using one of the following supported kernels:


2.
SLES 12 SP2 GA
RHEL 7.3 GA
Configure RoCEv2 as follows:
a.
Identify the GID index for RoCE v2.
b.
Configure the routing address for the server and client.
c.
Enable L3 routing on the switch.
NOTE
You can configure RoCE v1 and v2 by using RoCE v2 supported
kernels. These kernels allow you to run RoCE traffic over the same
subnet, as well as over different subnets such as RoCE v2 and any
routable environment. Only a few settings are required for RoCE v2,
and all other switch and adapter settings are common for RoCE v1
and v2.
Identifying RoCE v2 GID Index or Address
To find RoCE v1 and v2 specific GIDs, use either sys or class parameters, or run
RoCE scripts from the 41000 Series FastLinQ source package. To check the
default RoCE GID Index and address, issue the ibv_devinfo command and
compare it with the sys or class parameters. For example:
#ibv_devinfo -d qedr0 -v|grep GID
GID[ 0]:
fe80:0000:0000:0000:020e:1eff:fec4:1b20
GID[ 1]:
fe80:0000:0000:0000:020e:1eff:fec4:1b20
GID[ 2]:
0000:0000:0000:0000:0000:ffff:1e01:010a
GID[ 3]:
0000:0000:0000:0000:0000:ffff:1e01:010a
GID[ 4]:
3ffe:ffff:0000:0f21:0000:0000:0000:0004
70
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
GID[
GID[
GID[
5]:
6]:
7]:
3ffe:ffff:0000:0f21:0000:0000:0000:0004
0000:0000:0000:0000:0000:ffff:c0a8:6403
0000:0000:0000:0000:0000:ffff:c0a8:6403
Verifying RoCE v1 or v2 GID Index and Address from sys and class Parameters
Use one of the following options to verify the RoCE v1 or v2 GID Index and
address from the sys and class parameters:

Option 1
# cat /sys/class/infiniband/qedr0/ports/1/gid_attrs/types/0
IB/RoCE v1
# cat /sys/class/infiniband/qedr0/ports/1/gid_attrs/types/1
RoCE v2
# cat /sys/class/infiniband/qedr0/ports/1/gids/0
fe80:0000:0000:0000:020e:1eff:fec4:1b20
# cat /sys/class/infiniband/qedr0/ports/1/gids/1
fe80:0000:0000:0000:020e:1eff:fec4:1b20

Option 2
Use the scripts from the FastLinQ source package.
#/../fastlinq-8.x.x.x/add-ons/roce/show_gids.sh
DEV
PORT INDEX
GID
-----------qedr0 1 0
fe80:0000:0000:0000:020e:1eff:fec4:1b20
qedr0 1 1
fe80:0000:0000:0000:020e:1eff:fec4:1b20
qedr0 1 2
0000:0000:0000:0000:0000:ffff:1e01:010a
qedr0 1 3
0000:0000:0000:0000:0000:ffff:1e01:010a
qedr0 1 4
3ffe:ffff:0000:0f21:0000:0000:0000:0004
qedr0 1 5
3ffe:ffff:0000:0f21:0000:0000:0000:0004
qedr0 1 6
0000:0000:0000:0000:0000:ffff:c0a8:6403
qedr0 1 7
0000:0000:0000:0000:0000:ffff:c0a8:6403
qedr1 1 0
fe80:0000:0000:0000:020e:1eff:fec4:1b21
qedr1 1 1
fe80:0000:0000:0000:020e:1eff:fec4:1b21
IPv4
------------
30.1.1.10
30.1.1.10
192.168.100.3
192.168.100.3
VER
--v1
v2
v1
v2
v1
v2
v1
v2
v1
v2
DEV
--p4p1
p4p1
p4p1
p4p1
p4p1
p4p1
p4p1.100
p4p1.100
p4p2
p4p2
NOTE
You must specify the GID index values for RoCE v1 or v2 based on
server or switch configuration (Pause/PFC). Use GID index for the link
local IPv6 address, IPv4 address, or IPv6 address. To use VLAN
tagged frames for RoCE traffic, you must specify GID index values that
are derived from the VLAN IPv4 or IPv6 address.
71
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
Verifying RoCE v1 or v2 Functionality Through perftest Applications
This section shows how to verify RoCE v1 or v2 functionality through perftest
applications. In this example, the following server IP and client IP are used:


Server IP: 192.168.100.3
Client IP: 192.168.100.4
Verifying RoCE v1
Run over the same subnet and use the RoCE v1 GID index.
Server# ib_send_bw -d qedr0 -F -x 0
Client# ib_send_bw -d qedr0 -F -x 0
192.168.100.3
Verifying RoCE v2
Run over the same subnet and use the RoCE v2 GID index.
Server# ib_send_bw -d qedr0 -F -x 1
Client# ib_send_bw -d qedr0 -F -x 1
192.168.100.3
NOTE
If you are running through a switch PFC configuration, use VLAN GIDs for
RoCE v1 or v2 through the same subnet.
Verifying RoCE v2 Through Different Subnets
NOTE
You must first configure the route settings for the switch and servers. On the
adapter, set the RoCE priority and DCBX mode using the HII or UEFI user
interface.
To verify RoCE v2 through different subnets.
1.
Set the route configuration for the server and client using the DCBX-PFC
configuration.

System Settings:
Server VLAN IP : 192.168.100.3 and Gateway :192.168.100.1
Client VLAN IP : 192.168.101.3 and Gateway :192.168.101.1

Server Configuration:
#/sbin/ip link add link p4p1 name p4p1.100 type vlan id 100
#ifconfig p4p1.100 192.168.100.3/24 up
#ip route add 192.168.101.0/24 via 192.168.100.1 dev p4p1.100
72
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux

Client Configuration:
#/sbin/ip link add link p4p1 name p4p1.101 type vlan id 101
#ifconfig p4p1.101 192.168.101.3/24 up
#ip route add 192.168.100.0/24 via 192.168.101.1 dev p4p1.101
2.
Set the switch settings using the following procedure.

Use any flow control method (Pause, DCBX-CEE, or DCBX-IEEE),
and enable IP routing for RoCE v2. See “Preparing the Ethernet
Switch” on page 55 for RoCE v2 configuration, or refer to the vendor
switch documents.

If you are using PFC configuration and L3 routing, run RoCE v2 traffic
over the VLAN using a different subnet, and use the RoCE v2 VLAN
GID index.
Server# ib_send_bw -d qedr0 -F -x 5
Client# ib_send_bw -d qedr0 -F -x 5 192.168.100.3
Server Switch Settings:
Figure 6-2. Switch Settings, Server
73
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
Client Switch Settings:
Figure 6-3. Switch Settings, Client
Configuring RoCE v1 or v2 Settings for RDMA_CM Applications
To configure RoCE, use the following scripts from the FastLinQ source package:
# ./show_rdma_cm_roce_ver.sh
qedr0 is configured to IB/RoCE v1
qedr1 is configured to IB/RoCE v1
# ./config_rdma_cm_roce_ver.sh v2
configured rdma_cm for qedr0 to RoCE v2
configured rdma_cm for qedr1 to RoCE v2
Server Settings:
Figure 6-4. Configuring RDMA_CM Applications: Server
74
AH0054601-00 A
6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
Client Settings:
Figure 6-5. Configuring RDMA_CM Applications: Client
75
AH0054601-00 A
7
iWARP Configuration
Internet wide area RDMA protocol (iWARP) is a computer networking protocol
that implements RDMA for efficient data transfer over IP networks. iWARP is
designed for multiple environments, including LANs, storage networks, data
center networks, and WANs.
This chapter provides instructions for:

Configuring iWARP on Windows

Configuring iWARP on Linux
NOTE
Some iWARP features may not be fully enabled in the current release. For
details, refer to Appendix C Feature Constraints.
Configuring iWARP on Windows
This section provides procedures for configuring iWARP through HII, enabling
iWARP and verifying RDMA, and verifying iWARP traffic. For a list of OSs that
support iWARP, see Table 6-1 on page 53.
To configure iWARP through HII:
1.
Access the server BIOS System Setup, and then click Device Settings.
2.
On the Device Settings page, select a port for the 25G 41000 Series
Adapter.
3.
On the Main Configuration Page for the selected adapter, click NIC
Configuration.
4.
On the NIC Configuration page (Figure 7-1):
a.
Set the NIC + RDMA Mode to Enabled.
b.
Set the RDMA Protocol Support to iWARP.
76
AH0054601-00 A
7–iWARP Configuration
Configuring iWARP on Windows
c.
Click Back.
Figure 7-1. System Setup for iWARP: NIC Configuration
5.
On the Main Configuration Page, click Finish.
6.
In the Warning - Saving Changes message box, click Yes to save the
configuration.
7.
In the Success - Saving Changes message box, click OK.
8.
Repeat Step 2 through Step 7 to configure the NIC and DCB for the second
port.
9.
To complete adapter preparation of both ports:
a.
On the Device Settings page, click Finish.
b.
On the main menu, click Finish.
c.
Exit to reboot the system.
To enable iWARP and verify RDMA:
1.
To enable iWARP on the adapter:
a.
Open the Device Manager.
b.
Open the QLogic FastLinQ Adapter NDIS miniport properties.
c.
On the adapter properties, click the Advanced tab.
77
AH0054601-00 A
7–iWARP Configuration
Configuring iWARP on Windows
2.
d.
On the Advanced page under Property, select Network Direct
Functionality, and then select Enabled for the Value.
e.
Under Property, select RDMA Mode, and then select iWARP under
Value.
f.
Click OK to save your changes and close the adapter properties.
To verify that Network Direct Functionality is enabled, launch Windows
PowerShell, and then issue the Get-NetAdapterRdma command.
Figure 7-2 shows the Get-NetAdapterRdma command output listing
supported adapters.
Figure 7-2. Windows PowerShell Command: Get-NetAdapterRdma
3.
To verify that RDMA is enabled in the OS, launch Windows PowerShell, and
then issue the Get-NetOffloadGlobalSetting command.
Figure 7-3 shows the Get-NetOffloadGlobalSetting command output
showing NetworkDirect as enabled.
Figure 7-3. Windows PowerShell Command: Get-NetOffloadGlobalSetting
To verify iWARP traffic:
1.
Map SMB drives and run iWARP traffic.
2.
Launch Performance Monitor (Perfmon).
3.
In the Add Counters dialog box, click RDMA Activity, and then select the
adapter instances.
78
AH0054601-00 A
7–iWARP Configuration
Configuring iWARP on Windows
Figure 7-4 shows an example.
Figure 7-4. Perfmon: Add Counters
79
AH0054601-00 A
7–iWARP Configuration
Configuring iWARP on Windows
If iWARP traffic is running, counters appear as shown in the Figure 7-5
example.
Figure 7-5. Perfmon: Verifying iWARP Traffic
4.
To verify the SMB connection:
a.
At a command prompt, issue the net use command as follows:
C:\Users\Administrator&gt; net use
New connections will be remembered.
Status
Local
Remote
Network
--------------------------------------------------------OK
F:
\\192.168.10.10\Share1
Microsoft Windows Network
The command completed successfully.
b.
Issue the net -xan command as follows, where Share1 is mapped
as an SMB share:
C:\Users\Administrator&gt; net -xan
Active NetworkDirect Connections, Listeners, ShareEndpoints
Mode
IfIndex Type
Local Address
Foreign Address
PID
Kernel
56 Connection 192.168.11.20:16159 192.168.11.10:445
0
Kernel
56 Connection 192.168.11.20:15903 192.168.11.10:445
0
Kernel
56 Connection 192.168.11.20:16159 192.168.11.10:445
0
Kernel
56 Connection 192.168.11.20:15903 192.168.11.10:445
0
Kernel
60 Listener
[fe80::e11d:9ab5:a47d:4f0a%56]:445 NA
Kernel
60 Listener
192.168.11.20:445
Kernel
60 Listener
[fe80::71ea:bdd2:ae41:b95f%60]:445 NA
80
NA
0
0
0
AH0054601-00 A
7–iWARP Configuration
Configuring iWARP on Linux
Kernel
60 Listener
192.168.11.20:16159 192.168.11.10:445
0
Configuring iWARP on Linux
QLogic 41000 Series Adapters support iWARP on the Linux Open Fabric
Enterprise Distributions (OFEDs) listed in Table 6-1 on page 53.
iWARP configuration on a Linux system includes the following:

Installing the Driver

Detecting the Device

Supported iWARP Applications

Running Perftest for iWARP

Configuring NFS-RDMA
Installing the Driver
To install the driver, configure iWARP by selecting iWARP mode through the HII.
Install the RDMA drivers as shown in Chapter 3 Driver Installation.
Detecting the Device
To detect the device:
1.
To verify whether RDMA devices are detected, view the dmesg logs:
# dmesg |grep qedr
[10500.191047] qedr 0000:04:00.0: registered qedr0
[10500.221726] qedr 0000:04:00.1: registered qedr1
2.
Issue the ibv_devinfo command, and then verify the transport type.
If the command is successful, each PCI function will show a separate
hca_id. For example (if checking the second port of the above dual-port
adapter):
[root@localhost ~]# ibv_devinfo -d qedr1
hca_id: qedr1
transport:
iWARP (1)
fw_ver:
8.14.7.0
node_guid:
020e:1eff:fec4:c06e
sys_image_guid:
020e:1eff:fec4:c06e
vendor_id:
0x1077
vendor_part_id:
5718
hw_ver:
0x0
phys_port_cnt:
1
81
AH0054601-00 A
7–iWARP Configuration
Configuring iWARP on Linux
port:
1
state:
PORT_ACTIVE (4)
max_mtu:
4096 (5)
active_mtu:
1024 (3)
sm_lid:
0
port_lid:
0
port_lmc:
0x00
link_layer:
Ethernet
Supported iWARP Applications
Linux-supported RDMA applications for iWARP include the following:

ibv_devinfo, ib_devices

ib_send_bw/lat, ib_write_bw/lat, ib_read_bw/lat, ib_atomic_bw/lat
For iWARP, all applications must use the RDMA communication manager
(rdma_cm) using the -R option.

rdma_server, rdma_client

rdma_xserver, rdma_xclient

rping

NFS over RDMA (NFSoRDMA)

iSER (for details, see Chapter 11 iSER Configuration)
Running Perftest for iWARP
All perftest tools are supported over the iWARP transport type. You must run the
tools using RDMA connection manager (with the -R option).
Example:
1.
On one server, issue the following command (using the second port in this
example):
# ib_send_bw -d qedr1 -F -R
2.
On one client, issue the following command (using the second port in this
example):
[root@localhost ~]# ib_send_bw -d qedr1 -F -R 192.168.11.3
---------------------------------------------------------------------------Send BW Test
Dual-port
: OFF
Device
Number of qps
: 1
Transport type : IW
Connection type : RC
: qedr1
Using SRQ
82
: OFF
AH0054601-00 A
7–iWARP Configuration
Configuring iWARP on Linux
TX depth
: 128
CQ Moderation
: 100
Mtu
: 1024[B]
Link type
: Ethernet
GID index
: 0
Max inline data : 0[B]
rdma_cm QPs
: ON
Data ex. method : rdma_cm
---------------------------------------------------------------------------local address: LID 0000 QPN 0x0192 PSN 0xcde932
GID: 00:14:30:196:192:110:00:00:00:00:00:00:00:00:00:00
remote address: LID 0000 QPN 0x0098 PSN 0x46fffc
GID: 00:14:30:196:195:62:00:00:00:00:00:00:00:00:00:00
---------------------------------------------------------------------------#bytes
#iterations
65536
1000
BW peak[MB/sec]
BW average[MB/sec]
2250.38
2250.36
MsgRate[Mpps]
0.036006
----------------------------------------------------------------------------
NOTE
For latency applications (send/write), if the perftest version is the latest (for
example, perftest-3.0-0.21.g21dc344.x86_64.rpm), use the supported
inline size value: 0-128.
Configuring NFS-RDMA
NFS-RDMA for iWARP includes both server and client configuration steps.
To configure the NFS server:
1.
In the /etc/exports file for the directories that you must export using
NFS-RDMA on the server, make the following entry:
/tmp/nfs-server *(fsid=0,async,insecure,no_root_squash)
Ensure that you use a different file system identification (FSID) for each
directory that you export.
2.
Load the svcrdma module as follows:
# modprobe svcrdma
3.
Start the NFS service without any errors:
# service nfs start
83
AH0054601-00 A
7–iWARP Configuration
Configuring iWARP on Linux
4.
Include the default RDMA port 20049 into this file as follows:
# echo rdma 20049 &gt; /proc/fs/nfsd/portlist
5.
To make local directories available for NFS clients to mount, issue the
exportfs command as follows:
# exportfs -v
To configure the NFS client:
NOTE
This procedure for NFS client configuration also applies to RoCE.
1.
Load the xprtrdma module as follows:
# modprobe xprtrdma
2.
Mount the NFS file system as appropriate for your version:
For NFS Version 3:
#mount -o rdma,port=20049 192.168.2.4:/tmp/nfs-server /tmp/nfs-client
For NFS Version 4:
#mount -t nfs4 -o rdma,port=20049 192.168.2.4:/ /tmp/nfs-client
NOTE
The default port for NFSoRDMA is 20049. However, any other port that
is aligned with the NFS client will also work.
3.
Verify that the file system is mounted by issuing the mount command.
Ensure that the RDMA port and file system versions are correct.
#mount |grep rdma
84
AH0054601-00 A
8
iSCSI Configuration
This chapter provides the following iSCSI configuration information:

iSCSI Boot

Configuring iSCSI Boot

Configuring the DHCP Server to Support iSCSI Boot

Configuring iSCSI Boot from SAN for SLES 12

iSCSI Offload in Windows Server

iSCSI Offload in Linux Environments

Differences from bnx2i

Configuring qedi.ko

Verifying iSCSI Interfaces in Linux

Open-iSCSI and Boot from SAN Considerations
NOTE
iSCSI hardware offload is supported only on the QL4146x adapters. Some
iSCSI features may not be fully enabled in the current release. For details,
refer to Appendix C Feature Constraints.
iSCSI Boot
QLogic 4xxxx Series gigabit Ethernet (GbE) adapters support iSCSI boot to
enable network boot of operating systems to diskless systems. iSCSI boot allows
a Windows, Linux, or VMware operating system to boot from an iSCSI target
machine located remotely over a standard IP network.
For both Windows and Linux operating systems, iSCSI boot can be configured to
boot with two distinctive methods:

iSCSI SW (also known as non-offload path with Microsoft/Open-iSCSI
initiator).

iSCSI HW (offload path with QLogic offload iSCSI driver). This option can be
set using Boot Mode, under port-level configuration.
85
AH0054601-00 A
8–iSCSI Configuration
iSCSI Boot
iSCSI Boot Setup
The iSCSI boot setup includes:

Selecting the Preferred iSCSI Boot Mode

Configuring the iSCSI Target

Configuring iSCSI Boot Parameters
Selecting the Preferred iSCSI Boot Mode
Boot mode option is listed under of the adapter, and the setting is port specific.
Refer to the OEM user manual for direction on accessing the device level
configuration menu under UEFI HII.
NOTE
Boot from SAN boot is supported only in UEFI, and not in legacy BIOS.
Configuring the iSCSI Target
Configuring the iSCSI target varies by target vendors. For information on
configuring the iSCSI target, refer to the documentation provided by the vendor.
To configure the iSCSI target:
1.
Select the appropriate procedure based on your iSCSI target, either:

Create an iSCSI target for targets such as SANBlaze&reg; or IET&reg;.

Create a vdisk or volume for targets such as EqualLogic&reg; or EMC&reg;.
2.
Create a virtual disk.
3.
Map the virtual disk to the iSCSI target created in Step 1.
4.
Associate an iSCSI initiator with the iSCSI target. Record the following
information:





5.
iSCSI target name
TCP port number
iSCSI Logical Unit Number (LUN)
initiator iSCSI qualified name (IQN)
CHAP authentication details
After configuring the iSCSI target, obtain the following:






Target IQN
Target IP address
Target TCP port number
Target LUN
Initiator IQN
CHAP ID and secret
86
AH0054601-00 A
8–iSCSI Configuration
iSCSI Boot
Configuring iSCSI Boot Parameters
Configure the QLogic iSCSI boot software for either static or dynamic
configuration. For configuration options available from the General Parameters
window, see Table 8-1, which lists parameters for both IPv4 and IPv6. Parameters
specific to either IPv4 or IPv6 are noted.
NOTE
The availability of the IPv6 iSCSI boot is platform- and device-dependent.
Table 8-1. Configuration Options
Option
Description
TCP/IP parameters via DHCP
This option is specific to IPv4. Controls whether the iSCSI boot
host software acquires the IP address information using DHCP
(Enabled) or use a static IP configuration (Disabled).
iSCSI parameters via DHCP
Controls whether the iSCSI boot host software acquires its
iSCSI target parameters using DHCP (Enabled) or through a
static configuration (Disabled). The static information is
entered on the iSCSI Initiator Parameters Configuration page.
CHAP Authentication
Controls whether the iSCSI boot host software uses CHAP
authentication when connecting to the iSCSI target. If CHAP
Authentication is enabled, configure the CHAP ID and
CHAP Secret on the iSCSI Initiator Parameters Configuration
page.
IP Version
This option is specific to IPv6. Toggles between IPv4 and
IPv6. All IP settings are lost if you switch from one protocol
version to another.
DHCP Request Timeout
Allows you to specify a maximum wait time in seconds for a
DHCP request, and response to complete.
Target Login Timeout
Allows you to specify a maximum wait time in seconds for the
initiator to complete target login.
DHCP Vendor ID
Controls how the iSCSI boot host software interprets the
Vendor Class ID field used during DHCP. If the Vendor
Class ID field in the DHCP offer packet matches the value in
the field, the iSCSI boot host software looks into the DHCP
Option 43 fields for the required iSCSI boot extensions. If
DHCP is disabled, this value does not need to be set.
87
AH0054601-00 A
8–iSCSI Configuration
iSCSI Boot
Adapter UEFI Boot Mode Configuration
To configure the boot mode:
1.
Restart the system.
2.
Press the OEM hotkey to enter the System setup or configuration menu.
This is also known as UEFI HII. For example, the HPE Gen 9 systems use
F9 as a hotkey to access the System Utilities menu at boot time (Figure 8-1).
NOTE
SAN boot is supported in UEFI environment only. Make sure the
system boot option is UEFI, and not legacy.
Figure 8-1. Systems Utilities at Boot Time
3.
In System HII, select the QLogic device (Figure 8-2). Refer to the OEM user
guide on accessing the PCI device configuration menu. For example, on a
HPE Gen 9 server, the System Utilities for QLogic devices are listed on the
System Configuration menu.
Figure 8-2. Configuration Utility
88
AH0054601-00 A
8–iSCSI Configuration
iSCSI Boot
4.
On the Main Configuration Page, select Port Level Configuration
(Figure 8-3), and then press ENTER.
Figure 8-3. Selecting Port Level Configuration
5.
On the Port Level Configuration page (Figure 8-4), select Boot Mode, and
then press ENTER to select one of the following iSCSI boot modes:


iSCSI (SW)
iSCSI (HW)
89
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
Figure 8-4. Port Level Configuration, Boot Mode
NOTE
The iSCSI (HW) option is not listed if the iSCSI Offload feature is
disabled at port level. If the preferred boot mode is iSCSI (HW), make
sure the iSCSI offload feature is enabled. Not all adapter versions
support iSCSI offload and iSCSI (HW) offloaded boot. Additionally, not
all operating systems support iSCSI (HW) offloaded boot; some
support only iSCSI (SW) boot.
6.
Proceed with one of the following configuration options:

“Static iSCSI Boot Configuration” on page 91

“Dynamic iSCSI Boot Configuration” on page 97
Configuring iSCSI Boot
iSCSI boot configuration options include:

Static iSCSI Boot Configuration

Dynamic iSCSI Boot Configuration

Enabling CHAP Authentication
90
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
Static iSCSI Boot Configuration
In a static configuration, you must enter data for the following:



System’s IP address
System’s initiator IQN
Target parameters (obtained in “Configuring the iSCSI Target” on page 86)
For information on configuration options, see Table 8-1 on page 87.
To configure the iSCSI boot parameters using static configuration:
1.
In the Device HII Main Configuration Page, select iSCSI Configuration
(Figure 8-5), and then press ENTER.
Figure 8-5. Selecting iSCSI Boot Configuration
91
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
2.
In the iSCSI Boot Configuration Menu, select iSCSI General Parameters
(Figure 8-6), and then press ENTER.
Figure 8-6. Selecting General Parameters
3.
On the iSCSI General Parameters page (Figure 8-7), press the UP ARROW
and DOWN ARROW keys to select a parameter, and then press the ENTER
key to select or input the following values:







TCP/IP Parameters via DHCP: Disabled
iSCSI Parameters via DHCP: Disabled
CHAP Authentication: As required
IP Version: As required (IPv4 or IPv6)
ARP Redirect: Not applicable for boot
DHCP Request Timeout: Default value or as required
Target Login Timeout: Default value or as required
Figure 8-7. iSCSI General Configuration
4.
Return to the iSCSI Boot Configuration Menu, and then press the ESC key.
92
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
5.
Select iSCSI Initiator Parameters (Figure 8-8), and then press ENTER.
Figure 8-8. Selecting iSCSI Initiator Parameters
6.
On the iSCSI Initiator Configuration page (Figure 8-9), select the following
parameters, and then type a value for each:

IPv4* Address

IPv4* Subnet Mask

IPv4* Default Gateway

IPv4* Primary DNS

IPv4* Secondary DNS

VLAN ID (Optional). iSCSI traffic on the network may be isolated in a
Layer 2 VLAN to segregate it from general traffic. If this is the case,
make the iS2CSI interface on the adapter a member of that VLAN by
setting this value.

iSCSI Name. Corresponds to the iSCSI initiator name to be used by
the client system.

CHAP ID

CHAP Secret
93
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
NOTE
Note the following for the preceding items with asterisks (*):
 The label will change to IPv6 or IPv4 (default) based on the IP
version set on the iSCSI General Parameters page (Figure 8-7 on
page 92).
 Carefully enter the IP address. There is no error-checking
performed against the IP address to check for duplicates, incorrect
segment, or network assignment.
Figure 8-9. iSCSI Initiator Configuration
7.
Return to the iSCSI Boot Configuration Menu, and then press ESC.
8.
Select iSCSI First Target Parameters (Figure 8-10), and then press
ENTER.
Figure 8-10. iSCSI First Target Parameters
94
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
9.
On the iSCSI First Target Configuration page, set the Connect option to
Enabled to the iSCSI target.
10.
Type values for the following parameters for the iSCSI target, and then press
ENTER:






IPv4* Address
TCP Port
Boot LUN
iSCSI Name
CHAP ID
CHAP Secret
NOTE
For the preceding parameters with an asterisk (*), the label will change
to IPv6 or IPv4 (default) based on IP version set on the iSCSI General
Parameters page, as shown in Figure 8-11.
Figure 8-11. iSCSI First Target Configuration
11.
Return to the iSCSI Boot Configuration page, and then press ESC.
95
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
12.
If you want configure a second iSCSI target device, select iSCSI Second
Target Parameters (Figure 8-12), and enter the parameter values as you
did in Step 10. Otherwise, proceed to Step 13.
Figure 8-12. iSCSI Second Target Configuration
13.
Press ESC once, and a second time to exit.
96
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
14.
Press the Y key to save changes, or follow the OEM guidelines to save the
device-level configuration. For example, in a HPE Gen 9 system, press Y, c
to confirm setting change (Figure 8-13).
Figure 8-13. Saving iSCSI Changes
15.
After all changes have been made, reboot the system to apply the changes
to the adapter’s running configuration.
Dynamic iSCSI Boot Configuration
In a dynamic configuration, ensure that the system’s IP address and target (or
initiator) information are provided by a DHCP server (see IPv4 and IPv6
configurations in “Configuring the DHCP Server to Support iSCSI Boot” on
page 100).
Any settings on the following parameters are ignored and do not need to be
cleared (with the exception of the initiator iSCSI name for IPv4, CHAP ID, and
CHAP secret for IPv6):

Initiator Parameters

First Target Parameters or Second Target Parameters
97
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
For information on configuration options, see Table 8-1 on page 87.
NOTE
When using a DHCP server, the DNS server entries are overwritten by the
values provided by the DHCP server. This override occurs even if the locally
provided values are valid and the DHCP server provides no DNS server
information. When the DHCP server provides no DNS server information,
both the primary and secondary DNS server values are set to 0.0.0.0.
When the Windows OS takes over, the Microsoft iSCSI initiator retrieves the
iSCSI initiator parameters and statically configures the appropriate
registries. It will overwrite whatever is configured. Because the DHCP
daemon runs in the Windows environment as a user process, all TCP/IP
parameters must be statically configured before the stack comes up in the
iSCSI boot environment.
If DHCP Option 17 is used, the target information is provided by the DHCP server
and the initiator iSCSI name is retrieved from the value programmed from the
Initiator Parameters window. If no value was selected, the controller defaults to
the following name:
iqn.1995-05.com.qlogic.&lt;11.22.33.44.55.66&gt;.iscsiboot
The string 11.22.33.44.55.66 corresponds to the controller’s MAC address. If
DHCP Option 43 (IPv4 only) is used, any settings on the following windows are
ignored and do not need to be cleared:

Initiator Parameters

First Target Parameters, or Second Target Parameters
To configure the iSCSI boot parameters using dynamic configuration:

On the iSCSI General Configuration page, set the following options, as
shown in Figure 8-14:








TCP/IP Parameters via DHCP: Enabled
iSCSI Parameters via DHCP: Enabled
CHAP Authentication: As required
IP Version: As required (IPv4 or IPv6)
ARP Redirect: Not applicable for boot
DHCP Request Timeout: Default value or as required
Target Login Timeout: Default value or as required
DHCP Vendor ID: As required
98
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot
Figure 8-14. iSCSI General Configuration
Enabling CHAP Authentication
Ensure that the CHAP authentication is enabled on the target.
To enable CHAP authentication:
1.
Go to the iSCSI General Configuration page.
2.
Set CHAP Authentication to Enabled.
3.
In the Initiator Parameters window, type values for the following:

CHAP ID (up to 255 characters)

CHAP Secret (if authentication is required; must be 12 to 16
characters in length)
4.
Press ESC to return to the iSCSI Boot configuration page.
5.
On the iSCSI Boot configuration page, select iSCSI First Target
Parameters.
6.
In the iSCSI First Target Parameters window, type values used when
configuring the iSCSI target:

CHAP ID (optional if two-way CHAP)

CHAP Secret (optional if two-way CHAP; must be 12 to 16 characters
in length or longer)
7.
Press ESC to return to the iSCSI Boot configuration page.
8.
Press ESC, and then select confirm Save Configuration.
99
AH0054601-00 A
8–iSCSI Configuration
Configuring the DHCP Server to Support iSCSI Boot
Configuring the DHCP Server to Support iSCSI
Boot
The DHCP server is an optional component, and is only necessary if you will be
doing a dynamic iSCSI boot configuration setup (see “Dynamic iSCSI Boot
Configuration” on page 97).
Configuring the DHCP server to support iSCSI boot differs for IPv4 and IPv6:

DHCP iSCSI Boot Configurations for IPv4

Configuring DHCP iSCSI Boot for IPv6
DHCP iSCSI Boot Configurations for IPv4
DHCP includes a several options that provide configuration information to the
DHCP client. For iSCSI boot, QLogic adapters support the following DHCP
configurations:

DHCP Option 17, Root Path

DHCP Option 43, Vendor-specific Information
DHCP Option 17, Root Path
Option 17 is used to pass the iSCSI target information to the iSCSI client.
The format of the root path, as defined in IETC RFC 4173, is:
&quot;iscsi:&quot;&lt;servername&gt;&quot;:&quot;&lt;protocol&gt;&quot;:&quot;&lt;port&gt;&quot;:&quot;&lt;LUN&gt;&quot;:&quot;&lt;targetname&gt;&quot;
Table 8-2 lists the DHCP Option 17 parameters.
Table 8-2. DHCP Option 17 Parameter Definitions
Parameter
Definition
&quot;iscsi:&quot;
A literal string
&lt;servername&gt;
IP address or fully qualified domain name (FQDN) of the iSCSI target
&quot;:&quot;
Separator
&lt;protocol&gt;
IP protocol used to access the iSCSI target. Because only TCP is
currently supported, the protocol is 6.
&lt;port&gt;
Port number associated with the protocol. The standard port number for iSCSI is 3260.
&lt;LUN&gt;
Logical unit number to use on the iSCSI target. The value of the
LUN must be represented in hexadecimal format. A LUN with an
ID OF 64 must be configured as 40 within the Option 17 parameter
on the DHCP server.
100
AH0054601-00 A
8–iSCSI Configuration
Configuring the DHCP Server to Support iSCSI Boot
Table 8-2. DHCP Option 17 Parameter Definitions (Continued)
Parameter
&lt;targetname&gt;
Definition
Target name in either IQN or EUI format. For details on both IQN
and EUI formats, refer to RFC 3720. An example IQN name is
iqn.1995-05.com.QLogic:iscsi-target.
DHCP Option 43, Vendor-specific Information
DHCP Option 43 (vendor-specific information) provides more configuration
options to the iSCSI client than does DHCP Option 17. In this configuration, three
additional sub-options are provided that assign the initiator IQN to the iSCSI boot
client, along with two iSCSI target IQNs that can be used for booting. The format
for the iSCSI target IQN is the same as that of DHCP Option 17, while the iSCSI
initiator IQN is simply the initiator’s IQN.
NOTE
DHCP Option 43 is supported on IPv4 only.
Table 8-3 lists the DHCP Option 43 sub-options.
Table 8-3. DHCP Option 43 Sub-option Definitions
Sub-option
201
Definition
First iSCSI target information in the standard root path format:
&quot;iscsi:&quot;&lt;servername&gt;&quot;:&quot;&lt;protocol&gt;&quot;:&quot;&lt;port&gt;&quot;:&quot;&lt;LUN&gt;&quot;:
&quot;&lt;targetname&gt;&quot;
202
Second iSCSI target information in the standard root path format:
&quot;iscsi:”&lt;servername&gt;&quot;:&quot;&lt;protocol&gt;&quot;:&quot;&lt;port&gt;&quot;:&quot;&lt;LUN&gt;&quot;:
&quot;&lt;targetname&gt;&quot;
203
iSCSI initiator IQN
Using DHCP Option 43 requires more configuration than DHCP Option 17, but it
provides a richer environment and more configuration options. QLogic
recommends that customers use DHCP Option 43 when performing dynamic
iSCSI boot configuration.
101
AH0054601-00 A
8–iSCSI Configuration
Configuring the DHCP Server to Support iSCSI Boot
Configuring the DHCP Server
Configure the DHCP server to support either Option 16, 17, 43.
NOTE
The format of DHCPv6 Option 16 and Option 17 are fully defined in
RFC 3315.
If you use Option 43, you must also configure Option 60. The value of Option
60 must match the DHCP Vendor ID value, QLGC ISAN, as shown in iSCSI
General Parameters of the iSCSI Boot Configuration page.
Configuring DHCP iSCSI Boot for IPv6
The DHCPv6 server can provide several options, including stateless or stateful IP
configuration, as well as information for the DHCPv6 client. For iSCSI boot,
QLogic adapters support the following DHCP configurations:

DHCPv6 Option 16, Vendor Class Option

DHCPv6 Option 17, Vendor-Specific Information
NOTE
The DHCPv6 standard Root Path option is not yet available. QLogic
suggests using Option 16 or Option 17 for dynamic iSCSI boot IPv6 support.
DHCPv6 Option 16, Vendor Class Option
DHCPv6 Option 16 (vendor class option) must be present and must contain a
string that matches your configured DHCP Vendor ID parameter. The DHCP
Vendor ID value is QLGC ISAN, as shown in the General Parameters of the
iSCSI Boot Configuration menu.
The content of Option 16 should be &lt;2-byte length&gt; &lt;DHCP Vendor ID&gt;.
DHCPv6 Option 17, Vendor-Specific Information
DHCPv6 Option 17 (vendor-specific information) provides more configuration
options to the iSCSI client. In this configuration, three additional sub-options are
provided that assign the initiator IQN to the iSCSI boot client, along with two iSCSI
target IQNs that can be used for booting.
102
AH0054601-00 A
8–iSCSI Configuration
Configuring the DHCP Server to Support iSCSI Boot
Table 8-4 lists the DHCP Option 17 sub-options.
Table 8-4. DHCP Option 17 Sub-option Definitions
Sub-option
201
Definition
First iSCSI target information in the standard root path format:
&quot;iscsi:&quot;[&lt;servername&gt;]&quot;:&quot;&lt;protocol&gt;&quot;:&quot;&lt;port&gt;&quot;:&quot;&lt;LUN&gt; &quot;:
&quot;&lt;targetname&gt;&quot;
202
Second iSCSI target information in the standard root path format:
&quot;iscsi:&quot;[&lt;servername&gt;]&quot;:&quot;&lt;protocol&gt;&quot;:&quot;&lt;port&gt;&quot;:&quot;&lt;LUN&gt; &quot;:
&quot;&lt;targetname&gt;&quot;
203
iSCSI initiator IQN
Table Notes:
Brackets [ ] are required for the IPv6 addresses.
The content of Option 17 should be:
&lt;2-byte Option Number 201|202|203&gt; &lt;2-byte length&gt; &lt;data&gt;
Configuring VLANs for iSCSI Boot
iSCSI traffic on the network may be isolated in a Layer 2 VLAN to segregate it
from general traffic. If this is the case, make the iSCSI interface on the adapter a
member of that VLAN.
To configure VLAN for iSCSI boot:
1.
Go to the iSCSI Boot Configuration Menu for the port.
2.
Select iSCSI Initiator Configuration.
103
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot from SAN for SLES 12
3.
Select VLAN ID to enter and set the VLAN value, as shown in Figure 8-15.
Figure 8-15. iSCSI Initiator Configuration, VLAN ID
Configuring iSCSI Boot from SAN for SLES 12
Perform L2 to L4 iSCSI boot from SAN through Microsoft Multipath I/O (MPIO) on
FastLinQ 41000 Series Adapters for SLES 12 SP1 on UEFI-based systems.
To perform iSCSI boot from SAN:
1.
Configure adapter ports for L2 iSCSI Boot Firmware Table (iBFT)
configuration as follows:
a.
Open System Configuration, select the adapter port, and then select
Port Level Configuration.
b.
On the Port Level Configuration page, set the Boot Mode to iSCSI
(SW) and set the iSCSI Offload to Disabled, as shown in Figure 8-16.
Figure 8-16. System Configuration: Setting Boot Mode
104
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot from SAN for SLES 12
c.
On the iSCSI Boot Configuration Menu, select iSCSI Initiator
Parameters.
d.
On the iSCSI Initiator Configuration page, configure the parameters as
shown in Figure 8-17.
Figure 8-17. System Configuration: Setting Boot Mode
e.
On the iSCSI Boot Configuration Menu, select iSCSI General
Parameters.
f.
On the iSCSI General Configuration page, set the TC/IP Parameters
via DHCP to either Disabled for static configurations as shown in
Figure 8-18, or Enabled for DHCP configuration.
Figure 8-18. System Configuration: Setting DHCP
105
AH0054601-00 A
8–iSCSI Configuration
Configuring iSCSI Boot from SAN for SLES 12
g.
On the iSCSI Boot Configuration Menu, select iSCSI First Target
Parameters.
h.
On the iSCSI First Target Configuration page, configure the following
parameters:





i.
2.
Connect: Set to Enabled
IPv4 Address
TCP Port
Boot LUN
iSCSI Name
Save the settings and reboot the server.
Mount the OS ISO and proceed to install SL12-SP1 OS on iSCSI disk over
L2. At the beginning of the installation, pass the following boot parameter to
inject latest drivers:
dud=1
3.
To complete L2 BFS, follow the on-screen instructions.
4.
When installation is complete, boot into the OS using IBFT over L2.
5.
To migrate from L2 to L4 and configure MPIO settings to boot the OS over
offloaded interface, follow these steps:
a.
To update open-iscsi tools, issue the following command:
#rpm -ivh qlgc-open-iscsi-2.0_873.109-1.x86_64.rpm
--force
b.
Go to /etc/default/grub and change the rd.iscsi.ibft
parameter to rd.iscsi.firmware.
c.
Issue the following command:
grub2-mkconfig -o /boot/efi/EFI/suse/grub.cfg
d.
To load the multipath module, issue the following command:
modprobe dm_multipath
e.
To enable the multipath daemon, issue the following commands:
systemctl start multipathd.service
systemctl enable multipathd.service
systemctl start multipathd.socket
f.
To add devices to the multipath, issue the following commands:
multipath -a /dev/sda
multipath -a /dev/sdb
106
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
g.
To run the multipath utility, issue the following commands:
multipath (may not show the multipath devices because it is booted with
single path on L2)
multipath -ll
h.
To Inject the multipath module in initrd, issue the following command:
dracut --force --add multipath --include /etc/multipath
i.
6.
Reboot the server and enter system settings by pressing the F9 key
during the POST menu.
Change UEFI configuration to use L4 iSCSI boot:
a.
Open System Configuration, select the adapter port, and then select
Port Level Configuration.
b.
On the Port Level Configuration page (Figure 8-19):


Set the Boot Mode to iSCSI (HW).
Set the iSCSI Offload to Enabled.
Figure 8-19. System Configuration: Setting DHCP
c.
Save the setting, exit the System Setup Menu, and then reboot.
The OS should now boot through the offload interface.
iSCSI Offload in Windows Server
iSCSI offload is a technology that offloads iSCSI protocol processing overhead
from host processors to the iSCSI HBA. iSCSI offload increases network
performance and throughput while helping to optimize server processor use. This
section covers how to configure the Windows iSCSI offload feature for the QLogic
FastLinQ 41000 Series Adapters.
107
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
With the proper iSCSI offload licensing, you can configure your iSCSI-capable
FastLinQ 41000 Series Adapter to offload iSCSI processing from the host
processor. The following sections describe how to enable the system to take
advantage of QLogic’s iSCSI offload feature:

Installing QLogic Drivers

Installing the Microsoft iSCSI Initiator

Configuring Microsoft Initiator to Use QLogic’s iSCSI Offload

iSCSI Offload FAQs

Windows Server 2012, 2012 R2, and 2016 iSCSI Boot Installation

iSCSI Crash Dump
Installing QLogic Drivers
Install the Windows drivers as described in “Installing Windows Driver Software”
on page 19.
Installing the Microsoft iSCSI Initiator
Launch the Microsoft iSCSI initiator applet. At the first launch, the system prompts
for an automatic service start. Confirm the selection for the applet to launch.
Configuring Microsoft Initiator to Use QLogic’s iSCSI Offload
After the IP address is configured for the iSCSI adapter, you must use Microsoft
Initiator to configure and add a connection to the iSCSI target using the QLogic
iSCSI adapter. For more details on Microsoft Initiator, see the Microsoft user
guide.
To configure the Microsoft Initiator:
1.
Open Microsoft Initiator.
2.
To configure the initiator IQN name according to your setup, follow these
steps:
a.
On the iSCSI Initiator Properties, click the Configuration tab.
b.
On the Configuration page (Figure 8-20), click Change to modify the
initiator name.
108
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
Figure 8-20. iSCSI Initiator Properties, Configuration Page
c.
In the iSCSI Initiator Name dialog box, type the new initiator IQN
name, and then click OK. (Figure 8-21)
Figure 8-21. iSCSI Initiator Node Name Change
3.
On the iSCSI Initiator Properties, click the Discovery tab.
109
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
4.
On the Discovery page (Figure 8-22) under Target portals, click Discover
Portal.
Figure 8-22. iSCSI Initiator—Discover Target Portal
110
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
5.
In the Discover Target Portal dialog box (Figure 8-23):
a.
In the IP address or DNS name box, type the IP address of the target.
b.
Click Advanced.
Figure 8-23. Target Portal IP Address
6.
In the Advanced Settings dialog box (Figure 8-24), complete the following
under Connect using:
a.
For Local adapter, select the QLogic &lt;name or model&gt; Adapter.
b.
For Initiator IP, select the adapter IP address.
c.
Click OK.
111
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
Figure 8-24. Selecting the Initiator IP Address
7.
On the iSCSI Initiator Properties, Discovery page, click OK.
112
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
8.
Click the Targets tab, and then on the Targets page (Figure 8-25), click
Connect.
Figure 8-25. Connecting to the iSCSI Target
113
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
9.
On the Connect To Target dialog box (Figure 8-26), click Advanced.
Figure 8-26. Connect To Target Dialog Box
10.
In the Local Adapter dialog box, select the QLogic &lt;name or model&gt;
Adapter, and then click OK.
11.
Click OK again to close the Microsoft Initiator.
12.
To format the iSCSI partition, use Disk Manager.
NOTE
Some limitations of the teaming functionality include:
 Teaming does not support iSCSI adapters.
 Teaming does not support NDIS adapters that are in the boot path.
 Teaming supports NDIS adapters that are not in the iSCSI boot
path, but only for the SLB team type.
iSCSI Offload FAQs
Some of the frequently asked questions about iSCSI offload include:
Question: How do I assign an IP address for iSCSI offload?
Answer:
Use the Configurations page in QConvergeConsole GUI.
Question: What tools should be used to create the connection to the target?
Answer:
Use Microsoft iSCSI Software Initiator (version 2.08 or later).
Question: How do I know that the connection is offloaded?
Answer:
Use Microsoft iSCSI Software Initiator. From a command line, type
oiscsicli sessionlist. From Initiator Name, an iSCSI offloaded
connection will display an entry beginning with B06BDRV. A
non-offloaded connection displays an entry beginning with Root.
114
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Windows Server
Question: What configurations should be avoided?
Answer:
The IP address should not be the same as the LAN.
Windows Server 2012, 2012 R2, and 2016 iSCSI Boot
Installation
Windows Server 2012, 2012 R2, and 2016 support booting and installing in either
the offload or non-offload paths. QLogic requires that you use a slipstream DVD
with the latest QLogic drivers injected. See “Injecting (Slipstreaming) Adapter
Drivers into Windows Image Files” on page 131.
The following procedure prepares the image for installation and booting in either
the offload or non-offload path.
To set up Windows Server 2012/2012R2/2016 iSCSI boot:
1.
Remove any local hard drives on the system to be booted (remote system).
2.
Prepare the Windows OS installation media by following the slipstreaming
steps in “Injecting (Slipstreaming) Adapter Drivers into Windows Image
Files” on page 131.
3.
Load the latest QLogic iSCSI boot images into the NVRAM of the adapter.
4.
Configure the iSCSI target to allow a connection from the remote device.
Ensure that the target has sufficient disk space to hold the new OS
installation.
5.
Configure the UEFI HII to set the iSCSI boot type (offload or non-offload),
correct initiator, and target parameters for iSCSI boot.
6.
Save the settings and reboot the system. The remote system should
connect to the iSCSI target and then boot from the DVD-ROM device.
7.
Boot from DVD and begin installation.
8.
Follow the on-screen instructions.
At the window that shows the list of disks available for the installation, the
iSCSI target disk should be visible. This target is a disk connected through
the iSCSI boot protocol and located in the remote iSCSI target.
9.
To proceed with Windows Server 2012/2012R2/2016 installation, click Next,
and then follow the on-screen instructions. The server will undergo a reboot
multiple times as part of the installation process.
10.
After the server boots to the OS, QLogic recommends running the driver
installer to complete the QLogic drivers and application installation.
115
AH0054601-00 A
8–iSCSI Configuration
iSCSI Offload in Linux Environments
iSCSI Crash Dump
Crash dump functionality is currently supported only for offload iSCSI boot for the
FastLinQ 41000 Series Adapters. No additional configurations are required to
configure iSCSI crash dump generation when in offload iSCSI boot mode.
iSCSI Offload in Linux Environments
The QLogic FastLinQ 41000 Series iSCSI software consists of a single kernel
module called qedi.ko (qedi). The qedi module is dependent on additional parts of
the Linux kernel for specific functionality:

qed.ko is the Linux eCore kernel module used for common QLogic
FastLinQ 41000 Series hardware initialization routines.

scsi_transport_iscsi.ko is the Linux iSCSI transport library used for
upcall and downcall for session management.

libiscsi.ko is the Linux iSCSI library function needed for protocol data
unit (PDU) and task processing, as well as session memory management.

iscsi_boot_sysfs.ko is the Linux iSCSI sysfs interface that provides
helpers to export iSCSI boot information.

uio.ko is the Linux Userspace I/O interface, used for light L2 memory
mapping for iscsiuio.
These modules must be loaded before qedi can be functional. Otherwise, you
might encounter an “unresolved symbol” error. If the qedi module is installed in the
distribution update path, the requisite is automatically loaded by modprobe.
Differences from bnx2i
Some key differences exist between qedi—the driver for the QLogic FastLinQ
41000 Series Adapter (iSCSI)—and the previous QLogic iSCSI offload
driver—bnx2i for the QLogic 8400 Series Adapters. Some of these differences
include:

qedi directly binds to a PCI function exposed by the CNA.

qedi does not sit on top of the net_device.

qedi is not dependent on a network driver such as bnx2x and cnic.

qedi is not dependent on cnic, but it has dependency on qed.

qedi is responsible for exporting boot information in sysfs using
iscsi_boot_sysfs.ko, whereas bnx2i boot from SAN relies on the
iscsi_ibft.ko module for exporting boot information.
116
AH0054601-00 A
8–iSCSI Configuration
Configuring qedi.ko
Configuring qedi.ko
The qedi driver automatically binds to the exposed iSCSI functions of the CNA,
and the target discovery and binding is done through the open-iscsi tools. This
functionality and operation is similar to that of the bnx2i driver.
NOTE
For more information on how to install FastLinQ drivers, see Chapter 3
Driver Installation.
To load the qedi.ko kernel module, issue the following commands:
# modprobe qed
# modprobe libiscsi
# modprobe uio
# modprobe iscsi_boot_sysfs
# modprobe qedi
Verifying iSCSI Interfaces in Linux
After installing and loading the qedi kernel module, you must verify that the iSCSI
interfaces were detected correctly.
To verify iSCSI interfaces in Linux:
1.
To verify that the qedi and associated kernel modules are actively loaded,
issue the following command:
# lsmod | grep qedi
2.
qedi
114578
2
qed
697989
1 qedi
uio
19259
4 cnic,qedi
libiscsi
57233
2 qedi,bnx2i
scsi_transport_iscsi
99909
5 qedi,bnx2i,libiscsi
iscsi_boot_sysfs
16000
1 qedi
To verify that the iSCSI interfaces were detected properly, issue the following
command. In this example, two iSCSI CNA devices are detected with SCSI
host numbers 4 and 5.
# dmesg | grep qedi
[0000:00:00.0]:[qedi_init:3696]: QLogic iSCSI Offload Driver v8.15.6.0.
....
[0000:42:00.4]:[__qedi_probe:3563]:59: QLogic FastLinQ iSCSI Module qedi
8.15.6.0, FW 8.15.3.0
117
AH0054601-00 A
8–iSCSI Configuration
Verifying iSCSI Interfaces in Linux
....
[0000:42:00.4]:[qedi_link_update:928]:59: Link Up event.
....
[0000:42:00.5]:[__qedi_probe:3563]:60: QLogic FastLinQ iSCSI Module qedi
8.15.6.0, FW 8.15.3.0
....
[0000:42:00.5]:[qedi_link_update:928]:59: Link Up event
3.
Use open-iscsi tools to verify that IP is configured properly. Issue the
following command:
# iscsiadm -m iface | grep qedi
qedi.00:0e:1e:c4:e1:6d
qedi,00:0e:1e:c4:e1:6d,192.168.101.227,&lt;empty&gt;,iqn.1994-05.com.redhat:534ca9b6
adf
qedi.00:0e:1e:c4:e1:6c
qedi,00:0e:1e:c4:e1:6c,192.168.25.91,&lt;empty&gt;,iqn.1994-05.com.redhat:534ca9b6ad
f
4.
To ensure that the iscsiuio service is running, issue the following command:
# systemctl status iscsiuio.service
iscsiuio.service - iSCSI UserSpace I/O driver
Loaded: loaded (/usr/lib/systemd/system/iscsiuio.service; disabled; vendor
preset: disabled)
Active: active (running) since Fri 2017-01-27 16:33:58 IST; 6 days ago
Docs: man:iscsiuio(8)
Process: 3745
ExecStart=/usr/sbin/iscsiuio (code=exited, status=0/SUCCESS)
Main PID: 3747 (iscsiuio)
CGroup: /system.slice/iscsiuio.service !--3747 /usr/sbin/iscsiuio
Jan 27 16:33:58 localhost.localdomain systemd[1]: Starting iSCSI
UserSpace I/O driver...
Jan 27 16:33:58 localhost.localdomain systemd[1]: Started iSCSI UserSpace
I/O driver.
5.
To discover the iSCSI target, issue the iscsiadm command:
#iscsiadm -m discovery -t st -p 192.168.25.100 -I qedi.00:0e:1e:c4:e1:6c
192.168.25.100:3260,1 iqn.200304.com.sanblaze:virtualun.virtualun.target-05000007
192.168.25.100:3260,1
iqn.2003-04.com.sanblaze:virtualun.virtualun.target-05000012
192.168.25.100:3260,1
iqn.2003-04.com.sanblaze:virtualun.virtualun.target-0500000c
192.168.25.100:3260,1 iqn.200304.com.sanblaze:virtualun.virtualun.target-05000001
118
AH0054601-00 A
8–iSCSI Configuration
Verifying iSCSI Interfaces in Linux
192.168.25.100:3260,1
iqn.2003-04.com.sanblaze:virtualun.virtualun.target-05000002
6.
Log into the iSCSI target using the IQN obtained in Step 5. To initiate the
login procedure, issue the following command (where the last character in
the command is a lowercase letter “L”:
#iscsiadm -m node -p 192.168.25.100 -T
iqn.2003-04.com.sanblaze:virtualun.virtualun.target-0)000007 -l
Logging in to [iface: qedi.00:0e:1e:c4:e1:6c,
target:iqn.2003-04.com.sanblaze:virtualun.virtualun.target-05000007,
portal:192.168.25.100,3260] (multiple)
Login to [iface: qedi.00:0e:1e:c4:e1:6c, target:iqn.200304.com.sanblaze:virtualun.virtualun.target-05000007,
portal:192.168.25.100,3260] successful.
7.
To verify that the iSCSI session was created, issue the following command:
# iscsiadm -m session
qedi: [297] 192.168.25.100:3260,1
iqn.2003-04.com.sanblaze:virtualun.virtualun.target-05000007 (non-flash)
8.
To check for iSCSI devices, issue the iscsiadm command:
# iscsiadm -m session -P3
...
************************
Attached SCSI devices:
************************
Host Number: 59 State: running
scsi59 Channel 00 Id 0 Lun: 0
Attached scsi disk sdb
State: running scsi59 Channel 00 Id 0 Lun: 1
Attached scsi disk sdc
State: running scsi59 Channel 00 Id 0 Lun: 2
Attached scsi disk sdd
State: running scsi59 Channel 00 Id 0 Lun: 3
Attached scsi disk sde
State: running scsi59 Channel 00 Id 0 Lun: 4
Attached scsi disk sdf
State: running
For advanced target configurations, refer to the Open-iSCSI README at:
https://github.com/open-iscsi/open-iscsi/blob/master/README
119
AH0054601-00 A
8–iSCSI Configuration
Open-iSCSI and Boot from SAN Considerations
Open-iSCSI and Boot from SAN Considerations
In current distributions (for example, RHEL 6/7 and SLE 11/12) the inbox iSCSI
user space utility (Open-iSCSI tools) lacks support for qedi iSCSI transport and
cannot perform user space-initiated iSCSI functionality. During boot from SAN
installation, you can update the qedi driver using a driver update disk (DUD).
However, no interface or process exists to update userspace inbox utilities, which
causes the iSCSI target login and boot from SAN installation to fail.
To overcome this limitation, perform the initial boot from SAN with the pure L2
interface (do not use hardware offloaded iSCSI) using one of the following
procedures during the boot from SAN.
To boot from SAN using a software initiator:
1.
Complete the following in the adapter's preboot device configuration (iSCSI
Boot Configuration Menu):
a.
On all ports, set iSCSI Offload to Disable.
b.
On all ports, set the HBA Mode (which uses the iSCSI offload boot
pathway) to Disabled.
c.
Set the Boot Mode to iSCSI.
NOTE
The preceding step is required because DUDs contain qedi, which
binds to the iSCSI PF. After it is bound, open-iscsi infrastructure fails
due to unknown transport driver.
2.
Configure the initiator and target entries.
3.
At the beginning of the installation, pass the following boot parameter with
the DUD option:

For RHEL 7.x and SLES 12.0:
rd.iscsi.ibft
No separate options are required for older distributions of RHEL and
SLES.

For the FastLinQ DUD package (for example, on RHEL 7):
fastlinq-8.18.10.0-dd-rhel7u3-3.10.0_514.el7-x86_64.iso
Where the DUD parameter is dd for RHEL 7.x and dud=1 for
SLES 12.x.
4.
Install the OS on the target LUN.
120
AH0054601-00 A
8–iSCSI Configuration
Open-iSCSI and Boot from SAN Considerations
To migrate from a non-offload interface to an offload interface:
1.
Upgrade qedi transport-supported Open-iSCSI tools such as iscsiadm,
iscsid, iscsiuio, and iscsistart.
a.
Use the following Open-iSCSI RPM package:
qlgc-open-iscsi-2.0_873.107-1.x86_64.rpm
b.
Issue the following command:
rpm -ivh qlgc-open-iscsi-2.0_873.107-1.x86_64.rpm –force
c.
Enable iscsid and iscsiuio sockets and services as follows:
#systemctl enable iscsid.socket
#systemctl enable iscsid
#systemctl enable iscsiuio.socket
#systemctl enable iscsiuio
2.
Issue the following command:
cat /proc/cmdline
3.
Check if the OS has preserved any boot options, such as ip=ibft or
rd.iscsi.ibft.


4.
If there are preserved boot options, continue with Step 4.
If there are no preserved boot options, skip to Step 4 c.
Edit the /etc/default/grub file and modify the GRUB_CMDLINE_LINUX
value:
a.
Remove rd.iscsi.ibft (if present).
b.
Remove any ip=&lt;value&gt; boot option. (if present).
c.
For SLES 12.x and RHEL 7.x, replace iscsi_firmware with
rd.iscsi.firmware.
d.
5.
If the iscsi_firmware or rd.iscsi.firmare boot option is not
present, complete one of the following:

For RHEL 7.x and SLES 12.x, add rd.scsi.firmware.

For earlier versions of RHEL and SLES, add iscsi_firmware.
Create a backup of the original grub.cfg file, which is in the following
locations:

For legacy boot:
/boot/grub2/grub.cfg

For UEFI boot:
/boot/efi/EFI/redhat/grub.cfg or /boot/grub2/grub.cfg
121
AH0054601-00 A
8–iSCSI Configuration
Open-iSCSI and Boot from SAN Considerations
NOTE
Step 6 and Step 7 describe how to replace the correct grub.cfg file.
6.
Create a new grub.cfg by issuing the following command:
grub2-mkconfig -o &lt;name of new grub.cfg&gt;
7.
Compare the old grub.cfg file with the new grub.cfg file to verify that
your changes have been made.
8.
Replace the original grub.cfg file with the new grub.cfg file.
9.
Create a new initramfs image by issuing the following command:
dracut –-force
10.
On the adapter’s preboot iSCSI Boot Configuration Menu, change the value
of the iSCSI (offload):
a.
On the iSCSI Boot Configuration Menu, set iSCSI Offload to Enable.
b.
Set HBA Mode to Enable.
NOTE
The OS can now boot through the offload interface.
122
AH0054601-00 A
9
FCoE Configuration
This chapter provides the following Fibre Channel over Ethernet (FCoE)
configuration information:

FCoE Boot from SAN

Injecting (Slipstreaming) Adapter Drivers into Windows Image Files

Configuring Linux FCoE Offload

Differences Between qedf and bnx2fc

Configuring qedf.ko

Verifying FCoE Devices in Linux

Boot from SAN Considerations
NOTE
FCoE offload is supported only on the QL4546x adapters. Some FCoE
features may not be fully enabled in the current release.For details, refer to
Appendix C Feature Constraints.
FCoE Boot from SAN
This section describes the installation and boot procedures for the Windows,
Linux, and ESXi operating systems, including:

Preparing System BIOS for FCoE Build and Boot

Windows FCoE Boot from SAN
NOTE
FCoE Boot from SAN is not supported on ESXi 5.0 and 5.1. ESXi Boot from
SAN is supported on ESXi 5.5 and later. Not all adapter versions support
FCoE and FCoE Boot from SAN.
123
AH0054601-00 A
9–FCoE Configuration
FCoE Boot from SAN
Preparing System BIOS for FCoE Build and Boot
To prepare the system BIOS, modify the system boot order and specify the BIOS
boot protocol, if required.
Specifying the BIOS Boot Protocol
FCoE boot from SAN is supported in UEFI mode only. Set the platform in boot
mode (protocol) using the system BIOS configuration to UEFI.
NOTE
FCoE BFS is not supported in legacy BIOS mode.
Configuring Adapter UEFI Boot Mode
To configure the boot mode to FCOE:
1.
Restart the system.
2.
Press the OEM hot key to enter System Setup or the configuration menu
(Figure 9-1). This is also known as UEFI HII. For example, the HPE Gen 9
system have F9 as a hotkey to access the System Utilities menu at boot
time.
Figure 9-1. System Utilities
NOTE
SAN boot is supported in UEFI environment only. Make sure the
system boot option is UEFI, and not legacy.
124
AH0054601-00 A
9–FCoE Configuration
FCoE Boot from SAN
3.
In System HII, select the QLogic device (Figure 9-2). Refer to the OEM user
guide on accessing PCI device configuration menu. For example, on an
HPE Gen 9 server, the System Utilities for QLogic devices are listed under
the System Configuration menu.
Figure 9-2. System Configuration, Port Selection
4.
On the Main Configuration Page, select Port Level Configuration
(Figure 9-3), and then press ENTER.
Figure 9-3. Port Level Configuration
125
AH0054601-00 A
9–FCoE Configuration
FCoE Boot from SAN
5.
On the Port Level Configuration Page (Figure 9-4) page, select Boot Mode,
and then press ENTER to select FCoE as a preferred boot mode.
Figure 9-4. Boot Mode in Port Level Configuration
126
AH0054601-00 A
9–FCoE Configuration
FCoE Boot from SAN
NOTE
FCoE is not listed as a boot option if the FCoE Offload feature is disabled at
the port level. If the Boot Mode preferred is FCoE, make sure the FCoE
Offload feature is enabled as shown in Figure 9-5. Not all adapter versions
support FCoE.
Figure 9-5. FCoE Offload Enabled
To configure the FCoE boot parameters:
1.
On the Device HII Main Configuration Page, select FCoE Configuration,
and then press ENTER.
2.
In the FCoE Boot Configuration Menu, select FCoE General Parameters
(Figure 9-6), and then press ENTER.
Figure 9-6. Selecting General Parameters
127
AH0054601-00 A
9–FCoE Configuration
FCoE Boot from SAN
3.
In the FCoE General Parameters menu (Figure 9-7), press the UP ARROW
and DOWN ARROW keys to select a parameter, and then press ENTER to
select and input the following values:

FIP VLAN ID: As required (if not set, adapter will attempt FIP VLAN
discovery)

Fabric Login Retry Count: Default value or as required

Target Login Retry Count: Default value or as required
Figure 9-7. FCoE General Parameters
4.
Return to the FCoE Boot Configuration page.
5.
Press ESC, and then select FCoE Target Parameters.
6.
Press ENTER.
7.
In the FCoE Target Parameters Menu, enable Connect to the preferred
FCoE target.
8.
Type values for the following parameters (Figure 9-8) for the iSCSI target,
and then press ENTER:


WWPN n
Boot LUN n
Where the value of n is between 1 and 8, enabling you to configure 8
FCoE targets.
128
AH0054601-00 A
9–FCoE Configuration
FCoE Boot from SAN
Figure 9-8. FCoE Target Configuration
Windows FCoE Boot from SAN
FCoE boot from SAN information for Windows includes:

Windows Server 2012, 2012 R2, and 2016 FCoE Boot Installation

Configuring FCoE

FCoE Crash Dump
Windows Server 2012, 2012 R2, and 2016 FCoE Boot Installation
For Windows Server 2012/2012R2/2016 boot from SAN installation, QLogic
requires the use of a “slipstream” DVD, or ISO image, with the latest QLogic
drivers injected. See “Injecting (Slipstreaming) Adapter Drivers into Windows
Image Files” on page 131.
129
AH0054601-00 A
9–FCoE Configuration
FCoE Boot from SAN
The following procedure prepares the image for installation and booting in FCoE
mode.
To set up Windows Server 2012/2012R2/2016 FCoE boot:
1.
Remove any local hard drives on the system to be booted (remote system).
2.
Prepare the Windows OS installation media by following the slipstreaming
steps in “Injecting (Slipstreaming) Adapter Drivers into Windows Image
Files” on page 131.
3.
Load the latest QLogic FCoE boot images into the adapter NVRAM.
4.
Configure the FCoE target to allow a connection from the remote device.
Ensure that the target has sufficient disk space to hold the new OS
installation.
5.
Configure the UEFI HII to set the FCoE boot type on the required adapter
port, correct initiator, and target parameters for FCoE boot.
6.
Save the settings and reboot the system. The remote system should
connect to the FCoE target, and then boot from the DVD-ROM device.
7.
Boot from DVD and begin installation.
8.
Follow the on-screen instructions.
9.
On the window that shows the list of disks available for the installation, the
FCoE target disk should be visible. This target is a disk connected through
the FCoE boot protocol, located in the remote FCoE target.
10.
To proceed with Windows Server 2012/2012R2/2016 installation, select
Next, and follow the on-screen instructions. The server will undergo a reboot
multiple times as part of the installation process.
11.
After the server boots to the OS, QLogic recommends running the driver
installer to complete the QLogic drivers and application installation.
Configuring FCoE
By default, DCB is enabled on QLogic 41000 Series FCoE- and DCB-compatible
C-NICs. QLogic 41000 Series FCoE requires a DCB-enabled interface. For
Windows operating systems, use QCC GUI or a command line utility to configure
the DCB parameters.
FCoE Crash Dump
Crash dump functionality is currently supported for FCoE boot for the FastLinQ
41000 Series Adapters.
No additional configuration is required for FCoE crash-dump generation when in
FCoE boot mode.
130
AH0054601-00 A
9–FCoE Configuration
Injecting (Slipstreaming) Adapter Drivers into Windows Image Files
Injecting (Slipstreaming) Adapter Drivers into
Windows Image Files
To inject adapter drivers into the Windows image files:
1.
Obtain the latest driver package for the applicable Windows Server version
(2012, 2012 R2, or 2016).
2.
Extract the driver package to a working directory:
a.
Open a command line session and navigate to the folder that contains
the driver package.
b.
To start the driver installer, issue the following command:
setup.exe /a
c.
In the Network location field, enter the path of the folder to which to
extract the driver package. For example, type c:\temp.
d.
Follow the driver installer instructions to install the drivers in the
specified folder. In this example, the driver files are installed here:
c:\temp\Program File 64\QLogic Corporation\QDrivers
3.
Download the Windows Assessment and Deployment Kit (ADK) version 10
from Microsoft:
https://developer.microsoft.com/en-us/windows/hardware/
windows-assessment-deployment-kit
4.
Open a command line session (with administrator privilege), and navigate
through the release CD to the Tools\Slipstream folder.
5.
Locate the slipstream.bat script file, and then issue the following
command:
slipstream.bat &lt;path&gt;
Where &lt;path&gt; is the drive and subfolder that you specified in Step 2. For
example:
slipstream.bat &quot;c:\temp\Program Files 64\QLogic
Corporation\QDrivers&quot;
131
AH0054601-00 A
9–FCoE Configuration
Configuring Linux FCoE Offload
NOTE
Note the following regarding the operating system installation media:
 Operating system installation media is expected to be a local drive.
Network paths for operating system installation media are not
supported.
 The slipstream.bat script injects the driver components in all
the SKUs that are supported by the operating system installation
media.
6.
Burn a DVD containing the resulting driver ISO image file located in the
working directory.
7.
Install the Windows Server operating system using the new DVD.
Configuring Linux FCoE Offload
The QLogic FastLinQ 41000 Series Adapter FCoE software consists of a single
kernel module called qedf.ko (qedf). The qedf module is dependent on additional
parts of the Linux kernel for specific functionality:

qed.ko is the Linux eCore kernel module used for common QLogic
FastLinQ 41000 Series hardware initialization routines.

libfcoe.ko is the Linux FCoE kernel library needed to conduct FCoE
forwarder (FCF) solicitation and FCoE initialization protocol (FIP) fabric login
(FLOGI).

libfc.ko is the Linux FC kernel library needed for several functions,
including:


Name server login and registration

rport session management
scsi_transport_fc.ko is the Linux FC SCSI transport library used for
remote port and SCSI target management.
These modules must be loaded before qedf can be functional, otherwise errors
such as “unresolved symbol” can result. If the qedf module is installed in the
distribution update path, the requisite modules are automatically loaded by
modprobe. QLogic 41000 Series Adapters support FCoE offload.
132
AH0054601-00 A
9–FCoE Configuration
Differences Between qedf and bnx2fc
Differences Between qedf and bnx2fc
Significant differences exist between qedf—the driver for QLogic FastLinQ 41000
Series 10/25GbE Controller (FCoE)—and the previous QLogic FCoE offload
driver, bnx2fc. Differences include:

qedf directly binds to a PCI function exposed by the CNA.

qedf does not need the open-fcoe user space tools (fipvlan, fcoemon,
fcoeadm) to initiate discovery.

qedf issues FIP VLAN requests directly and does not need the fipvlan utility.

qedf does not need an FCoE interface created by fipvlan for fcoemon.

qedf does not sit on top of the net_device.

qedf is not dependent on network drivers (such as bnx2x and cnic).

qedf will automatically initiate FCoE discovery on link up (because it is not
dependent on fipvlan or fcoemon for FCoE interface creation).
Configuring qedf.ko
No explicit configuration is required for qedf.ko. The driver automatically binds to
the exposed FCoE functions of the CNA and begins discovery. This functionality is
similar to the functionality and operation of QLogic’s FC driver, qla2xx, as
opposed to the older bnx2fc driver.
NOTE
For more information on FastLinQ driver installation, see Chapter 3 Driver
Installation.
The load qedf.ko kernel module performs the following:
# modprobe qed
# modprobe libfcoe
# modprobe qedf
133
AH0054601-00 A
9–FCoE Configuration
Verifying FCoE Devices in Linux
Verifying FCoE Devices in Linux
Follow these steps to verify that the FCoE devices were detected correctly after
installing and loading the qedf kernel module.
To verify FCoE devices in Linux:
1.
Check lsmod to verify that the qedf and associated kernel modules were
loaded:
# lsmod | grep qedf libfcoe
69632
1
qedf libfc
143360
2
qedf,libfcoe scsi_transport_fc
65536
2
qedf,libfc qed
806912
1
qedf scsi_mod
262144 14
sg,hpsa,qedf,scsi_dh_alua,scsi_dh_rdac,dm_multipath,scsi_transport_fc,
scsi_transport_sas,libfc,scsi_transport_iscsi,scsi_dh_emc,libata,sd_mod,sr_mod
2.
Check dmesg to verify that the FCoE devices were detected properly. In this
example, the two detected FCoE CNA devices are SCSI host numbers 4
and 5.
# dmesg | grep qedf
[ 235.321185] [0000:00:00.0]: [qedf_init:3728]: QLogic FCoE Offload Driver
v8.18.8.0.
....
[ 235.322253] [0000:21:00.2]: [__qedf_probe:3142]:4: QLogic FastLinQ FCoE
Module qedf 8.18.8.0, FW 8.18.10.0
[
235.606443] scsi host4: qedf
....
[ 235.624337] [0000:21:00.3]: [__qedf_probe:3142]:5: QLogic FastLinQ FCoE
Module qedf 8.18.8.0, FW 8.18.10.0
[
235.886681] scsi host5: qedf
....
[
243.991851] [0000:21:00.3]: [qedf_link_update:489]:5: LINK UP (40 GB/s).
134
AH0054601-00 A
9–FCoE Configuration
Boot from SAN Considerations
3.
Check for discovered FCoE devices using lsblk -S:
# lsblk -S
NAME HCTL
TYPE
VENDOR
MODEL
REV TRAN
sdb
5:0:0:0
disk
SANBlaze
VLUN P2T1L0
V7.3 fc
sdc
5:0:0:1
disk
SANBlaze
VLUN P2T1L1
V7.3 fc
sdd
5:0:0:2
disk
SANBlaze
VLUN P2T1L2
V7.3 fc
sde
5:0:0:3
disk
SANBlaze
VLUN P2T1L3
V7.3 fc
sdf
5:0:0:4
disk
SANBlaze
VLUN P2T1L4
V7.3 fc
sdg
5:0:0:5
disk
SANBlaze
VLUN P2T1L5
V7.3 fc
sdh
5:0:0:6
disk
SANBlaze
VLUN P2T1L6
V7.3 fc
sdi
5:0:0:7
disk
SANBlaze
VLUN P2T1L7
V7.3 fc
sdj
5:0:0:8
disk
SANBlaze
VLUN P2T1L8
V7.3 fc
sdk
5:0:0:9
disk
SANBlaze
VLUN P2T1L9
V7.3 fc
Configuration information for the host is located in /sys/class/fc_host/hostX,
where X is the number of the SCSI host. In the preceding example, X could 4 or
5. The hostX file contains attributes for the FCoE function, such as worldwide
port name and fabric ID.
Boot from SAN Considerations
FCoE boot from SAN should work like FC boot from SAN, where the module
simply needs to be injected by the driver update disk (DUD) into the installation
environment. The disks from any remote targets are discovered automatically.
Installation can then proceed as if the remote disks are local disks.
135
AH0054601-00 A
10
SR-IOV Configuration
Single root input/output virtualization (SR-IOV) is a specification by the PCI SIG
that enables a single PCI Express (PCIe) device to appear as multiple, separate
physical PCIe devices. SR-IOV permits isolation of PCIe resources for
performance, interoperability, and manageability.
NOTE
Some SR-IOV features may not be fully enabled in the current release. For
details, refer to Appendix C Feature Constraints.
This chapter provides instructions for:

Configuring SR-IOV on Windows

Configuring SR-IOV on Linux

Configuring SR-IOV on VMware
Configuring SR-IOV on Windows
To configure SR-IOV on Windows:
1.
Access the server BIOS System Setup, and then click System BIOS
Settings.
2.
On the System BIOS Settings page, click Integrated Devices.
136
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Windows
3.
On the Integrated Devices page (Figure 10-1):
a.
Set the SR-IOV Global Enable option to Enabled.
b.
Click Back.
Figure 10-1. System Setup for SR-IOV: Integrated Devices
4.
On the Main Configuration Page for the selected adapter, click Device Level
Configuration.
5.
On the Main Configuration Page - Device Level Configuration (Figure 10-2):
a.
Set the Virtualization Mode to SR-IOV.
b.
Click Back.
Figure 10-2. System Setup for SR-IOV: Device Level Configuration
6.
On the Main Configuration Page, click Finish.
137
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Windows
7.
In the Warning - Saving Changes message box, click Yes to save the
configuration.
8.
In the Success - Saving Changes message box, click OK.
9.
To enable SR-IOV on the miniport adapter:
a.
Access Device Manager.
b.
Open the miniport adapter properties, and then click the Advanced
tab.
c.
On the Advanced properties page (Figure 10-3) under Property, select
SR-IOV, and then set the value to Enabled.
d.
Click OK.
Figure 10-3. Adapter Properties, Advanced: Enabling SR-IOV
10.
To create a Virtual Machine Switch with SR-IOV (Figure 10-4):
a.
Launch the Hyper-V Manager.
b.
Select Virtual Switch Manager.
c.
In the Name box, type a name for the virtual switch.
d.
Under Connection type, select External network.
138
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Windows
e.
Select the Enable single-root I/O virtualization (SR-IOV) check box,
and then click Apply.
NOTE
Be sure to enable SR-IOV when you create the vSwitch. This
option is unavailable after the vSwitch is created.
Figure 10-4. Virtual Switch Manager: Enabling SR-IOV
139
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Windows
f.
11.
The Apply Networking Changes message box advises you that
Pending changes may disrupt network connectivity. To save your
changes and continue, click Yes.
To get the virtual machine switch capability, issue the following Windows
PowerShell command:
PS C:\Users\Administrator&gt; Get-VMSwitch -Name SR-IOV_vSwitch | fl
Output of the Get-VMSwitch command includes the following SR-IOV
capabilities:
12.
IovVirtualFunctionCount
: 96
IovVirtualFunctionsInUse
: 1
To create a virtual machine (VM) and export the virtual function (VF) in the
VM:
a.
Create a virtual machine.
b.
Add the VMNetworkadapter to the virtual machine.
c.
Assign a virtual switch to the VMNetworkadapter.
140
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Windows
d.
In the Settings for VM &lt;VM_Name&gt; dialog box (Figure 10-5),
Hardware Acceleration page, under Single-root I/O virtualization,
select the Enable SR-IOV check box, and then click OK.
NOTE
After the virtual adapter connection is created, the SR-IOV
setting can be enabled or disabled at any time (even while traffic
is running).
Figure 10-5. Settings for VM: Enabling SR-IOV
141
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Windows
13.
Install the QLogic drivers for VF in the VM.
NOTE
Be sure to use the same driver package on both the VM and the host
system. For example, use the same qeVBD and qeND driver version
on the Windows VM and in the Windows Hyper-V host.
After installing the drivers, the QLogic adapter is listed in the VM.
Figure 10-6 shows an example.
Figure 10-6. Device Manager: VM with QLogic Adapter
14.
To view the SR-IOV VF details, issue the following Windows PowerShell
command:
PS C:\Users\Administrator&gt; Get-NetadapterSriovVf
142
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Linux
Figure 10-7 shows example output.
Figure 10-7. Windows PowerShell Command: Get-NetadapterSriovVf
Configuring SR-IOV on Linux
To configure SR-IOV on Linux:
1.
Access the server BIOS System Setup, and then click System BIOS
Settings.
2.
On the System BIOS Settings page, click Integrated Devices.
3.
On the System Integrated Devices page (see Figure 10-1 on page 137):
4.
a.
Set the SR-IOV Global Enable option to Enabled.
b.
Click Back.
On the System BIOS Settings page, click Processor Settings.
143
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Linux
5.
On the Processor Settings (Figure 10-8) page:
a.
Set the Virtualization Technology option to Enabled.
b.
Click Back.
Figure 10-8. System Setup: Processor Settings for SR-IOV
6.
On the System Setup page, select Device Settings.
7.
On the Device Settings page, select Port 1 for the QLogic adapter.
144
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Linux
8.
On the Device Level Configuration page (Figure 10-9):
a.
Set the Virtualization Mode to SR-IOV.
b.
Click Back.
Figure 10-9. System Setup for SR-IOV: Integrated Devices
9.
On the Main Configuration Page, click Finish, save your settings, and then
reboot the system.
145
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Linux
10.
To enable and verify virtualization:
a.
Open the grub.conf file and configure the intel_iommu parameter
as shown in Figure 10-10.
Figure 10-10. Editing the grub.conf File for SR-IOV
b.
Save the grub.conf file and then reboot the system.
c.
To verify that the changes are in effect, issue the following command:
dmesg | grep
-I
iommu
A successful input–output memory management unit (IOMMU)
command output should show, for example:
Intel-IOMMU: enabled
d.
To view VF details (number of VFs and total VFs), issue the find
/sys/|grep -I sriov command.
146
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Linux
11.
For a specific port, enable a quantity of VFs.
a.
Issue the following command to enable, for example, 8 VF on PCI
instance 04:00.0 (bus 4, device 0, function 0):
[root@ah-rh68 ~]# echo 8 &gt;
/sys/devices/pci0000:00/0000:00:02.0/0000:04:00.0/sriov_numvfs
b.
Review the command output (Figure 10-11) to confirm that actual VFs
were created on bus 4, device 2 (from the 0000:00:02.0 parameter),
functions 0 through 7. Note that the actual device ID is different on the
PFs (8070 in this example) versus the VFs (9090 in this example).
Figure 10-11. Command Output for sriov_numvfs
12.
To view a list of all PF and VF interfaces, issue the following command:
# ip link show/ifconfig -a
147
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Linux
Figure 10-12 shows example output.
Figure 10-12. Command Output for ip link show Command
13.
Assign and verify MAC addresses:
a.
To assign a MAC address to the VF, issue the following command:
ip link set &lt;pf device&gt; vf &lt;vf index&gt; mac &lt;mac address&gt;
b.
Ensure that the VF interface is up and running with the assigned MAC
address.
148
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on Linux
14.
Power off the VM and attach the VF. (Some OSs support hot-plugging of
VFs to the VM.)
a.
In the Virtual Machine dialog box (Figure 10-13), click Add Hardware.
Figure 10-13. RHEL68 Virtual Machine
b.
In the left pane of the Add New Virtual Hardware dialog box
(Figure 10-14), click PCI Host Device.
c.
In the right pane, select a host device.
149
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on VMware
d.
Click Finish.
Figure 10-14. Add New Virtual Hardware
15.
Power on the VM and then issue the following command:
check lspci -vv|grep -I ether
16.
If no inbox driver is available, install the driver.
17.
As needed, add more VFs in the VM.
Configuring SR-IOV on VMware
To configure SR-IOV on VMware:
1.
Access the server BIOS System Setup, and then click System BIOS
Settings.
2.
On the System BIOS Settings page, click Integrated Devices.
3.
On the Integrated Devices page (see Figure 10-1 on page 137):
a.
Set the SR-IOV Global Enable option to Enabled.
b.
Click Back.
4.
On the System Setup window, click Device Settings.
5.
On the Device Settings page, select a port for the 25G 41000 Series
Adapter.
6.
On the Device Level Configuration (see Figure 10-2 on page 137):
7.
a.
Set the Virtualization Mode to SR-IOV.
b.
Click Back.
On the Main Configuration Page, click Finish.
150
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on VMware
8.
Save the configuration settings and reboot the system.
9.
To enable the needed quantity of VFs per port (in this example, 16 on each
port of a dual-port adapter), issue the following command:
&quot;esxcfg-module -s &quot;max_vfs=16,16&quot; qedentv&quot;
NOTE
Each Ethernet function of the 41000 Series Adapter must have it's own
entry.
10.
Reboot the host.
11.
To verify that the changes are complete at the module level, issue the
following command:
&quot;esxcfg-module -g qedentv&quot;
[root@localhost:~] esxcfg-module -g qedentv
qedentv enabled = 1 options = 'max_vfs=16,16'
12.
To verify if actual VFs were created, issue the lspci command as follows:
[root@localhost:~] lspci | grep -i QLogic | grep -i 'ethernet\|network' | more
0000:05:00.0 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx 10/25
GbE Ethernet Adapter [vmnic6]
0000:05:00.1 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx 10/25
GbE Ethernet Adapter [vmnic7]
0000:05:02.0 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.0_VF_0]
0000:05:02.1 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.0_VF_1]
0000:05:02.2 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.0_VF_2]
0000:05:02.3 Network controller: QLogic Corp. QLogic FastLinQ QL41xQL41xxxxx
Series 10/25 GbE Controller (SR-IOV VF) [PF_0.5.0_VF_3]
.
.
.
0000:05:03.7 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.0_VF_15]
0000:05:0e.0 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.1_VF_0]
0000:05:0e.1 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.1_VF_1]
0000:05:0e.2 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.1_VF_2]
151
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on VMware
0000:05:0e.3 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.1_VF_3]
.
.
.
0000:05:0f.6 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.1_VF_14]
0000:05:0f.7 Network controller: QLogic Corp. QLogic FastLinQ QL41xxx Series
10/25 GbE Controller (SR-IOV VF) [PF_0.5.1_VF_15]
13.
To validate the VFs per port, issue the esxcli command as follows:
[root@localhost:~] esxcli network sriovnic vf list -n vmnic6
14.
15.
VF ID
Active
PCI Address
Owner World ID
-----
------
-----------
--------------
0
true
005:02.0
60591
1
true
005:02.1
60591
2
false
005:02.2
-
3
false
005:02.3
-
4
false
005:02.4
-
5
false
005:02.5
-
6
false
005:02.6
-
7
false
005:02.7
-
8
false
005:03.0
-
9
false
005:03.1
-
10
false
005:03.2
-
11
false
005:03.3
-
12
false
005:03.4
-
13
false
005:03.5
-
14
false
005:03.6
-
15
false
005:03.7
-
Attach VFs to the VM as follows:
a.
Power off the VM and attach the VF. (Some OSs support hot-plugging
of VFs to the VM.)
b.
Add a host to a VMware vCenter Server Virtual Appliance (vCSA).
c.
Click Edit Settings of the VM.
Complete the Edit Settings dialog box (Figure 10-15) as follows:
a.
In the New Device box, select Network, and then click Add.
b.
For Adapter Type, select SR-IOV Passthrough.
152
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on VMware
c.
For Physical Function, select the QLogic VF.
d.
To save your configuration changes and close this dialog box, click
OK.
Figure 10-15. VMware Host Edit Settings
16.
Power on the VM, and then issue the ifconfig -a command to verify that
the added network interface is listed.
153
AH0054601-00 A
10–SR-IOV Configuration
Configuring SR-IOV on VMware
17.
If no inbox driver is available, install the driver.
18.
As needed, add more VFs in the VM.
154
AH0054601-00 A
11
iSER Configuration
This chapter provides procedures for configuring iSCSI Extensions for RDMA
(iSER) for Linux (RHEL, SLES, and Ubuntu), including:

Before You Begin

Configuring iSER for RHEL

Configuring iSER for SLES 12

Using iSER with iWARP on RHEL and SLES

Configuring iSER for Ubuntu

Optimizing Linux Performance
Before You Begin
As you prepare to configure iSER, consider the following:

iSER is supported only in inbox OFED for following operating systems:



RHEL 7.2 and 7.3
SLES12 SP1 and SP2
Ubuntu 14.04 LTS and 16.04 LTS

After logging into the targets or while running I/O traffic, unloading the Linux
RoCE qedr driver may crash the system.

While running I/O, performing interface down/up tests or performing cable
pull-tests can cause driver or iSER module errors that may crash the
system. If this happens, reboot the system.
Configuring iSER for RHEL
To configure iSER for RHEL:
1.
Install inbox OFED as described in “RoCE Configuration for RHEL” on
page 63. Out-of-box OFEDs are not supported for iSER because the ib_isert
module is not available in the out-of-box OFED 3.18-2 GA/3.18-3 GA
155
AH0054601-00 A
11–iSER Configuration
Configuring iSER for RHEL
versions. The inbox ib_isert module does not work with any out-of-box
OFED versions.
2.
Unload any existing FastLinQ drivers as described in “Removing the Linux
Drivers” on page 13.
3.
Install the latest FastLinQ driver and libqedr packages as described in
“Installing the Linux Drivers with RDMA” on page 17.
4.
Load the RDMA services.
systemctl start rdma
modprobe qedr
modprobe ib_iser
modprobe ib_isert
5.
Verify that all RDMA and iSER modules loaded on the initiator and target
devices by issuing the lsmod | grep qed and lsmod | grep iser
commands.
6.
Verify that there are separate hca_id instances by issuing the
ibv_devinfo command, as shown in Step 6 on page 68.
7.
Check the RDMA connection on the initiator device and the target device.
a.
On the initiator device issue the following command:
rping -s -C 10 -v
b.
On the target device, issue the following command:
rping -c -a 192.168.100.99 -C 10 -v
Figure 11-1 shows an example of a successful RDMA ping.
Figure 11-1. RDMA Ping Successful
156
AH0054601-00 A
11–iSER Configuration
Configuring iSER for RHEL
8.
You can use a Linux TCM-LIO target to test iSER. The setup is the same for
any iSCSI target, except that you issue the command
enable_iser Boolean=true on the applicable portals. The portal
instances are identified as iser in Figure 11-2.
Figure 11-2. iSER Portal Instances
9.
Install Linux iSCSI Initiator Utilities using the
yum install iscsi-initiator-utils commands.
a.
To discover the iSER target, issue the iscsiadm command. For
example:
iscsiadm -m discovery -t st -p 192.168.100.99:3260
b.
To change the transport mode to iSER, issue the iscsiadm
command. For example:
iscsiadm -m node -T iqn.2015-06.test.target1 -o update -n
iface.transport_name -v iser
c.
To connect to or log in to the iSER target, issue the iscsiadm
command. For example:
iscsiadm -m node -l -p 192.168.100.99:3260 -T
iqn.2015-06.test.target1
157
AH0054601-00 A
11–iSER Configuration
Configuring iSER for RHEL
d.
Confirm that the Iface Transport is iser in the target connection,
as shown Figure 11-3. Issue the iscsiadm command; for example:
iscsiadm -m session -P2
Figure 11-3. Iface Transport Confirmed
e.
To check for a new iSCSI device, as shown Figure 11-4, issue the
lsscsi command.
Figure 11-4. Checking for New iSCSI Device
158
AH0054601-00 A
11–iSER Configuration
Configuring iSER for SLES 12
Configuring iSER for SLES 12
Because the targetcli is not inbox on SLES 12.x, you must complete the following
procedure.
To configure iSER for SLES 12:
1.
To install targetcli, copy and install the following RPMs from the ISO image
(x86_64 and noarch location):
lio-utils-4.1-14.6.x86_64.rpm
python-configobj-4.7.2-18.10.noarch.rpm
python-PrettyTable-0.7.2-8.5.noarch.rpm
python-configshell-1.5-1.44.noarch.rpm
python-pyparsing-2.0.1-4.10.noarch.rpm
python-netifaces-0.8-6.55.x86_64.rpm
python-rtslib-2.2-6.6.noarch.rpm
python-urwid-1.1.1-6.144.x86_64.rpm
targetcli-2.1-3.8.x86_64.rpm
2.
Before starting the targetcli, load all RoCE device drivers and iSER modules
as follows:
# modprobe qed
# modprobe qede
# modprobe qedr
# modprobe ib_iser
(Initiator)
# modprobe ib_isert (Target)
3.
Before configuring iSER targets, configure NIC interfaces and run L2 and
RoCE traffic, as described in Step 7 on page 68.
4.
Start the targetcli utility, and configure your targets on the iSER target
system.
NOTE
targetcli versions are different in RHEL and SLES. Be sure to use the
proper backstores to configure your targets:
 RHEL uses ramdisk
 SLES uses rd_mcp
159
AH0054601-00 A
11–iSER Configuration
Using iSER with iWARP on RHEL and SLES
Using iSER with iWARP on RHEL and SLES
Configure the iSER initiator and target similar to RoCE to work with iWARP. You
can use different methods to create a Linux-IO Target (LIO™); one is listed in this
section. You may encounter some difference in targetcli configuration in SLES 12
and RHEL 7.x because of the version.
To configure a target for LIO:
1.
Create an LIO target using the targetcli utility. Issue the following command:
# targetcli
targetcli shell version 2.1.fb41
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
2.
Issue the following commands:
/&gt; /backstores/ramdisk create Ramdisk1-1 1g nullio=true
/&gt; /iscsi create iqn.2017-04.com.org.iserport1.target1
/&gt; /iscsi/iqn.2017-04.com.org.iserport1.target1/tpg1/luns create
/backstores/ramdisk/Ramdisk1-1
/&gt; /iscsi/iqn.2017-04.com.org.iserport1.target1/tpg1/portals/ create
192.168.21.4 ip_port=3261
/&gt; /iscsi/iqn.2017-04.com.org.iserport1.target1/tpg1/portals/192.168.21.4:3261
enable_iser boolean=true
/&gt; /iscsi/iqn.2017-04.com.org.iserport1.target1/tpg1 set attribute
authentication=0 demo_mode_write_protect=0 generate_node_acls=1
cache_dynamic_acls=1
/&gt; saveconfig
Figure 11-5 shows the target configuration for LIO.
Figure 11-5. LIO Target Configuration
160
AH0054601-00 A
11–iSER Configuration
Configuring iSER for Ubuntu
To configure an initiator for iWARP:
1.
To discover the iSER LIO target using port 3261, issue the iscsiadm
command as follows:
# iscsiadm -m discovery -t st -p 192.168.21.4:3261 -I iser
192.168.21.4:3261,1 iqn.2017-04.com.org.iserport1.target1
2.
Change the transport mode to iser as follows:
# iscsiadm -m node -o update -T iqn.2017-04.com.org.iserport1.target1 -n
iface.transport_name -v iser
3.
Log into the target using port 3261:
# iscsiadm -m node -l -p 192.168.21.4:3261 -T iqn.2017-04.com.org.iserport1.target1
Logging in to [iface: iser, target: iqn.2017-04.com.org.iserport1.target1,
portal: 192.168.21.4,3261] (multiple)
Login to [iface: iser, target: iqn.2017-04.com.org.iserport1.target1, portal:
192.168.21.4,3261] successful.
4.
Ensure that those LUNs are visible by issuing the following command:
# lsscsi
[1:0:0:0]
storage HP
P440ar
3.56
-
[1:1:0:0]
disk
HP
LOGICAL VOLUME
3.56
/dev/sda
[6:0:0:0]
cd/dvd
hp
DVD-ROM DUD0N
UMD0
/dev/sr0
[7:0:0:0]
disk
LIO-ORG
Ramdisk1-1
4.0
/dev/sdb
Configuring iSER for Ubuntu
To configure iSER, you must first configure RoCE, and then configure the target
and initiator devices as described in the following sections:

Configuring LIO as Target

Configuring the Initiator
Configuring LIO as Target
To configure the Linux-IO (LIO) as a target:
1.
Install targetcli by issuing the following command:
# sudo apt-get install targetcli
2.
To enter the LIO CLI console, issue the targetcli command:
root@captain:~# targetcli
targetcli GIT_VERSION (rtslib GIT_VERSION)
Copyright (c) 2011-2013 by Datera, Inc.
161
AH0054601-00 A
11–iSER Configuration
Configuring iSER for Ubuntu
All rights reserved.
/&gt; ls
o- /...................................................[...]
o- backstores ...................................... [...]
| o- fileio ........................... [0 Storage Object]
| o- iblock ........................... [0 Storage Object]
| o- pscsi ............................ [0 Storage Object]
| o- rd_dr ............................ [0 Storage Object]
| o- rd_mcp ........................... [0 Storage Object]
o- ib_srpt ................................... [0 Targets]
o- iscsi ..................................... [0 Targets]
o- loopback .................................. [0 Targets]
o- qla2xxx ................................... [0 Targets]
o- tcm_fc .................................... [0 Targets]
/&gt;
3.
Create a disk of type rd_mcp with size 1G named iSERPort1-1 by issuing
the following command:
/&gt; backstores/rd_mcp create
name=iSERPort1-1
size=1G
Generating a wwn serial.
Created rd_mcp ramdisk iSERPort1-1 with size 1G.
/&gt;
/&gt; ls
o- / ................................................. [...]
o- backstores ...................................... [...]
| o- fileio ........................... [0 Storage Object]
| o- iblock ........................... [0 Storage Object]
| o- pscsi ............................ [0 Storage Object]
| o- rd_dr ............................ [0 Storage Object]
| o- rd_mcp ........................... [1 Storage Object]
|
o- iSERPort1-1 ................. [ramdisk deactivated]
o- ib_srpt ................................... [0 Targets]
o- iscsi ..................................... [0 Targets]
o- loopback .................................. [0 Targets]
o- qla2xxx ................................... [0 Targets]
o- tcm_fc .................................... [0 Targets]
/&gt;
4.
Create an iSCSI target by issuing the following command:
/&gt; iscsi/ create wwn=iqn.2004-01.com.qlogic.iSERPort1.Target1
162
AH0054601-00 A
11–iSER Configuration
Configuring iSER for Ubuntu
Created target iqn.2004-01.com.qlogic.iSERPort1.Target1.
Selected TPG Tag 1.
Successfully created TPG 1.
/&gt; ls
o- / ................................................. [...]
o- backstores ...................................... [...]
| o- fileio ........................... [0 Storage Object]
| o- iblock ........................... [0 Storage Object]
| o- pscsi ............................ [0 Storage Object]
| o- rd_dr ............................ [0 Storage Object]
| o- rd_mcp ........................... [1 Storage Object]
|
o- iSERPort1-1 ................. [ramdisk deactivated]
o- ib_srpt ................................... [0 Targets]
o- iscsi ...................................... [1 Target]
| o- iqn.2004-01.com.qlogic.iSERPort1.Target ..... [1 TPG]
|
o- tpgt1 ................................... [enabled]
|
o- acls ................................... [0 ACLs]
|
o- luns ................................... [0 LUNs]
|
o- portals ............................. [0 Portals]
o- loopback .................................. [0 Targets]
o- qla2xxx ................................... [0 Targets]
o- tcm_fc .................................... [0 Targets]
/&gt;
5.
Create a LUN by issuing the following command. Be sure that the RAM disk
is activated:
/&gt; /iscsi/iqn.2004-01.com.qlogic.iSERPort1.Target1/tpgt1/luns
create /backstores/rd_mcp/iSERPort1-1
Selected LUN 0.
Successfully created LUN 0.
/&gt; ls
o- / ................................................. [...]
o- backstores ...................................... [...]
| o- fileio ........................... [0 Storage Object]
| o- iblock ........................... [0 Storage Object]
| o- pscsi ............................ [0 Storage Object]
| o- rd_dr ............................ [0 Storage Object]
| o- rd_mcp ........................... [1 Storage Object]
|
o- iSERPort1-1 ................... [ramdisk activated]
o- ib_srpt ................................... [0 Targets]
163
AH0054601-00 A
11–iSER Configuration
Configuring iSER for Ubuntu
o- iscsi ...................................... [1 Target]
| o- iqn.2004-01.com.qlogic.iSERPort1.Target1 .... [1 TPG]
|
o- tpgt1 ................................... [enabled]
|
o- acls ................................... [0 ACLs]
|
o- luns .................................... [1 LUN]
|
| o- lun0 ........... [rd_mcp/iSERPort1-1 (ramdisk)]
|
o- portals ............................. [0 Portals]
o- loopback .................................. [0 Targets]
o- qla2xxx ................................... [0 Targets]
o- tcm_fc .................................... [0 Targets
6.
Create a portal by issuing the following commands. Supply the IP address of
the interface on which to run iSER (RoCE enabled).
/&gt; /iscsi/iqn.2004-01.com.qlogic.iSERPort1.Target1/tpgt1/
portals create 192.168.10.5
Using default IP port 3260
Successfully created network portal 192.168.10.5:3260.
/&gt; ls
o- / ................................................. [...]
o- backstores ...................................... [...]
| o- fileio ........................... [0 Storage Object]
| o- iblock ........................... [0 Storage Object]
| o- pscsi ............................ [0 Storage Object]
| o- rd_dr ............................ [0 Storage Object]
| o- rd_mcp ........................... [1 Storage Object]
|
o- iSERPort1-1 ................... [ramdisk activated]
o- ib_srpt ................................... [0 Targets]
o- iscsi ...................................... [1 Target]
| o- iqn.2004-01.com.qlogic.iSERPort1.Target1 .... [1 TPG]
|
o- tpgt1 ................................... [enabled]
|
o- acls ................................... [0 ACLs]
|
o- luns .................................... [1 LUN]
|
| o- lun0 ........... [rd_mcp/iSERPort1-1 (ramdisk)]
|
o- portals .............................. [1 Portal]
|
o- 192.168.10.5:3260 ......... [OK, iser disabled]
o- loopback .................................. [0 Targets]
o- qla2xxx ................................... [0 Targets]
o- tcm_fc .................................... [0 Targets]
/&gt;
164
AH0054601-00 A
11–iSER Configuration
Configuring iSER for Ubuntu
7.
Enable iSER on the portal by issuing the following command:
/&gt; /iscsi/iqn.2004-01.com.qlogic.iSERPort1.Target1/tpgt1/port
als/192.168.10.103:3260 iser_enable
iser operation has been enabled
/&gt; ls
o- / ................................................. [...]
o- backstores ...................................... [...]
| o- fileio ........................... [0 Storage Object]
| o- iblock ........................... [0 Storage Object]
| o- pscsi ............................ [0 Storage Object]
| o- rd_dr ............................ [0 Storage Object]
| o- rd_mcp ........................... [1 Storage Object]
|
o- iSERPort1-1 ................... [ramdisk activated]
o- ib_srpt ................................... [0 Targets]
o- iscsi ...................................... [1 Target]
| o- iqn.2004-01.com.qlogic.iSERPort1.Target1 .... [1 TPG]
|
o- tpgt1 ................................... [enabled]
|
o- acls ................................... [0 ACLs]
|
o- luns .................................... [1 LUN]
|
| o- lun0 ........... [rd_mcp/iSERPort1-1 (ramdisk)]
|
o- portals .............................. [1 Portal]
|
o- 192.168.10.103:3260 ........ [OK, iser enabled]
o- loopback .................................. [0 Targets]
o- qla2xxx ................................... [0 Targets]
o- tcm_fc .................................... [0 Targets]
/&gt;
8.
Configure the quantity of access control lists (ACLs) by issuing the following
command:
/&gt; /iscsi/iqn.2004-01.com.qlogic.iSERPort1.Target1/tpgt1/ set
attribute authentication=0 demo_mode_write_protect=0
generate_node_acls=1 cache_dynamic_acls=1
Parameter demo_mode_write_protect is now '0'.
Parameter authentication is now '0'.
Parameter generate_node_acls is now '1'.
Parameter cache_dynamic_acls is now '1'.
/&gt; ls
o- / ................................................. [...]
o- backstores ...................................... [...]
| o- fileio ........................... [0 Storage Object]
| o- iblock ........................... [0 Storage Object]
165
AH0054601-00 A
11–iSER Configuration
Configuring iSER for Ubuntu
| o- pscsi ............................ [0 Storage Object]
| o- rd_dr ............................ [0 Storage Object]
| o- rd_mcp ........................... [1 Storage Object]
|
o- iSERPort1-1 ................... [ramdisk activated]
o- ib_srpt ................................... [0 Targets]
o- iscsi ...................................... [1 Target]
| o- iqn.2004-01.com.qlogic.iSERPort1.Target1 .... [1 TPG]
|
o- tpgt1 ................................... [enabled]
|
o- acls ................................... [0 ACLs]
|
o- luns .................................... [1 LUN]
|
| o- lun0 ........... [rd_mcp/iSERPort1-1 (ramdisk)]
|
o- portals .............................. [1 Portal]
|
o- 192.168.10.103:3260 ........ [OK, iser enabled]
o- loopback .................................. [0 Targets]
o- qla2xxx ................................... [0 Targets]
o- tcm_fc .................................... [0 Targets]
/&gt;
9.
Save the configuration by issuing the following command:
/&gt; saveconfig
WARNING: Saving ratan-ProLiant-DL380p-Gen8 current
configuration to disk will overwrite your boot settings.
The current target configuration will become the default boot
config.
Are you sure? Type 'yes': yes
Making backup of fc/ConfigFS with timestamp:
2015-06-09_19:07:37.855693
Successfully updated default config /etc/target/fc_start.sh
Making backup of loopback/ConfigFS with timestamp:
2015-06-09_19:07:37.855693
Successfully updated default config
/etc/target/loopback_start.sh
Making backup of srpt/ConfigFS with timestamp:
2015-06-09_19:07:37.855693
Successfully updated default config /etc/target/srpt_start.sh
Making backup of qla2xxx/ConfigFS with timestamp:
2015-06-09_19:07:37.855693
Successfully updated default config
/etc/target/qla2xxx_start.sh
Making backup of LIO-Target/ConfigFS with timestamp:
2015-06-09_19:07:37.855693
166
AH0054601-00 A
11–iSER Configuration
Configuring iSER for Ubuntu
Generated LIO-Target config:
/etc/target/backup/lio_backup-2015-06-09_19:07:37.855693.sh
Making backup of Target_Core_Mod/ConfigFS with timestamp:
2015-06-09_19:07:37.855693
Generated Target_Core_Mod config:
/etc/target/backup/tcm_backup-2015-06-09_19:07:37.855693.sh
Successfully updated default config /etc/target/lio_start.sh
Successfully updated default config /etc/target/tcm_start.sh
/&gt;
Configuring the Initiator
To configure the initiator:
1.
Load the ib_iser module and confirm that it is loaded properly by issuing the
following commands:
# sudo modprobe ib_iser
# lsmod | grep ib_iser
ib_isert
56835
iscsi_target_mod
307333
2
6 ib_isert
target_core_mod
382144 22
target_core_iblock,tcm_qla2xxx,target_core_pscsi,iscsi_target_mod,tcm_fc,
ib_srpt,target_core_file,target_core_user,tcm_loop,ib_isert
ib_iser
52919
0
rdma_cm
48739
3 ib_iser,rdma_ucm,ib_isert
ib_core
98710 15
qedr,rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,ib_mad,ib_ucm,ib_iser,ib_srpt,ib_umad,i
b_uverbs,rdma_ucm,ib_ipoib,ib_isert
libiscsi
57498
scsi_transport_iscsi
100628
2.
3 libiscsi_tcp,iscsi_tcp,ib_iser
4 iscsi_tcp,ib_iser,libiscsi
Issue the iscsiadm command to discover the target and change the
transport mode to iSER.
# iscsiadm -m discovery -t st -p 192.168.10.5:3260 -I iser
192.168.10.5:3260,1 iqn.2004-01.com.qlogic.iSERPort1.Target1
# iscsiadm -m node -T iqn.2004-01.com.qlogic.iSERPort1.Target1
-o update -n iface.transport_name -v iser
3.
Log into the target device by issuing the following command:
# iscsiadm -m node -l
Logging in to [iface: default, target:
iqn.2004-01.com.qlogic.iSERPort1.Target1, portal:
192.168.10.5,3260] (multiple)
167
AH0054601-00 A
11–iSER Configuration
Optimizing Linux Performance
Login to [iface: default, target:
iqn.2004-01.com.qlogic.iSERPort1.Target1, portal:
192.168.10.5,3260] successful.
4.
Verify that the LUNs are visible by issuing the following commands:
# sudo apt-get install lsscsi
# lsscsi
[1:0:0:0]
cd/dvd
hp
DVD D
DS8D9SH
JHJ4
/dev/sr0
[2:0:0:0]
disk
HP
LOGICAL VOLUME
4.68
/dev/sda
[2:0:0:1]
disk
HP
LOGICAL VOLUME
4.68
-
[2:3:0:0]
storage HP
P420i
4.68
-
[3:0:0:0]
disk
RAMDISK-MCP
4.0
/dev/sdb
LIO-ORG
Optimizing Linux Performance
Consider the following Linux performance configuration enhancements described
in this section.

Configuring CPUs to Maximum Performance Mode

Configuring Kernel sysctl Settings

Configuring IRQ Affinity Settings

Configuring Block Device Staging
Configuring CPUs to Maximum Performance Mode
Configure the CPU scaling governor to performance by using the following script
to set all CPUs to maximum performance mode:
for CPUFREQ in
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f
$CPUFREQ ] || continue; echo -n performance &gt; $CPUFREQ; done
Verify that all CPU cores are set to maximum performance mode by issuing the
following command:
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
168
AH0054601-00 A
11–iSER Configuration
Optimizing Linux Performance
Configuring Kernel sysctl Settings
Set the kernel sysctl settings as follows:
sysctl -w net.ipv4.tcp_mem=&quot;4194304 4194304 4194304&quot;
sysctl -w net.ipv4.tcp_wmem=&quot;4096 65536 4194304&quot;
sysctl -w net.ipv4.tcp_rmem=&quot;4096 87380 4194304&quot;
sysctl -w net.core.wmem_max=4194304
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_default=4194304
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.ipv4.tcp_timestamps=0
sysctl -w net.ipv4.tcp_sack=1
sysctl -w net.ipv4.tcp_low_latency=1
sysctl -w net.ipv4.tcp_adv_win_scale=1
echo 0 &gt; /proc/sys/vm/nr_hugepages
Configuring IRQ Affinity Settings
The following example sets CPU core 0, 1, 2, and 3 to IRQ XX, YY, ZZ, and XYZ
respectively. Performs these steps for each IRQ assigned to a port (default is
eight queues per port).
systemctl disable irqbalance
systemctl stop irqbalance
cat /proc/interrupts | grep qedr
Shows IRQ assigned to each port queue
echo 1 &gt; /proc/irq/XX/smp_affinity_list
echo 2 &gt; /proc/irq/YY/smp_affinity_list
echo 4 &gt; /proc/irq/ZZ/smp_affinity_list
echo 8 &gt; /proc/irq/XYZ/smp_affinity_list
Configuring Block Device Staging
Set the block device staging settings for each iSCSI device or target as follows:
echo noop &gt; /sys/block/sdd/queue/scheduler
echo 2 &gt; /sys/block/sdd/queue/nomerges
echo 0 &gt; /sys/block/sdd/queue/add_random
echo 1 &gt; /sys/block/sdd/queue/rq_affinity
169
AH0054601-00 A
12
Windows Server 2016
This chapter provides the following information for Windows Server 2016:

Configuring RoCE Interfaces with Hyper-V

RoCE over Switch Embedded Teaming

Configuring QoS for RoCE

Configuring VMMQ

Configuring VXLAN

Configuring Storage Spaces Direct

Deploying and Managing a Nano Server
Configuring RoCE Interfaces with Hyper-V
In Windows Server 2016, Hyper-V with Network Direct Kernel Provider Interface
(NDKPI) Mode-2, host virtual network adapters (host virtual NICs) support RDMA.
NOTE
DCBX is required for RoCE over Hyper-V. To configure DCBX, either:
 Configure through the HII (see “Preparing the Adapter” on page 55).
 Configure using QoS (see “Configuring QoS for RoCE” on page 177).
RoCE configuration procedures in this section include:

Creating a Hyper-V Virtual Switch with an RDMA Virtual NIC

Adding a VLAN ID to Host Virtual NIC

Verifying If RoCE is Enabled

Adding Host Virtual NICs (Virtual Ports)

Mapping the SMB Drive and Running RoCE Traffic
170
AH0054601-00 A
12–Windows Server 2016
Configuring RoCE Interfaces with Hyper-V
Creating a Hyper-V Virtual Switch with an RDMA Virtual NIC
Follow the procedures in this section to create a Hyper-V virtual switch and then
enable RDMA in the host VNIC.
To create a Hyper-V virtual switch with an RDMA virtual NIC:
1.
Launch Hyper-V Manager.
2.
Click Virtual Switch Manager (see Figure 12-1).
Figure 12-1. Enabling RDMA in Host Virtual NIC
3.
Create a virtual switch.
4.
Select the Allow management operating system to share this network
adapter check box.
In Windows Server 2016, a new parameter—Network Direct (RDMA)—is added in
the Host virtual NIC.
To enable RDMA in a host virtual NIC:
1.
Open the Hyper-V Virtual Ethernet Adapter Properties window.
2.
Click the Advanced tab.
3.
On the Advanced page (Figure 12-2):
a.
Under Property, select Network Direct (RDMA).
b.
Under Value, select Enabled.
171
AH0054601-00 A
12–Windows Server 2016
Configuring RoCE Interfaces with Hyper-V
c.
Click OK.
Figure 12-2. Hyper-V Virtual Ethernet Adapter Properties
4.
To enable RDMA, issue the following Windows PowerShell command:
PS C:\Users\Administrator&gt; Enable-NetAdapterRdma &quot;vEthernet
(New Virtual Switch)&quot;
PS C:\Users\Administrator&gt;
Adding a VLAN ID to Host Virtual NIC
To add VLAN ID to a host virtual NIC:
1.
To find the host virtual NIC name, issue the following Windows PowerShell
command:
PS C:\Users\Administrator&gt; Get-VMNetworkAdapter -ManagementOS
Figure 12-3 shows the command output.
Figure 12-3. Windows PowerShell Command: Get-VMNetworkAdapter
172
AH0054601-00 A
12–Windows Server 2016
Configuring RoCE Interfaces with Hyper-V
2.
To set the VLAN ID to the host virtual NIC, issue the following Windows
PowerShell command:
PS C:\Users\Administrator&gt; Set-VMNetworkAdaptervlan
-VMNetworkAdapterName &quot;New Virtual Switch&quot; -VlanId 5 -Access
-Management05
NOTE
Note the following about adding a VLAN ID to a host virtual NIC:
 A VLAN ID must be assigned to a host virtual NIC. The same
VLAN ID must be assigned to all the interfaces, and on the switch.
 Make sure that the VLAN ID is not assigned to the physical
Interface when using a host virtual NIC for RoCE.
 If you are creating more than one host virtual NIC, you can assign
a different VLAN to each host virtual NIC.
Verifying If RoCE is Enabled
To verify if the RoCE is enabled:

Issue the following Windows PowerShell command:
Get-NetAdapterRdma
Command output lists the RDMA supported adapters as shown in
Figure 12-4.
Figure 12-4. Windows PowerShell Command: Get-NetAdapterRdma
Adding Host Virtual NICs (Virtual Ports)
To add host virtual NICs:
1.
To add a host virtual NIC, issue the following command:
Add-VMNetworkAdapter -SwitchName &quot;New Virtual Switch&quot; -Name
SMB - ManagementOS
2.
Enable RDMA on host virtual NICs as shown in “To enable RDMA in a host
virtual NIC:” on page 171.
173
AH0054601-00 A
12–Windows Server 2016
Configuring RoCE Interfaces with Hyper-V
3.
To assign a VLAN ID to the virtual port, issue the following command:
Set-VMNetworkAdapterVlan -VMNetworkAdapterName SMB -VlanId 5
-Access -ManagementOS
Mapping the SMB Drive and Running RoCE Traffic
To map the SMB drive and run the RoCE traffic:
1.
Launch the Performance Monitor (Perfmon).
2.
Complete the Add Counters dialog box (Figure 12-5) as follows:
a.
Under Available counters, select RDMA Activity.
b.
Under Instances of selected object, select the adapter.
c.
Click Add.
Figure 12-5. Add Counters Dialog Box
174
AH0054601-00 A
12–Windows Server 2016
RoCE over Switch Embedded Teaming
If the RoCE traffic is running, counters appear as shown in Figure 12-6.
Figure 12-6. Performance Monitor Shows RoCE Traffic
RoCE over Switch Embedded Teaming
Switch Embedded Teaming (SET) is Microsoft’s alternative NIC teaming solution
available to use in environments that include Hyper-V and the Software Defined
Networking (SDN) stack in Windows Server 2016 Technical Preview. SET
integrates limited NIC Teaming functionality into the Hyper-V Virtual Switch.
Use SET to group between one and eight physical Ethernet network adapters into
one or more software-based virtual network adapters. These adapters provide
fast performance and fault tolerance if a network adapter failure occurs. To be
placed on a team, SET member network adapters must all be installed in the
same physical Hyper-V host.
RoCE over SET procedures included in this section:

Creating a Hyper-V Virtual Switch with SET and RDMA Virtual NICs

Enabling RDMA on SET

Assigning a VLAN ID on SET

Running RDMA Traffic on SET
175
AH0054601-00 A
12–Windows Server 2016
RoCE over Switch Embedded Teaming
Creating a Hyper-V Virtual Switch with SET and RDMA Virtual NICs
To create a Hyper-V virtual switch with SET and RDMA virtual NICs:

To create a SET, issue the following Windows PowerShell command:
PS C:\Users\Administrator&gt; New-VMSwitch -Name SET
-NetAdapterName &quot;Ethernet 2&quot;,&quot;Ethernet 3&quot;
-EnableEmbeddedTeaming $true
Figure 12-7 shows command output.
Figure 12-7. Windows PowerShell Command: New-VMSwitch
Enabling RDMA on SET
To enable RDMA on SET:
1.
To view the SET on the adapter, issue the following Windows PowerShell
command:
PS C:\Users\Administrator&gt; Get-NetAdapter &quot;vEthernet (SET)&quot;
Figure 12-8 shows command output.
Figure 12-8. Windows PowerShell Command: Get-NetAdapter
2.
To enable RDMA on SET, issue the following Windows PowerShell
command:
PS C:\Users\Administrator&gt; Enable-NetAdapterRdma &quot;vEthernet
(SET)&quot;
Assigning a VLAN ID on SET
To assign a VLAN ID on SET:

To assign a VLAN ID on SET, issue the following Windows PowerShell
command:
PS C:\Users\Administrator&gt; Set-VMNetworkAdapterVlan
-VMNetworkAdapterName &quot;SET&quot; -VlanId 5 -Access -ManagementOS
176
AH0054601-00 A
12–Windows Server 2016
Configuring QoS for RoCE
NOTE
Note the following when adding a VLAN ID to a host virtual NIC:
 Make sure that the VLAN ID is not assigned to the physical Interface
when using host virtual NIC for RoCE.
 If you are creating more than one host virtual NIC, a different VLAN can
be assigned to each host virtual NIC.
Running RDMA Traffic on SET
For information about running RDMA traffic on SET, go to:
https://technet.microsoft.com/en-us/library/mt403349.aspx
Configuring QoS for RoCE
The two methods of configuring quality of service (QoS) include:

Configuring QoS by Disabling DCBX on the Adapter

Configuring QoS by Enabling DCBX on the Adapter
Configuring QoS by Disabling DCBX on the Adapter
All configuration must be completed on all of the systems in use before
configuring quality of service by disabling DCBX on the adapter. The
priority-based flow control (PFC), enhanced transition services (ETS), and traffic
classes configuration must be the same on the switch and server.
To configure QoS by disabling DCBX:
1.
Disable DCBX on the adapter.
2.
Using HII, set the RoCE Priority to 0.
3.
To install the DCB role in the host, issue the following Windows PowerShell
command:
PS C:\Users\Administrators&gt; Install-WindowsFeature
Data-Center-Bridging
4.
To set the DCBX Willing mode to False, issue the following Windows
PowerShell command:
PS C:\Users\Administrators&gt; set-NetQosDcbxSetting -Willing 0
177
AH0054601-00 A
12–Windows Server 2016
Configuring QoS for RoCE
5.
Enable QoS in the miniport as follows:
a.
Open the miniport window, and then click the Advanced tab.
b.
On the adapter’s Advanced Properties page (Figure 12-9) under
Property, select Quality of Service, and then set the value to
Enabled.
c.
Click OK.
Figure 12-9. Advanced Properties: Enable QoS
6.
Assign the VLAN ID to the interface as follows:
a.
Open the miniport window, and then click the Advanced tab.
b.
On the adapter’s Advanced Properties page (Figure 12-10) under
Property, select VLAN ID, and then set the value.
c.
Click OK.
178
AH0054601-00 A
12–Windows Server 2016
Configuring QoS for RoCE
NOTE
The preceding step is required for priority flow control (PFC).
Figure 12-10. Advanced Properties: Setting VLAN ID
7.
To enable priority flow control for RoCE on a specific priority, issue the
following command:
PS C:\Users\Administrators&gt; Enable-NetQoSFlowControl
-Priority 4
NOTE
If configuring RoCE over Hyper-V, do not assign a VLAN ID to the
physical interface.
179
AH0054601-00 A
12–Windows Server 2016
Configuring QoS for RoCE
8.
To disable priority flow control on any other priority, issue the following
commands:
PS C:\Users\Administrator&gt; Disable-NetQosFlowControl 0,1,2,3,5,6,7
PS C:\Users\Administrator&gt; Get-NetQosFlowControl
Priority
Enabled
PolicySet
IfIndex IfAlias
--------
-------
---------
------- -------
0
False
Global
1
False
Global
2
False
Global
3
False
Global
4
True
Global
5
False
Global
6
False
Global
7
False
Global
9.
To configure QoS and assign relevant priority to each type of traffic, issue
the following commands (where Priority 4 is tagged for RoCE and Priority 0
is tagged for TCP):
PS C:\Users\Administrators&gt; New-NetQosPolicy &quot;SMB&quot;
-NetDirectPortMatchCondition 445 -PriorityValue8021Action 4 -PolicyStore
ActiveStore
PS C:\Users\Administrators&gt; New-NetQosPolicy &quot;TCP&quot; -IPProtocolMatchCondition
TCP -PriorityValue8021Action 0 -Policystore ActiveStore
PS C:\Users\Administrator&gt; Get-NetQosPolicy -PolicyStore activestore
Name
: tcp
Owner
: PowerShell / WMI
NetworkProfile : All
Precedence
: 127
JobObject
:
IPProtocol
: TCP
PriorityValue
: 0
Name
: smb
Owner
: PowerShell / WMI
NetworkProfile : All
Precedence
: 127
JobObject
:
180
AH0054601-00 A
12–Windows Server 2016
Configuring QoS for RoCE
NetDirectPort
: 445
PriorityValue
: 4
10.
To configure ETS for all traffic classes defined in the previous step, issue the
following commands:
PS C:\Users\Administrators&gt; New-NetQosTrafficClass -name &quot;RDMA class&quot;
-priority 4 -bandwidthPercentage 50 -Algorithm ETS
PS C:\Users\Administrators&gt; New-NetQosTrafficClass -name &quot;TCP class&quot; -priority
0 -bandwidthPercentage 30 -Algorithm ETS
PS C:\Users\Administrator&gt; Get-NetQosTrafficClass
Name
Algorithm Bandwidth(%) Priority
----
--------- ------------ --------
PolicySet
---------
[Default]
ETS
20
2-3,5-7
Global
RDMA class
ETS
50
4
Global
TCP class
ETS
30
0
Global
11.
IfIndex IfAlias
------- -------
To see the network adapter QoS from the preceding configuration, issue the
following Windows PowerShell command:
PS C:\Users\Administrator&gt; Get-NetAdapterQos
Name
: SLOT 4 Port 1
Enabled
: True
Capabilities
:
Hardware
Current
--------
-------
MacSecBypass
: NotSupported NotSupported
DcbxSupport
: None
NumTCs(Max/ETS/PFC) : 4/4/4
OperationalTrafficClasses
OperationalFlowControl
: TC TSA
Bandwidth Priorities
-- ---
--------- ----------
0 ETS
20%
2-3,5-7
1 ETS
50%
4
2 ETS
30%
0
None
4/4/4
: Priority 4 Enabled
OperationalClassifications : Protocol
Port/Type Priority
--------
--------- --------
Default
0
NetDirect 445
4
181
AH0054601-00 A
12–Windows Server 2016
Configuring QoS for RoCE
12.
Create a startup script to make the settings persistent across the system
reboots.
13.
Run RDMA traffic and verify as described in “RoCE Configuration” on
page 53.
Configuring QoS by Enabling DCBX on the Adapter
All configuration must be completed on all of the systems in use. The PFC, ETS,
and traffic classes configuration must be the same on the switch and server.
To configure QoS by enabling DCBX:
1.
Enable DCBX (IEEE, CEE, or Dynamic).
2.
Using HII, set the RoCE Priority to 0.
3.
To install the DCB role in the host, issue the following Windows PowerShell
command:
PS C:\Users\Administrators&gt; Install-WindowsFeature
Data-Center-Bridging
NOTE
For this configuration, set DCBX Protocol to CEE.
4.
To set the DCBX Willing mode to True, issue the following command:
PS C:\Users\Administrators&gt; set-NetQosDcbxSetting -Willing 1
182
AH0054601-00 A
12–Windows Server 2016
Configuring QoS for RoCE
5.
Enable QoS in the miniport as follows:
a.
On the adapter’s Advanced Properties page (Figure 12-11) under
Property, select Quality of Service, and then set the value to
Enabled.
b.
Click OK.
Figure 12-11. Advanced Properties: Enabling QoS
6.
Assign the VLAN ID to the interface (required for PFC) as follows:
a.
Open the miniport window, and then click the Advanced tab.
b.
On the adapter’s Advanced Properties page (Figure 12-12) under
Property, select VLAN ID, and then set the value.
c.
Click OK.
183
AH0054601-00 A
12–Windows Server 2016
Configuring QoS for RoCE
Figure 12-12. Advanced Properties: Setting VLAN ID
7.
To configure the switch, issue the following Windows PowerShell command:
PS C:\Users\Administrators&gt; Get-NetAdapterQoS
Name
: Ethernet 5
Enabled
: True
Capabilities
:
Hardware
Current
--------
-------
MacSecBypass
: NotSupported NotSupported
DcbxSupport
: CEE
NumTCs(Max/ETS/PFC) : 4/4/4
OperationalTrafficClasses
OperationalFlowControl
: TC TSA
Bandwidth Priorities
-- ---
--------- ----------
0 ETS
5%
0-3,5-7
1 ETS
95%
4
CEE
4/4/4
: Priority 4 Enabled
OperationalClassifications : Protocol
Port/Type Priority
--------
--------- --------
184
AH0054601-00 A
12–Windows Server 2016
Configuring VMMQ
NetDirect 445
RemoteTrafficClasses
4
: TC TSA
Bandwidth Priorities
-- ---
--------- ----------
0 ETS
5%
0-3,5-7
1 ETS
95%
4
RemoteFlowControl
: Priority 4 Enabled
RemoteClassifications
: Protocol
Port/Type Priority
--------
--------- --------
NetDirect 445
4
NOTE
The preceding example is taken when the adapter port is connected to an
Arista 7060X switch. In this example, the switch PFC is enabled on
Priority 4. RoCE App TLVs are defined. The two traffic classes are defined
as TC0 and TC1, where TC1 is defined for RoCE. DCBX Protocol mode is
set to CEE. For Arista switch configuration, refer to “Preparing the Ethernet
Switch” on page 55“. When the adapter is in Willing mode, it accepts
Remote Configuration and shows it as Operational Parameters.
Configuring VMMQ
Virtual machine multiqueue (VMMQ) configuration information includes:

Enabling VMMQ on the Adapter

Setting the VMMQ Max QPs Default and Non-Default VPort

Creating a Virtual Machine Switch with or Without SR-IOV

Enabling VMMQ on the Virtual Machine Switch

Getting the Virtual Machine Switch Capability

Creating a VM and Enabling VMMQ on VMNetworkadapters in the VM

Default and Maximum VMMQ Virtual NIC

Enabling and Disabling VMMQ on a Management NIC

Monitoring Traffic Statistics
185
AH0054601-00 A
12–Windows Server 2016
Configuring VMMQ
Enabling VMMQ on the Adapter
To enable VMMQ on the adapter:
1.
Open the miniport window, and then click the Advanced tab.
2.
On the Advanced Properties page (Figure 12-13) under Property, select
Virtual Switch RSS, and then set the value to Enabled.
3.
Click OK.
Figure 12-13. Advanced Properties: Enabling Virtual Switch RSS
Setting the VMMQ Max QPs Default and Non-Default VPort
To set the VMMQ maximum QPs default and non-default VPort:
1.
Open the miniport window, and click the Advanced tab.
2.
On the Advanced Properties page (Figure 12-14) under Property, select
one of the following:

VMMQ Max QPs Default VPort

VMMQ Max QPs - Non-Default VPort
186
AH0054601-00 A
12–Windows Server 2016
Configuring VMMQ
3.
If applicable, adjust the Value for the selected property.
Figure 12-14. Advanced Properties: Setting VMMQ
4.
Click OK.
Creating a Virtual Machine Switch with or Without SR-IOV
To create a virtual machine switch with or without SR-IOV:
1.
Launch the Hyper-V Manager.
2.
Select Virtual Switch Manager (see Figure 12-15).
3.
In the Name box, type a name for the virtual switch.
187
AH0054601-00 A
12–Windows Server 2016
Configuring VMMQ
4.
Under Connection type:
a.
Click External network.
b.
Select the Allow management operating system to share this
network adapter check box.
Figure 12-15. Virtual Switch Manager
5.
Click OK.
188
AH0054601-00 A
12–Windows Server 2016
Configuring VMMQ
Enabling VMMQ on the Virtual Machine Switch
To enable VMMQ on the virtual machine switch:

Issue the following Windows PowerShell command:
PS C:\Users\Administrators&gt; Set-VMSwitch -name q1
-defaultqueuevmmqenabled $true -defaultqueuevmmqqueuepairs 4
Getting the Virtual Machine Switch Capability
To get the virtual machine switch capability:

Issue the following Windows PowerShell command:
PS C:\Users\Administrator&gt; Get-VMSwitch -Name ql | fl
Figure 12-16 shows example output.
Figure 12-16. Windows PowerShell Command: Get-VMSwitch
189
AH0054601-00 A
12–Windows Server 2016
Configuring VMMQ
Creating a VM and Enabling VMMQ on VMNetworkadapters in
the VM
To create a virtual machine (VM) and enable VMMQ on VMNetworksadapters
in the VM:
1.
Create a VM.
2.
Add the VMNetworkadapter to the VM.
3.
Assign a virtual switch to the VMNetworkadapter.
4.
To enable VMMQ on the VM, issue the following Windows PowerShell
command:
PS C:\Users\Administrators&gt; set-vmnetworkadapter -vmname vm1
-VMNetworkAdapterName &quot;network adapter&quot; -vmmqenabled $true
-vmmqqueuepairs 4
NOTE
For an SR-IOV capable virtual switch: If the VM switch and hardware
acceleration is SR-IOV-enabled, you must create 10 VMs with 8 virtual
NICs each to utilize VMMQ. This requirement is because the SR-IOV
has precedence over VMMQ.
Example output of 64 virtual functions and 16 VMMQs is shown here:
PS C:\Users\Administrator&gt; get-netadaptervport
Name
---Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
.
.
.
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
3
3
3
3
3
3
3
3
3
3
3
3
ID
-0
1
2
3
4
5
6
7
8
9
10
11
3
3
3
3
3
64
65
66
67
68
MacAddress
---------00-15-5D-36-0A-FB
00-0E-1E-C4-C0-A4
VID
---
ProcMask
-------0:0
0:8
0:0
0:0
0:0
0:0
0:0
0:0
0:0
0:0
0:0
0:0
0:0
0:0
0:16
1:0
0:0
00-15-5D-36-0A-04
00-15-5D-36-0A-05
00-15-5D-36-0A-06
190
FID
--PF
PF
0
1
2
3
4
5
6
7
8
9
62
63
PF
PF
PF
State
ITR
------Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
QPairs
-----Unknown
4
Adaptive 4
Unknown
1
Unknown
1
Unknown
1
Unknown
1
Unknown
1
Unknown
1
Unknown
1
Unknown
1
Unknown
1
Unknown
1
Unknown
Unknown
Adaptive
Adaptive
Adaptive
1
1
4
4
4
AH0054601-00 A
12–Windows Server 2016
Configuring VMMQ
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
Ethernet
3
3
3
3
3
3
3
3
3
3
3
3
69
70
71
72
73
74
75
76
77
78
79
80
00-15-5D-36-0A-07
00-15-5D-36-0A-08
00-15-5D-36-0A-09
00-15-5D-36-0A-0A
00-15-5D-36-0A-0B
00-15-5D-36-0A-F4
00-15-5D-36-0A-F5
00-15-5D-36-0A-F6
00-15-5D-36-0A-F7
00-15-5D-36-0A-F8
00-15-5D-36-0A-F9
00-15-5D-36-0A-FA
0:8
0:16
1:0
0:0
0:8
0:16
1:0
0:0
0:8
0:16
1:0
0:0
PF
PF
PF
PF
PF
PF
PF
PF
PF
PF
PF
PF
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Activated
Adaptive
Adaptive
Adaptive
Adaptive
Adaptive
Adaptive
Adaptive
Adaptive
Adaptive
Adaptive
Adaptive
Adaptive
4
4
4
4
4
4
4
4
4
4
4
4
PS C:\Users\Administrator&gt; get-netadaptervmq
Name
---Ethernet 4
InterfaceDescription
Enabled BaseVmqProcessor MaxProcessors NumberOfReceive
Queues
-------------------------- ---------------- ------------- --------------QLogic FastLinQ QL45212-DE...#238 False
0:0
16
1
Default and Maximum VMMQ Virtual NIC
According to the current implementation, a maximum quantity of 4 VMMQs is
available per virtual NIC; that is, up to 16 virtual NICs.
Four default queues are available as previously set using Windows PowerShell
commands. The maximum default queue can currently be set to 8. To verify the
maximum default queue, use the VMswitch capability.
Enabling and Disabling VMMQ on a Management NIC
To enable or disable VMMQ on a management NIC:

To enable VMMQ on a management NIC, issue the following command:
PS C:\Users\Administrator&gt; Set-VMNetworkAdapter –ManagementOS
–vmmqEnabled $true
The MOS VNIC has four VMMQs.

To disable VMMQ on a management NIC, issue the following command:
PS C:\Users\Administrator&gt; Set-VMNetworkAdapter –ManagementOS
–vmmqEnabled $false
A VMMQ will also be available for the multicast open shortest path first
(MOSPF).
Monitoring Traffic Statistics
To monitor virtual function traffic in a virtual machine, issue the following Windows
PowerShell command:
PS C:\Users\Administrator&gt; Use get-netadapterstatistics | fl
191
AH0054601-00 A
12–Windows Server 2016
Configuring VXLAN
Configuring VXLAN
VXLAN configuration information includes:

Enabling VXLAN Offload on the Adapter

Deploying a Software Defined Network
Enabling VXLAN Offload on the Adapter
To enable VXLAN offload on the adapter:
1.
Open the miniport window, and then click the Advanced tab.
2.
On the Advanced Properties page (Figure 12-17) under Property, select
VXLAN Encapsulated Task Offload.
Figure 12-17. Advanced Properties: Enabling VXLAN
3.
Set the Value to Enabled.
4.
Click OK.
Deploying a Software Defined Network
To take advantage of VXLAN encapsulation task offload on virtual machines, you
must deploy a Software Defined Networking (SDN) stack that utilizes a Microsoft
Network Controller.
192
AH0054601-00 A
12–Windows Server 2016
Configuring Storage Spaces Direct
For more details, refer to the following Microsoft TechNet link on Software Defined
Networking:
https://technet.microsoft.com/en-us/windows-server-docs/networking/sdn/
software-defined-networking--sdn-
Configuring Storage Spaces Direct
Windows Server 2016 introduces Storage Spaces Direct, which allows you to
build highly available and scalable storage systems with local storage.
For more information, refer to the following Microsoft TechnNet link:
https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces
/storage-spaces-direct-windows-server-2016
Configuring the Hardware
Figure 12-18 shows an example of hardware configuration.
Figure 12-18. Example Hardware Configuration
NOTE
The disks used in this example are 4 &times; 400G NVMe™, and 12 &times; 200G SSD
disks.
193
AH0054601-00 A
12–Windows Server 2016
Configuring Storage Spaces Direct
Deploying a Hyper-Converged System
This section includes instructions to install and configure the components of a
Hyper-Converged system using the Windows Server 2016. The act of deploying a
Hyper-Converged system can be divided into the following three high-level
phases:

Deploying the Operating System

Configuring the Network

Configuring Storage Spaces Direct
Deploying the Operating System
To deploy the operating systems:
1.
Install the operating system.
2.
Install the Windows server roles (Hyper-V).
3.
Install the following features:



4.
Failover
Cluster
Data center bridging (DCB)
Connect the nodes to domain and adding domain accounts.
Configuring the Network
To deploy Storage Spaces Direct, the Hyper-V switch must be deployed with
RDMA-enabled host virtual NICs.
NOTE
The following procedure assumes that there are four RDMA NIC ports.
To configure the network on each server:
1.
Configure the physical network switch as follows:
a.
Connect all adapter NICs to the switch port.
NOTE
If your test adapter has more than one NIC port, you must
connect both ports to the same switch.
b.
Enable the switch port and make sure that the switch port supports
switch-independent teaming mode, and is also part of multiple VLAN
networks.
194
AH0054601-00 A
12–Windows Server 2016
Configuring Storage Spaces Direct
Example Dell switch configuration:
no ip address
mtu 9416
portmode hybrid
switchport
dcb-map roce_S2D
protocol lldp
dcbx version cee
no shutdown
2.
Enable Network Quality of Service.
NOTE
Network Quality of Service is used to ensure that the Software Defined
Storage system has enough bandwidth to communicate between the
nodes to ensure resiliency and performance. To configure QoS on the
adapter, see “Configuring QoS for RoCE” on page 177.
3.
Create a Hyper-V virtual switch with SET and RDMA virtual NIC as follows:
a.
To identify the network adapters, issue the following command:
Get-NetAdapter | FT
Name,InterfaceDescription,Status,LinkSpeed
b.
To create the virtual switch connected to all of the physical network
adapters, and then enable the switch embedded teaming, issue the
following command:
New-VMSwitch -Name SETswitch -NetAdapterName
&quot;&lt;port1&gt;&quot;,&quot;&lt;port2&gt;&quot;,&quot;&lt;port3&gt;&quot;,&quot;&lt;port4&gt;&quot;
–EnableEmbeddedTeaming $true
c.
To add host virtual NICs to the virtual switch, issue the following
commands:
Add-VMNetworkAdapter –SwitchName SETswitch –Name SMB_1
–managementOS
Add-VMNetworkAdapter –SwitchName SETswitch –Name SMB_2
–managementOS
NOTE
The preceding commands configure the virtual NIC from the
virtual switch that you just configured for the management
operating system to use.
195
AH0054601-00 A
12–Windows Server 2016
Configuring Storage Spaces Direct
d.
To configure the host virtual NIC to use a VLAN, issue the following
commands:
Set-VMNetworkAdapterVlan -VMNetworkAdapterName &quot;SMB_1&quot;
-VlanId 5 -Access -ManagementOS
Set-VMNetworkAdapterVlan -VMNetworkAdapterName &quot;SMB_2&quot;
-VlanId 5 -Access -ManagementOS
NOTE
These commands can be on the same or different VLANs.
e.
To verify that the VLAN ID is set, issue the following command:
Get-VMNetworkAdapterVlan -ManagementOS
f.
To disable and enable each host virtual NIC adapter so that the VLAN
is active, issue the following command:
Disable-NetAdapter &quot;vEthernet (SMB_1)&quot;
Enable-NetAdapter &quot;vEthernet (SMB_1)&quot;
Disable-NetAdapter &quot;vEthernet (SMB_2)&quot;
Enable-NetAdapter &quot;vEthernet (SMB_2)&quot;
g.
To enable RDMA on the host virtual NIC adapters, issue the following
command:
Enable-NetAdapterRdma &quot;SMB1&quot;,&quot;SMB2&quot;
h.
To verify RDMA capabilities, issue the following command:
Get-SmbClientNetworkInterface | where RdmaCapable -EQ
$true
Configuring Storage Spaces Direct
Configuring Storage Spaces Direct in Windows Server 2016 includes the following
steps:

Step 1. Running Cluster Validation Tool

Step 2. Creating a Cluster

Step 3. Configuring a Cluster Witness

Step 4. Cleaning Disks Used for Storage Spaces Direct

Step 5. Enabling Storage Spaces Direct

Step 6. Creating Virtual Disks

Step 7. Creating or Deploying Virtual Machines
196
AH0054601-00 A
12–Windows Server 2016
Configuring Storage Spaces Direct
Step 1. Running Cluster Validation Tool
Run the cluster validation tool to make sure server nodes are configured correctly
to create a cluster using Storage Spaces Direct.
Issue the following Windows PowerShell command to validate a set of servers for
use as Storage Spaces Direct cluster:
Test-Cluster -Node &lt;MachineName1, MachineName2, MachineName3,
MachineName4&gt; -Include &quot;Storage Spaces Direct&quot;, Inventory,
Network, &quot;System Configuration&quot;
Step 2. Creating a Cluster
Create a cluster with the four nodes (which was validated for cluster creation) in
Step 1. Running Cluster Validation Tool.
To create a cluster, issue the following Windows PowerShell command.
New-Cluster -Name &lt;ClusterName&gt; -Node &lt;MachineName1, MachineName2,
MachineName3, MachineName4&gt; -NoStorage
The –NoStorage parameter is required. If it is not included, the disks are
automatically added to the cluster, and you must remove them before enabling
Storage Spaces Direct. Otherwise, they will not be included in the Storage Spaces
Direct storage pool.
Step 3. Configuring a Cluster Witness
QLogic recommends that you configure a witness for the cluster, so that this
four-node system can withstand two nodes failing or being offline. With these
systems, you can configure file share witness or cloud witness.
For more information, go to:
https://blogs.msdn.microsoft.com/clustering/2014/03/31/configuring-a-file-sharewitness-on-a-scale-out-file-server/
Step 4. Cleaning Disks Used for Storage Spaces Direct
The disks intended to be used for Storage Spaces Direct must be empty, and
without partitions or other data. If a disk has partitions or other data, it will not be
included in the Storage Spaces Direct system.
The following Windows PowerShell command can be placed in a Windows
PowerShell script (.PS1) file and executed from the management system in an
open Windows PowerShell (or Windows PowerShell ISE) console with
Administrator privileges.
NOTE
Running this script helps identify the disks on each node that can be used
for Storage Spaces Direct and removes all data and partitions from those
disks.
197
AH0054601-00 A
12–Windows Server 2016
Configuring Storage Spaces Direct
icm (Get-Cluster -Name HCNanoUSClu3 | Get-ClusterNode) {
Update-StorageProviderCache
Get-StoragePool |? IsPrimordial -eq $false | Set-StoragePool
-IsReadOnly:$false -ErrorAction SilentlyContinue
Get-StoragePool |? IsPrimordial -eq $false | Get-VirtualDisk |
Remove-VirtualDisk -Confirm:$false -ErrorAction SilentlyContinue
Get-StoragePool |? IsPrimordial -eq $false | Remove-StoragePool
-Confirm:$false -ErrorAction SilentlyContinue
Get-PhysicalDisk | Reset-PhysicalDisk -ErrorAction
SilentlyContinue
Get-Disk |? Number -ne $null |? IsBoot -ne $true |? IsSystem -ne
$true |? PartitionStyle -ne RAW |% {
$_ | Set-Disk -isoffline:$false
$_ | Set-Disk -isreadonly:$false
$_ | Clear-Disk -RemoveData -RemoveOEM -Confirm:$false
$_ | Set-Disk -isreadonly:$true
$_ | Set-Disk -isoffline:$true
}
Get-Disk |? Number -ne $null |? IsBoot -ne $true |? IsSystem -ne
$true |? PartitionStyle -eq RAW | Group -NoElement -Property
FriendlyName
} | Sort -Property PsComputerName,Count
Step 5. Enabling Storage Spaces Direct
After creating the cluster, issue the Enable-ClusterStorageSpacesDirect
Windows PowerShell cmdlet. The cmdlet places the storage system into the
Storage Spaces Direct mode and automatically does the following:

Creates a single large pool that has a name such as S2D on Cluster1.

Configures Storage Spaces Direct cache. If there is more than one media
type available for Storage Spaces Direct use, it configures the most efficient
type as cache devices (in most cases, read and write).

Creates two tiers—Capacity and Performance—as default tiers. The
cmdlet analyzes the devices and configures each tier with the mix of device
types and resiliency.
Step 6. Creating Virtual Disks
If the Storage Spaces Direct was enabled, it creates a single pool using all of the
disks. It also names the pool (for example S2D on Cluster1), with the name of the
cluster that is specified in the name.
198
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
The following Windows PowerShell command creates a virtual disk with both
mirror and parity resiliency on the storage pool:
New-Volume -StoragePoolFriendlyName &quot;S2D*&quot; -FriendlyName
&lt;VirtualDiskName&gt; -FileSystem CSVFS_ReFS -StorageTierfriendlyNames
Capacity,Performance -StorageTierSizes &lt;Size of capacity tier in
size units, example: 800GB&gt;, &lt;Size of Performance tier in size
units, example: 80GB&gt; -CimSession &lt;ClusterName&gt;
Step 7. Creating or Deploying Virtual Machines
You can provision the virtual machines onto the nodes of the hyper-converged
S2D cluster. Store the virtual machine’s files on the system’s CSV namespace (for
example, c:\ClusterStorage\Volume1), similar to clustered virtual machines
on failover clusters.
Deploying and Managing a Nano Server
Windows Server 2016 offers Nano Server as a new installation option. Nano
Server is a remotely administered server operating system optimized for private
clouds and data centers. It is similar to Windows Server in Server Core mode, but
is significantly smaller, has no local logon capability, and supports only 64-bit
applications, tools, and agents. The Nano Server takes less disk space, sets up
faster, and requires fewer updates and restarts than Windows Server. When it
does restart, it restarts much faster.
Roles and Features
Table 12-1 shows the roles and features that are available in this release of Nano
Server, along with the Windows PowerShell options that will install the packages
for them. Some packages are installed directly with their own Windows
PowerShell options (such as -Compute). Others are installed as extensions to the
-Packages option, which you can combine in a comma-separated list.
Table 12-1. Roles and Features of Nano Server
Role or Feature
Options
Hyper-V role
-Compute
Failover Clustering
-Clustering
Hyper-V guest drivers for hosting the Nano Server
as a virtual machine
-GuestDrivers
Basic drivers for a variety of network adapters and
storage controllers. This is the same set of drivers
included in a Server Core installation of Windows
Server 2016 Technical Preview.
-OEMDrivers
199
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
Table 12-1. Roles and Features of Nano Server (Continued)
Role or Feature
Options
File Server role and other storage components
-Storage
Windows Defender Antimalware, including a
default signature file
-Defender
Reverse forwarders for application compatibility;
for example, common application frameworks such
as Ruby, Node.js, and others.
-ReverseForwarders
DNS Server Role
-Packages Microsoft-NanoServer-DNSPackage
Desired State Configuration (DSC)
-Packages Microsoft-NanoServer-DSCPackage
Internet Information Server (IIS)
-Packages Microsoft-NanoServer-IISPackage
Host Support for Windows Containers
-Containers
System Center Virtual Machine Manager Agent
-Packages Microsoft-Windows-ServerSCVMM-Package
-Packages Microsoft-Windows-ServerSCVMM-Compute-Package
Note: Use this package only if you are monitoring
Hyper-V. If you install this package, do not use the
-Compute option for the Hyper-V role; instead
use the -Packages option to install -Packages
Microsoft-NanoServer-Compute-Package,
Microsoft-Windows-Server-SCVMMCompute-Package.
Network Performance Diagnostics Service (NPDS) -Packages Microsoft-NanoServer-NPDSPackage
Data Center Bridging
-Packages Microsoft-NanoServer-DCBPackage
The next sections describe how to configure a Nano Server image with the
required packages, and how to add additional device drivers specific to QLogic
devices. They also explain how to use the Nano Server Recovery Console, how to
manage a Nano Server remotely, and how to run Ntttcp traffic from a Nano Server.
200
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
Deploying a Nano Server on a Physical Server
Follow these steps to create a Nano Server virtual hard disk (VHD) that will run on
a physical server using the preinstalled device drivers.
To deploy the Nano Server:
1.
Download the Windows Server 2016 OS image.
2.
Mount the ISO.
3.
Copy the following files from the NanoServer folder to a folder on your hard
drive:


NanoServerImageGenerator.psm1
Convert-WindowsImage.ps1
4.
Start Windows PowerShell as an administrator.
5.
Change directory to the folder where you pasted the files from Step 3.
6.
Import the NanoServerImageGenerator script by issuing the following
command:
Import-Module .\NanoServerImageGenerator.psm1 -Verbose
7.
To create a VHD that sets a computer name and includes the OEM drivers
and Hyper-V, issue the following Windows PowerShell command:
NOTE
This command will prompt you for an administrator password for the
new VHD.
New-NanoServerImage –DeploymentType Host –Edition
&lt;Standard/Datacenter&gt; -MediaPath &lt;path to root of media&gt;
-BasePath
.\Base -TargetPath .\NanoServerPhysical\NanoServer.vhd
-ComputerName
&lt;computer name&gt; –Compute -Storage -Cluster -OEMDrivers
–Compute
-DriversPath “&lt;Path to Qlogic Driver sets&gt;”
Example:
New-NanoServerImage –DeploymentType Host –Edition Datacenter
-MediaPath C:\tmp\TP4_iso\Bld_10586_iso
-BasePath &quot;.\Base&quot; -TargetPath
&quot;C:\Nano\PhysicalSystem\Nano_phy_vhd.vhd&quot; -ComputerName
201
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
&quot;Nano-server1&quot; –Compute -Storage -Cluster -OEMDrivers
-DriversPath
&quot;C:\Nano\Drivers&quot;
In the preceding example, C:\Nano\Drivers is the path for QLogic drivers.
This command takes about 10 to 15 minutes to create a VHD file. A sample
output for this command is shown here:
Windows(R) Image to Virtual Hard Disk Converter for Windows(R) 10
Copyright (C) Microsoft Corporation.
All rights reserved.
Version 10.0.14300.1000.amd64fre.rs1_release_svc.160324-1723
INFO
: Looking for the requested Windows image in the WIM file
INFO
: Image 1 selected (ServerDatacenterNano)...
INFO
: Creating sparse disk...
INFO
: Mounting VHD...
INFO
: Initializing disk...
INFO
: Creating single partition...
INFO
: Formatting windows volume...
INFO
: Windows path (I:) has been assigned.
INFO
: System volume location: I:
INFO
: Applying image to VHD. This could take a while...
INFO
: Image was applied successfully.
INFO
: Making image bootable...
INFO
: Fixing the Device ID in the BCD store on VHD...
INFO
: Drive is bootable.
INFO
: Dismounting VHD...
INFO
: Closing Windows image...
INFO
: Done.
Cleaning up...
Done. The log is at:
C:\Users\ADMINI~1\AppData\Local\Temp\2\NanoServerImageGenerator.log
8.
Log in as an administrator on the physical server where you want to run the
Nano Server VHD.
9.
To copy the VHD to the physical server and configure it to boot from the new
VHD:
a.
Go to Computer Management &gt; Storage &gt; Disk Management.
b.
Right-click Disk Management and select Attach VHD.
c.
Provide the VHD file path.
d.
Click OK.
e.
Run bcdboot d:\windows.
202
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
NOTE
In this example, the VHD is attached under D:\.
f.
Right-click Disk Management and select Detach VHD.
10.
Reboot the physical server into the Nano Server VHD.
11.
Log in to the Recovery Console using the administrator and password you
supplied while running the script in Step 7.
12.
Obtain the IP address of the Nano Server computer.
13.
Use Windows PowerShell remoting (or other remote management) tool to
connect and remotely manage the server.
Deploying a Nano Server in a Virtual Machine
To create a Nano Server virtual hard drive (VHD) to run in a virtual machine:
1.
Download the Windows Server 2016 OS image.
2.
Go to the NanoServer folder from the downloaded file in Step 1.
3.
Copy the following files from the NanoServer folder to a folder on your hard
drive:

NanoServerImageGenerator.psm1

Convert-WindowsImage.ps1
4.
Start Windows PowerShell as an administrator.
5.
Change directory to the folder where you pasted the files from Step 3.
6.
Import the NanoServerImageGenerator script by issuing the following
command:
Import-Module .\NanoServerImageGenerator.psm1 -Verbose
7.
Issue the following Windows PowerShell command to create a VHD that
sets a computer name and includes the Hyper-V guest drivers:
NOTE
This following command will prompt you for an administrator password
for the new VHD.
New-NanoServerImage –DeploymentType Guest –Edition
&lt;Standard/Datacenter&gt; -MediaPath &lt;path to root of media&gt;
-BasePath
.\Base -TargetPath .\NanoServerPhysical\NanoServer.vhd
-ComputerName
203
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
&lt;computer name&gt; –GuestDrivers
Example:
New-NanoServerImage –DeploymentType Guest –Edition Datacenter
-MediaPath C:\tmp\TP4_iso\Bld_10586_iso
-BasePath .\Base -TargetPath .\Nano1\VM_NanoServer.vhd
-ComputerName
Nano-VM1 –GuestDrivers
The preceding command takes about 10 to 15 minutes to create a VHD file.
A sample output for this command follows:
PS C:\Nano&gt; New-NanoServerImage –DeploymentType Guest –Edition
Datacenter -MediaPath
C:\tmp\TP4_iso\Bld_10586_iso -BasePath .\Base -TargetPath
.\Nano1\VM_NanoServer.vhd -ComputerName Nano-VM1 –GuestDrivers
cmdlet New-NanoServerImage at command pipeline position 1
Supply values for the following parameters:
Windows(R) Image to Virtual Hard Disk Converter for Windows(R) 10
Copyright (C) Microsoft Corporation.
All rights reserved.
Version 10.0.14300. 1000.amd64fre.rs1_release_svc.160324-1723
INFO
: Looking for the requested Windows image in the WIM file
INFO
: Image 1 selected (ServerTuva)...
INFO
: Creating sparse disk...
INFO
: Attaching VHD...
INFO
: Initializing disk...
INFO
: Creating single partition...
INFO
: Formatting windows volume...
INFO
: Windows path (G:) has been assigned.
INFO
: System volume location: G:
INFO
: Applying image to VHD. This could take a while...
INFO
: Image was applied successfully.
INFO
: Making image bootable...
INFO
: Fixing the Device ID in the BCD store on VHD...
INFO
: Drive is bootable.
INFO
: Closing VHD...
INFO
: Deleting pre-existing VHD : Base.vhd...
INFO
: Closing Windows image...
INFO
: Done.
Cleaning up...
Done. The log is at:
C:\Users\ADMINI~1\AppData\Local\Temp\2\NanoServerImageGenerator.log
204
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
8.
Create a new virtual machine in Hyper-V Manager, and use the VHD created
in Step 7.
9.
Boot the virtual machine.
10.
Connect to the virtual machine in Hyper-V Manager.
11.
Log in to the Recovery Console using the administrator and password you
supplied while running the script in Step 7.
12.
Obtain the IP address of the Nano Server computer.
13.
Use Windows PowerShell remoting (or other remote management) tool to
connect and remotely manage the server.
Managing a Nano Server Remotely
Options for managing Nano Server remotely include Windows PowerShell,
Windows Management Instrumentation (WMI), Windows Remote Management,
and Emergency Management Services (EMS). This section describes how to
access Nano Server using Windows PowerShell remoting.
Managing a Nano Server with Windows PowerShell Remoting
To manage Nano Server with Windows PowerShell remoting:
1.
Add the IP address of the Nano Server to your management computer’s list
of trusted hosts.
NOTE
Use the recovery console to find the server IP address.
2.
Add the account you are using to the Nano Server’s administrators.
3.
(Optional) Enable CredSSP if applicable.
Adding the Nano Server to a List of Trusted Hosts
At an elevated Windows PowerShell prompt, add the Nano Server to the list of
trusted hosts by issuing the following command:
Set-Item WSMan:\localhost\Client\TrustedHosts &quot;&lt;IP address of Nano
Server&gt;&quot;
Examples:
Set-Item WSMan:\localhost\Client\TrustedHosts &quot;172.28.41.152&quot;
Set-Item WSMan:\localhost\Client\TrustedHosts &quot;*&quot;
NOTE
The preceding command sets all host servers as trusted hosts.
205
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
Starting the Remote Windows PowerShell Session
At an elevated local Windows PowerShell session, start the remote Windows
PowerShell session by issuing the following commands:
$ip = &quot;&lt;IP address of Nano Server&gt;&quot;
$user = &quot;$ip\Administrator&quot;
Enter-PSSession -ComputerName $ip -Credential $user
You can now run Windows PowerShell commands on the Nano Server as usual.
However, not all Windows PowerShell commands are available in this release of
Nano Server. To see which commands are available, issue the command
Get-Command -CommandType Cmdlet. To stop the remote session, issue the
command Exit-PSSession.
For more details on Nano Server, go to:
https://technet.microsoft.com/en-us/library/mt126167.aspx
Managing QLogic Adapters on a Windows Nano Server
To manage QLogic adapters in Nano Server environments, refer to the Windows
QConvergeConsole GUI and Windows QLogic Control Suite CLI management
tools and associated documentation, available from the QLogic Downloads and
Documentation page:
http://driverdownloads.qlogic.com
RoCE Configuration
To manage the Nano Server with Windows PowerShell remoting:
1.
Connect to the Nano Server through Windows PowerShell Remoting from
another machine. For example:
PS C:\Windows\system32&gt; $1p=&quot;172.28.41.152&quot;
PS C:\Windows\system32&gt; $user=&quot;172.28.41.152\Administrator&quot;
PS C:\Windows\system32&gt; Enter-PSSession -ComputerName $ip
-Credential $user
NOTE
In the preceding example, the Nano Server IP address is
172.28.41.152 and the user name is Administrator.
If the Nano Server connects successfully, the following is returned:
[172.28.41.152]: PS C:\Users\Administrator\Documents&gt;
2.
To determine if the drivers are installed and the link is up, issue the following
Windows PowerShell command:
206
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
[172.28.41.152]: PS C:\Users\Administrator\Documents&gt;
Get-NetAdapter
Figure 12-19 shows example output.
Figure 12-19. Windows PowerShell Command: Get-NetAdapter
3.
To verify whether the RDMA is enabled on the adapter, issue the following
Windows PowerShell command:
[172.28.41.152]: PS C:\Users\Administrator\Documents&gt;
Get-NetAdapterRdma
Figure 12-20 shows example output.
Figure 12-20. Windows PowerShell Command: Get-NetAdapterRdma
4.
To assign an IP address and VLAN ID to all interfaces of the adapter, issue
the following PowerShell commands:
[172.28.41.152]: PS C:\&gt; Set-NetAdapterAdvancedProperty
-InterfaceAlias &quot;slot 1 port 1&quot; -RegistryKeyword vlanid
-RegistryValue 5
[172.28.41.152]: PS C:\&gt; netsh interface ip set address
name=&quot;SLOT 1 Port 1&quot; static 192.168.10.10 255.255.255.0
5.
To create SMBShare on the Nano Server, issue the following Windows
PowerShell command:
[172.28.41.152]: PS C:\Users\Administrator\Documents&gt;
New-Item -Path c:\ -Type Directory -Name smbshare -Verbose
207
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
Figure 12-21 shows example output.
Figure 12-21. Windows PowerShell Command: New-Item
[172.28.41.152]: PS C:\&gt; New-SMBShare -Name &quot;smbshare&quot; -Path
c:\smbshare -FullAccess Everyone
Figure 12-22 shows example output.
Figure 12-22. Windows PowerShell Command: New-SMBShare
6.
To map the SMBShare as a network drive in the client machine, issue the
following Windows PowerShell command:
NOTE
The IP address of an interface on the Nano Server is 192.168.10.10.
PS C:\Windows\system32&gt; net use z: \\192.168.10.10\smbshare
This command completed successfully.
7.
To perform read/write on SMBShare and check RDMA statistics on the Nano
Sever, issue the following Windows PowerShell command:
[172.28.41.152]: PS C:\&gt;
(Get-NetAdapterStatistics).RdmaStatistics
208
AH0054601-00 A
12–Windows Server 2016
Deploying and Managing a Nano Server
Figure 12-23 shows the command output.
Figure 12-23. Windows PowerShell Command: Get-NetAdapterStatistics
209
AH0054601-00 A
13
Troubleshooting
This chapter provides the following troubleshooting information:

Troubleshooting Checklist

Verifying that Current Drivers Are Loaded

Testing Network Connectivity

Microsoft Virtualization with Hyper-V

Linux-specific Issues

Miscellaneous Issues

Collecting Debug Data
Troubleshooting Checklist
CAUTION
Before you open the server cabinet to add or remove the adapter, review the
“Safety Precautions” on page 8.
The following checklist provides recommended actions to resolve problems that
may arise while installing the 41000 Series Adapter or running it in your system.

Inspect all cables and connections. Verify that the cable connections at the
network adapter and the switch are attached properly.

Verify the adapter installation by reviewing “Installing the Adapter” on
page 9. Ensure that the adapter is properly seated in the slot. Check for
specific hardware problems, such as obvious damage to board components
or the PCI edge connector.

Verify the configuration settings and change them if they are in conflict with
another device.

Verify that your server is using the latest BIOS.

Try inserting the adapter in another slot. If the new position works, the
original slot in your system may be defective.
210
AH0054601-00 A
13–Troubleshooting
Verifying that Current Drivers Are Loaded

Replace the failed adapter with one that is known to work properly. If the
second adapter works in the slot where the first one failed, the original
adapter is probably defective.

Install the adapter in another functioning system, and then run the tests
again. If the adapter passes the tests in the new system, the original system
may be defective.

Remove all other adapters from the system, and then run the tests again. If
the adapter passes the tests, the other adapters may be causing contention.
Verifying that Current Drivers Are Loaded
Ensure that the current drivers are loaded for your Windows, Linux, or VMware
system.
Verifying Drivers in Windows
See the Device Manager to view vital information about the adapter, link status,
and network connectivity.
Verifying Drivers in Linux
To verify that the qed.ko driver is loaded properly, issue the following command:
# lsmod | grep -i &lt;module name&gt;
If the driver is loaded, the output of this command shows the size of the driver in
bytes. The following example shows the drivers loaded for the qed module:
# lsmod | grep -i qed
qed
qede
199238
1
1417947
0
If you reboot after loading a new driver, you can issue the following command to
verify that the currently loaded driver is the correct version:
modinfo qede
Or, you can issue the following command:
[root@test1]# ethtool -i eth2
driver: qede
version: 8.4.7.0
firmware-version: mfw 8.4.7.0 storm 8.4.7.0
bus-info: 0000:04:00.2
211
AH0054601-00 A
13–Troubleshooting
Testing Network Connectivity
If you loaded a new driver, but have not yet rebooted, the modinfo command will
not show the updated driver information. Instead, issue the following dmesg
command to view the logs. In this example, the last entry identifies the driver that
will be active upon reboot.
# dmesg | grep -i &quot;QLogic&quot; | grep -i &quot;qede&quot;
[
10.097526] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x
[
23.093526] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x
[
34.975396] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x
[
34.975896] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x
[ 3334.975896] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x
Verifying Drivers in VMware
To verify that the VMware ESXi drivers are loaded, issue the following command:
# esxcli software vib list
Testing Network Connectivity
This section provides procedures for testing network connectivity in Windows and
Linux environments.
NOTE
When using forced link speeds, verify that both the adapter and the switch
are forced to the same speed.
Testing Network Connectivity for Windows
Test network connectivity using the ping command.
To determine if the network connection is working:
1.
Click Start, and then click Run.
2.
In the Open box, type cmd, and then click OK.
3.
To view the network connection to be tested, issue the following command:
ipconfig /all
4.
Issue the following command, and then press ENTER.
ping &lt;ip_address&gt;
The displayed ping statistics indicate whether or not the network connection is
working.
212
AH0054601-00 A
13–Troubleshooting
Microsoft Virtualization with Hyper-V
Testing Network Connectivity for Linux
To verify that the Ethernet interface is up and running:
1.
To check the status of the Ethernet interface, issue the ifconfig
command.
2.
To check the statistics on the Ethernet interface, issue the netstat -i
command.
To verify that the connection has been established:
1.
Ping an IP host on the network. From the command line, issue the following
command:
ping &lt;ip_address&gt;
2.
Press ENTER.
The displayed ping statistics indicate whether or not the network connection is
working.
Microsoft Virtualization with Hyper-V
Microsoft Virtualization is a hypervisor virtualization system for Windows Server
2012 R2. For more information on Hyper-V, go to:
https://technet.microsoft.com/en-us/library/Dn282278.aspx
Linux-specific Issues
Problem: Errors appear when compiling driver source code.
Solution: Some installations of Linux distributions do not install the development
tools and kernel sources by default. Before compiling driver source
code, ensure that the development tools for the Linux distribution that
you are using are installed.
Miscellaneous Issues
Problem: The 41000 Series Adapter has shut down, and an error message
appears indicating that the fan on the adapter has failed.
Solution: The 41000 Series Adapter may intentionally shut down to prevent
permanent damage. Contact QLogic Technical Support for
assistance.
213
AH0054601-00 A
13–Troubleshooting
Troubleshooting Windows FCoE and iSCSI Boot from SAN
Troubleshooting Windows FCoE and iSCSI Boot
from SAN
If any USB flash drive is connected while Windows setup is loading files for
installation, an error message will appear when you provide the drivers and then
select the SAN disk for the installation. The most common error message that
Windows OS installer reports is shown at the bottom of the Windows Setup dialog
box, as shown in Figure 13-1.
Figure 13-1. Windows Setup Error Message
In other cases, the error message may indicate a need to ensure that the disk's
controller is enabled in the computer's BIOS menu. To avoid any of the depicted
error messages, ensure that there is no USB flash drive attached, until the setup
asks for the drivers.
After loading the drivers and viewing the SAN disk or disks, detach or disconnect
the USB flash drive immediately before selecting the disk for further installation.
214
AH0054601-00 A
13–Troubleshooting
Collecting Debug Data
Collecting Debug Data
Use the information in Table 13-1 to collect debug data.
Table 13-1. Collecting Debug Data Commands
Debug Data
Description
demesg-T
Kernel logs
ethtool-d
Register dump
sys_info.sh
System information; available in the driver bundle
215
AH0054601-00 A
A
Adapter LEDS
Table A-1 lists the LED indicators for the state of the adapter port link and activity.
Table A-1. Adapter Port Link and Activity LEDs
Port LED
Link LED
Activity LED
LED Appearance
Network State
Off
No link (cable disconnected)
Continuously illuminated
Link
Off
No port activity
Blinking
Port activity
216
AH0054601-00 A
B
Cables and Optical
Modules
This appendix provides the following information for the supported cables and
optical modules:

Supported Specifications

Tested Cables and Optical Modules

Tested Switches
Supported Specifications
The 41000 Series Adapters support a variety of cables and optical modules that
comply with SFF8024. Specific form factor compliance is as follows:



SFPs:

SFF8472 (for memory map)

SFF8419 or SFF8431 (low speed signals and power)
Quad small form factor pluggable (QSFPs):

SFF8636 (for memory map)

SFF8679 or SFF8436 (low speed signals and power)
Optical modules electrical input/output, active copper cables (ACC), and
active optical cables (AOC):

10G—SFF8431 limiting interface

25G—IEEE802.3by Annex 109B (25GAUI)
217
AH0054601-00 A
B–Cables and Optical Modules
Tested Cables and Optical Modules
Tested Cables and Optical Modules
QLogic does not guarantee that every cable or optical module that satisfies the
compliance requirements will operate with the 41000 Series Adapters. QLogic has
tested the components listed in Table B-1 and presents this list for your
convenience.
This list is based on cable and optics components that are available at the time of
product release, and is subject to change over time as new components enter the
market or are discontinued. To view the most current list of supported cables and
optical modules, view the QLogic FastLinQ 41000/45000 Series Interoperability
Matrix located here:
http://www.qlogic.com/Resources/Documents/LineCards/LC_41000-45000_Interoperability_Matrix.pdf
Table B-1. Tested Cables and Optical Modules
Speed/Form
Factor
Manufacturer
Part Number
Type
Cable
Gauge
Lengtha
Cables
Cisco
HP
10G
DAC b
Dell
COPQAA4JAA
SFP Twin-axial 10G
1
30
COPQAA6JAA
SFP Twin-axial 10G
3
30
COPQAA5JAA
SFP Twin-axial 10G
5
26
2074260-2
SFP Twin-axial 10G
1
30
AP784A
SFP Twin-axial 10G
3
26
AP820A
SFP Twin-axial 10G
5
26
CN-OW25W9-52204-36R0016-A00
SFP Twin-axial 10G
3
26
407-BBBK
SFP Twin-axial 10G
1
30
407-BBBI
SFP Twin-axial 10G
3
26
407-BBBP
SFP Twin-axial 10G
5
30
218
AH0054601-00 A
B–Cables and Optical Modules
Tested Cables and Optical Modules
Table B-1. Tested Cables and Optical Modules (Continued)
Speed/Form
Factor
Manufacturer
Amphenol&reg;
25G DAC
HP&reg;
40G DAC
Splitter (4 &times;
10G)
Dell
Part Number
Type
Cable Gauge
Lengtha
NDCCGJ0003
SFP28 to SFP28
3
26
NDCCGJ0005
SFP28 to SFP28
5
26
NDCCGF0001
SFP28 to SFP28
1
30
NDCCGF0003
SFP28 to SFP28
3
30
844471-B21
SFP28 to SFP28
0.5
26
844474-B21
SFP28 to SFP28
1
26
844477-B21
SFP28 to SFP28
3
26
844480-B21
SFP28 to SFP29
5
26
470-AAVO
QSFP40GB to
4XSFP10GB
1
26
470-AAXG
QSFP40GB to
4XSFP10GB
3
26
470-AAXH
QSFP40GB to
4XSFP10GB
5
26
219
AH0054601-00 A
B–Cables and Optical Modules
Tested Cables and Optical Modules
Table B-1. Tested Cables and Optical Modules (Continued)
Speed/Form
Factor
Manufacturer
Dell
FCI
100G DAC
Splitter (4 &times;
25G)
Amphenol
Arista
Part Number
Type
Cable Gauge
Lengtha
07R9N9 Rev A00
QSFP100GB to
4XSFP28GB
3
26
0YFNDD Rev A00
QSFP100GB to
4XSFP28GB
2
26
026FN3 Rev A00
QSFP100GB to
4XSFP28GB
1
26
10130795-4050LF
QSFP100GB to
4XSFP28GB
5
26
NDAQGJ-0005
QSFP100GB to
4XSFP28GB
5
26
NDAQGF-0003
QSFP100GB to
4XSFP28GB
3
30
NDAQGJ-0001
QSFP100GB to
4XSFP28GB
1
26
NDAQGF-0002
QSFP100GB to
4XSFP28GB
2
30
CAB-Q-4S-100G-3M
QSFP100GB to
4XSFP28GB
3
26
FTLX8571D3BCL-QL
SFP 10G Optical
Transceiver SR
N/A
N/A
FTLX1471D3BCL-QL
SFP 10G Optical
Transceiver LR
N/A
N/A
AFBR-703SMZ
SFP 10G Optical
Transceiver SR
N/A
N/A
AFBR-701SDZ
SFP 10G Optical
Transceiver LR
N/A
N/A
Optical Solutions
Finisar
10G Optical
Transceivers
Avago
220
AH0054601-00 A
B–Cables and Optical Modules
Tested Cables and Optical Modules
Table B-1. Tested Cables and Optical Modules (Continued)
Speed/Form
Factor
25G Optical
Transceivers
10G AOC c
25G AOC
Manufacturer
Part Number
Type
Cable Gauge
Lengtha
FTLF8536P4BCL
SFP28 Optical
Transceiver SR
N/A
N/A
Finisar
FTLF8538P4BCL
SFP28 Optical
Transceiver SR no
FEC
N/A
N/A
Mellanox
MMA2P00-AS
SFP28 Optical
Transceiver SR
N/A
N/A
HPE/Finisar
845398-B21 /
FTLF8536P4BCL
SFP28 Optical
Transceiver SR
N/A
N/A
470-ABLV
SFP 10G Optical
AOC
2
N/A
470-ABLZ
SFP 10G Optical
AOC
3
N/A
470-ABLT
SFP 10G Optical
AOC
5
N/A
470-ABML
SFP 10G Optical
AOC
7
N/A
470-ABLU
SFP 10G Optical
AOC
10
N/A
470-ABMD
SFP 10G Optical
AOC
15
N/A
470-ABMJ
SFP 10G Optical
AOC
15
N/A
TF-PY020-N00
SFP28 AOC
20
N/A
TF-PY003-N00
SFP28 AOC
3
N/A
Dell
InnoLight
a
Cable length is indicated In meters.
b
DAC is direct attach cable.
c
AOC is active optical cable.
221
AH0054601-00 A
B–Cables and Optical Modules
Tested Switches
Tested Switches
Table B-2 lists the switches that have been tested for interoperability with the
41000 Series Adapters. This list is based on switches that are available at the
time of product release, and is subject to change over time as new switches enter
the market or are discontinued. To view the most current list of supported
switches, view the QLogic FastLinQ 41000/45000 Series Interoperability Matrix
located here:
http://www.qlogic.com/Resources/Documents/LineCards/LC_41000-45000_Interoperability_Matrix.pdf
Table B-2. Switches Tested for Interoperability
Manufacturer
Arista
Ethernet Switch Model
7060X
Nexus 3132
Cisco
Nexus 5548 and 5596T
Nexus 6000
Dell EMC
Z9100
HPE
FlexFabric 5950
Mellanox
SN2700
222
AH0054601-00 A
C
Feature Constraints
This appendix provides information about feature constraints implemented in the
current release.
These feature coexistence constraints will be removed in the next release and
customers can use the feature combinations without any additional configuration
steps beyond what would be usually required to enable the features.
Concurrent FCoE and iSCSI Is Not Supported on the Same Port in
NPAR Mode
The current release does not support configuration of both FCoE and iSCSI on
PFs belonging to the same physical port when in NPAR Mode (concurrent FCoE
and iSCSI is only supported on the same port in Default Mode). Either FCoE or
iSCSI is allowed on a physical port in NPAR Mode.
After a PF with either iSCSI or FCoE personality has been configured on a port
using either HII or QLogic management tools, configuration of the storage
protocol on another PF is disallowed by those management tools.
Because storage personality is disabled by default, only the personality that has
been configured using HII or QLogic management tools is written in NVRAM
configuration. When this limitation is removed, users can configure additional PFs
on the same port for storage in NPAR Mode.
Concurrent RoCE and iWARP Is Not Supported on the Same Port
RoCE and iWARP are not supported on the same port. HII and QLogic
management tools do not allow users to configure both concurrently.
NPAR Configuration Is Not Supported if SR-IOV Is Already Configured
If SR-IOV is already configured, NPAR configuration is not allowed unless SR-IOV
is first disabled.

NPAR is configured using either HII or QLogic management tools. When
NPAR is enabled, device- and adapter-level configuration for multiple PCIe
functions are enumerated on all ports of the adapter.

SR-IOV is configured using either HII or QLogic management tools. When
SR-IOV is enabled, adapter-level configuration is not allowed if NPAR is
already configured.
223
AH0054601-00 A
C–Feature Constraints

For the HII mapping specification, the Virtualization Mode field supports
only the values None, NPAR, and SR-IOV. The value NPARSRIOV is not
currently supported.
NOTE
The 41000 Series Adapters will support the NPAR+SR-IOV combination in a
future software release.
RoCE and iWARP Configuration Is Not Supported if NPAR Is Already
Configured
If NPAR is already configured on the adapter, you cannot configure RoCE or
iWARP. Currently, RDMA can be enabled on all PFs and the RDMA transport type
(RoCE or iWARP) can be configured on a per-port basis. The per-port
configuration is reflected in the per-PF settings by HII and QLogic management
tools.
RDMANICModeOnPort can be enabled and disabled. However,
RDMANICModeOnPartition is currently set to disabled and cannot be enabled.
NOTE
The 41000 Series Adapters will support RDMA (RoCE and iWARP) in a
future software release.
NIC and SAN Boot to Base Is Supported Only on Select PFs
Ethernet and PXE boot are currently supported only on PF0 and PF1. In NPAR
configuration, other PFs do not support Ethernet and PXE boot.

When the Virtualization Mode is set to NPAR, non-offloaded FCoE boot is
supported on Partition 2 (PF2 and PF3) and iSCSI boot is supported on
Partition 3 (PF4 and PF5). iSCSI and FCoE boot is limited to a single target
per boot session. iSCSI boot target LUN support is limited to LUN ID 0 only.

When the Virtualization Mode is set to None or SR-IOV, boot from SAN is
not supported.
224
AH0054601-00 A
Glossary
ACPI
The Advanced Configuration and Power
Interface (ACPI) specification provides an
open standard for unified operating
system-centric device configuration and
power management. The ACPI defines
platform-independent interfaces for
hardware discovery, configuration, power
management, and monitoring. The specification is central to operating
system-directed configuration and Power
Management (OSPM), a term used to
describe a system implementing ACPI,
which therefore removes device management responsibilities from legacy firmware
interfaces.
bandwidth
A measure of the volume of data that can
be transmitted at a specific transmission
rate. A 1Gbps or 2Gbps Fibre Channel
port can transmit or receive at nominal
rates of 1 or 2Gbps, depending on the
device to which it is connected. This corresponds to actual bandwidth values of
106MB and 212MB, respectively.
BAR
Base address register. Used to hold
memory addresses used by a device, or
offsets for port addresses. Typically,
memory address BARs must be located in
physical RAM while I/O space BARs can
reside at any memory address (even
beyond physical memory).
adapter
The board that interfaces between the
host system and the target devices.
Adapter is synonymous with Host Bus
Adapter, host adapter, and board.
base address register
See BAR.
basic input output system
See BIOS.
adapter port
A port on the adapter board.
BIOS
Basic input output system. Typically in
Flash PROM, the program (or utility) that
serves as an interface between the
hardware and the operating system and
allows booting from the adapter at startup.
Advanced Configuration and Power
Interface
See ACPI.
data center bridging
See DCB.
225
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
data center bridging exchange
See DCBX.
dynamic host configuration protocol
See DHCP.
DCB
Data center bridging. Provides enhancements to existing 802.1 bridge specifications to satisfy the requirements of
protocols and applications in the data
center. Because existing high-performance
data centers typically comprise multiple
application-specific networks that run on
different link layer technologies (Fibre
Channel for storage and Ethernet for
network management and LAN connectivity), DCB enables 802.1 bridges to be
used for the deployment of a converged
network where all applications can be run
over a single physical infrastructure.
eCore
A layer between the OS and the hardware
and firmware. It is device-specific and
OS-agnostic. When eCore code requires
OS services (for example, for memory
allocation, PCI configuration space
access, and so on) it calls an abstract OS
function that is implemented in OS-specific
layers. eCore flows may be driven by the
hardware (for example, by an interrupt) or
by the OS-specific portion of the driver (for
example, loading and unloading the load
and unload).
EFI
Extensible firmware interface. A specification that defines a software interface
between an operating system and platform
firmware. EFI is a replacement for the
older BIOS firmware interface present in
all IBM PC-compatible personal
computers.
DCBX
Data center bridging exchange. A protocol
used by DCB devices to exchange configuration information with directly connected
peers. The protocol may also be used for
misconfiguration detection and for configuration of the peer.
enhanced transmission selection
See ETS.
device
A target, typically a disk drive. Hardware
such as a disk drive, tape drive, printer, or
keyboard that is installed in or connected
to a system. In Fibre Channel, a target
device.
Ethernet
The most widely used LAN technology that
transmits information between computers,
typically at speeds of 10 and 100 million
bits per second (Mbps).
DHCP
Dynamic host configuration protocol.
Enables computers on an IP network to
extract their configuration from servers
that have information about the computer
only after it is requested.
driver
The software that interfaces between the
file system and a physical data storage
device or network media.
226
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
ETS
Enhanced transmission selection. A
standard that specifies the enhancement
of transmission selection to support the
allocation of bandwidth among traffic
classes. When the offered load in a traffic
class does not use its allocated bandwidth,
enhanced transmission selection allows
other traffic classes to use the available
bandwidth. The bandwidth-allocation priorities coexist with strict priorities. ETS
includes managed objects to support
bandwidth allocation. For more information, refer to:
http://ieee802.org/1/pages/802.1az.html
human interface infrastructure
See HII.
extensible firmware interface
See EFI.
Internet Protocol
See IP.
FCoE
Fibre Channel over Ethernet. A new
technology defined by the T11 standards
body that allows traditional Fibre Channel
storage networking traffic to travel over an
Ethernet link by encapsulating Fibre
Channel frames inside Layer 2 Ethernet
frames. For more information, visit
www.fcoe.com.
Internet small computer system interface
See iSCSI.
HII
Human interface infrastructure. A specification (part of UEFI 2.1) for managing user
input, localized strings, fonts, and forms,
that allows OEMs to develop graphical
interfaces for preboot configuration.
IEEE
Institute of Electrical and Electronics
Engineers. An international nonprofit
organization for the advancement of
technology related to electricity.
Internet wide area RDMA protocol
See iWARP.
IP
Internet protocol. A method by which data
is sent from one computer to another over
the Internet. IP specifies the format of
packets, also called datagrams, and the
addressing scheme.
Fibre Channel over Ethernet
See FCoE.
IQN
iSCSI qualified name. iSCSI node name
based on the initiator manufacturer and a
unique device name section.
file transfer protocol
See FTP.
FTP
File transfer protocol. A standard network
protocol used to transfer files from one
host to another host over a TCP-based
network, such as the Internet. FTP is
required for out-of-band firmware uploads
that will complete faster than in-band
firmware uploads.
iSCSI
Internet small computer system interface.
Protocol that encapsulates data into IP
packets to send over Ethernet connections.
iSCSI qualified name
See IQN.
227
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
iWARP
Internet wide area RDMA protocol. A
networking protocol that implements
RDMA for efficient data transfer over IP
networks. iWARP is designed for multiple
environments, including LANs, storage
networks, data center networks, and
WANs.
LSO
Large send offload. LSO Ethernet adapter
feature that allows the TCP\IP network
stack to build a large (up to 64KB) TCP
message before sending it to the adapter.
The adapter hardware segments the
message into smaller data packets
(frames) that can be sent over the wire: up
to 1,500 bytes for standard Ethernet
frames and up to 9,000 bytes for jumbo
Ethernet frames. The segmentation
process frees up the server CPU from
having to segment large TCP messages
into smaller packets that will fit inside the
supported frame size.
jumbo frames
Large IP frames used in high-performance
networks to increase performance over
long distances. Jumbo frames generally
means 9,000 bytes for Gigabit Ethernet,
but can refer to anything over the IP MTU,
which is 1,500 bytes on an Ethernet.
maximum transmission unit
See MTU.
large send offload
See LSO.
message signaled interrupts
See MSI, MSI-X.
Layer 2
Refers to the data link layer of the multilayered communication model, Open
Systems Interconnection (OSI). The
function of the data link layer is to move
data across the physical links in a network,
where a switch redirects data messages at
the Layer 2 level using the destination
MAC address to determine the message
destination.
MSI, MSI-X
Message signaled interrupts. One of two
PCI-defined extensions to support
message signaled interrupts (MSIs), in
PCI 2.2 and later and PCI Express. MSIs
are an alternative way of generating an
interrupt through special messages that
allow emulation of a pin assertion or
deassertion.
MSI-X (defined in PCI 3.0) allows a device
to allocate any number of interrupts
between 1 and 2,048 and gives each interrupt separate data and address registers.
Optional features in MSI (64-bit
addressing and interrupt masking) are
mandatory with MSI-X.
MTU
Maximum transmission unit. Refers to the
size (in bytes) of the largest packet (IP
datagram) that a specified layer of a
communications protocol can transfer.
228
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
QoS
Quality of service. Refers to the methods
used to prevent bottlenecks and ensure
business continuity when transmitting data
over virtual ports by setting priorities and
allocating bandwidth.
network interface card
See NIC.
NIC
Network interface card. Computer card
installed to enable a dedicated network
connection.
quality of service
See QoS.
NIC partitioning
See NPAR.
PF
non-volatile random access memory
See NVRAM.
Physical function.
RDMA
Remote direct memory access. The ability
for one node to write directly to the
memory of another (with address and size
semantics) over a network. This capability
is an important feature of VI networks.
NPAR
NIC partitioning. The division of a single
NIC port into multiple physical functions or
partitions, each with a user-configurable
bandwidth and personality (interface type).
Personalities include NIC, FCoE, and
iSCSI.
reduced instruction set computer
See RISC.
NVRAM
Non-volatile random access memory. A
type of memory that retains data (configuration settings) even when power is
removed. You can manually configure
NVRAM settings or restore them from a
file.
remote direct memory access
See RDMA.
RISC
Reduced instruction set computer. A
computer microprocessor that performs
fewer types of computer instructions,
thereby operating at higher speeds.
PCI™
Peripheral component interface. A 32-bit
local bus specification introduced by
Intel&reg;.
RDMA over Converged Ethernet
See RoCE.
RoCE
RDMA over Converged Ethernet. A
network protocol that allows remote direct
memory access (RDMA) over a converged
or a non-converged Ethernet network.
RoCE is a link layer protocol that allows
communication between any two hosts in
the same Ethernet broadcast domain.
PCI Express (PCIe)
A third-generation I/O standard that allows
enhanced Ethernet network performance
beyond that of the older peripheral component interconnect (PCI) and PCI extended
(PCI-X) desktop and server slots.
229
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
A target is a device that responds to a
requested by an initiator (the host system).
Peripherals are targets, but for some
commands (for example, a SCSI COPY
command), the peripheral may act as an
initiator.
SCSI
Small computer system interface. A
high-speed interface used to connect
devices, such as hard drives, CD drives,
printers, and scanners, to a computer. The
SCSI can connect many devices using a
single controller. Each device is accessed
by an individual identification number on
the SCSI controller bus.
TCP
Transmission control protocol. A set of
rules to send data in packets over the
Internet protocol.
SerDes
Serializer/deserializer. A pair of functional
blocks commonly used in high-speed
communications to compensate for limited
input/output. These blocks convert data
between serial data and parallel interfaces
in each direction.
TCP/IP
Transmission control protocol/Internet
protocol. Basic communication language
of the Internet.
TLV
Type-length-value. Optional information
that may be encoded as an element inside
of the protocol. The type and length fields
are fixed in size (typically 1–4 bytes), and
the value field is of variable size. These
fields are used as follows:
 Type—A numeric code that indicates
the kind of field that this part of the
message represents.
 Length—The size of the value field
(typically in bytes).
 Value—Variable-sized set of bytes that
contains data for this part of the
message.
serializer/deserializer
See SerDes.
single root input/output virtualization
See SR-IOV.
small computer system interface
See SCSI.
SR-IOV
Single root input/output virtualization. A
specification by the PCI SIG that enables a
single PCIe device to appear as multiple,
separate physical PCIe devices. SR-IOV
permits isolation of PCIe resources for
performance, interoperability, and
manageability.
transmission control protocol
See TCP.
target
The storage-device endpoint of a SCSI
session. Initiators request data from
targets. Targets are typically disk-drives,
tape-drives, or other media devices.
Typically a SCSI peripheral device is the
target but an adapter may, in some cases,
be a target. A target can contain many
LUNs.
transmission control protocol/Internet
protocol
See TCP/IP.
type-length-value
See TLV.
230
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
UDP
User datagram protocol. A connectionless
transport protocol without any guarantee
of packet sequence or delivery. It functions
directly on top of IP.
VLAN
Virtual logical area network (LAN). A group
of hosts with a common set of requirements that communicate as if they were
attached to the same wire, regardless of
their physical location. Although a VLAN
has the same attributes as a physical LAN,
it allows for end stations to be grouped
together even if they are not located on the
same LAN segment. VLANs enable
network reconfiguration through software,
instead of physically relocating devices.
UEFI
Unified extensible firmware interface. A
specification detailing an interface that
helps hand off control of the system for the
preboot environment (that is, after the
system is powered on, but before the
operating system starts) to an operating
system, such as Windows or Linux. UEFI
provides a clean interface between
operating systems and platform firmware
at boot time, and supports an architecture-independent mechanism for initializing add-in cards.
VM
Virtual machine. A software implementation of a machine (computer) that executes
programs like a real machine.
wake on LAN
See WoL.
unified extensible firmware interface
See UEFI.
WoL
Wake on LAN. An Ethernet computer
networking standard that allows a
computer to be remotely switched on or
awakened by a network message sent
usually by a simple program executed on
another computer on the network.
user datagram protocol
See UDP.
VF
Virtual function.
VI
Virtual interface. An initiative for remote
direct memory access across Fibre
Channel and other communication protocols. Used in clustering and messaging.
virtual interface
See VI.
virtual logical area network
See VLAN.
virtual machine
See VM.
231
AH0054601-00 A
Index
A
address
MAC, permanent and virtual 37
PCI 37
ADK, downloading Windows 131
Advanced Configuration and Power Interface,
See ACPI 225
affinity settings, IRQ 169
agency certifications xxii
agreements, license xx
allocating bandwidth 47
AOC specifications, supported 217
applications for adapter management 3
architecture, host hardware 7
assigning VLAN ID on SET 176
audience, intended for guide xvi
auto_fw_reset parameter 28
ACC specifications, supported 217
ACPI
definition of 225
manageability feature supported 2
activity LED indicator 216
adapter
See also adapter port
definition of 225
drivers installation 11
drivers,
enable RoCE 60
feature support 1
functional description 1
installation 7, 9
installation steps 9
LED indicators 216
management 4, 34
management tools 3, 4
model supported xvi
preboot configuration 34
properties (Windows) 20
specifications 5
adapter port
See also adapter
definition of 225
alignment, connectors 10
configuring device settings 35
FCoE boot type on 130
link and activity LEDs 216
link status, confirming 32
server, VLAN ID 60
Add Counters dialog box 174
adding
host VNIC 173
VLAN ID to host VNIC 172
B
bandwidth
definition of 225
allocations, configuring 47, 49
supported on PCIe 7
BAR
definition of 225
support for 3
base address register, See BAR
basic input output system, See BIOS
BIOS
definition of 225
installation error 214
specifying boot protocol 124
verifying version 9, 210
blinking LEDs, indicators 216
block device staging, configuring 169
bnx2fc driver, differences from qedf 133
232
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
Coalesce Tx Microseconds 19
collecting debug data 215
compliance
product laser safety xxii
product safety xxiv
configuring
adapter UEFI boot mode 124
DHCP server for iSCSI boot 100
FCoE 123
FCoE boot parameters 127
FCoE crash dump 130
FCoE DCB parameters 130
hardware 193
iSCSI boot from SAN 104
iSCSI target 86
iSER for Ubuntu 161
iSER initiator 167
iSER on RHEL 155
iSER on SLES 12 159
iSER with iWARP 160
iWARP 76
iWARP on Linux 81
iWARP on Windows 76
kernel sysctl settings 169
Linux FCoE offload 132
Linux for optimal performance 168
qedf.ko 133
QoS by disabling DCBX 177
QoS by enabling DCBX 182
RoCE interfaces with Hyper-V 170
RoCE on Windows Nano Server 206
SR-IOV on Linux 143
SR-IOV on VMware 150
SR-IOV on Windows 136
Storage Spaces Direct 193
VXLAN 192
connections
DAC, SerDes interface 2
inspecting 210
L2, verifying 68
network, verifying 213
RDMA, SMB Direct 62
RoCE, verifying 69
constraints on features in this release 223
boot
See also boot code and boot from SAN
installation, Windows Server 129
mode, adapter UEFI, configuring 124
parameters, configuring FCoE 127
protocol, BIOS 124
boot code
See also boot and boot from SAN
downloading updates xx
upgrading with scripts 31
boot from SAN
See also boot and boot code
FCoE 123
FCoE considerations 135
FCoE for Windows 129
iSCSI, configuring on SLES 12 104
troubleshooting 214
bracket types for adapter 5
C
cables
inspecting 210
pull-tests, performing 155
supported specifications 217
tested and supported 218
CEE
DCBX protocol 55
DCBX protocol, configuring 59
protocol supported 55
protocol, DCBX 59
CentOS
host requirement 8
Linux drivers, installing 16
OS support 8
certification, KCC Class A xxiii
checklist
preinstallation 9
troubleshooting 210
chip type 37
Class A certification xxiii
class map 55, 56
Coalesce Rx Microseconds 18
233
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
debug parameter 18, 28
definitions of terms 225
deploying
Hyper-Converged system 194
SDN 192
Windows Nano Server 199, 201, 203
device
definition of 226
FCoE, verifying in Linux 134
ID 37
name 37
Device Manage, verifying Windows driver 211
DHCP 98
definition of 226
dynamic iSCSI boot 97
enabling on interface for SLES 16
IP address, assigning 67
iSCSI configuration 87
iSCSI general parameters 92, 98
iSCSI, configuring 45
Option 17 98
Option 43 98
server, configuring for iSCSI boot 100
disable_tpa parameter 28
disabling DCBX to configure QoS 177
dmesg
verifying driver installation 212
verifying FCoE detected 134
verifying iSCSI devices detected 117
verifying RoCE devices detected 67
verifying SR-IOV devices detected 146
documentation
conventions xviii
downloading xx
guide description xvi
intended audience xvi
related materials xvii
downloading
Linux drivers from QLogic 13, 17
QCC GUI and QCS CLI 206
QLogic support documents 7, 8
VMware driver from VMware 26
Windows ADK 131
Windows drivers 19
contacting QLogic xxi
controller, power management options 22
conventions, documentation xviii
converged enhanced Ethernet, See CEE
CPUs, configuring for maximum performance
168
crash dump functionality, FCoE boot 130
creating
DCB map 57
grub.cfg backup file 121
Hyper-V virtual switch 171, 176
initramfs image 122
iSCSI target 86, 162
Linux-IO Target 160
LUN for iSER 163
portal for iSER 164
rd_mcp disk 162
rules file for RoCE 65
vdisk for target 86
virtual disk 86
Virtual Machine Switch with SR-IOV 138
VM for SR-IOV 140
VM on VMNetworkadapters 190
VM switch (with or without SR-IOV) 187
D
DAC cables, supported 218
data center bridging exchange, See DCBX
data center bridging, See DCB
DCB
definition of 226
feature support 2
map, creating 57
parameters, configuring for FCoE 130
DCBX
definition of 226
disabling to configure QoS 177
enabling 55
enabling to configure QoS 182
protocol 59
protocol supported 55
debug data, collecting 215
234
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
driver removal
Linux, non-RoCE 14
Linux, RoCE 14
VMware 30
Windows 20
dynamic host configuration protocol, See
DHCP
Windows Server 2016 OS image 201
updates xx
driver
See also driver installation
See also driver removal
definition of 226
downloading updates xx
for Linux, verifying loaded 211
for VMware, verifying loaded 212
for Windows, verifying loaded 211
injecting into Nano image 23
injecting into Windows image files 131
installing for Nano Server 22
Linux files 12
Linux Ubuntu, installing 16
Linux, description of 12
Linux, downloading from QLogic 13, 17
Linux, installing 11
messages (Linux) 19
packages, VMware ESXi 25
parameters (Linux) 18
parameters (VMware) 27
qedf VMware FCoE 30
qedi, configuring qedi.ko 117
qedil VMware iSCSI 30
source code errors, troubleshooting 213
statistics (Linux) 19
troubleshooting Linux 211
troubleshooting VMware 212
troubleshooting Windows 211
upgrade (VMware) 27
VMware 25
VMware, downloading from VMware 26
VMware, parameter defaults 29
driver installation 11
kmod RPM 16
Linux without RoCE 13
RoCE, inbox OFED 17
source RPM 15
TAR file, Linux 16
VMware 26
Windows 19
E
eCore
definition of 226
qed.ko Linux kernel module 116, 132
EFI
definition of 226
driver version 38
EMC requirements xxiii
EMI requirements xxiii
enable_vxlan_offld parameter 29
enabling
DCBX to configure QoS 182
VMMQ on adapter 186
enhanced transmission selection, See FTP
ESXi driver packages 25
Ethernet
definition of 226
drivers, loading 66
interface, verifying 213
ETS
definition of 227
configuring on switch 57
feature support 2
EULA xx
eVBD driver
upgrading on Windows 32
upgrading on Windows Nano 33
F
fan failure, troubleshooting 213
FAQs, iSCSI offload 114
FastLinQ Driver Install package, downloading
19
235
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
G
FastLinQ ESXCLI VMware Plug-in adapter
management tool 5
FCoE
definition of 227
boot configuration parameters 43
boot from SAN 123
boot from SAN considerations 135
boot from SAN, troubleshooting 214
boot from SAN, Windows 129
boot installation, Windows Server 129
boot mode 42
boot mode, configuring adapter UEFI 124
boot parameters, configuring 127
configuring 123
crash dump functionality 130
DCB parameters, configuring 130
devices, verifying in Linux 134
Linux offload driver 12
Linux offload, configuring 132
native offload driver 25
qedf driver, VMware support 30
system BIOS, preparing 124
Windows offload driver 23
FDA notice xxii
features
constraints in current release 223
Windows Nano Server 199
Fibre Channel over Ethernet, See FCoE
file transfer protocol, See FTP
filtering, VXLAN 28
firmware
downloading updates xx
properties 38
reset 28
upgrading on Linux 32
upgrading on Windows 32
upgrading on Windows Nano 33
Firmware Upgrade Utility, running in Nano
Server 24
flow control 18, 29
Flow Control, troubleshooting Linux issue 213
FTP
definition of 227
network installation 13
functional description of adapter 1
GID
index values, VLAN 69
VLAN interfaces, configuring RoCE 62
global bandwidth allocation 47
global ID, See GID
glossary terms and definitions 225
guide
documentation conventions xviii
guide, overview of contents xvi
H
hardware
host architecture 7
installation 7
requirements, host 7
Windows configuration 193
HII
application, starting 34
DCBX, enabling with 55
definition of 227
iWARP, configuring 76
UEFI, configuring FCoE boot mode 124
host
hardware requirements 7
OS requirements 8
VNIC, adding 173
VNIC, adding VLAN ID to 172
HTTP network installation 13
human interface infrastructure, See HII
hw_vlan parameter 27, 29
Hyper-Converged system
deploying 194
Storage Spaces Direct, configuring 196
Storage Spaces Direct, deploying 194
Hyper-V
Microsoft Virtualization 213
RoCE interfaces, configuring with 170
hypervisor virtualization for Windows 213
236
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
I
IP
checksum offloads support 2
IPv4 standards support 6
IPv6 standards support 6
IP, definition of 227
IQN
definition of 227
initiator 101, 102, 103
iSCSI initiator 86
iSCSI target 86
system initiator 91
target name 101
IRQ affinity settings 169
iSCSI
definition of 227
boot configuration 44, 100
boot from SAN, configuring on SLES 12 104
boot from SAN, troubleshooting 214
legacy offload driver 25
Linux offload driver 12
offload, FAQs 114
qedil driver, VMware support 30
target, configuring 86
Windows offload driver 23
iSCSI qualified name, See IQN
iscsi_boot_sysfs.ko in Linux iSCSI offload 116
iSER
configuration, preparation for 155
configuring for Ubuntu 161
feature support 1
initiator, configuring 167
RHEL, configuring 155
SLES 12 configuration 159
iWARP
definition of 228
configuring on Linux 81
configuring on Windows 76
enabling 77
initiator, configuring 161
iSER with on Linux 160
IEEE
DCBX, enabling 182
definition of 227
feature support 2
standards specifications 5
image files, injecting drivers into 131
image verification, firmware upgrade 31
inbox OFED, OS support 53
information level 28
initiator
iWARP, configuring 161
initiator, configuring for iSER 167
injecting drivers into Nano image 23
injecting drivers into Windows image files 131
installation
error message, Windows Setup dialog box
214
pre-checklist 9
installing
adapters 9
drivers, Linux with RDMA 17
drivers, Nano Server 22
drivers, qedf for Linux 12
drivers, qedi for Linux 12
drivers, Ubuntu Linux 16
drivers, VMware 26
drivers, Windows 19
hardware 7
Institute of Electrical and Electronics
Engineers, See IEEE
intended audience of guide xvi
interfaces
RoCE, configuring 170
verifying Ethernet 213
Internet Protocol, definition of 227
Internet small computer system interface, See
iSCSI
Internet wide area RDMA protocol, See
iWARP
interoperability, switches 222
237
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
J
Linux (continued)
kernel module qed.ko 116, 132
minimum host OS 8
performance, optimizing 168
SR-IOV, configuring 143
Linux drivers
See also Linux
downloading from QLogic 13, 17
installing 11
installing with RDMA 17
optional, qede 18
LIO
target for, configuring 160
target, configuring as 161
TCM target, testing iSER with 157
lnxfwnx2 utility, firmware upgrade utility for
Linux 31
load qedf.ko kernel actions 133
log level 28
LRO, VMware driver parameter 29
LSO
definition of 228
feature support 2
jumbo frames
definition of 228
support for 3
jumbo packet 60
K
KCC Class A certification xxiii
kernel sysctl settings 169
kmod RPM package 16
knowledge database, QLogic xxii
L
large send offload, See LRO
large send offload, See LSO
laser safety xxii
Layer 2
definition of 228
VLAN segregation 93, 103
LED, port state indicators 216
legal notices xxii
libfc.ko, in Linux FCoE offload 132
libfcoe.ko, in Linux FCoE offload 132
libiscsi.ko in Linux iSCSI offload 116
license agreements xx
link
LED indicator 216
speed 18, 29
status 37
Linux
See also Linux drivers
drivers, installing Ubuntu 16
FCoE devices, verifying 134
FCoE offload, configuring 132
firmware, updating on 32
iSCSI boot from SAN, configuring for SLES
12 104
iSER and iWARP on 160
issues, troubleshooting 213
iWARP, configuring 81
M
MAC address, permanent and virtual 37
maintenance mode 26
management applications 4
management NIC, VMMQ on 191
management tools for adapters 3
manufacturers of tested cables and optical
modules 218
map
DCB 57
network QoS class 56
QoS class 56
QoS policy 56
queuing class 56
mapping SMB drive 174
max_vfs parameter 28
maximum bandwidth, allocating 47
maximum transmission unit, See MTU
238
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
non-volatile random access memory, See
NVRAM
notice level 28
notices, legal xxii
NPAR
definition of 229
configuring 47
feature support 1
num_queues parameter 28, 29
NVMe-oF feature support 1
NVRAM
boot images, loading FCoE 130
boot images, loading iSCSI 115
definition of 229
feature support 3
memory required for host hardware 7
message signaled interrupts, See MSI, MSI-X
messages, Linux driver 19
MFW version 38
Microsoft Virtualization with Hyper-V 213
minimum bandwidth, allocating 47
models, supported adapters xvi
modules, tested optical 218
monitoring traffic statistics 191
MSI
definition of 228
Linux driver parameter 18
MSI-X
definition of 228
interrupt mode, qede driver 18
Linux driver parameter 18
VMware driver parameter 29
MTU
definition of 228
size, Linux driver 18
size, network direct 60
size, VMware driver parameter 29
multi_rx_filters parameter 28
O
OFED
inbox, RHEL 63
inbox, SLES 64
limitations 54
OS support 53
working with qedr driver 12
offload
checksum support 2
iSCSI, FAQs 114
Linux FCoE, configuring 132
open fabric enterprise distribution, See OFED
operating system
OFED support
requirements, host 8
RoCE v1/v2 support
optical modules
supported specifications 217
tested and supported 218
optimizing Linux performance 168
optional VMware driver parameters 27
overview, product 1
N
Nano Server, See Windows Nano Server
NDKPI, RoCE interfaces 170
network
connectivity 212
installations 13
network connectivity, testing
Linux 213
Windows 212
Network Direct Functionality, enabling 78
Network Direct Kernel Provider Interface,
RoCE interfaces 170
network interface card, See NIC
network state indicators 216
Nexus 6000 Ethernet switch 55
NFS network installation 13
NIC partitioning, See NPAR
239
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
P
precautions, safety 8
preface, guide introduction xvi
preinstallation checklist 9
priority
PFC 57
RoCE 55
priority-based flow control, See PFC
product
functional description 1
overview of 1
safety compliance information xxiv
training from QLogic xxi
product safety compliance xxiv
packet, jumbo 60
parameters
Linux driver defaults 18
Linux qede, optional 18
VMware driver, default 29
VMware driver, optional 27
partitions
bandwidth, configuring 47
PCI
definition of 229
address 37
PCI Express, See PCIe
PCIe
bandwidth supported 7
card for adapter 5
connector slot 9
definition of 229
host hardware requirement 7
standards specifications 5
Performance Monitor, viewing RoCE traffic
175
performance, RoCE 62
peripheral component interface, See PCI
PFC
feature support 2
priority 57
PFs
definition of 229
max PFs per 28
physical characteristics, adapter 5
physical function, See PF
pnputil tool
adding and installing driver package with 22
deleting oem0.inf package with 23
upgrading or installing drivers with 24
policy map 55, 56
port activity indicator 216
port LED, link and activity indicators 216
power management, setting options 22
PowerShell, Windows 61
preboot configuration 34
Q
QCC GUI
description of 4
downloading 206
QCC vCenter Plug-In 4
QConvergeConsole GUI adapter
management tool 4
QConvergeConsole GUI, See QCC GUI
QConvergeConsole PowerKit adapter
management tool 5
QConvergeConsole vCenter Plug-in adapter
management tool 4
QCS CLI
downloading 206
management applications 4
qed driver
defaults, Linux 18
description of 11
upgrading on Linux 32
qed.ko, in Linux FCoE offload 132
qed.ko, in Linux iSCSI offload 116
qede driver
defaults, Linux 18
Linux, description of 12
optional parameters 18
upgrading on Linux 32
qedentv driver for VMware 25
240
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
R
qedf driver
differences from bnx2fx 133
for VMware 25
installing 12
qedf driver, FCoE driver for VMware 30
qedf.ko, configuring 133
qedi driver
configuring qedi.ko 117
differences from bnx2i 116
installing 12
qedi.ko, configuring qedi driver 117
qedil driver for VMware 25
qedil driver, iSCSI driver for VMware 30
qedr driver installation 12
qedrntv driver for VMware 25
qeVBD driver
upgrading on Windows 32
upgrading on Windows Nano 33
QLogic
contacting Technical Support xxi
knowledge database xxii
training, obtaining xxi
QLogic adapter management tools,
downloading 206
QLogic Control Suite CLI, adapter
management tool 4
QLogic Control Suite, See QCS CLI
QLogic FastLinQ 41000/45000 Series
Interoperability Matrix, accessing 218,
222
QoS
definition of 229
configuring 56
configuring by disabling DCBX 177
configuring by enabling DCBX 182
QPs, setting VMMQ max 186
QSFPs, supported specifications 217
quality of service, See QoS
queues, Tx/Rx 28, 29
RDMA 53
definition of 229
adapters, supported 61
applications 54
connections per port 62
drivers, loading 66
iWARP, configuring 76
library code 12
Linux driver 12
Linux drivers, installing 17
modules, loading 66
offload, native 25
on Ubuntu host 64
packages for Ubuntu 64
running traffic on SET 177
services, Linux 67
starting 67
user space applications 17
verifying 77
virtual switch, creating Hyper-V 171
RDMA over Converged Ethernet, See RoCE
reduced instruction set computer, See RISC
related materials xvii
remote direct memory access, See RDMA
removing
VMware driver 30
Windows drivers 20
requirements
EMI and EMC xxiii
host hardware 7
host OS, minimum 8
RHEL
iSER with iWARP, configuring 160
RHEL, configuring iSER 155
RISC
definition of 229
feature support 3
RoCE
configuration for RHEL 63
configuration for SLES 64
configuring on Windows Nano Server 206
definition of 229
241
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
SET
creating virtual switch with 176
defined 175
procedures 175
RDMA traffic, running 177
VLAN ID, assigning 176
SFF8024-compliant specifications, supported
217
SFP+ Flow Control, troubleshooting Linux
issue 213
SFPs, specifications supported 217
single root input/output virtualization, See
SR-IOV
SLES, iSER with iWARP, configuring 160
SLES 12, configuring iSER 159
slipstream.bat script file 131
slipstreaming drivers into Windows image files
131
small computer system interface, See SCSI
SMB drive 62
SMB drive, mapping 174
software
downloading updates xx
EULA xx
software-defined network, deploying 192
specifications, supported cables and optical
modules 217
specifications, supported standards 5
src RPM package 15
SR-IOV
definition of 230
configuring on Linux 143
configuring on VMware 150
configuring on Windows 136
device-level parameter configuration 38
enabling 139
VM switch, creating 187
standards 5
standards specifications 5
statistics, driver 19
Storage Spaces Direct, configuring 193, 196
support, accessing technical xx
RoCE (continued)
driver 12
interfaces, configuring with Hyper-V 170
limitations 54
Linux driver removal 14
priority, specifying 55
statistics, viewing 69
traffic, running 174
verification 67
RoCE over SET procedures 175
RPM package 15, 16
RSS
enabling for virtual switch 186
VMware driver parameter 28, 29
running
RDMA traffic on SET 177
RoCE traffic 174
Rx filters 28
Rx queue 28
Rx Ring Size 18, 29
S
safety precautions 8
safety, product compliance xxiv
scripts, firmware upgrade 31
SCSI
host number 135
host numbers 117, 134
Linux FC transport library 132
stack, VMware 30
scsi_transport_fc.ko, in Linux FCoE offload
132
scsi_transport_iscsi.ko in Linux iSCSI offload
116
SCSI, definition of 230
SDN
deploying 192
in RoCE over SET 175
serializer/deserializer, See SerDes
server message block, See SMB
242
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
TLV
definition of 230
iSCSI lossless 55
parameters 58
RoCE, defined 185
TPA feature, enabling/disabling 28
traffic
RDMA, running on SET 177
RoCE, running 174
statistics, monitoring 191
training from QLogic xxi
transmission control protocol, See TCP
transmission control protocol/Internet protocol,
definition of 230
transmit send offload, See TSO
troubleshooting
checklist 210
fan failure 213
FCoE/iSCSI boot from SAN, Windows 214
Linux driver 211
Linux-specific issues 213
miscellaneous issues 213
VMware driver 212
Windows driver 211
trusted hosts, adding Nano Server to 205
TSO
Linux driver operations 19
VMware driver parameter 29
VXLAN 29
Tx Ring Size 18, 29
type-length-value, See TLV
supported
cables 218
optical modules 218
OS for OFED
products xvi
switches 222
switch
configuration 55
Nexus 6000 Ethernet 55
tested for interoperability with adapter 222
Z9100 Ethernet 57
Switch Embedded Teaming, defined 175
system BIOS, preparing for FCoE build and
boot 124
system requirements for adapter installation 7
T
TAR file, installing Linux drivers 16
target
definition of 230
FCoE boot, configuring 43
ISCSI, configuration 44
iSCSI, configuring 86
targetcli, installing 161
TCP
checksum offloads support 2
definition of 230
TCP segmentation offload, See TSO
TCP/IP, definition of 230
technical support xx
contacting xxi
knowledge database xxii
training from xxi
warranty xxii
terms defined 225
tested cables and optical modules 218
tested switches 222
testing network connectivity
Linux 213
Windows 212
U
Ubuntu
Debian package for drivers 12
drivers, installing 16
drivers, removing 14
iSER, configuring 161
OS support 8
UDP
checksum offloads support 2
definition of 231
243
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
virtual switch
assigning to VMNetworkadapter 140
creating 171
creating with SR-IOV 138
Hyper-V, creating 171, 176
RSS enabling 186
Virtual Switch Manager, creating VM switch
187
VLAN
definition of 231
interface 69
VLAN ID 55, 57, 59
adding to host VNIC 172
assigning on SET 176
VM
definition of 231
creating 190
creating for SR-IOV 140
Device Manager showing adapter 142
drivers for VF, installing 142
hardware, adding 149
setting hardware acceleration 141
switch, SR-IOV-capable 190
VFs, attaching to 152
viewing installed adapter in 142
Windows Nano Server, deploying 203
VM switch
creating with or without SR-IOV 187
switch capability, getting 189
VMMQ, enabling 189
VMMQ
enabling in VM 190
enabling on adapter 186
enabling on VM switch 189
enabling/disabling on management NIC 191
max QPs, setting 186
VNIC quantities 191
VMNetworkadapters in VM 190
VMware
driver parameters, defaults 29
driver parameters, optional 27
driver, FCoE (qedf) 30
driver, installing 26
driver, iSCSI (qedil) 30
UEFI
definition of 231
boot mode, configuring for adapter 124
driver version 38
FCoE boot from SAN 124
HII, configuring FCoE boot mode with 124
uio.ko in Linux iSCSI offload 116
unified extensible firmware interface, See
UEFI
Universal RDMA, support for 1
updates, downloading xx
upgrading
adapter firmware on Linux 32
adapter firmware on Windows 32
adapter firmware on Windows Nano 33
user datagram protocol, See UDP
V
vCenter Plug-In 4
verbose level 28
verifying
Ethernet interface 213
FCoE devices in Linux 134
network connection 213
RDMA 77
VMware driver 212
VF
attaching to VM 152
drivers for VM, installing 142
maximum per PF 28
traffic, monitoring 191
definition of 231
VI, definition of 231
.vib file 27, 26
virtual extensible LAN, See VXLAN
virtual function, See VF
virtual interface, definition of 231
virtual LAN, See VLAN
virtual logical area network, See VLAN
virtual machine, See VM
virtual NIC, See VNIC
244
AH0054601-00 A
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 41000 Series
Windows (continued)
SR-IOV, configuring 136
Storage Spaces Direct, configuring 193
Windows Nano Server
deploying and managing 199
deploying in VM 203
deploying on physical server 201
drivers, installing 22
firmware, updating on 33
management tools for 206
managing remotely 205
Nano ISO image, creating 23
RoCE, configuring 206
roles and features 199
trusted host, adding to 205
Windows PowerShell, managing Nano Server
205
Windows PowerShell, starting session 206
Windows Server
FCoE boot installation 129
minimum host OS requirements 8
Windows Server R2, Microsoft Virtualization
with Hyper-V 213
Windows Setup dialog box, installation error
message 214
wnfwnx2 utility, firmware upgrade utility for
Windows 31
WoL
definition of 231
VMware driver parameter 29
VMware (continued)
driver, removing 30
drivers and packages 25
drivers, downloading 26
drivers, verifying current 212
ESXi driver packages 25
minimum host OS requirements 8
SR-IOV, configuring 150
VMware Update Manager, installing driver with
27
VNIC
adding VLAN ID to 172
creating with RDMA 176
host, adding 173
RDMA, enabling for Hyper-V vSwitch 171
VMMQ quantities 191
VPort, VMMQ max QPs, setting 186
VUM, installing driver with 27
VXLAN
configuring 192
encapsulated task offload 192
filtering 28
offload, enabling 192
SDN, deploying 192
tunneled traffic 28
vxlan_filter_en parameter 28
W
wake on LAN, See WoL
warranty, product xxii
what’s in this guide xvi
Windows
Assessment and Deployment Kit,
downloading 131
drivers, installing 19
drivers, removing 20
drivers, verifying current 211
FCoE boot from SAN 129
firmware, updating on 32
Hyper-Converged system, deploying 194
image files, injecting 131
iWARP, configuring 76
Z
Z9100 Ethernet switch 57
245
AH0054601-00 A
Corporate Headquarters
Cavium, Inc.
2315 N. First Street
San Jose, CA 95131
408-943-7100
International Offices UK | Ireland | Germany | France | India | Japan | China | Hong Kong | Singapore | Taiwan | Israel
Copyright &copy; 2017 Cavium, Inc. All rights reserved worldwide. QLogic Corporation is a wholly owned subsidiary of Cavium, Inc. QLogic, FastLinQ, QConvergeConsole, and QLogic Control Suite
are registered trademarks or trademarks of Cavium, Inc. All other brand and product names are trademarks or registered trademarks of their respective owners.
This document is provided for informational purposes only and may contain errors. Cavium reserves the right, without notice, to make changes to this document or in product design or specifications.
Cavium disclaims any warranty of any kind, expressed or implied, and does not guarantee that any results or performance described in the document will be achieved by you. All statements
regarding Cavium’s future direction and intent are subject to change or withdrawal without notice and represent goals and objectives only.
">
Hi! I've reviewed the Cavium FastLinQ 41000 User's Guide. This guide covers installation, driver installation for various OSes (including Windows, Linux, and VMware), RDMA/RoCE configuration, and troubleshooting for these network adapters. Ask me about the FastLinQ 41000.
Show related prompts
Assistant typing
Assistant is out of keyboard. Please, try again later.
Key features

iSCSI Support
FCoE Support
RoCE Support
iWARP Support
SR-IOV Support
VMMQ Support
VXLAN Offload
Frequently asked questions

The FastLinQ 41000 series network adapters support a variety of protocols including iSCSI, FCoE, RoCE, and iWARP.
The adapters are compatible with a range of operating systems, including Windows, Linux, and VMware.
Key features include SR-IOV, VMMQ, and VXLAN offload.
You can configure the adapters using the QLogic Control Suite CLI, QLogic QConvergeConsole GUI, or FastLinQ ESXCLI VMware Plug-in.
What protocols are supported by the FastLinQ 41000 series network adapters?

What operating systems are compatible with the FastLinQ 41000 series network adapters?

What are some of the key features of the FastLinQ 41000 series network adapters?

How do I configure the FastLinQ 41000 series network adapters?

Related

Cavium QL41162HLRJ-BK User manual

Dell QLogic Family of Adapters User's Guide

Fujitsu CS-IUP-R2545M Manual

QLogic FastLinQ 45000 Series Intelligent Ethernet Adapters User’s Guide

HPE FlexFabric Adapters User Guide: AI Chat & PDF

QLogic FastLinQ 45000 Series User's Guide

Fujitsu CS-IDP-R2545M Manual

Dell QLogic Family of Adapters Benutzerhandbuch

Cavium FastLinQ 45000 Series Network Adapter User's Guide

Dell QLogic Family of Adapters 사용자 설명서

Dell QLogic Family of Adapters Guía del usuario

Panasonic EB-TX320 Series Cordless Phone Operating Instructions
