QLogic FastLinQ 45000 QL45211HLCU-CK/SP/BK, QL45212HLCU-CK/SP/BK, QL45411HLCU-CK/SP/BK, QL45412HLCU-CK/SP/BK, QL45611HLCU-CK/SP/BK Intelligent Ethernet Adapter User’s Guide

Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Testing Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Testing Network Connectivity for Windows . . . . . . . . . . . . . . . . . . . . .

Testing Network Connectivity for Linux . . . . . . . . . . . . . . . . . . . . . . . .

Microsoft Virtualization with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Miscellaneous. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Adapter LEDS

Cables and Optical Modules

Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

125

ix BC0154501-00 F

User’s Guide—25Gb/40Gb/100Gb Intelligent Ethernet Adapters

QLogic FastLinQ 45000 Series

List of Figures

Figure

3-1 Power Management Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-1 Main Configuration Page—Default Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-2 Main Configuration Page—NPAR Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-3 Firmware Information Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-4 Device Level Configuration Pages (25Gb and 40Gb Adapters). . . . . . . . . . . . . . . .

5-5 Device Level Configuration Page (100Gb Adapter) . . . . . . . . . . . . . . . . . . . . . . . . .

5-6 Partitions Configuration Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-7 Global Bandwidth Allocation Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-8 Partition 1 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Page

x BC0154501-00 F

User’s Guide—25Gb/40Gb/100Gb Intelligent Ethernet Adapters

QLogic FastLinQ 45000 Series

List of Tables

Table Page

2-1 Host Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2-2 Host Operating System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3-1 QLogic 45000 Series Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3-2 qede Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3-3 Linux Driver Parameter Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3-4 VMware Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3-5 VMware Driver Parameter Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-1 Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-2 Device Level Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-1 Operating System Support for RoCE v1/v2 and OFED . . . . . . . . . . . . . . . . . . . . . .

8-1 Roles and Features of Nano Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

108

A-1 Adapter Port Link and Activity LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

123

B-1 Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

125

x BC0154501-00 F

Preface

NOTE

QLogic

now supports QConvergeConsole

(QCC) GUI as the only GUI management tool across all QLogic adapters. QLogic Control Suite (QCS)

GUI is no longer supported for the QLogic FastLinQ™ 45000 Series

Adapters and adapters based on 57xx/57xxx controllers, and has been replaced by the QCC GUI management tool. The QCC GUI provides single-pane-of-glass GUI management for all QLogic adapters.

In Windows environments, when you run QCS CLI and Management Agents

Installer, it will uninstall the QCS GUI (if installed on the system) and any related components from your system. To obtain the new GUI, download

QCC GUI for your adapter from the QLogic Downloads Web page: driverdownloads.qlogic.com

Overview

This user’s guide describes the following QLogic FastLinQ products:

 25Gb Intelligent Ethernet Adapters:

 QL45211HLCU-CK/SP/BK

 QL45212HLCU-CK/SP/BK



40Gb Intelligent Ethernet Adapters

 QL45411HLCU-CK/SP/BK

 QL45412HLCU-CK/SP/BK

 100Gb Intelligent Ethernet Adapters:



QL45611HLCU-CK/SP/BK

Intended Audience

This guide is intended for system administrators and other technical staff members responsible for configuring and managing adapters installed on servers in Windows

, Linux

, or VMware

environments.

Linux and VMware only xi BC0154501-00 F

Preface

What Is in This Guide

This preface specifies the intended audience, explains the typographic conventions used in this guide, lists related documents, and provides technical support and contact information. The remainder of this guide is organized into the following sections:



Chapter 1, Product Overview

provides a product functional description, a list of features, a list of supported operating systems, and the adapter specifications.



Chapter 2, Installing the Hardware

describes how to install the adapter including the list of system requirements and a preinstallation checklist.



Chapter 3, Installing Drivers describes the installation of the adapter drivers.



Chapter 4, Firmware Upgrade Utility

describes the use of the utility to upgrade adapter firmware and boot code.



Chapter 5, Preboot Adapter Configuration describes the preboot adapter

configuration tasks using the Human Infrastructure Interface (HII) application.



Chapter 6, Configuring RoCE describes how to configure the adapter, the

Ethernet switch, and the host to use RDMA over converged Ethernet

(RoCE).



Chapter 7, iSCSI Extensions for RDMA

describes how to configure iSCSI

Extensions for RDMA (iSER) for RHEL and SLES.



Chapter 8, Windows Server 2016

describes the Windows Server 2016 features.



Chapter 9, Troubleshooting describes a variety of troubleshooting methods

and resources.



Appendix A, Adapter LEDS

describes the adapter LEDs and their significance.



Appendix B, Cables and Optical Modules

describes the cables and optical modules that the 45000 Series Adapters support.

xii BC0154501-00 F

Preface

Documentation Conventions

This guide uses the following documentation conventions:



NOTE

provides additional information.

CAUTION

without an alert symbol indicates the presence of a hazard that could cause damage to equipment or loss of data.



CAUTION

with an alert symbol indicates the presence of a hazard that could cause minor or moderate injury.



WARNING

injury or death.

indicates the presence of a hazard that could cause serious

 Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this guide, and links to Web sites are shown in underlined blue . For example:

 Table 9-2 lists problems related to the user interface and remote agent.

 See “Installation Checklist” on page 6 .

 For more information, visit www.qlogic.com

 Text in bold font indicates user interface elements such as a menu items, buttons, check boxes, or column headings. For example:

 Click the Start button, point to Programs, point to Accessories, and then click Command Prompt.

 Under Notification Options, select the Warning Alarms check box.

 Text in Courier font indicates a file name, directory path, or command line text. For example:

 To return to the root directory from anywhere in the file structure:

Type cd /root and press ENTER.

 Enter the following command: sh ./install.bin

 Key names and key strokes are indicated with UPPERCASE:

 Press CTRL+P.

 Press the UP ARROW key.

xiii BC0154501-00 F

Preface

License Agreements

 Text in italics indicates terms, emphasis, variables, or document titles. For example:

 For a complete listing of license agreements, refer to the QLogic

Software End User License Agreement.

 What are shortcut keys?

 To enter the date type mm/dd/yyyy (where mm is the month, dd is the day, and yyyy is the year).

 Topic titles between quotation marks identify related topics either within this manual or in the online help, which is also referred to as the help system throughout this document.

License Agreements

Refer to the QLogic Software End User License Agreement for a complete listing of all license agreements affecting this product.

Technical Support

Customers should contact their authorized maintenance provider for technical support of their QLogic products. QLogic-direct customers may contact QLogic

Technical Support; others will be redirected to their authorized maintenance provider. Visit the QLogic support Web site listed in

Contact Information

for the latest firmware and software updates.

For details about available service plans, or for information about renewing and extending your service, visit the Service Program Web page: www.qlogic.com/Support/Pages/ServicePrograms.aspx

Downloading Updates

The QLogic Web site provides periodic updates to product firmware, software, and documentation.

To download firmware, software, and documentation:

Go to the QLogic Downloads and Documentation page: driverdownloads.qlogic.com

Type the QLogic model name in the search box.

In the search results list, locate and select the firmware, software, or documentation for your product.

View the product details Web page to ensure that you have the correct firmware, software, or documentation. For additional information, click

Read Me and Release Notes under Support Files.

xiv BC0154501-00 F

Preface

Technical Support

Click Download Now.

Save the file to your computer.

If you have downloaded firmware, software, drivers, or boot code, follow the installation instructions in the Read Me file.

Instead of typing a model name in the search box, you can perform a guided search as follows:

Click the product type tab: Adapters, Switches, Routers, or ASICs.

Click the corresponding button to search by model or operating system.

Click an item in each selection column to define the search, and then click

Go.

Locate the firmware, software, or document you need, and then click the item’s name or icon to download or open the item.

Training

QLogic Global Training maintains a Web site at www.qlogictraining.com

offering online and instructor-led training for all QLogic products. In addition, sales and technical professionals may obtain Associate and Specialist-level certifications to qualify for additional benefits from QLogic.

Contact Information

QLogic Technical Support for products under warranty is available during local standard working hours excluding QLogic Observed Holidays. For customers with extended service, consult your plan for available hours. For Support phone numbers, see the Contact Support link: support.qlogic.com

Support Headquarters

QLogic Web Site

Technical Support Web Site

Technical Support E-mail

Technical Training E-mail

QLogic Corporation

12701 Whitewater Drive

Minnetonka, MN 55343 USA www.qlogic.com

support.qlogic.com

[email protected]

xv BC0154501-00 F

Preface

Legal Notices

Knowledge Database

The QLogic knowledge database is an extensive collection of QLogic product information that you can search for specific solutions. QLogic is constantly adding to the collection of information in the database to provide answers to your most urgent questions. Access the database from the QLogic Support Center: support.qlogic.com

Legal Notices

Warranty

For warranty details, please check the QLogic Web site: www.qlogic.com/Support/Pages/Warranty.aspx

Laser Safety

FDA Notice

This product complies with DHHS Rules 21CFR Chapter I, Subchapter J. This product has been designed and manufactured according to IEC60825-1 on the safety label of laser product.

CLASS I LASER

Class 1 Laser Product

Appareil laser de classe 1

Produkt der Laser Klasse 1

Luokan 1 Laserlaite

Caution—Class 1 laser radiation when open. Do not view directly with optical instruments

Attention—Radiation laser de classe 1. Ne pas regarder directement avec des instruments optiques.

Vorsicht—Laserstrahlung der Klasse 1 bei geöffneter Abdeckung. Direktes Ansehen mit optischen Instrumenten vermeiden.

Varoitus—Luokan 1 lasersäteilyä, kun laite on auki. Älä katso suoraan laitteeseen käyttämällä optisia instrumenttej.

xvi BC0154501-00 F

Preface

Legal Notices

Agency Certification

The following sections summarize the EMC and EMI test specifications performed on the 45000 Series Adapters.

EMI and EMC Requirements

FCC Part 15 compliance: Class A

FCC compliance information statement: This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation.

ICES-003 Compliance: Class A

This Class A digital apparatus complies with Canadian ICES-003.Cet appareil numériqué de la classe A est conformé à la norme NMB-003 du Canada.

CE Mark 2004/108/EC EMC Directive Compliance:

EN55022:2010 Class A1:2007/CISPR22:2006: Class A

EN55024:2010

EN61000-3-2: Harmonic Current Emission

EN61000-3-3: Voltage Fluctuation and Flicker

Immunity Standards

EN61000-4-2: ESD

EN61000-4-3: RF Electro Magnetic Field

EN61000-4-4: Fast Transient/Burst

EN61000-4-5: Fast Surge Common/ Differential

EN61000-4-6: RF Conducted Susceptibility

EN61000-4-8: Power Frequency Magnetic Field

EN61000-4-11: Voltage Dips and Interrupt

VCCI: 2010-04 Class A

AS/NZS CISPR22: Class A

KCC: Class A

Korea RRA Class A Certified

Product Name/Model: Converged Network

Adapters and Intelligent Ethernet Adapters

Certification holder: QLogic Corporation

Manufactured date: Refer to date code listed on product

Manufacturer/Country of origin: QLogic

Corporation/USA xvii BC0154501-00 F

Preface

Legal Notices

A class equipment

(Business purpose info/telecommunications equipment)

As this equipment has undergone EMC registration for business purpose, the seller and/or the buyer is asked to beware of this point and in case a wrongful sale or purchase has been made, it is asked that a change to household use be made.

Korean Language Format—Class A

Product Safety Compliance

UL, cUL product safety:

UL60950-1 (2nd Edition), 2007-03-3-27

UL CSA C22.2 60950-1-07 (2nd Edition)

Use only with listed ITE or equivalent.

Complies with 21 CFR 1040.10 and 1040.11.

2006/95/EC low voltage directive:

TUV EN60950-1:2006+A11+A1+A12

IEC60950-1 2nd Edition (2005) CB

CB Certified to IEC 60950-1 2nd Edition xviii BC0154501-00 F

Product Overview

This chapter provides information about the following topics:



Functional Description



Features



Adapter Management



Adapter Specifications

1 BC0154501-00 F

1–Product Overview

Functional Description

The QLogic FastLinQ 45000 Series Adapter is a 25/40/100Gb Intelligent Ethernet

Adapter that is designed to perform accelerated data networking for server systems. The 45000 Series Adapter includes a 25/40/100Gb Ethernet MAC with full-duplex capability.

Using the operating system’s teaming feature, you can split your network into virtual LANs (VLANs) as well as group multiple network adapters together into teams to provide network load balancing and fault tolerance. For more information about teaming, see your operating system documentation.

Features

The 45000 Series Adapters provide the following features. Some features may not be available on all adapters.

 NIC partitioning (NPAR)

 Remote direct memory access over converged Ethernet, versions 1 and 2

(RoCE v1/v2)

 Data center bridging (DCB)

 Enhanced transmission selection (ETS; IEEE 802.1Qaz)

 Priority-based Flow Control (PFC; IEEE 802.1Qbb)

 Data center bridging eXchange protocol (DCBX; CEE version 1.01,

IEEE)

 Single-chip solution

 25/40/100Gb MAC

 SerDes interface for direct attach copper (DAC) transceiver connection

 PCI Express Gen3 x8 (2x25G NIC)/x16 (2x40G/1x100G NIC)

 Zero copy capable hardware

 Performance features

 TCP, IP, UDP checksum offloads

 TCP segmentation offload (TSO)

 Large segment offload (LSO)

 Generic segment offload (GSO)

 HW based Generic Receive Offload (HW-GRO)

 Large receive offload (LRO)

 Receive segment coalescing (RSC)

2 BC0154501-00 F

1–Product Overview

Features

 Microsoft

dynamic virtual machine queue (VMQ) and Linux multiqueue

 Adaptive interrupts

 Transmit/receive side scaling (TSS/RSS)

 Stateless offloads for NVGRE/VXLAN L2/L3 GRE tunneled traffic



Manageability

 System management bus (SMB) controller

 Advanced Configuration and Power Interface (ACPI) 1.1a compliant

(multiple power modes)



Network controller-sideband interface (NC-SI) support

 Advanced network features



Jumbo frames (up to 9,600 bytes). The OS and the link partner must support jumbo frames.

 Virtual LANs (VLAN)

 Flow control (IEEE Std 802.3x)



Logical link control (IEEE Std 802.2)

 High-speed on-chip reduced instruction set computer (RISC) processor



Integrated 96KB frame buffer memory

 1,024 classification filters

 Support for multicast addresses through 128-bit hashing hardware function

 Serial flash NVRAM memory



PCI Power Management Interface (v1.1)

 64-bit base address register (BAR) support

 EM64T processor support

 Virtualization

This feature requires support for Hypervisor to use the offloads.

3 BC0154501-00 F

1–Product Overview

Adapter Management

The following applications are available to manage 45000 Series Adapters:



QLogic Control Suite CLI



QLogic QConvergeConsole Graphical User Interface



QLogic QConvergeConsole vCenter

Plug-In



FastLinQ ESXCLI VMware Plug-in

QLogic Control Suite CLI

The QCS CLI is a console application that you can run from a Windows command prompt or a Linux terminal console. Use the QCS CLI to manage QLogic 45000

Series Intelligent Ethernet Adapters or any QLogic adapter based on 57xx/57xxx controllers on both local and remote computer systems. For information about installing and using the QCS CLI, see the QLogic Control Suite CLI User’s Guide.

NOTE

Although QLogic Control Suite CLI does not run on a Windows Nano Server shell, you can install the QLNXRemote agent on a Nano Server system and manage it remotely through QCS CLI using the addhost (Add Host) and

removehost (Remove Host) commands. For QCS CLI to connect to the remote QLNXRemote Nano agent, you must disable or correctly configure the firewall on the Nano system. For instructions on installing the

QLNXRemote agent on a Nano Server, refer to the QConvergeConsole

Windows Agent Installers Readme located inside the QCS CLI for Windows download available on www.qlogic.com

QLogic QConvergeConsole Graphical User Interface

The QCC GUI is a Web-based management tool for configuring and managing

QLogic Fibre Channel adapters and Intelligent Ethernet adapters. You can use the

QCC GUI on Windows and Linux platforms to manage QLogic 45000 Series

Intelligent Ethernet Adapters on both local and remote computer systems. For information about installing the QCC GUI, see the QConvergeConsole GUI

Installation Guide. For information about using the QCC GUI, see the online help.

4 BC0154501-00 F

1–Product Overview

Adapter Specifications

QLogic QConvergeConsole vCenter

Plug-In

The QCC vCenter

Plug-In is a Web-based management tool that is integrated into the VMware vCenter Server for configuring and managing QLogic Fibre

Channel adapters and Intelligent Ethernet adapters in a virtual environment. You can use the vCenter Plug-in VMware vSphere clients to manage QLogic 45000

Series Intelligent Ethernet Adapters. For information about installing and using the vCenter Plug-in, see the QConvergeConsole Plug-ins for vSphere User’s Guide.

FastLinQ ESXCLI VMware Plug-in

The FastLinQ ESXCLI plug-in extends the capabilities of the ESX® command line interface to manage QLogic 3400, 8400, and 45000 Series Adapters installed in

VMware ESX/ESXi hosts. For information about installing and using the ESXCLI plug-in, see the FastLinQ ESXCLI VMware Plug-in User’s Guide.

Adapter Specifications

Physical Characteristics

The 45000 Series Adapter is a standard PCI Express ® card and ships with either a full-height or a low-profile bracket for use in a standard PCIe

slot.

Standards Specifications



PCI Express Base Specification, rev. 3.0

 PCI Express Card Electromechanical Specification, rev. 3.0

 PCI Bus Power Management Interface Specification, rev. 1.2

 802.1q (VLAN)



802.1AX (Link Aggregation)

 802.1ad (QinQ)

 802.1p (Priority Encoding)

 IPv4 (RFQ 791)

 IPv6 (RFC 2460)

 Applicable IEEE802.3 Ethernet

5 BC0154501-00 F

Installing the Hardware



System Requirements



Safety Precautions



Preinstallation Checklist



Installing the Adapter

6 BC0154501-00 F

2–Installing the Hardware

System Requirements

Before you install a QLogic 45000 Series Adapter, verify that your system meets the hardware and operating system requirements shown in

Tables 2-1

and

2-2 :.

For a complete list of supported operating systems, see driverdownloads.qlogic.com.

Table 2-1. Host Hardware Requirements

Hardware

Architecture

PCIe

Memory

Cables and Optical

Modules

Requirement

IA-32 or EMT64 that meets operating system requirements

PCIe Gen2 x8 (2x25G NIC)/x16 (2x40G/1x100G NIC)

PCIe Gen3 x8 (2x25G NIC)/x16 (2x40G/1x100G NIC)

Full dual-port 40Gb and single port 100Gb bandwidth is supported on PCIe Gen3 x16 or faster slots.

Full dual-port 25Gb bandwidth is supported on PCIe Gen3 x8 or faster slots.

8GB RAM (minimum)

The 45000 Series Adapters support a variety of 10G, 25G, and

100G cables and optical modules. See

“Tested Cables and Optical Modules” on page 125

VMware

Table 2-2. Host Operating System Requirements

Operating

System

Windows Server

Linux

Requirement

2012 R2, 2016, Nano

RHEL 6.7, 6.8, 7.1, 7.2

SLES 11 SP4, 12, 12 SP1

CentOS

7.2 and later

Ubuntu

14.04 LTS, 16.04 LTS

ESXi 6.0 U2 and later for 25G adapters

ESXi 5.5.x, 6.0.x and later for 40G adapters a

25G and 100G only

7 BC0154501-00 F

2–Installing the Hardware

Safety Precautions

WARNING

The adapter is being installed in a system that operates with voltages that can be lethal. Before you open the case of your system, observe the following precautions to protect yourself and to prevent damage to the system components.

 Remove any metallic objects or jewelry from your hands and wrists.

 Make sure to use only insulated or nonconducting tools.

 Verify that the system is powered OFF and is unplugged before you touch internal components.

 Install or remove adapters in a static-free environment. The use of a properly grounded wrist strap or other personal antistatic devices and an antistatic mat is strongly recommended.

8 BC0154501-00 F

2–Installing the Hardware

Preinstallation Checklist

Before installing the adapter, do the following:

Verify that your system meets the hardware and software requirements

listed under “System Requirements” on page 7 .

Verify that your system is using the latest BIOS.

NOTE

If you acquired the adapter software on a disk or from the QLogic Web

Site driverdownloads.qlogic.com

, verify the path to the adapter driver files.

If your system is active, shut it down.

When system shutdown is complete, turn off the power and unplug the power cord.

Remove the adapter from its shipping package and place it on an anti-static surface.

Check the adapter for visible signs of damage, particularly on the edge connector. Never attempt to install a damaged adapter.

Installing the Adapter

The following instructions apply to installing the QLogic 45000 Series Adapters in most systems. Refer to the manuals that were supplied with your system for details about performing these tasks.

Review “Safety Precautions” on page 8

and

“Preinstallation Checklist” on page 9 . Before you install the adapter, ensure that the system power is OFF,

the power cord is unplugged from the power outlet, and that you are following proper electrical grounding procedures.

Open the system case, and select the slot that matches the adapter size, which can be PCIe Gen2 x8 or PCIe Gen3 x8. A lesser width adapter can be seated into a greater width slot (x8 in a x16), but a greater width adapter cannot be seated into a lesser width slot (x8 in a x4). If you do not know how to identify a PCIe slot, refer to your system documentation.

Remove the blank cover-plate from the slot that you selected.

Align the adapter connector edge with the PCI Express connector slot in the system.

9 BC0154501-00 F

2–Installing the Hardware

Installing the Adapter

Applying even pressure at both corners of the card, push the adapter card into the slot until it is firmly seated. When the adapter is properly seated, the adapter port connectors are aligned with the slot opening, and the adapter faceplate is flush against the system chassis.

CAUTION

Do not use excessive force when seating the card, as this may damage the system or the adapter. If you have difficulty seating the adapter, remove it, realign it, and try again.

Secure the adapter with the adapter clip or screw.

Close the system case and disconnect any personal anti-static devices.

Connect the host system Ethernet segment to the adapter fiber optic connector.

10 BC0154501-00 F

Installing Drivers

NOTE

QLogic now supports QCC GUI as the only GUI management tool across all

QLogic adapters. QCS GUI is no longer supported for the 45000 Series

Adapters and adapters based on 57xx/57xxx controllers, and has been replaced by the QCC GUI management tool. The QCC GUI provides single-pane-of-glass GUI management for all QLogic adapters.

In Windows environments, when you run QCS CLI and Management Agents

Installer, it will uninstall the QCS GUI (if installed on the system) and any related components from your system. To obtain the new GUI, download

QCC GUI for your adapter from the QLogic Downloads Web page: http://driverdownloads.qlogic.com

This chapters provides information about the following topics:



Linux Driver Software



Windows Driver Software



VMware Driver Software

11 BC0154501-00 F

3–Installing Drivers

Linux Driver Software

This section describes how to install Linux drivers with or without RoCE. It also describes the Linux driver optional parameters, default values, messages, and statistics.



Installing the Linux Drivers without RoCE



Installing the Linux Drivers with RoCE



Linux Driver Optional Parameters



Linux Driver Parameter Defaults



Linux Driver Messages



Statistics

The 45000 Series Adapter Linux drivers and supporting documentation are available at driverdownloads.qlogic.com

Table 3-1 describes the 45000 Series

Adapter Linux drivers.

Table 3-1. QLogic 45000 Series Linux Drivers

Linux

Driver

qed

Description

qede qedr

The Linux core module manages all PCI device resources (registers, host interface queues, and so on). The qed core module directly controls the firmware, handles interrupts, and is the interface for the qede driver. The qed core module requires Linux kernel version 2.6.32 or later. Testing was concentrated on the x86_64 architecture.

Linux Ethernet driver for the 45000 Series Adapter. This driver directly controls the hardware and is responsible for sending and receiving Ethernet packets on behalf of the Linux host networking stack. This driver also receives and processes device interrupts, on behalf of itself (for L2 networking). The qede driver requires Linux kernel version 2.6.32 or later.

Testing was concentrated on the x86_64 architecture.

Linux RDMA over converged Ethernet (RoCE) driver. This driver works in the open fabric enterprise distributions (OFED) environment in conjunction with the qed core module and the qede Ethernet driver. RDMA user space applications also requires that the libqedr user library be installed on the server.

The Linux drivers can be installed using a source Red Hat packet manager (RPM) package or a kmod RPM package. The RHEL RPM packages are as follows:

 qlgc-fastlinq-<version>.<OS>.src.rpm

 qlgc-fastlinq-kmp-default-<version>.<arch>.rpm

12 BC0154501-00 F

3–Installing Drivers

Linux Driver Software

The SLES source and kmp RPM packages are as follows:

 qlgc-fastlinq-<version>.<OS>.src.rpm

 qlgc-fastlinq-kmp-default-<version>.<OS>.<arch>.rpm

The Ubuntu DEB package is as follows:

 qlgc-fastlinq-dkms-<version>_all.deb

The following kmod RPM installs Linux drivers on SLES hosts running the XEN hypervisor:

 qlgc-fastlinq-kmp-xen-<version>.<OS>.<arch>.rpm

The following source RPM installs the RDMA library code on RHEL and SLES hosts:

 qlgc-libqedr-<version>.<OS>.<arch>.src.rpm

The following source code tar BZ2 installs Linux drivers on RHEL and SLES hosts: fastlinq-linux-<version>.tar.bz2

NOTE

For network installations through NFS, FTP, or HTTP (using a network boot disk), a driver disk that contains the qede driver may be needed. Linux boot drivers can be compiled by modifying the Makefile and the make environment.

Installing the Linux Drivers without RoCE

NOTE

When using the following procedures in a CentOS environment, follow the instructions for RHEL. When performing these tasks in an Ubuntu environment, see the Install.txt document in the Ubuntu package that is available at driverdownloads.qlogic.com

for your adapter.

To install the Linux drivers without RoCE, do the following:

Download the 45000 Series Adapter Linux drivers from driverdownloads.qlogic.com

Remove the existing Linux drivers, as described in “Removing the Linux

Drivers” on page 14 .

Install the new Linux drivers using one of the following methods:



“Installing Linux Drivers Using the src RPM Package” on page 16

13 BC0154501-00 F

3–Installing Drivers

Linux Driver Software



“Installing Linux Drivers Using the kmp/kmod RPM Package” on page 17



“Installing Linux Drivers Using the TAR File” on page 17

Removing the Linux Drivers

There are two procedures for removing Linux drivers: one for a non-RoCE environment and another for a RoCE environment. Choose the procedure that matches your environment.



Removing Linux Drivers in a non-RoCE Environment



Removing Linux Drivers in a RoCE Environment

Removing Linux Drivers in a non-RoCE Environment

To remove Linux drivers in a non-RoCE environment:

Unload and remove the Linux drivers.

 If the Linux drivers were installed using an RPM package, type the following commands:

rmmod qede rmmod qed depmod -a

rpm -e qlgc-fastlinq-kmp-default-<version>.<arch>

 If the Linux drivers were installed using a TAR file, type the following commands:

rmmod qede rmmod qed depmod -a

 For Ubuntu, type the following command: dpkg -r fastlinq-dkms

Delete the qed.ko, qede.ko, and qedr.ko files from the directory in which they reside. For example, in SLES, type the following commands:

cd /lib/modules/<version>/updates/qlgc-fastlinq rm -rf qed.ko

rm -rf qede.ko

rm -rf qedr.ko

depmod -a

14 BC0154501-00 F

3–Installing Drivers

Linux Driver Software

Removing Linux Drivers in a RoCE Environment

To remove Linux drivers in a RoCE environment:

Unload and remove the Linux drivers.

modprobe -r qedr modprobe -r qede modprobe -r qed depmod -a

Remove driver module files.

 If the drivers were installed using an RPM package, type the following command: rpm -e qlgc-fastlinq-kmp-default-<version>.<arch>

 If the drivers were installed using a TAR file, type the following commands for your operating system:

For RHEL and CentOS: cd /lib/modules/<version>/extra/qlgc-fastlinq rm -rf qed.ko qede.ko qedr.ko

For SLES: cd /lib/modules/<version>/updates/qlgc-fastlinq rm -rf qed.ko qede.ko qedr.ko

15 BC0154501-00 F

3–Installing Drivers

Linux Driver Software

Installing Linux Drivers Using the src RPM Package

To install Linux drivers using the src RPM package:

Type the following at a command prompt:

rpm -ivh qlgc-fastlinq-<version>.src.rpm

Change the directory to the RPM path and build the binary RPM for the kernel.

For RHEL and CentOS:

cd /root/rpmbuild rpmbuild -bb SPECS/fastlinq-<version>.spec

For SLES:

cd /usr/src/packages rpmbuild -bb SPECS/fastlinq-<version>.spec

Install the newly compiled RPM:

rpm -ivh RPMS/<arch>/qlgc-fastlinq-<version>.<arch>.rpm

NOTE

The --force option may be needed on some Linux distributions if conflicts are reported.

The drivers will be installed in the following paths.

For SLES:

/lib/modules/<version>/updates/qlgc-fastlinq

For RHEL and CentOS:

/lib/modules/<version>/extra/qlgc-fastlinq

Turn on all ethX interfaces.

ifconfig <ethX> up

For SLES, use YaST to configure your Ethernet interfaces to automatically start at boot by setting a static IP address or enabling DHCP on the interface.

16 BC0154501-00 F

3–Installing Drivers

Linux Driver Software

Installing Linux Drivers Using the kmp/kmod RPM Package

To install kmod RPM package:

Type the following command at a command prompt:

rpm -ivh qlgc-fastlinq-<version>.<arch>.rpm

Reload the driver:

modprobe -r qede modprobe qede

Installing Ubuntu Linux Drivers

To install Ubuntu Linux drivers, type the following command:

dpkg -i fastlinq-dkms_<version>_all.deb

Installing Linux Drivers Using the TAR File

To install Linux drivers using the TAR file:

Create a directory and extract the TAR files to the directory:

tar xjvf fastlinq-minor-<version>.tar.bz2

Change to the recently created directory, and then install the drivers:

cd fastlinq-minor-<version> make clean; make install

The qed and qede drivers will be installed in the following paths.

For SLES:

/lib/modules/<version>/updates/qlgc-fastlinq

For RHEL and CentOS:

/lib/modules/<version>/extra/qlgc-fastlinq

Test the drivers by loading them (unload the existing drivers first, if necessary):

rmmod qede rmmod qed modprobe qed modprobe qede

17 BC0154501-00 F

3–Installing Drivers

Linux Driver Software

Installing the Linux Drivers with RoCE

NOTE

for your adapter.

To install Linux drivers in an inbox OFED environment:

Download the 45000 Series Adapter Linux drivers from driverdownloads.qlogic.com

Configure RoCE on the adapter, as described in “Configuring RoCE on the

Adapter for Linux” on page 54

Remove existing Linux drivers, as described in “Removing the Linux Drivers” on page 14 .

Install the new Linux drivers using one of the following methods:



“Installing Linux Drivers Using the src RPM Package” on page 16



“Installing Linux Drivers Using the kmp/kmod RPM Package” on page 17



“Installing Linux Drivers Using the TAR File” on page 17

Install libqedr libraries to work with inbox OFED. The libqedr RPM is available only for inbox OFED. Type the following command: rpm –ivh qlgc-libqedr-<version>.<arch>.rpm

Test the drivers by loading them: modprobe qed modprobe qede modprobe qedr

18 BC0154501-00 F

3–Installing Drivers

Linux Driver Software

Linux Driver Optional Parameters

Table 3-2 describes the qede driver optional parameters.

Parameter

debug int_mode gro_disable no_tracing

Table 3-2. qede Driver Optional Parameters

Description

Specifies the debug message level.

Forces interrupts on all 45000 Series adapters in the system using legacy INTx mode (1) or MSI mode (2).

Enables (0) or disables (1) the HW GRO feature. The default is 0.

Enables (0) or disables (1) the transmission of all driver logs to the ftrace trace buffer.

Linux Driver Parameter Defaults

Table 3-3 lists the qed and qede Linux driver parameter defaults.

Speed

MSI/MSI-X

Flow Control

Table 3-3. Linux Driver Parameter Defaults

Parameter

MTU

Rx Ring Size

Tx Ring Size

Coalesce Rx Microseconds

Coalesce Tx Microseconds

TSO

qed Driver Default

Auto-negotiation with speed advertised

Enabled

—

qede Driver Default

Auto-negotiation with speed advertised

Enabled

Auto-negotiation with Rx and Tx advertised

1500 (range is 46–9600)

8191 (range is 128 - 8191)

4078 (range is 128 - 8191)

24 (range is 0–255)

Enabled

19 BC0154501-00 F

3–Installing Drivers

Linux Driver Software

Linux Driver Messages

To set the Linux driver message detail level, use one of the following commands:

 ethtool -s <interface> msglvl <value>

 modprobe qede debug=<value> where <value> represents bits 0–15, which are standard Linux networking values, and bits 16 and greater are driver specific.

Statistics

Detailed statistics and configuration information can be viewed using the ethtool utility. See the ethtool man page for more information.

20 BC0154501-00 F

3–Installing Drivers

Windows Driver Software

NOTE

Currently, Windows supports only 25G and 100G 45000 Series Adapters.



Installing the Windows Drivers



Removing the Windows Drivers



Managing Adapter Properties



Setting Power Management Options



Setting Power Management Options

Installing the Windows Drivers

NOTE

For information on how to build the Windows Nano virtual hard disk, go to: https://technet.microsoft.com/en-us/windows-server-docs/compute/nano-ser ver/getting-started-with-nano-server

To install the Windows drivers:

Go to driverdownloads.qlogic.com

and download the Windows device drivers for the 45000 Series Adapter.

Launch the QLogic FastLinQ™ Driver Install package (setup.exe) that was downloaded.

In the InstallShield Wizard Welcome dialog box, click Next. Follow the wizard instructions, accepting the terms of the license agreement and clicking Install to start the installation. When the installation is complete, click Finish.

Verify that the Windows drivers have been installed. Click Start, Control

Panel, Programs, Programs and Features, and then locate QLogic

FastLinQ Driver Installer in the list of programs.

Removing the Windows Drivers

To remove the Windows drivers:

In Control Panel, click Programs, Programs and Features.

In the list of programs, select QLogic FastLinQ Driver Installer, and then click Uninstall. Follow the instructions to remove the drivers.

21 BC0154501-00 F

3–Installing Drivers

Windows Driver Software

Managing Adapter Properties

To view or change the 45000 Series Adapter properties

In Control Panel, click Device Manager.

Click the Advanced section of the chosen port.

Setting Power Management Options

You can set power management options to allow the operating system to turn off the controller to save power or to allow the controller to wake up the computer. If the device is busy doing something (servicing a call, for example), the operating system will not shut down the device. The operating system attempts to shut down every possible device only when the computer attempts to go into hibernation. To have the controller stay on at all times, do not click the Allow the computer to

turn off the device to save power check box ( Figure 3-1

Figure 3-1. Power Management Options

NOTE



The Power Management tab is available only for servers that support power management.



Do not select Allow the computer to turn off the device to save

power for any adapter that is a member of a team.

22 BC0154501-00 F

3–Installing Drivers

Windows Driver Software

Installing Drivers for Windows Nano Server

NOTE

You will need to extract the drivers from the Windows Driver Installer from driverdownloads.qlogic.com

to obtain the individual driver files available prior to starting Step 1 below.

Use the following procedure to install the drivers for Windows Nano Server:

Add and install the driver package by entering the following command:

pnputil.exe -i -a a:\usbcam\USBCAM.INF

Delete the oem0.inf package by entering the following command:

 Option 1: pnputil.exe -d oem0.inf

 Option 2: pnputil.exe -f -d oem0.inf

Creating a Nano ISO Image, Injecting Drivers, and Updating the Multi-boot/Flash Image on Nano Server

Use the following procedure to create a Nano ISO image, inject drivers, and update the MBI/Flash Image on Nano:

Create Nano Server image.

NOTE

For more information on how to create a Nano Server image, refer to the Microsoft documentation at: https://technet.microsoft.com/en-us/windows-server-docs/compute/na no-server/getting-started-with-nano-server .

Complete the following to inject the QLogic drivers into the Nano image.

Download the Windows Driver Installer and extract the relevant drivers from driverdownloads.qlogic.com

Place the extracted individual files in a temporary folder.

NOTE

You will use the QLogic drivers files during Nano Server image creation. Refer to the Injecting Drivers section in above Microsoft link from Step 1 for more information on a specific command.

23 BC0154501-00 F

3–Installing Drivers

VMware Driver Software

Complete the following to use Microsoft's pnputil tool to upgrade or install the QLogic drivers: a.

Copy the QLogic driver files to Nano Server.

After creating a Nano Server image, mount it, and copy over the files.

Navigate to the QLogic driver directory, and run the following command:

pnputil /add-driver c:\drivers\*.inf /install

NOTE

The c:\drivers in the command above, is where the QLogic drivers were copied to.

Complete the following to run the Firmware Upgrade Utility in Nano Server environment: a.

Copy the utility into the Nano Server.

After creating Nano Server image, mount it, and copy the utility.

NOTE

Refer to the Using Windows PowerShell section in above

Microsoft link from Step 1 for more information on connecting toa

Nano Server remotely.

Once connected remotely, navigate to the utility directory, and run it.

VMware Driver Software

This section describes the qedentv VMware ESXi driver for the 45000 Series

Adapter.



Installing VMware Driver



VMware Driver Optional Parameters



VMware Driver Parameter Defaults



Removing the VMware Driver

Installing VMware Driver

You can use the driver zip file to install a new driver or update an existing driver.

Be sure to install the entire driver set from the same driver zip file. Mixing drivers from different zip files will cause problems.

24 BC0154501-00 F

3–Installing Drivers

VMware Driver Software

To install the VMware driver

Go to www.vmware.com/support.html

and download the VMware driver for the 45000 Series Adapter.

Power up the ESX host, and then log into an account with administrator authority.

Unzip the driver zip file, and then extract the .vib file.

Copy the .vib file to the ESX server. You can place the file anywhere that is accessible to the ESX console shell. Type the following command to use the

Linux scp utility to copy a .vib file from a local system into the /tmp directory on an ESX server with IP address 10.10.10.10:

#scp qedentv-1.0.3.11-1OEM.550.0.0.1331820.x86_64.vib

[email protected]:/tmp

Place the host in maintenance mode by typing the following command:

#esxcli --maintenance-mode

You can install the .vib directly on an ESX server using the command line interface, or with the VMware Update Manager (VUM). For information about using the VUM, see the knowledge base article Updating an

ESXi/ESX host using VMware vCenter Update Manager 4.x and 5.x

(1019545) .

To install the .vib file using the command line interface, type the following command. Be sure to specify the full .vib file path.

#esxcli software vib install -v

/tmp/qedentv-1.0.3.11-1OEM.550.0.0.1331820.x86_64.vib

To upgrade an existing driver, follow the steps for a new installation, except

replace the command in Step 6

with the following:

#esxcli software vib update -v

/tmp/qedentv-1.0.3.11-1OEM.550.0.0.1331820.x86_64.vib

VMware Driver Optional Parameters

Table 3-4 describes the optional parameters that can be supplied as a command

line arguments to the esxcfg-module command.

25 BC0154501-00 F

3–Installing Drivers

VMware Driver Software

Table 3-4. VMware Driver Optional Parameters

Parameter Description

hw_vlan num_queues

The hw_lan parameter globally enables (1) or disables (0)

HW VLAN insertion and removal. Disable this parameter when the upper layer needs to send or receive fully formed packets. hw_vlan=1 is the default.

The num_queues parameter specifies the number of Tx/Rx queue pairs. num_queues can be 1–11 or one of the following:

 –1: Allow the driver to determine the optimal number of queue pairs (default)

 0: Use the default queue.

You can specify multiple values delimited by commas for multiport or multifunction configurations.

multi_rx_filters

The multi_rx_filters parameter specifies the number of

Rx filters per Rx queue, excluding the default queue. multi_rx_filters can be 1–4 or one of the following values:

 –1: Use the default number of Rx filters per queue.

 0: Disable Rx filters.

disable_tpa max_vfs

RSS

Enables (0) or disables (1) the TPA (LRO) feature. disable_tpa=0 is the default.

Specifies the number of virtual functions (VFs) per physical function (PF). max_vfs can be 0 (disable), or 8 or 16

(enable).

The RSS parameter specifies the number of receive side scaling queues used by the host or virtual extensible LAN

(VxLAN) tunneled traffic for a PF. RSS can be 2, 3, or 4, or one of the following values:

 –1: Use the default number of queues.

 0 or 1: Disable RSS queues.

You can specify multiple values delimited by commas for multiport or multifunction configurations.

26 BC0154501-00 F

3–Installing Drivers

VMware Driver Software

Table 3-4. VMware Driver Optional Parameters (Continued)

Parameter Description

debug auto_fw_reset vxlan_filter_en enable_vxlan_offl d

The debug parameter specifies the level of data that the driver records in the vmkernel log file. debug can have the following values, shown in increasing amounts of data:

 0x80000000: Notice level

 0x40000000: Information level (includes the Notice level)

 0x3FFFFFFF: Verbose level for all driver submodules

(includes the Information and Notice levels)

The auto_fw_reset parameter enables (1) or disables (0) the driver automatic firmware recovery capability. When this parameter is enabled, the driver attempts to recover from events such as transmit timeouts, firmware asserts, and adapter parity errors. The default is auto_fw_reset=1.

The vxlan_filter_en parameter enables (1) or disables (0) the

VxLAN filtering based on the outer MAC, the inner MAC, and the VxLAN network (VNI), directly matching traffic to a specific queue. The default is vxlan_filter_en=1. You can specify multiple values delimited by commas for multiport or multifunction configurations.

The enable_vxlan_offld parameter enables (1) or disables (0) the VxLAN tunneled traffic checksum offload and

TCP segmentation offload (TSO) capability. The default is enable_vxlan_offld=1. You can specify multiple values delimited by commas for multiport or multifunction configurations.

VMware Driver Parameter Defaults

Table 3-5 lists the VMware driver parameter defaults.

Speed

Flow Control

MTU

Table 3-5. VMware Driver Parameter Defaults

Parameter Default

Autonegotiation with all speeds advertised. The speed parameter must be the same on all ports.

If auto-negotiation is enabled on the device, then all of the device ports will use auto-negotiation.

Autonegotiation with Rx and Tx advertised

1,500 (range 46–9,600)

27 BC0154501-00 F

3–Installing Drivers

VMware Driver Software

Table 3-5. VMware Driver Parameter Defaults (Continued)

Parameter

Rx Ring Size

Tx Ring Size

MSI-X

Transmit Send Offload (TSO)

Large Receive Offload (LRO)

RSS

HW VLAN

Number of Queues

Wake on LAN (WoL)

Default

8,192 (range 128–8,192)

Enabled

Enabled (four Rx queues)

Enabled

Enabled (eight Rx/Tx queue pairs)

Disabled

Removing the VMware Driver

To remove the .vib file (qedentv), type the following command

#esxcli software vib remove --vibname qedentv

To remove the driver, type the following command:

#vmkload_mod -u qedentv

28 BC0154501-00 F

Firmware Upgrade Utility

QLogic provides scripts to automate the adapter firmware and boot code upgrade process for Windows and Linux systems. Each script identifies all 45000 Series

Adapters and upgrades all firmware components. To upgrade adapter firmware on

VMware systems, see the FastLinQ ESXCLI VMware Plug-in User’s Guide or the

QConvergeConsole Plug-ins for vSphere User’s Guide.

CAUTION

Upgrading the adapter firmware may require several minutes. To avoid damaging your adapters, do not exit, reboot, or power cycle your system during the upgrade.

Each script executes its respective utility: lnxfwnx2 for Linux and winfwnx2 for

Windows. Both utilities are console applications that can be called from other processes or used as a command line tool with parameters. For more information about these utilities, their command syntax, and parameters, see their respective

ReadMe files.

This chapters provides information about the following topics:



Upgrading Adapter Firmware on Linux



Upgrading Adapter Firmware on Windows

29 BC0154501-00 F

4–Firmware Upgrade Utility

Upgrading Adapter Firmware on Linux

To upgrade adapter firmware on a Linux system:

Install the qed and qede drivers, and confirm that they are running.

Download the Firmware Upgrade Utility for Linux from driverdownloads.qlogic.com

Unzip the Firmware Upgrade Utility on the system where the adapter is installed. All files should reside in the same folder.

Confirm that the adapter port link status is up (ifconfig ethx up).

Run the Linux upgrade utility script by typing the following command:

./LnxQlgcUpg.sh

Upgrading Adapter Firmware on Windows

To upgrade adapter firmware on a Windows system, type the following command:

Install the Window eVBD driver.

Download the Firmware Upgrade Utility for Windows from driverdownloads.qlogic.com

Unzip the Firmware Upgrade Utility on the system where the adapter is installed. All files should reside in the same folder.

Run the Windows upgrade utility script by type the following command:

C:\Windows_FWupg\AMD64\WinQlgcUpg.bat

30 BC0154501-00 F

Preboot Adapter

Configuration

Getting Started

During the host boot process, you have the opportunity to pause and perform adapter management tasks using the Human Infrastructure Interface (HII) application. These tasks include the following:



Displaying Firmware Image Properties



Device Level Configuration

NOTE

The HII screens in this chapter are representative and may not match the screens that you see on your system.

31 BC0154501-00 F

5–Preboot Adapter Configuration

Getting Started

To start the HII application:

Open the System Setup window for your platform. For information about launching the System Setup, consult the user guide for your platform.

In the System Setup window, select Device Settings, and then press

ENTER.

In the Device Settings window, select the 45000 Series Adapter port that you want to configure, and then press ENTER.

32 BC0154501-00 F

5–Preboot Adapter Configuration

Getting Started

The Main Configuration Page presents the adapter management options. If you are not using NPAR, set Partitioning Mode to Default, as shown in

Figure 5-1 . Setting Partitioning Mode to NPAR adds the Partitions

Configuration option to the Main Configuration Page, as shown in

Figure 5-2 .

Figure 5-1. Main Configuration Page—Default Mode

Figure 5-2. Main Configuration Page—NPAR Mode

33 BC0154501-00 F

5–Preboot Adapter Configuration

Displaying Firmware Image Properties

In addition to the management options, the Main Configuration Page presents the adapter properties shown in

Table 5-1 .

Adapter Property

Device Name

Chip Type

PCI Device ID

PCI Address

Blink LEDs

Link Status

Permanent MAC

Address

Virtual MAC Address

Table 5-1. Adapter Properties

Description

Factory-assigned device name

ASIC version

Unique vendor specific PCI device ID

PCI device address in bus-device function format

User-defined blink count for the port LED

External link status

Manufacturer-assigned permanent device MAC address

User-defined device MAC address

Displaying Firmware Image Properties

To display firmware image properties, select Firmware Image Properties from the Main Configuration Page, and then press ENTER. The Firmware Image

Properties window (

Figure 5-3 ) presents the following:

 Family Firmware Version: multiboot image version, which comprises several firmware images.

 MFW Version: management firmware version

 UEFI Driver Version: Unified extensible firmware interface

(UEFI)/extensible firmware interface (EFI) driver version

Figure 5-3. Firmware Information Window

34 BC0154501-00 F

5–Preboot Adapter Configuration

Device Level Configuration

NOTE

The iSCSI physical function (PF) is enumerated when the iSCSI Offload is enabled. If the iSCSI Offload is disabled from Hll pre-boot level on a QLogic

QL45452-40G, the iSCSI PF will not be enumerated.

Device level configuration involves the following parameters:

 Link speed

 Forward error correction (FEC) Mode

 Boot mode

 SR-IOV

 DCBX protocol

 RoCE priority

To configure the device level parameters:

Select Device Level Configuration in the Main Configuration Page

(

Figure 5-1

), and then press ENTER.

Choose values for the device level parameters. The available parameters will vary depending on whether you are using a 25Gb, 40Gb, or a 100Gb adapter. For example, 25Gb and 40Gb adapters show Link Speed, FEC

Mode (if Link Speed is 25Gbps), SR-IOV, Boot Mode, DCBX Protocol, and

RoCE Priority, as shown in

Figure 5-4 .

Figure 5-4. Device Level Configuration Pages (25Gb and 40Gb Adapters)

35 BC0154501-00 F

5–Preboot Adapter Configuration

Device Level Configuration

In contrast, a 100Gb adapter shows Link Speed, FEC Mode (if Link Speed is

100Gbps), and Boot Mode, as shown in Figure 5-5

Figure 5-5. Device Level Configuration Page (100Gb Adapter)

Click Back.

When prompted, click Yes to save the changes. Changes will take effect after a system reset.

Table 5-2 describes the device level parameters for 25Gb and 100Gb adapters.

Parameter

Link Speed

Table 5-2. Device Level Parameters

Description

The link speed you choose applies to both adapter ports. The link speed and the forward error correction (FEC) must match that of the connected switch or device port.

 Auto Negotiated (default)—Dual-port 25Gb adapters and single-port 100Gb adapters auto-negotiate available speeds and link parameters (FEC, Pause) over DAC connections

. FEC is automatically enabled.

 10Gbps—Sets the port link speed at 10Gbps (25Gb and 40Gb adapters only). FEC is not supported.

 25Gbps—Sets the port link speed at 25Gbps (25Gb adapter only). BaseR-FEC (Firecode) is optional. RS-FEC is not supported.

 40Gbps—Sets the port link speed at 40Gbps (40Gb adapters only). FEC is optional.

 100Gbps—Sets the port link speed at 100Gbps (100Gb adapter only). FEC is required for standard connection, but can be turned off.

36 BC0154501-00 F

5–Preboot Adapter Configuration

Device Level Configuration

Table 5-2. Device Level Parameters (Continued)

Parameter Description

FEC Mode FEC options are available when the Link Speed is set explicitly to

25Gbps on 25Gb adapters, 40Gbps on 40Gb adapters, or 100Gbps on 100Gb adapters:

 None—Disables FEC

 Fire Code—Enables FEC (BaseR-FEC) on 25Gb and 40Gb adapters

 Reed Solomon—Enables Reed Solomon FEC on 100Gb adapters. This setting is required for optical connections.

Enables (UNDI) or disables (Disabled) UEFI PXE boot.

Enables (Enabled) or disables (Disabled) SR-IOV.

Boot Mode

SR-IOV

DCBX Protocol Specifies the DCBX protocol:

 Disabled: Disables DCB

 IEEE: IEEE protocol

 CEE: Converged Enhanced Ethernet (CEE) protocol

 Dynamic: Dynamically applies CEE or IEEE to match the attached link partner

RoCE Priority Specifies a value (0–7) for the RoCE priority. This value must match the RoCE priority on the connected switch.

For information about available link speeds and optical connections, see the driver Release Notes.

37 BC0154501-00 F

5–Preboot Adapter Configuration

Partitions Configuration

To configure the maximum and minimum bandwidth allocations for each partition on a 25Gb adapter:

Select Partitions Configuration in the Main Configuration Page

(

Figure 5-2

), and then press ENTER.

In the Partitions Configuration page (

Figure 5-6 ), select Global Bandwidth

Allocation.

Figure 5-6. Partitions Configuration Page

38 BC0154501-00 F

5–Preboot Adapter Configuration

Partitions Configuration

In the Global Bandwidth Allocation page (

Figure 5-7 ), click on each partition

minimum and maximum TX bandwidth field for which you want to allocate bandwidth. There are eight partitions per port.

Figure 5-7. Global Bandwidth Allocation Page

 Partition N Minimum TX Bandwidth is the minimum transmit bandwidth of the selected partition expressed as a percentage of the maximum physical port link speed. Values can be 0–100. When DCBX

ETS mode is enabled, the per traffic class DCBX ETS minimum bandwidth value overrides the per partition minimum TX bandwidth value.The total of the minimum TX bandwidth values of all partitions on a single port must equal 100 or be all zeros.

 Partition N Maximum TX Bandwidth is the maximum transmit bandwidth of the selected partition expressed as a percentage of the maximum physical port link speed. Values can be 1–100. The per partition maximum TX bandwidth value applies regardless of the

DCBX ETS mode setting. If the maximum TX bandwidth for a partition is set to 0, the effective bandwidth is one percent.

Type a value in each selected field, and then click Back.

When prompted, click Yes to save the changes. Changes will take effect after a system reset.

39 BC0154501-00 F

5–Preboot Adapter Configuration

Partitions Configuration

To examine a specific partition configuration, in the Partitions Configuration page (

Figure 5-6 ), select Partition n Configuration. For example, selecting

Partition 1 Configuration, opens the Partition 1 Configuration page

(

Figure 5-8

), which shows the partition 1 personality, PCI device ID, PCI bus address, permanent MAC address, and virtual MAC address.

Figure 5-8. Partition 1 Configuration

40 BC0154501-00 F

Configuring RoCE

This chapter describes RoCE (v1 and v2) configuration on the 45000 Series

Adapter, the Ethernet switch, and the Windows or Linux host.

This chapter describes the following topics:



Supported Operating Systems and OFED



Planning



Preparing the Adapter



Preparing the Ethernet Switch



Configuring RoCE on the Adapter for Windows Server



Configuring RoCE on the Adapter for Linux

41 BC0154501-00 F

6–Configuring RoCE

Supported Operating Systems and OFED

Table 6-1 shows the operating system support for RoCE v1/v2 and OFED:

Table 6-1. Operating System Support for RoCE v1/v2 and OFED

Operating System

Windows 2012 R2

Windows 2016

RHEL 6.7, 6.8, 7.1, 7.2

SLES 11 SP4, 12, 12 SP1

CentOS 7.2

Ubuntu 14.04 LTS,

16.04 LTS

RoCE v1 RoCE v2 Inbox OFED

Yes

Out-of-Box

OFED

42 BC0154501-00 F

6–Configuring RoCE

Planning

As you prepare to implement RoCE, consider the following limitations:

 If you are using the inbox OFED, the operating system should be the same on the server and client systems. Some of the applications may work between different operating systems, but there is no guarantee. This is an

OFED limitation.

 For OFED applications (most often perftest applications), server and client applications should use the same options and values. Problems can arise if the operating system and the perftest application have different OFED versions. To confirm the perftest OFED version, type the following command:

# ib_send_bw --version

 For any RDMA application problem that you encounter, compare your results with those reported for other brands of adapters to determine whether your problem is common. Refer to the OFED ReadMe and Release

Notes for information about known issues and limitations.

 Building libqedr in INBOX OFED requires installing libibverbs-devel.

 Running user space applications in INBOX OFED requires installing the

Infiniband Support group, by yum groupinstall "Infiniband Support" that contain libibcm, libibverbs, and more.

 OFED/RDMA applications that depend on libibverbs also require the QLogic

RDMA user space library, libqedr. libqedr can be installed using libqedr RPM or source packages

 RoCE supports only little endian.

 RoCE will not work over a VF in an SR-IOV environment.

43 BC0154501-00 F

6–Configuring RoCE

Preparing the Adapter

The following steps describe how to enable DCBX, and specify the RoCE priority using the HII management application. For information about the HII application,

see Chapter 5, Preboot Adapter Configuration

In the Main Configuration Page, select Data Center Bridging (DCB)

Settings, and then click Finish.

In the Data Center Bridging (DCB) Settings window, click the DCBX

Protocol drop-down. The 45000 Series Adapter supports both CEE and

IEEE protocols. This value should match the corresponding value on the

DCB switch. In this example, select CEE.

Click the RoCE Priority field, and type a priority value. This value should match the corresponding value on the DCB switch. In this example, type 4.

Click Back.

When prompted, click Yes to save the changes. Changes will not take effect unit after a system reset.

Preparing the Ethernet Switch

This section describes how to configure a Cisco

Nexus

6000 Ethernet Switch and a Dell

Z9100 Ethernet switch for RoCE.



Cisco Nexus 6000 Ethernet Switch



Dell Z9100 Ethernet Switch



Arista 7060X Switch

Cisco Nexus 6000 Ethernet Switch

Configuring the Cisco Nexus 6000 Ethernet Switch for RoCE comprises configuring class maps, configuring policy maps, applying the policy, and assigning a VLAN ID to the switch port.

Open a config terminal session.

Switch# config terminal switch(config)#

Configure quality of service (QoS) class maps, setting the RoCE priority to match the adapter (4).

switch(config)# class-map type qos class-roce switch(config)# match cos 4

44 BC0154501-00 F

6–Configuring RoCE

Preparing the Ethernet Switch

Configure queuing class maps.

switch(config)# class-map type queuing class-roce switch(config)# match qos-group 3

Configure network QoS class maps.

switch(config)# class-map type network-qos class-ro switch(config)# match qos-group 3

Configure QoS policy maps.

switch(config)# policy-map type qos roce switch(config)# class type qos class-roc switch(config)# set qos-group 3

Configure queuing policy maps to assign network bandwidth. In this example, use a value of 50 percent.

switch(config)# policy-map type queuing roce switch(config)# class type queuing class-roce switch(config)# bandwidth percent 50

Configure network QoS policy maps to set priority flow control for no drop traffic class.

switch(config)# policy-map type network-qos roce switch(config)# class type network-qos class-roce switch(config)# pause no-drop

Apply the new policy at the system level.

switch(config)# system qos switch(config)# service-policy type qos input roce switch(config)# service-policy type queuing output roce switch(config)# service-policy type queuing input roce switch(config)# service-policy type network-qos roce

Assign a VLAN ID to the switch port to match the VLAN ID assigned to the adapter (5).

switch(config)# interface ethernet x/x switch(config)# switchport mode trunk switch(config)# switchport trunk allowed vlan 1,5

45 BC0154501-00 F

6–Configuring RoCE

Preparing the Ethernet Switch

Dell Z9100 Ethernet Switch

Configuring the Dell Z9100 Ethernet Switch for RoCE comprises configuring a

DCB map for RoCE, configuring priority-based flow control (PFC) and enhanced transmission selection (ETS), verifying the DCB map, applying the DCB map to the port, verifying PFC and ETS on the port, specifying the DCB protocol, and assigning a VLAN ID to the switch port.

Create a DCB map.

Dell# configure

Dell(conf)# dcb-map roce

Dell(conf-dcbmap-roce)#

Configure two ETS traffic classes in the DCB map with 50 percent bandwidth assigned for RoCE (group 1).

Dell(conf-dcbmap-roce)# priority-group 0 bandwidth 50 pfc off

Dell(conf-dcbmap-roce)# priority-group 1 bandwidth 50 pfc on

Configure the PFC priority to match the adapter priority (4).

Dell(conf-dcbmap-roce)# priority-pgid 0 0 0 0 1 0 0 0

Verify the DCB map configuration. Priority group

Dell(conf-dcbmap-roce)# do show qos dcb-map roce

-----------------------

State :Complete

PfcMode :ON

--------------------

PG:0 TSA:ETS BW:40 PFC:OFF

Priorities:0 1 2 3 5 6 7

PG:1 TSA:ETS BW:60 PFC:ON

Priorities:4

Apply the DCB map to the port.

Dell(conf)# interface twentyFiveGigE 1/8/1

Dell(conf-if-tf-1/8/1)# dcb-map roce

46 BC0154501-00 F

6–Configuring RoCE

Preparing the Ethernet Switch

Verify the ETS and PFC configuration on the port.

Dell(conf-if-tf-1/8/1)# do show interfaces twentyFiveGigE 1/8/1 ets summary

Interface twentyFiveGigE 1/8/1

Max Supported TC is 4

Number of Traffic Classes is 8

Admin mode is on

Admin Parameters :

------------------

Admin is enabled

PG-grp Priority# BW-% BW-COMMITTED BW-PEAK TSA

% Rate(Mbps) Burst(KB) Rate(Mbps) Burst(KB)

------------------------------------------------------------------------

0 0,1,2,3,5,6,7 40 - - - - ETS

1 4 60 - - - - ETS

2 - - - - - -

3 - - - - - -

Dell(Conf)# do show interfaces twentyFiveGigE 1/5/1 pfc detail

Interface twentyFiveGigE 1/5/1

Admin mode is on

Admin is enabled, Priority list is 4

Remote is enabled, Priority list is 4

Remote Willing Status is enabled

Local is enabled, Priority list is 4

Oper status is init

PFC DCBX Oper status is Up

State Machine Type is Feature

TLV Tx Status is enabled

PFC Link Delay 65535 pause quntams

Application Priority TLV Parameters :

--------------------------------------

FCOE TLV Tx Status is disabled

ISCSI TLV Tx Status is enabled

Local FCOE PriorityMap is 0x0

Local ISCSI PriorityMap is 0x10

Remote ISCSI PriorityMap is 0x10

66 Input TLV pkts, 99 Output TLV pkts, 0 Error pkts, 0 Pause Tx pkts, 0 Pause Rx pkts

66 Input Appln Priority TLV pkts, 99 Output Appln Priority TLV pkts, 0 Error Appln

Priority TLV Pkts

Configure the DCBX protocol (CEE in this example).

Dell(conf)# interface twentyFiveGigE 1/8/1

Dell(conf-if-tf-1/8/1)# protocol lldp

Dell(conf-if-tf-1/8/1-lldp)# dcbx version cee

47 BC0154501-00 F

6–Configuring RoCE

Preparing the Ethernet Switch

Assign a VLAN ID to the switch port to match the VLAN ID assigned to the adapter (5).

Dell(conf)# interface vlan 5

Dell(conf-if-vl-5)# tagged twentyFiveGigE 1/8/1

Arista 7060X Switch

Configuring RoCE APP TLVs for both RoCE v1 and RoCE v2

The following is an example on configuring RoCE APP TLV for both RoCE v1 and and RoCE v2:

Arista-7060X-EIT(config)#dcbx application ether 0x8915 priority 4

Arista-7060X-EIT(config)#dcbx application udp 4791 priority 4

Mapping Priority to Traffic Class

In the example below, Priority 4 is mapped to traffic class 1, and the rest of the priorities are mapped to traffic class 0:

Arista-7060X-EIT(config)#dcbx ets qos map cos 0 traffic-class 0

Arista-7060X-EIT(config)#dcbx ets qos map cos 1 traffic-class 0

Arista-7060X-EIT(config)#dcbx ets qos map cos 2 traffic-class 0

Arista-7060X-EIT(config)#dcbx ets qos map cos 3 traffic-class 0

Arista-7060X-EIT(config)#dcbx ets qos map cos 4 traffic-class 1

Arista-7060X-EIT(config)#dcbx ets qos map cos 5 traffic-class 0

Arista-7060X-EIT(config)#dcbx ets qos map cos 6 traffic-class 0

Arista-7060X-EIT(config)#dcbx ets qos map cos 7 traffic-class 0

Setting up ETS

In example below, traffic class 0 is given bandwidth 5%, and traffic class 1 is given bandwidth 95%:

Arista-7060X-EIT(config)#dcbx ets traffic-class 0 bandwidth 5

Arista-7060X-EIT(config)#dcbx ets traffic-class 1 bandwidth 95

Configuring Interface

The same VLAN ID needs to be assigned to the Server Adapter Ports. The configuration below needs to be done on all the switch interfaces where the

Server Adapter Ports are connected. DCBX Protocol mode is set to CEE. PFC is enabled for Priority 4.

Arista-7060X-EIT(config)#interface ethernet 26/1

Arista-7060X-EIT(config-if-Et26/1)#dcbx mode cee

Arista-7060X-EIT(config-if-Et26/1)# priority-flow-control mode on

Arista-7060X-EIT(config-if-Et26/1)# priority-flow-control priority 4 no-drop

Arista-7060X-EIT(config-if-Et26/1)# switchport mode trunk

Arista-7060X-EIT(config-if-Et26/1)# switchport trunk allowed vlan 1,5

48 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Windows Server

Configuring RoCE on the Adapter for Windows

Server

Configuring RoCE on the adapter for Windows Server comprises enabling RoCE on the adapter and verifying the Network Direct MTU size.

To configure RoCE on a Windows Server host:

Enable RoCE on the adapter.

Open the Windows Device Manager, and then open the 45000 Series

NDIS Miniport Properties.

Select Network Direct Functionality from the Property menu, and then select Enabled in the Value field. Click OK.

49 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Windows Server

Select Network Direct Mtu Size from the Property menu. The network direct MTU size must be less than the jumbo packet size. In this example, select 1024. Click OK.

Select RoCE Mode from the Property menu. In the Value: pull-down menu, select RoCE v1 or RoCE v2. Click OK.

50 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Windows Server

Verify that RoCE is enabled on the adapter using Windows PowerShell

The Get-NetAdapterRdma command lists the adapters that support

RDMA—both ports are enabled.

PS C:\Users\Administrator> Get-NetAdapterRdma

Name InterfaceDescription Enabled

----- -------------------- -------

SLOT 4 3 Port 1 QLogic FastLinQ QL45212... True

SLOT 4 3 Port 2 QLogic FastLinQ QL45212... True

Verify that RoCE is enabled on the host operating system using PowerShell.

The Get-NetOffloadGlobalSetting command shows NetworkDirect is enabled.

PS C:\Users\Administrators> Get-NetOffloadGlobalSetting

ReceiveSideScaling : Enabled

ReceiveSegmentCoalescing : Enabled

Chimney : Disabled

TaskOffload : Enabled

NetworkDirect : Enabled

NetworkDirectAcrossIPSubnets : Blocked

PacketCoalescingFilter : Disabled

Connect a server message block (SMB) drive, run RoCE traffic, and verify the results.

Map an SMB drive, and then run RoCE traffic.

Launch Performance Monitor.

51 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Windows Server c.

Select <Local computer> under Select counters from computer:.

Select Add RDMA Activity, and then select <All instances> under

Instances of selected object;.

The following figure shows the presence of RDMA traffic.

52 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Windows Server d.

Verify the SMB connection using Windows PowerShell. Type the net use and netstat -xan commands to show Share1 is mapped as

SMB share.

PS C:\Users\Administrator> net use

Status Local Remote Network

-------------------------------------------------------------

OK F: \\192.168.10.10\Share1 Microsoft Window

The command completed successfully.

PS C:\Users\Administrator> netstat -xan

Active NetworkDirect Connections, Listeners, SharedEndpoints

Mode IfIndex Type Local Address Foreign Address PID

Kernel 56 Connection 192.168.11.20.16159 192.168.11.10.445 0

Kernel 56 Connection 192.168.11.20.15903 192.168.11.10.445 0

Kernel 60 Connection 192.168.10.20.16159 192.168.10.10.445 0

Kernel 60 Connection 192.168.10.20.15903 192.168.10.10.445 0

Kernel 56 Listener [fe80:e11d:9ab5:a47d:4f0a%56]:445 NA 0

Kernel 56 Listener 192.168.11.20.445 NA 0

Kernel 60 Listener [fe80:71ea:bdd2:ae41:b95f%60]:445 NA 0

Kernel 60 Connection 192.168.10.20.445 NA 0

By default, Microsoft's SMB Direct establishes two RDMA connections per port, which provides good performance, including line rate at higher block size (for example, 64KB). To optimize performance, you can change the number of RDMA connections per RDMA interface to four (or greater).

To increase the number of RDMA connections to four, type the following command in PowerShell:

PS C:\Users\Administrator> Set-ItemProperty -Path `

"HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Pa rameters" ConnectionCountPerRdmaNetworkInterface -Type DWORD

-Value 4 –Force

53 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Linux

This section describes the RoCE configuration procedure for RHEL and SLES. It also describes how to verify the RoCE configuration and provides some guidance about using group IDs (GIDs) with VLAN interfaces.

RoCE Configuration for RHEL

To configure RoCE on the adapter, the Open Fabrics Enterprise Distribution

(OFED) must be installed and configured on the RHEL host.

To prepare inbox OFED for RHEL:

While installing or upgrading the operating system, select the InfiniBand

® and OFED support packages.

Install the following RPMs from RHEL ISO image: libibverbs-devel-x.x.x.x86_64.rpm

(required for libqedr library) perftest-x.x.x.x86_64.rpm

(required for IB bandwidth and latency applications) or, using Yum, install the inbox OFED:

yum groupinstall "Infiniband Support" yum install perftest

yum install tcl tcl-devel tk zlib-devel libibverbs

libibverbs-devel

NOTE

During installation, if you already selected the previously mentioned packages, you need not reinstall them. The inbox OFED and support packages may vary depending on the operating system version.

Install the new Linux drivers, as described in “Installing the Linux Drivers with RoCE” on page 18 .

54 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Linux

RoCE Configuration for SLES

To configure RoCE on the adapter for an SLES host, OFED must be installed and configured on the SLES host.

To install inbox OFED for SLES Linux:

While installing or upgrading the operating system, select the Infiniband support packages.

Install the following RPMs from the corresponding SLES SDK kit image: libibverbs-devel-x.x.x.x86_64.rpm

(required for libqedr installation) perftest-x.x.x.x86_64.rpm

(required for bandwidth and latency applications)

Install the Linux drivers, as described in “Installing the Linux Drivers with

RoCE” on page 18 .

RoCE Configuration for Ubuntu

To configure RoCE on the adapter for an Ubuntu host, OFED must be installed and configured on the Ubuntu host.

To install inbox OFED for Ubuntu Linux:

Use the appropriate procedure below:



Ubuntu 14.04 LTS

Install libibverbs.

Note that this is only required for Ubuntu 14.04 in case libibverbs-dev package is not installed through apt-get.

Enable root access on terminal by typing the following commands: user@captain:~$ sudo bash

[sudo] password for user: root@captain:~#

Get the libibverbs source by typing the following commands: root@captain:~/libibverbs# mkdir libibverbs root@captain:~ # cd libibverbs root@captain:~/libibverbs# apt-get source

libibverbs

cd libibverbs-1.1.7

./configure ; make install

55 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Linux b.

Install the RDMA packages by typing the following command: root@captain:~sudo apt-get -y --force-yes install

libibcm1 libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils ibsim-utils ibutils libcxgb3-1 libibmad5 libibumad3 libmlx4-1 libmthca1 libnes1 rds-tools infiniband-diags libibcommon1 mstflint opensm libopensm5 perftest srptools



Ubuntu 16.04 LTS

Install the RDMA packages by typing the following command:

$ apt-get install dkms infiniband-diags libibverbs* libibverbs-dev ibacm librdmacm* mstflint libibcm.* libibmad.* libibumad* opensm srptools librdmacm-dev rdmacm-utils ibverbs-utils perftest ibutils libnl-utils

Install the following additional build tools:

$ apt-get install libtool autoconf automake build-essential ibutils ibverbs-utils infiniband-diags numactl libnuma-dev autoconf automake libtool pkg-config

If the file /etc/udev/rules.d/40-rdma.rules file does not exist, create it with the following content:

KERNEL=="umad*", NAME="infiniband/%k"

KERNEL=="issm*", NAME="infiniband/%k"

KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"

KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"

KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"

KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"

Edit the /etc/security/limits.conf file to increase the size of memory, which can be locked by non-root user. Add the following lines, and then logout.

* soft memlock unlimited

* hard memlock unlimited root soft memlock unlimited root hard memlock unlimited

56 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Linux

ulimit -l

You should get the output as "unlimited”.

Reboot the system.

Install the FastLinQ package to allow the device to be recognized as an

InfiniBand device that can be used by OFED by typing the following commands:

cd fastlinq-linux-X.X.X.X

make clean make install

Configure userlib for qedr by typing the following commands:

 Option 1

cd libqedr-X.X.X/src

If installed through Source (apt-get source libibverbs):

./configure --prefix=/usr --libdir=${exec_prefix}/lib

--sysconfdir=/usr/local/etc

If installed through pkg (apt-get install libibverbs-dev):

./configure --prefix=/usr --libdir=${exec_prefix}/lib

--sysconfdir=/etc make install

 Option 2 cd fastlinq-linux-X.X.X.X

make libqedr_install

Load the qedr driver by typing the following command:

# modprobe -v qedr

57 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Linux

10. Load the RDMA modules by typing the following commands. You must perform this step every time you reboot the system.

# RDMA stack modules

# sudo modprobe rdma_cm

# sudo modprobe ib_uverbs

# sudo modprobe rdma_ucm

# sudo modprobe ib_ucm

# sudo modprobe ib_umad

# sudo modprobe ib_ipoib

11.

Assign IP addresses to the Ethernet interfaces:

 To assign the static IP address, edit the

/etc/network/interfaces file with the following entries: iface eth0 inet static address 192.168.10.5

netmask 255.255.255.0

gateway 192.168.10.254

 To assign the DHCP IP address, edit /etc/network/interfaces file with the following entries: auto eth0 iface eth0 inet dhcp

Verifying the RoCE Configuration on Linux

After installing OFED, installing the Linux driver, and loading the RoCE drivers, verify that the RoCE devices were detected on all Linux operating systems:

Stop firewall tables using service/systemctl commands.

Verify that the RDMA services has started using the service rdma status command. If RDMA has not started, type the following command:

# service rdma start

If RDMA does not start, type either of the following alternative commands:

# /etc/init.d/rdma start or

# systemctl start rdma.service

58 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Linux

Verify that the RoCE devices were detected by examining the dmesg logs.

# dmesg|grep qedr

[87910.988411] qedr: discovered and registered 2 RoCE funcs

Verify that all of the modules have been loaded. For example:

# lsmod|grep qedr

qedr 89871 0

qede 96670 1 qedr

qed 2075255 2 qede,qedr

ib_core 88311 16 qedr, rdma_cm, ib_cm,

ib_sa,iw_cm,xprtrdma,ib_mad,ib_srp,

ib_ucm,ib_iser,ib_srpt,ib_umad,

ib_uverbs,rdma_ucm,ib_ipoib,ib_isert

Configure the IP address and enable the port using a configuration method, such as ifconfig:

# ifconfig ethX 192.168.10.10/24 up

Type the ibv_devinfo command. For each PCI function, you should see a separate hca_id, as shown in the following example: root@captain:~# ibv_devinfo hca_id: qedr0

transport: InfiniBand (0)

fw_ver: 8.3.9.0

node_guid: 020e:1eff:fe50:c7c0

sys_image_guid: 020e:1eff:fe50:c7c0

vendor_id: 0x1077

vendor_part_id: 5684

hw_ver: 0x0

phys_port_cnt: 1

port: 1

state: PORT_ACTIVE (1)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

Verify the L2 and RoCE connectivity between all servers: one server acts as a server, another acts as a client.

59 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Linux

 Verify the L2 connection using a simple ping command.

 Verify the RoCE connection by performing an RDMA ping:

On the server, type the following command: ibv_rc_pingpong -d <ib-dev> -g 0

On the client, type the following command: ibv_rc_pingpong -d <ib-dev> -g 0 <server L2 IP address>

The following are examples of successful ping pong tests on the server and the client:

Server: root@captain:~# ibv_rc_pingpong -d qedr0 -g 0 local address: LID 0x0000, QPN 0xff0000, PSN 0xb3e07e, GID fe80::20e:1eff:fe50:c7c0 remote address: LID 0x0000, QPN 0xff0000, PSN 0x934d28, GID fe80::20e:1eff:fe50:c570

8192000 bytes in 0.05 seconds = 1436.97 Mbit/sec

1000 iters in 0.05 seconds = 45.61 usec/iter

Client: root@lambodar:~# ibv_rc_pingpong -d qedr0 -g 0 192.168.10.165 local address: LID 0x0000, QPN 0xff0000, PSN 0x934d28, GID fe80::20e:1eff:fe50:c570 remote address: LID 0x0000, QPN 0xff0000, PSN 0xb3e07e, GID fe80::20e:1eff:fe50:c7c0

8192000 bytes in 0.02 seconds = 4211.28 Mbit/sec

1000 iters in 0.02 seconds = 15.56 usec/iter

 To display RoCE statistics, type the following commands, where X is the device number:

> mount -t debugfs nodev /sys/kernel/debug

> cat /sys/kernel/debug/qedr/qedrX/stats

60 BC0154501-00 F

6–Configuring RoCE

Configuring RoCE on the Adapter for Linux

VLAN Interfaces and GID Index Values

If you are using VLAN interfaces on both server and client, then configure the same VLAN ID on the switch. If you are running traffic through a switch, the

Infiniband applications must use the correct GID value, which is based on the

VLAN ID and VLAN IP address.

Based on the following results, the GID value (-x 4 / -x 5) should be used for any perftest applications.

# ibv_devinfo -d qedr0 -v|grep GID

GID[ 0]: fe80:0000:0000:0000:020e:1eff:fe50:c5b0

GID[ 1]:

0000:0000:0000:0000:0000:ffff:c0a8:0103

GID[ 2]:

2001:0db1:0000:0000:020e:1eff:fe50:c5b0

GID[ 3]:

2001:0db2:0000:0000:020e:1eff:fe50:c5b0

GID[ 4]:

0000:0000:0000:0000:0000:ffff:c0a8:0b03 IP address for VLAN interface

GID[ 5]: fe80:0000:0000:0000:020e:1e00:0350:c5b0 VLAN ID 3

NOTE

The default GID value is zero (0) for back-to-back/Pause settings. For server/switch configurations, you must identify the proper GID Value. If you are using a switch, refer to the corresponding switch configuration documents for the proper settings.

61 BC0154501-00 F

iSCSI Extensions for RDMA

This chapter describes the procedure for configuring iSCSI Extensions for RDMA

(iSER) for RHEL and SLES:



Configuring iSER for RHEL



Configuring iSER for SLES 12 Linux



Configuring iSER for Ubuntu



Optimizing Linux Performance

As you prepare to configure iSER, consider the following:

 iSER is supported only in inbox OFED for following operating systems:

 RHEL 7.1, 7.2

 SLES 12, 12 SP1

 CentOS 7.2

 Ubuntu 14.04 LTS, 16.04 LTS

 After logging into the targets or while running I/O traffic, unloading the Linux

RoCE qedr driver may crash the system.

 While running I/O, performing interface down/up tests or performing cable pull-tests can cause driver or iSER module errors that may crash the system. If this happens, reboot the system.

62 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for RHEL

To configure iSER for RHEL Linux:

Install inbox OFED as described in

“RoCE Configuration for RHEL” on page 54 . Out-of-box OFEDs are not supported for iSER because the ib_isert

module is not available in the out-of-box OFED-3.18/3.81-1 GA versions.

The inbox ib_isert module does not work with any out-of-box OFED versions.

Unload any existing FastLinQ drivers as described in

“Removing Linux

Drivers in a RoCE Environment” on page 15 .

Install the latest FastLinQ driver and libqedr packages as described in

“Installing the Linux Drivers with RoCE” on page 18 .

Load the RDMA services.

systemctl start rdma modprobe qed modprobe qede modprobe qedr modprobe ib_iser modprobe ib_isert

Verify all RDMA and iSER modules loaded on the initiator and target devices using the lsmod | grep qed and lsmod | grep iser commands.

Verify that there are separate hca_id instances by typing the ibv_devinfo command, as shown in

Step 6

page 59

Check the RDMA connection on the initiator device and the target device.

Initiator device: rping -s -C 10 -v

Target device: rping -c -a 192.168.100.99 -C 10 -v

63 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for RHEL

The following figure shows an example of a successful RDMA ping.

You can use a Linux TCM-LIO target with which to test iSER. The setup is the same for any iSCSI target, except you type command enable_iser Boolean=true on the applicable portals. The portal instances are identified as "iser" as shown in the following figure:

Install Linux iSCSI Initiator Utilities using the yum install iscsi-initiator-utils commands.

Discover the iSER target by typing command such as the following:

iscsiadm -m discovery -t st -p 192.168.100.99:3260

Change the transport mode to iSER by typing command such as the following:

iscsiadm -m node -T iqn.2015-06.test.target1 -o update -n iface.transport_name -v iser

64 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for RHEL c.

Connect to or login to the iSER target by typing a command such as the following:

iscsiadm -m node -l -p 192.168.100.99:3260 -T iqn.2015-06.test.target1

Confirm that the Iface Transport is iser in the target connection, as shown in the following figure. Type a command such as the following:

iscsiadm -m session -P2

Check for a new iSCSI device, as shown in the following figure. Type command such as the following: lsscsi

65 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for SLES 12 Linux

The TargetCLI is not inbox on SLES12.x, therefore you must do the following:

To install TargetCLI, copy and install the following RPMs from ISO image

(x86_64 and noarch location).

lio-utils-4.1-14.6.x86_64.rpm

python-configobj-4.7.2-18.10.noarch.rpm

python-PrettyTable-0.7.2-8.5.noarch.rpm

python-configshell-1.5-1.44.noarch.rpm

python-pyparsing-2.0.1-4.10.noarch.rpm

python-netifaces-0.8-6.55.x86_64.rpm python-rtslib-2.2-6.6.noarch.rpm

python-urwid-1.1.1-6.144.x86_64.rpm

targetcli-2.1-3.8.x86_64.rpm

Before starting the TargetCLI, load all RoCE device drivers and iSER modules.

# modprobe qed

# modprobe qede

# modprobe qedr

# modprobe ib_iser (Initiator)

# modprobe ib_isert (Target)

Configure NIC interfaces and run L2 and RoCE traffic before configuring iSER targets as described in

Step 7

page 59

Start the TargetCLI utility, and configure your targets on the iSER target system.

NOTE

TargetCLI versions are different in RHEL and SLES. Be sure to use the proper backstores to configure your targets. RHEL uses ramdisk;

SLES uses rd_mcp.

66 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

To configure iSER, you must first configure RoCE, and then configure the target and initiator devices.

LIO Configuration as Target

To configure the Linux-IO (LIO) as target:

Install Targetcli by typing the following command:

# sudo apt-get install targetcli

Type the targetcli command to enter the LIO CLI console: root@captain:~# targetcli targetcli GIT_VERSION (rtslib GIT_VERSION)

/> ls o- /...................................................[...]

o- backstores ...................................... [...]

| o- fileio ........................... [0 Storage Object]

| o- iblock ........................... [0 Storage Object]

| o- pscsi ............................ [0 Storage Object]

| o- rd_dr ............................ [0 Storage Object]

| o- rd_mcp ........................... [0 Storage Object]

o- ib_srpt ................................... [0 Targets]

o- iscsi ..................................... [0 Targets]

o- loopback .................................. [0 Targets]

o- qla2xxx ................................... [0 Targets]

o- tcm_fc .................................... [0 Targets]

67 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

Create a disk of type rd_mcp with size 1G named iSERPort1-1 by typing the following command:

/> backstores/rd_mcp create name=iSERPort1-1 size=1G

Generating a wwn serial.

Created rd_mcp ramdisk iSERPort1-1 with size 1G.

/> ls o- / ................................................. [...]

o- backstores ...................................... [...]

| o- fileio ........................... [0 Storage Object]

| o- iblock ........................... [0 Storage Object]

| o- pscsi ............................ [0 Storage Object]

| o- rd_dr ............................ [0 Storage Object]

| o- rd_mcp ........................... [1 Storage Object]

| o- iSERPort1-1 ................. [ramdisk deactivated]

o- ib_srpt ................................... [0 Targets]

o- iscsi ..................................... [0 Targets]

o- loopback .................................. [0 Targets]

o- qla2xxx ................................... [0 Targets]

o- tcm_fc .................................... [0 Targets]

68 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

Create an iSCSI target by typing the following command:

/> iscsi/ create wwn=iqn.2004-01.com.qlogic.iSERPort1.Target1

Created target iqn.2004-01.com.qlogic.iSERPort1.Target1.

Selected TPG Tag 1.

Successfully created TPG 1.

/> ls o- / ................................................. [...]

o- backstores ...................................... [...]

| o- fileio ........................... [0 Storage Object]

| o- iblock ........................... [0 Storage Object]

| o- pscsi ............................ [0 Storage Object]

| o- rd_dr ............................ [0 Storage Object]

| o- rd_mcp ........................... [1 Storage Object]

| o- iSERPort1-1 ................. [ramdisk deactivated]

o- ib_srpt ................................... [0 Targets]

o- iscsi ...................................... [1 Target]

| o- iqn.2004-01.com.qlogic.iSERPort1.Target ..... [1 TPG]

| o- tpgt1 ................................... [enabled]

| o- acls ................................... [0 ACLs]

| o- luns ................................... [0 LUNs]

| o- portals ............................. [0 Portals]

o- loopback .................................. [0 Targets]

o- qla2xxx ................................... [0 Targets]

o- tcm_fc .................................... [0 Targets]

69 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

Create a LUN by typing the following command. Be sure that the RAM disk is activated:

/> /iscsi/iqn.2004-01.com.qlogic.iSERPort1.Target1/tpgt1/luns

create /backstores/rd_mcp/iSERPort1-1

Selected LUN 0.

Successfully created LUN 0.

/> ls o- / ................................................. [...]

o- backstores ...................................... [...]

| o- fileio ........................... [0 Storage Object]

| o- iblock ........................... [0 Storage Object]

| o- pscsi ............................ [0 Storage Object]

| o- rd_dr ............................ [0 Storage Object]

| o- rd_mcp ........................... [1 Storage Object]

| o- iSERPort1-1 ................... [ramdisk activated]

o- ib_srpt ................................... [0 Targets]

o- iscsi ...................................... [1 Target]

| o- iqn.2004-01.com.qlogic.iSERPort1.Target1 .... [1 TPG]

| o- tpgt1 ................................... [enabled]

| o- acls ................................... [0 ACLs]

| o- luns .................................... [1 LUN]

| | o- lun0 ........... [rd_mcp/iSERPort1-1 (ramdisk)]

| o- portals ............................. [0 Portals]

o- loopback .................................. [0 Targets]

o- qla2xxx ................................... [0 Targets]

o- tcm_fc .................................... [0 Targets]

70 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

Create a portal by typing the following commands. Supply the IP address of the interface on which to run iSER (RoCE enabled).

/> /iscsi/iqn.2004-01.com.qlogic.iSERPort1.Target1/tpgt1/port

als create 192.168.10.5

Using default IP port 3260

Successfully created network portal 192.168.10.5:3260.

/> ls o- / ................................................. [...]

o- backstores ...................................... [...]

| o- fileio ........................... [0 Storage Object]

| o- iblock ........................... [0 Storage Object]

| o- pscsi ............................ [0 Storage Object]

| o- rd_dr ............................ [0 Storage Object]

| o- rd_mcp ........................... [1 Storage Object]

| o- iSERPort1-1 ................... [ramdisk activated]

o- ib_srpt ................................... [0 Targets]

o- iscsi ...................................... [1 Target]

| o- iqn.2004-01.com.qlogic.iSERPort1.Target1 .... [1 TPG]

| o- tpgt1 ................................... [enabled]

| o- acls ................................... [0 ACLs]

| o- luns .................................... [1 LUN]

| | o- lun0 ........... [rd_mcp/iSERPort1-1 (ramdisk)]

| o- portals .............................. [1 Portal]

| o- 192.168.10.5:3260 ......... [OK, iser disabled]

o- loopback .................................. [0 Targets]

o- qla2xxx ................................... [0 Targets]

o- tcm_fc .................................... [0 Targets]

71 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

Enable iSER on the portal by typing the following command:

/> /iscsi/iqn.2004-01.com.qlogic.iSERPort1.Target1/tpgt1/port

als/192.168.10.103:3260 iser_enable

iser operation has been enabled

/> ls o- / ................................................. [...]

o- backstores ...................................... [...]

| o- fileio ........................... [0 Storage Object]

| o- iblock ........................... [0 Storage Object]

| o- pscsi ............................ [0 Storage Object]

| o- rd_dr ............................ [0 Storage Object]

| o- rd_mcp ........................... [1 Storage Object]

| o- iSERPort1-1 ................... [ramdisk activated]

o- ib_srpt ................................... [0 Targets]

o- iscsi ...................................... [1 Target]

| o- iqn.2004-01.com.qlogic.iSERPort1.Target1 .... [1 TPG]

| o- tpgt1 ................................... [enabled]

| o- acls ................................... [0 ACLs]

| o- luns .................................... [1 LUN]

| | o- lun0 ........... [rd_mcp/iSERPort1-1 (ramdisk)]

| o- portals .............................. [1 Portal]

| o- 192.168.10.103:3260 ........ [OK, iser enabled]

o- loopback .................................. [0 Targets]

o- qla2xxx ................................... [0 Targets]

o- tcm_fc .................................... [0 Targets]

72 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

Set No ACLs by typing the following command:

/> /iscsi/iqn.2004-01.com.qlogic.iSERPort1.Target1/tpgt1/ set

attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

Parameter demo_mode_write_protect is now '0'.

Parameter authentication is now '0'.

Parameter generate_node_acls is now '1'.

Parameter cache_dynamic_acls is now '1'.

/> ls o- / ................................................. [...]

o- backstores ...................................... [...]

| o- fileio ........................... [0 Storage Object]

| o- iblock ........................... [0 Storage Object]

| o- pscsi ............................ [0 Storage Object]

| o- rd_dr ............................ [0 Storage Object]

| o- rd_mcp ........................... [1 Storage Object]

| o- iSERPort1-1 ................... [ramdisk activated]

o- ib_srpt ................................... [0 Targets]

o- iscsi ...................................... [1 Target]

| o- iqn.2004-01.com.qlogic.iSERPort1.Target1 .... [1 TPG]

| o- tpgt1 ................................... [enabled]

| o- acls ................................... [0 ACLs]

| o- luns .................................... [1 LUN]

| | o- lun0 ........... [rd_mcp/iSERPort1-1 (ramdisk)]

| o- portals .............................. [1 Portal]

| o- 192.168.10.103:3260 ........ [OK, iser enabled]

o- loopback .................................. [0 Targets]

o- qla2xxx ................................... [0 Targets]

o- tcm_fc .................................... [0 Targets]

73 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

Save the configuration by typing the following command:

/> saveconfig

WARNING: Saving ratan-ProLiant-DL380p-Gen8 current configuration to disk will overwrite your boot settings.

The current target configuration will become the default boot config.

Are you sure? Type 'yes': yes

Making backup of fc/ConfigFS with timestamp:

2015-06-09_19:07:37.855693

Successfully updated default config /etc/target/fc_start.sh

Making backup of loopback/ConfigFS with timestamp:

2015-06-09_19:07:37.855693

Successfully updated default config

/etc/target/loopback_start.sh

Making backup of srpt/ConfigFS with timestamp:

2015-06-09_19:07:37.855693

Successfully updated default config /etc/target/srpt_start.sh

Making backup of qla2xxx/ConfigFS with timestamp:

2015-06-09_19:07:37.855693

Successfully updated default config

/etc/target/qla2xxx_start.sh

Making backup of LIO-Target/ConfigFS with timestamp:

2015-06-09_19:07:37.855693

Generated LIO-Target config:

/etc/target/backup/lio_backup-2015-06-09_19:07:37.855693.sh

Making backup of Target_Core_Mod/ConfigFS with timestamp:

2015-06-09_19:07:37.855693

Generated Target_Core_Mod config:

/etc/target/backup/tcm_backup-2015-06-09_19:07:37.855693.sh

Successfully updated default config /etc/target/lio_start.sh

Successfully updated default config /etc/target/tcm_start.sh

74 BC0154501-00 F

7–iSCSI Extensions for RDMA

Configuring iSER for Ubuntu

Initiator Configuration

To configure the initiator:

Load the ib_iser module and confirm that it is loaded properly by typing the following commands:

# sudo modprobe ib_iser

# lsmod | grep ib_iser ib_isert 56835 2 iscsi_target_mod 307333 6 ib_isert target_core_mod 382144 22 target_core_iblock,tcm_qla2xxx,target_core_pscsi,iscsi_target

_mod,tcm_fc,ib_srpt,target_core_file,target_core_user,tcm_loo p,ib_isert ib_iser 52919 0 rdma_cm 48739 3 ib_iser,rdma_ucm,ib_isert ib_core 98710 15 qedr,rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,ib_mad,ib_ucm,ib_iser, ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert libiscsi 57498 3 libiscsi_tcp,iscsi_tcp,ib_iser scsi_transport_iscsi 100628 4 iscsi_tcp,ib_iser,libiscsi

Type the iscsiadm command to discover the target and change the transport mode to iSER.

# iscsiadm -m discovery -t st -p 192.168.10.5:3260 -I iser

192.168.10.5:3260,1 iqn.2004-01.com.qlogic.iSERPort1.Target1

# iscsiadm -m node -T iqn.2004-01.com.qlogic.iSERPort1.Target1

-o update -n iface.transport_name -v iser

# iscsiadm -m node -l

Logging in to [iface: default, target: iqn.2004-01.com.qlogic.iSERPort1.Target1, portal:

192.168.10.5,3260] (multiple)

192.168.10.5,3260] successful.

75 BC0154501-00 F

7–iSCSI Extensions for RDMA

Optimizing Linux Performance

Verify that the LUNs are visible by typing the following commands:

# sudo apt-get install lsscsi

# lsscsi

[1:0:0:0] cd/dvd hp DVD D DS8D9SH JHJ4 /dev/sr0

[2:0:0:0] disk HP LOGICAL VOLUME 4.68 /dev/sda

[2:0:0:1] disk HP LOGICAL VOLUME 4.68 -

[2:3:0:0] storage HP P420i 4.68 -

[3:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdb

Optimizing Linux Performance

Consider the Linux performance enhancements.

Set CPUs to Maximum Performance Mode

Set the CPU scaling governor to performance by using the following script to set all CPUs to maximum Performance mode for CPUFREQ in

/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f

$CPUFREQ ] || continue; echo -n performance > $CPUFREQ; done

Verify that all CPU cores were set to maximum performance mode by typing the following command: cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Set Kernel sysctl Settings

Set the kernel sysctl settings as follows: sysctl -w net.ipv4.tcp_mem="4194304 4194304 4194304" sysctl -w net.ipv4.tcp_wmem="4096 65536 4194304" sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304" sysctl -w net.core.wmem_max=4194304 sysctl -w net.core.rmem_max=4194304 sysctl -w net.core.wmem_default=4194304 sysctl -w net.core.rmem_default=4194304 sysctl -w net.core.netdev_max_backlog=250000 sysctl -w net.ipv4.tcp_timestamps=0 sysctl -w net.ipv4.tcp_sack=1 sysctl -w net.ipv4.tcp_low_latency=1 sysctl -w net.ipv4.tcp_adv_win_scale=1 echo 0 > /proc/sys/vm/nr_hugepages

76 BC0154501-00 F

7–iSCSI Extensions for RDMA

Optimizing Linux Performance

IRQ Affinity Settings

This example sets CPU core 0, 1, 2 and 3 to IRQ XX, YY, ZZ and XYZ respectively. This will need to be done for each IRQ assigned to a port (default is eight queues per port) systemctl disable irqbalance systemctl stop irqbalance cat /proc/interrupts | grep qedr Shows IRQ assigned to each port queue echo 1 > /proc/irq/XX/smp_affinity_list echo 2 > /proc/irq/YY/smp_affinity_list echo 4 > /proc/irq/ZZ/smp_affinity_list echo 8 > /proc/irq/XYZ/smp_affinity_list

Block Device Staging

Set the block device staging settings for each iSCSI device/target: echo noop > /sys/block/sdd/queue/scheduler echo 2 > /sys/block/sdd/queue/nomerges echo 0 > /sys/block/sdd/queue/add_random echo 1 > /sys/block/sdd/queue/rq_affinity

77 BC0154501-00 F

Windows Server 2016



Configuring RoCE Interfaces with Hyper-V (NDKPI Mode-2)



RoCE Over Switch Embedded Teaming (SET)



Configuring Quality of Service for RoCE



Configuring VMMQ



Configuring VXLAN



Configuring Storage Spaces Direct



Nano Server

78 BC0154501-00 F

8–Windows Server 2016

Configuring RoCE Interfaces with Hyper-V (NDKPI Mode-2)

Configuring RoCE Interfaces with Hyper-V

(NDKPI Mode-2)

In Windows Server 2016, Hyper-V host virtual network adapter (Host virtual NICs) supports RDMA.

Creating a Hyper-V Virtual Switch with an RDMA Virtual NIC

Use the following procedure to create a Hyper-V virtual switch with an RDMA virtual NIC:

Launch Hyper-V Manager.

Click Virtual Switch Manager.

Create a virtual switch.

Check the Allow management operating system to share this network

adapter check box.

79 BC0154501-00 F

8–Windows Server 2016

Configuring RoCE Interfaces with Hyper-V (NDKPI Mode-2)

Enabling RDMA in Host Virtual NIC

In Windows Server 2016, a new parameter Network Direct (RDMA) has been added in Host virtual NIC.

Use the following to enable RDMA in host virtual NIC:

Open the Hyper-V Virtual Ethernet Adapter Properties window.

Click the Advanced tab.

Select Network Direct (RDMA) in the Property list.

Select Enabled in the Value drop-down option.

Click OK.

Use the following PowerShell command to enable RDMA:

PS C:\Users\Administrator> Enable-NetAdapterRdma “vEthernet (New

Virtual Switch)”

PS C:\Users\Administrator>

80 BC0154501-00 F

8–Windows Server 2016

Configuring RoCE Interfaces with Hyper-V (NDKPI Mode-2)

Add VLAN ID to Host Virtual NIC

Use the following procedure to add VLAN ID to host virtual NIC:

Enter the following PowerShell command to find the host virtual NIC name:

PS C:\Users\Administrator> Get-VMNetworkAdapter

-ManagementOS

Enter the following PowerShell command to set VLAN ID to host virtual NIC.

PS C:\Users\Administrator> Set-VMNetworkAdaptervlan

-VMNetworkAdapterName “New Virtual Switch” -VlanId 5

-Access -Management05

NOTE

A VLAN ID needs to be assigned to a host virtual NIC. The same

VLAN ID should be assigned to all the interfaces, and on the switch.

Verifying If RoCE is Enabled

Use the following procedure to verify if the RoCE is enabled:

 Enter the following PowerShell command:

Get-NetAdapterRdma

This will list the RDMA supported adapters as shown below.

81 BC0154501-00 F

8–Windows Server 2016

Configuring RoCE Interfaces with Hyper-V (NDKPI Mode-2)

Adding More Host Virtual NICs (Virtual Ports)

Use the following procedure to add more host virtual NICs:

Enter the following command to add a host virtual NIC:

Add-VMNetworkAdapter -SwitchName “New Virttual Switch”

-Name SMB - ManagementOS

Enable RDMA on host virtual NICs as shown in “Enabling RDMA in Host

Virtual NIC” on page 80

Enter the following command to assign a VLAN ID to the virtual port:

Set-VMNetworkAdapterVlan -VMNetworkAdapterName SMB

-VlanId 5 -Access -ManagementOS

82 BC0154501-00 F

8–Windows Server 2016

Configuring RoCE Interfaces with Hyper-V (NDKPI Mode-2)

Mapping the SMB Drive and Running RoCE Traffic

Use the following procedure to map the SMB drive, and run the RoCE traffic:

Launch Perfmon.

Select Host vNIC add RDMA Activity for the adapters as shown below:

If the RoCE traffic is running, counters will look the following:

83 BC0154501-00 F

8–Windows Server 2016

RoCE Over Switch Embedded Teaming (SET)

Creating a Hyper-V Virtual Switch with SET and RDMA Virtual NICs

Use the following procedure to create a Hyper-V virtual switch with SET and

RDMA virtual NICs:

 Enter the following PowerShell command to create a SET:

PS C:\Users\Administrator> New-VMSwitch -Name SET

-NetAdapterName “Ethernet 2”,”Ethernet 3”

-EnableEmbeddedTeaming $true

Enabling RDMA on SET

Use the following procedure to enable RDMA on SET:

Enter the following PowerShell command to enable RDMA on SET:

PS C:\Users\Administrator> Get-NetAdapter “vEthernet (SET)”

Enter the following PowerShell command to enable RDMA on SET:

PS C:\Users\Administrator> Enable-NetAdapterRdma “vEthernet

(SET)”

Assigning VLAN ID on SET

Use the following PowerShell command to assign VLAN ID on SET:

PS C:\Users\Administrator> New-VMSwitch -Name SET -NetAdapterName

“Ethernet 2”,”Ethernet 3” -EnableEmbeddedTeaming $true

Running RDMA Traffic on SET

See https://technet.microsoft.com/en-us/library/mt403349.aspx

for more information.

84 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

There are two methods of configuring quality of service:



Configuring Quality of Service by Disabling DCBX on the Adapter



Configuring Quality of Service by Enabling DCBX on the Adapter

Configuring Quality of Service by Disabling DCBX on the Adapter

All the configuration needs to be completed on all the systems in use before configuring quality of service by disabling DCBX on the adapter. The priority-based flow control (PFC), enhanced transition services (ETS), and traffic classes configuration should be the same on the switch and server.

Disable DCBX.

Set RoCE Priority as 0 through HII.

Enter the following PowerShell command to install DCB Role in host:

PS C:\Users\Administrators> Install-WindowsFeature

Data-Center-Bridging

Enter the following PowerShell command to set DCBX Willing mode to

False:

PS C:\Users\Administrators> set-NetQosDcbxSetting

-Willing 0

Enable quality of service in mini port by completing the following: a.

Open the mini port window, and click the Advanced tab.

Select Quality of Service in the Property list.

Select Enabled in the Value drop-down option.

Click OK.

85 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

Assign VLAN ID to the interface by completing the following: a.

Open the mini port window, and click the Advanced tab.

Select VLAN ID in the Property list.

Set a value in the Value box.

Click OK.

NOTE

This is required for priority flow control (PFC).

86 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

Enter the following PowerShell command to enable priority flow control for

RoCE on a specific priority:

PS C:\Users\Administrators> Enable-NetQoSFlowControl

-Priority 4

87 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

Enter the following PowerShell command to disable priority flow control on any other priority:

PS C:\Users\Administrators> Disable-NetQosFlowControl

0,1,2,3,5,6,7

PS C:\Users\Administrator> Get-NetQosFlowControl

Priority Enabled PolicySet IfIndex IfAlias

-------- ------- --------- ------- -------

0 False Global

1 False Global

2 False Global

3 False Global

4 True Global

5 False Global

6 False Global

7 False Global

Enter the following PowerShell command to configure quality of service and assign relevant priority to each type of traffic:

NOTE

In the example below, Priority 4 is tagged for RoCE and Priority 0 for

TCP.

PS C:\Users\Administrators> New-NetQosPolicy “SMB”

-NetDirectPortMatchCondition 445 -PriorityValue8021Action 4

-PolicyStore ActiveStore

PS C:\Users\Administrators> New-NetQosPolicy “TCP”

-IPProtocolMatchCondition TCP -PriorityValue8021Action 0

-Policystore ActiveStore

PS C:\Users\Administrator> Get-NetQosPolicy -PolicyStore activestore

Name : tcp

Owner : PowerShell / WMI

NetworkProfile : All

Precedence : 127

JobObject :

IPProtocol : TCP

PriorityValue : 0

Name : smb

Owner : PowerShell / WMI

NetworkProfile : All

Precedence : 127

JobObject :

NetDirectPort : 445

PriorityValue : 4

88 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

10. Enter the following PowerShell command to configure ETS for all traffic classes defined above:

PS C:\Users\Administrators> New-NetQosTrafficClass -name "RDMA class"

-priority 4 -bandwidthPercentage 50 -Algorithm ETS

PS C:\Users\Administrators> New-NetQosTrafficClass -name "TCP class" -priority

0 -bandwidthPercentage 30 -Algorithm ETS

PS C:\Users\Administrator> Get-NetQosTrafficClass

Name

----

Algorithm Bandwidth(%) Priority PolicySet IfIndex IfAlias

--------- ------------ -------- --------- ------- -------

[Default] ETS 20 2-3,5-7

RDMA class ETS 50 4

TCP class ETS 30 0

Global

11.

Enter the following PowerShell command to see the network adapter quality of service from the above configuration:

PS C:\Users\Administrator> Get-NetAdapterQos

Name : SLOT 4 Port 1

Enabled : True

Capabilities : Hardware Current

-------- -------

MacSecBypass : NotSupported NotSupported

DcbxSupport : None None

NumTCs(Max/ETS/PFC) : 4/4/4 4/4/4

OperationalTrafficClasses : TC TSA Bandwidth Priorities

-- --- --------- ----------

0 ETS 20% 2-3,5-7

1 ETS 50% 4

2 ETS 30% 0

OperationalFlowControl : Priority 4 Enabled

OperationalClassifications : Protocol Port/Type Priority

-------- --------- --------

Default 0

NetDirect 445 4

12. Create a startup script to make the settings persistent across the system reboots.

89 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

13. Run RDMA traffic and verify as mentioned in

“Configuring RoCE” on page 41 .

Configuring Quality of Service by Enabling DCBX on the Adapter

All the configuration needs to be completed on all the systems in use. The priority-based flow control (PFC), enhanced transition services (ETS), and traffic classes configuration should be the same on the switch and server.

Enable DCBX (IEEE/CEE/Dynamic).

Set RoCE Priority as 0 through HII.

Enter the following PowerShell command to install DCB Role in host:

PS C:\Users\Administrators> Install-WindowsFeature

Data-Center-Bridging

NOTE

For this configuration, set DCBX Protocol as CEE.

Enter the following PowerShell command to set the DCBX Willing mode to

True:

PS C:\Users\Administrators> set-NetQosDcbxSetting

-Willing 1

Enable quality of service in mini port by completing the following: a.

Open the mini port window, and click the Advanced tab.

Select Quality of Service in the Property list.

Select Enabled in the Value drop-down option.

Click OK.

90 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

Assign VLAN ID to the interface by completing the following: a.

Open the mini port window, and click the Advanced tab.

Select VLAN ID in the Property list.

Set a value in the Value box.

Click OK.

NOTE

This is required for priority flow control (PFC).

91 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

Enter the following PowerShell command to configure the switch:

NOTE

The example below is taken when the adapter port is connected to

Arista 7060X switch. In this example, on switch, PFC is enabled on

Priority 4. RoCE App TLVs are defined. There are two traffic classes defined TC0 and TC1. TC1 is defined for RoCE. DCBX Protocol

mode is set to CEE. For Arista switch configuration, refer to “Preparing the Ethernet Switch” on page 44

“. The adapter, being in Willing mode, accepts Remote Configuration and shows it as Operational

Parameters.

92 BC0154501-00 F

8–Windows Server 2016

Configuring Quality of Service for RoCE

PS C:\Users\Administrators> Get-NetAdapterQoS

Name : Ethernet 5

Enabled : True

Capabilities : Hardware Current

-------- -------

MacSecBypass : NotSupported NotSupported

DcbxSupport : CEE CEE

NumTCs(Max/ETS/PFC) : 4/4/4 4/4/4

OperationalTrafficClasses : TC TSA Bandwidth Priorities

-- --- --------- ----------

0 ETS 5% 0-3,5-7

1 ETS 95% 4

OperationalFlowControl : Priority 4 Enabled

OperationalClassifications : Protocol Port/Type Priority

-------- --------- --------

NetDirect 445 4

RemoteTrafficClasses : TC TSA Bandwidth Priorities

-- --- --------- ----------

0 ETS 5% 0-3,5-7

1 ETS 95% 4

RemoteFlowControl : Priority 4 Enabled

RemoteClassifications : Protocol Port/Type Priority

-------- --------- --------

NetDirect 445 4

93 BC0154501-00 F

8–Windows Server 2016

Configuring VMMQ

Enabling VMMQ on Adapter

Use the following procedure to enable VMMQ on the adapter:

Open the mini port window, and click the Advanced tab.

Select Virtual Switch RSS in the Property list.

Select Enabled in the Value drop-down option.

Click OK.

94 BC0154501-00 F

8–Windows Server 2016

Configuring VMMQ

Creating Virtual Machine Switch With or Without SRIOV

Use the following procedure to create a virtual machine switch with or without

SRIOV:

Launch the Hyper-V Manager.

Select Virtual Switch Manager.

Type a value in the Name box.

Select External network.

Check the Allow management operating system to share this network

adapter check box.

Click OK.

95 BC0154501-00 F

8–Windows Server 2016

Configuring VMMQ

Enabling VMMQ on the Virtual Machine Switch

Use the following PowerShell command to enable VMMQ on the virtual machine switch:

PS C:\Users\Administrators> Set-VMSwitch -name q1

-defaultqueuevmmqenabled $true -defaultqueuevmmqqueuepairs 4

Getting the Virtual Machine Switch Capability

Use the following PowerShell command to get the virtual machine switch capability:

PS C:\Users\Administrator> Get-VMSwitch -Name ql | fl

96 BC0154501-00 F

8–Windows Server 2016

Configuring VMMQ

Creating a Virtual Machine and Enabling VMMQ on

VMNetworkadapters in the Virtual Machine

Use the following procedure to create a virtual machine and enable VMMW on

VMNetworksadapters in the virtual machine:

Create a virtual machine.

Add VMNetworkadapter to the virtual machine.

Assign a virtual switch to the VMNetworkadapter.

Enter the following PowerShell command to enable VMMQ on virtual machine:

PS C:\Users\Administrators> set-vmnetworkadapter -vmname vm1 -VMNetworkAdapterName "network adapter" -vmmqenabled

$true -vmmqqueuepairs 4

NOTE

For SR-IOV capable virtual switch: If the virtual machine switch and hardware acceleration is SR-IOV enabled, then you will need to create

10 virtual machines, with 8 virtual NICs each, to utilize VMMQ. This is because the SR-IOV has precedence over VMMQ.

The output of 64 virtual functions and 16 VMMQs is shown below:

PS C:\Users\Administrator> get-netadaptervport

Name ID MacAddress VID ProcMask FID State ITR QPairs

---- -- ---------- --- -------- --- ----- --- ------

Ethernet 3 0 00-15-5D-36-0A-FB 0:0 PF Activated Unknown 4

Ethernet 3 1 00-0E-1E-C4-C0-A4 0:8 PF Activated Adaptive 4

Ethernet 3 2 0:0 0 Activated Unknown 1

Ethernet 3 3 0:0 1 Activated Unknown 1

Ethernet 3 4 0:0 2 Activated Unknown 1

Ethernet 3 5 0:0 3 Activated Unknown 1

Ethernet 3 6 0:0 4 Activated Unknown 1

Ethernet 3 7 0:0 5 Activated Unknown 1

Ethernet 3 8 0:0 6 Activated Unknown 1

Ethernet 3 9 0:0 7 Activated Unknown 1

Ethernet 3 10 0:0 8 Activated Unknown 1

Ethernet 3 11 0:0 9 Activated Unknown 1

Ethernet 3 12 0:0 10 Activated Unknown 1

Ethernet 3 13 0:0 11 Activated Unknown 1

Ethernet 3 14 0:0 12 Activated Unknown 1

Ethernet 3 15 0:0 13 Activated Unknown 1

Ethernet 3 16 0:0 14 Activated Unknown 1

Ethernet 3 17 0:0 15 Activated Unknown 1

Ethernet 3 18 0:0 16 Activated Unknown 1

Ethernet 3 19 0:0 17 Activated Unknown 1

Ethernet 3 20 0:0 18 Activated Unknown 1

Ethernet 3 21 0:0 19 Activated Unknown 1

Ethernet 3 22 0:0 20 Activated Unknown 1

Ethernet 3 23 0:0 21 Activated Unknown 1

97 BC0154501-00 F

8–Windows Server 2016

Configuring VMMQ

Ethernet 3 24 0:0 22 Activated Unknown 1

Ethernet 3 25 0:0 23 Activated Unknown 1

Ethernet 3 26 0:0 24 Activated Unknown 1

Ethernet 3 27 0:0 25 Activated Unknown 1

Ethernet 3 28 0:0 26 Activated Unknown 1

Ethernet 3 29 0:0 27 Activated Unknown 1

Ethernet 3 30 0:0 28 Activated Unknown 1

Ethernet 3 31 0:0 29 Activated Unknown 1

Ethernet 3 32 0:0 30 Activated Unknown 1

Ethernet 3 33 0:0 31 Activated Unknown 1

Ethernet 3 34 0:0 32 Activated Unknown 1

Ethernet 3 35 0:0 33 Activated Unknown 1

Ethernet 3 36 0:0 34 Activated Unknown 1

Ethernet 3 37 0:0 35 Activated Unknown 1

Ethernet 3 38 0:0 36 Activated Unknown 1

Ethernet 3 39 0:0 37 Activated Unknown 1

Ethernet 3 40 0:0 38 Activated Unknown 1

Ethernet 3 41 0:0 39 Activated Unknown 1

Ethernet 3 42 0:0 40 Activated Unknown 1

Ethernet 3 43 0:0 41 Activated Unknown 1

Ethernet 3 44 0:0 42 Activated Unknown 1

Ethernet 3 45 0:0 43 Activated Unknown 1

Ethernet 3 46 0:0 44 Activated Unknown 1

Ethernet 3 47 0:0 45 Activated Unknown 1

Ethernet 3 48 0:0 46 Activated Unknown 1

Ethernet 3 49 0:0 47 Activated Unknown 1

Ethernet 3 50 0:0 48 Activated Unknown 1

Ethernet 3 51 0:0 49 Activated Unknown 1

Ethernet 3 52 0:0 50 Activated Unknown 1

Ethernet 3 53 0:0 51 Activated Unknown 1

Ethernet 3 54 0:0 52 Activated Unknown 1

Ethernet 3 55 0:0 53 Activated Unknown 1

Ethernet 3 56 0:0 54 Activated Unknown 1

Ethernet 3 57 0:0 55 Activated Unknown 1

Ethernet 3 58 0:0 56 Activated Unknown 1

Ethernet 3 59 0:0 57 Activated Unknown 1

Ethernet 3 60 0:0 58 Activated Unknown 1

Ethernet 3 61 0:0 59 Activated Unknown 1

Ethernet 3 62 0:0 60 Activated Unknown 1

Ethernet 3 63 0:0 61 Activated Unknown 1

Ethernet 3 64 0:0 62 Activated Unknown 1

Ethernet 3 65 0:0 63 Activated Unknown 1

Ethernet 3 66 00-15-5D-36-0A-04 0:16 PF Activated Adaptive 4

Ethernet 3 67 00-15-5D-36-0A-05 1:0 PF Activated Adaptive 4

Ethernet 3 68 00-15-5D-36-0A-06 0:0 PF Activated Adaptive 4

Ethernet 3 69 00-15-5D-36-0A-07 0:8 PF Activated Adaptive 4

Ethernet 3 70 00-15-5D-36-0A-08 0:16 PF Activated Adaptive 4

Ethernet 3 71 00-15-5D-36-0A-09 1:0 PF Activated Adaptive 4

Ethernet 3 72 00-15-5D-36-0A-0A 0:0 PF Activated Adaptive 4

Ethernet 3 73 00-15-5D-36-0A-0B 0:8 PF Activated Adaptive 4

Ethernet 3 74 00-15-5D-36-0A-F4 0:16 PF Activated Adaptive 4

Ethernet 3 75 00-15-5D-36-0A-F5 1:0 PF Activated Adaptive 4

Ethernet 3 76 00-15-5D-36-0A-F6 0:0 PF Activated Adaptive 4

Ethernet 3 77 00-15-5D-36-0A-F7 0:8 PF Activated Adaptive 4

Ethernet 3 78 00-15-5D-36-0A-F8 0:16 PF Activated Adaptive 4

Ethernet 3 79 00-15-5D-36-0A-F9 1:0 PF Activated Adaptive 4

Ethernet 3 80 00-15-5D-36-0A-FA 0:0 PF Activated Adaptive 4

PS C:\Users\Administrator> get-netadaptervmq

Name InterfaceDescription

---- --------------------

Ethernet 4 QLogic FastLinQ QL45212-DE...#238 False 0:0

Queues

16 1

98 BC0154501-00 F

8–Windows Server 2016

Configuring VMMQ

Default and Maximum VMMQ Virtual NIC

Maximum VMMQ per Virtual NICs

4 VMMQs are available per virtual NICs, up to 16 virtual NICs according to current implementation

Default Queue

4 Default queues are available as set earlier using PowerShell commands. The maximum default queue can be set to 8 as of now. Use the VMswitch capability to verify maximum default queue as shown above.

Enabling/Disabling VMMQ on Management NIC

Use the following procedure to enable or disable VMMW on management NIC:

 Enabling VMMW on Management NIC:

PS C:\Users\Administrator> Set-VMNetworkAdapter

–ManagementOS –vmmqEnabled $true

MOS vnic will have 4 VMMQ as well

 Disabling VMMW on Management NIC:

PS C:\Users\Administrator> Set-VMNetworkAdapter

–ManagementOS –vmmqEnabled $false

A VMQ will be available for MOS PF as well

Traffic Statistics

Use the following PowerShell command to monitor virtual function traffic in virtual machine:

PS C:\Users\Administrator> Use get-netadapterstatistics | fl

99 BC0154501-00 F

8–Windows Server 2016

Configuring VXLAN

Enable VXLAN Offload on the Adapter

Use the following procedure to enable VXLAN offload on the adapter:

Open the mini port window, and click the Advanced tab.

Select VXLAN Encapsulated Task Offload in the Property list.

Select Enabled in the Value drop-down option.

Click OK.

Deploy a Software Defined Network (SDN)

In order to take advantage of VXLAN encapsulation task offload on virtual machines, you must deploy a software defined network (SDN) which utilizes a

Microsoft Network Controller.

Refer to Microsoft TechNet link on Software Defined Networking for more details: https://technet.microsoft.com/en-us/windows-server-docs/networking/sdn/softwar e-defined-networking--sdn-

100 BC0154501-00 F

8–Windows Server 2016

Configuring Storage Spaces Direct

Windows Server 2016 introduces Storage Spaces Direct, which allows you to build highly available, and scalable storage systems with local storage.

Refer to Microsoft TechnNet link for more information: https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces

/storage-spaces-direct-windows-server-2016

Hardware Configuration

The following is a sample hardware configuration:

NOTE

The disks used in this example are 4 x 400G NVMe, and 12 x 200G SSD disks.

101 BC0154501-00 F

8–Windows Server 2016

Configuring Storage Spaces Direct

Hyper-Converged Deployment Procedure

This section includes instructions to install, and configure the components of a

Hyper-Converged system using the Windows Server 2016. The act of deploying a

Hyper-Converged system can be divided into the following three high level phases:



Deploying the Operating System



Configuring the Network



Configuring Storage Spaces Direct

Deploying the Operating System

Use the following procedure to deploy the operating systems:

Install the operating system.

Install the Windows server roles (Hyper-V).

Install the following features:

 failover

 cluster

 data center bridging (DCB)

Connect the nodes to domain and adding domain accounts

Configuring the Network

In order to deploy Storage Spaces Direct, the Hyper-V switch must be deployed with RDMA-enabled host virtual NICs.

NOTE

The following assumes 4 RDMA NIC ports (E4 Baffin bay adapters configured in 4x25G mode).

102 BC0154501-00 F

8–Windows Server 2016

Configuring Storage Spaces Direct

Use the following procedure to configure the network on each server:

Complete the following to configure the physical network switch: a.

Connect all adapter NICs to the switch port.

NOTE

If your test adapter has more than one NIC port, you will need to connect both ports to same switch.

Enable the switch port and make sure the switch port supports switch independent teaming mode, and also part of multiple VLAN networks.

The following is a sample Dell switch configuration:

no ip address mtu 9416 portmode hybrid switchport dcb-map roce_S2D protocol lldp dcbx version cee no shutdown

Enable Network Quality of Service.

NOTE

Network Quality of Service is used to ensure that the Software Defined

Storage system has enough bandwidth to communicate between the nodes to ensure resiliency and performance. Refer to

“Configuring

Quality of Service for RoCE” on page 85

“ to configure quality of service on adapter

Complete the following to create a Hyper-V virtual switch with SET and

RDMA virtual NIC: a.

Enter the following command to identify the network adapters:

Get-NetAdapter | FT

Name,InterfaceDescription,Status,LinkSpeed

Enter the following command to create the virtual switch connected to all the physical network adapter, and enable the switch embedded teaming:

New-VMSwitch -Name SETswitch -NetAdapterName

“<port1>”,“<port2>”,“<port3>”,“<port4>”

–EnableEmbeddedTeaming $true

103 BC0154501-00 F

8–Windows Server 2016

Configuring Storage Spaces Direct c.

Enter the following command to add host virtual NICs to the virtual switch:

Add-VMNetworkAdapter –SwitchName SETswitch –Name SMB_1

–managementOS

Add-VMNetworkAdapter –SwitchName SETswitch –Name SMB_2

–managementOS

NOTE

This configures the virtual NIC, from the virtual switch that you just configured for the management operating system to use.

Enter the following command to configure the host virtual NIC to use a

VLAN:

Set-VMNetworkAdapterVlan -VMNetworkAdapterName "SMB_1"

-VlanId 5 -Access -ManagementOS

Set-VMNetworkAdapterVlan -VMNetworkAdapterName "SMB_2"

-VlanId 5 -Access -ManagementOS

NOTE

These can be on the same or different VLANs.

Enter the following command to verify the VLAN ID is set:

Get-VMNetworkAdapterVlan -ManagementOS

Enter the following command to disable and enable each host virtual

NIC adapter so that the VLAN is active:

Disable-NetAdapter “vEthernet (SMB_1)”

Enable-NetAdapter “vEthernet (SMB_1)”

Disable-NetAdapter “vEthernet (SMB_2)”

Enable-NetAdapter “vEthernet (SMB_2)”

Enter the following command to enable RDMA on the host virtual NIC adapters:

Enable-NetAdapterRdma "SMB1","SMB2"

Enter the following command to verify RDMA capabilities:

Get-SmbClientNetworkInterface | where RdmaCapable -EQ

$true

104 BC0154501-00 F

8–Windows Server 2016

Configuring Storage Spaces Direct

Configuring Storage Spaces Direct in Windows Server 2016 includes the following steps:



Step 1 - Running Cluster Validation Tool



Step 2 - Creating a Cluster r



Step 3 - Configuring a Cluster Witness



Step 4 - Cleaning Disks Used for Storage Spaces Direct



Step 5 - Enabling Storage Spaces Direct



Step 6 - Creating Virtual Disks



Step 7 - Creating or Deploying Virtual Machines

Step 1 - Running Cluster Validation Tool

Run the cluster validation tool to make sure server nodes are configured correctly to create a cluster using Storage Spaces Direct.

Use the following PowerShell command to validate a set of servers for use as

Storage Spaces Direct cluster:

Test-Cluster -Node <MachineName1, MachineName2, MachineName3,

MachineName4> -Include "Storage Spaces Direct", Inventory,

Network, "System Configuration"

Step 2 - Creating a Cluster

Create a cluster with the four nodes (which was validated for cluster creation) in the Step 1.

Use the following PowerShell command to create a cluster.

New-Cluster -Name <ClusterName> -Node <MachineName1, MachineName2,

MachineName3, MachineName4> -NoStorage

The –NoStorage parameter is important here. Otherwise, the disks will be added automatically to the cluster, and you will need to remove them before enabling

Storage Spaces Direct, or they will not be included in the Storage Spaces Direct storage pool.

Step 3 - Configuring a Cluster Witness

It is recommended that to configure a witness for the cluster, so that this four node system can withstand two nodes failing or being offline. With these systems you can configure file share witness or cloud witness.

For more information, see https://blogs.msdn.microsoft.com/clustering/2014/03/31/configuring-a-file-share-w itness-on-a-scale-out-file-server/

105 BC0154501-00 F

8–Windows Server 2016

Configuring Storage Spaces Direct

Step 4 - Cleaning Disks Used for Storage Spaces Direct

The disks intended to be used for Storage Spaces Direct need to be empty, and without partitions, or other data. If a disk has partitions or other data, it will not be included in the Storage Spaces Direct system.

The following PowerShell command can be placed in a PowerShell script (.PS1) file, and executed from the management system in a open PowerShell (or

PowerShell ISE) console with Administrator privileges.

NOTE

Running this script will help identify the disks on each node that can be used for Storage Spaces Direct, and removes all data and partitions from those disks.

icm (Get-Cluster -Name HCNanoUSClu3 | Get-ClusterNode) {

Update-StorageProviderCache

Get-StoragePool |? IsPrimordial -eq $false |

Set-StoragePool -IsReadOnly:$false -ErrorAction

SilentlyContinue

Get-StoragePool |? IsPrimordial -eq $false |

Get-VirtualDisk | Remove-VirtualDisk -Confirm:$false

-ErrorAction SilentlyContinue

Get-StoragePool |? IsPrimordial -eq $false |

Remove-StoragePool -Confirm:$false -ErrorAction

SilentlyContinue

Get-PhysicalDisk | Reset-PhysicalDisk -ErrorAction

SilentlyContinue

Get-Disk |? Number -ne $null |? IsBoot -ne $true |?

IsSystem -ne $true |? PartitionStyle -ne RAW |% {

$_ | Set-Disk -isoffline:$false

$_ | Set-Disk -isreadonly:$false

$_ | Clear-Disk -RemoveData -RemoveOEM -Confirm:$false

$_ | Set-Disk -isreadonly:$true

$_ | Set-Disk -isoffline:$true

}

Get-Disk |? Number -ne $null |? IsBoot -ne $true |?

IsSystem -ne $true |? PartitionStyle -eq RAW | Group

-NoElement -Property FriendlyName

} | Sort -Property PsComputerName,Count

106 BC0154501-00 F

8–Windows Server 2016

Configuring Storage Spaces Direct

Step 5 - Enabling Storage Spaces Direct

After creating the cluster, use the Enable-ClusterStorageSpacesDirect

PowerShell cmdlet, which will put the storage system into the Storage Spaces

Direct mode, and do the following automatically:

 Create a single large Pool that has a name like S2D on Cluster1.

 Configures Storage Spaces Direct cache. If there is more than one media type available for Storage Spaces Direct use, it will configure the most efficient as cache devices (read and write in most cases).

 Creates two tiers as default tiers. One is called Capacity and the other called Performance. The cmdlet analyzes the devices and configures each tier with the mix of device types and resiliency.

Step 6 - Creating Virtual Disks

If the Storage Spaces Direct was enabled, it creates a single pool using all the disks. It will also name the pool (for example S2D on Cluster1), with the name of the cluster that is specified in the name.

The following PowerShell command creates a virtual disk with both mirror and parity resiliency on storage pool:

New-Volume -StoragePoolFriendlyName "S2D*" -FriendlyName

<VirtualDiskName> -FileSystem CSVFS_ReFS -StorageTierfriendlyNames

Capacity,Performance -StorageTierSizes <Size of capacity tier in size units, example: 800GB>, <Size of Performance tier in size units, example: 80GB> -CimSession <ClusterName>

Step 7 - Creating or Deploying Virtual Machines

At this point, you can provision the virtual machines onto the nodes of the hyper-converged S2D cluster.

The virtual machine's files should be stored on the systems CSV namespace (for example: c:\ClusterStorage\Volume1), just like clustered virtual machines on failover clusters.

107 BC0154501-00 F

8–Windows Server 2016

Nano Server

Introduction

Windows Server 2016 offers Nano Server as a new installation option. Nano

Server is a remotely administered server operating system optimized for private clouds and data centers. It is similar to Windows Server in Server Core mode, but significantly smaller, has no local logon capability, and only supports 64-bit applications, tools, and agents. The Nano Server takes less disk space, sets up faster, and requires fewer updates and restarts than Windows Server. When it does restart, it restarts much faster.

Table 8-1 shows the roles and features that are available in this release of Nano

Server, along with the Windows PowerShell options that will install the packages for them. Some packages are installed directly with their own Windows

PowerShell options (such as -Compute). Others are installed as extensions to the -Packages option, which you can combine in a comma-separated list.

Table 8-1. Roles and Features of Nano Server

Role or Feature Options

Hyper-V role

Failover Clustering

Hyper-V guest drivers for hosting

Nano Server as a virtual machine

Basic drivers for a variety of network adapters and storage controllers. This is the same set of drivers included in a Server Core installation of Windows Server

2016 Technical Preview.

-Compute

-Clustering

-GuestDrivers

-OEMDrivers

File Server role and other storage components

Windows Defender Antimalware, including a default signature file

Reverse forwarders for application compatibility, for example common application frameworks such as Ruby, Node.js, etc.

-ReverseForwarders

DNS Server Role

Desired State Configuration

(DSC)

Internet Information Server (IIS)

-Storage

-Defender

-Packages Microsoft-NanoServer-DNS-Package

-Packages Microsoft-NanoServer-DSC-Package

-Packages Microsoft-NanoServer-IIS-Package

108 BC0154501-00 F

8–Windows Server 2016

Nano Server

Table 8-1. Roles and Features of Nano Server (Continued)

Role or Feature

Host Support for Windows Containers

System Center Virtual Machine

Manager Agent

-Containers

Options

-Packages Microsoft-Windows-Server-SCVMM-

Package

-Packages Microsoft-Windows-Server-SCVMM-

Compute-Package

Note: Use this package only if you are monitoring Hyper-V. If you install this package, do not use the -Compute option for the Hyper-V role; instead use the -Packages option to install -Packages Microsoft-NanoServer-

Compute-Package, Microsoft-Windows-Server-

SCVMM-Compute-Package.

-Packages Microsoft-NanoServer-NPDS-Package

Network Performance Diagnostics Service (NPDS)

Data Center Bridging

-Packages Microsoft-NanoServer-DCB-Package

The next sections will describe how to configure a Nano Server image with the packages you will need, and how to add additional device drivers specific to

QLogic devices. It will also explain how to use Nano Server Recovery Console, how to manage Nano Server remotely, and how to run Ntttcp traffic from Nano

Server.

Deploying Nano Server on a Physical Server

Use the following procedure to create a Nano Server virtual hard disk (VHD) that will run on a physical server using the pre-installed device drivers:

Download Windows Server 2016 OS image.

Mount the ISO.

Copy the following files from the NanoServer folder to a folder on your hard drive:

 NanoServerImageGenerator.psm1

 Convert-WindowsImage.ps1

Start Windows PowerShell as an administrator.

Change directory to the folder where you pasted the files from Step 3.

109 BC0154501-00 F

8–Windows Server 2016

Nano Server

Import the NanoServerImageGenerator script by entering the following command:

Import-Module .\NanoServerImageGenerator.psm1 -Verbose

Enter the following power shell command to create a VHD that sets a computer name and includes the OEM drivers and Hyper-V:

NOTE

This command will prompt you for an administrator password for the new VHD.

New-NanoServerImage –DeploymentType Host –Edition

<Standard/Datacenter> -MediaPath <path to root of media> -BasePath

.\Base -TargetPath .\NanoServerPhysical\NanoServer.vhd -ComputerName

<computer name> –Compute -Storage -Cluster -OEMDrivers –Compute

-DriversPath “<Path to Qlogic Driver sets>”

For Example:

New-NanoServerImage –DeploymentType Host –Edition Datacenter

-MediaPath C:\tmp\TP4_iso\Bld_10586_iso

-BasePath ".\Base" -TargetPath

"C:\Nano\PhysicalSystem\Nano_phy_vhd.vhd" -ComputerName

"Nano-server1" –Compute -Storage -Cluster -OEMDrivers -DriversPath

“C:\Nano\Drivers"

In the example above, C:\Nano\Drivers is the path for QLogic drivers.

This command takes about 10-15 minutes to create a VHD file. A sample output for this command is displayed below:

Windows(R) Image to Virtual Hard Disk Converter for Windows(R) 10

Version 10.0.14300.1000.amd64fre.rs1_release_svc.160324-1723

INFO : Looking for the requested Windows image in the WIM file

INFO : Image 1 selected (ServerDatacenterNano)...

INFO : Creating sparse disk...

INFO : Mounting VHD...

INFO : Initializing disk...

INFO : Creating single partition...

INFO : Formatting windows volume...

INFO : Windows path (I:) has been assigned.

INFO : System volume location: I:

INFO : Applying image to VHD. This could take a while...

110 BC0154501-00 F

8–Windows Server 2016

Nano Server

INFO : Image was applied successfully.

INFO : Making image bootable...

INFO : Fixing the Device ID in the BCD store on VHD...

INFO : Drive is bootable. Cleaning up...

INFO : Dismounting VHD...

INFO : Closing Windows image...

INFO : Done.

Done. The log is at:

C:\Users\ADMINI~1\AppData\Local\Temp\2\NanoServerImageGenerator.log

Nano Server VHD.

Complete the following procedure to copy the VHD to the physical server and configure it to boot from the new VHD: i.

Go to Computer Management > Storage > Disk Management.

Right-click on Disk Management and select Attach VHD.

Provide the VHD file path.

Click OK.

Run bcdboot d:\windows.

NOTE

In this example, the VHD is attached under D:\.

Right-click on Disk Management and select Detach VHD.

10. Reboot the physical server into the Nano Server VHD.

11.

12. Obtain the IP address of the Nano Server computer.

13. Use Windows PowerShell remoting (or other remote management) tool to connect and remotely manage the server.

111 BC0154501-00 F

8–Windows Server 2016

Nano Server

Deploying Nano Server in a Virtual Machine

Use the following procedure to create a Nano Server virtual hard drive (VHD) that will run in a virtual machine:

Download the Windows Server 2016 OS image.

Go to the NanoServer folder from the downloaded file from Step 1.

Copy the following files from the NanoServer folder to a folder on your hard drive:

 NanoServerImageGenerator.psm1

 Convert-WindowsImage.ps1

Start Windows PowerShell as an administrator.

Change directory to the folder where you pasted the files from Step 3.

Import the NanoServerImageGenerator script by entering the following command:

Import-Module .\NanoServerImageGenerator.psm1 -Verbose

Enter the following power shell command to create a VHD that sets a computer name and includes the Hyper-V guest drivers:

NOTE

This command will prompt you for an administrator password for the new VHD.

New-NanoServerImage –DeploymentType Guest –Edition

<Standard/Datacenter> -MediaPath <path to root of media> -BasePath

.\Base -TargetPath .\NanoServerPhysical\NanoServer.vhd -ComputerName

<computer name> –GuestDrivers

For Example:

New-NanoServerImage –DeploymentType Guest –Edition Datacenter

-MediaPath C:\tmp\TP4_iso\Bld_10586_iso

-BasePath .\Base -TargetPath .\Nano1\VM_NanoServer.vhd -ComputerName

Nano-VM1 –GuestDrivers

This command takes about 10-15 minutes to create a VHD file. A sample output for this command is displayed below:

PS C:\Nano> New-NanoServerImage –DeploymentType Guest –Edition

Datacenter -MediaPath

C:\tmp\TP4_iso\Bld_10586_iso -BasePath .\Base -TargetPath

.\Nano1\VM_NanoServer.vhd -ComputerName Nano-VM1 –GuestDrivers cmdlet New-NanoServerImage at command pipeline position 1

112 BC0154501-00 F

8–Windows Server 2016

Nano Server

Supply values for the following parameters:

Windows(R) Image to Virtual Hard Disk Converter for Windows(R) 10

Version 10.0.14300. 1000.amd64fre.rs1_release_svc.160324-1723

INFO : Looking for the requested Windows image in the WIM file

INFO : Image 1 selected (ServerTuva)...

INFO : Creating sparse disk...

INFO : Attaching VHD...

INFO : Initializing disk...

INFO : Creating single partition...

INFO : Formatting windows volume...

INFO : Windows path (G:) has been assigned.

INFO : System volume location: G:

INFO : Applying image to VHD. This could take a while...

INFO : Image was applied successfully.

INFO : Making image bootable...

INFO : Fixing the Device ID in the BCD store on VHD...

INFO : Drive is bootable. Cleaning up...

INFO : Closing VHD...

INFO : Deleting pre-existing VHD : Base.vhd...

INFO : Closing Windows image...

INFO : Done.

Done. The log is at:

C:\Users\ADMINI~1\AppData\Local\Temp\2\NanoServerImageGenerator.log

Create a new virtual machine in Hyper-V Manager, and use the VHD created in Step 7.

Boot the virtual machine.

10. Connect to the virtual machine in Hyper-V Manager.

11.

12. Obtain the IP address of the Nano Server computer.

13. Use Windows PowerShell remoting (or other remote management) tool to connect and remotely manage the server.

113 BC0154501-00 F

8–Windows Server 2016

Nano Server

Managing Nano Server Remotely

There are a variety of options for managing Nano Server remotely, including

Windows PowerShell, Windows Management Instrumentation (WMI), Windows

Remote Management, and Emergency Management Services (EMS). This section will discuss how to access Nano Server using Windows PowerShell remoting.

Managing Nano Server with Windows PowerShell Remoting

Use the following procedure to manage Nano Server with Windows PowerShell remoting:

Add the IP address of the Nano Server to your management computer’s list of trusted hosts.

NOTE

Use the recovery console to find the server IP address.

Add the account you are using to the Nano Server’s administrators.

Enable CredSSP if applicable. (optional)

Adding the Nano Server to List of Trusted Hosts

Enter the following command at an elevated Windows PowerShell prompt to add the Nano Server to the list of trusted hosts:

Set-Item WSMan:\localhost\Client\TrustedHosts "<IP

address of Nano Server>"

The following are examples:

 Example 1:

Set-Item WSMan:\localhost\Client\TrustedHosts "172.28.41.152"

 Example 2:

Set-Item WSMan:\localhost\Client\TrustedHosts "*"

NOTE

This command sets all host server as a trusted host.

114 BC0154501-00 F

8–Windows Server 2016

Nano Server

Starting the Remote Windows PowerShell Session

Enter the following commands at an elevated local Windows PowerShell session to start the remote Windows PowerShell session:

$ip = “<IP address of Nano Server>”

$user = “$ip\Administrator”

Enter-PSSession -ComputerName $ip -Credential $user

You can now run Windows PowerShell commands on the Nano Server as normal.

However, not all Windows PowerShell commands are available in this release of

Nano Server. To see which are available, run Get-Command -CommandType

Cmdlet. To stop the remote session with the command Exit-PSSession.

For more details on Nano Server refer to: https://technet.microsoft.com/en-us/library/mt126167.aspx

Managing QLogic Adapters on Windows Nano Server

To manage QLogic adapters in a Nano Server environments, refer to the Windows

QCC GUI / Windows QCS CLI Management Tools and associated documentation on driverdownloads.qlogic.com

RoCE Configuration

In this example, the Nano Server is managed through PowerShell Remoting feature.

Connect to the Nano Server through PowerShell Remoting from another machine as shown below.

PS C:\Windows\system32> $1p=”172.28.41.152”

PS C:\Windows\system32> $user=”172.28.41.152\Administrator”

PS C:\Windows\system32> Enter-PSSession -ComputerName $ip

-Credential $user

NOTE

The IP of the Nano Server is 172.28.41.152 in this example.

Administrator is the User Name.

115 BC0154501-00 F

8–Windows Server 2016

Nano Server

You will get the following if the Nano Server connects successfully:

[172.28.41.152]: PS C:\Users\Administrator\Documents>

Enter the following PowerShell command to check if the drivers are installed and the link is up:

[172.28.41.152]: PS C:\Users\Administrator\Documents>

Get-NetAdapter

Enter the following PowerShell command to check if RDMA is enabled on the adapter:

[172.28.41.152]: PS C:\Users\Administrator\Documents>

Get-NetAdapterRdma

Enter the following PowerShell command to assign IP address and VLAN ID to all the interfaces of adapter:

[172.28.41.152]: PS C:\> Set-NetAdapterAdvancedProperty

-InterfaceAlias “slot 1 port 1” -RegistryKeyword vlanid

-RegistryValue 5

[172.28.41.152]: PS C:\> netsh interface ip set address name=”SLOT 1 Port 1” static 192.168.10.10 255.255.255.0

Enter the following PowerShell command to create SMBShare on the Nano

Server:

[172.28.41.152]: PS C:\Users\Administrator\Documents>

New-Item -Path c:\ -Type Directory -Name smbshare -Verbose

116 BC0154501-00 F

8–Windows Server 2016

Nano Server

[172.28.41.152]: PS C:\> New-SMBShare -Name “smbshare” -Path c:\smbshare -FullAccess Everyone

Enter the following PowerShell command to map the SMBShare as Network

Drive in Client Machine:

NOTE

The IP address of an interface on the Nano Server is 192.168.10.10.

PS C:\Windows\system32> net use z: \\192.168.10.10\smbshare

Enter the following PowerShell command to perform Read/Write on

SMBShare, and check RDMA statistics on Nano Sever:

[172.28.41.152]: PS C:\>

(Get-NetAdapterStatistics).RdmaStatistics

117 BC0154501-00 F

Troubleshooting

This chapter provides information about the following topics:



Troubleshooting Checklist



Verifying that Current Drivers are Loaded



Testing Network Connectivity



Microsoft Virtualization with Hyper-V



Linux



Miscellaneous

118 BC0154501-00 F

9–Troubleshooting

Troubleshooting Checklist

CAUTION

Before you open the server cabinet to add or remove the adapter, review

“Safety Precautions” on page 8

The following checklist provides recommended actions to resolve problems that may arise while installing the 45000 Series Adapter or running it in your system.

 Inspect all cables and connections. Verify that the cable connections at the network adapter and the switch are attached properly.

 Verify the adapter installation by reviewing

“Installing the Adapter” on page 9 . Verify that the adapter is properly seated in the slot. Check for

specific hardware problems, such as obvious damage to board components or the PCI edge connector.

 Verify the configuration settings and change them if they are in conflict with another device.

 Verify that your server is using the latest BIOS.

 Try inserting the adapter in another slot. If the new position works, the original slot in your system may be defective.

 Replace the failed adapter with one that is known to work properly. If the second adapter works in the slot where the first one failed, the original adapter is probably defective.

 Install the adapter in another functioning system, and then run the tests again. If the adapter passes the tests in the new system, the original system may be defective.

 Remove all other adapters from the system, and then run the tests again. If the adapter passes the tests, the other adapters may be causing contention.

119 BC0154501-00 F

9–Troubleshooting

Verifying that Current Drivers are Loaded

Windows

See the Device Manager to view vital information about the adapter, link status, and network connectivity.

Linux

To verify that the qed.ko driver is loaded properly, run:

# lsmod | grep -i <module name>

If the driver is loaded, the output of this command shows the size of the driver in bytes. The following example shows the drivers loaded for the qed module:

# lsmod | grep -i qed qed 199238 1 qede 1417947 0

If you reboot after loading a new driver, you can use the following command to verify that the currently loaded driver is the correct version: modinfo qede

Or, you can use the following command:

[root@test1]# ethtool -i eth2 driver: qede version: 8.4.7.0

firmware-version: mfw 8.4.7.0 storm 8.4.7.0

bus-info: 0000:04:00.2

If you loaded a new driver, but have not yet booted, the modinfo command will not show the updated driver information. Instead, type the following dmesg command to view the logs. In this example, the last entry identifies the driver that will be active upon reboot.

# dmesg | grep -i "QLogic" | grep -i "qede"

[ 10.097526] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x

[ 23.093526] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x

[ 34.975396] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x

[ 34.975896] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x

[ 3334.975896] QLogic FastLinQ 4xxxx Ethernet Driver qede x.x.x.x

120 BC0154501-00 F

9–Troubleshooting

Testing Network Connectivity

NOTE

When using forced link speeds, verify that both the adapter and the switch are forced to the same speed.

Testing Network Connectivity for Windows

Network connectivity can be tested using the ping command to determine if the network connection is working.

Click Start, and then click Run.

Type cmd in the Open box, and then click OK.

Type ipconfig /all to view the network connection to be tested.

Type ping <ip_address>, and then press ENTER.

The ping statistics that are displayed indicate whether the network connection is working or not.

ethtool <pf> can also be used to determine the link status.

Testing Network Connectivity for Linux

To verify that the Ethernet interface is up and running, run ifconfig to check the status of the Ethernet interface. Use netstat -i to check the statistics on the

Ethernet interface.

Ping an IP host on the network to verify that the connection has been established.From the command line, type ping <ip_address>, and then press

ENTER.

The ping statistics that are displayed indicate whether or not the network connection is working.

Microsoft Virtualization with Hyper-V

Microsoft Virtualization is a hypervisor virtualization system for Windows Server

2012 R2. For more information on Hyper-V, see https://technet.microsoft.com/en-us/library/Dn282278.aspx

121 BC0154501-00 F

9–Troubleshooting

Linux

Problem: 45000 Series devices with SFP+ Flow Control default to Off rather than

Rx/Tx Enable.

Solution: The Flow Control default setting for revision 1.6.x and newer has been changed to Rx Off and Tx Off because SFP+ devices do not support auto-negotiation for Flow Control.

Problem: Errors appear when compiling driver source code.

Solution: Some installations of Linux distributions do not install the development tools by default. Ensure the development tools for the Linux distribution you are using are installed before compiling driver source code.

Miscellaneous

Problem: The 45000 Series Adapter has shut down, and an error message appears indicating that the fan on the adapter has failed.

Solution: The 45000 Series Adapter shut down to prevent permanent damage.

Contact QLogic Technical Support for assistance.

122 BC0154501-00 F

Adapter LEDS

The state of the adapter port link and activity is indicated by the LEDs as

described in Table A-1 .

Port LED

Link LED

Activity LED

Table A-1. Adapter Port Link and Activity LEDs

LED Appearance

Off

Continuously illuminated

Off

Blinking

Network State

No link (cable disconnected)

Link

No port activity

Port activity

123 BC0154501-00 F

Cables and Optical

Modules

Specifications

The 45000 Series Adapters support a variety of cables and optical modules that comply with SFF8024. Specific form factor compliance is as follows:

 SFPs

 SFF8472 (for memory map)

 SFF8419 or SFF8431 (low speed signals and power)

 QSFPs

 SFF8636 (for memory map)

 SFF8679 or SFF8436 (low speed signals and power)

 Optical modules electrical input/output, active copper cables (ACC), and active optical cables (AOC)

 10G—SFF8431 limiting interface

 25G—IEEE802.3by Annex 109B (25GAUI)

 40G—IEEE802.3 Annex 86A (nPPI)

 100G—IEEE802.3 Annex 83E (CAUI4)

124 BC0154501-00 F

B–Cables and Optical Modules

Tested Cables and Optical Modules

QLogic does not guarantee that every cable or optical module that satisfies the compliance requirements will operate with the 45000 Series Adapters. QLogic has

tested the following components ( Table B-1 ) and presents this list for your

convenience. This list is based on cable and optics components that are available at the time of product release, and is subject to change over time as new components enter the market or are discontinued..

25G DAC

Table B-1. Tested Cables and Optical Modules

Speed/

Form Factor

10G DAC

Manufacturer

Cisco

Dell

Amphenol

Dell

Leoni ParaLink

InnoLight

Description/Part Number

10G Twin-axial Cable 3.0M

10G Twin-axial Cable with Cisco SFP

(CN-0J564N-52204-51F-ACU1-A00 3.0M)

NDCCGF-00011.0m

NDCCGF-0005 3.0m

NDAQGF-0001 1m Splitter DAC

NDAQGF-0003 3m Splitter DAC

07R9N9 Rev A00 3m Splitter DAC (100G)

0YFNDD Rev A00 2m Splitter DAC (100G)

026FN3 Rev A00 1m Splitter DAC (100G)

LA0SF064-SD-R

TF-PY020-N00 SFP28 AOC 20m 25G Optical

(active optical cable only)

40G DAC

40G Optical

Tripp Lite

Finisar

QSFP+ N282-03M-BK, passive 3M

FTL410QE2C

125 BC0154501-00 F

B–Cables and Optical Modules

Tested Cables and Optical Modules

Table B-1. Tested Cables and Optical Modules (Continued)

Speed/

Form Factor

100G DAC

Manufacturer Description/Part Number

100G Optical

Molex

LuXshare

Mellanox

Dell

™

Finisar

Avago

(Broadcom

Ltd.)

1002971301 3.0 M 30AWG (prototype)

1002971101 1.0 M 30AWG (prototype)

ICT 100G LP9QF001-SD-R 3.0m_26AWG

LP9QF002-SD-R 1.0m_30AWG

100G DACMCP 1600-C003 Rev A2 3.0m

0P7C7N Rev A00 1.0M (100G)

076V43 Rev A00 1.0M (100G)

FTLC9551REPM

AFBR-89CDDZ

126 BC0154501-00 F

Glossary

adapter

The board that interfaces between the host system and the target devices.

Adapter is synonymous with Host Bus

Adapter, host adapter, and board.

adapter port

A port on the adapter board.

alternative routing-ID interpretation (ARI)

The PCI-SIG ARI provides a mechanism for a single PCIe device to support up to

256 physical functions (PFs). Current OS's allow up to 16 PFs (per device) to be used with ARI enabled in the system BIOS, and up to 8 PFs (per device) to be used when

ARI is disabled.

basic input output system (BIOS)

The program (or utility) that serves as an interface between the hardware and the operating system, and allows booting from the adapter at startup.

data center bridging (DCB)

Provides enhancements to existing 802.1 bridge specifications to satisfy the requirements of protocols and applications in the data center. Because existing high-performance data centers typically comprise multiple application-specific networks that run on different link layer technologies

(Fibre Channel for storage and Ethernet for network management and LAN connectivity), DCB enables 802.1 bridges to be used for the deployment of a converged network where all applications can be run over a single physical infrastructure.

data center bridging exchange (DCBX)

A protocol used by DCB devices to exchange configuration information with directly connected peers. The protocol may also be used for misconfiguration detection and for configuration of the peer.

device

A computer subsystem, such as an adapter card, that mediates data in a computer network. The term device is used interchangeably with target and target device.

driver

The software that interfaces between the file system and a physical data storage device or network media.

127 BC0154501-00 F

User’s Guide—25Gb/40Gb/100Gb Intelligent Ethernet Adapters

QLogic FastLinQ 45000 Series

enhanced transmission selection (ETS)

A standard that specifies the enhancement of transmission selection to support the allocation of bandwidth among traffic classes. When the offered load in a traffic class does not use its allocated bandwidth, enhanced transmission selection allows other traffic classes to use the available bandwidth. The bandwidth- allocation priorities coexist with strict priorities. ETS includes managed objects to support bandwidth allocation. The per traffic class

ETS minimum bandwidth mode pre-empts the per partition minimum TX bandwidth settings. For more information, refer to http://ieee802.org/1/pages/802.1az.html

Ethernet

The most widely used LAN technology that transmits information between computers, typically at speeds of 10 and 100 million bits per second (Mbps).

extensible firmware interface (EFI)

A specification that defines a software interface between an operating system and platform firmware. EFI is a replacement for the older BIOS firmware interface present in all IBM PC-compatible personal computers.

file transfer protocol (FTP)

A standard network protocol used to transfer files from one host to another host over a TCP-based network, such as the

Internet. FTP is required for out-of-band firmware uploads that will complete faster than in-band firmware uploads.

human interface infrastructure (HII)

A specification (part of UEFI 2.1) for managing user input, localized strings, fonts, and forms, that allows OEMs to develop graphical interfaces for preboot configuration.

Institute of Electrical and Electronics

Engineers (IEEE)

An international nonprofit organization for the advancement of technology related to electricity.

maximum transmission unit (MTU)

Refers to the size (in bytes) of the largest packet (IP datagram) that a specific layer of a communications protocol can transfer.

message signaled interrupts (MSI)

Two PCI-defined extensions to support message signaled interrupts (MSI), in PCI

2.2 and later and PCI Express. MSIs are an alternative way of generating an interrupt through special messages that allow the emulation of a pin assertion or deassertion.

network interface controller/card (NIC)

A computer circuit board or card that is installed in a computer so that it can be connected to a network.

NIC partitioning (NPAR)

On QLogic Converged Network Adapters and Intelligent Ethernet adapters, refers to the partitioning of a single NIC port into up to four individual physical functions or partitions, each with a user-configurable bandwidth and personality (interface type).

Personalities can include NIC (with or without RDMA), FCoE, and iSCSI.

PCI Express (PCIe)

A third-generation I/O standard that allows enhanced Ethernet network performance beyond that of the older peripheral component interconnect (PCI) and PCI extended

(PCI-X) desktop and server slots.

128 BC0154501-00 F

User’s Guide—25Gb/40Gb/100Gb Intelligent Ethernet Adapters

QLogic FastLinQ 45000 Series

quality of service (QoS)

Refers to the methods used to prevent bottlenecks and ensure business continuity when transmitting data over virtual ports by setting priorities and allocating bandwidth.

RDMA over Converged Ethernet (RoCE)

A network protocol that allows remote direct memory access (RDMA) over a converged or a non-converged Ethernet network. RoCE is a link layer protocol that allows communication between any two hosts in the same Ethernet broadcast domain.

reduced instruction set computer (RISC)

A computer microprocessor that performs fewer types of computer instructions, thereby operating at higher speeds.

remote direct memory access (RDMA)

The ability for one node to write directly to the memory of another (with address and size semantics) over a network. This capability is an important feature of VI networks.

serializer/deserializer (SerDes)

A pair of functional blocks commonly used in high-speed communications to compensate for limited input/output. These blocks convert data between serial data and parallel interfaces in each direction.

unified extensible firmware interface (UEFI)

A specification detailing an interface that helps hand off control of the system for the preboot environment (that is, after the system is powered on, but before the operating system starts) to an operating system, such as Windows or Linux. UEFI provides a clean interface between operating systems and platform firmware at boot time, and supports an architecture-independent mechanism for initializing add-in cards.

virtual interface (VI)

An initiative for remote direct memory access across Fibre Channel and other communication protocols. Used in clustering and messaging.

129 BC0154501-00 F

Index

Symbols

.vib file

A

adapter enable RoCE

installation

management 4

5 ,

properties (Windows)

address, MAC

address, PCI

architecture

auto_fw_reset parameter

B

bandwidth allocation

38 ,

bracket

C

cable specifications

Coalesce Rx Microseconds 19

supported 125

CEE 44 ,

checklist preinstallation

troubleshooting

119

chip type

class map

, 45

Coalesce Tx Microseconds 19

130

converged enhanced Ethernet, See CEE.

D

DCB map

DCBX

device

name

Device Manage

disable_tpa parameter

driver

installation (VMware) 24

installation (Windows)

Linux files

messages (Linux)

parameter (VMware) 25

parameters (Linux)

removal (VMware) 28

source code errors

enable_vxlan_offld parameter 27

statistics (Linux) 20

upgrade (VMware) 25

verification 120

driver installation

kmod RPM 17

Linux without RoCE 13

RoCE, inbox OFED 18

source RPM 16

TAR file

BC0154501-00 D

User’s Guide—25Gb/40Gb/100Gb Intelligent Ethernet Adapters

QLogic FastLinQ 45000 Series driver removal

Linux, non-RoCE

Linux, RoCE

Windows

E

EFI 34

enhanced transmission selection, See ETS.

ETS

Extensible Firmware Interface, See EFI.

F

fan failure

human interface infrastructure, See HII.

filtering, VxLAN 27

firmware

properties 34

reset

firmware upgrade

Linux 30

Windows

flow control

, 27

G

GID

61 global ID, See GID.

gro_disable parameter 19

H

hardware 7

HII

31 ,

hw_vlan parameter 26 ,

Hyper-V

QoS policy 45 queuing class 45

I

inbox OFED

information level 27

int_mode parameter 19

link speed

19 ,

status 34

Linux operating system

log level

LRO

M

MAC address

maintenance mode 25

management applications

, 5

map

DCB 46

network QoS class

QoS class

max_vfs parameter

maximum transmission unit, See MTU.

memory

messages 20

MSI 19

131 BC0154501-00 D

User’s Guide—25Gb/40Gb/100Gb Intelligent Ethernet Adapters

QLogic FastLinQ 45000 Series

MSI-X 28

MTU size 19

27 ,

multi_rx_filters parameter 26

priority

PFC

RoCE

priority-based flow control, See PFC.

N

network

connectivity 121

installations

Nexus 6000 Ethernet switch

no_tracing parameter

notice level 27

NPAR 38

num_queues parameter 26 ,

O

OFED 12 ,

inbox, RHEL

inbox, SLES

limitations 43

open fabric enterprise distribution, See OFED.

optical module specifications

physical function, See PF.

supported 125

P

packet, jumbo

parameters, Linux 19

Performance Monitor 51

performance, RoCE 53

PFC

physical characteristics

policy map 44 ,

power management 22

PowerShell, Windows

preinstallation checklist

Q

QCC GUI

QCC vCenter Plug-In

QConvergeConsole GUI, See QCC GUI.

QCS CLI 4

qed driver 12 ,

qede driver

12 ,

qedr driver

QLogic Control Suite, See QCS CLI.

QoS 44

QSFPs 124

quality of service, See QoS.

queues, Tx/Rx 26 ,

R

RDMA

RoCE

configuration for RHEL 54

configuration for SLES 55

driver

limitations

Linux driver removal

Rx queue

S

safety 8

scripts, firmware upgrade

server message block, See SMB.

SFP+ Flow Control

132 BC0154501-00 D

User’s Guide—25Gb/40Gb/100Gb Intelligent Ethernet Adapters

QLogic FastLinQ 45000 Series

SFPs