Broadcom Emulex Drivers for Linux for LightPulse Adapters User Guide

Broadcom Emulex Drivers for Linux for LightPulse Adapters User Guide

Emulex® Drivers for Linux for LightPulse® Adapters User Guide

Version 11.2

December 30, 2016 pub-005778

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Corporate Headquarters San Jose, CA Website www.broadcom.com

Broadcom, the pulse logo, Connecting everything, Avago Technologies, Avago, the A logo, Emulex, LightPulse, OneCommand, OneConnect, and ExpressLane are among the trademarks of Broadcom in the United States, certain other countries and/or the EU.

Copyright © 2003–2017 Broadcom. All Rights Reserved.

The term "Broadcom" refers to Broadcom Limited and/or its subsidiaries. For more information, please visit www.broadcom.com

.

Broadcom reserves the right to make changes without further notice to any products or data herein to improve reliability, function, or design.

Information furnished by Broadcom is believed to be accurate and reliable. However, Broadcom does not assume any liability arising out of the application or use of this information, nor the application or use of any product or circuit described herein, neither does it convey any license under its patent rights nor the rights of others.

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table of Contents

Table of Contents

Chapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Chapter 2: Installing and Uninstalling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1 General Installation Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Binary RPM FC and FCoE Driver Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 Ethernet Driver Kit for LPe16202/OCe15100 Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4 Booting from a Nonzero LUN Attached to an Emulex FC or FCoE Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.5 Emulex OneCommand Manager Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Chapter 3: Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1 FC and FCoE Driver Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.2 VPort Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.3 Ethernet Driver Configuration for LPe16202/OCe15100 Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.4 FC and FCoE Driver Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.5 Network Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Chapter 4: Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.1 Hardware Situations and Resolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.2 FC and FCoE Driver Situations and their Resolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.3 Ethernet Driver Situations and their Resolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.4 Log Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Appendix A: Ethtool -S Option Statistics for LPe16202/OCe15100 Adapters in NIC Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

Appendix B: lpfc Driver BlockGuard Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

B.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

B.2 Enabling BlockGuard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 B.3 SCSI Command Support – SCSI Operation Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

B.4 lpfc Driver Module Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Broadcom - 3 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Chapter 1: Introduction

1.1

1.2

Chapter 1: Introduction Overview

Overview

This guide provides installation, updating, uninstalling, configuring, and troubleshooting procedures for several types of Emulex®-supported drivers for Linux.

This guide is applicable to several versions of drivers, operating systems, firmware, and adapters.

 For supported firmware versions and their latest release, refer to the Documents and Downloads area of http://www.broadcom.com

for the specific adapter. This product supports the following Emulex LightPulse® host bus adapters (HBAs) and converged fabric adapters (CFAs):     LPe12000-series adapters LPe16000-series adapters, including LPe16202/OCe15100 adapters LPe31000-series adapters LPe32000-series adapters For a list of Emulex adapters that are compatible with the stand-alone driver kits, refer to the Documents and Downloads area of http://www.broadcom.com

for the specific driver.

Abbreviations

AIC ANSI API ASIC BIOS CLI CPU CQ CQE DCBX DHCHAP DIF DIMM DMA ELS EQ FC FCF FCFI FCoE adaptive interrupt coalescing American National Standards Institute application programming interface application-specific integrated circuit basic input/output system command line interface central processing unit completion queue completion queue entry Data Center Bridging Capabilities Exchange Diffie-Hellman Challenge Handshake Authentication Protocol Data Integrity Field dual in-line memory module direct memory access Extended Link Service event queue Fibre Channel Fibre Channel over Ethernet Forwarder Fibre Channel Forwarder Indicator Fibre Channel over Ethernet Broadcom - 4 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 PF POST QoS RHEL RPI RPM RQ RSCN RSS Rx SAN I/O IOCB IOV IP IRQ iSCSI KVM LAN FCP FSB FW Gb/s GnuPG GRO GUI HBA LBA LLC LOM LRO LUN MAC Mb/s MSI MSI-X MTU NAA NIC NPIV NUMA PCI PCIe Fibre Channel Protocol front side bus firmware gigabits per second GNU Privacy Guard Generic Receive Offload Graphical User Interface host bus adapter input/output input/output control block I/O virtualization Internet Protocol interrupt request internet Small Computer System Interface kernel-based virtual machine local area network logical block address logical link control LAN on motherboard large receive offload logical unit number Media Access Control megabits per second message signaled interrupts message signaled interrupts – extended maximum transmission unit Network Address Authority network interface card (or controller) N_Port ID Virtualization non-uniform memory access Peripheral Component Interconnect Peripheral Component Interconnect Express physical function power-on self-test quality of service Red Hat Enterprise Linux remote port indicator resource package manager receive queue registered state change notification receive-side scaling receive storage area network Broadcom - 5 Chapter 1: Introduction Abbreviations

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 TSO Tx UDP UEFI VEB VEPA VF VGT SCSI SGE SLES SLI SNMP SR-IOV tar TCP VPort VLAN VM VPD WWPN XRI Small Computer System Interface Oracle (formerly Sun) grid engine SUSE Linux Enterprise Server Service Level Interface Simple Network Management Protocol single-root I/O virtualization tape archive Transmission Control Protocol TCP Segmentation Offload transmit User Datagram Protocol Unified Extensible Firmware Interface virtual Ethernet bridging virtual Ethernet port aggregator virtual function virtual guest tagging virtual port virtual local area network virtual machine vital product data World Wide Port Name extensible resource indicator Chapter 1: Introduction Abbreviations Broadcom - 6 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 2: Installing and Uninstalling General Installation Requirements

Chapter 2: Installing and Uninstalling

Emulex releases Linux binary RPMs that are digitally signed using the GNU Privacy Guard (GnuPG) standard. This will allow certification of the contents of the RPMs and verification that the contents have not been modified since they were created by Emulex. The RPMs have been digitally signed by Emulex with a GnuPG private key that is only held by Emulex. Instructions for creating the GnuPG public key file are located at: http://www.broadcom.com/docs/elx-rpm-public-key

2.1

General Installation Requirements

NOTE

You must install the driver before updating the firmware.

Prior to driver installation, follow these general requirements:   Install a supported Emulex adapter in the system. Refer to the adapter's installation guide for specific hardware installation instructions. Use a supported operating system. Refer to the Documents and Downloads area of http://www.broadcom.com

for supported operating systems.

2.2

2.2.1

Binary RPM FC and FCoE Driver Kit

The binary RPM FC and FCoE driver kit contains the following:  A zipped tar file that includes the driver binary RPMs for a specific driver version and Linux distribution  

NOTE

Use only officially released Linux distribution kernels. The binary RPM packages support only officially released Linux distribution kernels and do not support pre-release distribution kernels.

An installation script, elx_lpfc_install.sh

, that installs by default the FC and FCoE driver binary RPM that corresponds to the target system's architecture and kernel memory variant A README file that provides a description of the kit structure, its contents, and distribution support scope

Installing the Binary RPM FC and FCoE Driver Kit

NOTE

You must uninstall any FC driver kits that are not part of this distribution. For example, you must uninstall any previous FC and FCoE driver kits before installing this driver kit. This installation fails if a previous version of the FC and FCoE driver kit is detected. For more

information, see Section 2.2.2, Uninstalling the Binary RPM FC and FCoE Driver Kit

.

To install the binary RPM FC and FCoE driver: 1.

2.

3.

Download the appropriate driver kit from the Documents and Downloads area of http://www.broadcom.com

.

Log in as

root

to a terminal, and unpack the tarball: tar xzf elx-lpfc-dd-<

Linux distribution version

>-<

driver version

>.tar.gz

Change to the directory into which the tarball was extracted: cd elx-lpfc-dd-<

Linux distribution version

>-<

driver version

>/ Broadcom - 7 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

2.2.2

Chapter 2: Installing and Uninstalling Ethernet Driver Kit for LPe16202/OCe15100 Adapters 4.

5.

Run the elx_lpfc_install.sh

script without options to install the driver kit: ./elx_lpfc_install.sh

After the elx_lpfc_install.sh

script has finished running successfully, the Emulex FC and FCoE driver is loaded, and devices that are connected to the system are accessible. Reboot the system to enable the newly added driver options in the ramdisk. You can also reboot the system later.

Uninstalling the Binary RPM FC and FCoE Driver Kit

NOTE

You must run the uninstall script that shipped with the version of the driver kit you want to remove.

To uninstall the binary RPM FC and FCoE driver: 1.

2.

3.

Log in as

root

.

If possible, exit all applications that use FC-attached drives, then unmount the drives. If you cannot exit all applications that use FC-attached drives, the uninstall script works properly, but you must reboot after the uninstall is complete.

Run the elx_lpfc_install.sh

script with the --uninstall option: ./elx_lpfc_install.sh --uninstall

2.3

2.3.1

Ethernet Driver Kit for LPe16202/OCe15100 Adapters

The Ethernet driver kit includes the driver that supports the NIC protocol. The Ethernet driver kit contains the following:  A zipped tar file that includes the driver binary RPMs for a specific driver version, and for all of the supported Linux distribution kernels  

NOTE

Use only officially released Linux distribution kernels. The binary RPM packages support only officially released Linux distribution kernels, and do not support pre-release distribution kernels.

An installation script, lpnic_install.sh

, which installs (by default) the Ethernet driver binary RPM that corresponds to the target system's architecture and kernel memory variant A README file that provides a description of the kit structure, its contents, and distribution support scope

Installing the Ethernet Driver Kit

NOTE

Remove any previously installed Ethernet driver kits (that is, those that were not part of a distribution’s kernel), before proceeding. For more information on uninstalling the driver, see

Section 2.3.2, Uninstalling the Ethernet Driver Kit .

To install the Ethernet driver: 1.

2.

3.

Download the appropriate driver kit from the Documents and Downloads area of Log in as

root

to a terminal, and unpack the tarball: tar xzf elx-lpnic-dd-<

driver version

>.tar.gz

Change to the directory that is extracted: cd elx-lpnic-dd-<

driver version

>/ http://www.broadcom.com

.

Broadcom - 8 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

2.3.2

Chapter 2: Installing and Uninstalling Booting from a Nonzero LUN Attached to an Emulex FC or FCoE Adapter 4.

5.

Run the lpnic_install.sh

script without options to install the driver kit: ./lpnic_install.sh

After the lpnic_install.sh

script has finished running successfully, the Emulex Ethernet driver is loaded, and devices that are connected to the system are accessible. Reboot the system to enable the newly added driver options in the ramdisk. You can also reboot the system later.

Uninstalling the Ethernet Driver Kit

NOTE

You must run the uninstall script that shipped with the version of the driver kit you want to remove.

To uninstall the Ethernet driver: 1.

2.

3.

Log in as

root

.

If possible, exit all applications that use Ethernet-attached drives, then unmount the drives. If you cannot exit all applications that use Ethernet-attached drives, the uninstall works properly, but you must reboot after the uninstall is complete.

Run the lpnic_install.sh

script with the --uninstall option: ./lpnic_install.sh --uninstall

2.4

2.5

Booting from a Nonzero LUN Attached to an Emulex FC or FCoE Adapter

To configure SLES 11 SPx to boot from an FC-attached disk device other than /dev/sda , refer to the

Boot for the Fibre Channel Protocol User Guide

or the

Boot for LPe16202/OCe15100 Adapters User Guide

, both of which are available at the Documents and Downloads area of http://www.broadcom.com

.

Emulex OneCommand Manager Application

The Emulex OneCommand® Manager application is a powerful and centralized adapter management suite. It provides discovery, reporting, and management of local and remote adapters from a single console anywhere in the SAN and across platforms. Both a GUI and CLI are provided. For instructions on installing and using the Emulex OneCommand Manager application, refer to the

OneCommand Manager Application for LightPulse Adapters User Guide

, which is available at the Documents and Downloads area of http://www.broadcom.com

.

Broadcom - 9 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Chapter 3: Configuration

Chapter 3: Configuration FC and FCoE Driver Configuration

3.1

FC and FCoE Driver Configuration

The following section describes how to configure parameters for the FC and FCoE driver.

3.1.1

FC and FCoE Driver Parameters

The FC and FCoE driver parameters determine some aspects of the driver’s behavior. There are two main types, static and dynamic. Changes to the static parameters require a driver reload for the change to take effect. Changes to most dynamic parameters take effect immediately; some do not take effect until there is a link-down/link-up sequence.

For more information on driver parameters, see

Section 3.1.1.1, Static FC and FCoE Driver Parameters , and

Section 3.1.1.2, Dynamic FC and FCoE Driver Parameters

.

3.1.1.1

Static FC and FCoE Driver Parameters

Changes to static parameters require a driver reload for the change to take effect. The following table lists the static FC and FCoE driver parameters.

Table 1 Static FC and FCoE Driver Parameters

lpfc_ack0

Parameter

lpfc_discovery_threads lpfc_enable_bg lpfc_enable_da_id lpfc_enable_hba_heartbeat

Description

sysfs

Visible

When enabled, ACK0 is used for Class 2. The enabled value is 1.

The disabled value is 0 (default).

Yes Specifies the maximum number of ELS commands that can be outstanding for a discovery.

No

NOTE

The lpfc_discovery_threads parameter defaults to a value of 64 for private loop topologies regardless of the configured value. If multiple ports are configured on the host, the value of 64 is used only for those ports that are connected in a private loop topology. The configured value is used for all other ports.

The minimum value is 1. The maximum value is 64. The default value is 32.

Enables the BlockGuard (T10-DIF) feature.

The minimum value is 0 (default). The maximum value is 1.

When enabled, the FC and FCoE driver issues a DA_ID CT command to the fabric when VPorts log out of the fabric. The enabled value is 1. The disabled value is 0 (default).

Yes No When enabled, the heartbeat logic in the FC and FCoE driver is able to detect whether the adapter is functional. If the heartbeat logic detects the adapter is not functional, the driver will shut down the adapter. Yes The disabled value is 0 (default). The enabled value is 1.

Applicable Adapters

SLI-3 only All All All All

Activation

Driver reload Driver reload Driver reload Driver reload Driver reload Broadcom - 10 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration

Table 1 Static FC and FCoE Driver Parameters (Continued) Parameter

lpfc_enable_hba_reset lpfc_enable_npiv lpfc_enable_SmartSAN lpfc_EnableXLane lpfc_fcp_class lpfc_fcp_io_channel lpfc_fdmi_on lpfc_hba_queue_depth lpfc_lun_queue_depth lpfc_max_luns

Description

sysfs

Visible

When enabled, the FC and FCoE driver can pass resets to the adapter. This action is typically used for debugging purposes.

The enabled value is 1 (default). The disabled value is 0.

When enabled, the FC and FCoE driver can use NPIV to create VPorts (if supported by the fabric).

The enabled value is 1 (default). The disabled value is 0.

Yes Yes All All Sets up FDMI support for SmartSAN. The disabled value is 0 (default). The enabled value is 1.

Yes All

Applicable Adapters

For Emulex LPe16000-series, LPe31000-series, LPe32000-series, and LPe16202/OCe15100 adapters, specifies if the ExpressLane™ feature is enabled or disabled.

The minimum value is 0 (default). The maximum value is 1.

Yes SLI-4 only Specifies either FC Class 2 or 3 for FCP data transmission. For Class 2, the value is 2. For Class 3, the value is 3 (default).

Yes For Emulex LPe16000-series, LPe31000-series, LPe32000-series, and LPe16202/OCe15100 adapters, defines the number of I/O channels supported by the driver. For more information, see

Section 3.4, FC and FCoE Driver Performance Tuning

.

Yes The default value is 4 I/O channels.

The minimum value is 1 I/O channel. The maximum value is 16 I/O channels.

For Emulex LPe12000-series adapters, this parameter is not applicable and has no effect.

All SLI-4 only Specifies if FDMI support is enabled or disabled.

The enabled value is 1 (default). The disabled value is 0.

Yes All Limits the number of outstanding commands per HBA. The minimum value is 32. The maximum value is 8192 (also default).

Yes All All Specifies the default maximum number of commands sent to a single logical unit (disk drive).

Yes The minimum value is 1. The maximum value is 512. The default value is 30.

Specifies the highest available LUN ID that is valid, per target. For example, a value of 19 means that LUN IDs from 0 to 19 are valid for the target. The SCSI layer scans each target until it reaches this specified LUN ID.

Yes The minimum value is 0. The maximum value is 65535. The default value is 255.

All

Activation

Driver reload Driver reload Driver reload Driver reload Driver reload Driver reload Driver reload Driver reload Broadcom - 11 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration

Table 1 Static FC and FCoE Driver Parameters (Continued) Parameter

lpfc_max_scsicmpl_time lpfc_restrict_login lpfc_scan_down lpfc_sg_seg_cnt lpfc_sli_mode lpfc_tgt_queue_depth lpfc_use_msi

Description

sysfs

Visible

Uses command completion time to control queue depth. The units are in milliseconds.

Yes The minimum value is 0 (default). The maximum value is 6000.

When enabled, restricts VPorts login to remote initiators.

The enabled value is 1 (default). The disabled value is 0.

No When enabled, selects the SCSI ID.

scan down

method (scanning the AL_PA from high to low) to assign a Yes The enabled value is 1 (default). The disabled value is 0.

Controls the scatter/gather maximum segment count passed to the FC and FCoE driver. This variable is applicable per SCSI command.

For Emulex LPe12000-series adapters, the minimum value is 64 (default), and the maximum value is 4096.

For all other adapters, the minimum value is 64 (default), and the maximum value is 510.

Yes For Emulex LPe12000-series adapters, this parameter allows you to force the SLI mode requested by the adapter driver. The possible values are:   0 = Auto-select (default) 2 = SLI-2  3 = SLI-3 No

NOTE

For all other adapters, this parameter is not applicable and has no effect.

Limits the number of outstanding commands per target port. The minimum value is 10. The maximum value is 65535 (also default).

Yes When enabled, determines whether the driver uses MSI or MSI-X.

 0 = MSI disabled; INTx mode is used (default for the FC RHEL 5.x driver).

  1 = MSI; allows a maximum of 32 interrupts.

2 = MSI-X; allows a maximum of 2048 interrupts (default for FC RHEL 6.x/SLES 11 SPx drivers).

Yes

NOTE

The default is 2. This value reverts to 1 if the system does not support MSI-X. This value reverts to 0 if the system does not support MSI.

All All All All All

Applicable Adapters

FC adapters only SLI-3 only

Activation

Driver reload Driver reload Driver reload Driver reload Driver reload Driver reload Driver reload Broadcom - 12 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration

3.1.1.2

Dynamic FC and FCoE Driver Parameters

Changes to the dynamic parameters take effect immediately. All lpfc dynamic parameters are read/write using sysfs. The following table lists the dynamic FC and FCoE driver parameters.

Table 2 Dynamic FC and FCoE Driver Parameters Parameter

lpfc_cr_count lpfc_cr_delay lpfc_devloss_tmo lpfc_fcp_imax lpfc_fcp_io_sched lpfc_first_burst_size

Description

sysfs

Visible

For Emulex LPe12000-series adapters, this parameter determines the value for I/O coalescing for lpfc_cr_count outstanding commands. The minimum value is 1 (default). The maximum value is 255.

Yes

NOTE

For all other adapters, this parameter is not applicable and has no effect.

For Emulex LPe12000-series adapters, this parameter determines the value for I/O coalescing for lpfc_cr_delay (milliseconds) outstanding commands.

The minimum value is 0 (default). The maximum value is 63.

Yes

NOTE

For all other adapters, this parameter is not applicable and has no effect.

Specifies the number of seconds to hold an I/O error when a device disappears.

The minimum value is 0. The maximum value is 255. The default value is 30.

For Emulex LPe16000-series, LPe31000-series, LPe32000-series, and LPe16202/OCe15100 adapters, specifies the maximum number of fast-path FCP interrupts per second.

The minimum value is 636. The maximum value is 651042. The default value is 5000. Yes Yes

NOTE

For Emulex LPe12000 adapters, this parameter is not applicable and has no effect.

For Emulex LPe16000-series, LPe31000-series, LPe32000-series, and LPe16202/OCe15100 adapters, determines which algorithm to use when scheduling an FCoE I/O to an I/O channel. For more information, see

Section 3.4, FC and FCoE Driver Performance Tuning

.

The default value is 0, configuration by round-robin scheduling. A value of 1 sets configuration to CPU scheduling.

For Emulex LPe12000-series adapters, this parameter is not applicable and has no effect.

Yes Configures first burst size on targets that support first burst. The minimum value is 0 (default). The maximum value is 65536.

Yes

Applicable Adapters

SLI-3 only SLI-3 only All SLI-4 only SLI-4 only All

Activation

Port reset Port reset Dynamic Dynamic Dynamic Dynamic Broadcom - 13 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration

Table 2 Dynamic FC and FCoE Driver Parameters (Continued) Parameter

lpfc_link_speed lpfc_log_verbose lpfc_throttle_log_cnt lpfc_throttle_log_time lpfc_topology lpfc_use_adisc lpfc_XLanePriority

Description

      Specifies the FC link speed. The possible values are: 0 = Auto-select (default) 2 = 2Gb/s 4 = 4Gb/s 8 = 8Gb/s 16 = 16Gb/s 32 = 32Gb/s

NOTE

For LPe32000-series adapters running 11.2 firmware or later, this parameter is ignored.

NOTE

Setting this option incorrectly can cause the adapter to fail to initialize.

sysfs

Visible

Yes

Applicable Adapters

FC adapters only

NOTE

If you configure the link speed in a BIOS utility, the link speed may be overwritten by the operating system according to its own configuration settings. To avoid this issue, configure the link speed in both the operating system driver and the Boot BIOS or UEFI driver.

Specifies the log verbosity level of the messages posted by the driver. Extra activity logging (bit mask).

The minimum value is 0x0 (default). The maximum value is 0xFFFFFFFF.

Yes Specifies the number of messages logged within throttle_log_time .

The minimum value is 1. The maximum value is 1000. The default value is 10.

All Yes All Specifies the time limit for throttle_log_cnt .

The minimum value is 1. The maximum value is 60. The default value is 1.

Yes Yes    For FC adapters, this parameter sets the link topology. The possible values are:  0x0 = Loop first; if loop fails, then point-to-point (default) 0x2 = Point-to-point only 0x4 = Loop only 0x6 = Point-to-point first; if point-to-point fails, then loop All FC adapters only

NOTE

For the LPe16202/OCe15100 adapter in FCoE mode, this parameter is not applicable and has no effect.

When enabled, an ADISC is sent instead of a PLOGI for device discovery or RSCN. The enabled value is 1.

The default value is 0 (disabled).

Yes For Emulex LPe16000-series, LPe31000-series, LPe32000-series, and LPe16202/OCe15100 adapters, when enabled, this parameter defines the frame priority to be used for the Express Lane feature.

The minimum value is 0 (default). The maximum value is 0x7f.

Yes All

Activation

Port reset Dynamic Dynamic Dynamic Port reset Dynamic Dynamic Broadcom - 14 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration

3.1.2

3.1.2.1

3.1.2.2

3.1.2.3

Configuring FC and FCoE Driver Parameters

   You can configure the FC and FCoE driver parameters using any of the following methods: The modprobe Linux program for temporary configuration The lpfc.conf

file for persistent configuration The sysfs interface (to view and modify parameters after loading the FC and FCoE driver) 

NOTE

Not all parameters visible in the sysfs directory can be modified; some are read-only.

The Emulex OneCommand Manager application (refer to the

OneCommand Manager Application for LightPulse Adapters User Guide

for more information)

NOTE

The FC and FCoE driver parameter changes made using the Emulex OneCommand Manager application remain after the FC and FCoE driver is uninstalled. To return to the default settings, you must reset them using the OneCommand Manager application.

Temporary Configuration with modprobe

When you manually load the FC and FCoE driver as a module using the modprobe command, and you change one or more driver parameter values in the command line, the configuration is temporary. These changes are considered temporary because they are valid only for the current session or until the FC and FCoE driver is unloaded.

The modprobe program uses the lpfc.conf

file, but parameters passed to it using the command line override the parameters in the lpfc.conf

file. Values can be expressed in hexadecimal or decimal notation.

If you want to temporarily set lun_queue_depth to 20 (the default is 30) for all HBAs in your system, load the FC and FCoE driver with the following command: modprobe lpfc lpfc_lun_queue_depth=20

Persistent Configuration with the lpfc.conf file

To make the FC and FCoE driver parameters persist across module loads and reboots, perform these steps: 1.

2.

In the /etc/modprobe.d

the verbose flag: directory, create a file with the driver name options lpfc lpfc_log_verbose=0x3ffff lpfc.conf

.

In /etc/modprobe.d/lpfc.conf

, use the options command to add the appropriate FC and FCoE driver parameters and their desired values. For example, adding the following command to the lpfc.conf

file sets If driver parameters are added to the lpfc.conf

file, the FC and FCoE driver must be reloaded for the parameters to take effect. Also, a new ramdisk image is required if you want the changes to take effect in the next boot. For

information on creating a new ramdisk, See Section 3.1.3, Creating a New Ramdisk Image .

If the same parameter is specified on the modprobe command line and in the lpfc.conf

file, the value specified in the modprobe command line takes precedence.

Configure Parameters with a Read/Write to sysfs

Sysfs is a virtual file system that exposes the structure of the system. It also includes interfaces to driver parameters through which the FC and FCoE driver parameters can be viewed and modified. Because these interfaces are available only after driver load, only dynamic FC and FCoE driver parameters can be changed. However, both static and dynamic FC and FCoE driver parameters can be read through sysfs.

Broadcom - 15 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration

NOTE

  Sysfs changes exist only during driver load and are lost when the FC and FCoE driver is unloaded or the system is rebooted.

Driver parameters that are set through module parameters are global; setting them through sysfs is on a SCSI host (adapter port) basis.

For example: echo 0x7f >> /sys/class/scsi_host/host7/lpfc_log_verbose only affects host 7.

modprobe lpfc lpfc_log_verbose=0x7f applies to all SCSI host (ports) managed by the lpfc driver.

Viewing Parameters with sysfs

The sysfs file system is mounted and available as /sys . You must first identify the scsi_host that represents the adapter for which you want to modify the FC and FCoE driver parameters. All scsi_hosts bound to the FC and FCoE driver can be viewed with the following command: ls -d /sys/bus/pci/drivers/lpfc/*/host* Assuming you are interested in adapter scsi_host 7, you can list the FC and FCoE driver parameters for this particular adapter as: ls -l /sys/class/scsi_host/host7/lpfc* An example output follows: -r--r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_ack0 -r--r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_fcp_class -rw-r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/ lpfc_fdmi_on -r--r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_link_speed -rw-r--r-- 1 root root 4096 Feb 28 15:34 /sys/class/scsi_host/host7/lpfc_log_verbose -r--r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_lun_queue_depth -rw-r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_max_luns -rw-r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_devloss_tmo -rw-r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_scan_down -r--r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_topology -rw-r--r-- 1 root root 4096 Feb 28 17:03 /sys/class/scsi_host/host7/lpfc_use_adisc

Temporary Configuration Parameters with sysfs

In the previous example, notice that the FC and FCoE driver parameters are available as files. Reading a file displays the current value of a driver parameter. If the permissions allow it, you can write a value to the file and it will take effect immediately.

Broadcom - 16 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration

3.1.3

3.1.4

3.1.5

3.1.5.1

Reading the lpfc_log_verbose file may show that its value is 0 : cat /sys/class/scsi_host/host7/lpfc_log_verbose 0 To modify the lpfc_log_verbose value to 0xffffffff: echo 0xffffffff > /sys/class/scsi_host/host7/lpfc_log_verbose Reading the lpfc_log_verbose file now shows a value of 0xffffffff: cat /sys/class/scsi_host/host7/lpfc_log_verbose 0xffffffff

NOTE

Setting the lpfc_log_verbose value to 0xffffffff might degrade system performance.

Creating a New Ramdisk Image

The lpfc-install script creates a ramdisk image containing the FC and FCoE driver for the currently running kernel.

NOTE

You must create a new ramdisk image whenever the lpfc options in /etc/modprobe.conf

are changed and you want the change to take effect on the next reboot.

To create a new initial ramdisk image for inbox FC and FCoE drivers and installed binary RPM FC driver kits, type: dracut -f /boot/initramfs-<

kernel-version

>.img <

kernel-version

>

Dynamically Recognizing LUNs and Targets (Using scan)

The FC and FCoE driver enables you to dynamically recognize LUNs and targets without unloading or reloading the FC/FCoE module and without resetting the adapter.

To rescan an adapter’s targets with sysfs, given the adapter's host number (in this example, 3), type: echo "- - -" > /sys/class/scsi_host/host3/scan To limit the rescan to a particular target, given the adapter's host number (in this example, 3) and the target number (in this example, 2), type: echo "- 2 -" > /sys/class/scsi_host/host3/scan You can also use the lun_scan script in the /usr/sbin/lpfc directory.

Persistent Naming

The generic device manager for the Linux kernel is

udev

, which primarily manages device nodes in the /dev directory.

Using udev to Discover Logical to Physical Mappings for sd Devices

In Linux, the driver for SCSI disk drives is

sd

. A disk device name has an sd prefix. Persistent names for sd devices are provided in the /dev/disk/by-id directory. To find the persistent udev name for the disk, which is currently

sdc

, type: cd /dev/disk/by-id ls -l | grep sdc Broadcom - 17 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration

3.1.5.2

3.1.5.3

The sample output is: lrwxrwxrwx 1 root root 9 2006-08-01 19:08 scsi-32000000c5005d6e6 -> ../../sdc In the previous example, the disk has no partitions. If the disk had two partitions, the output would look like the following: lrwxrwxrwx 1 root root 9 2006-08-01 19:08 scsi-32000000c5005d6e6 -> ../../sdc lrwxrwxrwx 1 root root 10 2006-08-01 19:08 scsi-32000000c5005d6e6-part1 -> ../../sdc1 lrwxrwxrwx 1 root root 10 2006-08-01 19:08 scsi-32000000c5005d6e6-part2 -> ../../sdc2

Configuring the System to Boot Using Persistent Names For SLES 11 SPx and SLES 12 NOTE

SLES 11 SPx and SLES 12 are configured by default with udev to provide persistent names for hard disks, including FC-attached disks.

To use a persistent name for a boot device: 1.

2.

3.

In /boot/grub/menu.lst

, find the kernel line for the default boot. For example: kernel /boot/vmlinuz root=/dev/sda2 vga=0x314 Find the persistent name for the root partition (following root= on the kernel line) by using the instructions in

Section 3.1.5.1, Using udev to Discover Logical to Physical Mappings for sd Devices

.

In the same file, /boot/grub/menu.lst

, replace the text after root= with the partition’s persistent name. For example: kernel /boot/vmlinuz root=/dev/disk/by-id/scsi-32000000c5005d6e6-part2 vga=0x314 4.

Change any mounts listed in /etc/fstab that refer to this root partition by either its /dev/sd name or a file system label to use the persistent name as well.

For RHEL 6.x and RHEL 7.x

To use a persistent name for a boot device: 1.

2.

3.

4.

In /boot/grub/grub.conf

, find the kernel line for the default boot. For example: kernel /boot/vmlinuz -<

kernel version

> ro root=/dev/sda2 Find the persistent name for the root partition (following root= on the kernel line) by using the instructions in

Section 3.1.5.1, Using udev to Discover Logical to Physical Mappings for sd Devices

.

In the same file, /boot/grub/menu.lst

, replace the text after root= with the partition's persistent name. For example: kernel /boot/vmlinuz -<

kernel version

> ro root=/dev/disk/by-id/scsi-32000000c5005d6e6-part2 Change any mounts listed in /etc/fstab which refer to this root partition by either its /dev/sd name or a file system label to use the persistent name as well.

Using udev with st Devices

In Linux, the driver for SCSI tape drives is

st

. A tape device name has an

st

prefix. The udev rules for tape devices are the same as for disk devices. A unique ID must exist that persists across initiator reboots and persists regardless of discovery order.

Broadcom - 18 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration You must consider whether the tape device is an FC tape device or an FC-SCSI tape device (in which multiple SCSI tape devices reside behind an FC controller). If it is an FC tape device, then the WWPN is unique and can be used to create the persistent name. In this case, the scsi_id command should return this as the unique identifier with a single digit prefix. If the FC controller has multiple SCSI tape devices behind it, the WWPN is not unique, and the persistent name must use multiple information elements to build the unique ID.

FC Tape Device Examples

and

FC-SCSI Tape Device Example are examples of each scenario.

FC Tape Device Examples

The following examples use the scsi_id command to retrieve and generate a unique SCSI identifier: scsi_id [options] For these examples, the following [ options ] are used:   -g Treats the device as white listed. It is needed on the command line or in the scsi_id.config

file for the scsi_id command to generate any output. In the examples, the -g option is needed on the command line because the vendor and model for this tape device were not in the /etc/scsi_id.config

file.

-s Generates an id for the sysfs-device. Note that -s is an invalid option for scsi_id version 147.

NOTE

Since the [options] can vary depending on the version of the scsi_id command, refer to the

scsi_id

man page on your system for the correct and complete list of the [options].‘ The following example is an FC tape device using the SCSI generic driver (sg) rather than the SCSI tape driver. The value returned has a leading prefix of 3, which is the NAA type. The remaining digits represent the FC controller’s WWPN. scsi_id -g -s /sys/class/scsi_generic/sg0 350060b000029b592 The following example is an FC tape device using the SCSI tape driver. The value returned is the same as the previous example.

scsi_id -g -s /sys/class/scsi_tape/nst0 350060b000029b592 The following example uses a different FC tape vendor. Notice that the value returned is similar to the previous examples, with respect to the leading digit and the WWPN.

/sbin/scsi_id -g -s sys/class/scsi_tape/nst0 35005076300015101

FC-SCSI Tape Device Example

The following is an example of a FC controller with multiple SCSI tape devices behind it (FC-SCSI tape device). When the Emulex driver is loaded, the SCSI mid-level discovers the SCSI tape devices as follows: scsi scan: INQUIRY to host 14 channel 0 id 0 lun 0 scsi: unknown device type 12 Vendor: ADIC Model: SNC 4000 Rev: 42d4 Type: RAID ANSI SCSI revision: 03 Attached scsi generic sg5 at scsi14, channel 0, id 0, lun 0, type 12 scsi scan: INQUIRY to host 14 channel 0 id 0 lun 1 Vendor: ADIC Model: Scalar 24 Rev: 227A Type: Medium Changer ANSI SCSI revision: 02 Attached scsi generic sg6 at scsi14, channel 0, id 0, lun 1,type 8 scsi scan: INQUIRY to host 14 channel 0 id 0 lun 2 Vendor: IBM Model: ULTRIUM-TD2 Rev: 38D0 Type: Sequential-Access ANSI SCSI revision: 03 Attached scsi tape st0 at scsi14, channel 0, id 0, lun 2 Broadcom - 19 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Configuration st0: try direct i/o: yes (alignment 512 B), max page reachable by HBA 4503599627370495 Attached scsi generic sg7 at scsi14, channel 0, id 0, lun 2, type 1 scsi scan: INQUIRY to host 14 channel 0 id 0 lun 3 Vendor: IBM Model: ULTRIUM-TD2 Rev: 38D0 Type: Sequential-Access ANSI SCSI revision: 03 Attached scsi tape st1 at scsi14, channel 0, id 0, lun 3 st1: try direct i/o: yes (alignment 512 B), max page reachable by HBA 4503599627370495 Attached scsi generic sg8 at scsi14, channel 0, id 0, lun 3, type 1 This log output shows a controller at LUN 0, the medium changer at LUN 1, and two SCSI tape devices at LUNs 2 and 3. The following example is the result of a scsi_id call: scsi_id -g -s /sys/class/scsi_tape/nst0 1IBM ULTRIUM-TD2 1110133831 scsi_id -g -s /sys/class/scsi_tape/nst1 1IBM ULTRIUM-TD2 1110133994 Notice that the unique ID contains three values with space delimiters. A udev rule must have a unique ID for the device, so that all three parts of this returned string are required. To do this, use the following command: scsi_id -u -g -s /sys/class/scsi_tape/nst0 1IBM_____ULTRIUM-TD2_____1110133831 scsi_id -u -g -s /sys/class/scsi_tape/nst1 1IBM_____ULTRIUM-TD2_____1110133994

Creating the udev Persistent Name for SCSI Tape Device

After you know the SCSI ID call needed to extract a unique ID, use the same process to create a udev persistent name for a SCSI tape device as on a SCSI disk device.

The rule for the FC tape device is BUS="scsi", SYSFS{vendor}="HP", SYSFS{model}="ULTRIUM 3-SCSI", PROGRAM="/sbin/scsi_id -p 0x83 -u -g -s /sys/class/scsi_tape/nst%n",RESULT="350060b000029b592", SYMLINK="fc_lun_st%n" The rule for the FC-SCSI tape device is: BUS="scsi", SYSFS{vendor}="IBM", SYSFS{model}="ULTRIUM-TD2", PROGRAM="/sbin/scsi_id -p 0x83 -u -g -s /sys/class/scsi_tape/nst%n",RESULT="1IBM_____ULTRIUM-TD2_____1110133831", SYMLINK="fc_lun_st%n" BUS="scsi", RESULT="1IBM_____ULTRIUM-TD2_____1110133994", SYMLINK="fc_lun_st%n" Create a new file named /etc/udev/rules.d/45-local.rules, and put the appropriate rule in it. Then run udevtrigger to reload the udev rules, and the output of the rules will follow: udevtrigger ls -al /dev/fc* lrwxrwxrwx 1 root root 3 Apr 7 15:03 fc_lun_st0 -> st0 lrwxrwxrwx 1 root root 3 Apr 7 15:03 fc_lun_st1 -> st1 Broadcom - 20 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration VPort Configuration

3.1.5.4

3.1.6

Persistent Naming References

See the following references for more information on persistent naming:   http://www.reactivated.net/udevrules.php

by Daniel Drake (dsd) http://kernel.org/pub/linux/utils/kernel/hotplug/udev_vs_devfs by Greg Kroah-Hartman

ExpressLane Support

The host servers do not distinguish between lower and higher priority workloads being sent to LUNs. For flash storage deployed in mixed storage environments, the combination of data from rotating media and flash devices can cause congestion on the network and reduced performance for the flash storage devices.

The ExpressLane feature enables special priority queuing for selected LUNs (ExpressLane LUNs). The ExpressLane LUN performance is superior to that of regular LUNs. Mixed storage environments can use the ExpressLane feature to alleviate congestion, reduce latency, and improve throughput, ensuring that key applications receive highest priority.

The following requirements must be met to use the ExpressLane feature:   The ExpressLane feature is supported only on Emulex LPe16000-series, LPe31000-series, LPe32000-series, and LPe16202/OCe15100 adapters.

An ExpressLane LUN must be enabled in the driver before it can be used by the Emulex OneCommand Manager application. Additionally, the priority value to mark each of the ExpressLane frames must be specified to the FC/FCoE driver through the appropriate driver parameters.

For additional information, refer to the

OneCommand Manager for LightPulse Adapters User Guide

.

3.2

3.2.1

3.2.2

VPort Configuration

This section describes how to create, delete, and display VPorts.

VPort Configuration Prerequisites

Before configuring VPorts, consider the following points:  Ensure that you are using the latest recommended firmware for VPort functionality. For the latest firmware, check the Documents and Downloads area of http://www.broadcom.com.

 

NOTE

Before performing a firmware update, driver installation is required. For more information on installing the driver, see

Chapter 2 , Installing and Uninstalling

.

Loop devices and NPIV are not supported on the same port simultaneously. If you are running a loop topology and you create a VPort, the VPort link state is offline.

VPorts do not persist across system reboots.

Creating, Deleting, and Displaying VPorts

VPorts are created through sysfs entries that are presented in the physical port's sysfs directory. There are three scripts create, delete, and display VPorts. The scripts reside in the / usr/sbin/lpfc directory and are part of the OneCommand Manager application kit.

When NPIV is enabled and VPorts are configured, it can take longer for the adapter to finish discovery in some cases because each VPort must perform discovery independently. As more VPorts are configured, the amount of time that Broadcom - 21 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration VPort Configuration

3.2.2.1

3.2.2.2

3.2.2.3

the driver and adapter take to finish discovery of remote ports on the SAN increases. To compensate for this extended amount of time taken in discovery, set the lpfc_devloss_tmo parameter to 60 seconds when NPIV is enabled.

Creating VPorts Using the mkvport.sh Script

You can use the mkvport script to create VPorts. To see the usage information, run the script with no parameters specified. The mkvport.sh

script uses the following syntax: ./mkvport.sh <

Physical Port's Host number

> <

Port Name

> <

Node Name

> You must supply the physical port’s host number, WWPN, and WWNN when using the mkvport.sh

script. For example, to create a VPort with port name of 10000000c94ac63a and a node name of 20010000c94ac63a on the physical port with scsi_host name

host7

, type: ./mkvport.sh host7 10000000c94ac63a 20010000c94ac63a This script fails if the VPort is not created.

NOTE

It is possible for a VPort to be created successfully but to be in a failed state. For example, loop devices and NPIV are not supported on the same port simultaneously. If you are running a loop topology and you create a VPort, the VPort’s link state will be offline.

Deleting VPorts Using the rmvport.sh Script NOTE

You must unmap, unmount, and flush I/Os to VPort-connected devices before deleting the VPort. You can use the rmvport script to delete VPorts. To see the usage information, run the script with no parameters specified. The rmvport.sh script uses the following syntax: ./rmvport.sh <

VPort's Host number

> Or ./rmvport.sh <

Port Name

> <

Node Name

> To delete the VPort with a port name of 10000000c94ac63a and a node name of 20010000c94ac63a, type: ./rmvport.sh 10000000c94ac63a 20010000c94ac63a This script may take up to 30 seconds to finish. The script fails if the VPort is not deleted.

Displaying VPorts Using the lsvport.sh Script

You can use the lsvport script to display the VPorts and physical ports that are present on the system. Run the script with no parameters to display port information. For example: ./lsvport.sh lpfc0: host6 10000000c93a5b5e:20000000c93a5b5e LP10000 NPIV Not Supported lpfc1: host7 10000000c93a5b5d:20000000c93a5b5d LP10000 NPIV Not Supported lpfc2: host8 10000000c93cc8dd:20000000c93cc8dd LPe12000 NPIV Physical lpfc3: host9 10000000c93cc8dc:20000000c93cc8dc LPe12000 NPIV Physical lpfc4: host10 10000000c94ac63a:20010000c94ac63a NPIV Virtual (VPI 1) In reference to the previous example:    For LPFC0 and LPFC1,

NPIV Not Supported

indicates that this adapter/firmware combination does not support the creation of VPorts.

For LPFC2 and LPFC3,

NPIV Physical

refers to a physical port of this adapter.

For LPFC4,

NPIV Virtual

refers to a VPort of this adapter.

Broadcom - 22 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration VPort Configuration

3.2.3

3.2.3.1

VPort sysfs

This section describes the VPort sysfs structure and VPort sysfs entries.

VPort sysfs Tree

When a VPort is created, three new directories are created in the class tree: /sys/class/scsi_host/hostY/ /sys/class/fc_host/hostY/ /sys/class/fc_vports/vport-X:0-Z/ Creating a new VPort also creates a new sysfs directory in the bus and devices tree: /sys/bus/pci/drivers/lpfc/0000:A:B:C/hostX/vport-X:0-Z/hostY /sys/devices/pci0000:A/0000:A:B:C/hostX/vport-X:0-Z/hostY In both directories, there is a hostY directory that contains the remote ports that this new host can access: /sys/bus/pci/drivers/lpfc/0000:A:B:C/hostX/vport-X:0-Z/hostY /sys/bus/pci/drivers/lpfc/0000:A:B:C/hostX/vport-X:0-Z/hostY/rport-Y:0-0 /sys/bus/pci/drivers/lpfc/0000:A:B:C/hostX/vport-X:0-Z/hostY/rport-Y:0-1 /sys/bus/pci/drivers/lpfc/0000:A:B:C/hostX/vport-X:0-Z/hostY/rport-Y:0-2    In this example: X indicates the host value for the parent fc_host from which this VPort was created.

Y indicates the new host value for the VPort that was created.

Z indicates the instance of VPort created from the parent fc_host. A, B, and C indicate the PCI hierarchy for each physical FC or FCoE port.

hostY

is the new host created for the new VPort.

vport-X:0-Z

uniquely identifies the VPort and indicates the parent host structure (X) that created this VPort.

For example, if a VPort is created from host5, a new scsi_host, a new fc_host, a new fc_vport, and a new entry under the bus tree are created as well.

ls /sys/class/scsi_host/ host0 host1 host4 host5 host6 ls /sys/class/fc_host/ host4 host5 host6 ls /sys/class/fc_vports/ vport-5:0-0 Broadcom - 23 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration VPort Configuration

3.2.3.2

VPort sysfs Entries NOTE Table 3 VPort sysfs Entries VPort sysfs Entries

lpfc_peer_port_login lpfc_restrict_login max_npiv_vports node_name VPort sysfs entries shown in the following table are present only if the driver was loaded with lpfc_enable_npiv enabled.

Type

Read/Write Read/Write Read-only Read-only

Range/Input

0=Off (default) 1=On 0=Off 1=On (default) integers 16-byte hexadecimal value

Location and Description

/sys/class/scsi_host/hostX/lpfc_peer_port_lo gin This entry sets the port behavior when discovering targets in the SAN. The default behavior (value=0) will log in only to N_Ports that are physically located on a different port. The port will still attempt to log in to targets on all other ports (including the other port in a dual-port adapter). If this parameter is turned on (value=1), the port attempts to log in to all N_Ports, even if they are physically located on the same port.

NOTE

This parameter was created to reduce the amount of hardware resources (for example, RPIs) that the driver requires. In a configuration where there are many VPorts on one physical port, this capability greatly reduces the number of RPIs that the driver uses.

/sys/class/scsi_host/hostX/lpfc_restrict_log in (VPorts only) This entry sets the VPort behavior when discovering targets in the SAN. The default behavior (value=1) prevents the VPort from logging into other initiator ports in the SAN. It also rejects logins from other ports in the SAN, because it assumes that all ports that send a PLOGI are initiators.

If this sysfs entry is turned off, the driver attempts to log in to every port that it can access in the SAN, and accepts logins from all ports.

NOTE

This parameter was created to reduce the amount of hardware resources (for example, RPIs) that the driver requires. In a SAN where other initiators exist, this capability greatly reduces the number of RPIs that the driver uses.

/sys/class/fc_host/hostX/max_npiv_vports This entry displays the maximum number of VPorts that are supported by the underlying fc_host hardware. This sysfs entry exists only if the vport_create and vport_delete sysfs entries exist. If an fc_host does not support NPIV, this sysfs entry might not exist. Use this sysfs entry with the created on this fc_host.

npiv_vports_inuse entry to determine whether the maximum number of VPorts have been For the FC RHEL 5.x driver: /sys/class/fc_host/hostX/node_name For FC RHEL 6.x/SLES 11 SPx drivers: /sys/class/fc_host/hostX/node_name /sys/class/fc_vports/vport-X:0-Z/node_name These entries display the physical or VPort node name. You assign this value when the VPort is created, and it is transmitted to the fabric upon fabric login.

Broadcom - 24 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration VPort Configuration

Table 3 VPort sysfs Entries (Continued) VPort sysfs Entries

npiv_vports_inuse

Type

Read-only port_name vport_create vport_delete

3.2.4

Read-only Write-only Write- only

Range/Input

integers 16-byte hexadecimal value WWPN; WWNN WWPN; WWNN

Location and Description

/sys/class/fc_host/hostX/npiv_vports_inuse This entry displays the number of VPorts that were created on this fc_host. This sysfs entry exists only if the vport_create and vport_delete sysfs entries exist. If an fc_host does not support NPIV, this sysfs entry may not exist. Use this sysfs entry with this fc_host.

max_npiv_vports to determine whether the maximum number of VPorts have been created on /sys/class/fc_host/hostX/port_name /sys/class/fc_vports/vport-X:0-Z/port_name This entry displays the physical or VPort port name. You assign this value when the VPort is created, and it is transmitted to the fabric upon fabric login.

/sys/class/fc_host/hostX/vport_create This entry creates a VPort on the physical port on which hostX is located. The new VPort will have a WWPN and WWNN present on the fabric based on the WWPN and WWNN that are entered with this sysfs entry.

This entry returns a 0 if the VPort creation was successful. A nonzero value indicates that the VPort was not created. If an fc_host does not support NPIV, then this sysfs entry may not exist.

NOTE

It is possible for the VPort creation to succeed but for the VPort to be in a failed or inoperative state. Use the new sysfs tree created by the new VPort to check the state of the new VPort.

/sys/class/fc_host/hostX/vport_delete This entry deletes a VPort on the physical port on which hostX is located. The VPort matching the WWPN and WWNN is immediately deleted.

This entry returns a 0 if the VPort deletion was successful. A nonzero value indicates that the VPort was not deleted. If an fc_host does not support NPIV, then this sysfs entry may not exist.

NOTE

This entry deletes the VPort even if there are mounted file systems being accessed through this VPort, or if there are open files on it.

Monitoring VPorts with fc_vport

The transport creates an fc_vports directory that you can use to monitor VPorts. This directory is populated entirely of VPorts and has links from each to the fc_host associated with that VPort.

ls /sys/class/fc_vports/ vport-5:0-0 ls -d /sys/bus/pci/drivers/lpfc/*/host*/*/host* /sys/bus/pci/drivers/lpfc/0000:03:06.1/host5/vport-5:0-0/host6 ls /sys/devices/pci*/*/host5/vport-5*/host6 power rport-6:0-0 rport-6:0-1 rport-6:0-2 uevent ls /sys/devices/pci*/*/host5/vport-5*/host6/rport-* /sys/devices/pci00:03/00:03:06.1/host5/vport-5:0-0/host6/rport-6:0-0: Broadcom - 25 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

3.2.5

Chapter 3: Configuration VPort Configuration power uevent /sys/devices/pci00:03/00:03:06.1/host5/vport-5:0-0/host6/rport-6:0-1: power uevent /sys/devices/pci00:03/00:03:06.1/host5/vport-5:0-0/host6/rport-6:0-2: power target6:0:0 uevent In this example:    A new entry exists in the fc_vports directory for the VPort (vport-5:0-0). The vport-5:0-0 entry indicates that the VPort was created from host5 and it is the first (0) VPort to be created on that fc_host. The new host for the VPort is host6, and it will appear in the usual directories.

A new directory exists in the bus tree. This new directory indicates that host6 was created under vport-5:0-0 (which was created from host5).

VPort Configuration Limits

   VPort configuration limits are designated as enforced or unenforced. Enforced limits are limits that the driver enforces and prevents you from exceeding. Unenforced limits are limits that the driver cannot enforce, but configurations that exceed them are unsupported.

The following VPort configuration limits have been tested with and are supported by the Emulex driver. Configurations that exceed one or more of these limits are unsupported.

  Before the VPort is deleted or the driver is unloaded, I/O devices accessed through a VPort must be stopped and file systems must be unmounted.

For all LPe12000-series, LPe16000-series (including LPe16202/OCe15100), LPe31000-series, and LPe32000-series adapters the maximum number of VPorts configurable on a physical port is 255.

The maximum number of LUNs supported on each driver port is 256.

The maximum number of targets supported for each driver port is 255.

The maximum number of driver ports in one zone is 64. This limit is based on the system’s ability to recover from link events within the time constraints of the default timers. The NPIV use-cases that involve virtual server environment include associating a VPort with a virtual machine, and placing the virtual machine in its own zone, which results in one VPort per zone. In the case of load-balanced environments, this can increase typically to two VPorts per virtual machine, to a practical limit of something far less than 50. In the NPIV cases not related to virtual server environments, zoning is typically initiator-zoning, again resulting in one VPort, or a low number of VPorts in the case of load-balancing, within a given zone. If there are too many VPorts within a single zone, expected behavior includes devices being lost after link events.

 The minimum lifetime of a VPort is 60 seconds. An unenforced limit of 60 seconds exists between the creation of a VPort and the deletion of the same VPort. VPorts are designed to exist for a long time in the system, and the creation of VPorts is asynchronous, which means that a VPort might not be finished with FC or SCSI discovery when the command to create a VPort is finished. Broadcom - 26 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters

3.3

Ethernet Driver Configuration for LPe16202/OCe15100 Adapters 3.3.1

The following section describes how to configure parameters for the Ethernet driver.

Ethernet Driver Configuration Parameters

The Ethernet driver configurable parameters are described in the following table.

Table 4 Ethernet Driver Configuration Parameters Parameter

num_vfs rss_on_mc rx_frag_size

Description

In systems supporting SR-IOV, when IOV is enabled, this parameter indicates the number of VFs to be enabled per PF. For configuring SR-IOV, see

Section 3.3.7, SR-IOV Configuration .

The default value is 0 (SR-IOV is not enabled). The possible values are 0 to 63.

NOTE

This parameter is obsolete. Use the sysfs method to enable or disable VFs.

Enables receive-side scaling (RSS) on multichannel functions that have the capability.

The default value is 0 (disabled). The enabled value is 1.

NOTE

Currently, this parameter is ignored by the driver. The driver enables RSS on multichannel functions that have the capability by default.

The size of fragments used to DMA received data. The possible values are 2048 (default), 4096 and 8192.

You can configure Linux to automatically load the driver with any of these options after each reboot. To do so, add a line to /etc/modprobe.conf

with the required options. For example, to load the driver with the fragment size of 4096, add the following line: options lpnic rx_frag_size=4096

3.3.2

Support for Ethtool set-channels/get-channels Commands

The Ethtool get-channels command – ethtool -l <

ethx

> displays the following information:   The current number of Rx/Tx/interrupt queue pairs (a

combined channel

) created by the NIC function The maximum number of channels supported by the NIC function The Ethtool set-channels command ethtool -L <

ethx

> combined N configures the number of channels requested for the NIC function.

NOTE

   You can increase (up to the maximum supported limit) or decrease the number of channels used by the NIC function.

The NIC driver supports only

combined

channels.

The maximum number of channels supported by an interface is also limited to the number of cpu-cores in the server.

NOTE

In older Linux distributions, use the sysfs interface as follows:  The maximum number of channels supported by NIC function:  cat /sys/class/net/<

ethx

>/max_qs The current number of channels configured: cat /sys/class/net/<

ethx

>/num_qs Broadcom - 27 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

3.3.3

3.3.4

3.3.5

Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters  To configure the 'N' number of channels requested for the NIC function: echo N > /sys/class/net<

ethx

>/num_qs

Support for Ethtool set-dump Command

The Ethtool set-dump command ethtool -W <

ethx

> N can be used as follows:   To trigger a FW dump: ethtool -W <

ethx

> 1 To delete an existing dump: ethtool -W <

ethx

> 2

NOTE NOTE

 Delete an existing dump first before triggering a new dump.

In older Linux distributions, use the sysfs interface as follows:  To trigger a FW dump:  # echo 1 > /sys/class/net/<

ethx

>/trigger_dump To delete an existing dump: # echo 1 > /sys/class/net/<

ethx

>/delete_dump

Transmit/Receive Queue Counts

Use the command ethtool -l <

ethx

> to view the Transmit and Receive queue counts. For example: ethtool -l <

ethx

> Output Channel parameters for enp9s0f0: Pre-set maximums: RX: 7 TX: 7 Other: 0 Combined: 8 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 8

Support for Ethtool set-rxfh/get-rxfh Commands

The Ethtool get-rxfh and set-rxfh commands are available in kernel versions of RHEL 7.1, 3.16.0 and higher.

The Ethool get-rxfh command – ethtool –x <

ethx

> shows the following information:   The current indirection table entries The programmed RSS hash-key Broadcom - 28 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters

3.3.6

3.3.7

3.3.7.1

The Ethtool set-rxfh command – ethtool –X <

ethx

> [hkey aa:bb:cc...] [equal N | weight W0 W1...] sets the following parameters:   Indirection table entries RSS hash-key

NOTE

In older Linux distributions, use the sysfs interface as follows:  To show the currently configured RSS hash key:  cat /sys/class/net/<

ethx

>/rss_hashkey To configure the new RSS hash key: echo -e "aa:bb:cc...." > /sys/class/net/<

ethx

>/rss_hashkey

Support for Showing Onboard Die Temperature

Onboard die temperature is exported through the sysfs interface. On kernels that support the hardware monitoring framework, the temperature is exported using the hwmon sysfs interface. Temperatures exported using the hwmon interface can also be read using the lm-sensors utility with the sensors command.

To show the current board temperature using the sysfs node: cat /sysclass/hwmon/<

hwmonx

>/device/temp1_input Otherwise, the current board temperature is exported with the regular sysfs interface: cat /sys/class/net/<

ethx

>/device/temp1_input

SR-IOV Configuration

Introduction

This section contains requirements and instructions to use SR-IOV with the following host operating systems:            Red Hat Enterprise Linux 6.6 – 64-bit KVM Red Hat Enterprise Linux 6.7 – 64-bit KVM Red Hat Enterprise Linux 6.8 – 64-bit KVM Red Hat Enterprise Linux 7.1 – 64-bit KVM Red Hat Enterprise Linux 7.2 – 64-bit KVM SuSE Linux Enterprise Server 11 SP3 – 64-bit Xen and KVMRed Hat Enterprise Linux 7.3 - 64-bit KVM SuSE Linux Enterprise Server 11 SP3 – 64-bit Xen and KVM SuSE Linux Enterprise Server 11 SP4 – 64-bit Xen and KVM SuSE Linux Enterprise Server 12SP1 – 64-bit Xen and KVM SuSE Linux Enterprise Server 12 SP2 – 64-bit Xen and KVM SLES and RHEL supported with Xen These environments support capabilities of the Emulex OneConnect® adapter to enable multiple PCIe virtual functions (VFs) for a PCIe physical function (PF). Each of these VFs can be assigned to virtual machines (VMs). A VF enables the guest operating system direct access to the Emulex OneConnect adapter, such that guest performance is not limited by the overheads of the hypervisor.

Broadcom - 29 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters

3.3.7.2

    With SR-IOV, VMs directly drive I/Os on the NIC. Therefore, SR-IOV has the following advantages over traditional virtualized I/O: Improved device performance for virtual guests Increased scalability Reduced CPU utilization Reduced latency Known issues include:  The Kdump (kernel dump) feature is not supported when SR-IOV is enabled.

Setting Up SR-IOV Prerequisites (except FLEX10)

To set up SR-IOV on your system, you need the following:  A server or blade with an SR-IOV-capable motherboard BIOS  

NOTE

Configuration mechanisms for parameters, such as MAC address, VLAN, and QoS for VF are supported in RHEL 6 and later, and SLES 11 and later distributions.

An Emulex LPe16202/OCe15100 adapter in FCoE+NIC mode If a KVM hypervisor is installed, it must contain the qemu-kvm packages.

Depending on your system, perform one or more of the following tasks to set up your BIOS. For more information, refer to the manufacturer’s instructions for your system.

  Enable SR-IOV in the system BIOS, which supports SR-IOV functionality.

Enable Intel Virtualization Technology support for Direct I/O VT-d.

You can use the PXESelect utility, the Emulex OneCommand Manager application, or the UEFI (HII) utility to set up SR-IOV.

To enable and configure SR-IOV using either the UEFIBoot utility or the PXESelect utility, refer to the

Boot for LPe16202/OCe15100 Adapters User Guide

.

1.

2.

3.

4.

5.

Install the required Linux operating system that serves as a hypervisor.

For legacy boot, perform the following step: Update the /boot/grub/menu.lst

file to include the following command line load parameter for the Linux kernel: intel_iommu=on For UEFI boot, perform the following step: Update the /etc/default/grub file to include the following command line load parameter for the Linux kernel: intel_iommu=on Install the appropriate version of the Emulex Ethernet driver that supports SR-IOV for the operating system version that you are using.

Reboot the server for the new changes to become operational.

Use the lspci -vvv output command to check if SR-IOV is properly enabled.

The lspci -vvv command returns an SR-IOV capability section for each Ethernet PF. The Initial VFs and Total VFs should be nonzero. Make a note of the lspci command output in the hypervisor. This output is needed in step

10

. Specifically, make note of the pci-id of the VFs that have been created.

Broadcom - 30 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters 6.

7.

8.

The following is an example of the SR-IOV capabilities section output of the PF with SR-IOV enabled: Capabilities: [180] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy IOVSta: Migration Initial VFs: 32, Total VFs: 32, Number of VFs: 0, Function Dependency Link: 00 VF offset: 0, stride: 1, Device ID: 0710 Supported Page Size: 00000557, System Page Size: 00000001 Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 The following is an example of the SR-IOV capabilities section output of the PF with SR-IOV disabled: Capabilities: [180] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy IOVSta: Migration Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Dependency Link: 00 VF offset: 0, stride: 1, Device ID: 0710 Supported Page Size: 00000557, System Page Size: 00000001 Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Create a VM using the Virtual Machine Manager utility, and install any supported operating system on the VM.

Shut down the VM.

Echo the number of VFs/PF to the sriov_numvfs file, where

X

is the number of VFs per PF: echo

X

> /sys/bus/pci/devices/..../sriov_numvfs The possible values are 0 to 64 per physical port. The default value is 0 (SR-IOV is not enabled). With the latest kernels, the sysfs path to enable VFs is /sys/bus/pci/devices/<

device-id

>/sriov_numvfs and for earlier kernels, the sysfs path to enable VFs is /sys/bus/pci/devices/<

device-id

>/net/<

ifaceName

>/sriov_numvfs The total number of VFs can be distributed among available ports as required, but each port has a maximum of 64 VFs.

Table 5 on page 34 lists the total number of VF counts that are supported for various adapter configurations.

The maximum number of VFs supported per PF can be read from: cat/sys/bus/pci/devices/..../sriov_totalvfs 9.

NOTE

VFs are supported only for network functions; they are not supported for storage functions.

The total VF count that is supported for a two-port LPe16002/OCe15100 adapter in NIC plus FCoE mode (two NIC PFs and two FcoE PFs) is 128. The VF count is symmetrically distributed across the ports, which means that the VF count per NIC port is 64. Detach VFs from the host before adding them to the guest.

Example 1 (using only one VF per physical port): RHEL 6.x KVM: View Emulex PCI devices with the lspci command: lspci | grep Emulex 16:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30)<-- LOM Port 0 (Function 0) 16:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30)<-- LOM Port 1 (Function 1) 16:04.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30)<-- VF LOM Port 0 (Function 0) 16:08.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30)<-- VF LOM Port 1 (Function 1) # lspci -nn | grep Emulex 16:00.0 Ethernet controller: [0200]: Emulex Corporation OneConnect Broadcom - 31 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters

3.3.7.3

NIC (Lancer)[10df:0e220] (rev 03) <-- LOM Port 0 (Function 0) 16:00.1 Ethernet controller: [0200]: Emulex Corporation OneConnect NIC (Lancer)[10df:0e220] (rev 03) <-- LOM Port 1 (Function 1) 16:04.0 Ethernet controller: [0200]: Emulex Corporation OneConnect NIC (Lancer)[10df:0e220] (rev 03) <-- VF LOM Port 0 (Function 0) 16:08.0 Ethernet controller: [0200]: Emulex Corporation OneConnect NIC (Lancer)[10df:0e220] (rev 03) <-- VF LOM Port 1 (Function 1) Detach desired VFs: # virsh nodedev-dettach pci_0000_16_04_0 Device pci_0000_16_04_0 detached <-- VF LOM Port 0 (Function 0) # virsh nodedev-dettach pci_0000_16_08_0 Device pci_0000_16_08_0 detached <-- VF LOM Port 1 (Function 1) Example 2 (using only one VF per physical port): SLES 11.x Xen: View Emulex PCI devices with the lspci command: lspci | grep Emulex 16:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30)<-- LOM Port 0 (Function 0) 16:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30)<-- LOM Port 1 (Function 1) 16:04.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30)<-- VF LOM Port 0 (Function 0) 16:08.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30)<-- VF LOM Port 1 (Function 1) # lspci -nn | grep Emulex 16:00.0 Ethernet controller: [0200]: Emulex Corporation OneConnect NIC (Lancer)[10df:0e220] (rev 03) <-- LOM Port 0 (Function 0) 16:00.1 Ethernet controller: [0200]: Emulex Corporation OneConnect NIC (Lancer)[10df:0e220] (rev 03) <-- LOM Port 1 (Function 1) 16:04.0 Ethernet controller: [0200]: Emulex Corporation OneConnect NIC (Lancer)[10df:0e220] (rev 03) <-- VF LOM Port 0 (Function 0) 16:08.0 Ethernet controller: [0200]: Emulex Corporation OneConnect NIC (Lancer)[10df:0e220] (rev 03) <-- VF LOM Port 1 (Function 1) List Emulex PCI device by device code: # virsh nodedev-list | grep 10df pci_10df_e220 <-- LOM Port 0 (Function 0) pci_10df_e220_0 <-- LOM Port 1 (Function 1) pci_10df_e220_1 <-- VF LOM Port 0 (Function 0) pci_10df_e220_2 <-- VF LOM Port 1 (Function 1) # virsh nodedev-dettach pci_10df_e220_1 Device pci_10df_e220_1 detached <-- VF LOM Port 0 (Function 0) # virsh nodedev-dettach pci_10df_e220_2 Device pci_10df_e220_2 detached <-- VF LOM Port 1 (Function 1)

10. Use the Virtual Machine Manager GUI to attach the VF (step 5 on page 30) to the guest operating system by using

the add physical device option.

NOTE

To reconfigure a system that is already set up, remove the attached VF from the guest operating system by selecting the VF and using the remove option. Refer to the documentation for the host operating system for information on using the Virtual Machine Manager to attach and remove virtual interfaces.

11. Start the RHEL 6.x guest operating system. After the guest operating system is booted, use the lspci command to confirm the visibility of the NIC function. The output shows a NIC function, for example: 06:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) 12. The Ethernet driver automatically loads with the out-of-box driver, and creates the network interfaces. Use the ifconfig command to verify that the interface is created.

13. After configuring the network interfaces with proper IP addresses, you can send and receive network traffic from the VM. Refer to the documentation for the host and guest operating systems for information on network configuration.

Assigning VFs to a VM on the SLES Operating System

To assign VFs to the VM in the SLES Xen kernel, the VF must be unbound from the NIC module and then bound to the pciback module.

Broadcom - 32 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters 1.

2.

3.

4.

5.

6.

NOTE

In the following steps,

0000:07:0b.5

is used as an example. To match those instances to the port that you want to use, you need to select the entry that matches the PCI bus, device, or the function that corresponds to the port that you want to assign. Use the ethtool utility to determine this information, such as ethtool -i eth0 (where eth0 is the interface you want to assign).

Load the pciback driver

modprobe pciback

.

Navigate to the /sys/bus/pci/drivers/pciback directory, and ensure that the following is displayed: Panama-Sles11sp2:/sys/bus/pci/drivers/pciback # ls -lrt total 0 --w------- 1 root root 4096 Sep 5 15:29 unbind --w------- 1 root root 4096 Sep 5 15:29 uevent -r-------- 1 root root 4096 Sep 5 15:29 slots --w------- 1 root root 4096 Sep 5 15:29 remove_slot --w------- 1 root root 4096 Sep 5 15:29 remove_id -rw------- 1 root root 4096 Sep 5 15:29 quirks -rw------- 1 root root 4096 Sep 5 15:29 permissive --w------- 1 root root 4096 Sep 5 15:29 new_id lrwxrwxrwx 1 root root 0 Sep 5 15:29 module -> ../../../../module/pciback --w------- 1 root root 4096 Sep 5 15:34 new_slot --w------- 1 root root 4096 Sep 5 15:34 bind Navigate to the /sys/bus/pci/drivers/lpnic directory, and ensure that the following is displayed: --w------- 1 root root 4096 Sep 5 15:32 uevent --w------- 1 root root 4096 Sep 5 15:32 remove_id --w------- 1 root root 4096 Sep 5 15:32 new_id lrwxrwxrwx 1 root root 0 Sep 5 15:32 module -> ../../../../module/lpnic --w------- 1 root root 4096 Sep 5 15:32 bind --w------- 1 root root 4096 Sep 5 15:33 unbind lrwxrwxrwx 1 root root 0 Sep 5 15:32 0000:07:0b.5 -> ../../../../devices/pci0000:00/0000:00:09.0/0000:07:0b.5

lrwxrwxrwx 1 root root 0 Sep 5 15:32 0000:07:0b.4 -> ../../../../devices/pci0000:00/0000:00:09.0/0000:07:0b.4

lrwxrwxrwx 1 root root 0 Sep 5 15:32 0000:07:0b.3 -> ../../../../devices/pci0000:00/0000:00:09.0/0000:07:0b.3

lrwxrwxrwx 1 root root 0 Sep 5 15:32 0000:07:0b.2 -> ../../../../devices/pci0000:00/0000:00:09.0/0000:07:0b.2

lrwxrwxrwx 1 root root 0 Sep 5 15:32 0000:07:0b.1 -> ../../../../devices/pci0000:00/0000:00:09.0/0000:07:0b.1

lrwxrwxrwx 1 root root 0 Sep 5 15:32 0000:07:0b.0 -> ../../../../devices/pci0000:00/0000:00:09.0/0000:07:0b.0

Unbind the VF from the lpnic driver.

echo –n “0000:07:0b.5” > /sys/bus/pci/drivers/lpnic/unbind Bind the driver to the pciback module.

echo –n “0000:07:0b.5” > /sys/bus/pci/drivers/pciback/new_slot echo –n “0000:07:0b.5” > /sys/bus/pci/drivers/pciback/bind Navigate to the /sys/bus/pci/drivers/pciback directory, and ensure that the device

0000:07:0b.5

is listed under it. In addition, check that dmesg logs report the same device.

You can now launch qemu-kvm and attach VF

0000:07:0b.5

to any desired VM.

Broadcom - 33 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters

3.3.7.4

3.3.7.5

Link State Reporting with SR-IOV

When VEB is used to switch traffic between the functions of an SR-IOV-enabled port, the link status reported to the operating system stack when the physical port is not connected varies with the adapter type. The VFs indicate the link as DOWN when the physical port is not linked up. This behavior enables two or more VFs to be configured in a bond.

The following table summarizes the default link status of the PFs and VFs when SR-IOV is enabled in BIOS and in the driver.

Table 5 Default Link Status of the PFs and VFs Function

Physical function (PF) Virtual function (VF)

Physical Link Is Not Connected

LPe16000 Down Down The default behavior of VFs can be changed using the ip link command in distributions of Linux that support iproute version 3.11.0 or later. For details on configuring the link state for VFs, see

Section 3.3.7.6, Link State Configuration .

When VEPA is configured to switch traffic between the functions instead of VEB, the VF link state always reflects the physical state of the associated port and this behavior cannot be changed.

Configuring VFs

In operating system distributions with newer IP commands that support VF configuration options, the host administrator can perform the following by using the ip link set command:      Change the default MAC address Configure VLAN Configure the transmission rate Configure the link state Set the QoS parameter on VFs

MAC Address Configuration

The LPe16202/OCe15100 adapter in NIC+FCoE mode is shipped with factory-configured MAC addresses for the network interfaces corresponding to the PFs. The driver generates random MAC addresses for the network interfaces corresponding to the VFs based on the factory-configured MAC address. Other MAC addresses can be assigned for the interfaces corresponding to the VF using IP utility commands in the hypervisor.

To configure the MAC address for the virtual function, run the following command in the hypervisor: # ip link set eth<

X

> vf <

VFN

> mac <

MAC-ADDR

>    where: eth

is the interface corresponding to the physical function.

<

VFN

> is the VF number (0-based) corresponding to the interface for which you are configuring the MAC address.

<

MAC-ADDR

> is the MAC address you are configuring.

For example, to configure the MAC address for VF number 0 on eth0(PF), run the following command in the hypervisor: # ip link set eth0 vf 0 mac 00:16:88:AA:BB:AA Broadcom - 34 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters

3.3.7.6

If the VM is already running and the VF driver is loaded, the VF MAC address can be updated using one of the following two methods in the VM:  If the VF interface is configured using ifcfg script: a.

Update the ifcfg script for the VF, with the MAC address configured in the hypervisor in the earlier step. For example, update the HWADDR line in the VF ifcfg-eth<

X

> script.

b.

HWADDR=00:16:88:AA:BB:AA Restart the network service.

# service network restart Or  If the VF interface is not configured using ifcfg script: Update the VF MAC address using the ifconfig or ip link command.

# ifconfig eth0 hw ether 00:16:88:AA:BB:AA

VLAN Configuration

This section includes examples for transparent tagging and virtual guest tagging (VGT).

When transparent tagging is configured for a VF, the NIC transparently tags all non-VLAN traffic from the VF with the configured transparent VLAN ID. The VM is not aware of the VLAN tag.

To assign a transparent VLAN ID to the VF, run the following IP command in the hypervisor: # ip link set eth<

X

> vf <

VFN

> vlan <

VLAN

>    where: eth <

X

> is the interface corresponding to the physical function <

VFN

> is the VF number corresponding to the interface for which you are configuring the VLAN <

VLAN

> is the VLAN ID you are configuring Example: # ip link set eth0 vf 0 vlan 5

NOTE

Guest VLAN tagging is not allowed on the VF in the guest operating system, when transparent VLAN tagging is enabled on the VF.

Transmission Rate Configuration

Configure the transmission rate limit (TX-RATE) on a VF interface from the hypervisor using the following IP command syntax: # ip link set eth<

X

> vf <

VFN

> rate <

TX-RATE

>    where: eth <

X

> is the interface corresponding to the physical function <

VFN

> is the VF number corresponding to the interface that you are configuring the TX-rate <

TX-RATE

> is the transmission rate limit, in Mb/s For example, to set a TX-RATE of 5000 Mb/s for the VF 0, run the following command: # ip link set eth0 vf 0 rate 5000

Link State Configuration NOTE

The link state configuration feature is supported on RHEL 7.x and SLES 12 and later and is not supported in earlier kernels.

This section provides link state configuration for a VF. Broadcom - 35 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Ethernet Driver Configuration for LPe16202/OCe15100 Adapters

3.3.7.7

3.3.7.8

3.3.8

Configure the link state on a VF interface from the hypervisor using the following IP command syntax: #ip link set eth<

X

> vf <

VFN

> state < auto | enable | disable >      where: eth <

X

> is the interface corresponding to the physical function <

VFN

> is the VF number corresponding to the interface for which the link state is being configured auto – VF link state will reflect the PF link state enable – VF link state will be always up disable – VF link state will be always down

Spoof Check Configuration

Configure the HW packet source MAC spoof check on a VF interface from the hypervisor using the following IP command syntax: # ip link set eth<

X

> vf <

VFN

> spoofchk [on|off] where:   eth <

X

> is the interface corresponding to the physical function <

VFN

> is the VF number corresponding to the interface for which you are configuring the spoof check For example, to turn on spoof checking for the VF 0, run the following command: # ip link set eth0 vf 0 spoofchk on

Viewing VF Properties

To view the properties configured to VFs attached to a PF, use the following IP command syntax: # ip link show eth<

X

> where eth <

X

> is the interface corresponding to the physical function.

For example, to view the properties of a PF at eth0 (which has VFs 0, 1 associated with it), run the following command in the hypervisor: # ip link show eth0 Expected example output: eth0: mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:00:c9:bb:16:ee brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:c9:9d:90:80, tx rate 10000 (Mbps) vf 1 MAC 00:00:c9:9d:90:81, tx rate 10000 (Mbps)

Bonding Considerations

VFs can be added to a bonding configuration. But because VFs are typically assigned to a VM, it might not be possible to identify two VFs from the same PF or physical port, which could lead to incorrect configurations. To avoid this, the driver exports a physical port identifier corresponding to a given network interface. VFs from the same physical port will have the same identifier.

For example, to obtain the physical port identifier of a VF eth0 in a VM, read the corresponding sysfs file phys_port_id in the VM as shown here. The number in the output shown is the physical port identifier of eth0.

# cat /sys/class/net/eth0/phys_port_id 0100000000000000000000383838314643 Broadcom - 36 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Performance Tuning The NetworkManager reports incorrect bonding configuration with a message like this: "NetworkManager: (bond0): slave eth1 shares a physical port with existing slave eth0"

3.4

3.4.1

3.4.1.1

3.4.1.2

FC and FCoE Driver Performance Tuning

This section describes how to tune the FC and FCoE driver for best performance.

Overview

The following configurable parameters can enhance performance:    lpfc_fcp_io_channel lpfc_fcp_io_sched lpfc_fcp_imax These features are available through module parameters that are defined in the FC and FCoE driver as well as sysfs entries defined by the Linux kernel.

In addition, you can use the lpfc_vector_map.sh

script to map a specific I/O channel to a specific CPU. Determination of the mapping of the I/O channel to a specific CPU is also handled by the driver.

This section provides more information about how the tuning parameters and script can improve Emulex adapter performance.

The parameters in this section do not apply to LPe12000-series adapters.

lpfc_fcp_io_channel

The lpfc_fcp_io_channel module parameter can be configured at driver load time. It defines the number of I/O channels supported by the driver. The driver is capable of supporting parallel I/O paths, and each I/O path is capable of posting and completing FCP commands independent of the other.

Emulex adapters that are running in MSI-X interrupt mode can use more than one I/O path. Each I/O channel is composed of a unique MSI-X vector- EQ/CQ/WQ tuple. This parameter will override the value of the lpfc_fcp_eq_channel parameter.

NOTE

The Emulex LPe12000 adapters support only one I/O path, so this parameter has no effect on them.

By default, the driver is configured for four I/O channels per port. The driver will also limit the number of I/O channels to not exceed the number of online

logical

CPUs (as reported by /proc/cpuinfo ).

lpfc_fcp_io_sched

The lpfc_fcp_io_sched module parameter can be configured at driver load time. It also can be set dynamically as an sysfs entry. The driver uses the parameter to determine which algorithm to use when scheduling an FCP I/O to an I/O channel.

When multiple I/O channels are in use, I/Os can be scheduled to an I/O channel in a round-robin fashion, or by determining which CPU is running when the I/O is submitted.

The default value (0) configures the driver for round-robin scheduling. A value of 1 configures scheduling by CPU.

Broadcom - 37 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration FC and FCoE Driver Performance Tuning

3.4.1.3

3.4.1.4

lpfc_fcp_imax

The lpfc_fcp_imax can be configured at driver load time. It also can be set dynamically as an sysfs entry. This parameter defines the maximum number of interrupts per second that each adapter port will support.

Considerations

   The lower the value set, the more completions are coalesced by the adapter, which causes the driver to handle multiple completions under the context of one interrupt. The higher the value, the faster an interrupt is generated for a completed command. Therefore, a balanced or

tuned

system must be found.

A lower value equals higher interrupt latency; a higher value equals lower interrupt latency.

Faster completions consume more system resources and CPU cycles, as the overhead of one interrupt completes fewer commands. The value is divided by the number of I/O channels, and each I/O channel is separately configured for its own interrupt latency.

By default, the module parameter is configured for 50,000 interrupts per second per adapter port. Older driver versions have a default value of 10,000.

lpfc_vector_map.sh

The lpfc_vector_map.sh

script uses kernel sysfs entry points to map a specific I/O channel (MSI-X vector-EQ/CQ/WQ tuple) to a specific CPU. Run the script immediately after the driver is loaded.

This script resides in /usr/sbin/lpfc . Run this script by adding the following line to /etc/modprobe.d/lpfc.conf

and through /etc/rc.d

for the initial boot: install lpfc /sbin/modprobe --ignore-install lpfc; /usr/sbin/lpfc/lpfc_vector_map.sh

The script maps each interrupt vector allocated by the driver to a CPU, thereby spreading the interrupt load of the ports across multiple CPUs. Each vector, with its associated I/O channel, is sequentially mapped to a CPU in a round-robin fashion. The number of vectors assigned to each adapter port is defined by the lpfc_fcp_io_channel module parameter.

I/O channels, which correspond to vectors, are typically mapped to unique CPUs to enhance the ability of the driver to run multiple FCP commands in parallel. In addition, running this script forces I/O scheduling to be by CPU ( lpfc_fcp_io_sched = 1 ), which increases performance when a specific I/O channel is mapped to a specific CPU.

The script has two modes of operation: Driver mode and HBA mode. By default, the script runs in Driver mode.

Driver Mode

Driver mode maps all vectors for all driver ports, starting with CPU0, sequentially assigning a new CPU for each vector belonging to the entire driver. If more interrupt vectors exist than CPUs, the vector assignment wraps back to CPU0 as needed.

HBA Mode

HBA mode maps all vectors for each specific adapter port, starting with CPU0, sequentially assigning a new CPU for each vector belonging to that adapter port.

Broadcom - 38 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Network Performance Tuning

3.5

3.5.1

3.5.1.1

3.5.1.2

3.5.1.3

3.5.1.4

3.5.2

Network Performance Tuning

This section describes optimizing network performance.

Memory Bandwidth Considerations

The availability of higher memory bandwidth leads to better network performance. The following sections describe how memory bandwidth can be increased.

Enabling Optimal Bandwidth Options

Most computers offer multiple distinct memory channels, or memory interleaves, that may not be enabled by default. Check the manufacturer's documentation and BIOS parameters for details on enabling optimal memory bandwidth options.

Populating DIMM Slots

Typically, all the dual inline memory module (DIMM) slots must be populated to make use of all the memory channels. As a general rule, using more DIMMs provides better performance by allowing a higher degree of memory-access interleaving to occur.

Disabling Memory Mirroring

Some servers may allow memory mirroring, where the total memory is divided in half and each location is stored twice. This situation allows fault recovery if one memory location detects an error, but it greatly reduces the perceived memory bandwidth of the system. Consider disabling memory mirroring if it is not needed.

Using a Fast Clock Speed for the Front Side Bus (FSB)

Nearly any desktop or low-end server has enough memory bandwidth for Emulex LPe16202/OCe15100 adapters in NIC mode to support DMA at 20Gb/s of data (10Gb/s read, 10Gb/s write). However, most of the memory demands come from the processor accessing the data for either packet copies in the non-offloaded networking stack or application accesses. All processor memory accesses use the FSB. The clock speed of this bus is critical for allowing efficient memory bandwidth. A system with a faster processor FSB clock speed performs better than a system with a slower FSB clock speed.

Network Memory Limits

The default values of tunable parameters in the Linux network stack are optimal for most network applications involving several TCP/UDP streams. The optimal size for the network queues and buffers depend on several factors such as protocol, number of streams (connections), request size, and application behavior. The following network configuration settings are a good combination to get the best uni-directional transmit and receive performance with six or more TCP connections/UDP streams: echo 4096 87380 4194304 > /proc/sys/net/ipv4/tcp_rmem echo 4096 16384 4194304 > /proc/sys/net/ipv4/tcp_wmem echo 64000000 > /proc/sys/net/core/rmem_default echo 64000000 > /proc/sys/net/core/rmem_max echo 32000000 > /proc/sys/net/core/wmem_default echo 32000000 > /proc/sys/net/core/wmem_max These settings assume ideal conditions such as low latency, zero (or close-to-zero) packet loss in the network, enough free memory, and 10Gb/s path-to-peer system. Broadcom - 39 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

3.5.3

3.5.4

Chapter 3: Configuration Network Performance Tuning These tcp_rmem and tcp_wmem values are also the default values in recent RHEL 5 distributions. If your application requires best throughput with very small number of connections (less than four), it may help to increase the tcp_rmem and tcp_wmem to much larger values: echo 4096 87380 16777216 > /proc/sys/net/ipv4/tcp_rmem echo 4096 65536 16777216 > /proc/sys/net/ipv4/tcp_wmem

TCP Segmentation Offload (TSO)

In low-loss networks, TSO considerably improves performance, and therefore must be enabled. TSO is enabled by default in the OneConnect network driver. The /proc/sys/net/ipv4/tcp_tso_win_divisor process variable controls how aggressive the network stack can be in making TSO requests. For low-loss networks, the TSO divisor values must be in the range of 2 to 16. In most distributions, the default value of 3 may be the optimal choice for a no-loss network.

Smaller divisor values result in larger TSO chunks and better throughput, as well as better CPU utilization. However, if the receiver or the network is dropping frames (too many retransmits on the transmit side as indicated by netstat -st ), it might help to make TSO chunks smaller (by increasing the divisor value) or to turn TSO off. For example, to set the divisor level to a value of 8, run: echo 8 > /proc/sys/net/ipv4/tcp_tso_win_divisor To turn TSO on or off, run one of the following ethtool commands: ethtool -K eth<

N

> tso on ethtool -K eth<

N

> tso off where eth <

N

> is the name of the Ethernet device you are working on (for example, eth0).

Flow Control

LPe16202/OCe15100 adapters in NIC mode support IEEE 802.3x standard flow control, which uses control packets to temporarily pause the transmission of packets between two endpoints. These control messages are point-to-point and are not forwarded by switches or routers. The adapter can respond to flow control packets by temporarily pausing transmits. The adapter can send flow control pause packets when the transmitter is overwhelming the system’s receive bandwidth. Flow control can greatly improve performance, as described in the following examples:   The adapter is installed in 4x PCIe slot or an underpowered server system.

If the PCIe bus does not provide 10Gb/s of throughput due to chipset limitations or the bus width, the adapter cannot maintain 10Gb/s of incoming receive data. It starts dropping packets quickly. In this situation, it may be beneficial to enable receive flow control in the adapter, and enable flow control in the attached switch for all devices. This action helps to slow down the transmitters.

The adapter transmits to 1Gb/s devices, especially when using a non-TCP protocol.

If the adapter transmits to a 10Gb/s switch with attached 1Gb/s clients, the adapter may overwhelm the switch. The switch is then forced to start dropping packets because, although it may receive a 10Gb/s stream, the client can only sink a 1Gb/s stream. In this situation, it may be beneficial to enable transmit flow control in the adapter and enable flow control for the 10Gb/s switch port.

You can configure the adapter to respond to flow control pause frames from the other side (switch or router) using the following ethtool commands: ethtool -A eth<

N

> pause rx on ethtool -A eth<

N

> pause rx off where eth<

N

> is the name of the Ethernet device you are working on (for example, eth0 ).

Broadcom - 40 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

3.5.5

3.5.6

3.5.7

Chapter 3: Configuration Network Performance Tuning You can configure the adapter to send flow control pause frames using the following ethtool commands: ethtool -A eth<

N

> pause tx on ethtool -A eth<

N

> pause tx off where eth <

N

> is the name of the Ethernet device you are working on (for example, eth0).

RX and TX flow control are enabled by default. When priority flow control is enabled in the adapter, normal flow control cannot be enabled.

Refer to the switch or router documentation to determine how link level flow control can be configured on the switch or router to which the adapter port is connected.

NOTE

In multichannel configurations where multiple PCI functions are exposed for a single 10GbE port, the flow control parameter for a port can be configured through any interface associated with the physical port, and the configured property will apply to all interfaces associated with the port.

RX Frame Coalescing

The Ethernet driver coalesces regular-sized TCP segments to a large frame before passing it to the network stack, which might improve TCP receive performance. RX frame coalescing is implemented using the GRO mechanism (in Linux driver versions that support GRO) or the LRO mechanism (in older Linux driver versions). RX frame coalescing is enabled by default. In some configurations where the end point for the TCP connection to which the packets belong is not in the current server (for example, the end point is a router), RX coalescing should not be enabled.

GRO can be disabled using the –K option with the ethtool command: ethtool –K eth<

N

> gro off LRO can be disabled using the –C option with the ethtool command: ethtool -C eth<

N

> rx-frames 1 where eth <

N

> is the name of the Ethernet device you are working on (for example, eth0 ).

Maximum Transmission Unit (MTU)

The Ethernet driver supports MTUs between 256 bytes and 9000 bytes. The default MTU is set to 1500. If other elements in the network path support a larger MTU, you can increase the MTU up to 9000 using the ifconfig command. To do this run: ifconfig eth<

N

> mtu 9000 where eth <

N

> is the name of the Ethernet device you are working on (for example, eth0).

Interrupt Coalescing

The Ethernet driver tries to reduce the number of interrupts by delaying the interrupts from the adapter, which reduces CPU utilization during a high traffic rate. The interrupt delay duration can be set to change dynamically within a range of values, depending on the receive rate (known as Adaptive Interrupt Coalescing (AIC)), or can be set to a constant value. Broadcom - 41 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Network Performance Tuning

3.5.7.1

3.5.7.2

3.5.8

3.5.9

Setting the Interrupt Delay Duration to a Range of Values (AIC)

For receive interrupts, AIC is enabled by default. When AIC is enabled, the default low limit is 0 microseconds and the default high limit is 96 microseconds. In low traffic, the interrupt delay is set to 0 for best latency. As the number of interrupts per second increases, the delay is increased to higher values proportional to the receive rate, up to the default high limit of 96 microseconds. You can change the low and high limits using ethtool . For example, to set a low limit of 8 and a high limit of 40, run: ethtool -C eth<

N

> rx-usecs-low 8 ethtool -C eth<

N

> rx-usecs-high 40 where eth <

N

> is the name of the Ethernet device you are working on (for example, eth0 ).

For transmit interrupts, AIC is not supported.

To disable AIC and set the interrupt delay duration to a constant value, see the following section.

Setting the Interrupt Delay Duration to a Constant Value

The interrupt delay duration can be set to a constant value for both receive and transmit interrupts. The possible interrupt delay duration values are 0 to 96 microseconds, in 8 microsecond increments.

For receive interrupts, disable AIC (because it is enabled by default) and set the interrupt delay duration using ethtool . For example, to disable AIC and set the constant RX interrupt delay to 8-microseconds, run: ethtool -C eth<

N

> adaptive-rx off rx-usec 8 where eth <

N

> is the number of the Ethernet interface you are working on.

If your application requires low or predictive latency, turn off AIC and set rx-usecs to 0.

For transmit interrupts, the default interrupt delay duration is 96 microseconds. You can change this value using ethtool . For example, to set the transmit interrupt delay to 64 microseconds, run: ethtool -C eth<

N

> tx-usec 64 where eth <

N

> is the number of the Ethernet interface on which you are working.

Receive-Side Scaling (RSS)

Distributing the incoming traffic across several receive rings with separate interrupt vectors helps to distribute the receive processing across several CPU cores. This distribution could reduce the packet drop and improve the packet rate in certain applications. RSS is enabled in non-SR-IOV and non-multichannel configurations. In multichannel configurations, RSS is enabled in the first section of each port.

Analyzing Performance Issues

MSI-x interrupts are required for RSS to work. If your motherboard and operating system version support MSI-X, the Ethernet driver automatically uses MSI-X interrupts. If not enough MSI-X vectors are available, the Ethernet driver uses INTx interrupts, which might decrease performance. The proc node /proc/interrupts shows the interrupts and their types.

The Linux performance

top

utility can monitor the CPU utilization while troubleshooting performance issues. A low idle CPU percentage in any CPU core is an indication of excessive processing load for that CPU. The proc node /proc/interrupts shows the distribution of the interrupts across the CPU cores. If you see too many interrupts per second directed to one CPU, check to see if the irqbalance program is running. The irqbalance program is normally started at system boot. In some cases, you can get better performance by disabling irqbalance and manually distributing interrupts. You can manually distribute the interrupt load across the available CPU cores by setting the CPU affinity for any interrupt vector by setting the mask in the proc node /proc/irq/<

int-vector

>/smp_affinity .

Broadcom - 42 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 3: Configuration Network Performance Tuning Use the netstat command to look for excessive TCP retransmits or packet drops in the network stack.

In systems having more than one NUMA node, you can get better performance by pinning interrupts to the NUMA node local to the PCIe device.

Use the –S option of ethtool to see all statistics counters maintained by the Ethernet and driver. Excessive drop or error counters are an indication of a bad link or defective hardware. See

Table 11, Ethtool -S Option Statistics , and

Table 12, Transmit/Receive Queue Statistics .

Turning off auditing and SELinux can improve CPU utilization and, in some cases, increase throughput. You can disable auditing by appending audit=0 in the boot command line. You can turn off SELinux by specifying selinux=0 in the boot command line. For example, the following command boots the Linux kernel with the SELinux and auditing options disabled: kernel /boot/vmlinux-2.6.18 ro root=/dev/md0 selinux=0 audit=0 You can get better CPU utilization and, in some cases, better throughput, by disabling kernel debug options, such as CONFIG_DEBUG_SLAB . This situation requires you to build the kernel image and modules. Turning off the firewall and disabling Hyper-Threading can also improve performance.

Broadcom - 43 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Hardware Situations and Resolutions

Chapter 4: Troubleshooting

This section explains some of the situations in which your system can operate in an unexpected manner and some possible resolutions.

4.1

Hardware Situations and Resolutions

Table 6

lists the hardware situations and their resolutions.

Table 6 Hardware Situations and their Resolutions Situation Resolution

An unapproved optical transceiver is used to connect the adapter.

The system log can generate one or more of these events if an unapproved optical transceiver is detected:  Unsupported module  Optics faulted/incorrectly installed/not installed   Incompatible optics Unknown port SFB status To resolve this issue, do the following: 1. 2. 3. Power off the system.

Replace the unapproved optical transceiver with an approved on.

Power on the system.

NOTE

For more information on replacing an optical transceiver, refer to the hardware guide of the adapter available at the Documents and Downloads area of http://www.broadcom.com

.

4.2

FC and FCoE Driver Situations and their Resolutions

Table 7

lists the FC and FCoE driver situations and their resolutions.

Table 7 FC and FCoE Driver Situations and their Resolutions Situation Resolution

FC link fails to come up.

If an FC link fails to come up, verify that an 8Gb/s or 16Gb/s adapter is not attempting to connect to a 1Gb/s device. Only 2Gb/s, 4Gb/s, and 8Gb/s devices are supported on 8Gb/s adapters. Only 2Gb/s, 4Gb/s, 8Gb/s, and 16Gb/s devices are supported on 16Gb/s adapters.

“Authentication is enabled but authentication service is not running.” error message If you see this message in /var/log/messages and the adapter is in an error state, the fcauthd daemon probably is not running. To determine whether fcauthd is running, run: /etc/init.d/fcauthd status.

To start fcauthd, run /etc/init.d/fcauthd start.

If a SAN configuration has 256 targets mapped by the FC and FCoE driver, any additional added targets do not get a target ID mapping by the driver and cause target discovery to fail.

Removing targets or reinitializing the link does not solve this issue.

Unload and reload the driver to reset available target IDs. Ensure that the SAN configuration is correct prior to rebooting the driver. This action clears the driver consistent binding table and frees target IDs for new target nodes.

Broadcom - 44 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting FC and FCoE Driver Situations and their Resolutions

Table 7 FC and FCoE Driver Situations and their Resolutions (Continued) Situation Resolution

rmmod fails to unload FC and FCoE driver module due to “ERROR: Module lpfc is in use.” This message can appear when you attempt to remove the driver and a Logical Volume Group is dependent on the driver. If you have configured boot from a SAN, you must reboot the system. Otherwise, use these steps to resolve this situation: 1. Make the Logical Volume Group unavailable. Type: lvchange -a n

xxxxxxx

2. The

xxxxxxx

parameter is the Volume Group Name.

Stop the Emulex OneCommand Manager application.

rmmod hangs and module reference count is 0.

Due to a small race condition in the kernel, it is possible for an rmmod command to hang. Issue the rmmod -w command. If this does not help, reboot the computer.

rmmod fails to unload driver due to device or resource busy.

This message occurs when you attempt to remove the driver without first stopping the Emulex OneCommand Manager application or the fcauthd daemon when the Emulex OneCommand Manager application is installed and running, or when FC disks connected to an Emulex LightPulse adapter are mounted. To resolve this situation: 1. Stop the Emulex OneCommand Manager application before attempting to unload the driver. The script is located in the ./stop_ocmanager /usr/sbin/ocmanager directory. Type: 2. 3. Unmount any disks connected to the adapter. Unload the driver. Type: rmmod lpfc An lspci shows recent Emulex adapters as unknown.

Slow targets or extended link faults on the storage side might result in storage being marked offline by the mid-level and remaining offline (not recovered) when the link faults are corrected.

This situation occurs because of the delay of getting new product IDs into the Linux development cycle.

There is no resolution at this time.

This version of the driver should eliminate this issue. However, if you experience offline device issues, increase the SCSI command timeout to a value greater than or equal to 60 seconds. Emulex also provides a script that addresses this issue. To access the lun_change_state.sh script , go to the Documents and Downloads area of http://www.broadcom.com

.

Under certain conditions of an I/O load, some targets cannot complete an I/O issued by a Linux initiator within the default timeout of 30 seconds given by the SCSI mid-level.

If the situation is not corrected, the initiator-to-target condition deteriorates into abort/recovery storms, leading to I/O failures in the block layer. These types of failures are preceded by a SCSI I/O error of hex 6000000.

Emulex provides a script that addresses this issue. To access the set_target_timeout.sh script , go to the Documents and Downloads area of http://www.broadcom.com

.

The FC and FCoE driver fails to recognize an adapter and logs “unknown IOCB” messages in the system log during driver load. The adapter is running outdated firmware.

Install the latest firmware on the adapter.

NOTE

Before performing a firmware update, driver installation is required. For more information on installing the driver, see

Section 2.2.1, Installing the Binary RPM FC and FCoE Driver Kit .

Loading the FC and FCoE driver on SLES 11 SPx reports “unsupported module, tainting kernel” in system log.

This message is logged by the kernel whenever a module that is not shipped with the kernel is loaded. Ignore this message.

The system panics when it is booted with a failed adapter installed.

Remove the failed adapter and reboot the system.

Broadcom - 45 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting FC and FCoE Driver Situations and their Resolutions

Table 7 FC and FCoE Driver Situations and their Resolutions (Continued) Situation Resolution

Unloading the FC and FCoE driver on SLES 11 SPx might cause a message to be logged in the system log, such as the following: umount: /dev/disk/bypath/pci-0000:02:04.0-scs i-0:0:1:0: not mounted These messages are normal output from the SLES 11 SPx hotplug scripts. Ignore them.

Driver installation fails. The lpfc-install script fails to install the driver. The install script may fail for the following reasons:  A previous version of the driver is installed. Run the lpfc-install --uninstall script and then try to install the driver.

  The current driver is already installed.

Run a supported RHEL or SLES kernel.

“No module lpfc found for kernel KERNELVERSION” RPM error message when upgrading the kernel.

A recently upgraded kernel cannot find the ramdisk. After upgrading the kernel, the kernel cannot find the ramdisk, which halts or panics the system.

These three situations can be resolved by upgrading the kernel. There are two ways to install the driver into an upgraded kernel. The method you use depends on whether you are updating the driver.

  Upgrade the kernel using the same version of the driver.

Upgrade the kernel using a new version of the driver.

The driver is not loaded after a system reboot after upgrading the kernel.

For more information on upgrading the kernel, see

Chapter 2

,

Installing and Uninstalling

.

Driver uninstall fails.    The lpfc-install --uninstall script fails with an error.

Try the following solutions:  Uninstall the Emulex OneCommand Manager application; refer to the

OneCommand Manager Application for LightPulse Adapters User Guide

instructions.

for Unmount all FC disk drives.

Unload the lpfcdfc and FC and FCoE driver.

Use rpm -e lpfcdriver and -e ocmanager and uninstall the new kits.

Broadcom - 46 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Ethernet Driver Situations and their Resolutions

Table 7 FC and FCoE Driver Situations and their Resolutions (Continued) Situation Resolution

lpfc-install script exit code. The Emulex driver for Linux does not load in ramdisk for a custom-built kernel.

The lpfc-install script contains exit codes that can be useful in diagnosing installation issues. Refer to the lpfc-install script for a complete listing of codes and definitions.

Custom built kernels are not supported by Emulex. However, the Emulex install script attempts to install the driver into a ramdisk that follows the naming scheme used by RHEL or SLES kernels.

 The RHEL naming scheme for ramdisk images is:  /boot/initrd-

KERNELVERSION

.img.

The SLES naming scheme for ramdisk images is: /boot/initrd.

If a custom built kernel has a ramdisk image that does not follow the appropriate naming scheme, change the name of the image using the following procedure: 1. Change the name of the ramdisk image to match the SLES naming scheme.

2. 3. 4. 5. Update any file links to the ramdisk image.

Edit the boot loader configuration file (for example, /etc/lilo.conf

, /etc/yaboot.conf

name.

, /boot/grub/grub.conf

, /boot/grub/menu.lst

), find any references to the old ramdisk image name, and replace them with the new Reboot the system to verify the changes.

Install the Emulex Linux driver kit.

The Linux SCSI subsystem sees only eight LUNs when more are present.

Some SCSI drivers do not scan past eight LUNs when the target reports itself as a SCSI-2 device.

To resolve this situation, force a SCSI bus scan with the following command: /usr/sbin/lpfc/lun_scan.

SUSE supplies a /bin/rescan-scsi-bus.sh

script, which can be changed to scan everything.

4.3

Ethernet Driver Situations and their Resolutions

The following table lists the Ethernet driver situations and their resolutions. This section applies to LPe16202/OCe15100 adapters in NIC mode only.

Table 8 Ethernet Driver Situations and their Resolutions Situation

The ethtool configuration settings are not restored after system reboot.

The Ethernet driver works but the transmit and receive data rates are not near a 10Gb/s line rate.

When MILI and SNMP daemons start, they trigger warning messages within SELinux for certain operations.

Resolution

The ethtool settings are not designed to persist across reboot. For persistence, invoke configuration commands from a boot script that is executed at system start, such as /etc/rc.local

.

There could be several reasons for poor performance. For best performance practices,

see Section 3.5, Network Performance Tuning

.

This issue is known and no solution is available. However, to avoid SELinux warning messages, you can disable SELinux. To disable SELinux, open a terminal and enter the following command at the prompt: echo 0 > /selinux/enforce To enable SELinux, use the following command: echo 1 > /selinux/enforce

In addition, see Section 3.5.9, Analyzing Performance Issues .

Broadcom - 47 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

4.4

Log Messages 4.4.1

FC and FCoE Driver Log Messages

4.4.1.1

The following section describes retrieving and interpreting FC and FCoE driver log messages.

Retrieving FC and FCoE Driver Log Messages

LPFC error log messages are logged in the /var/log/messages file.

An example of an LPFC message: Jul 2 04:23:34 daffy kernel: lpfc 0000:03:06.0: 0:1305 Link Down Event x2f2 received Data: x2f2 x20 x110    In this example: lpfc 0000:03:06.0

– Identifies the PCI location of the particular LPFC hardware port.

0: – Indicates Emulex adapter 0 1305 – Indicates a log message number of 1305.

NOTE

  If Data: is present in a log message, any information following Data: is intended only for Broadcom® Technical Support or Engineering use.

If an error message instructs you to perform a firmware update, ensure that the driver is installed first. For more information on

installing the driver, see Chapter 2

,

Installing and Uninstalling .

4.4.1.2

LPFC Error Log Messages and their Descriptions

The following table lists the LPFC error log messages and their descriptions.

Table 9 LPFC Error Log Messages and their Descriptions

0111: Dropping received ELS cmd The driver decided to drop an ELS Response ring entry.

Data: (1) ulpStatus (2) ulpWord[4] (3) ulpTimeout Severity: Log: Action: Error Always This error could indicate a software driver or firmware issue. If this issue persists, report these errors to Broadcom Technical Support.

0113: An FLOGI ELS command was received from DID in Loop Mode While in Loop Mode an unknown or unsupported ELS command was received.

Data: None Severity: Log: Action: Error Always Check device DID.

0115: Unknown ELS command received from NPORT Received an unsupported ELS command from a remote N_Port.

Data: None Severity: Log: Action: Error Always Check the remote N_Port for a potential issue.

Broadcom - 48 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0125: FDISC Failed (x%x). Fabric out of resources The fabric rejected an FDISC because the switch cannot support additional virtual ports.

Data: lsRjtError Severity: Log: Action: Error Always Reconfigure the switch to support more NPIV logins. If this issue persists, contact Broadcom Technical Support.

0126: FDISC failed ulpStatus ulpWord4 Data: Severity: Log: Action: lsRjtError Error Always Reconfigure the switch to support more NPIV logins. If this issue persists, contact Broadcom Technical Support.

0127: ELS timeout An ELS IOCB command was posted to a ring and did not complete within ULP timeout seconds.

Data: Severity: Log: Action: (1) elscmd (2) remote_id (3) ulpcommand (4) ulpIoTag Error Always If no ELS command is going through the adapter, reboot the system. If the issue persists, contact Broadcom Technical Support.

0133: PLOGI: no memory for reg_login Memory allocation error.

Data: (1) nlp_DID (2) nlp_state (3) nlp_flag (4) nlp_rpi Severity: Log: Action: Error LOG_ELS Memory allocation error. Check system resources. Unload unused modules.

0134: PLOGI cannot issue reg_login The ELS PLOGI mailbox command has failed.

Data: (1) nlp_DID (2) nlp_state (3) nlp_flag (4) nlp_rpi Severity: Log: Action: Error LOG_ELS Check the port and switch configuration.

0135: cannot format reg_login Could not allocate an RPI or DMA buffer for the mailbox command.

Data: (1) nlp_DID (2) nlp_state (3) nlp_flag (4) nlp_rpi Severity: Log: Action: Error LOG_ELS None required.

0136: PLOGI completes to NPort completion A PLOGI has completed for which there is no NDLP.

Data: (1) ulpStatus (2) ulpWord[4] Severity: Log: Action: Error LOG_ELS None required.

0137: No retry ELS command to remote Data: Severity: Log: Action: (1) ulpStatus (2) ulpWord[4] Error LOG_ELS None required.

Broadcom - 49 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0138: ELS rsp: Cannot issue reg_login for REG_LOGIN mailbox command failed.

Data: (1) nlp_DID (2) nlp_state (3) nlp_flag (4) nlp_rpi Severity: Log: Action: Error LOG_ELS None required.

0139: Ignoring ELS cmd tag completion Data This ELS command was aborted.

Data: (1) ulpStatus (2) ulpWord[4] (3) ulpTimeout Severity: Log: Action: Error LOG_ELS None required.

0140: PLOGI Reject: invalid name Invalid node WWN provided.

Data: None Severity: Log: Action: Error LOG_ELS None required.

0141: PLOGI Reject: invalid pname Invalid port WWN provided.

Data: None Severity: Log: Action: Error LOG_ELS None required. 0142: PLOGI RSP: Invalid WWN The PLOGI sent to the port by a remote port had an invalid WWN. Data: None Severity: Log: Action: Error LOG_ELS None required.

0143: SLI4 Adapter Hardware Error Data: / The HBA has encountered an unrecoverable error.

Data: None Severity: Log: Action: Error LOG_INIT Use hbacmd to retrieve a dump file.

0144: Not a valid WCQE code: The completion queue handler detected an invalid type.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0147: Failed to allocate memory for RSCN event Memory could not be allocated to send the RSCN event to the management application.

Data: None Severity: Log: Action: Error LOG_ELS None required.

Broadcom - 50 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0148: Failed to allocate memory for LOGO event Memory could not be allocated to send the LOGO event to the FC transport.

Data: None Severity: Log: Action: Error LOG_ELS None required.

0149: Failed to allocate memory for ELS event Memory could not be allocated to send the ELS event to the FC transport.

Data: None Severity: Log: Action: Error LOG_ELS None required.

0154: Authentication not complete Authentication was restarted because the previous authentication did not complete.

Data: None Severity: Log: Action: Error LOG_DISCOVERY Check the switch configuration.

0200: CONFIG_LINK bad hba state A CONFIG_LINK mailbox command completed and the driver was not in the right state.

Data: None Severity: Log: Action: Error Always Software driver error. If this issue persists, report these errors to Broadcom Technical Support.

0203: Devloss timeout on WWPN

NPort A remote N_Port that was discovered by the driver disappeared for more than lpfc_devloss_tmo seconds.

Data: (1) nlp_flag (2) nlp_state (3) nlp_rpi Severity: Log: Action: Error Always If the device generating this message is not a target to which the HBA is connected, this error will not affect the data integrity of the I/O between the HBA and the attached storage and can be ignored.

0206: Device discovery completion error This indicates that an uncorrectable error was encountered during device discovery after a link up. FC devices will not be accessible if this message is displayed.

Data: Severity: None Error Log: Action: Always Reboot the system. If this issue persists, report the error to Broadcom Technical Support. Run with verbose mode enabled for more information.

0207: Device () sent invalid service parameters. Ignoring device.

Invalid service parameters were received from DID. This remote port is ignored.

Data: DID, WWN Severity: Log: Action: Error Always Verify the remote port's configuration. If this issue persists, report the error to Broadcom Technical Support. Run with verbose mode on for more details.

Broadcom - 51 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0217: Block sgl registration required DMAsize great than a page The request to post SGL pages does not fit on a page.

Data: None Severity: Warning Log: LOG_INIT 0221: FAN timeout A link up event was received without the login bit set, so the driver waits E_D_TOV for the fabric to send a FAN. If no FAN if received, a FLOGI will be sent after the timeout. Data: None Severity: Warning Action: None required. The driver recovers from this condition by issuing a FLOGI to the fabric.

0222: Initial FLOG/FDISKI timeout The driver sent the initial FLOGI or FDISK to the fabric and never got a response back.

Data: None Severity: Log: Action: Error Always Check fabric configuration. The driver recovers from this situation and continues with device discovery.

0223: Timeout while waiting for NameServer login Our login request to the NameServer was not acknowledged within R_A_TOV.

Data: None Severity: Log: Action: Error Always Check the fabric configuration. The driver recovers from this and continues with device discovery.

0224: NameServer Query timeout Node authentication timeout, node Discovery timeout. A NameServer Query to the fabric or discovery of reported remote N_Ports is not acknowledged within R_A_TOV. Data: Severity: (1) fc_ns_retry (2) fc_max_ns_retry Error Log: Action: Always Check fabric configuration. The driver recovers from this situation and continues with device discovery.

0226: Device discovery completion error This indicates that an uncorrectable error was encountered during device discovery after a link up. FC devices will not be accessible if this message is displayed.

Data: Severity: None Error Log: Action: Always Reboot the system. If this issue persists, report the error to Broadcom Technical Support. Run with verbose mode on for more details.

0227: Node Authentication timeout The driver has lost track of what N_Ports are being authenticated.

Data: None Severity: Log: Action: Error Always None required. The driver should recover from this event.

Broadcom - 52 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0228: CLEAR LA timeout The driver issued a CLEAR_LA that never completed.

Data: None Severity: Log: Action: Error Always None required. The driver should recover from this event.

0230: Unexpected timeout, hba linkstate Discovery has timed out, and the HBA state is not ready.

Data: None Severity: Log: Action: Error LOG_DISCOVERY None required.

0231: RSCN timeout The driver has lost track of which N_Ports have RSCNs pending.

Data: (1) fc_ns_retry (2) lpfc_max_ns_retry Severity: Log: Action: Error Always None required. The driver should recover from this event.

0233: Nodelist not empty Driver unloaded or hotplug detected a node still in use.

Data: None Severity: Log: Action: Error LOG_DISCOVERY None required.

0237: Pending Link Event during Discovery: State Received link event during discovery. Causes discovery restart.

Data: None Severity: Warning Action: None required, unless this issue persists. If persistent, check cabling.

0241: NameServer rsp error The driver received a NameServer response containing a status error.

Data: (1) CommandResponse.bits.CmdRsp (2) ReasonCode (3) Explanation (4) fc_flag Severity: Log: Action: Error LOG_DISCOVERY verbose Check the fabric configuration. The driver recovers from this situation and continues with device discovery.

0246: RegLogin failed The firmware returned a failure for the specified RegLogin.

Data: (1) Did (2) mbxStatus (3) hbaState Severity: Log: Action: Error Always This message indicates that the firmware could not do RegLogin for the specified DID. There might be a limitation on how many nodes an HBA can see. 0249: Cannot issue Register Fabric login: Err %d\ Could not issue the fabric register login, the error value is unique for each possible failure.

Data: None Severity: Log: Action: Error LOG_ELS None required.

Broadcom - 53 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0251: NameServer login: no memory Could not allocate memory for the NDLP structure.

Data: None Severity: Log: Action: Error LOG_ELS None required.

0252: Cannot issue NameServer login Could not issue an ELS PLOGI to the NameServer DID.

Data: None Severity: Log: Action: Error LOG_ELS Check the port connection and the switch configuration.

0253: Register VPI: Can’t send mbox\ Could not issue the REG_VPI mailbox command for this VPort.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

0254: Register VPI: no memory” goto mbox_err_exit Could not allocate memory for the REG_VPI mailbox command.

Data: None Severity: Log: Action: Error LOG_MBOX None required. 0255: Issue FDISC: no IOCB All of the preallocated IOCBs are in use.

Data: None Severity: Log: Action: Error LOG_ELS None required.

0256: Issue FDISC: Cannot send IOCB\ Unable to send the fabric IOCB.

Data: None Severity: Log: Action: Error LOG_ELS None required.

0257: GID_FT Query error The GID_FT CT request for the NameServer has failed.

Data: None Severity: Log: Action: Error LOG_ELS Check the switch configuration.

0258: Register Fabric login error: The REG_LOGIN for the fabric has failed. Data: None Severity: Log: Action: Error LOG_MBOX Check the port connection and the switch configuration.

Broadcom - 54 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0259: No NPIV Fabric support The switch to which the port is connected does not support NPIV.

Data: None Severity: Log: Action: Error LOG_ELS Check the switch configuration.

0260: Register NameServer error: The REG_LOGIN mailbox command has failed for the NameServer.

Data: None Severity: Log: Action: Error LOG_ELS Check the switch configuration 0261: Cannot register NameServer login: Either a memory allocation issue or an invalid parameter was sent to the REG_LOGIN.

Data: None Severity: Log: Action: Error LOG_ELS At least one message (0142 0121 0133 0134 0135) should precede this message.

0262: No NPIV Fabric support The switch to which the port is connected does not support NPIV.

Data: None Severity: Log: Action: Error LOG_ELS Check the switch configuration.

0263: Discovery Mailbox error: state: Either the driver could not allocate resources or it could not send sparam_mbox or cfglink_mbox.

Data: (1) address of sparam_mbox command (2) address of cfglink_mbox command Severity: Log: Action: Error LOG_MBOX Attempt to unload and reload the driver when it is convenient.

0264: No NPIV Fabric support The switch to which the port is connected does not support NPIV.

Data: None Severity: Log: Action: Error LOG_ELS Check the switch configuration.

0266: Issue NameServer Req err Data: The driver was unable to send the NameServer CT command.

Data: (1) vports fc_flag (2) vports fc_rscn_id_cnt Severity: Log: Action: Error LOG_DISCOVERY Check the port and switch configurations. 0267: NameServer GFF Rsp Error ( ) Data: The NameServer GFF CT request failed.

Data: (1) vports fc_flag (2) vports fc_rscn_id_cnt Severity: Log: Action: Error LOG_DISCOVERY Check the port and switch configurations. Broadcom - 55 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0268: NS cmd Error ( ) The NameServer CT request failed.

Data: None Severity: Log: Action: Error LOG_DISCOVERY Check the port and switch configurations. 0271: Illegal State Transition: node event , state Data: The current node state does not have a handler for this event.

Data: (1) nlp_rpi (2) nlp_flag Severity: Log: Action: Error LOG_DISCOVERY Verify that all targets are still visible to the SCSI mid-layer.

0272: Illegal State Transition: node event , state Data: The driver is completing a PLOGI but does not have the rcv_plogi flag set.

Data: (1) nlp_rpi (2) nlp_flag Severity: Log: Action: Error LOG_DISCOVERY Verify that all targets are still visible to the SCSI mid-layer.

0273: Unexpected discovery timeout, vport State x%x The discovery process has timed out.

Data: None Severity: Log: Action: Error LOG_DISCOVERY Verify that all targets are still visible.

0274: lpfc_nlp_put: ndlp:x%pusgmap:x%x refcnt:%d, void *)ndlp, ndlp->nlp_usg_map, atom ic_read(&ndlp->kref.refcount) Data: None Severity: Warning Log: LOG_NODE 0275: lpfc_nlp_put: ndlp:x%pusgmap:x%x refcnt:%d, void *)ndlp, ndlp->nlp_usg_map, atom ic_read(&ndlp->kref.refcount) A kref_put was called again after the node was already inactive.

Data: None Severity: Warning Log: LOG_NODE 0276: lpfc_nlp_get: ndlp:x%pusgmap:x%x refcnt:%d, void *)ndlp, ndlp->nlp_usg_map, atom ic_read(&ndlp->kref.refcount) A kref_get was attempted on a node that was being released.

Data: None Severity: Warning Log: LOG_NODE Broadcom - 56 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0277: lpfc_enable_node: ndlp:x%pusgmap:x%x refcnt:%d, void *)ndlp, ndlp->nlp_usg_map, atom ic_read(&ndlp->kref.refcount) Enable node was attempted on an inactive node.

Data: None Severity: Warning Log: LOG_NODE 0278: lpfc_enable_node: ndlp:x%pusgmap:x%x refcnt:%d, void *)ndlp, ndlp->nlp_usg_map, atom ic_read(&ndlp->kref.refcount) Enable node was attempted on an inactive node.

Data: None Severity: Warning Log: LOG_NODE 0280: lpfc_cleanup_node: ndlp:x%pusgmap:x%x refcnt:%d, void *)ndlp, ndlp->nlp_usg_map, atom ic_read(&ndlp->kref.refcount) Node clean-up was attempted on a node that has already been marked for memory free.

Data: None Severity: Warning Log: LOG_NODE 0281: lpfc_cleanup_node: ndlp:x%pusgmap:x%x refcnt:%d, void *)ndlp, ndlp->nlp_usg_map, atom ic_read(&ndlp->kref.refcount) Node clean-up was called to prepare the node for release.

Data: None Severity: Warning Log: LOG_NODE 0282: ldid:x%x ndlp:x%pusgmap:x%x refcnt:%d, ndlp->nlp_DID, (void *)ndlp, lpfc_init.c-ndlp->nlp_us g_map, Driver clean-up has found a node that is still on the node list during driver unload or PCI hot plug removal.

Data: None Severity: Log: Action: Error LOG_NODE None required. 0283: Failed to allocate mbox cmd memory Mailbox allocation error.

Data: None Severity: Log: Action: Error LOG_INIT None required. 0285: Allocated DMA memory size is less than the requested DMA memorysize Memory allocation was truncated. Data: None Severity: Log: Action: Error LOG_INIT None required. Broadcom - 57 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0286: lpfc_nlp_state_cleanup failed to allocate statistical data buffer Memory allocation failed for node’s statistical data.

Data: None Severity: Log: Action: Error LOG_INIT None required. 0287: lpfc_alloc_bucket failed to allocate statistical data buffer DID Memory allocation failed for node’s statistical data.

Data: None Severity: Log: Action: Error LOG_NODE None required. 0288: Unknown FCoE event type event tag The firmware has detected an unknown FCoE event.

Data: None Severity: Log: Action: Error LOG_SLI Check the FCoE switch configuration and the HBA DCBX mode.

0289: Issue Register VFI failed: Err The driver could not register the Virtual Fabric Index for the FCFI.

Data: None Severity: Log: Action: Error LOG_ELS Check the switch and port configurations.

0290: The SLI4 DCBX asynchronous event is not handled yet The SLI-4 DCBX asynchronous event is not handled yet.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0291: Allocated DMA memory size (x%x) is less than the requested DMA memory size (x%x) The asynchronous DCBX events are not handled in the driver.

Data: None Severity: Log: Action: Error LOG_INIT Check the switch configuration.

0293: PM resume failed to start worker thread: error= The PCI resume (hot plug) could not start the worker thread for the driver.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver.

0294: PM resume Failed to enable interrupt The PCI resume (hot plug) could not get an interrupt vector.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver.

Broadcom - 58 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0297: Invalid device group While unloading the driver, the driver detected a PCI device that it should not have claimed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0299: Invalid SLI revision While processing a host attention or unrecoverable error, the driver detected an invalid SLI revision.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0300: LATT: Cannot issue READ_LA: Data: The link attention handler could not issue a READ_LA mailbox command.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

0301: READ_SPARAM: no buffers The driver attempted to issue a READ_SPARAM mailbox command to the adapter, but there were no buffers available.

Data: None Severity: Warning Action: This message indicates: (1) Kernel virtual memory is depleted. Check that the system meets minimum RAM requirements for the adapter. Try closing other applications to free some memory. (2) A possible driver buffer management issue. If this issue persists, report the error to Broadcom Technical Support.

0302: REG_LOGIN: no buffers The driver attempted to issue a REG_LOGIN mailbox command to the adapter, but no buffers were available.

Data: (1) Did, (2) flag Severity: Warning Action: This message indicates: (1) Kernel virtual memory is depleted. Check that the system meets minimum RAM requirements for the adapter. Try closing other applications to free some memory. (2) A possible driver buffer management issue. If this issue persists, report the error to Broadcom Technical Support.

0313: Ring handler: unexpected Rctl Type received The RCTL/Type of a received frame did not match any for the configured masks for the specified ring. Data: None Severity: Warning Action: This error could indicate a software driver, firmware, or hardware issue. Report these errors to Broadcom Technical Support.

0303: Ring handler: portRspPut is bigger then rsp ring The port rsp ring put index is larger than the size of the rsp ring.

Data: None Severity: Log: Action: Error Always This error could indicate a software driver, firmware, or hardware issue. Report these errors to Broadcom Technical Support.

Broadcom - 59 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0304: Stray mailbox interrupt, mbxCommand mbxStatus Received a mailbox completion interrupt and there are no outstanding mailbox commands.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0306: CONFIG_LINK mbxStatus error HBA state The driver issued a CONFIG_LINK mailbox command to the HBA that failed.

Data: None Severity: Log: Action: Error Always This error could indicate a firmware or hardware issue. Report these errors to Broadcom Technical Support.

0310: Mailbox command timeout A mailbox command was posted to the adapter and did not complete within 30 seconds.

Data: (1) hba_state (2) sli_flag (3) mbox_active Severity: Log: Action: Error Always This error could indicate a software driver or firmware issue. If no I/O is going through the adapter, reboot the system. If this issue persists, report the error to Broadcom Technical Support.

0312: Ring handler: portRspPut is bigger then rsp ring The IOCB command rings put pointer is ahead of the get pointer.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0313: Ring handler: unexpected Rctl Type received The RCTL/Type of a received frame did not match any for the configured masks for the specified ring. Data: None Severity: Log: Action: Warning LOG_SLI verbose This error could indicate a software driver, firmware or hardware issue. Report these errors to Broadcom Technical Support.

0315: Ring issue: portCmdGet is bigger then cmd ring The port command ring get index is greater than the size of the command ring.

Data: None Severity: Log: Action: Error Always This error could indicate a software driver, firmware or hardware issue. Report these errors to Broadcom Technical Support.

0317: iotag is out of range: max iotag wd0 The IoTag in the completed IOCB is out of range.

Data: None Severity: Log: Action: Error Always This error could indicate a software driver, firmware or hardware issue. Report these errors to Broadcom Technical Support.

Broadcom - 60 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0318: Failed to allocate IOTAG. last IOTAG is The driver cannot allocate an IoTag. Display the last value used.

Data: None Severity: Log: Action: Error Always This message indicates the adapter HBA I/O queue is full. Typically this happens when heavy I/O is running on a low-end (3-digit) adapter. Upgrade to a higher-end adapter.

0319: READ_SPARAM mbxStatus error hba state The driver issued a READ_SPARAM mailbox command to the HBA that failed.

Data: None Severity: Log: Action: Error Always This error could indicate a firmware or hardware issue. Report these errors to Broadcom Technical Support.

0320: CLEAR_LA mbxStatus error hba state The driver issued a CLEAR_LA mailbox command to the HBA that failed.

Data: None Severity: Log: Action: Error Always This error could indicate a firmware or hardware issue. Report these errors to Broadcom Technical Support.

0322: Ring handler: unexpected completion IoTag The driver could not find a matching command for the completion received on the specified ring. Data: (1) ulpStatus, (2) ulpWord[4], (3) ulpCommand, (4) ulpContext Severity: Warning Action: This error could indicate a software driver or firmware issue. If this issue persists, report these errors to Broadcom Technical Support.

0323: Unknown Mailbox command Cmpl A unknown mailbox command completed.

Data: None Severity: Log: Action: Error Always This error could indicate a software driver, firmware, or hardware issue. Report these errors to Broadcom Technical Support.

0324: Config port initialization error, mbxCmd READ_NVPARM, mbxStatus READ_NVPARMS mailbox command failed during port configuration.

Data: None Severity: Log: Action: Error Always This error could indicate a software driver, firmware, or hardware issue. Report these errors to Broadcom Technical Support.

0328: Rsp Ring error: IOCB Data: The firmware has returned an error for this IOCB.

Data: (1) , (2) Severity: Log: Action: Warning LOG_SLI None required.

Broadcom - 61 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0330: IOCB wake NOT set The completion handler associated with the IOCB was never called.

Data: (1) timeout (2) timeleft/jiffies Severity: Log: Action: Error Always This error could indicate a software driver, firmware, or hardware issue. If this issue persists, report the error to Broadcom Technical Support.

0334: Unknown IOCB command Received an unknown IOCB command completion.

Data: (1) type (2) ulpCommand (3) ulpStatus (4) ulpIoTag (5) ulpContext) Severity: Log: Action: Error Always This error could indicate a software driver or firmware issue. If this issue persists, report these errors to Broadcom Technical Support.

0335: Unknown IOCB command Received an unknown IOCB command completion.

Data: (1) ulpCommand (2) ulpStatus (3) ulpIoTag (4) ulpContext) Severity: Log: Action: Error Always This error could indicate a software driver or firmware issue. If this issue persists, report these errors to Broadcom Technical Support 0336: Rsp Ring error: IOCB An IOCB error has occurred on the specified ring.

Data: (1) ulpWord[0], (2) ulpWord[1], (3) ulpWord[2], (4) ulpWord[3], (5) ulpWord[4], (6) ulpWord[5], (7) irsp+6, (8) irsp+7 Severity: Log: Action: Warning LOG_SLI verbose If this issue persists, check the targets. If the targets are not the problem, report the error to Broadcom Technical Support.

0340: Adapter temperature is OK now Adapter temperature has reverted to normal range.

Data: Temperature in Celsius Severity: Log: Action: Error LOG_TEMP verbose No action needed, informational.

0341: Ring Cannot find buffer for an unsolicited iocb tag No more preallocated buffers are available to handle unsolicited buffers.

Data: None Severity: Log: Action: Error LOG_SLI Ensure that this port is not being managed by multiple ports.

0342: Ring Cannot find buffer for an unsolicited iocb tag This is a multiple IOCB unsolicited command, and sufficient buffer space cannot be allocated for it.

Data: None Severity: Log: Action: Error LOG_SLI None required.

Broadcom - 62 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0343: Ring Cannot find buffer for an unsolicited iocb tag No more preallocated buffers are available to handle unsolicited buffers.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0344: Ring Cannot find buffer for an unsolicited iocb tag No more preallocated buffers are available to handle unsolicited buffers.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0345: Resetting board due to mailbox timeout iocb. tag 0x%x A mailbox command failed to complete. The driver is resetting the port.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI If the mailbox command fails again, set the lpfc_log_verbose to LOG_MBOX and retry.

0346: Ring handler: unexpected ASYNC_STATUS evt_code W0 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 The HBA received an asynchronous event that was not a temperature event.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0347: Adapter is very hot, please take corrective action Adapter temperature is above normal range.

Data: Temperature in Celsius Severity: Log: Action: Error LOG_TEMP verbose Shutdown and remove the HBA. Contact Broadcom Technical Support.

0348: NameServer login: node freed The enable mode failed to free up the NameServer login.

Data: None Severity: Log: Action: Error LOG_ELSI None required.

0349: rc should be MBX_SUCCESS The next mailbox command on the mailbox queue has failed.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

0350: rc should have been MBX_BUSY Attempting to unregister a default RPI from an interrupt context and the mailbox state is not busy.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

Broadcom - 63 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0351: Config MSI mailbox command failed, mbxCmd , mbxStatus The mailbox command sent to the firmware to configure the adapter to use MSI-X has failed.

Data: None Severity: Log: Action: Warning LOG_MBOX Ensure that the hardware platform supports MSI-X.

0352: Config MSI mailbox command failed, mbxCmd , mbxStatus The mailbox command sent to the firmware to configure the HBA to use MSI-X has failed.

Data: None Severity: Log: Action: Error LOG_MBOX Ensure that the hardware platform supports MSI-X.

0353: Active Mailbox cleared - mailbox timeout exiting The mailbox timeout handler has determined that the driver is in the process of completing this mailbox command.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

0357: MSI-X interrupt with no EQE SLI-4 adapter interrupt on the slow path but there is no associated EQE.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

0358: MSI-X interrupt with no EQE SLI-4 adapter interrupt on the fast path but there is no associated EQE.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

0359:Not a valid slow-path completion event: majorcode=x%x, minorcode=x%x\n , bf_get(lpfc_eqe_major_code, eqe), bf_get(lpfc_eqe_minor_code, eqe)); SLI-4: The EQE is invalid.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0360:Unsupported EQ count. Cannot create an event queue of this size.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0361:Unsupported CQ count. Cannot create an completion queue of this size.

Data: None Severity: Log: Action: Error LOG_SLI None required.

Broadcom - 64 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0362:Unsupported MQ count. Cannot create MQ of this size.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0364:Invalid param: SLI-4: The post-SGL function was passed an invalid XRI.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0365:Slow-path CQ identifier does not exist: The Completion Queue ID passed in the event queue entry does not reference a valid completion queue.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0366:Not a valid fast-path completion event: majorcode=, minor-code= The major or minor code in the Event Queue field is invalid.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0367: Fast-path completion queue does not exist The fast-path completion queue referenced by the CQID does not exist.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0368: Mis-matched fast-path completion queue identifier: eqcqid=%d, fcpcqid=%d The CQID in the event queue entry does not match the fcp_cqid that was passed into the routine. Data: None Severity: Log: Action: Error LOG_SLI None required.

0369: No entry from fast-path completion queue fcpcqid= No completions in the completion queue were referenced by fcp_cqid.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0370: Invalid completion queue type The event queue entry is not for a mailbox or a work queue entry.

Data: None Severity: Log: Action: Error LOG_SLI None required.

Broadcom - 65 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0371: No entry from the CQ: identifier , type No completion queue event existed for this event queue entry.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0372: iotag is out of range: max iotag () The IOCB lookup cannot be performed because the iocb_tag is out of range.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0373: FCP complete error: status= hw_status=, total_data_specified=, parameter=, word3= Logs the FCP failure. Status and parameter are equivalent to ulpStatus and ulpWord[4].

Data: None Severity: Log: Action: Warning LOG_SLI None required.

0374: FCP complete with no corresponding cmdiocb: iotag No IOCB was on the in-progress list that matched this iotag.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

0375: FCP cmdiocb not callback function iotag: The IOCB found for this iotag does not have a completion handler set in it.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

0377: Error parsing vpd. Using defaults.

Could not parse the VPD data, so the driver is using the default values.

Data: None Severity: Log: Action: Error Always None required.

0378: No support for fcpi mode.

Could not configure the port to run in FCP initiator mode.

Data: None Severity: Log: Action: Warning LOG_MBOX, LOG_SLI None required.

0379: Feature Mismatch Data: The features passed in to the driver as module parameters do not match what the firmware can do. Setting to default values.

Data: None Severity: Log: Action: Warning LOG_MBOX, LOG_SLI None required.

Broadcom - 66 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0381: Error %d during queue setup.

Could not set up all the queues that the driver requires to exchange I/Os with the HBA.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI Reload the driver.

0382: READ_SPARAM command failed status , mbxStatus The READ_SPARAM mailbox command has failed during initialization. The HBA has been set to error state.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI: Perform a dump with hbacmd and then try reloading the driver.

0383: Error during scsi sgl post operation The SGL entries could not be registered with the adapter.

Data: None Severity: Log: Action: Warning LOG_MBOX, LOG_SLI Reset the adapter using hbacmd.

0384: There is pending active mailbox cmd The mailbox commands have overlapped. This command should have been added to the mailbox queue.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

0385: rc should have been MBX_BUSY The completion handler for REG_LOGIN detected the IMMED_UNREG flag and tried to issue the UNREG_LOGIN command from an interrupt level. The mailbox status should still be busy.

Data: Severity: None Error Log: Action: LOG_MBOX, LOG_SLI None required.

0386: ELS complete with no corresponding cmdiocb: iotag The completion that the ISR is handling cannot find a tag associated with the IOTAG.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

0387:Failed to allocate an iocbq Failed to get an IOCBQ from the list of available IOCBQs.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0388:Not a valid WCQE code: x The event code is invalid. This event will be dropped.

Data: None Severity: Log: Action: Error LOG_SLI Ensure that the adapter firmware is current.

Broadcom - 67 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0391:Error during rpi post operation The driver was trying to post pages to the firmware to keep target login information and encountered a failure.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI Unload and reload the driver.

0393:Error during rpi post operation The driver was trying to post pages to the firmware to keep target login information and encountered a failure.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI Unload and reload the driver.

0394: Failed to allocate CQ_EVENT entry The asynchronous event handler was unable to allocate an event queue entry to which to transfer the asynchronous event.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI This could be a V-LINK clear from the switch or a fatal error from the firmware. Perform a dump from the Emulex OneCommand Manager application.

0395: The mboxq allocation failed The asynchronous link event handler could not allocate a mailbox command to issue the READ_LA (read link attention) mailbox command.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0396:The lpfc_dmabuf allocation failed The asynchronous link event handler could not allocate a mailbox command to issue the READ_LA mailbox command. Data: None Severity: Log: Action: Error LOG_SLI None required.

0397:The mbuf allocation failed The asynchronous link event handler could not allocate direct memory access for the READ_LA mailbox command.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0398 Invalid link fault code: The attempt to read the link attention register has returned an unknown value.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0399 Invalid link attention type: The READ_LA mailbox command has returned an invalid link type.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 68 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0401: Ignoring change to nodev_tmo because devloss_tmo is set Attempting to change the nodev timeout when the devloss has already been set.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0402:Cannot find virtual addr for buffer tag on ring A DMA buffer is unavailable for this unsolicited command.

Data: (1) tag (2) next (3) prev (4) postbufq_cnt Severity: Log: Action: Error LOG_INIT None required.

0404: lpfc_devloss_tmo attribute cannot be set to , allowed range is [, ] Attempt to set the devloss timeout value is outside the allowed range.

Data: None Severity: Log: Action: Error LOG_INIT Set the devloss timeout between the minimum and maximum devloss range.

0405: lpfc_link_speed attribute cannot be set to %d, allowed values are ["LPFC_LINK_SPEED_STRING"] Attempt to set the link speed value is outside the allowed range. Data: None Severity: Log: Action: Error LOG_INIT Set the link speed between 0 and the maximum.

0406: Adapter maximum temperature exceeded , taking this port offline The driver has received an error for the HBA indicating that the maximum allowable temperature has been exceeded.

Data: (1) work_hs (2) work_status[0] (3) work_status[1] Severity: Log: Action: Error LOG_INIT Ensure that the server fans are not blocked. Shut down the server if the airflow is restricted.

0407: Ignoring nodev_tmo module parameter because devloss_tmo is set.

Both module parameters (nodev and devloss) were set so the driver is ignoring the nodev parameter.

Data: None Severity: Log: Action: Error LOG_INIT Set only one of these parameters.

0410: Cannot find virtual addr for mapped buf on ring The driver cannot find the specified buffer in its mapping table. Therefore, it cannot find the virtual address needed to access the data.

Data: (1) phys (2) next (3) prev (4) postbufq_cnt Severity: Log: Action: Error Always This error could indicate a software driver or firmware issue. If this issue persists, report these errors to Broadcom Technical Support.

0421: MSI-X slow-path request_irq failed The kernel API to request an IRQ has failed.

Data: None Severity: Log: Action: Warning LOG_INIT Use module parameter lpfc_use_msi=0 (INTx).

Broadcom - 69 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0422: lpfc_restrict_login attribute cannot be set to , allowed range is [0, 1] Attempt to set the restrict login parameter to something other than on or off.

Data: None Severity: Log: Action: Error LOG_INIT Use 0 (Off ) or 1 (On) 0423: lpfc_"#attr" attribute cannot be set to %d, allowed range is ["#minval", "#maxval"] This compile time macro is used by several module parameters during initialization. Each module parameter has its own minimum and maximum values that are displayed.

Data: Severity: None Error Log: Action: LOG_INIT Set the module parameter between the minimum and maximum values. 0424:lpfc_"#attr" attribute cannot be set to %d, allowed range is ["#minval", "#maxval"] This is a compile time macro that is used by several module parameters to set the value.

Data: None Severity: Log: Action: Error LOG_INIT Set the module parameter between the minimum and maximum values. 0425:lpfc_restrict_login attribute cannot be set to %d, allowed range is [0, 1] The module parameter lpfc_restrict_login can only be set to 0 (off ) or 1 (on).

Data: None Severity: Log: Action: Error LOG_INIT Set lpfc_restrict_login=[0,1].

0426: Failed to enable interrupt The driver failed to start the interrupt.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver. 0427: Cannot re-enable interrupt after slot reset The driver was unable to enable the interrupt after an HBA reset.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver.

0429: MSI-X fast-path request_irq failed () The driver received an error for the request_irq_call.

Data: None Severity: Log: Action: Warning LOG_INIT Unload and reload the driver.

0430: PM resume Failed to enable interrupt The driver power management resume function could not enable the interrupt.

Data: None Severity: Log: Action: Error LOG_INIT Perform another PM suspend and resume or HBA reset.

Broadcom - 70 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0431: Failed to enable interrupt.

The driver failed to start the interrupt.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver.

0433: Wakeup on signal: rc= A signal other than the LPFC_DATA_READY was received on the worker thread.

Data: None Severity: Log: Action: Error LOG_ELS Unload and reload the driver.

0434: PM resume failed to start worker thread: error= The driver's power management resume function could not start the worker thread.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver.

0435: Adapter failed to get Option ROM version status The driver could not read the HBA option ROM.

Data: None Severity: Log: Action: Error LOG_INIT Reset the HBA. Ensure the adapter firmware is current.

0436: Adapter failed to init, timeout, status reg The adapter failed during power-up diagnostics after it was reset.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0437: Adapter failed to init, chipset, status reg The adapter failed during power-up diagnostics after it was reset.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0438: Adapter failed to init, chipset, status reg The adapter failed during power-up diagnostics after it was reset.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0439: Adapter failed to init, mbxCmd READ_REV, mbxStatus Adapter initialization failed when issuing a READ_REV mailbox command.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

Broadcom - 71 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0440: Adapter failed to init, READ_REV has missing revision information A firmware revision initialization error was detected.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. Install the latest firmware revision. If this issue persists, report the error to Broadcom Technical Support.

0442: Adapter failed to init, mbxCmd CONFIG_PORT, mbxStatus Adapter initialization failed when issuing a CONFIG_PORT mailbox command.

Data: (1) hbainit Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0446: Adapter failed to init, mbxCmd CFG_RING, mbxStatus , ring Adapter initialization failed when issuing a CFG_RING mailbox command.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0447: Adapter failed init, mbxCmd CONFIG_LINK mbxStatus Adapter initialization failed when issuing a CONFIG_LINK mailbox command.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0448: Adapter failed to init, mbxCmd READ_SPARM, mbxStatus Adapter initialization failed when issuing a READ_SPARM mailbox command.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0449: lpfc_%attr attribute cannot be initialized to %d, allowed range is [%min, %max] The sysfs attribute value written exceeds attribute range.

Data: (1) attribute name (2) value written (3) minimum value (3) maximum value Severity: Log: Action: Error Always Write a value within the supported range.

0450: lpfc_%attr attribute cannot be set to %d, allowed range is [%min, %max] The sysfs attribute value written exceeds attribute range.

Data: (1) attribute name (2) value written (3) minimum value (3) maximum value Severity: Log: Action: Error Always Write a value within the supported range.

0451: Enable interrupt handler failed The driver attempted to register the HBA interrupt service routine with the host operating system, but failed.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or driver issue. If this issue persists, report the error to Broadcom Technical Support.

Broadcom - 72 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0453: Adapter failed to init, mbxCmd READ_CONFIG, mbxStatus Adapter initialization failed when issuing a READ_CONFIG mailbox command.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0454: Adapter failed to init, mbxCmd INIT_LINK, mbxStatus Adapter initialization failed when issuing an INIT_LINK mailbox command.

Data: None Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0456: Adapter failed to issue ASYNCEVT_ENABLE mbox status x%x The mailbox command to enable an asynchronous event notification failed.

Data: None Severity: Log: Action: Error LOG_INIT Ensure the adapter firmware is current. Reload the driver.

0457: Adapter Hardware Error The driver received an interrupt indicating a possible hardware issue.

Data: (1) status (2) status1 (3) status2 Severity: Log: Action: Error Always This error could indicate a hardware or firmware issue. If this issue persists, report the error to Broadcom Technical Support.

0458: Bring adapter online The FC driver has received a request to bring the adapter online. This issue might occur when running lputil.

Data: None Severity: Log: Action: Warning LOG_INIT verbose None required.

0459: Adapter heartbeat failure, taking this port offline.

The Heartbeat mailbox command failed.

Data: None Severity: Log: Action: Error LOG_INIT Ensure that the adapter firmware is current. Reload the driver.

0460: Bring adapter offline The FC driver has received a request to bring the adapter offline. This issue might occur when running lputil.

Data: None Severity: Log: Action: Warning LOG_INIT verbose None required.

0466: Outstanding I/O when bringing Adapter offline The I/O is still pending while attempting to stop the driver.

Data: None Severity: Log: Action: Warning LOG_INIT None required.

Broadcom - 73 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0467: lpfc_topology attribute cannot be set to %d, allowed range is [0, 6], phba->brd_no, val.

The lpfc_topology module parameter is invalid.

Data: None Severity: Log: Action: Error LOG_INIT Use a value in the valid range.

0468: lpfc_restrict_login must be 0 for Physical ports. “vport->cfg_restrict_login = 0; Cannot restrict the login for the physical port.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0469: lpfc_link_speed attribute cannot be set to %d, allowed range is [0, 8] The link speed module parameter is invalid.

Data: None Severity: Log: Action: Error LOG_INIT Use a link speed parameter in the valid range.

0472: PCI channel I/O permanent failure The PCI bus has detected an error. Data: None Severity: Log: Action: Error LOG_INIT Issue an HBA reset. 0474: Unable to allocate memory for issuing MBOX_CONFIG_MSI command Mailbox memory pool allocation error. Data: None Severity: Log: Action: Error LOG_INIT None required.

0475: Not configured for supporting MSI-X cfg_use_msi: 0x%x The lpfc_use_msi module parameter should have been set to 2.

Data: None Severity: Log: Action: Error LOG_INIT Set module parameter lpfc_use_msi = 2.

0476: HBA not supporting SLI-3 or later SLI Revision: The HBA does not support SLI-3 or SLI-4.

Data: None Severity: Log: Action: Error LOG_INIT This HBA does not support msi. Set lpfc_use_msi=0.

0478: MSI request_irq failed ().

The request_irq kernel API has failed.

Data: None Severity: Log: Action: Warning LOG_INIT Set lpfc_use_msi=0.

Broadcom - 74 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0479: Deferred Adapter Hardware Error An adapter hardware error was sent to the driver.

Data: (1) work_hs, (2) work_status[0], (3) work_status[1] Severity: Log: Action: Error LOG_INIT Perform a dump using hbacmd.

0483:Invalid link-attention link speed: x%x", bf_get(lpfc_acqe_link_speed, acqe_link).

The link speed reported in the link attention interrupt is invalid.

Data: None Severity: Log: Action: Error LOG_INIT Check the switch configuration.

0485: MSI-X slow-path request_irq failed ().

The request_irq kernel API has failed.

Data: None Severity: Log: Action: Warning LOG_INIT Set module parameter lpfc_use_msi=0.

0486: MSI-X fast-path () request_irq failed ().

The request_irq kernel API has failed.

Data: None Severity: Log: Action: Warning LOG_INIT Set module parameter lpfc_use_msi=0.

0490: MSI request_irq failed ().

The request_irq kernel API has failed.

Data: None Severity: Log: Action: Warning LOG_INIT Set module parameter lpfc_use_msi=0.

0492: Unable to allocate memory for issuing SLI_CONFIG_SPECIAL mailbox command Mailbox memory pool allocation error.

Data: None Severity: Log: Action: Error LOG_INIT None required. 0493: SLI_CONFIG_SPECIAL mailbox failed with status Mailbox command failed. Data: None Severity: Log: Action: Error LOG_INIT Ensure that the adapter firmware is current. Unload and reload the driver.

0494: Unable to allocate memory for issuing "SLI_FUNCTION_RESET mailbox command" Mailbox memory pool allocation error.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 75 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0495: SLI_FUNCTION_RESET mailbox failed with status add_status , mbx status .

Mailbox command failed. Data: None Severity: Log: Action: Error LOG_INIT Reset the HBA.

0496: Failed allocate slow-path EQ The event queue for the slow path was not allocated.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver.

0497: Failed allocate fast-path EQ The event queue for the fast path was not allocated.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver.

0499: Failed allocate fast-path FCP CQ ().

The completion queue event for the fast path could not be allocated.

Data: None Severity: Log: Action: Error LOG_INIT Unload and reload the driver. 0500: Failed allocate slow-path mailbox CQ Failed to allocate slow-path mailbox CQ.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0501: Failed allocate slow-path ELS CQ Failed to allocate slow-path ELS CQ.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0503: Failed allocate fast-path FCP Failed to allocate fast-path FCP.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0504: Failed allocate slow-path ELS WQ Failed to allocate slow-path ELS WQ.

Data: None Severity: Log: Action: Error LOG_INIT None required. Chapter 4: Troubleshooting Log Messages Broadcom - 76 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0505: Failed allocate slow-path ELS MQ Data: Severity: Log: Action: None Error LOG_INIT None required. 0506: Failed allocate receive HRQ\n Data: Severity: Log: Action: None Error LOG_INIT None required.

0507: Failed allocate receive DRQ Failed to allocate receive DRQ.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0520: Slow-path EQ not allocated The slow-path EQ not allocated.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0522: Fast-path EQ not allocated The fast-path EQ is not allocated.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0523: Failed setup of fast-path EQ , rc = The fast-path EQ setup failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0526: Fast-path FCP CQ not allocated The fast-path FCP is not allocated.

Data: None Severity: Log: Action: Error LOG_INIT None required. 0527: Failed setup of fast-path FCP CQ , rc = The fast-path FCP CQ setup failed.

Data: None Severity: Log: Action: Error LOG_INIT None required. Broadcom - 77 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0528: Mailbox CQ not allocated The mailbox CQ is not allocated.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0530: ELS CQ not allocated The ELS CQ is not allocated Data: None Severity: Log: Action: Error LOG_INIT None required.

0534: Fast-path FCP WQ not allocated The fast-path FCP WQ is not allocated.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0535: Failed setup of fast-path FCP WQ , rc = The fast-path FCP WQ setup failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0536: Slow-path ELS WQ not allocated The slow-path ELS WQ is not allocated.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0538: Slow-path MQ not allocated The slow-path MQ is not allocated.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0540: Receive Queue not allocated The Receive Queue is not allocated.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0542: lpfc_create_static_vport failed to allocate mailbox memory Failed to allocate mailbox memory for VPort creation.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 78 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0543: lpfc_create_static_vport failed to allocate vport_info\n")) Failed to allocate vport_info.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0544: lpfc_create_static_vport failed to issue dump mailbox command ret status Failed to issue a dump mailbox command for static VPort creation.

Data: None Severity: Log: Action: Warning LOG_INIT None required.

0545: lpfc_create_static_vport bad information header 0x%x 0x%x\n"\, le32_to_cpu(vport_info->signature), le32_to_cpu(vport_info->rev) & VPORT_INFO_REV_MASK); Invalid information header; the signature or revision is invalid.

Data: None Severity: Log: Action: Error LOG_INIT None required.

0546: lpfc_create_static_vport failed to create vport Failed to create a VPort.

Data: None Severity: Log: Action: Warning LOG_INIT None required.

0582: Error during sgl post operation The SGL post operation failed.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_IP verbose None required.

0602: Failed to allocate CQ_EVENT entry Failed to allocate a CQ_EVENT entry.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0603: Invalid work queue CQE subtype (x%x)\n", cq- Invalid work queue CQE.

Data: None Severity: Log: Action: Error LOG_SLI None required.

0700: Bus Reset on target failed The bus reset for the specified target failed.

Data: None Severity: Log: Action: Error LOG_FCP None required.

Broadcom - 79 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0704: At limitation of preallocated command buffers The maximum number of command buffers have already been allocated.

Data: None Severity: Log: Action: Warning LOG_FCP verbose None required.

0705: Allocation request of command buffers will exceed max of . Reducing allocation request to The number of command buffers requested will exceed the maximum so a smaller quantity will be allocated.

Data: None Severity: Log: Action: Warning LOG_FCP verbose None required.

0708: Allocation request of command buffers did not succeed. Allocated buffers.

The allocation request for the specified command buffers did not succeed. However, the specified number of buffers has been allocated.

Data: None Severity: Log: Action: Warning LOG_FCP None required.

0711: detected queue full - lun queue depth adjusted to%d The driver detected a queue full status on a SCSI command response. New LUN queue depth is reported.

Data: (1) New LUN queue depth Severity: Log: Action: Warning LOG_FCP verbose This may indicate an oversubscribed target array. Check your SAN configuration and I/O workload.

0713: SCSI layer issued Device Reset (%d, %d) A device reset was issued.

Data: None Severity: Log: Action: Error LOG_FCP None required.

0714: SCSI layer issued bus reset The SCSI layer is requesting the driver to abort all I/Os to all targets on this HBA.

Data: (1) ret Severity: Log: Action: Error Always Check the state of the targets in question.

0720: FCP command residual overrun error A residual overrun error has occurred while processing the specified FCP command.

Data: (1) request_bufflen (2) resid Severity: Log: Action: Warning LOG_FCP verbose If this issue persists, check the targets for errors.

0721: Device Reset rport failure: rdata The reset of the R_Port failed.

Data: None Severity: Log: Action: Error LOG_FCP None required.

Broadcom - 80 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0722: Target Reset rport failure: rdata The reset of the target failed.

Data: None Severity: Log: Action: Error LOG_FCP None required.

0723: SCSI layer issued Target Reset (%d, %d) The SCSI layer issued a target reset.

Data: None Severity: Log: Action: Error LOG_FCP None required.

0724: I/O flush failure for context <“LUN”,”TGT”,”HOST”,”Unknown”>: cnt The I/O flush to the LUN, target, or host has failed.

Data: None Severity: Log: Action: Error LOG_FCP None required.

0727: TMF to TGT LUN failed (, ) The task management command failed.

Data: None Severity: Log: Action: Error LOG_FCP None required. 0729: FCP cmd failed / status: result: The specified device failed an FCP command. Data: (1) ulpContext (2) iotag Severity: Log: Action: Warning LOG_FCP verbose Check the state of the target in question.

0730: FCP command failed: RSP The FCP command failed with a response error.

Data: (1) resp_info (2) scsi_status (3) ResId (4) SnsLen (5) RspLen (6)rspInfo3 Severity: Log: Action: Warning LOG_FCP verbose Check the state of the target in question.

0734: FCP read check error The issued FCP command returned a read check error.

Data: (1) fcpDl (2) rspResId (3) fcpi_parm (4) cmd[0] Severity: Log: Action: Warning LOG_FCP verbose Check the state of the target in question.

0735: FCP Read Check Error and Underrun Data HBA reported under run from storage array.

Data: (1) vpi (2) fcpDI (3) res_id (4) fcpi_parm Severity: Log: Action: Warning LOG_FCP_ERROR verbose No action needed, informational.

Broadcom - 81 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

0748: Abort handler timed out waiting for abort to complete:ret D LUN The abort handler timed out waiting for abort to complete.

Data: None Severity: Log: Action: Error Always None required.

0749: SCSI layer issued abort device The SCSI layer aborted a device.

Data: (1) ret, (2) id, (3) lun, (4) snum Severity: Log: Action: Warning LOG_FCP verbose None required.

0915 Register VPI failed: Could not register the VPI.

Data: None Severity: Log: Action: Error LOG_MBOX None required. 1005: AUTHENTICATION_FAILURE Nport: The system detected DHCHAP authentication failure on a port.

Data: (1) nlp_DID Severity: Log: Action: Error LOG_SECURITY Verify authentication settings and keys on local and remote port.

1006: Bad Name tag in auth message < message > DHCHAP Authentication process failed when invalid tag was detected.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1007: Bad Name length in auth message < message > DHCHAP Authentication process failed when an invalid name was detected.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1008: Bad Number of Protocols DHCHAP Authentication process failed due to unexpected protocol number.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1009: Bad param type DHCHAP Authentication process failed when invalid protocol was detected.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

Broadcom - 82 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1010: Bad Tag 1 DHCHAP Authentication process failed when bad Tag was detected.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1011: Auth_neg no hash function chosen DHCHAP Authentication process failed when an incorrect hash function was specified.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1012: Auth_negotiate Bad Tag DHCHAP Authentication process failed due to bad Tag for auto negotiation.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1013: Auth_negotiate no DH_group found DHCHAP Authentication process failed when an incorrect or a missing DH Group was detected.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to the Broadcom Technical Support.

1014: dhchap challenge bad name tag DHCHAP Authentication process failed when incorrect challenge name tag was detected.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to the Broadcom Technical Support.

1015: dhchap challenge bad name length DHCHAP Authentication process failed due to unexpected challenge name length.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1016: dhchap challenge Hash ID not Supported DHCHAP Authentication process failed due to uncorroborated challenge hash ID.

Data: (1) message Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1017: dhchap challenge could not find DH Group DHCHAP Authentication process failed due to uncorroborated challenge group.

Data: None Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

Broadcom - 83 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1018: dhchap challenge No Public key for non-NULL DH Group There is no public key for the non-NULL DH group.

Data: None Severity: Log: Action: Error LOG_SECURITY None required.

1019: Request tranid timed out A transaction with storage array could not complete due to timeout.

Data: (1) tran_id Severity: Log: Action: Warning LOG_SECURITY verbose Software driver warning. If this issue persists, report these errors to Broadcom Technical Support.

1021: ERROR: attempted to queue security work, when no workqueue created Driver encountered missing queue required for processing security information.

Data: None Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report these errors to Broadcom Technical Support.

1022: Security request does not exist A security request operation failed because no match was found for such request. Data: None Severity: Log: Action: Warning LOG_SECURITY Software driver warning. If this issue persists, report these errors to Broadcom Technical Support.

1023: Warning - data may have been truncated. Data: reqdl: mesdl: A security message exchange operation failed because the response was missing or unreliable.

Data: Severity: Log: Action: None Warning LOG_SECURITY Software driver warning. If this issue persists, report these errors to Broadcom Technical Support.

1028: Start Authentication: No buffers The authentication failed because some memory resources were not allocated.

Data: None Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1029: Reauthentication Failure The driver encountered errors and there was a failure to re-authenticate.

Data: None Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1031: Start Authentication: Get config failed The authentication failed due to some error during port configuration.

Data: None Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

Broadcom - 84 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1032: Start Authentication: get config timed out The node authentication was aborted because waiting for port configuration to complete, timed out.

Data: None Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1033: Received auth_negotiate from Nport: < nlp_DID> Unsolicited authentication negotiation message received from a port.

Data: (1) nlp_DID Severity: Log: Action: Warning LOG_SECURITY No action, this message is informational.

1034: Not Expecting Challenge - Rejecting Challenge Unsolicited authentication challenge received from a port was rejected.

Data: None Severity: Log: Action: Warning LOG_SECURITY Software driver warning. If this issue persists, report errors to Broadcom Technical Support.

elx_mag1036: Authentication transaction reject - re-auth request reason exp An authentication was rejected and requested again due to reason as displayed with explanation.

Data: (1) reason (2) explanation.

Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1037: Authentication transaction reject - restarting authentication, reason exp An authentication process was rejected then restarted and an authentication requested was again due to reason as displayed with the explanation.

Data: Severity: (1) reason (2) explanation.

Error Log: Action: LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1039: Not Expecting Reply - rejecting. State An unanticipated reply was received during authentication and was subsequently rejected.

Data: (1) auth_state.

Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1040: Bad Reply trans_id- rejecting. Trans_id < trans_id > Expecting: < trans_id> Unexpected transaction id was received during authentication and was subsequently rejected.

Data: (1) auth_state Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1043: Authentication LS_RJT The authentication request was rejected.

Data: None Severity: Log: Action: Error LOG_ELS None required.

Broadcom - 85 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1045: Issue AUTH_NEG failed Status:%x The authentication negotiation failed.

Data: None Severity: Log: Action: Error LOG_ELS None required.

1048: Issue AUTH_REJECT failed Could not issue the reject for the authentication request.

Data: None Severity: Log: Action: Error LOG_ELS None required.

1049: Authentication is enabled but authentication service is not running Discovery failed because DHCHAP authentication was enabled while no authentication service was established.

Data: None Severity: Log: Action: Error LOG_SECURITY Start the authentication daemon (fcauthd).

elx_ msg1050: Authentication mode is disabled, but is required by the fabric Discovery failed because the switch fabric required authentication, but either authentication was not configured or the authentication mode for this port pair is disabled. Data: Severity: None Error Log: Action: LOG_SECURITY Configure the driver to authenticate with the switch or disable authentication on the switch to this port.

1053: Start Authentication: Security service offline The authentication failed because security service was unavailable.

Data: None Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

1055: Authentication parameter is disabled, but is required by the fabric FLOGI failed because the fabric has indicated that authentication is required, but authentication has not yet been configured or enabled on the HBA.

Data: Severity: None Error Log: Action: LOG_SECURITY Configure authentication on this HBA.

1057: Authentication transaction reject. reason exp An authentication was rejected and requested again due to the reason as displayed with the explanation.

Data: (1) reason (2) explanation.

Severity: Log: Action: Error LOG_SECURITY Software driver error. If this issue persists, report errors to Broadcom Technical Support.

Broadcom - 86 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1058: Waiting for authentication service A delay occurred when the authentication service was not initially available as expected.

Data: None Severity: Log: Action: Warning LOG_SECURITY Software driver warning. If this issue persists, report these errors to Broadcom Technical Support.

1059: Authentication became available The authentication service came online, but it was not initially available as expected.

Data: None Severity: Log: Action: Warning LOG_SECURITY Software driver warning. If this issue persists, report these errors to Broadcom Technical Support.

1201: Failed to allocate dfc_host Failed to allocate memory for the dfc_host_struct.

Data: None Severity: Log: Action: Error LOG_ELS None required.

1210: Invalid cmd size: cmd cmdsz rspsz The management command for LPFC 2100 has failed.

Data: None Severity: Log: Action: Error LOG_LIBDFC None required.

1211: genreq alloc failed\n"); Resource allocation failure.

Data: (1) return code Log: Severity: Action: LOG_LIBDFC Error Kernel memory resources are too low.

1213: FCoE cmd overflow: off <#> + cnt <#> > cmdsz <#> Application has tried to read more data than originally requested.

Data: (1) response offset (2) size (3) cmd size Log: Severity: Action: LOG_LIBDFC Error The application might have sent a invalid command.

1214: Cannot issue FCoE cmd SLI not active: <#> rc= -EACCESS The SLI layer has not been initialized.

Data: (1) offset Log: Severity: Action: LOG_LIBDFC Error Restart the HBA.

1215: Cannot issue FCoE cmd: not ready or not in maint mode Either the external link is unplugged, link down, and the FCoE is not in maintenance mode.

Data: (1) current offset (2) return code.

Log: Severity: Action: LOG_LIBDFC Error Plug external cable in or set FCoE in maintenance mode.

Broadcom - 87 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1216: FCoE IOCB failed: off <#> rc <#> FCoE command generated by the application has failed.

Data: (1) offset (2) return code.

Log: Severity: Action: LOG_LIBDFC Error Application should retry the command.

1223: menlo_write: couldn't alloc genreq Resource allocation failure.

Data: None Log: Severity: Action: LOG_LIBDFC Error Kernel memory resources too low.

1224: FCoE iocb failed off <#> rc=<#>", FCoE command failed in SLI.

Data: (1) offset (2) return code Log: Severity: Action: LOG_LIBDFC Informational.

Retry the command, if it fails again, reset HBA when convenient.

1227: FCoE IOCB TMO: handler set for The management command for the LPFC 2100 has timed out.

Data: None Severity: Log: Action: Warning LOG_LIBDFC None required.

1228: FCoE IOCB TMO: handler set for A management IOCB for the LPFC 2100 has timed out.

Data: None Severity: Log: Action: Warning LOG_LIBDFC None required.

1229: Waiting for menlo mnt Waiting for the LPFC 2100 to enter maintenance method.

Data: None Severity: Log: Action: Warning LOG_LIBDFC None required.

1230: Could not find buffer for FCoE cmd:off <#> indmp off <#> Could not find resources associated with this FCoE command.

Data: (1) current offset (2) buffer desc pointer (3) size Severity: Log: Action: Error LOG_LIBDFC Try reloading the driver when convenient.

1231: bad bpl: An invalid buffer list was detected upon completion.

Data: None Severity: Log: Action: Error LOG_LIBDFC None required.

Broadcom - 88 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1235: Could not find buffer for FCoE cmd: off:<#> poff:<#> cnt:<#> mlastcnt:<#> addl: addh: mdsz:<#> FCoE command failed because it could not find the resource.

Data: (1) current offset (2)previous offset (3) count (4) last count (5) address low (6) address high Severity: Log: Action: Error LOG_LIBDFC No action needed, informational.

1238: FCoE IOCB failed: off <#> rc=<#> The command generated by the driver to check the FCoE has failed.

Data: (1) offset (2) return code Log: Severity: Action: LOG_LIBDFC Error Make sure link is up or the adapter has set menlo in maintenance mode. 1240: Unable to allocate command buffer memory Could not allocate memory for the command buffer.

Data: None Severity: Log: Action: Error LOG_LINK_EVENT None required.

1243: Menlo command error. code=%d.\n", mlorsp->code The Menlo maintenance command failed.

Data: None Severity: Log: Action: Error LOG_LINK_EVENT None required.

1244: Unable to allocate response buffer memory.

Could not allocate memory for the management command response.

Data: None Severity: Log: Action: Error LOG_LINK_EVENT None required.

1246: FCoE chip is running golden firmware. Update FCoE chip firmware immediately The FCoE is running the golden firmware.

Data: (1) firmware-type Severity: Log: Action: Error LOG_LINK_EVENT Try resetting the FCoE to operational mode and disable maintenance mode.

1247: FCoE chip is running diagnostic firmware. Operational use suspended. The FCoE is running a diagnostic.

Data: (1) firmware-type Severity: Log: Action: Error LOG_LINK_EVENT Try resetting the FCoE to operational mode.

1248: FCoE chip is running unknown firmware. The FCoE is running an unknown firmware version.

Data: (1) firmware-type Severity: Log: Action: Error LOG_LINK_EVENT Try resetting the FCoE to operational mode. Try loading the latest FCoE firmware.

Broadcom - 89 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1249: Invalid FRU data found on adapter. Return adapter to Emulex for repair.

The FRU data on the FCoE chip is invalid.

Data: (1) firmware-type Severity: Log: Action: Error LOG_LINK_EVENT Try resetting the FCoE to operational mode. Try loading the latest FCoE firmware or send the HBA back to Broadcom for repair.

1250: Menlo command error. code=<#> The IOCB driver sent to check FCoE state has bad header size.

Data: (1) return code Log: Severity: Action: LOG_LINK_EVENT Error Try resetting the FCoE to operational mode.

1251: Menlo command error. code=<#> The IOCB driver sent to check FCoE state has failed, no resources.

Data: (1) return code Log: Severity: Action: LOG_LINK_EVENT Error Try resetting the FCoE to operational mode.

1252: Menlo command error. code=<#> The IOCB driver sent to check FCoE state has failed.

Data: (1) return code Log: Severity: Action: LOG_LINK_EVENT Error Try resetting the FCoE to operational mode.

1257: lpfc_menlo_issue_iocb: handler set for .

Data: Log: Severity: Action: None LOG_LIBDFC Warning None required.

1259: mbox: Issued mailbox cmd while in stopped state.

Only the dump mailbox command and reset adapter mailbox command are allowed when in the stopped state.

Data: None Severity: Log: Action: Warning LOG_MBOX None required.

1262: Failed to allocate dfc_host Could not allocate memory the dfc_host_struct.

Data: None Log: Severity: Action: LOG_LIBDFC Error None required. 1268: Find ndlp returned NULL for oxid:x%x SID:x%x, oxid, sid.(int)off, rc.

Could not find the node for this DID.

Data: None Severity: Log: Action: Warning LOG_ELS None required.

Broadcom - 90 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1302: Invalid speed for this board: Reset link speed to auto: The driver is reinitializing the link speed to auto-detect.

Data: None Severity: Log: Action: Warning LOG_LINK_EVENT verbose None required.

1303: Link Up Event received A link up event was received. It is possible for multiple link events to be received together. Data: (1) fc_eventTag (2) granted_AL_PA (3) UlnkSpeed (4) alpa_map[0] Detail: If link events received, log (1) last event number received, (2) ALPA granted, (3) Link speed, (4) number of entries in the loop init LILP ALPA map. An ALPA map message is also recorded if LINK_EVENT verbose mode is set. Each ALPA map message contains 16 AL_PAs. Severity: Log: Action: Error Always If numerous link events are occurring, check the physical connections to the FC network.

1304: Link Up Event ALPA map A link up event was received.

Data: (1) wd1, (2) wd2, (3) wd3, (4) wd4 Severity: Log: Action: Warning LOG_LINK_EVENT verbose If numerous link events are occurring, check the physical connections to the FC network.

1305: Link Down Event received A link down event was received.

Data: (1) fc_eventTag (2) hba_state (3) fc_flag Severity: Log: Action: Error Always If numerous link events are occurring, check the physical connections to the FC network.

1306: Link Up Event in loop back mode x%x received Data: x%x x%x x%x x%x Link up notification; configured for loopback.

Data: (1) fc_eventTag (2) granted_AL_PA (3) UlnkSpeed (4) alpa_map[0] Severity: Log: Action: Error LOG_LINK_EVENT None required.

1308: Menlo Maint Mode Link up Event x%x rcvd Data: x%x x%x x%x Link down notification; configured for loopback.

Data: (1) fc_eventTag (2) port_state (3) vport fc_flag Severity: Log: Action: Error LOG_LINK_EVENT None required.

1309: Link Up Event npiv not supported in loop topology NPIV is not supported in loop topology.

Data: None Severity: Log: Action: Error LOG_LINK_EVENT None required.

Broadcom - 91 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1310: Menlo Maint Mode Link up Event rcvd The link is up in maintenance mode; only management commands are allowed.

Data: (1) fc_eventTag (2) port_state (3) vport fc_flag Severity: Log: Action: Error LOG_LINK_EVENT None required.

1312: Link Down Event received Maintenance mode link up notification received without entering link down.

Data: (1) fc_eventTag (2) port_state (3) vport fc_flag Severity: Log: Action: Error LOG_LINK_EVENT None required.

1400: Failed to initialize sgl list.

Failed to initialize SGL list during initialization.

Data: None Severity: Log: Action: Error LOG_INIT None required. 1401: Failed to enable pci device.

Failed to enable PCI device during initialization.

Data: None Severity: Log: Action: Error LOG_INIT None required. 1402: Failed to set up pci memory space.

PCI initialization failed.

Data: None Severity: Log: Action: Error LOG_INIT None required. 1403: Failed to set up driver resource.

Driver resource initialization failed.

Data: None Severity: Log: Action: Error LOG_INIT None required. 1404: Failed to set up driver resource.

Driver resource initialization failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1405: Failed to initialize iocb list.

Driver resource initialization failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 92 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1406: Failed to set up driver resource.

Initialization failed to set up driver resource.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1407: Failed to create scsi host.

Initialization failed to create SCSI host.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1408: Failure HBA POST Status: sta_reg=0x%x, ""perr=x%x, sfi=x%x, nip=x%x, ipc=x%x, xrom=x%x, ""dl=x%x, pstatus=x%x\n", sta_reg.word0, bf_get(lpfc_hst_state_perr, &sta_reg), The HBA power-on self-test has failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1409: Failed to enable pci device.

Failed to enable PCI device during initialization.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1410: Failed to set up pci memory space.

Initialization failed to set up PCI memory space.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1411: Failed to set up driver resource.

Data: Severity: Log: Action: None Error LOG_INIT None required.

1412: Failed to set up driver resource.

Initialization failed to set up driver resource.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1413: Failed to initialize iocb list.

Initialization failed to initialize the IOCB list.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 93 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1414: Failed to set up driver resource.

Initialization failed to set up driver resource.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1415: Failed to create scsi host.

Initialization failed to create SCSI host.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1416: Failed to allocate sysfs attr Initialization failed to sysfs attribute.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1418: Invalid HBA PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1419: Invalid HBA PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1420: Invalid HBA PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1421: Failed to set up hba Initialization failed to set up the HBA.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1422: HBA Unrecoverable error: uerr_lo_reg=, uerr_hi_reg=, online0_reg=, online1_reg= The HBA has notified the driver that it has encountered an unrecoverable error.

Data: None Severity: Log: Action: Error LOG_INIT Perform a dump from the Emulex OneCommand Manager application. Then, unload and reload the driver.

Broadcom - 94 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1423: HBA Unrecoverable error: uerr_lo_reg=, uerr_hi_reg=, online0_reg=, online1_reg= The HBA has notified the driver that it has encountered an unrecoverable error.

Data: None Severity: Log: Action: Error LOG_INIT Perform a dump from the Emulex OneCommand Manager application. Then, unload and reload the driver.

1424: Invalid PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1425: Invalid PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1426: Invalid PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1427: Invalid PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1428: Invalid PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1429: Invalid PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1430: Failed to initialize sql list Failed to initialize SQL list.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 95 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1431: Invalid HBA PCI device group: Invalid HBA PCI device group detected.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1432: Failed to initialize rpi headers.

Failed to initialize RPI headers.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1476: Failed to allocate sysfs attr Failed to allocate sysfs attrIbutes.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1477: Failed to set up hba Failed to set up the HBA.

Data: None Severity: Log: Action: Error LOG_INIT None required.

1603: Loopback test did not receive expected data length. actual length expected length .

The loopback test did not receive the same amount of data that it transmitted.

Data: None Severity: Log: Action: Error LOG_LIBDFC None required.

1800: Could not issue unreg_vpi Driver attempt to unregister VPI failed.

Data: None Severity: Log: Action: Error LOG_VPORT verbose Software driver error. If this issue persists, report these errors to Broadcom Technical Support.

1801: Create vport work array FAILED: cannot do scsi_host_get The driver was unable to get a reference to a SCSI host.

Data: None Severity: Log: Action: Warning LOG_VPORT verbose Software driver warning. If this issue persists, report these errors to Broadcom Technical Support.

1816: FLOGI NPIV supported, response data The fabric reports support for NPIV upon FLOGI.

Data: (1) response_multiple_NPort Severity: Log: Action: Warning LOG_VPORT verbose No action needed, informational.

Broadcom - 96 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1817: Fabric does not support NPIV - configuring single port mode The fabric reports no support for NPIV upon FLOGI.

Data: None Severity: Log: Action: Warning LOG_VPORT verbose No action needed, informational.

1818: VPort failed init, mbxCmd READ_SPARM mbxStatus , rc = A pending mailbox command that was issued to initialize the port failed.

Data: (1) mbxCommand (2) mbxStatus (3) rc Severity: Log: Action: Error LOG_VPORT verbose Software driver error. If this issue persists, report these errors to Broadcom Technical Support.

1819: Unrecognized lpfc_sli_mode parameter: An attempt was made to set the SLI mode to an invalid value. The valid values for the SLI mode are 0, 2, and 3. Data: (1) lpfc_sli_mode Severity: Log: Action: Error LOG_VPORT verbose Correct the lpfc_sli_mode driver parameter setting. Valid values are 0, 2, and 3.

1820: Unable to select SLI-3. Not supported by adapter.

The HBA is incapable of operating in a given mode.

Data: None Severity: Log: Action: Error LOG_VPORT verbose SLI-3 mode is only available on some HBAs. Do not attempt to force the SLI mode to 3 on HBAs that do not support SLI-3 mode. This is an informational message. HBAs that do not support SLI-3 will be configured to run in SLI-2 mode, but it is recommended to use the auto setting (0).

1821: Create VPORT failed. Invalid WWN format The port could not be created due to an invalid WWNN or WWPN format.

Data: None Severity: Log: Action: Error LOG_VPORT verbose Provide a valid WWN when creating Vports.

1822: Invalid : An invalid WWN was used when creating a VPort.

Data: (1) type_name (2) wwn[1] (3) wwn[3] (3) wwn[5] (4) wwn[7] Severity: Log: Action: Error LOG_VPORT verbose When creating a VPort, you must furnish a valid WWN.

1823: Create VPORT failed. Duplicate WWN on HBA.

The port could not be created because it would duplicate an existing WWNN HBA address. The resources for the port were discarded.

Data: None Severity: Log: Action: Error LOG_VPORT verbose Provide a unique WWN.

Broadcom - 97 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1824: NPIV enabled: Override lpfc_sli_mode parameter () to auto(0) The lpfc_enable_npiv and lpfc_sli_mode driver parameter settings conflict. The HBA must be configured for SLI-3 mode to support NPIV.

Data: (1) lpfc_sli_mode Severity: Log: Action: Error LOG_VPORT verbose This is an informational message that indicates that the lpfc_enable_npiv and lpfc_sli_mode parameter settings are not compatible. Resolve the parameter conflict by setting the SLI mode to 0 or 3 or, if SLI-2 mode is required, disable NPIV.

1825: Vport Created.

This message indicates that a port was created in the system. It is displayed at this level to ensure that it always appears at all log levels.

Data: None Severity: Log: Action: Error LOG_VPORT verbose No action, informational.

1826: Vport Disabled.

The port had to be disabled in the system.

Data: None Severity: Log: Action: Error LOG_VPORT verbose No action, informational.

1827: Vport Enabled The port was enabled after possible recovery from some errors.

Data: None Severity: Log: Action: Error LOG_VPORT verbose No action, informational.

1828: Vport Deleted A Vport was deleted.

Data: None Severity: Log: Action: Error LOG_VPORT verbose No action, informational.

1829: CT command failed to delete objects on fabric. A command issued to the fabric to delete an associated resource for an object, such as for a port, failed.

Data: None Severity: Log: Action: Error LOG_VPORT verbose Software driver error. If this issue persists, report these errors to Broadcom Technical Support.

1830: Signal aborted mbxCmd A pending mailbox command was aborted because the thread received a signal.

Data: None Severity: Log: Action: Error LOG_VPORT verbose Retry the attempted command.

1831: Create VPORT Interrupted The port creation process was unexpectedly interrupted at a critical time, and the operation was unsuccessful.

Data: None Severity: Log: Action: Error LOG_VPORT verbose The process was interrupted while creating a VPort. Retry the command.

Broadcom - 98 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

1832: No pending MBOX command to handle Data: Severity: Log: Action: None Error LOG_MBOX None required.

1835: Vport discovery quiesce failed: state fc_flags wait msecs Could not pause discovery on this VPort.

Data: None Severity: Log: Action: Error LOG_VPORT None required.

1836: Could not issue unreg_login(all_rpis) status The unreg_login cannot be issued.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_VPORT None required.

1837: Vport_delete failed: Cannot delete static vport Static VPorts cannot be deleted.

Data: None Severity: Log: Action: Error LOG_VPORT None required. 1838: Failed to INIT_VPI on vpi status Failed to INIT_VPI.

Data: None Severity: Log: Action: Error LOG_VPORT None required.

2000: Failed to allocate mbox for read_FCF cmd Failed to allocate mailbox for READ_FCF command.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2001: Unable to allocate memory for issuing SLI_CONFIG_SPECIAL mailbox command Unable to allocate memory for issuing the SLI_CONFIG_SPECIAL mailbox command.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2002: Error Could not grow rpi count An error occurred because the RPI count could not be increased.

Data: None Severity: Log: Action: Error LOG_SLI None required.

Broadcom - 99 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2004: Failed to allocate XRI.last XRITAG is Max XRI is , Used XRI is .

All XRIs are in use.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

2005: Unable to deregister pages from HBA: The SGL pages could not be unregistered from the firmware.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2007: Only Limited Edition cmd Format supported The SGL pages could not be unregistered from the firmware.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2008: Error posting all rpi headers The RPI headers could not be posted to the firmware.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2009: Failed to allocate mbox for ADD_FCF cmd Failed to allocate mailbox for ADD_FCF command.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2010: Resume RPI Mailbox failed status , mbxStatus Data: Severity: Log: Action: None Error LOG_SLI None required.

2011: Unable to allocate memory for issuing SLI_CONFIG_SPECIAL mailbox command Data: None Severity: Log: Action: Error LOG_SLI None required.

2012: Mailbox failed , mbxCmd READ_CONFIG, mbxStatus The READ_CONFIG mailbox command failed.

Data: None Severity: Log: Action: Error LOG_SLI None required.

Broadcom - 100 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2013: Could not manually add FCF record 0, status Could not add an FCF record to the FCF list.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

2014: Invalid command The IOCB command is invalid.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2015: Invalid CT %x command Invalid command type in the IOCB is not supported.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2017: REG_FCFI mbxStatus error HBA state The REG_FCFI mailbox command has failed.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

2018: REG_VFI mbxStatus error HBA state The REG_VFI mailbox command has failed.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

2020: Failed to allocate mbox for ADD_FCF cmd Failed to allocate mailbox for ADD_FCF command.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2022: VPI Mailbox failed status , mbxStatus The INIT VPI mailbox command has failed.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2401: Failed to allocate memory for ELS XRI management array of size .

Initialization failed to allocate memory for the ELS XRI management array.

Data: None Severity: Log: Action: Error LOG_SLI None required.

Broadcom - 101 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2500: EQ_CREATE mailbox failed with status add_status , mbx status The mailbox command sent to create the event queue has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2501: CQ_CREATE mailbox failed with status add_status , mbx status The mailbox command sent to create the completion queue has failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2502: MQ_CREATE mailbox failed with status add_status , mbx status The mailbox command sent to create the mailbox queue has failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2503: WWQ_CREATE mailbox failed with status add_status , mbx status The mailbox command sent to create the work queue has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2504: RQ_CREATE mailbox failed with status add_status , mbx status The mailbox command sent to create the receive queue has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2505: EQ_DESTROY mailbox failed with status add_status , mbx status The mailbox command sent to delete the event queue has failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2506: CQ_DESTROY mailbox failed with status add_status , mbx status The mailbox command sent to delete the completion queue has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2507: MQ_DESTROY mailbox failed with status add_status , mbx status The mailbox command sent to delete the mailbox queue has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 102 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2508: WQ_DESTROY mailbox failed with status add_status , mbx status The mailbox command sent to delete the work queue has failed.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2509: RQ_DESTROY mailbox failed with status add_status , mbx status The mailbox command sent to delete the work queue has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2510: RQ_DESTROY mailbox failed with status add_status , mbx status The mailbox command sent to delete the work queue has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2511: POST_SGL mailbox failed with status add_status , mbx status n The mailbox command sent to post the SGL pages to the firmware has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2512: REMOVE_ALL_SGL_PAGES mailbox failed with status add_status , mbx status The mailbox command sent to delete the SGL pages from the firmware has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2513: POST_SGL_BLOCK mailbox command failed status add_status mbx status The mailbox command sent to post the SGL pages to the firmware has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2514: POST_RPI_HDR mailbox failed with status add_status , mbx status The mailbox command sent to post the RPUI header pages to the firmware has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2515: ADD_FCF_RECORD mailbox failed with status The mailbox command to add the FCF record has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

Chapter 4: Troubleshooting Log Messages Broadcom - 103 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2516: DEL FCF of default FCF Index failed mbx status , status add_status The mailbox command to delete the FCF record has failed. Data: None Severity: Log: Action: Error LOG_SLI None required.

2517: Unregister FCFI command failed status %d, mbxStatus x%x", rc, bf_get(lpfc_mqe_status, &mbox->u.mqe) The driver was unable to unregister the FCFI from the firmware. Data: None Severity: Log: Action: Error LOG_SLI None required.

2518: Requested to send 0 NOP mailbox cmd Data: Severity: Log: Action: None Warning LOG_INIT None required.

2519: Unable to allocate memory for NOP mailbox command Memory allocation for this mailbox command has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2520: NOP mailbox command failed status x%x add_status x%x mbx status x%x, shdr_status, shdr_add_status, rc.

The NOP mailbox command has failed.

Data: None Severity: Log: Action: Warning LOG_INIT None required.

2521: READ_FCF_RECORD mailbox failed with status add_status , mbx The READ_FCF_RECORD mailbox command failed.

Data: None Severity: Log: Action: Error LOG_INIT None required. 2523: Allocated DMA memory size () is less than the requested DMA memory size () The ADD_FCF_RECORD mailbox command failed to retrieve the length required from the firmware. Data: None Severity: Log: Action: Error LOG_INIT None required.

2524: Failed to get the non-embedded SGE virtual address The READ_FCF_RECORD mailbox command could not retrieve the SGE that was requested.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

Chapter 4: Troubleshooting Log Messages Broadcom - 104 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2527: Failed to allocate non-embedded SGE array.

Failed to allocate the non-embedded SGE array.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

2528: Mailbox command cannot issue The mailbox command could not be issued because the mailbox interrupt is disabled.

Data: (1) mbxCommand (2) sli_flag (3) flag Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

2529: Mailbox command cannot issue Data: Severity: Log: Action: (1) mbxCommand (2) sli_flag (3) flag Error LOG_MBOX, LOG_SLI None required.

2530: Mailbox command cannot issue The SLI layer in the driver is inactive.

Data: (1) mb.mbxCommand (2) sli_flag (3) flag Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

2531: Mailbox command cannot issue Data: Severity: Log: Action: (1) mb.mbxCommand (2) sli_flag (3) flag Error LOG_MBOX, LOG_SLI None required.

2532: Mailbox command () cannot issue The mailbox bootstrap code detected that the SLI layer is active.

Data: (1) sli4_mbox_opcode (2) sli_flag,(3) MBX_POLL Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

2533: Mailbox command () cannot issue Data: Severity: Log: Action: (1) sli4_mbox_opcode (2) sli_flag (3) MBX_NOWAIT Error LOG_MBOX, LOG_SLI None required.

2535: Unsupported RQ count. () The receive queue ring can only be 512, 1024, 2048, or 4096.

Data: None Severity: Log: Action: Error LOG_SLI None required.

Broadcom - 105 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2536: Unsupported RQ count. () The receive queue ring can only be 512, 1024, 2048, or 4096.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2537: Receive Frame Truncated!

The receive unsolicited handler detected a truncated frame.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2539: Dropped frame rctl:%s type:%s\n An unsupported frame was received by the port and dropped.

Data: (1) rctl_names[fc_hdr->fh_r_ctl], (2) type_names[fc_hdr->fh_type] Severity: Log: Action: Error Always No action needed, informational.

2540: Ring handler: unexpected Rctl Type The received frame has an unsupported RCTL or FH_TYPE.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

2541: Mailbox command () cannot issue Data: Severity: Log: Action: (1) sli_mbox_opcode (2) sli_flag (3) flag Error LOG_MBOX, LOG_SLI None required.

2542: Try to issue mailbox command () synchronously ahead of async mailbox command queue Attempting to send a synchronous mailbox command ahead of the asynchronous mailbox commands.

Data: (1) sli4_mbx_opcode or sli_mbox_opcode, (2) sli_flag, (3) flag Severity: Log: Action: Warning LOG_MBOX, LOG_SLI None required.

2543: Mailbox command () cannot issue The mailbox command does not have all of the fields set correctly.

Data: (1) sli_mbox_opcode (2) sli_flag (3) flag Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

2544: Mailbox command () cannot issue The HBA cannot be accessed on the PCI bus.

Data: (1) sli_mbox_opcode (2) sli_flag (3) flag Severity: Log: Action: Error LOG_MBOX, LOG_SLI None required.

Chapter 4: Troubleshooting Log Messages Broadcom - 106 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2546: New FCF found index tag A new FCF has been found.

Data: None Severity: Log: Action: Error LOG_DISCOVERY None required.

2547: Read FCF record failed Could not read the FCF record from the firmware.

Data: None Severity: Log: Action: Error LOG_DISCOVERY None required.

2548: FCF Table full count tag The FCF table is full.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2549: FCF disconnected from network index tag The FCF has disconnected from the network.

Data: None Severity: Log: Action: Error LOG_DISCOVERY None required.

2550: UNREG_FCFI mbxStatus error HBA state .

The UNREG_FCFI mailbox command has failed.

Data: None Severity: Log: Action: Error LOG_DISCOVERY. LOG_MBOX None required.

2551: UNREG_FCFI mbox allocation failed HBA state The allocation for the UNREG_FCFI mailbox command has failed.

Data: None Severity: Log: Action: Error LOG_DISCOVERY, LOG_MBOX None required.

2552: UNREG_FCFI issue mbox failed rc HBA state .

The UNREG_FCFI mailbox command has failed.

Data: None Severity: Log: Action: Error LOG_DISCOVERY, LOG_MBOX None required.

2553: lpfc_unregister_unused_fcf failed to read FCF record HBA state.

Data: Severity: Log: Action: None Error LOG_DISCOVERY, LOG_MBOX None required.

Broadcom - 107 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2554: Could not allocate memory for fcf record Data: Severity: Log: Action: None Error LOG_MBOX, LOG_SLI None required.

2555: UNREG_VFI mbxStatus error HBA state The UNREG_VFI mailbox command has failed.

Data: None Severity: Log: Action: Error LOG_DISCOVERY, LOG_MBOX None required.

2556: UNREG_VFI mbox allocation failed HBA state Could not allocate memory for UNREG_VFI mailbox command.

Data: None Severity: Log: Action: Error LOG_DISCOVERY, LOG_MBOX None required.

2557: UNREG_VFI issue mbox failed rc HBA state Could not issue the UNREG_VFI mailbox command.

Data: None Severity: Log: Action: Error LOG_DISCOVERY, LOG_MBOX None required.

2558: ADD_FCF_RECORD mailbox failed with status add_status The ADD_FCF_RECORD mailbox command has failed. Data: None Severity: Log: Action: Error LOG_INIT None required.

2559: Block sgl registration required DMA size great than a page.

Attempting to register more SGEs with the firmware than can fit in a page.

Data: None Severity: Log: Action: Warning LOG_INIT None required.

2560: Failed to allocate mbox cmd memory\n Failed to allocate mailbox command memory.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2561: Allocated DMA memory size () is less than the requested DMA memory size () Could not get the memory required for the number of XRIs that are attempting to be posted.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 108 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2562: No room left for SCSI XRI allocation max_xri=, els_xri=n The number of allocated XRIs has reached the max_xri value.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2563: Failed to allocate memory for SCSI XRI management array of size .

Initialization could not allocate memory to hold the XRIs.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2564: POST_SGL_BLOCK mailbox command failed status add_status mbx status The list of XRI SGEs failed to be registered with the firmware.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2566: Failed to allocate table entry Failed to allocate connection table entry.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2567: Config region 23 has bad signature The driver was unable to read Config Region 23 because it has an invalid signature.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2568: Config region 23 has bad version The driver was unable to read Config Region 23 because it is an invalid version.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2569: lpfc_dump_fcoe_param: memory allocation failed Memory allocation has failed.

Data: None Severity: Log: Action: Warning LOG_MBOX None required.

2570: Failed to read FCoE parameters The driver failed to read FCoE parameters.

Data: None Severity: Log: Action: Error LOG_MBOX, LOG_INIT None required.

Chapter 4: Troubleshooting Log Messages Broadcom - 109 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2572: Failed allocate memory for fast-path per-EQ handle array Failed to allocate memory for the fast-path per-EQ handle array.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2573: Failed allocate memory for msi-x interrupt vector entries The driver was unable to allocate memory during initialization of the MSI-X interrupt array.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2574: Not enough EQs () from the pci function for supporting FCP EQs () Failed to create the minimum fast-path event queues.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2575: Not enough EQs () from the pci function for supporting the requested FCP EQs (), the actual FCP EQs can be supported: The driver was not configured with enough fast-path event queues.

Data: None Severity: Log: Action: Warning LOG_INIT None required.

2576: Failed allocate memory for fast-path EQ record array Failed to allocate memory for the fast-path EQ record array.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2577: Failed allocate memory for fast-path CQ record array Failed to allocate memory for the fast-path EQ record array.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2578: Failed allocate memory for fast-path WQ record array Failed to allocate memory for the fast-path EQ record array.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2579: Slow-path wqe consume event carries miss-matched qid: wcqe-qid=, sp-qid= The consumed entry does not have the queueID of the slow-path.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

Broadcom - 110 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2580: Fast-path wqe consume event carries miss-matched qid: wcqe-qid=.

The consumed entry does not have the queueID of the fast path.

Data: None Severity: Log: Action: Warning LOG_SLI None required.

2581: Not enough WQs () from the pci function for supporting FCP WQs () The driver was not configured with the minimum number of fast-path work queues.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2582: Not enough WQs () from the pci function for supporting the requested FCP WQs (), the actual FCP WQs can be supported: The driver was not configured with enough fast-path work queues.

Data: None Severity: Log: Action: Warning LOG_INIT None required.

2593: The FCP EQ count() cannot be greater than the FCP WQ count(), limiting the FCP EQ count to The fast-path event queue cannot be greater than the fast-path work queue count.

Data: None Severity: Log: Action: Warning LOG_INIT None required.

2597: Mailbox command () cannot issue Synchronou(2) sli_flag (3) flag Data: Severity: Log: Action: None Error LOG_MBOX, LOG_SLI None required.

2598: Adapter Link is disabled.

The adapter link has been disabled.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2599: Adapter failed to issue DOWN_LINK mbox command rc The driver was unable to issue the DOWN_LINK mailbox command.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2600: lpfc_sli_read_serdes_param failed to allocate mailbox memory Failed to allocate mailbox memory.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 111 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2605: lpfc_dump_static_vport: memory allocation failed Failed to allocate mailbox memory.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

2606: No NPIV Fabric support No NPIV Fabric support.

Data: None Severity: Log: Action: Error LOG_ELS None required.

2607: Failed to allocate init_vpi mailbox Failed to allocate the INIT_VPI mailbox command.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

2608: Failed to issue Init VPI mailbox The driver was unable to send an INIT_VPI mailbox command.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

2609: Init VPI mailbox failed The INIT_VPI mailbox command failed.

Data: None Severity: Log: Action: Error LOG_MBOX None required.

2610: UNREG_FCFI mbox allocation failed Failed to allocate mailbox memory.

Data: None Severity: Log: Action: Error LOG_DISCOVERY, LOG_MBOX None required.

2611: UNREG_FCFI issue mbox failed Could not issue the UNREG_FCFI mailbox command.

Data: None Severity: Log: Action: Error LOG_DISCOVERY, LOG_MBOX None required.

2619: Config region 23 has bad signature Configuration region 23 has an invalid signature.

Data: None Severity: Log: Action: Error LOG_INIT None required.

Broadcom - 112 Chapter 4: Troubleshooting Log Messages

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2620: Config region 23 has bad version Configuration region 23 has an invalid version.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2621: Failed to allocate mbox for query firmware config cmd Failed to allocate mailbox memory.

Data: None Severity: Log: Action: Error LOG_INIT None required.

2622: Query Firmware Config failed mbx status , status add_status Could not read the firmware configuration.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2623: FCoE Function not supported by firmware. Function mode = > FCoE is not supported by this firmware.

Data: None Severity: Log: Action: Error LOG_SLI Use the Emulex OneCommand Manager application to update to the latest firmware.

2707: Ring handler: Failed to allocate iocb Rctl Type received The driver was unable to allocate memory to send a query config mailbox command.

Data: None Severity: Log: Action: Error LOG_SLI None required.

2717: CT context array entry [] over-run: oxid:, sid: All of the array slots to hold buffers that are passed to the application are in use.

Data: None Severity: Log: Action: Warning LOG_ELS None required.

2718: Clear Virtual Link Received for VPI tag A Clear virtual link was received from the fabric for this VPI.

Data: None Severity: Log: Action: Error LOG_DISCOVERY None required.

Chapter 4: Troubleshooting Log Messages Broadcom - 113 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 9 LPFC Error Log Messages and their Descriptions (Continued)

2719: Invalid response length: tgt lun cmnd rsplen The response length for this FCP command is not supported.

Data: None Severity: Log: Action: Error LOG_FCP None required.

2721: ndlp null for oxid %x SID %x\n, icmd->ulpContext, dfchba->ct_ctx[tag].SID); The Node value for this SID is not in the node list.

Data: None Severity: Log: Action: Warning LOG_ELS None required.

2726: READ_FCF_RECORD Indicates empty FCF table The driver requested the firmware provide a list of FCF entries to connect to and the firmware responded that the FCF table is empty.

Data: None Severity: Log: Action: Error LOG_INIT None required.

4.4.2

4.4.2.1

4.4.2.2

Ethernet Driver Log Messages for LPe16202/OCe15100 Adapters

The following section describes how to retrieve and interpret Ethernet log messages.

Retrieving Ethernet Driver Log Messages

Ethernet driver messages are logged in the /var/log/messages file. This log file is an ASCII text file and can be viewed and searched with a text editor such as vim. A log file automatically rotates as it gets larger. Rotated log files are named messages.x

, where

x

is an integer.

To search the log file for error messages, at the command prompt type: # cd /var/log # vim messages A message is displayed similar to the following: Aug 15 09:57:48 S74 kernel: Invalid MTU requested. Must be between 256 and 8974 bytes

Ethernet Driver Log Messages and their Descriptions

When reporting an issue with the adapter, check the kernel message log using the dmesg command or the /var/log/messages file, and report any of these entries that may be present. There are three types of Ethernet log messages: error, information, and warning. Ethernet driver warning messages logged by an adapter start with lpnic <

BID

> , where <

BID

> is the PCI bus identifier string. For example: lpnic 0000:0d:00.1: MTU must be between 256 and 9000 bytes.

Broadcom - 114 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

NOTE

In the following table, <

D

>, <

DD

>, or <

DDD

> refers to decimal values that appear in the log messages, and <

S

> refers to strings.

Table 10 Ethernet Driver Log Messages of Type Ethernet Log Message

initialization failed : Physical link Adapter busy, could not reset FW. Reboot server to activate new FW Adapter in non recoverable error Adapter recovery failed

Description Type

Initialization of the adapter or allocation of a resource for initializing the driver failed. In most cases, this message is accompanied by a more specific error message. Try rebooting the system after power cycling. If the issue persists, this could be a symptom of a hardware issue or corrupted firmware.

Error This is an informational message about a change in physical link status. Information After flashing firmware on the adapter, the adapter is busy activating the new firmware. Reboot the machine for the new firmware to be active.

Error Resetting the adapter failed, as adapter has gone into non-recoverable state. Error The adapter error recovery failed.

Error Adapter recovery succeeded The a apter has successfully recovered after an error.

Information Cannot set rx filter flags Interface is capable of flags only Could not create sysfs group Failed to change RX filter settings on an interface as requested, as interface does not have the capability.

Failed to enable VLAN promiscuous mode Could not get CRC for

region The creation of the flash_fw entry under the /sys/class/net/eth<

x

> failed. The driver is fully functional, but you cannot install later firmware versions on the adapter.

Firmware command failed to enable VLAN promiscuous mode.

Could not get

MSI-x vector(s) Using
vector(s) Could not set PCI DMA Mask Warning Error Error The driver could not get enough information from the adapter to decide whether a region from a firmware image should be flashed. The driver skips updating this section. This is a very unlikely error.

Error Enabled only a subset of MSI-x vectors requested.

Information created

RX queue(s) created
TX queue(s)
VFs are already enabled Debug data event - , , , The operating system call to set the DMA mask failed.

Error An informational message logging number of receive rings was created. Information An informational message logging number of transmit rings was created.

Information Information When unloading the driver while a VF is still assigned to VM, the VFs remain enabled. When the driver is reloaded, this message is logged with the number of VFs that are enabled and active.

These log messages are generated by firmware to identify certain infrequent hardware and firmware events and aid further analysis if required. These messages by themselves are not indicative of any problem. Data from these logs might help Broadcom troubleshoot other problems, if any, that are observed along with these messages.

Information The number of MSI-x vectors enabled for the NIC function.

Information enabled (out of

) MSI-x vector(s) for NIC Enable VLAN promiscuous mode Disabling VLAN promiscuous mode This is an informational log message when the driver enables or disables VLAN promiscuous mode.

Information Broadcom - 115 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 10 Ethernet Driver Log Messages of Type (Continued) Ethernet Log Message

Error detected in the card ERR: sliport status ERR: sliport error1 ERR: sliport error2 An SLIPORT error was detected in the adapter, followed by status and error messages.

Description

Error

Type

Error in mapping PCI BARs Failed to optimize SR-IOV resources FW config: function_mode=

, function_caps=
Initialization of the adapter failed due to an error while mapping PCI BARs.

Error When the adapter is in an SRIOV-capable configuration, driver failed to redistribute the PF pool resources among the PF and requested number of VFs.

Error A message indicating the function mode and function capabilities set on the adapter during initialization.

Information FW dump deleted successfully FW dump initiated successfully FW dump not generated Firmware flashed successfully Previously generated FW dump is successfully deleted.

FW dump is successfully initiated on the adapter.

Failed to initiate FW dump on the adapter.

This is an informational message that the firmware on the adapter has been updated.

Information Information Error Information Firmware load error Firmware load not allowed (interface is down) Firmware update in progress Updating the adapter with new firmware failed. Usually this message is accompanied by a detailed message on the failure.

Error Flashing firmware failed because the status of the interface is down.

Firmware update is in progress on the adapter.

Error Information Flashing firmware file Flashing section type

failed This is an informational message that the firmware in the adapter is being updated with the firmware image in the file indicated.

Information The driver could not get enough information from the FW UFI image to flash a section in the UFI image, after which the FW download will be aborted.

Error Error FW image size should be multiple of 4 The FW UFI image size is invalid. Download the correct FW image and try again.

FW version is This is an informational message to log the FW version loaded on the adapter.

Information INTx request IRQ failed – err Invalid Cookie. FW image may be corrupted Invalid digital signature invalid mailbox completion Lancer FW download timed out Link down event type:

The request for INTx interrupt registration failed. The driver is non functional if the INTx interrupt cannot be registered.

Error The firmware image under /lib/firmware/< Copy the proper file and try flashing again.

filename

> does not have the expected cookie. The firmware in this file will not be flashed. Error Updating the adapter with new firmware failed. The firmware image being updated does not have a valid digital signature.

The driver received an unexpected completion status for a firmware command.

Error Error FW command timed out, while downloading FW image.

The reason that the ASIC signaled the link status as down. Possible values are: 0 = Link down due to reasons other than those listed here.

1 = Link down caused by Dynamic Control channel protocol.

3 = Link down triggered by Virtual NIC configuration (for example, a zero bandwidth is assigned to a VNIC).

4 = Link down caused by Ethernet Pause frame flooding.

5 = Link down due to physical thermal temperature going up.

Error Information Broadcom - 116 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 10 Ethernet Driver Log Messages of Type (Continued) Ethernet Log Message Description Type

Link down/Link up LPVID

This is an informational message about a change in link status.

VLAN filter configured in BIOS in multichannel configuration.

Mac address assignment failed for VF

MAC M set on VF
Failed Firmware command fails to add a MAC address to a virtual function during initialization or when requested.

MAC address change to failed Failed to change to new MAC address as requested.

Information Information Error Mac hash table alloc failed Max: txqs

, rxqs
, rss
, eqs
, vfs
Max: uc-macs
, mc-macs
, vlans
An informational message logging maximum resources available to the function. Resources include TX rings, RX rings, RX rings with RSS capability, number of VFs supported, unicast MACs, multicast MACs and VLAN filters, respectively.

Information Memory allocation failure The hash table to remember learned MAC addresses could not be allocated.

The driver could not allocate the memory required for the requested operation.

Error Error Memory allocation failure during GET_MAC_LIST Failed to allocate memory to issue GET_MAC_LIST FW command.

Warning Error Missing digital signature Module param rx_frag_size must be 2048/4096/8192.

Using 2048 MSIx enable failed MAC address changed to A request for enabling MSIx interrupts registration failed. The driver will then use INTx interrupts.

Warning MAC address is changed successfully as requested.

Information Warning MSIX request IRQ failed –err The request for MSI-X interrupt registration failed. The driver will use INTx interrupts.

MTU changed from to bytes This is an informational message that the MTU value changed as requested.

Information MTU must be between 256 and 9000 bytes Updating the adapter with new firmware failed. The firmware image being updated does not have a digital signature.

An unsupported receive buffer size was passed for the rx_frag_size module parameter. The driver ignores the specified value and uses the default RX buffer size of 2048.

Error Warning A request to change the MTU was issued with an invalid MTU value. The request failed and MTU will not be changed.

Warning opcode - failed: status

-
A firmware command with opcode failed with the indicated status code and extended status code.

Error PCIe error reporting enabled POST timeout; stage=

PCIe error reporting is successfully enabled on this function.

The power-on self-test of the adapter failed. This is an indication of a hardware or firmware issue. Try rebooting the system after a reset.

Information Error Previous dump not cleared, not forcing dump Initiating dump on the adapter failed, because the previous dump is present. Clear the previous dump (using the ethtool -W eth<

x

> 2 command) before initiating new dump.

Error queue_setup failed Reboot server to activate new FW Firmware command failed to create the requested number of queues.

Error After upgrading the firmware when virtual functions are loaded, the driver cannot reset the adapter to activate the new firmware. Reset the server to activate new firmware after upgrading the firmware on the adapter successfully. Information Information Resetting adapter to activate new FW After upgrading the FW image on the adapter, driver is resetting the adapter to activate new FW without resetting the server. Setting HW VLAN filtering failed Adding a VLAN filter to HW failed.

SRIOV enable failed Could not enable SRIOV because the call to enable SRIOV failed.

Error Error Broadcom - 117 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 10 Ethernet Driver Log Messages of Type (Continued) Ethernet Log Message Description Type

TX-rate must be between 100 and Mbps Request to change transmission rate was issued with an invalid TX rate value. The request failed, and speed will not be changed.

TX-rate must be a multiple of Mbps Transmission rate on a virtual function should be given as multiples of %1Mb/s of link speed.

TX-rate setting not allowed when link is down Transmission rate on a virtual function cannot be modified when the physical link is down.

Error Error Error Tx-rate setting of Mbps on VF

failed : The firmware command failed to change transmission rate as requested.

Error txq: cleaning pending tx-wrbs The driver did not get completions for some transmit requests from the adapter while unloading the driver. This usually indicates an issue with the adapter.

Error Flash image is not compatible with adapter The requested UFI image is not compatible with the chip on which the flash was requested.

Error Unknown debug event

Other than QNQ type debug event, the driver logs as unknown for other asynchronous debug events received.

Warning Unqualified SFP+ detected on from part no: The SFP module indicated in the message is not qualified or supported by Broadcom.

Information Unrecoverable Error detected in the adapter Please reboot server to recover UE LOW: bit set UE HIGH: bit set An unrecoverable error is detected in the adapter that requires a reboot to recover. Low and high bits are set in the data path in which error occurred.

Error User has aborted FW download The user requested abort when FW download is in progress.

Error Information Using profile

VF
has FILTMGMT privilege VF is not privileged to issue opcode
-
VF does not have enough privileges to issue opcode mentioned in the log message.

VFs are assigned to VMs: not disabling VFs An informational message of the profile type currently enabled in the adapter.

VF is provided with FILTMGMT privilege to program MAC/VLAN filters.

Do not disable virtual functions on the port during driver unload, on which VFs are assigned to guests that are powered ON.

Information Warning Warning VF setup failed Failed to create VFs as FW commands failed to provide required resources.

VLAN config on VF

failed Firmware command failed to set VLAN filter as requested.

Waiting for FW to be ready after EEH reset After a PCI EEH reset, wait until firmware becomes ready.

Error Error Information Waiting for POST aborted Waiting for POST, s elapsed Waiting for power-on self-test of the adapter is aborted.

Error This is an informational log that logs the seconds elapsed while waiting for power-on self-test of the adapter.

Information Disable/re-enable i/f in VM to clear Transparent VLAN tag After clearing transparent VLAN tagging for a VF, disable and reenable the VF interface in guest operating system to clear VLAN-tagging for the traffic from guest.

Warning Cannot disable VFs while they are assigned Cannot disable VFs on the PF, when any of the VF is assigned to guest. Detach any VFs from guest to disable VFs on that PF.

Error Invalid FW UFI file The firmware UFI file is corrupted. Try flashing after copying correct UFI file.

Error RSS hash key is longer than

bytes The requested RSS hash key is longer than 40 bytes. Request for a 40-byte RSS hash key.

Error Broadcom - 118 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Chapter 4: Troubleshooting Log Messages

Table 10 Ethernet Driver Log Messages of Type (Continued) Ethernet Log Message

Invalid RSS hash key format RSS hash key is too short (

<
) version is

Description

The hash key format must be in xx:yy:zz:aa:bb:cc format, meaning both the nibbles of a byte should be mentioned even if a nibble is zero.

Error

Type

The requested RSS hash key is less than 40 bytes. Request only a 40-byte RSS hash key.

Error This is an informational message of the loaded lpnic driver version.

Information MGMT_STATUS_FLASHROM_SAVE_FAILED (0x17) / MGMT_ADDI_STATUS_DIGITAL_SIGNATURE_ MISSING(0x57) The firmware being used for flash is not digitally signed.

MGMT_STATUS_FLASHROM_SAVE_FAILED (0x17) / MGMT_ADDI_STATUS_DIGITAL_SIGNATURE_ INVALID (0x56) The digital signature on the firmware being used for flash cannot be verified.

Error Error Broadcom - 119 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Appendix A: Ethtool -S Option Statistics for LPe16202/OCe15100 Adapters in NIC Mode

Appendix A: Ethtool -S Option Statistics for LPe16202/OCe15100 Adapters in NIC Mode

The following table contains a list of ethtool -S option statistics and their descriptions.

Table 11 Ethtool -S Option Statistics Name

dma_map_errors eth_red_drops forwarded_packets jabber_events link_down_reason pmem_fifo_overflow_drop rx_address_filtered rx_alignment_symbol_errors rx_compl_err rx_control_frames rx_crc_errors rx_dropped_header_too_small rx_dropped_runt rx_dropped_tcp_length rx_dropped_too_short rx_dropped_too_small rx_drops_mtu rx_drops_no_erx_descr rx_drops_no_pbuf rx_drops_no_tpre_descr rx_drops_too_many_frags rx_frame_too_long rx_in_range_errors rx_ip_checksum_errs, rx_tcp_checksum_errs, rx_udp_checksum_errs

Description

The number of packets dropped due to DMA mapping errors.

Received packets dropped due to the ASIC’s Random Early Drop policy.

The number of packets generated by the ASIC internally. These packets are not handed to the host. This counter is shared across ports and all functions (NIC/FCoE/iSCSI).

The number of jabber packets received. Jabber packets are packets that are longer than the maximum size Ethernet frames and that have bad CRC.

The reason that the ASIC signaled the link status as down. The various values are:  0 – Link down due to reasons other than those listed here.

  1 – Link down caused by Dynamic Control channel protocol.

3 - Link down triggered by Virtual NIC configuration (for example: zero bandwidth assigned to a VNIC).

  4 – Link down caused by Ethernet Pause frame flooding.

5 – Link down due to physical thermal temperature going up.

Received packets dropped when an internal FIFO going into main packet buffer tank (PMEM) overflows.

Received packets dropped when they do not pass the unicast or multicast address filtering.

The number of packets dropped due to L1 alignment errors. This counter is on a per-port basis.

The number of RX completion errors received.

The number of control frames received.

The number of packets dropped due to CRC errors.

Received packets dropped when the IP header length field is less than 5.

Dropped receive packets due to runt packets (for example, packets shorter than the Ethernet standard).

Received packets dropped when the TCP header length field is less than 5 or the TCP header length + IP header length is more than IP packet length.

Received packets dropped when IP length field is greater than the actual packet length.

Received packets dropped when IP packet length field is less than the IP header length field.

Received packets dropped when the frame length is more than 9018 bytes.

Received packets dropped due to the input receive buffer descriptor FIFO overflowing.

Packets dropped due to lack of available HW packet buffers used to temporarily hold the received packets.

Packets dropped because the internal FIFO to the offloaded TCP receive processing block is full. This could happen only for offloaded iSCSI or FCoE traffic.

Received packets dropped when they need more than 8 receive buffers. This counter is always 0.

Received packets dropped when they are longer than 9216 bytes.

Received packets dropped when the Ethernet length field is not equal to the actual Ethernet data length.

Packets dropped due to TCP, IP, or UDP checksum errors.

Broadcom - 120 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Appendix A: Ethtool -S Option Statistics for LPe16202/OCe15100 Adapters in NIC Mode

Table 11 Ethtool -S Option Statistics (Continued) Name

rx_out_range_errors rx_pause_frames rx_priority_pause_frames rx_switched_unicast_packets, rx_switched_multicast_packets, rx_switched_broadcast_packets rxpp_fifo_overflow_drop, rx_input_fifo_overflow_drop tx_controlframes tx_dma_err tx_hdr_parse_err tx_internal_parity_err tx_pauseframes tx_priority_pauseframes tx_qinq_err tx_spoof_check_err tx_tso_err

Description

Received packets dropped when their length field is >= 1501 bytes and <= 1535 bytes.

The number of Ethernet pause frames (flow control) received.

The number of Ethernet priority pause frames (priority flow control) received per port.

The number of unicast, multicast, and broadcast packets switched internally.

The number of received packets dropped when a FIFO for descriptors going into the packet demux block overflows. In normal operation, this FIFO must never overflow.

The number of Ethernet control frames transmitted per port.

The number of errors occurred in the DMA operation associated with the transmit request from the host to the device.

The number of errors while parsing the packet header of a transmit request.

The number of parity errors in the transmit request.

The number of Ethernet pause frames (flow control) transmitted per port.

The number of Ethernet priority pause frames transmitted per port.

The number of transmit requests with Q-in-Q style VLAN tagging, when such tagging is not expected on the outgoing interface.

The number of spoof TX request failures, when MAC or VLAN spoof checking is enabled on the interface.

The number of transmit request errors, while performing TSO offload.

The following table contains a list of transmit/receive statistics per receive queue basis.

Table 12 Transmit/Receive Queue Statistics Statistic

rxq:rx_bytes rxq:rx_pkts rxq:rx_compl rxq:rx_mcast_pkts rxq:rx_post_fail rxq:rx_drops_no_skbs rxq:rx_drops_no_frags txq:tx_compl txq:tx_bytes txq:tx_pkts txq:tx_reqs txq:tx_stops txq:tx_drv_drops

Description

The number of bytes received by the driver.

The number of packets received by the driver.

The number of receive completions signaled to the driver by the ASIC.

The number of multicast packets received by the driver.

The number of times the driver could not post received buffers to the ASIC.

The number of times the driver could not allocate socket buffers.

Packets dropped due to insufficient buffers posted by the driver.

The number of transmit completions signaled by the ASIC.

The number of bytes transmitted by the driver.

The number of packets transmitted by the driver.

The number of transmit request generated by the driver.

The number of times the driver requests the host to stop giving further transmit requests because the hardware transmit queue is filled up.

The number of transmit packets dropped by the driver.

Broadcom - 121 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Appendix B: lpfc Driver BlockGuard Functionality Overview

Appendix B: lpfc Driver BlockGuard Functionality

This appendix describes how to enable BlockGuard and set lpfc driver module parameters.

B.1

Overview

The BlockGuard feature checks the integrity of data read from and written to the host to the disk and back through the SAN. This check is implemented through the Data Integrity Field (DIF) defined in the ANSI T10 standard.

The Emulex lpfc driver supports T10 DIF Type 1. In the Type 1 implementation, the 8-byte DIF consists of a Ref Tag (or LBA), an App Tag, and a Guard Tag (or CRC). A Type 1 DIF is defined as a having a 2-byte Guard Tag, a 2-byte App tag, and a 4-byte Ref tag, which consists of the lower 32 bits of the logical block address.

The following figure shows a data block (with a 512 byte sector) with the 8-byte footer attached to the end. The contents of the 8-byte footer are shown with the fields that make up the Type 1 DIF; the Guard Tag, the App Tag, and the Ref Tag. The App Tag is not used by the lpfc driver.

Figure 1 Data Block showing Type 1 DIF

When data is written, the DIF is generated by the Host, or by the adapter, based on the block data and the logical block address. The DIF field is added to the end of each data block, and the data is sent through the SAN to the storage target. The storage target validates the CRC and Ref tag and, if correct, stores both the data block and DIF on the physical media. If the CRC does not match the data, then the data was corrupted during the write. A Check Condition is returned back to the host with the appropriate error code. The host records the error and retransmits the data to the target. In this way, data corruption is detected immediately on a write and never committed to the physical media. On a read, the DIF is returned along with the data block to the host, which validates the CRC and Ref tags. Because this validation is done by the hardware, it adds a very small amount of latency to the I/O.

The format of the Guard Tag can optionally be an IP Checksum instead of the CRC mandated by T10 DIF. This can be beneficial because the Initiator Host uses less CPU overhead to generate an IP Checksum than it does with a CRC. The IP Checksum is typically passed as the Guard Tag between the Initiator Host and the adapter. The adapter hardware will translate the IP Checksum into a CRC, or visa versa, on data being sent/received on the wire. The CRC is called a DIF protection type, and the IP Checksum is referred to as DIX protection type.

Broadcom - 122 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016

B.2

Enabling BlockGuard

Appendix B: lpfc Driver BlockGuard Functionality Enabling BlockGuard BlockGuard is disabled by default. To enable it, the parameter lpfc_enable_bg must be passed to the driver as follows: insmod lpfc.ko lpfc_enable_bg=1 For a permanent configuration that will persist across system reboots, create the /etc/modprobe.d/lpfc.conf

file, and place the following line into it: options lpfc lpfc_enable_bg=1 Additional module parameters may be added to this line, separated by spaces.

B.3

SCSI Command Support – SCSI Operation Codes

When there are both READ and WRITE requests, the command descriptor block (CDB) passed to the adapter from the Initiator Host has a read protect/write protect (RDPROTECT/WRPROTECT) field that indicates to the target whether to perform data integrity verification. It also indicates whether to transfer protection data between initiator and target. The adapter does not know if a target supports protection information or with which type of protection it is formatted. The Initiator Host, which has this knowledge, will always prepare a CDB with the appropriate RDPROTECT/WRPROTECT information, depending on target format and capabilities. The request will also include information about with which protection type the target has been formatted.

In addition, the Initiator Host will also provide the adapter with an operation code that tells the controller how to place the protection data for the type of I/O to perform. Each I/O is logically a two-step process. The data is transferred between the Initiator Host and the adapter (over the PCI bus) and between the adapter and the target (over the SAN) as shown in the following figure. The type of operation defines whether the data transfer has protection data.

Figure 2 Data Transfer between Initiator Host and the Adapter

Broadcom - 123 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Appendix B: lpfc Driver BlockGuard Functionality lpfc Driver Module Parameters The initiator operations are listed in the following table.

Table 13 Initiator Operations Initiator Operation

NORMAL READ_INSERT

Initiator Host <-> Adapter

Unprotected Protected

Adapter <-> Target

Unprotected Unprotected READ_PASS READ_STRIP Protected Unprotected WRITE_INSERT Unprotected WRITE_PASS WRITE_STRIP Protected Protected Protected Protected Protected Protected Unprotected

Comment

Used for unprotected I/O.

Reads the data from the target. The adapter then generates the protection data and transfers both data and protection data to the Initiator host. No protection data is sent on the SAN. The adapter can insert the protection data guard tag as CRC or IP CSUM.

Reads the data and protection data from the target on the SAN. The adapter will verify data integrity and transfer both data and protection data to the Initiator host. The adapter can convert the protection data guard tag from CRC to IP CSUM.

Reads data and protection data from the target. The adapter will verify data integrity, discard protection data, and only transfers the data to the Initiator host. It does not send the protection data to the Initiator host. Protection data is only sent on the SAN.

Transfers the data from the Initiator host. The adapter will then generate protection data and write both the data and protection data to the target. Protection data is only sent on the SAN.

Transfers the data and protection data from the Initiator host to the adapter. The adapter will verify protection data and write both data and protection data to the target on the SAN. The adapter can convert the protection data guard tag from IP CSUM to CRC.

Transfers data and protection data from the Initiator host. The adapter will verify data integrity, discard protection data, and writes only the data to the target. No protection data is sent on the SAN.

B.4

B.4.1

lpfc Driver Module Parameters

The lpfc driver has two module parameters: lpfc_prot_mask and lpfc_prot_guard . Using these parameters, you can control which DIF capabilities the lpfc driver registers with the Linux SCSI subsystem. This, in turn, controls which initiator operations (BlockGuard profiles) are used during I/O operations. These parameters are set up when the driver loads and cannot be changed while the driver is running.

lpfc_prot_mask

This parameter controls the DIF operations that the driver registers with the operating system. The operating system selects an operation to use for each I/O command that matches the adapter DIF capability. The driver indicates its capabilities by the operations it registers with the operating system.

If the parameter is not passed to the driver, the default DIX_TYPE0 is used.

Broadcom - 124 -

Emulex Drivers for Linux for LightPulse Adapters User Guide December 30, 2016 Appendix B: lpfc Driver BlockGuard Functionality lpfc Driver Module Parameters The SCSI layer will typically use the bit masks listed in the following table to determine how to place the protection data associated with I/Os to the SCSI Host. The default value for lpfc_prot_mask is to allow all of the options.

Table 14

lpfc_prot_mask

Protection Types Flag

SHOST_DIF_TYPE1_PROTECTION SHOST_DIX_TYPE0_PROTECTION SHOST_DIX_TYPE1_PROTECTION 1 8 16

Value Indicates

Adapter supports T10 DIF Type 1 Adapter supports DIX Type 0 Adapter supports DIX Type 1

Description

Adapter to target Type 1 protection Host to adapter protection only Host to adapter Type 1 protection The following table shows how protection data gets placed for each supported profile.

Table 15 Protection Data Placement for Supported Profiles Flag

SHOST_DIF_TYPE1_PROTECTION SHOST_DIX_TYPE0_PROTECTION SHOST_DIX_TYPE1_PROTECTION SHOST_DIF_TYPE1_PROTECTION 1 8 17

Value

A1 AST2 AST1 / C1

BlockGuard Profile Operation

READ_STRIP / WRITE_INSERT READ_INSERT / WRITE_STRIP READ_PASS / WRITE_PASS

B.4.2

lpfc_prot_guard

This parameter correlates to the SCSI_host_guard_type of the Linux kernel. This parameter specifies the type of CRC the Linux operating system will pass to the lpfc driver. The following table shows the two guard types: CRC and IP-CSUM, with values of 0x1 and 0x2, respectively.

Table 16

lpfc_prot_guard

Guard Types Flag

SHOST_DIX_GUARD_CRC SHOST_DIX_GUARD_IP 1 2

Value Indicates

Adapter supports T10 DIF CRC Adapter supports both T10 DIF CRC and IP-CSUM The default value for lpfc_prot_guard is SHOST_DIX_GUARD_IP. This value defines the format for the guard tag when the data is transferred between the Host and the adapter. When data is transferred on the wire, the protection data guard tag is always translated into a T10 DIF CRC. To override the default, you can pass a module parameter value with either insmod or modprobe .

The SCSI layer will typically use an IP-CSUM as the method for computing the protection data guard tag because it uses less CPU overhead.

Broadcom - 125 -

Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement