Intel® Omni-Path Fabric Software — Installation Guide

Intel® Omni-Path Fabric Software — Installation Guide
Intel® Omni-Path Fabric Software
Installation Guide
November 2015
Order No.: H76467-1.0
You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described
herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed
herein.
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and
roadmaps.
The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications.
Current characterized errata are available on request.
Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or visit http://
www.intel.com/design/literature.htm.
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn
more at http://www.intel.com/ or from the OEM or retailer.
No computer system can be absolutely secure.
Intel, the Intel logo, Intel Xeon Phi, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
Copyright
©
2015, Intel Corporation. All rights reserved.
Intel® Omni-Path Fabric Software
Installation Guide
2
November 2015
Order No.: H76467-1.0
Revision History—Intel® Omni-Path Fabric
Revision History
Date
Revision
November 2015
1.0
Document has been updated for Revision 1.0.
September 2015
0.7
Document has been updated for Revision 0.7.
April 2015
0.5
Initial Alpha release of document.
November 2015
Order No.: H76467-1.0
Description
Intel® Omni-Path Fabric Software
Installation Guide
3
Intel® Omni-Path Fabric—Contents
Contents
Revision History..................................................................................................................3
Preface............................................................................................................................... 9
Intended Audience....................................................................................................... 9
Documentation Set.......................................................................................................9
Documentation Conventions........................................................................................ 10
License Agreements....................................................................................................10
Technical Support.......................................................................................................11
1.0 Overview.................................................................................................................... 12
1.1 Installation Packages Overview...............................................................................12
1.1.1 Intel® Omni-Path Fabric Host Software........................................................ 12
1.1.2 Intel® Omni-Path Fabric Suite Software....................................................... 15
1.1.3 Intel® Omni-Path Fabric Suite Fabric Manager GUI........................................ 15
2.0 Fabric Software Pre-Installation................................................................................. 16
2.1 Installation Prerequisites........................................................................................16
3.0 Download and Extract Installation Packages.............................................................. 19
3.1 Download the Intel® Omni-Path Software.................................................................19
3.2 Unpack the Tar File............................................................................................... 19
4.0 Install the Intel® Omni-Path Fabric Software............................................................. 20
4.1 Before You Install................................................................................................. 20
4.2 Intel® Omni-Path Software Installation Using the Install TUI.......................................20
4.3 Intel® Omni-Path Software Installation Using CLI Commands..................................... 23
4.3.1 Install Command...................................................................................... 24
4.4 IPoIB IPV4 Configuration Using the Software TUI...................................................... 27
4.5 IPoIB IPV4 Configuration Using CLI Commands.........................................................28
4.6 Install IPoIB IPV6................................................................................................. 28
5.0 Configure Intel® Omni-Path Chassis........................................................................... 30
5.1
5.2
5.3
5.4
Intel Chassis Configuration Pre-requisites.................................................................30
Configure Chassis Using Intel® Omni-Path Fabric Suite FastFabric............................... 31
Configure Chassis Using CLI Commands...................................................................41
Verify Chassis Configuration Using CLI Commands.................................................... 43
6.0 Configure Firmware on the Externally Managed Intel® Omni-Path Switches............... 46
6.1 Switch Configuration Pre-Requisites........................................................................ 46
6.2 Configure Externally Managed Switches................................................................... 46
7.0 Install Host Software on the Remaining Hosts Using the FastFabric TUI.....................54
8.0 Install Host Software on the Remaining Hosts using CLI Commands.......................... 60
9.0 Verify Intel® Omni-Path Fabric Host Software on the Remaining Servers................... 63
10.0 Verify Intel® Omni-Path Fabric Host Software is installed on the Remaining
Servers using CLI Commands.................................................................................. 68
11.0 Installation of Additional Fabric Management Nodes................................................ 72
Intel® Omni-Path Fabric Software
Installation Guide
4
November 2015
Order No.: H76467-1.0
Contents—Intel® Omni-Path Fabric
12.0 Configure and Initialize Health Check Tools.............................................................. 74
13.0 Running High Performance Linpack.......................................................................... 75
14.0 Install Intel® Omni-Path Fabric Suite Fabric Manager GUI........................................ 77
14.1 Windows* Installation..........................................................................................77
14.1.1 System Requirements for a Windows* Environment..................................... 77
14.1.2 Download the Intel® Omni-Path Fabric Suite Fabric Manager GUI.................. 77
14.1.3 Use the Installation Wizard to Install the Fabric Manager GUI........................ 78
14.2 Linux* Installation...............................................................................................81
14.2.1 System Requirements for a Linux* Environment..........................................81
14.2.2 Download the Intel® Omni-Path Fabric Suite Fabric Manager GUI.................. 81
14.2.3 Use RPM to Install the Fabric Manager GUI................................................. 81
14.3 Start the Fabric Manager GUI Application............................................................... 82
14.3.1 Windows* Procedure............................................................................... 82
14.3.2 Linux* Procedure.................................................................................... 82
14.4 Configure Startup Options.................................................................................... 82
14.5 Uninstall the Intel® Omni-Path Fabric Suite Fabric Manager GUI................................ 83
14.5.1 Windows* Procedure............................................................................... 83
14.5.2 Linux* Procedure.................................................................................... 83
15.0 Installation Verification and Additional Settings....................................................... 84
15.1
15.2
15.3
15.4
LED Link and Data Indicators................................................................................84
Adapter and Other Settings.................................................................................. 84
ARP Neighbor Table Setup for Large Clusters.......................................................... 84
SM Loop Test......................................................................................................85
16.0 Configuration............................................................................................................ 86
16.1 Virtual Fabrics.................................................................................................... 86
16.1.1 Virtual Fabrics, Switch Configuration.......................................................... 86
16.1.2 Virtual Fabrics, Fabric Manager Configuration.............................................. 86
16.1.3 Virtual Fabrics, Fabric Host Software Configuration...................................... 87
16.1.4 Virtual Fabrics, Application, and ULP Configuration.......................................88
16.1.5 Virtual Fabrics, Fabric Manager GUI Configuration........................................89
16.2 Congestion Analysis.............................................................................................89
16.2.1 Congestion Analysis, Switch Configuration.................................................. 89
16.2.2 Congestion Analysis, Fabric Manager Configuration...................................... 90
16.2.3 Congestion Analysis, Management Node Configuration................................. 90
16.3 Adaptive Routing.................................................................................................90
16.4 Adaptive Routing, Switch Configuration.................................................................. 90
16.5 Dispersive Routing.............................................................................................. 90
16.5.1 Dispersive Routing, Fabric Manager Configuration........................................91
16.5.2 Dispersive Routing, PSM Configuration....................................................... 91
16.6 Distributed SA Provider........................................................................................ 91
16.6.1 Distributed SA Provider, Fabric Manager Configuration................................. 91
16.6.2 Distributed SA, Fabric Host Software Configuration...................................... 91
16.6.3 Distributed SA Provider, Application and ULP Configuration........................... 91
17.0 Upgrade the Management Node................................................................................ 93
17.1 Preinstallation.....................................................................................................93
17.2 Intel® Omni-Path Fabric Suite Upgrade.................................................................. 93
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
5
Intel® Omni-Path Fabric—Contents
18.0 Upgrade the Fabric Software.....................................................................................96
19.0 Upgrade from IntelOPA-Basic to IntelOPA-IFS..........................................................99
Appendix A Multi-Subnet Fabrics.................................................................................... 101
A.1 Primarily Independent Subnets............................................................................. 101
A.2 Overlapping Subnets........................................................................................... 103
Appendix B Client/Server Configuration to Boot Over Fabric.......................................... 106
B.1 Server Configuration........................................................................................... 106
B.1.1 Configuring IFS.......................................................................................106
B.1.2 Installing a DHCP Server.......................................................................... 106
B.1.3 Installing a TFTP Server........................................................................... 107
B.1.4 Installing a PXE Server............................................................................ 107
B.2 Intel® Omni-Path Boot Client................................................................................108
Appendix C IFS Software Installation Checklist............................................................. 109
C.1 Pre-Installation...................................................................................................109
C.2 Install the Intel® Omni-Path Fabric Software on a Cluster.........................................109
Intel® Omni-Path Fabric Software
Installation Guide
6
November 2015
Order No.: H76467-1.0
Figures—Intel® Omni-Path Fabric
Figures
1
FastFabric OPA Switch Setup/Admin Menu.................................................................. 47
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
7
Intel® Omni-Path Fabric—Tables
Tables
1
Sample Fabric Manager vFabric Combinations..............................................................87
Intel® Omni-Path Fabric Software
Installation Guide
8
November 2015
Order No.: H76467-1.0
Preface—Intel® Omni-Path Fabric
Preface
This manual is part of the documentation set for the Intel® Omni-Path Fabric (Intel®
OP Fabric), which is an end-to-end solution consisting of adapters, edge switches,
director switches and fabric management and development tools.
The Intel® OP Fabric delivers a platform for the next generation of High-Performance
Computing (HPC) systems that is designed to cost-effectively meet the scale, density,
and reliability requirements of large-scale HPC clusters.
Both the Intel® OP Fabric and standard InfiniBand* are able to send Internet Protocol
(IP) traffic over the fabric, or IPoFabric. In this document, however, it is referred to as
IP over IB or IPoIB. From a software point of view, IPoFabric and IPoIB behave the
same way and, in fact, use the same ib_ipoib driver to send IP traffic over the ib0
and/or ib1 ports.
Intended Audience
The intended audience for the Intel® Omni-Path (Intel® OP) document set is network
administrators and other qualified personnel.
Documentation Set
The following are the list of the complete end-user publications set for the Intel®
Omni-Path product. These documents can be downloaded from https://
downloadcenter.intel.com/.
•
•
November 2015
Order No.: H76467-1.0
Hardware Documents:
—
Intel® Omni-Path Fabric Switches Hardware Installation Guide
—
Intel® Omni-Path Fabric Switches GUI User Guide
—
Intel® Omni-Path Fabric Switches Command Line Interface Reference Guide
—
Intel® Omni-Path Edge Switch Platform Configuration Reference Guide
—
Intel® Omni-Path Fabric Managed Switches Release Notes
—
Intel® Omni-Path Fabric Externally-Managed Switches Release Notes
—
Intel® Omni-Path Host Fabric Interface Installation Guide
—
Intel® Omni-Path Host Fabric Interface Release Notes
Software Documents:
—
Intel® Omni-Path Fabric Software Installation Guide
—
Intel® Omni-Path Fabric Suite Fabric Manager User Guide
—
Intel® Omni-Path Fabric Suite FastFabric User Guide
—
Intel® Omni-Path Fabric Host Software User Guide
—
Intel® Omni-Path Fabric Suite Fabric Manager GUI Online Help
Intel® Omni-Path Fabric Software
Installation Guide
9
Intel® Omni-Path Fabric—Preface
—
Intel® Omni-Path Fabric Suite Fabric Manager GUI User Guide
—
Intel® Omni-Path Fabric Suite FastFabric Command Line Interface Reference
Guide
—
Intel® Performance Scaled Messaging 2 (PSM2) Programmer's Guide
—
Intel® Omni-Path Fabric Performance Tuning User Guide
—
Intel® Omni-Path Host Fabric Interface Platform Configuration Reference
Guide
—
Intel® Omni-Path Fabric Software Release Notes
—
Intel® Omni-Path Fabric Manager GUI Release Notes
Documentation Conventions
This guide uses the following documentation conventions:
•
Note: provides additional information.
•
Caution: indicates the presence of a hazard that has the potential of causing
damage to data or equipment.
•
Warning: indicates the presence of a hazard that has the potential of causing
personal injury.
•
Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this
guide. Links to Web sites are also shown in blue. For example:
See License Agreements on page 10 for more information.
For more information, visit www.intel.com.
•
Text in bold font indicates user interface elements such as a menu items, buttons,
check boxes, or column headings. For example:
Click the Start button, point to Programs, point to Accessories, and then click
Command Prompt.
•
Text in Courier font indicates a file name, directory path, or command line text.
For example:
Enter the following command: sh ./install.bin
•
Key names and key strokes are shown in underlined bold uppercase letters. For
example:
Press CTRL+P and then press the UP ARROW key.
•
Text in italics indicates terms, emphasis, variables, or document titles. For
example:
For a complete listing of license agreements, refer to the Intel® Software End User
License Agreement.
License Agreements
This software is provided under one or more license agreements. Please refer to the
license agreement(s) provided with the software for specific detail. Do not install or
use the software until you have carefully read and agree to the terms and conditions
of the license agreement(s). By loading or using the software, you agree to the terms
of the license agreement(s). If you do not wish to so agree, do not install or use the
software.
Intel® Omni-Path Fabric Software
Installation Guide
10
November 2015
Order No.: H76467-1.0
Preface—Intel® Omni-Path Fabric
Technical Support
Technical support for Intel® Omni-Path products is available 24 hours a day, 365 days
a year. Please contact Intel Customer Support or visit www.intel.com for additional
detail.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
11
Intel® Omni-Path Fabric—Overview
1.0
Overview
The installation of the Intel® Omni-Path Software is accomplished using a Text User
Interface (TUI) to guide you through the installation process. You also have the ability
to use command line interface (CLI) commands to accomplish the installation or install
rpms individually.
1.1
Installation Packages Overview
The following software installation packages are available for an Intel® Omni-Path
Fabric:
1.1.1
•
Intel® Omni-Path Fabric Host Software – This is the basic installation package that
installs the Intel® Omni-Path Fabric Host Software components needed to set up a
compute node with drivers, stacks, and basic tools for local configuration and
monitoring.
•
Intel® Omni-Path Fabric Suite Software – This installation package provides
special features and includes the Intel® Omni-Path Fabric Host Software (IFS)
package, along with the Intel® Omni-Path Fabric Suite FastFabric Toolset
(FastFabric) and the Intel® Omni-Path Fabric Suite Fabric Manager (Fabric
Manager).
•
Intel® Omni-Path Fabric Suite Fabric Manager GUI – This installation package
provides a number of features for viewing and monitoring the fabric or multiple
fabrics.
Intel® Omni-Path Fabric Host Software
The Intel® Omni-Path Fabric Host Software installation package (IntelOPABasic.DISTRO.VERSION.tgz) includes the following components:
•
OFED OPA Stack
•
OFED IBACM
•
Intel HFI Components
•
OPA Tools
•
OFED OPA Development
•
OFED IP over IB
•
MVAPICH2 (verbs, gcc)
•
OpenMPI (verbs, gcc)
•
GASNet (hfi, gcc)
•
OpenSHMEM (hfi, gcc)
•
MVAPICH2 (hfi, gcc)
•
MVAPICH2 (hfi, PGI)
•
MVAPICH2 (hfi, Intel)
Intel® Omni-Path Fabric Software
Installation Guide
12
November 2015
Order No.: H76467-1.0
Overview—Intel® Omni-Path Fabric
•
OpenMPI (hfi, gcc)
•
OpenMPI (hfi, PGI)
•
OpenMPI (hfi, Intel)
•
MPI Source
•
OFED Debug Info
Note:
There is a separate Intel® Omni-Path Fabric Host Software installation package for
each of the supported Linux* distributions. Refer to the release notes of the package
version being installed for a list of supported Linux* distributions.
1.1.1.1
OS RPMs Installation Prerequisites
In addition to normal OS installation options, the following OS RPMs must be installed
before installing the Intel® Omni-Path software.
Note:
Use the version distributed with your OS distribution.
RHEL*
Install the following packages using yum from the RHEL* distro.
•
libibmad
•
libibverbs
•
librdmacm
•
libibcm
•
qperf
•
perftest
•
rdma
•
infinipath-psm
•
expat
•
elfutils-libelf-devel
•
libstdc++-devel
•
gcc-gfortran
•
atlas
•
tcl
•
expect
•
tcsh
•
sysfsutils
•
pciutils
•
opensm-devel (RHEL* 7.1)
•
opensm-libs (RHEL* 7.1)
For compiling IFS software, the following packages are required.
•
libibverbs-devel
•
libibmad-devel
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
13
Intel® Omni-Path Fabric—Overview
•
librdmacm-devel
•
ibacm-devel
•
openssl-devel (1.0.1 or higher)
•
libuuid-devel
•
expat-devel
•
infinipath-psm-devel
SLES*
Install the following packages using zypper from the SLES * distro.
•
libibmad5
•
libibverbs1
•
librdmacm1
•
libibcm1
•
qperf
•
perftest
•
rdma
•
libexpat1
•
opensm-devel
•
opensm-libs3
•
libpsm_infinipath1
•
libelf-devel
•
libudev-devel
•
texlive-latex
•
texlive-babel-english
•
texlive-psnfss
•
texlive-courier
For compiling IFS software, the following packages are required.
•
libibverbs-devel
•
libibmad-devel
•
librdmacm-devel
•
ibacm-devel
•
libopenssl-devel (1.0.1 or higher)
•
libuuid-devel
•
libexpat-devel
•
infinipath-psm-devel
•
rpm-build
Depending on which packages you choose, there may be additional prerequisites. For
additional information, refer to the Release Notes for your specific release and
installation type.
Intel® Omni-Path Fabric Software
Installation Guide
14
November 2015
Order No.: H76467-1.0
Overview—Intel® Omni-Path Fabric
1.1.2
Intel® Omni-Path Fabric Suite Software
The Intel® Omni-Path Fabric Suite (IntelOPA-IFS.DISTRO.VERSION.tgz)
installation package installs the installation package listed in Intel® Omni-Path Fabric
Host Software on page 12, and requires the same prerequisites listed in OS RPMs
Installation Prerequisites on page 13. It also installs:
•
FastFabric
•
OPA FM
For details on using the Intel® Omni-Path Fabric Suite FastFabric Toolset, refer to the
Intel® Omni-Path Fabric Suite FastFabric User Guide. For details on using the Intel®
Omni-Path Fabric Suite Fabric Manager, refer to the Intel® Omni-Path Fabric Suite
Fabric Manager User Guide.
Note:
There is a separate IFS installation package for each of the supported Linux*
distributions. Refer to the release notes of the version being installed for a list of
supported Linux* distributions.
1.1.3
Intel® Omni-Path Fabric Suite Fabric Manager GUI
The Intel® Omni-Path Fabric Suite Fabric Manager GUI (Fabric Manager GUI)
installation package installs the application for monitoring the fabric from a computer
external to the fabric.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
15
Intel® Omni-Path Fabric—Fabric Software Pre-Installation
2.0
Fabric Software Pre-Installation
This section provides the information and procedures needed prior to installing,
configuring, and verifying the fabric software. Typically, the site implementation
engineer performs the tasks described in this section to ensure that the fabric is ready
for the software installation.
The procedures will be marked with one of the following qualifications when required:
•
(Linux) – Tasks are only applicable when Linux* is being used.
•
(Host) – Tasks are only applicable when Intel® Omni-Path Fabric Host Software
(Intel's OFED Delta RPMs) or Intel® Omni-Path Fabric Suite is being used on the
hosts.
•
(Switch) – Tasks are applicable only when Intel® Omni-Path Switches or Chassis
are being used.
•
(All) – Tasks are generally applicable to all environments.
Note:
Some of the Linux* steps may be applicable to other Unix-like operating systems if it
is required to enable use of non-specific Intel® Omni-Path Fabric Suite FastFabric
Toolset commands (such as opacmdall) against the given hosts.
2.1
Installation Prerequisites
Design of the Fabric
Prior to the installation and setup of the fabric, it is important that the design and
installation of the hardware be planned carefully. The design plan must include the
following information:
•
Identification of servers that will function as the administration or fabric
management nodes, where the Intel® Omni-Path Fabric Suite (IFS) will be
installed.
•
Server memory requirements based on the software being used:
—
IFS, including the Intel® Omni-Path Fabric Suite Fabric Manager, requires at
least 500 MB of physical memory for each Fabric Manager instance. When
managing a cluster of 500 nodes or more, 1GB of memory per Fabric Manager
instance is required.
—
When running multiple Fabric Manager instances on a single management
node, the physical memory requirements should be multiplied by the number
of Fabric Manager instances.
•
Swap disk space allowance should follow recommendations for the given version
of Linux*. Swap space should be twice the size of the physical memory on the
server running the Fabric Manager.
•
Ensure at least one central processing unit (CPU) core is available per Fabric
Manager instance. For example, four Fabric Manager instances on a single
management node would require four CPU cores.
Intel® Omni-Path Fabric Software
Installation Guide
16
November 2015
Order No.: H76467-1.0
Fabric Software Pre-Installation—Intel® Omni-Path Fabric
Set Up the Fabric
The following steps provide the information to set up the fabric. For information about
the configuration files used by FastFabric, refer to Configuration on page 86.
1.
(All) Ensure all hardware is installed:
•
Servers
•
Core and edge switches.
Note: When installing externally managed switches such as the Intel® OmniPath Edge Switch 100 Family without a management module, the Node
GUID is required. The Node GUID can be found on a label on the case of
the switch and will be needed to configure and manage the switches
with the IFS.
2. (All) Ensure an Intel® Omni-Path Host Fabric Interface (HFI) is installed in each
server. Refer to the Intel® Omni-Path Host Fabric Interface Installation Guide for
instructions.
3. (All) The hardware configuration should be reviewed to ensure everything has
been installed according to plan. After the software installation, Intel® Omni-Path
Fabric Suite FastFabric tools may be used to help verify the installation.
4. (Linux) Ensure the required Operating System (OS) version (with the same
kernel version) is installed on all hosts. The fabric management node(s) (hosts
that will run FastFabric) should have a full installation and must include the Tcl
and Expect OS RPMs.
For MPI clusters, install the C and Fortran compilers along with their associated
tools on the fabric management nodes.
Note: Refer to the Intel® Omni-Path Fabric Software Release Notes for a list of
supported OS versions.
5. (Linux) Enable remote login as root on each host. In order for FastFabric to
manage the hosts, the fabric management node must be able to securely log in as
root to each host. This can be accomplished using ssh.
Note: To simplify the use of FastFabric, set up root access on the nodes in the
fabric using ssh. The same root password must be set on all nodes in the
fabric. After root access through ssh has been set up using FastFabric, the
root passwords can be changed.
6. (All) Resolve the TCP/IP Host Names.
FastFabric and TCP/IP must resolve host names to the management network
and/or IPoIB IP addresses. If the management network is not IPoIB, each host
must have both a management network name and an IPoIB network name. To do
this, use the actual host name as the management network name and HOSTNAMEopa as the IPoIB network name, where HOSTNAME is the management network
name of the given host.
Name resolution is accomplished by configuring a DNS server on the management
network, with both management network and IPoIB addresses for each host and
each Intel internally-managed chassis. Alternatively, an /etc/hosts file can be
created on the fabric management node; FastFabric can then propagate
this /etc/hosts file to all the other hosts.
If using the /etc/hosts file approach:
•
November 2015
Order No.: H76467-1.0
On the master node, add all the Ethernet and IPoIB addresses into the /etc/
hosts file.
Intel® Omni-Path Fabric Software
Installation Guide
17
Intel® Omni-Path Fabric—Fabric Software Pre-Installation
•
For the IPoIB convention, use HOSTNAME-opa.
•
The localhost line should not be edited.
•
The /etc/hosts file should not have any node-specific data.
•
Copy the file to every node, as described in Copy /etc/hosts to all hosts on
page 56.
If using Domain Name System (DNS):
•
Refer to the documentation for the domain name system (DNS) server being
used. Make sure to edit the /etc/resolv.conf configuration file on the
fabric management node to use the proper DNS server.
•
Refer to the Linux* OS documentation for more information about configuring
the /etc/resolv.conf file. This file is typically configured during OS
installation.
•
If /etc/resolv.conf must be manually configured for each host, FastFabric
can aid in copying the file to all the hosts.
•
The /etc/resolv.conf file created on the fabric management node must
not have any node-specific data and must be appropriate for use on all hosts.
•
Copying the /etc/resolv.conf file to all the nodes is accomplished during
the OS installation.
•
If the /etc/resolv.conf file was not set up on all the hosts during the OS
installation, the FastFabric Copy a file to all hosts operation can be used
during the Install Host Software on the Remaining Hosts Using the FastFabric
TUI procedures to copy the /etc/resolv.conf file from the fabric
management node to all the other nodes.
7. (All) Set up a Network Time Protocol (NTP) server.
Configure an NTP server for the cluster, and set all Linux* hosts and internally
managed chassis to sync to the NTP server. The setup of the internally managed
chassis is described in Configure Intel® Omni-Path Chassis on page 30.
Intel® Omni-Path Fabric Software
Installation Guide
18
November 2015
Order No.: H76467-1.0
Download and Extract Installation Packages—Intel® Omni-Path Fabric
3.0
Download and Extract Installation Packages
The following sections discuss downloading the software package from an Intel web
page or other Intel secured location, and unpacking the included tar file.
3.1
3.2
Download the Intel® Omni-Path Software
1.
Using a Web browser, type intel.com in the address field and press Enter.
2.
From the selection bar at the bottom of the screen, click on Download Center.
3.
In the "Search downloads" field type Omni-Path and select the search icon.
4.
Select the Omni-Scale product you are wanting to download.
5.
On the Available Downloads page select the file(s) you need for the OSs you have
installed on your fabric.
6.
Follow the instructions to download the file(s)
Unpack the Tar File
Use the following procedure to unpack the tar file.
Basic
IntelOPA-Basic.DISTRO.VERSION.tgz
IFS
IntelOPA-IFS.DISTRO.VERSION.tgz
1.
Open an SSH client session and log into the host where the package is being
installed.
2.
Copy the tar file to the /root directory.
3. Change directory to /root.
cd /root
4.
Unpack the tar file.
Basic
tar xvfz IntelOPA-Basic.DISTRO.VERSION.tgz
IFS
tar xvfz IntelOPA-IFS.DISTRO.VERSION.tgz
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
19
Intel® Omni-Path Fabric—Install the Intel® Omni-Path Fabric Software
4.0
Install the Intel® Omni-Path Fabric Software
This section provides information and procedures to install the Intel® Omni-Path
Fabric on the Management Node. There are two ways of installing the software:
•
Install using the Install TUI (Recommended). Refer to Intel® Omni-Path Software
Installation Using the Install TUI on page 20
•
Install using CLI commands. Refer to Intel® Omni-Path Software Installation Using
CLI Commands on page 23
All pre-installation requirements must be met prior to installing the software. When
installing a fabric/cluster, Intel recommends installing the software on the Fabric
Management Node using the Install TUI, and then using FastFabric to configure the
Fabric Management Node and to install the software on the remaining hosts.
The installation, configuration, and verification procedures provide step-by-step
instructions for a typical, single subnet fabric. For information about the installation
and verification of multiple subnet fabrics, see Multi-Subnet Fabrics on page 101.
4.1
Before You Install
Refer to the Release Notes for a list of compatible OSs.
If the managed cluster has IPoIB settings on the compute nodes that are incompatible
with the Fabric Management Node, do not run IPoIB on the Fabric Management Nodes.
For example, when compute nodes use 12K Maximum Transmission Unit (MTU) and
the Fabric Management Nodes use a 8K MTU, you would not run IPoIB settings on the
Fabric Management Nodes. Refer to the Intel® Omni-Path Fabric Host Software User
Guide, “Intel® Omni-Path Cluster Setup and Administration” section, "MTU Size"
subsection for detailed information.
4.2
Intel® Omni-Path Software Installation Using the Install
TUI
The installation of the Intel® Omni-Path Software (Intel® OPA Software) using the
Install TUI installs the basic software or the IFS software that was downloaded to your
fabric. The following procedure goes step-by-step through the installation and defines
the different items for the basic and IFS installation. When needed, the procedure
specifies Basic or IFS accordingly. This procedure presumes that you are logged in
and have completed all pre-installation requirements.
Perform the following procedure to install the Intel® OPA Software:
1. Change directory to IntelOPA-[Basic|IFS].DISTRO.VERSION directory.
Basic
cd IntelOPA-Basic.DISTRO.VERSION
Intel® Omni-Path Fabric Software
Installation Guide
20
November 2015
Order No.: H76467-1.0
Install the Intel® Omni-Path Fabric Software—Intel® Omni-Path Fabric
IFS
cd IntelOPA-IFS.DISTRO.VERSION
2.
Start the Install TUI:
./INSTALL
The Intel OPA VERSION Software main menu appears.
Intel OPA VERSION Software
1)
2)
3)
4)
6)
Install/Uninstall Software
Reconfigure OFED IP over IB
Reconfigure Driver Autostart
Generate Supporting Information for Problem Report
FastFabric (Host/Chassis/Switch Setup/Admin)
X) Exit
3.
Select 1) Install/Uninstall Software.
Screen 1 of 2 of the Intel OPA Install Menu appears.
Intel OPA Install (VERSION release) Menu
Please Select Install Action (screen
0) OFED OPA Stack
[
Install
1) OFED IBACM
[
Install
2) Intel HFI Components[
Install
3) OPA Tools
[
Install
4) OFED OPA Development[
Install
5) FastFabric
[
Install
6) OFED IP over IB
[
Install
7) OPA FM
[
Install
8) MVAPICH2 (hfi,gcc) [
Install
9) MVAPICH2 (hfi,Intel)[
Install
a) MVAPICH2 (hfi,PGI) [
Install
b) OpenMPI (hfi,gcc)
[
Install
c) OpenMPI (hfi,Intel) [
Install
d) OpenMPI (hfi,PGI)
[
Install
N)
P)
R)
X)
Next Screen
Perform the selected actions
Re-Install All
Return to Previous Menu (or ESC)
1 of 2):
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
I) Install All
U) Uninstall All
Note: The previous menu is an example of the IntelOPA-IFS Install TUI. The
Basic Install TUI would show 6) FastFabric and 8) OPA FM as [Don't
Install].
Note: Intel HFI Components selection contains the enhanced Intel® OmniPath Host Fabric Interface (HFI) driver optimized stack for MPI (PSM) on
HFIs and OpenMPI, as well as user tools.
4.
Review the items to be installed; the default value is in brackets (Install or
Don't Install). To change a value, type the alphanumeric character associated
with the item.
5. Press N to go to the next screen.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
21
Intel® Omni-Path Fabric—Install the Intel® Omni-Path Fabric Software
Screen 2 of 2 of the Intel OPA Install Menu appears.
Intel OPA Install (VERSION release) Menu
Please Select Install Action (screen 2 of 2):
0) GASNet (hfi,gcc)
[
Install
][Available]
1) OpenSHMEM (hfi,gcc) [
Install
][Available]
2) MVAPICH2(verbs,gcc) [
Install
][Available]
3) OpenMPI (verbs,gcc) [
Install
][Available]
4) MPI Source
[
Install
][Available]
5) OFED Debug Info
[Don't Install][Available]
N)
P)
R)
X)
Next Screen
Perform the selected actions
Re-Install All
Return to Previous Menu (or ESC)
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
I) Install All
U) Uninstall All
6. Review the items to be installed; the default value is in brackets (Install or
Don't Install). To change a value, type the alphanumeric character associated
with the item.
7.
Press P to perform the selected actions from the two screens.
The system prompts:
Preparing OFED VERSION release for Install...
Rebuild OFED SRPMs (a=all, p=prompt per SRPM, n=only as needed?) [n]:
8. Select the default by pressing Enter.
The system displays prompts that require your response as it is going through the
installation
9.
For each system prompt, select the default by pressing Enter.
The Intel OPA Autostart Menu appears.
Intel OPA Autostart (VERSION release) Menu
Please Select Autostart Option:
0) OFED OPA Stack ()
1) OFED IBACM (ibacm)
2) Intel HFI Components ()
3) OFED IP over IB ()
4) OPA FM (opafm)
P) Perform the autostart changes
S) Autostart All
X) Return to Previous Menu (or ESC)
[Enable ]
[Disable]
[Enable ]
[Enable ]
[Enable ]
R) Autostart None
Note: The previous menu is an example of the IntelOPA-IFS Autostart Menu. The
IntelOPA-Basic Autostart Menu does not show 5) OPA FM (opafm).
10. Review the items for Autostart; the default value is in brackets (Enable or
Disable). To change a value, type the alphanumeric character associated with
the item.
Intel recommends leaving all of the Autostart selections set to the default values.
11. Press P to perform the selected actions from the screen.
The system prompts:
Hit any key to continue...
Intel® Omni-Path Fabric Software
Installation Guide
22
November 2015
Order No.: H76467-1.0
Install the Intel® Omni-Path Fabric Software—Intel® Omni-Path Fabric
12. Press any key to proceed with the installation.
The system prompts:
Firmware Update unnecessary
Hit any key to continue...
13. Press any key to proceed with the installation.
The system prompts:
A System Reboot is recommended to activate the software changes
Hit any key to continue...
14. Press any key to proceed with the installation.
The installation completes and returns to the main menu.
15. Press X to exit.
16. Restart the network.
ifdown ib0
ifup ib0
If you have finished the IntelOPA-IFS installation of the Management Node, continue
setting up your Management Node by proceeding to the Configure Intel Chassis
section
4.3
Intel® Omni-Path Software Installation Using CLI
Commands
The installation of the Intel® Omni-Path Software (Intel® OPA Software) using CLI
commands can install the basic software or the IFS software that was downloaded to
your fabric. The ./INSTALL command has many options for installation including
installing single components, uninstalls, and enabling and disabling autostart of
components. For more information on the ./INSTALL syntax and options refer to
section Install Command. The following procedure steps through the installation and
defines the different items for the Basic and IFS installations. When needed, the
procedure specifies Basic or IFS accordingly. This procedure presumes that you are
logged in and have completed all pre-installation requirements.
Perform the following procedure to install the Intel® OPA Software in a default
configuration:
1. Change directory to IntelOPA-[Basic|IFS].DISTRO.VERSION directory.
Basic
cd IntelOPA-Basic.DISTRO.VERSION
IFS
cd IntelOPA-IFS.DISTRO.VERSION
2.
November 2015
Order No.: H76467-1.0
Start the default installation by typing ./INSTALL -n.
Intel® Omni-Path Fabric Software
Installation Guide
23
Intel® Omni-Path Fabric—Install the Intel® Omni-Path Fabric Software
The installation will complete with the following system prompt:
A System Reboot is recommended to activate the software changes
Done Installing OPA Software.
3. Reboot the server.
reboot
If you have finished the IntelOPA-IFS installation of the Management Node, continue
setting up your Management Node by proceeding to the Configure Intel Chassis
section
4.3.1
Install Command
The ./INSTALL command for the Basic and IFS installations are issued from the
following directories:
•
Intel Basic directory:
IntelOPA-Basic.DISTRO.VERSION
•
Intel IFS directory:
IntelOPA-IFS.DISTRO.VERSION
Basic
cd IntelOPA-Basic.DISTRO.VERSION
IFS
cd IntelOPA-IFS.DISTRO.VERSION
Syntax
./INSTALL [-r root] [-v|-vv] [-a|-n|-U|-u|-s|-O|-N|-i comp|-e comp] [-E comp]
[-D comp] [--user_configure_options 'options'] [--kernel_configure_options
'options']
[--prefix dir] [--without-depcheck] [--rebuild] [--force] [--answer keyword=value]
or
./INSTALL -C
or
./INSTALL -V
Options
No option selected
Intel® Omni-Path Fabric Software
Installation Guide
24
Display the Intel OPA Software TUI.
November 2015
Order No.: H76467-1.0
Install the Intel® Omni-Path Fabric Software—Intel® Omni-Path Fabric
-a
Installs all ULPs and drivers with the default options.
-n
Installs all ULPs and drivers with the default options,
but does not change the autostart options.
-U
Upgrades/re-installs all presently installed ULPs and
drivers with the default options, and does not
change the autostart options.
-i comp
Installs the given component with the default
options. This option can appear multiple times on a
command line.
-u
Uninstalls all ULPs and drivers with the default
options.
-s
Enables autostart for all installed software.
-e comp
Uninstalls the given component with the default
options. This option can appear multiple times on a
command line.
-E comp
Enables autostart of given component. This option
can appear with -D or multiple times on a command
line.
Note: To permit control over which installed
software is configured for autostart, combined
this option with -a, -n, -i, -e and -U
options.
-D comp
Disables autostart of the given component. This
option can appear with -E or multiple times on a
command line.
Note: To permit control over which installed
software is configured for autostart, combined
this option with -a, -n, -i, -e and -U
options.
-v
Verbose logging. Logs to the /var/log/opa.log
file.
-vv
Very verbose debug logging. Logs to
the /var/log/opa.log file.
-C
Shows the list of supported component names.
-V
Outputs the version number of the software.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
25
Intel® Omni-Path Fabric—Install the Intel® Omni-Path Fabric Software
-r dir
Specifies an alternate root directory. The default
is /.
Note: This option is to permit boot images to be
constructed that include Intel® Omni-Path
Software so that the boot images can later be
used for network boot of Intel® Omni-Path
Fabric enabled nodes.
Note: Intel® Omni-Path Fabric Suite FastFabric use
is not permitted in this environment.
-O
Keeps the current modified rpm configuration file
-N
Uses a new default rpm configuration file
--rebuild
Forces a rebuild of OFED_DELTA srpms.
--user_queries
Permits non-root users to query the fabric. This is
the default.
--no_user_queries
Non-root users cannot query the fabric.
--user_configure_options
options
Specifies additional OFED build options for user
space srpms. This forces a rebuild of all user srpms.
--kernel_configure
_options options
Specifies additional OFED build options for driver
srpms. This forces a rebuild of all driver srpms .
--prefix dir
Specifies an alternate directory prefix for the
OFED_Delta installation. Default is /usr. This
causes a rebuild of needed srpms.
--without-depcheck
Disables the check of OS dependencies.
--force
Forces the installation, even if the distributions do
not match. Use of this option can result in undefined
behaviors .
--answer keyword=value
Provides an answer to a question which might occur
during the operation. Answers to questions which
are not asked are ignored. Invalid answers will
result in prompting for interactive installations or
use of the default for non-interactive installations.
Possible Questions:
UserQueries Allow non-root users to access the
UMAD interface.
Intel® Omni-Path Fabric Software
Installation Guide
26
November 2015
Order No.: H76467-1.0
Install the Intel® Omni-Path Fabric Software—Intel® Omni-Path Fabric
Note: Allowing access to umadX
device files may present a
security risk. However, this
allows tools such as
opasaquery, opaportinfo,
etc to be used by non-root
users.
Default options retain existing
configuration files
Other Information
Supported component (comp) names:
opa_stack, ibacm, mpi_selector, intel_hfi, ib_wfr_lite, oftools, opa_stack_dev,
fastfabric, delta_ipoib,
opafm, mvapich2, openmpi, gasnet, openshmem, mvapich2_gcc_hfi, mvapich2_pgi_hfi,
mvapich2_intel_hfi,
openmpi_gcc_hfi, openmpi_pgi_hfi, openmpi_intel_hfi, delta_mpisrc, delta_debug
Supported component (comp) name aliases:
opa, ipoib, mpi, mpisrc, opadev
4.4
IPoIB IPV4 Configuration Using the Software TUI
Prerequisites: Have a list of the IP addresses and netmasks for each interface you
are going to set up.
The following procedure can be done during the Software TUI installation or after the
installation is complete. If you are configuring the IPoIB IPV4 during the Software TUI
installation start with Step 2 and when complete go back to the Software TUI
installation and continue where you left off.
The following procedure presumes you are logged on and at the Intel OPA Software
main menu.
1.
Select 2) Reconfigure OFED IP over IB from the Intel OPA Software main
menu.
The system prompts:
Configure OFED IP over IB IPV4 addresses now? [n]:
2. Press y to configure OFED IP over IB IPV4.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
27
Intel® Omni-Path Fabric—Install the Intel® Omni-Path Fabric Software
The system prompts:
You may configure an OFED IP over IB interface for
Or you may select to have OFED IP over IB only run
Or you may select to configure redundant HFI ports
pair of HFI ports running a single OFED IP over IB
How many OFED IP over IB interfaces would you like
3.
each HFI port
on some HFI ports
with a
interface
to configure? [1]:
Press Enter for the default of 1 or type in the number of interfaces you are going
to configure.
If you selected the default, the system prompts:
About to create OFED IP over IB ifcfg files in
/etc/sysconfig/network-scripts
Use interface name ib0? [y]:
If you selected something other than the default, the system prompts:
About to create OFED IP over IB ifcfg files in
/etc/sysconfig/network-scripts
Configure interface names sequentially starting with ib0? [y]:
4.
4.5
Continue the setup by answering and following the prompts to set up the OFED IP
over IB interfaces.
Setup completes and returns to the main menu
IPoIB IPV4 Configuration Using CLI Commands
Prerequisites: Have a list of the IP addresses and netmasks for each interface you
are going to set up.
The following procedure presumes you are logged on and at the Intel OPA Software
main menu.
1.
4.6
Type opahostadmin -f /etc/sysconfig/opa/hosts configipoib.
This creates the ifcfg-ibX configuration file for each host, using the host IP
addresses from the /etc/hosts file.
Install IPoIB IPV6
To install IPoIB for IPV6 on the management node use the following procedures for
the OS on the Fabric management node.
On Red Hat*:
1.
Edit file /etc/sysconfig/network to add the following line:
NETWORKING_IPV6=yes
2. Edit file ifcfg-if-name to add the following lines:
IPV6INIT=yes
IPV6ADDR=”ipv6addr/prefix-length”
Intel® Omni-Path Fabric Software
Installation Guide
28
November 2015
Order No.: H76467-1.0
Install the Intel® Omni-Path Fabric Software—Intel® Omni-Path Fabric
IPV6 address should look like the following:
3ffe::6/64
3.
Restart the network.
On SUSE* Enterprise:
1.
Edit ifcfg-ifname to add the following line:
IPADDR=”ipv6addr/prefix-length”
IPV6 address should look like the following:
3ffe::6/64
2. Restart the network.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
29
Intel® Omni-Path Fabric—Configure Intel® Omni-Path Chassis
5.0
Configure Intel® Omni-Path Chassis
Use Intel® Omni-Path Fabric Suite FastFabric to install and configure internally
managed switches, such as the Intel® Omni-Path Director Switch 100 Family. See the
switch documentation for information about installing and configuring switches made
by other manufacturers.
5.1
Intel Chassis Configuration Pre-requisites
Ensure the internally managed switches are configured to use the Intel® Omni-Path
Fabric Suite FastFabric tool set, by performing the following steps. Refer to the Intel®
Omni-Path Fabric Switches Hardware Installation Guide for more details:
1.
(Switch) Connect each chassis to the management network through its Ethernet
management port. For chassis with redundant management, connect both
Ethernet management ports.
2.
(Switch) Set up the netmask and gateway addresses on each Intel chassis,
following the procedures in the Intel® Omni-Path Fabric Switches GUI User Guide.
3.
(Switch) Assign each Intel chassis a unique IP address, and appropriately
configure the Ethernet management port network settings.
4.
(Switch) For a chassis with redundant management, assign a unique IP address
for each Intel Management Module or Intel Management Spine, and configure their
Ethernet management port network settings.
5.
(Switch) Select a unique name for each Intel chassis, Management Module, and
Spine. This name should be configured in DNS or /etc/hosts as the TCP/IP
name for the Ethernet management port.
Note: The chassis node description will be set later.
6. (Switch) Configure the administrator password on each Intel chassis.
Note: All versions of Intel® Omni-Path Chassis 100 Family firmware permit SSH
keys to be configured within the chassis for secure password-less login. To
simplify the configuration of SSH security using FastFabric, configure all
chassis with the same initial administrator password (or leave the default
“adminpass” until FastFabric has set up ssh keys), configure the SSH keys,
then change the administrator passwords. After ssh has been set up using
FastFabric, Intel recommends you change the admin passwords.
7. (Switch) Copy the relevant chassis firmware files onto the FastFabric
management node. When performing the Chassis Configuration procedures, the
*.spkg files will be used to upgrade the firmware on each chassis.
Note: Place all files at a given firmware level into a single directory, whose name
indicates the firmware revision number.
Intel® Omni-Path Fabric Software
Installation Guide
30
November 2015
Order No.: H76467-1.0
Configure Intel® Omni-Path Chassis—Intel® Omni-Path Fabric
5.2
Configure Chassis Using Intel® Omni-Path Fabric Suite
FastFabric
Refer to the Intel® Omni-Path Fabric Suite FastFabric User Guide for more information
on how to use the FastFabric TUI.
1. (Switch) Type opafastfabric and press Enter.
The Intel FastFabric OPA Tools menu is displayed.
Intel FastFabric OPA Tools
Version: X.X.X.X.X
1)
2)
3)
4)
5)
Chassis Setup/Admin
Externally Managed Switch Setup/Admin
Host Setup
Host Verification/Admin
Fabric Monitoring
X) Exit
2. (Switch) Press 1.
The FastFabric OPA Chassis Setup/Admin Menu is displayed.
FastFabric OPA Chassis Setup/Admin Menu
Chassis File: /etc/sysconfig/opa/chassis
Setup:
0) Edit config and select/edit Chassis file
1) Verify Chassis via Ethernet ping
2) Update Chassis firmware
3) Setup Chassis basic configuration
4) Setup password-less ssh/scp
5) Reboot Chassis
6) Get basic Chassis configuration
7) Configure Chassis Fabric Manager (FM)
8) Update Chassis FM security files
9) Get Chassis FM security files
Admin:
a) Check OPA Fabric status
b) Control Chassis Fabric Manager (FM)
c) Generate all Chassis Problem Report Info
d) Run a command on all chassis
Review:
e) View opachassisadmin result files
P) Perform the selected actions
X) Return to Previous Menu (or ESC)
[
[
[
[
[
[
[
[
[
[
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
]
]
]
]
]
]
]
]
]
]
[
[
[
[
Skip
Skip
Skip
Skip
]
]
]
]
[ Skip
]
N) Select None
3. (Switch) Select the items in the Setup section that are required or needed for
your fabric.
Type the alphanumeric character associated with the item to toggle the selection
from Skip to Perform.
4.
Press P.
Perform the items that were selected in the sub-sections as follows.
Edit the Configuration and Select/Edit Chassis File
(Switch) The Edit Config and Select/Edit Chassis File selection will permit the
chassis, ports, and FastFabric configuration files to be edited.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
31
Intel® Omni-Path Fabric—Configure Intel® Omni-Path Chassis
•
When placed in the editor for opafastfabric.conf, review the settings.
—
Especially review the FF_CHASSIS_LOGIN_METHOD and
FF_CHASSIS_ADMIN_PASSWORD. Refer to "Configuration Files for FastFabric"
section of the Intel® Omni-Path Fabric Suite FastFabric User Guide for more
information about the opafastfabric.conf file.
Note: FastFabric will provide the opportunity to enter the chassis password
interactively when needed. It is not necessary to place it within
opafastfabric.conf. If the Intel chassis admin password is placed in
opafastfabric.conf, change the opafastfabric.conf
permissions to be 0x600 (root-only access).
Note: All versions of Intel® Omni-Path Chassis 100 Family firmware permit ssh
keys to be configured within the chassis for secure password-less login.
There is no need to configure a FF_CHASSIS_ADMIN_PASSWORD, and
FF_CHASSIS_LOGIN_METHOD can be set to ssh (the default) when
using the newer versions of the chassis firmware.
•
Select the location for the result files from FastFabric with the FF_RESULT_DIR
parameter. The default is the directory from which a given session of FastFabric is
invoked. Alternatively, it can be set to a directory relative to your home directory.
For example:
export FF_RESULT_DIR=${FF_RESULT_DIR:-$HOME/
fastfabric_results}
Note: Refer to "Configuration Files for FastFabric" section of the Intel® Omni-Path
Fabric Suite FastFabric User Guide for more information about the
opafastfabric.conf file.
•
When placed in the editor for ports, review the file. For typical single-subnet
clusters, the default of “0:0” may be used. This will use the first active port on the
Fabric Management Node to access the fabric. For more information on configuring
a cluster with multiple subnets, see Multi-Subnet Fabrics on page 101. For further
details about the file format, refer to the Intel® Omni-Path Fabric Suite FastFabric
Command Line Interface Reference Guide.
•
When placed in the editor for chassis, create the file with a list of the chassis
names (the TCP/IP Ethernet management port names assigned) or IP addresses
(use of names is recommended). Enter one chassis name or IP address per line.
For example:
Chassis1
Chassis2
Note:
Refer to Generate or Update Switch File on page 48 to generate a list of the
externally managed switches presently in the fabric.
Note:
Do not list externally managed switches in this file. Those will be covered in section
Configure Firmware on the Externally Managed Intel® Omni-Path Switches on page
46.
For further details about the file format refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide.
Intel® Omni-Path Fabric Software
Installation Guide
32
November 2015
Order No.: H76467-1.0
Configure Intel® Omni-Path Chassis—Intel® Omni-Path Fabric
Verify Chassis via Ethernet Ping
(Switch) The Verify Chassis via Ethernet Ping selection will ping each selected
chassis over the management network. If all chassis were found, continue to the next
step. If some chassis were not found, exit the menu and review the following list for
those chassis which were not found:
•
Is chassis powered on and booted?
•
Is chassis connected to management network?
•
Are chassis IP address and network settings consistent with DNS or /etc/hosts?
•
Is Management node connected to the management network?
•
Are Management node IP address and network settings correct?
•
Is management network itself up (switches, routers, etc)?
•
Is correct set of chassis listed in the chassis file (the previous step may be
repeated to review and edit the file as needed)?
Update Chassis Firmware
(Switch) The Update Chassis Firmware selection will permit the chassis firmware
version to be verified and updated as needed.
Notes:
Before continuing, refer to the relevant switch release notes for any prerequisites:
•
Intel® Omni-Path Fabric Managed Switches Release Notes
•
Intel® Omni-Path Fabric Externally-Managed Switches Release Notes
1. When this procedure is started, the following system prompt will be displayed:
Multiple Firmware files and/or Directories may be space separated
Shell wildcards may be used
For Directories all .spkg files in the directory tree will be used
Enter Files/Directories to use (or none):
2. Specify the directory where the relevant firmware files have been stored and press
Enter.
This can be the mount point of the CD or the directory to which the files were
copied in a previous step.
System prompts:
Would you like to run the firmware now? [n]:
3. Type y and press Enter.
FastFabric will ensure that all chassis are running the firmware level provided, and
will install and/or reboot each chassis as needed.
If any chassis fails to be updated, use the View opachassisadmin result files
option to review the result files from the update. Refer to the Intel® Omni-Path
Fabric Suite FastFabric Command Line Interface Reference Guide for more details.
Set Up Chassis Basic Configuration
(Switch) The Setup Chassis Basic Configuration will permit the typical chassis
setup operations to be performed for all chassis.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
33
Intel® Omni-Path Fabric—Configure Intel® Omni-Path Chassis
Perform the following procedure:
1.
When this procedure is started, the following system prompt will be displayed:
Would you like to be prompted for chassis' password? [n]:
2. Press Enter to select default (n).
System prompts:
Do you wish to configure a syslog server? [y]:
3.
Press Enter to select default (y).
System prompts:
Enter IP address for syslog server:
4.
Enter the IP address of the syslog server that is to receive log messages from all
chassis.
System prompts:
Do you wish to configure the syslog TCP/UDP port number? [n]:
5. Press Enter to select default (n).
System prompts:
Do you wish to configure the syslog facility? [n]:
6. Press Enter to select default (n).
System prompts:
Logging to the syslog can be configured in a Quiet Mode, where only user
actionable
events are logged. Otherwise, in Normal Mode, all events will be logged
Do you wish to configure Quiet Mode for syslog? [n]:
7.
Press Enter to select default (n).
System prompts:
Do you wish to configure an NTP server? [y]:
8. Press Enter to select default (y).
System prompts:
Enter IP address for NTP server:
9.
Enter the IP address of an NTP server that can supply a consistent time base for
use by all chassis.
System prompts:
Do you wish to configure timezone and DST information? [y]:
Intel® Omni-Path Fabric Software
Installation Guide
34
November 2015
Order No.: H76467-1.0
Configure Intel® Omni-Path Chassis—Intel® Omni-Path Fabric
10. Press Enter to select default (y).
System prompts:
Do you want to use the local timezone information from the local server? [y]:
11. Press Enter to select default (y).
This causes the time zone of the local server (e.g., the Fabric Management Node)
to be replicated to all the chassis to specify their time zones.
System prompts:
Do you wish to configure the chassis maximum packet MTU size? [n]:
12. Press Enter to select default (n).
This causes the default MTU of 2048 to be used for all chassis. If chassis have
previously been manually configured for a different MTU size, this option will keep
the previously configured MTU size. Refer to the Intel® Omni-Path Fabric Host
Software User Guide, for detailed information.
System prompts:
Do you wish to configure the chassis VL Capability? [n]:
13. Press Enter to select default (n).
This causes the default VL Capability of 1 to be used for all chassis. If chassis have
previously been manually configured for a different VL Capability, this option will
keep the previously configured VL Capability size. See the Intel® Omni-Path Fabric
Switches GUI User Guide for more information.
System prompts:
Do you wish to configure the VL Credit Distribution? [n]:
Note: Always select the default (n) for the question Do you wish to
configure the VL Credit Distribution? unless instructed to do
otherwise by Intel Technical Support.
14. Press Enter to select default (n).
System prompts:
Do you wish to configure the chassis link width? [n]:
15. Press Enter to select default (n).
This causes the default link width supported value of 1x/4x/8x to be used for
all ports on all chassis. If any chassis have been previously manually configured
for a different link width supported, those setting will remain unchanged.
System prompts:
Do you wish to configure Switch Desc to match ethernet chassis name? [y]:
16. Press Enter to select default (y).
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
35
Intel® Omni-Path Fabric—Configure Intel® Omni-Path Chassis
This causes the chassis name entered in the /etc/sysconfig/opa/chassis
file to be used as the Switch Description for the chassis, making the management
network and OPA network names for the chassis consistent.
If the /etc/sysconfig/opa/chassis file has IP addresses instead of names,
enter n to this question.
System prompts:
Do you wish to configure Switch Desc Format? [y]:
17. Press Enter to select default (y).
System prompts:
Do you wish to use concise Switch Desc format? [y]:
18. Press Enter to select default (y).
This causes the chassis Switch Descriptions to use concise naming for the Leafs
and Spines such as L01 or S01A (as opposed to “Leaf 1" or “Spine 1, Chip A”).
System prompts:
Do you wish to configure the port counter auto-clear feature? [n]:
If
Then
You are not using opatop.
Continue with 19 on page 36
You will be using opatop.
Skip to 20 on page 36
19. Press Enter to select default (n).
Selecting the default will leave the auto clear feature as previously configured.
Skip to 21 on page 36.
20. Press Y to select to disable this feature.
This causes the port counter auto-clear feature to be disabled on all chassis.
21. Continue with Setup Password-less ssh/scp on page 36.
Setup Password-less ssh/scp
(Switch) The Setup Password-less ssh/scp selection will set up secure passwordless ssh, such that the Fabric Management Node can securely log into all the chassis
as admin through the management network, without requiring a password.
Reboot Chassis
(Switch) The Reboot Chassis selection will reboot all the selected chassis and
ensure they reboot fully (as verified through ping over the management network).
When the chassis come back up following the reboot, they will be running with all the
new configuration settings.
Intel® Omni-Path Fabric Software
Installation Guide
36
November 2015
Order No.: H76467-1.0
Configure Intel® Omni-Path Chassis—Intel® Omni-Path Fabric
Get Basic Chassis Configuration
(Switch) The Get Basic Chassis Configuration selection will retrieve basic
information from chassis such as syslog, NTP configuration, timezone info, MTU
Capability, VL Capability, VL Credit Distribution, Link Width, and node description. The
following is an example of the output from this selection:
Performing Chassis Admin: Get basic Chassis configuration
Executing: /usr/sbin/opachassisadmin -F /etc/sysconfig/opa/chassis getconfig
Executing getconfig Test Suite (getconfig) day mmm dd hh:mm:ss timezone yyyy ...
Executing TEST SUITE getconfig CASE (getconfig.xx.xx.xx.xx.getconfig) get
xx.xx.xx.xx ...
TEST SUITE getconfig CASE (getconfig.xx.xx.xx.xx.getconfig) get xx.xx.xx.xx
xx.xx.xx.xx:
Firmware Active
: xx.xx.xx.xx
Firmware Primary
: xx.xx.xx.xx
Syslog Configuration
: Syslog host set to: 0.0.0.0 port 514 facility 22
NTP
: Configured to use the local clock
Time Zone
: Time zone offset has not been configured
LinkWidth Support
: 4X
Node Description
: Node_Name
Link CRC Mode
: 48b_or_14b_or_16b
PASSED
TEST SUITE getconfig: 1 Cases; 1 PASSED
TEST SUITE getconfig PASSED
Done getconfig Test Suite day mmm dd hh:mm:ss timezone yyyy
Hit any key to continue (or ESC to abort)...
Configure Chassis Fabric Manager (FM)
(Switch) The Configure Chassis Fabric Manager (FM) selection will assist in
configuring the Fabric Manager for any Intel® Omni-Path Chassis 100 Family.
System prompts:
Performing Chassis Admin: Configure Chassis Fabric Manager
Enter FM Config file to use (or none or generate):
1. Enter generate.
This will perform the config_generate operation to guide you through selecting
FM configuration options. See the Intel® Omni-Path Fabric Suite Fabric Manager
User Guide for more information about config_generate.
2.
After responding to the prompts for config_generate, the following system
prompt will display:
You have selected to use: ./opafm.xml
Syntax Checking ./opafm.xml...
Executing: /opt/hfi/fm_tools/config_check -s -c ./opafm.xml
Valid FM Config file: ./opafm.xml
After push, the FM may be started/restarted
Would you like to restart the FM? [n]:
3.
Enter y.
This causes the FM to be started with the new configuration.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
37
Intel® Omni-Path Fabric—Configure Intel® Omni-Path Chassis
System prompts:
Would you like to run the FM on slave MMs? [n]:
4. Refer to the following If/Then table:
If
Then
Your fabric has a single chassis running the
Fabric Manager, you can run the Fabric Manager
on the slave management module (MM). This
causes the Fabric Manager to be started in the
applicable chassis.
Type y
Your fabric has multiple chassis running the
Fabric Manager, Intel recommends to run Fabric
Manager on the master management module.
This causes the Fabric Manager to be started
only on the master management module in the
applicable chassis.
Type n
System prompts:
There
Doing
Doing
Would
will be a disruption as FMs are restarted
the operation in parallel (on multiple chassis) will finish the fastest
it serially may reduce disruption
you like to do the operation in parallel? [y]:
5. Press Enter to select the default option: y.
Intel recommends doing the operation in parallel.
System prompts:
You have selected to perform the push and FM restart in parallel
Would you like to enable FM start at boot? [n]:
6. Enter y.
This causes the Fabric Manager to be started on all applicable chassis each time
those chassis boot.
System prompts:
Would you like to enable FM start on slave MMs at boot? [n]:
7. Refer to the following If/Then table:
Then
If
Your fabric has a single chassis running the
Fabric Manager, you can run the Fabric Manager
on the slave management module. This causes
the Fabric Manager to be started in the
applicable chassis.
Type y
Your fabric has multiple chassis running the
Fabric Manager. Intel recommends to run Fabric
Manager on the master management module.
This causes the Fabric Manager to only be
started on the master management module in
the applicable chassis.
Type n
Intel® Omni-Path Fabric Software
Installation Guide
38
November 2015
Order No.: H76467-1.0
Configure Intel® Omni-Path Chassis—Intel® Omni-Path Fabric
System prompts:
Would you like to be prompted for chassis' password? [n]:
8.
Press Enter to select the default n option.
System prompts:
Are you sure you want to proceed? [n]:
9. Enter y.
This updates the Fabric Manager.
System prompts:
Hit any key to continue (or ESC to abort)...
10. Press any key to complete this procedure
Update Chassis FM Security Files
(Switch) The Update Chassis FM Security Files selection runs the opachassisadmin
fmsecurityfiles command to permit the chassis security files to be verified and
updated as needed.
Note:
The FM security files are the private key, public key, and certificate files required by
the FM, in order to support secure socket connection via OpenSSL. Refer the Intel®
Omni-Path Fabric Suite Fabric Manager User Guide for instructions in the
administration tasks required to support these files.
Prompts will guide you through the options:
•
push - Ensures given security files are pushed to each chassis
Additional options prompted for:
•
selection of security files or directory containing pem files
•
parallel vs serial update
•
chassis password (default is to have password in fastfabric.conf or to use
password-less ssh)
If any chassis fails to be updated, use the View opachassisadmin results
files option to review the result files from the update. Refer to View
opachassisadmin result files on page 40 for more information.
Refer to Intel® Omni-Path Fabric Suite FastFabric Command Line Interface Reference
Guide for more details on the opachassisadmin update command.
Get Chassis FM Security Files
(Switch) The Get Chassis FM Security Files selection runs the
opachassisadmin fmgetsecurityfiles command to permit the chassis FM
security files to be retrieved from the chassis.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
39
Intel® Omni-Path Fabric—Configure Intel® Omni-Path Chassis
Check OPA Fabric status
(All) The Check OPA Fabric status selection prompts you to:
•
Perform a fabric error analysis,
•
Clear the error counters after generating a report
•
Perform a fabric link speed error analysis
•
Check for links configured to run slower than supported
•
Check for links connected with mismatched speed potential
•
Enter a filename for the results or save the results to the default location (/root/
ffres/linkanalysis.res
For more information refer to the Intel® Omni-Path Fabric Suite FastFabric User
Guide.
Control Chassis Fabric Manager (FM)
(Switch) The Control Chassis Fabric Manager (FM) selection prompts you to:
•
Restart the Fabric Manager
•
Run the Fabric Manager on slave management modules
•
Restart the Fabric Manager on all management modules
•
Perform this operation in parallel
•
Change the Fabric Manager boot state to enable the Fabric Manager to start at
boot
•
Enable Fabric Manager to start on slave management modules at boot
•
Be prompted for chassis’ password
For more information, refer to the Intel® Omni-Path Fabric Suite FastFabric User
Guide.
Generate all Chassis Problem Report Information
(Switch) The Generate all Chassis Problem Report Info selection generates the
chassis problem report. For more information, refer to the Intel® Omni-Path Fabric
Suite FastFabric User Guide.
Run a command on all chassis
(Switch) If there are any other operations that need to be performed on all chassis,
they may be performed using the Run a command on all chassis option. Each time
this is executed, a single chassis CLI command may be specified to be executed
against all selected chassis. When using these commands, additional setup or
verification of the chassis may be performed.
View opachassisadmin result files
(Switch) The View opachassisadmin result files selection opens the
punchlist.csv, test.res, and test.log files to be viewed. For more information,
refer to the Intel® Omni-Path Fabric Suite FastFabric User Guide.
Intel® Omni-Path Fabric Software
Installation Guide
40
November 2015
Order No.: H76467-1.0
Configure Intel® Omni-Path Chassis—Intel® Omni-Path Fabric
5.3
Configure Chassis Using CLI Commands
Prerequisites: Refer to Intel Chassis Configuration Pre-requisites on page 30 for prerequisites to this procedure.
This procedure provides step-by-step information to configure the chassis.
1. (Optional) Edit or review the /etc/sysconfig/opa/opafastfabric.conf file.
Review the complete file. Especially review the FF_CHASSIS_LOGIN_METHOD and
FF_CHASSIS_ADMIN_PASSWORD sections. Refer to "Configuration Files for
FastFabric" section of the Intel® Omni-Path Fabric Suite FastFabric User Guide for
more information about the opafastfabric.conf file.
2.
(Optional) Edit or review the /etc/sysconfig/opa/ports file.
For typical single-subnet clusters, the default of “0:0” may be used. This will use
the first active port on the Fabric Management Node to access the fabric. For more
information on configuring a cluster with multiple subnets, see Multi-Subnet
Fabrics on page 101. For further details about the file format, refer to the Intel®
Omni-Path Fabric Suite FastFabric Command Line Interface Reference Guide
3.
(Optional) Create, edit, or review the /etc/sysconfig/opa/chassis file.
Create/review the file with a list of the chassis names (the TCP/IP Ethernet
management port names assigned) or IP addresses (use of names is
recommended). Enter one chassis name or IP address per line. For example:
Chassis1
Chassis2
Note: Refer to Generate or Update Switch File on page 48 to generate a list of
the externally managed switches presently in the fabric.
Note: Do not list externally managed switches in this file. Those will be covered in
the Configure Firmware on the Externally Managed Intel® Omni-Path
Switches on page 46 section.
For further details about the file format refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide.
4. Verify the chassis is accessible over the management network.
opapingall -C -p -f /etc/sysconfig/opa/chassis
If some chassis were not found review the following list for those chassis which
were not found:
November 2015
Order No.: H76467-1.0
•
Is chassis powered on and booted?
•
Is chassis connected to management network?
•
Are chassis IP address and network settings consistent with DNS or /etc/
hosts?
•
Is Management node connected to the management network?
•
Are Management node IP address and network settings correct?
•
Is the management network itself up (switches, routers, etc)?
•
Is correct set of chassis listed in the chassis file (the previous step may be
repeated to review and edit the file as needed)?
Intel® Omni-Path Fabric Software
Installation Guide
41
Intel® Omni-Path Fabric—Configure Intel® Omni-Path Chassis
5.
Update the chassis firmware.
Run the firmware update in parallel use:
opachassisadmin -S -F chassisfile -P package -a run upgrade
Run the firmware update in serial use:
FF_MAX_PARALLEL=0 opachassisadmin -S -F chassisfile -P package
-a run upgrade
During run the following files are produced:
•
test.res - appended with summary results of run
•
test.log - appended with detailed results of run
•
save_tmp/ - contains a directory per failed operation with detailed logs
•
test_tmp*/ - intermediate result files while operation is running
If any chassis fails to be updated, refer to the View opachassisadmin result
files section to review the result files from the update.
Refer to the Intel® Omni-Path Fabric Suite FastFabric Command Line Interface
Reference Guide for more details.
6. Set up the chassis with the basic configuration settings.
opachassisadmin -S -F chassisfile configure
This wizard gives you the option of setting up the following items on the chassis:
•
Syslog server
•
NTP server
•
Timezone and DST information
•
Chassis maximum packet MTU size
•
Chassis VL Capability
•
VL Credit Distribution
•
Chassis link width
•
Configure Switch Desc to match ethernet chassis name
•
Configure Switch Desc Format
•
Use concise Switch Desc format
•
Configure the port counter auto-clear feature
Follow the system prompts to select and configure the items required.
7.
Set up secure password-less ssh, such that the Fabric Management Node can
securely log into all the chassis as admin through the management network,
without requiring a password.
opasetupssh -p -S -C -F chassisfile
8. Reboot all the selected chassis and ensure they reboot fully (as verified through
ping over the management network).
opachassisadmin -S -F chassisfile reboot
When the chassis come back up following the reboot, they will be running with all
the new configuration settings.
Intel® Omni-Path Fabric Software
Installation Guide
42
November 2015
Order No.: H76467-1.0
Configure Intel® Omni-Path Chassis—Intel® Omni-Path Fabric
9.
Configure the Chassis Fabric Manager for any Intel® Omni-Path Chassis 100
Family chassis.
To configure the Chassis Fabric Manager in parallel use:
opachassisadmin -S -F chassisfile -P package -a run fmconfig
To configure the Chassis Fabric Manager in serial use:
FF_MAX_PARALLEL=0 opachassisadmin -S -F chassisfile -P package
-a run fmconfig
10. Retrieve basic information from chassis such as syslog, NTP configuration,
timezone info, MTU Capability, VL Capability, VL Credit Distribution, Link Width,
and node description.
opachassisadmin -F chassisfile getconfig
5.4
Verify Chassis Configuration Using CLI Commands
Prerequisites: Chassis has been configured.
This procedure provides step-by-step information to verify the chassis configuration
and view the result files.
1.
Check the OPA fabric status.
opalinkanalysis -U -x snapshot_suffix reports verifyall >
results_file 2>&1
This step performs the following operations:
•
Perform a fabric error analysis
•
Clear the error counters after generating a report
•
Perform a fabric link speed error analysis
•
Check for links configured to run slower than supported
•
Check for links connected with mismatched speed potential
•
Enter a filename for the results or save the results to the default location (/
root/ffres/linkanalysis.res
2. To control the Chassis Fabric Manager you can perform any or all of the following
steps:
a.
(optional) Restart all of the Chassis Fabric Managers.
Note: Doing it serially may reduce disruption
To restart all the Chassis Fabric Managers and run the Fabric Manager on the
slave Management Module in parallel use:
opachassisadmin -S -F chassisfile -a restartall fmcontrol
To restart all the Chassis Fabric Managers and run the Fabric Manager on the
slave Management Module in serial use:
FF_MAX_PARALLEL=0 opachassisadmin -S -F chassisfile -a
restartall fmcontrol
There will be a disruption as FMs are restarted.
b.
(optional) Restart the master Chassis Fabric Manager and stopping the slaves.
Note: Doing it serially may reduce disruption
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
43
Intel® Omni-Path Fabric—Configure Intel® Omni-Path Chassis
To restart the Master Chassis Fabric Managers and stop the Fabric Manager on
the slave Management Modules in parallel use:
opachassisadmin -S -F chassisfile -a restart fmcontrol
To restart all the Chassis Fabric Managers and stop the Fabric Manager on the
slave Management Modules in serial use:
FF_MAX_PARALLEL=0 opachassisadmin -S -F chassisfile -a
restart fmcontrol
c.
(optional) Stop the Chassis Fabric Managers from running.
To stop the Master Chassis Fabric Managers from running in parallel use:
opachassisadmin -S -F chassisfile -a stop fmcontrol
To stop the Master Chassis Fabric Managers from running in serial use:
FF_MAX_PARALLEL=0 opachassisadmin -S -F chassisfile -a stop
fmcontrol
d.
(optional) Make sure all of the Chassis Fabric Managers are running.
To make sure all of the Chassis Fabric Managers are running including the
slaves in parallel use:
opachassisadmin -S -F chassisfile -a runall fmcontrol
To make sure all of the Chassis Fabric Managers are running including the
slaves in serial use:
FF_MAX_PARALLEL=0 opachassisadmin -S -F chassisfile -a
runall fmcontrol
e.
(optional) Make sure the Master Chassis Fabric Manager is running.
To make sure the Master Chassis Fabric Manager are running and the slaves
are stopped in parallel use:
opachassisadmin -S -F chassisfile -a run fmcontrol
To make sure the Master Chassis Fabric Manager are running and the slaves
are stopped in serial use:
FF_MAX_PARALLEL=0 opachassisadmin -S -F chassisfile -a run
fmcontrol
f.
(optional) Change the Fabric Manager boot state to enable the Master Chassis
Fabric Manager to start at boot.
opachassisadmin -S -F chassisfile -I enable
g.
(optional) Change the Fabric Manager boot state to enable the all of the
Chassis Fabric Managers to start at boot.
opachassisadmin -S -F chassisfile -I enableall
3. Generate all chassis problem report information.
opacaptureall -p -D 4 -f hostfile
4.
Run a command on all chassis.
opacmdall -C -S -p -T timelimit -F chassisfile STDIN
5. View opachassisadmin result files
editor(for example vi) result_dir/result_file
The following default files are created:
Intel® Omni-Path Fabric Software
Installation Guide
44
November 2015
Order No.: H76467-1.0
Configure Intel® Omni-Path Chassis—Intel® Omni-Path Fabric
November 2015
Order No.: H76467-1.0
•
punchlist.csv
•
test.res
•
test.log
Intel® Omni-Path Fabric Software
Installation Guide
45
Intel® Omni-Path Fabric—Configure Firmware on the Externally Managed Intel® Omni-Path
Switches
6.0
Configure Firmware on the Externally Managed
Intel® Omni-Path Switches
If the fabric contains Intel® Omni-Path Switch 100 Family externally managed
switches, Intel® Omni-Path Fabric Suite FastFabric is used to aid in the installation and
configuration of the switches. If the fabric contains another vendor’s switches, please
refer to the vendor’s documentation to configure the firmware on the externally
managed switches.
6.1
Switch Configuration Pre-Requisites
Prior to using Intel® Omni-Path Fabric Suite FastFabric, the following minimal steps
need to be performed:
1.
(Switch) Select a unique name to be used for each switch. This name will be
configured as the Switch Description for the switch in the following steps.
Note: Externally managed switches do not have an Ethernet port and therefore
will not have a TCP/IP name.
2.
(Switch) Copy the relevant switch firmware files onto the Intel® Omni-Path Fabric
Suite FastFabric management node. For the following steps, the *.emfw files will
be used to upgrade the firmware on each switch.
Note: When copying files, it is best to place all files at a given firmware level into a
single directory whose name indicates the firmware revision number
6.2
Configure Externally Managed Switches
Once the pre-requisites have been completed, configure the switches using Intel®
Omni-Path Fabric Suite FastFabric in the following procedure.
1. (Switch) If the Intel FastFabric OPA Tools menu is not displayed, type
opafastfabric at a command prompt and press Enter.
2. (Switch) Press 2 to display the FastFabric OPA Switch Setup/Admin Menu
(Figure 1 on page 47).
Intel® Omni-Path Fabric Software
Installation Guide
46
November 2015
Order No.: H76467-1.0
Configure Firmware on the Externally Managed Intel® Omni-Path Switches—Intel® Omni-Path
Fabric
Figure 1.
FastFabric OPA Switch Setup/Admin Menu
FastFabric OPA Switch Setup/Admin Menu
Externally Managed Switch File: /etc/sysconfig/opa/switches
Setup:
0) Edit Config and Select/Edit Switch File
1) Generate or Update Switch File
2) Test for Switch Presence
3) Verify Switch Firmware
4) Update Switch Firmware
5) Setup Switch Basic Configuration
6) Reboot Switch
7) Report Switch Firmware & Hardware Info
8) Get Basic Switch configuration
Admin:
9) Report Switch VPD Information
a) Generate all Switch Problem Report Info
Review:
b) View opaswitchadmin result files
P) Perform the selected actions
X) Return to Previous Menu (or ESC)
[
[
[
[
[
[
[
[
[
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
]
]
]
]
]
]
]
]
]
[ Skip
[ Skip
]
]
[ Skip
]
N) Select None
3. (Switch) Select the items in the Setup section that are required or needed for
your fabric
4. Press P.
5.
Perform the items selected using the following sections.
Edit Config and Select/Edit Switch File
(Switch) The Edit Config and Select/Edit Switch File selection will take you to
edit mode, where you can edit the switches, ports, and FastFabric configuration files.
While in the editor, review all the settings in opafastfabric.conf. Refer to Intel®
Omni-Path Fabric Suite FastFabric User Guide Configuration Files section for more
information about opafastfabric.conf.
When in the editor for ports, review the file. For typical single-subnet clusters, the
default of “0:0” may be used. This will use the first active port on the Fabric
Management Node to access all externally managed switches. For more information on
configuring a cluster with multiple subnets, see Multi-Subnet Fabrics on page 101. For
further details about the file format, refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide.
When in the editor for switches, create the file with a list of the switch node GUID and
required switch names. Enter one switch node GUID and required switch name per
line. For example:
0x00066a00d9000138,edge1
0x00066a00d9000139,edge2
Note:
Per the previous example, when typing a new name, do not use any spaces before or
after the comma.
Note:
The Generate or Update Switch File menu item or opagenswitches may be used
to generate a list of the externally managed switches presently in the fabric. For
example, when using the vi editor, the command :r ! opagenswitches may be
used to add the output from this command to the file.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
47
Intel® Omni-Path Fabric—Configure Firmware on the Externally Managed Intel® Omni-Path
Switches
Note:
Do not list internally managed chassis in this file. Those are covered in the Configure
Intel Chassis section.
Note:
Intel® Omni-Path Fabric Suite FastFabric is topology-aware when updating externally
managed switch firmware or resetting the switches. The update or restart will start at
the switches farthest from the FastFabric node and then work toward the FastFabric
node. This way, switches that are rebooted will not be in the path between the
FastFabric node and others that are being updated or reset.
For further details regarding the file format, refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide.
If needed, an SA query, such as the following, can be used to get a list of all switches.
This includes both internally and externally managed switches. Consequently, the
output must be edited to leave only the Intel externally managed switches:
opasaquery -t sw -o nodeguid
Generate or Update Switch File
(Switch) The Generate or Update Switch File selection generates or updates the
switches file. It can also update switch names in the switches file by comparing the
actual fabric to topology xml data.
Test for Switch Presence
(Switch) The Test for Switch Presence selection will verify that each Externally
Managed Switch specified in the switches file can be accessed by the Fabric
Management Node through the Fabric Network. If all switches were found, continue to
the next step. If some switches were not found, exit the menu and review the
following for those switches which were not found:
•
Is switch powered on and booted?
•
Is switch connected to Intel® Omni-Path Fabric?
•
Is Subnet Manager running?
•
Is Fabric Management node’s Port active?
•
Is Fabric Management node connected to the correct Intel® Omni-Path Fabric?
•
Is correct set of switches listed in the switches file (the previous step may be
repeated to review and edit the file as needed)?
Verify Switch Firmware
(Switch) The Verify Switch Firmware selection will check that each externally
managed switch is operational and that its firmware is valid and accessible.
Intel® Omni-Path Fabric Software
Installation Guide
48
November 2015
Order No.: H76467-1.0
Configure Firmware on the Externally Managed Intel® Omni-Path Switches—Intel® Omni-Path
Fabric
Update Switch Firmware
(Switch) The Update Switch Firmware selection will permit the switch firmware
version to be updated and the switch node name set. When this selection is
performed, the following message will be displayed:
Multiple Firmware files and/or Directories may be space separated
Shell wildcards may be used
For Directories all .emfw files in the directory tree will be used
Enter Files/Directories to use (or none):
Note:
Refer to the Intel® Omni-Path Fabric Externally-Managed Switches Release Notes to
ensure that any prerequisites for the upgrade to the new firmware level have been
met prior to performing the upgrade through FastFabric.
Perform the following procedure:
1. Specify the directory where the relevant firmware files have been stored. This can
be the mount point of the CD or the directory to which the files were copied in a
previous step.
The following message will display:
After upgrade, the switch may be optionally rebooted.
Would you like to reboot the switch after the update? [n]:
2. Type y
The following message will display:
The firmware on the switch will be checked, and if the running version is the
same
as the version being used for the update, the update operation will be
skipped.
Would you like to override this check, and force the update to occur? [n]:
3.
Press Enter to select default (n).
The fabric is not yet operational.
The following message will display:
You have selected to update the switch firmware and reboot.
There will be a disruption as switch or switches are rebooted.
Doing the operation in parallel (on multiple switches) will finish the
fastest.
Doing it serially may reduce disruption.
Would you like to do the operation in parallel? [y]:
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
49
Intel® Omni-Path Fabric—Configure Firmware on the Externally Managed Intel® Omni-Path
Switches
Note: Because the Intel® Omni-Path Fabric itself is used to update externally
managed switches, updating multiple switches with the reboot option may
disrupt parallel update operations. If there are not any selected externally
managed switches in the path from the Fabric Management node to any
other externally managed switch (for example, if the Fabric Management
node is connected directly to a core switch and externally managed switches
are only at the edges), parallel operations can be established. To control the
order of the rebooting of externally managed switches by FastFabric, see
the discussion of the distance option for the switches file or command in
the Intel® Omni-Path Fabric Suite FastFabric User Guide, Appendix A,
“Externally Managed Switch List File” subsection.
4.
Press Enter
or
Type n and press Enter if in doubt.
Note: Be aware that non-parallel operation for a fabric with many externally
managed switches could take a significant amount of time.
FastFabric will update the firmware on all switches and set the node names, as per the
switches file created in a previous step. Each switch will then be rebooted.
If any switch fails to be updated, use the View opaswitchadmin result files option
to review the result files from the update. Refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide for more details.
Set Up Switch Basic Configuration
(Switch) The Setup Switch Basic Configuration selection will permit the typical
switch setup operations to be performed for all switches.
Perform the following procedure:
1. When this procedure is started, the following message will be displayed:
Do you wish to configure the switch maximum MTU size? [n]:
2.
Press ENTER to select default (n).
This causes the default MTU of 8k to be used for all switches. If the switches have
previously been manually configured for a different MTU size, this option will keep
the previously configured MTU size. Refer to the Intel® Omni-Path Fabric Host
®
Software User Guide, “Intel Omni-Path Cluster Setup and Administration”, "MTU
Size" subsection for detailed information.
The following message will display:
Do you wish to configure the switch VL Capability? [n]:
3. Press ENTER to select default (n).
This causes the default VL Capability of 1 data VL to be used for all switches. If
switches have previously been manually configured for a different VL Capability,
this option will keep the previously configured VL Capability. See the Intel® OmniPath Fabric Switches GUI User Guide for more information.
Note: This operation is only applicable to Intel® Omni-Path Switch 100 Family
switches.
Intel® Omni-Path Fabric Software
Installation Guide
50
November 2015
Order No.: H76467-1.0
Configure Firmware on the Externally Managed Intel® Omni-Path Switches—Intel® Omni-Path
Fabric
The following message will display:
Do you wish to configure the switch VL Credit Distribution Method? [n]:
4. Press ENTER to select default (n).
This causes the default VL Credit Distribution Method setting of 4 to be used for all
switches. If switches have previously been manually configured for a different VL
Credit Distribution Method setting, those settings will remain unchanged. See the
Intel® Omni-Path Fabric Switches GUI User Guide for more information.
Note: This operation is only applicable to Intel® Omni-Path Switch 100 Family
switches.
The following message will display:
Do you wish to configure the switch Link Width Options? [n]:
5. Press ENTER to select default (n).
This causes the default switch Link Width Options to be used for all switches. If
switches have previously been manually configured for different switch Link Width
Options, this option will keep the previously configured switch Link Width Options.
See the Intel® Omni-Path Fabric Switches GUI User Guide for more information.
Selecting (y) will prompt for setting the switch link width supported setting for all
ports on all switches.
Note: This operation is only applicable to Intel® Omni-Path Switch 100 Family
switches.
The following message will display:
Do you wish to configure the switch Link Speed Options? [n]:
6. Press ENTER to select default (n).
This causes the default switch Link Speed Options to be used for all switches. If
switches have previously been manually configured for different switch Link Speed
Options, this option will keep the previously configured switch Link Speed Options.
See the Intel® Omni-Path Fabric Switches GUI User Guide for more information.
Selecting (y) will prompt for setting the switch link speed supported setting for all
ports on all switches.
Note: This operation is only applicable to Intel® Omni-Path Switch 100 Family
switches.
The following message will display:
Do you wish to configure the switch Node Description as it is set in the
switches file? [n]:
7. Press ENTER to select default (n).
This causes the default switch Node Description on each switch to be used. If the
switches have previously been manually configured for a customized switch Node
Description, this option will keep the previously configured switch Node
Descriptions. See the Intel® Omni-Path Fabric Switches GUI User Guide for more
information.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
51
Intel® Omni-Path Fabric—Configure Firmware on the Externally Managed Intel® Omni-Path
Switches
Selecting (y) causes the Node Description on each switch to be updated as
specified by the switches file.
Note: Only node descriptions on Intel® Omni-Path Switch 100 Family switches can
be changed in this step.
Reboot Switch
(Switch) The Reboot Switch selection will reboot all switches. This will ensure that
all the configuration changes become effective and are discovered by the Intel® OmniPath Fabric Suite Fabric Manager.
Report Switch Firmware and Hardware Info
(Switch) The Report Switch Firmware and Hardware Info selection will report
the firmware and hardware versions for each switch, along with the Capability (QDR,
DDR, or SDR), Fan Status, and Power Supply Status. Review the results against the
expected models and firmware versions.
(Switch) If any Intel® Omni-Path Switch 100 Family switches were purposely
skipped, these sections should be repeated for those switches. In this case, Intel
recommends that you create a separate file with a name other than switches. The
following is an example of the results:
Performing Switch Admin: Report Switch firmware & hardware info
Executing: /usr/sbin/opaswitchadmin -L /etc/sysconfig/opa/switches info
Executing report switch info Test Suite (switchinfo) day mmm dd hh:mm:ss timezone
yyyy ...
Executing TEST SUITE report switch info CASE (switchinfo.
0x00117500ff513121,Node_Name.i2c.extmgd.switchinfo)
retrieve switch 0x00117500ff513121,Node_Name ...
TEST SUITE report switch info CASE (switchinfo.
0x00117500ff513121,Node_Name.i2c.extmgd.switchinfo)
retrieve switch 0x00117500ff513121,Node_Name
0x00117500ff513121,hds1swb8171:
F/W ver:xx.xx.xx.xx H/W ver:XXX H/W pt num:NNNNNN-NNN
Fan status:Normal/Normal/Normal/Normal/Normal/Normal PS1 Status:N/A PS2
Status:ONLINE
PASSED
TEST SUITE report switch info: 1 Cases; 1 PASSED
TEST SUITE report switch info PASSED
Done report switch info Test Suite day mmm dd hh:mm:ss timezone yyyy
Get Basic Switch configuration
(Switch) The Get Basic Switch configuration selection executes the command to
get the switch configuration report (switchgetportconfig) for all of the ports. The
results show the number of cases, how many of the cases passed, and how many of
the cases failed. It also gives an overall summary of configuration and passed or
failed, as shown in the following example:
Performing Switch Admin: Get basic Switch configuration
Executing: /usr/sbin/opaswitchadmin -L /etc/sysconfig/opa/switches getconfig
Executing report switch getconfig Test Suite (switchgetportconfig) day mmm dd
hh:mm:ss timezone yyyy ...
Executing TEST SUITE report switch getconfig CASE (switchgetportconfig.
0x00117500ff513121,hds1swb8171.i2c
.extmgd.switchgetportconfig) retrieve switch 0x00117500ff513121,Node_Name ...
TEST SUITE report switch getconfig CASE (switchgetportconfig.
0x00117500ff513121,Node_Name.i2c
.extmgd.switchgetportconfig) retrieve switch 0x00117500ff513121,Node_Name
Link Width
: 1,2,3,4
Intel® Omni-Path Fabric Software
Installation Guide
52
November 2015
Order No.: H76467-1.0
Configure Firmware on the Externally Managed Intel® Omni-Path Switches—Intel® Omni-Path
Fabric
Link Speed
FM Enabled
Link CRC Mode
vCU
External Loopback Allowed
Node Description
:
:
:
:
:
:
25Gb
Yes
14-bit,16-bit,48-bit
0
Yes
Node_Name
PASSED
TEST SUITE report switch getconfig: 1 Cases; 1 PASSED
TEST SUITE report switch getconfig PASSED
Done report switch getconfig Test Suite day mmm dd hh:mm:ss timezone yyyy
Report Switch VPD Information
(Switch) The Report Switch VPD (vital product data) Information selection
executes the report switch hardware VPD Test Suite command (opaswitchadmin) for
all of the nodes listed in /etc/sysconfig/opa/switches. The results display the
VPD hardware information as shown in the following example:
Performing Switch Admin: Report Switch VPD information
Executing: /usr/sbin/opaswitchadmin -L /etc/sysconfig/opa/switches hwvpd
Executing report switch hwvpd Test Suite (switchhwvpd) day mmm dd hh:mm:ss
timezone yyyy ...
Executing TEST SUITE report switch hwvpd CASE (switchhwvpd.
0x00117500ff513121,Node_Name.i2c
.extmgd.switchhwvpd) retrieve switch 0x00117500ff513121,Node_Name ...
TEST SUITE report switch hwvpd CASE (switchhwvpd.0x00117500ff513121,Node_Name.i2c
.extmgd.switchhwvpd) retrieve switch 0x00117500ff513121,Node_Name
0x00117500ff513121,hds1swb8171: H/W VPD serial number: USFU13150000D
0x00117500ff513121,hds1swb8171: H/W VPD part number : NNNNNN-NNN
0x00117500ff513121,hds1swb8171: H/W VPD model
: 100SWE48QF2
0x00117500ff513121,hds1swb8171: H/W VPD h/w version : 004
0x00117500ff513121,hds1swb8171: H/W VPD manufacturer : Intel Corporation
0x00117500ff513121,hds1swb8171: H/W VPD prod desc
: 100 OP Edge 48p Q7
forward 2PSU
0x00117500ff513121,hds1swb8171: H/W VPD mfg id
: 001175
0x00117500ff513121,hds1swb8171: H/W VPD mfg date
: m-dd-yyyy
0x00117500ff513121,hds1swb8171: H/W VPD mfg time
: hh:mm
PASSED
TEST SUITE report switch hwvpd: 1 Cases; 1 PASSED
TEST SUITE report switch hwvpd PASSED
Done report switch hwvpd Test Suite day mmm dd hh:mm:ss timezone yyyy
View opaswitchadmin result files
(Switch) The View opaswitchadmin result files selection starts the editor to view
opaswitchadmin result files. The three files that are opened in the editor are /root/
punchlist.csv, /root/test.res, and /root/test.log.
When you exit the vi editor, the following question is displayed:
Would you like to remove test.res test.log test_tmp* and save_tmp
in /root ? [n]:
After you answer the question it takes you back to the TUI.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
53
Intel® Omni-Path Fabric—Install Host Software on the Remaining Hosts Using the FastFabric TUI
7.0
Install Host Software on the Remaining Hosts
Using the FastFabric TUI
In this procedure the Intel® Omni-Path Fabric Suite FastFabric is used to install and
configure the remaining hosts with the Host Softwareand verify overall operation of
the fabric.
Note:
Intel® Omni-Path Fabric Suite FastFabric is also used to install the Intel® Omni-Path
Fabric Stack Tools on the remaining hosts when using other variations of OFED. In this
case, OFED must be installed on each host prior to installing the OPA software.
1. (All) If the Intel FastFabric Tools menu is not displayed, type opafastfabric
at a command prompt and press Enter.
2.
(Linux) Press 3 to select Host Setup.
Displays the FastFabric OPA Host Setup Menu.
FastFabric OPA Host Setup Menu
Host File: /etc/sysconfig/opa/hosts
Setup:
0) Edit Config and Select/Edit Host File
1) Verify hosts pingable
2) Setup Password-less ssh/scp
3) Copy /etc/hosts to all hosts
4) Show uname -a for all hosts
5) Install/Upgrade OPA Software
6) Configure IPoIB IP Address
7) Build Test Apps and Copy to Hosts
8) Reboot Hosts
Admin:
9) Refresh ssh Known Hosts
a) Rebuild MPI Library and Tools
b) Run a command on all hosts
c) Copy a file to all hosts
Review:
d) View opahostadmin result files
P) Perform the selected actions
X) Return to Previous Menu (or ESC)
[
[
[
[
[
[
[
[
[
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
]
]
]
]
]
]
]
]
]
[
[
[
[
Skip
Skip
Skip
Skip
]
]
]
]
[ Skip
]
N) Select None
3. Select items 0 through 2 and 4 through 8.
4.
Press P.
Note: If /etc/hosts will be used for name resolution (as opposed to using DNS),
also select 3) Copy /etc/hosts to all hosts
5.
Perform the items selected using the descriptions in the following sections.
Edit Config and Select/Edit Host File
(All) The Edit Config and Select/Edit Host File selection will permit the hosts and
FastFabric configuration files to be edited. When placed in the editor for
opafastfabric.conf, review all the settings. In particular:
Intel® Omni-Path Fabric Software
Installation Guide
54
November 2015
Order No.: H76467-1.0
Install Host Software on the Remaining Hosts Using the FastFabric TUI—Intel® Omni-Path Fabric
•
FF_IPOIB_SUFFIX
•
FF_IPOIB_NETMASK
•
FF_PRODUCT
•
FF_PACKAGES
•
FF_INSTALL_OPTIONS
•
FF_UPGRADE_OPTIONS
•
FF_ALL_ANALYSIS
Refer to "Configuration Files for FastFabric" section of the Intel® Omni-Path Fabric
Suite FastFabric User Guide for more information about the opafastfabric.conf
file.
Note:
During setup of password-less ssh, FastFabric will provide the opportunity to enter the
host root password interactively when needed. Therefore, Intel recommends that you
do not place it within the opafastfabric.conf file. If you are required to keep the
root password for the hosts in the opafastfabric.conf file, Intel recommends that
you change the opafastfabric.conf permissions to be 0x600 (root-only access).
When placed in the editor for hosts, create the file with a list of the hosts names (the
TCP/IP management network names), except the Fabric Management Node from
which Intel® Omni-Path Fabric Suite FastFabric is presently being run. Enter one
host’s name per line. For example:
host1
host2
Note:
Do not list the Fabric management node itself (the node where Intel® Omni-Path
Fabric Suite FastFabric is currently running).
If additional Fabric Management Nodes are to be used, they may be listed at this time,
and Intel® Omni-Path Fabric Suite FastFabric can aid in their initial installation and
verification.
For further details about the file format, refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide.
Verify hosts pingable
(All) The Verify hosts pingable selection will ping each selected host over the
management network.
The following system prompt will be seen when you perform this selection:
Performing Host Setup: Verify hosts pingable
Would you like to verify hosts are ssh-able? [n]:
Press ENTER to select the default (n).
If all hosts were found, continue to the next step. If some hosts were not found, exit
the menu and review the following for those hosts that were not found:
•
Host powered on and booted?
•
Host connected to management network?
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
55
Intel® Omni-Path Fabric—Install Host Software on the Remaining Hosts Using the FastFabric TUI
•
Host management network IP address and network settings consistent with DNS
or /etc/hosts?
•
Management node connected to the management network?
•
Management node IP address and network settings correct?
•
Management network itself up (switches, routers, etc)?
•
Correct set of hosts listed in the hosts file (the previous step may be repeated to
review and edit the file as needed)?
Setup Password-less ssh/scp
(Linux) The Setup Password-less ssh/scp selection will set up secure passwordless ssh (root password) such that the Fabric Management Node can securely log in to
all the other hosts as root through the management network without requiring a
password.
The following system prompt will be seen when you perform this selection:
Performing Host Setup: Setup password-less ssh/scp
Executing: /usr/sbin/opasetupssh -S -p -i '' -f /etc/sysconfig/opa/hosts
Password for root on all hosts:
Type the password for root on all host and press ENTER
Password-less ssh is required by Intel® Omni-Path Fabric Suite FastFabric, MPI test
applications, and most versions of MPI (including OpenMPI and MVAPICH2).
Copy /etc/hosts to all hosts
(Linux) The Copy /etc/hosts to all hosts selection will copy the /etc/hosts file
on this host to all the other selected hosts.
Note:
If DNS is being used, skip this step.
Note:
Typically, /etc/resolv.conf is set up as part of OS installation for each host.
However, if /etc/resolv.conf was not set up on all the hosts during OS
installation, the FastFabric Copy a file to all hosts operation could be used at this
time to copy /etc/resolv.conf from the Fabric Management Node to all the other
nodes.
Show uname -a for all hosts
(Linux) The Show uname -a for all hosts selection will show the OS version on all
the hosts. Review the results carefully to verify all the hosts have the expected OS
version. In typical clusters, all hosts will be running the same OS and kernel version.
If any hosts are identified with an incorrect OS version, the OS on those hosts should
be corrected at this time and operation of this sequence should be aborted when
prompted. As necessary, all the preceding setup steps should then be repeated for
those hosts (there is no harm in repeating them for all the hosts).
Intel® Omni-Path Fabric Software
Installation Guide
56
November 2015
Order No.: H76467-1.0
Install Host Software on the Remaining Hosts Using the FastFabric TUI—Intel® Omni-Path Fabric
Install/Upgrade Intel OPA Software
(Host) The Install/Upgrade Intel OPA Software selection will install the Intel®
Omni-Path Fabric Host Software on all the hosts. By default, it will look in the current
directory for the IntelOPA-Basic.DISTRO.VERSION.tgz file. If the tarball is not
found in the current directory, the installer application will prompt for input of a
directory name where this file can be found.
Note:
An initial installation will uninstall any existing OFED Delta or IFS software. Initial
installs must be performed when installing on a clean system or on a system which
has stock OFED installed. To upgrade the fabric, refer to Upgrade the Fabric Software
on page 96.
Perform the following steps to install the selected hosts:
1.
The Install/Upgrade Intel Software selection will start with the following
system prompts:
Performing Host Setup: Install/Upgrade Intel OPA Software
Do you want to use ./IntelOPA-Basic.DISTRO.VERSION.tgz? [y]:
2.
Press ENTER to accept the default (y).
System prompts:
Would you like to do an upgrade/reinstall? [y]:
3.
Type n and press ENTER.
System prompts:
Would you like to do an initial installation? [n]:
4. Type y and press ENTER to proceed.
System prompts:
You have selected to perform an initial installation
This will uninstall any existing Intel OPA software on the selected nodes
Are you sure you want to proceed? [n]:
5.
Type y and press ENTER to proceed.
System prompts:
Executing: /sbin/opahostadmin -f /etc/sysconfig/opa/hosts -d .. load
.
.
.
Hit any key to continue (or ESC to abort)...
6. Press any key to proceed.
Once the installation is complete on the selected hosts the FastFabric OPA Host
Setup Menu appears.
The installation is complete.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
57
Intel® Omni-Path Fabric—Install Host Software on the Remaining Hosts Using the FastFabric TUI
Note:
If any hosts fail to be installed, for example:
TEST SUITE load: 1 Cases; 0 PASSED; 1 FAILED
TEST SUITE load FAILED
Use the View opahostadmin result files option to review the result files from the
update. For more details, refer to the Intel® Omni-Path Fabric Suite FastFabric
Command Line Interface Reference Guide.
Configure IPoIB IP Address
(Host) The Configure IPoIB IP Address selection will create the ifcfg-ib0 files
on each host (previous non-OFED releases created the ifcfg-ib1 file). The file will
be created with a statically assigned IPv4 address. The IPoIB IP address for each host
will be determined by the resolver (Linux* host command). If not found through the
resolver, /etc/hosts on the given host will be checked.
Build Test Apps and Copy to Hosts
(Host) The Build Test Apps and Copy to Hosts selection will build the MPI and/or
SHMEM sample applications on the Fabric Management Node and copy the resulting
object files to all the hosts. This is in preparation for execution of MPI and/or SHMEM
performance tests and benchmarks in a later step.
Note:
This option is only available when using the Intel® Omni-Path Fabric Host Software
packaging (OFED Delta).
Reboot Hosts
(Linux) The Reboot Hosts selection will reboot all the selected hosts and ensure
they fully reboot, (as verified through ping over the management network). When the
hosts come back up, they will be running the newly installed Fabric software.
Refresh ssh Known Hosts
(Linux) The Refresh ssh Known Hosts will run the opasetupssh -U “”
command to refresh the ssh known hosts list on this server for the Management
Network. This may be used to update security for this host if hosts are replaced,
reinstalled, renamed, or repaired.
Rebuild MPI Library and Tools
(Host) The Rebuild MPI Library and Tools will rebuild the MPI Library and related
tools (such as mpirun), and install the resulting rpms on all the hosts. This will be
performed using the do_build tool supplied with the MPI Source. When rebuilding
MPI, do_build will prompt you for selection of which MPI (openmpi or mvapich2) to
rebuild, and will provide choices as to which available compiler to use. Refer to (Intel®
Omni-Path Host Fabric Interface Installation Guide) and Intel® Omni-Path Fabric Host
Software User Guide for more information.
Note:
This option is only available when using with the Intel® Omni-Path Fabric Host
Software (OFED Delta).
Intel® Omni-Path Fabric Software
Installation Guide
58
November 2015
Order No.: H76467-1.0
Install Host Software on the Remaining Hosts Using the FastFabric TUI—Intel® Omni-Path Fabric
Run a command on all hosts
(Linux) If there are any other setup operations that need to be performed on all
hosts, they may be performed using the Run a command on all hosts option. Each
time this is executed, a Linux* shell command (or sequence of commands separated
by semicolons) may be specified to be executed against all selected hosts.
Note:
Intel recommends that you run the date command on all hosts to verify that the date
and time are consistent on all hosts. If needed, the Copy a file to all hosts option
may be used to copy the appropriate files to all hosts in order to enable and configure
NTP.
Copy a file to all hosts
(Linux) The Copy a file to all hosts selection will run the opascpall command. A
file on the local host may be specified to be copied to all selected hosts.
View opahostadmin result files
(All) The View opahostadmin result file permits viewing of the test.log and
test.res files that reflect the results from opahostadmin runs (such as for
installing Fabric software or rebooting all hosts per Reboot Hosts selection described
previously). You are also given the option to remove these files after viewing them.
If not removed, subsequent runs of opachassisadmin, opahostadmin, or
opaswitchadmin from within the current directory will continue to append to these
files.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
59
Intel® Omni-Path Fabric—Install Host Software on the Remaining Hosts using CLI Commands
8.0
Install Host Software on the Remaining Hosts
using CLI Commands
This procedure provides step-by-step information to install the Intel® Omni-Path
Fabric Host Software on the remaining servers.
1.
(Optional) Edit or review the /etc/sysconfig/opa/opafastfabric.conf file.
Review the complete file. In particular review the following:
•
FF_IPOIB_SUFFIX
•
FF_IPOIB_NETMASK
•
FF_PRODUCT
•
FF_PACKAGES
•
FF_INSTALL_OPTIONS
•
FF_UPGRADE_OPTIONS
•
FF_ALL_ANALYSIS
Refer to "Configuration Files for FastFabric" section of the Intel® Omni-Path Fabric
Suite FastFabric User Guide for more information about the
opafastfabric.conf file.
Note: During setup of password-less ssh, FastFabric will provide the opportunity to
enter the host root password interactively when needed. Therefore, Intel
recommends that you do not place it within the opafastfabric.conf file.
If you are required to keep the root password for the hosts in the
opafastfabric.conf file, Intel recommends that you change the
opafastfabric.conf permissions to be 0x600 (root-only access).
2.
(Optional) Create, Edit, or review the /etc/sysconfig/opa/hosts file.
When placed in the editor for hosts, create the file with a list of the hosts names
(the TCP/IP management network names), except the Fabric Management Node
from which Intel® Omni-Path Fabric Suite FastFabric is presently being run. Enter
one host’s name per line. For example:
host1
host2
Note: Do not list the Fabric management node itself (the node where Intel® OmniPath Fabric Suite FastFabric is currently running).
If additional Fabric Management Nodes are to be used, they may be listed at this
time, and Intel® Omni-Path Fabric Suite FastFabric can aid in their initial
installation and verification.
For further details about the file format, refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide.
3.
Verify the hosts is pingable over the management network.
Intel® Omni-Path Fabric Software
Installation Guide
60
November 2015
Order No.: H76467-1.0
Install Host Software on the Remaining Hosts using CLI Commands—Intel® Omni-Path Fabric
opapingall -p
If all hosts were found, continue to the next step. If some hosts were not found,
exit the menu and review the following for those hosts that were not found:
4.
•
Host powered on and booted?
•
Host connected to management network?
•
Host management network IP address and network settings consistent with
DNS or /etc/hosts?
•
Management node connected to the management network?
•
Management node IP address and network settings correct?
•
Management network itself up (switches, routers, etc)?
•
Correct set of hosts listed in the hosts file (the previous step may be repeated
to review and edit the file as needed)?
Set up secure password-less ssh, such that the Fabric Management Node can
securely log into all the hosts as root through the management network, without
requiring a password.
opasetupssh -S -p -i ipoib_suffix -f hostfile
5. (Optional) Copy the /etc/hosts file on this host to all the other selected hosts.
opascpall -p -f hostfile /etc/hosts /etc/hosts
Note: If DNS is being used, skip this step.
6.
(Optional) Copy the /etc/resolv.conf file on this host to all the other selected
hosts.
opascpall -p -f hostfile /etc/resolv.conf /etc/resolv.conf
7.
Show the OS version on all the hosts.
opacmdall -T 60 -f hostfile 'uname -a'
8.
Install the Intel® Omni-Path Fabric Host Software on all the hosts.
opahostadmin -f hostfile -d dir load
By default, it will look in the current directory for the IntelOPABasic.DISTRO.VERSION.tgz file.
Note: An initial installation will uninstall any existing OFED Delta or IFS software.
Initial installs must be performed when installing on a clean system or on a
system which has stock OFED installed. For additional details, refer to
Upgrade the Fabric Software on page 96.
Note: If any hosts fail to be installed, for example:
TEST SUITE load: 1 Cases; 0 PASSED; 1 FAILED
TEST SUITE load FAILED
Use the View opahostadmin result files option to review the result files from
the update. For more details, refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide.
9. Create the ifcfg-ib0 files on each host (previous non-OFED releases created
the ifcfg-ib1 file).
opahostadmin -f hostfile configipoib
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
61
Intel® Omni-Path Fabric—Install Host Software on the Remaining Hosts using CLI Commands
The file will be created with a statically assigned IPv4 address. The IPoIB IP
address for each host will be determined by the resolver (Linux* host command).
If not found through the resolver, /etc/hosts on the given host will be checked.
10. Build the MPI and/or SHMEM sample applications on the Fabric Management Node
and copy the resulting object files to all the hosts.
opascpall -t -p -f hostfile source_dir dest_dir
Note: This is in preparation for execution of MPI and/or SHMEM performance tests
and benchmarks in a later step.
Note: This option is only available when using the Intel® Omni-Path Fabric Host
Software packaging of OFED (OFED Delta).
11. Reboot all the selected hosts and ensure they fully reboot, (as verified through
ping over the management network).
opahostadmin -f hostfile reboot
When the hosts come back up, they will be running the newly installed Fabric
software.
Intel® Omni-Path Fabric Software
Installation Guide
62
November 2015
Order No.: H76467-1.0
Verify Intel® Omni-Path Fabric Host Software on the Remaining Servers—Intel® Omni-Path
Fabric
9.0
Verify Intel® Omni-Path Fabric Host Software on
the Remaining Servers
Upon completion of the preceding sections, the hosts are all booted, installed and
operational. The subsequent steps will verify the operation of the hosts and fabric.
1.
(All) If the FastFabric OPA Host Verification/Admin Menu is not displayed,
type opafastfabric and press Enter.
2.
(All) Press 4.
Displays the FastFabric OPA Host Verification/Admin Menu.
FastFabric OPA Host Verification/Admin Menu
Host File: /etc/sysconfig/opa/allhosts
Validation:
0) Edit Config and Select/Edit Host File
1) Summary of Fabric Components
2) Verify hosts pingable, sshable and active
3) Perform Single Host verification
4) Verify OPA Fabric status and topology
5) Verify Hosts see each other
6) Verify Hosts ping via IPoIB
7) Refresh ssh Known Hosts
8) Check MPI Performance
9) Check Overall Fabric Health
a) Start or Stop Bit Error Rate Cable Test
Admin:
b) Generate all Hosts Problem Report Info
c) Run a command on all hosts
Review:
d) View opahostadmin result files
P) Perform the selected actions
X) Return to Previous Menu (or ESC)
3.
[
[
[
[
[
[
[
[
[
[
[
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
Skip
]
]
]
]
]
]
]
]
]
]
]
[ Skip
[ Skip
]
]
[ Skip
]
N) Select None
Select the items 0 through 8 in the Validation section of the menu.
4. Press P.
Edit Config and Select/Edit Host File
(All) The Edit Config and Select/Edit Host File section will permit the hosts, ports,
and FastFabric configuration files to be edited. When placed in the editor for
opafastfabric.conf, review all the settings. Especially review the settings for the
following:
•
FF_TOPOLOGY_FILE
•
FF_IPOIB_SUFFIX
Refer to Intel® Omni-Path Fabric Suite FastFabric User Guide Configuration Files
section for more information about opafastfabric.conf.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
63
Intel® Omni-Path Fabric—Verify Intel® Omni-Path Fabric Host Software on the Remaining
Servers
If required, a FastFabric topology file may be created as /etc/sysconfig/opa/
topology.0:0.xml to describe the intended topology of the fabric. The file can also
augment assorted fabric reports with customer-specific information, such as cable
labels and additional details about nodes, SMs, links, ports, and cables. Refer to the
Intel® Omni-Path Fabric Suite FastFabric Command Line Interface Reference Guide for
more information about topology verification files.
Review the following parameters which will be used for overall fabric health checks:
•
FF_ANALYSIS_DIR
•
FF_ALL_ANALYSIS
•
FF_FABRIC_HEALTH
•
FF_CHASSIS_CMDS
•
FF_CHASSIS_HEALTH
•
FF_ESM_CMDS
FF_ALL_ANALYSIS should be updated to reflect the type of SM (esm or hostsm).
When placed in the editor for ports, review the file. For typical single-subnet clusters,
the default of 0:0 may be used. This will use the first active port on the Fabric
Management node to access the fabric. For more information on configuring a cluster
with multiple subnets, see Multi-Subnet Fabrics on page 101. For further details about
the file format, refer to the "Selection of Ports" section in the Intel® Omni-Path Fabric
Suite FastFabric Command Line Interface Reference Guide.
When placed in the editor for allhosts, create the file with the Fabric Management
node's hosts names (the TCP/IP management network names) (shown as mgmthost
for example) and include the hosts file previously created. Enter one per line. For
example:
mgmthost
include /etc/sysconfig/opa/hosts
For further details about the file format, refer to the "Selection of Hosts" section in the
Intel® Omni-Path Fabric Suite FastFabric Command Line Interface Reference Guide.
Summary of Fabric Components
(All) The Summary of Fabric Components selection will provide a brief summary of
the counts of components in the fabric, including how many switch chips, hosts, and
links are in the fabric. It will also indicate if any 1x links were found (which could
indicate a poorly seated or bad cable). Review the results against the expected
configuration of the cluster.
If components are missing or 1x links are found, they should be corrected.
Subsequent steps will aid in locating any 1x links.
Verify hosts pingable, sshable, and active
(All) The Verify hosts pingable, sshable, and active selection will verify each host
and provide a concise summary of the bad hosts found. Interactive prompts allow you
to select ping, ssh, and port active verification. After completion of this test, you will
have the option of using the resulting good hosts file for the remainder of the
operations within this TUI session.
Intel® Omni-Path Fabric Software
Installation Guide
64
November 2015
Order No.: H76467-1.0
Verify Intel® Omni-Path Fabric Host Software on the Remaining Servers—Intel® Omni-Path
Fabric
Perform Single Host verification
(All) The Perform Single Host verification uses the opaverifyhosts command to
perform a single host test on all hosts. Refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide for information on
opaverifyhosts.
Verify OPA Fabric status and topology
(All) The Verify OPA Fabric status and topology selection can run the following
checks:
•
Perform a fabric error analysis
•
Clear error counters after generating the report
•
Perform a fabric link speed error analysis
•
Check for links that are configured to run slower than supported
•
Check links that are connected with mismatched speed potential
•
Verify the fabric topology
•
Verify all aspects of the topology including links, nodes, and sms
•
Include unexpected devices in the punch-list
Verify Hosts see each other
(Host) The Verify Hosts see each other selection will verify that each host can see
all the others through queries to the Subnet Administrator.
Verify Hosts ping via IPoIB
(Host) The Verify Hosts ping via IPoIB selection will verify that IPoIB is properly
configured and running on all the hosts. This is accomplished through the Fabric
management node pinging each host through IPoIB.
Note:
This operation requires that IPoIB is enabled on the Fabric Management Node as well
as each host selected for verification.
1.
The management host needs to have IPoIB configured.
2. Depending on the MTU of the fabric, this may not be successful.
Refresh ssh Known Hosts
(Linux) The Refresh ssh Known Hosts selection will refresh the ssh known_hosts
file on the Fabric management node to include the IPoIB hostnames of all the hosts.
Note:
This operation requires that IPoIB is enabled on the Fabric Management Node as well
as each host selected for verification.
Check MPI Performance
(Host) The MPI Performance selection will do a quick check of PCIe and MPI
performance through end-to-end latency and bandwidth tests.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
65
Intel® Omni-Path Fabric—Verify Intel® Omni-Path Fabric Host Software on the Remaining
Servers
1.
When MPI Performance is selected, the following prompt is displayed:
Test Latency and Bandwidth deviation between all hosts? [y]:
2.
Press Enter to select default (y).
The following prompt is displayed:
View Load on hosts prior to test? [y]:
3. Press Enter to select default (y).
This displays the results of pair-wise analysis of latency and bandwidth for the
selected hosts and reports pairs outside an acceptable tolerance range. By default,
performance is compared relative to other hosts in the fabric (with the assumption
that all hosts selected for a given run should have comparable fabric performance).
Failing hosts will be clearly indicated.
The results are also written to the test.res file, which may be viewed through the
View opahostadmin result files option. Refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide for more details.
If any hosts fail, carefully examine the failing hosts to verify the HFI models, PCIe slot
used, BIOS settings, and any motherboard jumpers related to devices on PCIe buses
or slot speeds. Also verify the HFI and riser cards are properly seated.
Check Overall Fabric Health
(ALL) The Check Overall Fabric Health selection will permit the present fabric
configuration to be base-lined for use in future fabric health checks. This should be
performed after configuring any additional Fabric Management Nodes. Refer to
Configure and Initialize Health Check Tools on page 74 for more information.
Start or Stop Bit Error Rate Cable Test
(ALL) The Start or Stop Bit Error Rate Cable Test selection performs host and/or
ISL cable testing. The test allows for starting and stopping an extended Bit Error Rate
test. Intel recommends that this test be run for seven hours for a thorough test.
Generate all Hosts Problem Report Info
(Host) The Generate all Hosts Problem Report Info will run the captureall
command to collect configuration and status information from all hosts and generate a
single *.tgz file which can be sent to the Support Representative.
Based on the answer to the following prompt, various levels of detail about the fabric
can be included in the capture.
Capture detail level (1-Normal, 2-Fabric, 3-Fabric+FDB, 4-Analysis):
The Detail levels are:
•
1-Normal – Obtains local information from each host
•
2-Fabric – In addition to “Normal”, also obtains basic fabric information by
queries to the SM and fabric error analysis using iba_report.
Intel® Omni-Path Fabric Software
Installation Guide
66
November 2015
Order No.: H76467-1.0
Verify Intel® Omni-Path Fabric Host Software on the Remaining Servers—Intel® Omni-Path
Fabric
•
3-Fabric+FDB – In addition to “Fabric”, also obtains all the switch forwarding
tables and OPA multicast membership lists from the SM.
•
Note:
4-Analysis – In addition to “Fabric+FDB”, also obtains all_analysis results.
If all_analysis has not yet been run, it is run as part of the capture.
Detail levels 2-4 can be used when fabric operational problems occur. If the problem
is most likely node specific, detail level 1 should be sufficient. Detail levels 2-4 require
an operational Fabric Manager. Typically your support representative will request a
given detail level. If a given detail level takes excessively long or fails to be gathered,
try a lower detail level.
For detail levels 2-4, the additional information is only gathered on the node running
the captureall command. The information is gathered for every fabric specified in
the /etc/sysconfig/opa/ports file.
Run a command on all hosts
(Linux) The Run a command on all hosts will run the cmdall command. A Linux*
shell command (or sequence of commands separated by semicolons) may be specified
to be executed against all selected hosts.
View opahostadmin result files
(All) The View opahostadmin result files permits viewing of the test.log and
test.res files, which reflect the results from opahostadmin runs (such as those for
installing fabric software or rebooting all hosts). You are also given the option to
remove these files after viewing them.
If not removed, subsequent runs of opachassisadmin, opahostadmin, or
opaswitchadmin from within the current directory will continue to append to these
files.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
67
Intel® Omni-Path Fabric—Verify Intel® Omni-Path Fabric Host Software is installed on the
Remaining Servers using CLI Commands
10.0
Verify Intel® Omni-Path Fabric Host Software is
installed on the Remaining Servers using CLI
Commands
Prerequisites: Installation of the Intel® Omni-Path Fabric Host Software has been
completed on the remaining servers.
This procedure provides step-by-step information to verify that Intel® Omni-Path
Fabric Host Software is installed and running on the remaining servers using CLI
Commands.
1.
(Optional) Edit or review the /etc/sysconfig/opa/opafastfabric.conf file.
Review all of the settings. In particular review the following:
•
FF_TOPOLOGY_FILE
•
FF_IPOIB_SUFFIX
•
ff_host_basename_to_ipoib
•
ff_host_basename
Refer to "Configuration Files for FastFabric" section of the Intel® Omni-Path Fabric
Suite FastFabric User Guide for more information about the
opafastfabric.conf file.
2. (Optional) Edit, or review the /etc/sysconfig/opa/ports file.
When placed in the editor for ports, review the file. For typical single-subnet
clusters, the default of 0:0 may be used. This will use the first active port on the
Fabric Management node to access the fabric.Enter one host’s name per line. For
further details about the file format, refer to the "Selection of Ports" section in the
Intel® Omni-Path Fabric Suite FastFabric Command Line Interface Reference
Guide
3. (Optional) Edit, or review the /etc/sysconfig/opa/allhosts file.
When placed in the editor for allhosts, review the file, it should include the Fabric
Management node's hosts names (the TCP/IP management network names)
(shown as mgmthost for example) and include the hosts file previously created.
Enter one per line. For example:
mgmthost
include /etc/sysconfig/opa/hosts
For further details about the file format, refer to the Selection of Hosts section in
the Intel® Omni-Path Fabric Suite FastFabric Command Line Interface Reference
Guide.
4. Provide a brief summary of the counts of components in the fabric, including how
many switch chips, hosts, and links are in the fabric.
fabric_info
Intel® Omni-Path Fabric Software
Installation Guide
68
November 2015
Order No.: H76467-1.0
Verify Intel® Omni-Path Fabric Host Software is installed on the Remaining Servers using CLI
Commands—Intel® Omni-Path Fabric
It will also indicate if any 1x links were found (which could indicate a poorly seated
or bad cable). Review the results against the expected configuration of the cluster.
If components are missing or 1x links are found, they should be corrected.
Subsequent steps will aid in locating any 1x links.
5.
Verify each host is pingable.
opapingall -p -f hostfile
If all hosts were found, continue to the next step. If some hosts were not found,
exit the menu and review the following for those hosts that were not found:
•
Host powered on and booted?
•
Host connected to management network?
•
Host management network IP address and network settings consistent with
DNS or /etc/hosts?
•
Management node connected to the management network?
•
Management node IP address and network settings correct?
•
Management network itself up (switches, routers, etc)?
•
Correct set of hosts listed in the hosts file (the previous step may be repeated
to review and edit the file as needed)?
6. Verify each host is pingable, ssh'able and active on the Intel® Omni-Path Fabric
and produce a list of good hosts meeting all criteria.
opafindgood -R -A -Q -f hostfile
The following files are created in opasorthosts order with all duplicates
removed in the OPA_CONFIG_DIR/ directory:
•
good
•
alive
•
running
•
active
•
bad
The resulting good file can then be used in as input to create mpi_hosts files for
use running mpi_apps and the HFI-SW cable test.
7.
Perform a single host test on all hosts.
opaverifyhosts -k -c -u hostverify.res -T timelimit -f
hostfile test
Refer to the Intel® Omni-Path Fabric Suite FastFabric Command Line Interface
Reference Guide for information on opaverifyhosts.
The results can be seen in the $FF_RESULT_DIR/verifyhosts.res file. A
punch list of failures is also appended to the $FF_RESULT_DIR/punchlist.csv
file.
8.
Verify OPA Fabric status and topology.
opalinkanalysis -U -x snapshot_suffix all verifyall >
$FF_RESULT_DIR/linkanalysis.res 2>&1
To clear error counters after generating the report add clearerrors and
clearhwerrors options to the opalinkanalysis run.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
69
Intel® Omni-Path Fabric—Verify Intel® Omni-Path Fabric Host Software is installed on the
Remaining Servers using CLI Commands
The following items are verified:
•
Perform a fabric error analysis
•
Perform a fabric link speed error analysis
•
Check for links that are configured to run slower than supported
•
Check links that are connected with mismatched speed potential
•
Verify the fabric topology
•
Verify all aspects of the topology including links, nodes, and sms
•
Include unexpected devices in the punch-list
Refer to the Intel® Omni-Path Fabric Suite FastFabric Command Line Interface
Reference Guide for information on opalinkanalysis.
The results can be seen in the $FF_RESULT_DIR/linkanalysis.res file. A
punch list of issues is appended to the $FF_RESULT_DIR/punchlist.csv file.
9.
Verify that each host can see all the others through queries to the Subnet
Administrator.
opahostadmin -f hostfile sacache
10. Verify that IPoIB is properly configured and running on all the hosts.
opahostadmin -f hostfile ipoibping
If this is unsuccessful, verify that
•
The management host has IPoIB configured.
•
The MTU of the fabric must be correct.
11. Refresh the ssh known_hosts file on the Fabric Management Node to include the
IPoIB hostnames of all the hosts.
opasetupssh -p -U -f hostfile
12. Perform a quick check of PCIe and MPI performance through end-to-end latency
and bandwidth tests.
opahostadmin -f hostfile mpiperfdeviation
This displays the results of pair-wise analysis of latency and bandwidth for the
selected hosts and reports pairs outside an acceptable tolerance range. By default,
performance is compared relative to other hosts in the fabric (with the assumption
that all hosts selected for a given run should have comparable fabric
performance). Failing hosts will be clearly indicated.
The results can be seen in the test.res file.
Refer to the Intel® Omni-Path Fabric Suite FastFabric Command Line Interface
Reference Guide for more details on opahostadmin.
If any hosts fail, carefully examine the failing hosts to verify the HFI models, PCIe
slot used, BIOS settings, and any motherboard jumpers related to devices on PCIe
buses or slot speeds. Also verify the HFI and riser cards are properly seated.
13. Baseline the present fabric configuration for use in future fabric health checks.
opaallanalysis -b
This should be performed after configuring any additional Fabric Management
Nodes. Refer to Configure and Initialize Health Check Tools on page 74 for more
information.
Intel® Omni-Path Fabric Software
Installation Guide
70
November 2015
Order No.: H76467-1.0
Verify Intel® Omni-Path Fabric Host Software is installed on the Remaining Servers using CLI
Commands—Intel® Omni-Path Fabric
14. Perform host and/or ISL cable testing.
opacabletest stop -C -A -n numprocs -f hostfile start
The test allows for starting and stopping an extended Bit Error Rate test. Intel
recommends that this test be run for seven hours for a thorough test.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
71
Intel® Omni-Path Fabric—Installation of Additional Fabric Management Nodes
11.0
Installation of Additional Fabric Management
Nodes
If the fabric has more than one Fabric Management Node, use the procedures in this
section to set up additional management nodes. Previous sections in this installation
guide described how to perform basic software installation, setup and verification on
the nodes in a fabric. This section describes how to install and configure the
management software itself.
Note:
The following steps assume a symmetrical configuration where all Fabric management
nodes have the same connectivity and capabilities. In asymmetrical configurations
where the Fabric management nodes are not all connected to the same set of
management networks and subnets, the files copied to each management node may
need to be slightly different. For example, configuration files for fabric analysis may
indicate different port numbers, or host files used for FastFabric, and MPI may need to
list different hosts. For multiple-subnet configurations, refer to Multi-Subnet Fabrics on
page 101.
Repeat the following steps on each additional Fabric Management Node:
1.
(All) Upgrade the software to add additional components using the procedure
documented in Upgrade from IntelOPA-Basic to IntelOPA-IFS on page 99. The
Fabric Management node must have at least Intel® Omni-Path Fabric Suite
FastFabric, the Intel® Omni-Path Fabric Stack, and should have IPoIB installed
and configured. For MPI clusters, the Fabric Management node should also include
at least OFED openmpi or OFED mvapich2. If you plan to rebuild MPI, the Intel®
Omni-Path Fabric Host Software Fabric Development package and MPI Source
packages will also be required.
Note: Do not uninstall or replace existing configuration files that were previously
created, especially IPoIB-related configuration files.
2. (All) Copy the FastFabric configuration files from the initial Fabric Management
Node. At a minimum, the following files should be copied:
/etc/sysconfig/opa/opafastfabric.conf
/etc/sysconfig/opa/ports
/etc/sysconfig/opa/topology*.xml
/etc/sysconfig/opa/hosts
/etc/sysconfig/opa/allhosts
/etc/sysconfig/opa/switches
/etc/sysconfig/opa/chassis
After copying the files, edit the hosts and allhosts files such that the file on
each Fabric Management node omits itself from the hosts files (but lists all other
Fabric Management nodes) and specifies itself in the allhosts file.
See Intel® Omni-Path Fabric Suite FastFabric User Guide Configuration Files
section for a complete list of FastFabric configuration files.
Intel® Omni-Path Fabric Software
Installation Guide
72
November 2015
Order No.: H76467-1.0
Installation of Additional Fabric Management Nodes—Intel® Omni-Path Fabric
3.
(All) Copy the Fabric Manager configuration file (/etc/sysconfig/opafm.xml)
from the initial Fabric Management Node, if the Fabric Manager is also going to be
run. After copying the file, edit the file on each Fabric Management node as
needed.
Refer to the Intel® Omni-Path Fabric Suite Fabric Manager User Guide for more
information on how to configure the Fabric Manager.
4.
(Linux) Perform the Setup Password-less ssh/scp option in the Host Setup
via FastFabric menu.
5.
Perform the Refresh ssh Known Hosts option in the Host Admin via
FastFabric menu.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
73
Intel® Omni-Path Fabric—Configure and Initialize Health Check Tools
12.0
Configure and Initialize Health Check Tools
For more information on the health check tools, see the detailed discussion in Intel®
Omni-Path Fabric Suite FastFabric Command Line Interface Reference Guide. The
health check tools may be run on one or more Fabric management nodes within the
cluster. This procedure should be followed on each Fabric management node from
which the health check tools will be used.
1.
(All) Edit opafastfabric.conf and review the following parameters:
•
FF_ANALYSIS_DIR
•
FF_ALL_ANALYSIS
•
FF_FABRIC_HEALTH
•
FF_CHASSIS_CMDS
•
FF_CHASSIS_HEALTH
•
FF_ESM_CMDS
FF_ALL_ANALYSIS should be updated to reflect the type of SM (esm or hsm).
Note: If you are running in a back-to-back configuration, FF_ALL_ANALYSIS
should not contain chassis.
2.
(All) Create /etc/sysconfig/opa/esm_chassis, listing the chassis that are
running SMs if using Embedded SM(s) in theIntel® Omni-Path Fabric Chassis.
Create the file with a list of the chassis names using the assigned TCP/IP Ethernet
management port names. IP addresses can also be used; however, Intel
recommends that you use names instead. For example:
Chassis1
Chassis2
For further details about the file format, refer to the Intel® Omni-Path Fabric Suite
FastFabric Command Line Interface Reference Guide.
3.
(All) Perform a health check using opaallanalysis -e. If any errors are
encountered, resolve the errors and rerun opaallanalysis -e until a clean run
occurs.
4. (All) Create a cluster configuration baseline using: opaallanalysis -b.
Note: This may also be done using the FastFabric TUI menu by selecting Check
Overall Fabric Health and press y to Baseline present
configuration? [n]: question.
5. (All) If required, schedule regular runs of opaallanalysis through cron or other
mechanisms. Refer to the Linux* OS documentation for more information on cron.
Also refer to the Intel® Omni-Path Fabric Suite FastFabric Command Line Interface
Reference Guide for more information about opaallanalysis and its automated
use.
Intel® Omni-Path Fabric Software
Installation Guide
74
November 2015
Order No.: H76467-1.0
Running High Performance Linpack—Intel® Omni-Path Fabric
13.0
Running High Performance Linpack
A set of common MPI benchmarks are installed as part of the installation process. One
of the more popular measures of overall performance is High Performance Linpack
(HPL). This is the application used to rate systems on the Top 500 list. The following
steps enable you to make initial runs of HPL and gather preliminary baseline
numbers.. The defaults provided should perform within 10 – 20% of optimal HPL
results for the cluster. Tuning for that additional 10 – 20% is beyond the scope of this
document.
1.
(Host) To run HPL, first select a configuration file appropriate to your cluster. It is
best to start with a small configuration to verify HPL has been properly compiled:
cd /opt/opa/src/mpi_apps
./config_hpl 2t
This command will configure a two process test run of HPL.
2.
(Host) Create the file /opt/opa/src/mpi_apps/mpi_hosts listing the host
names of all the hosts.
Note: Use mpi_hosts.sample as a starting point for creating the mpi_hosts
file.
3. (Host) Run HPL:
./run_hpl 2
This initial run is a very small problem size to determine if the run will be
successful. Performance of this run is expected to be low.
If the initial run was successful, you are ready to move onto full scale HPL runs.
Assorted sample HPL.dat files are provided in the /opt/opa/src/mpi_apps/hplconfig folder. These files are a good starting point for most clusters and should get
within 10 – 20% of the optimal performance for the cluster. The problem sizes used
assume a cluster with 1GB of physical memory per processor (e.g., for a 2 processor
node, 2GB of node memory is assumed). For each cluster size, 4 files are provided:
•
t – A very small test run (5000 problem size)
•
s – A small problem size on the low end of optimal problem sizes
•
m – A medium problem size
•
l – A large problem size
The sample files can be selected using config_hpl. The following command displays
the pre-configured problem sizes available:
./config_hpl
For example, to do a small run for a 256 processor cluster (i.e., 128 nodes of dual
CPU systems):
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
75
Intel® Omni-Path Fabric—Running High Performance Linpack
1.
Type ./config_hpl 256s and press Enter.
2. Type ./run_hpl 256 and press Enter.
During these runs you should use the top command on a node to monitor
memory and CPU usage. The xhpl should use 98 – 99% of the CPU. If any other
processes are taking more than 1 – 2%, review the host configuration and stop
these extra processes if possible. HPL is very sensitive to swapping. If a lot of
swapping is seen, and xhpl is dropping below 97% for long durations, this may
indicate a problem size that is too large for the memory and OS configuration.
You can continue to tune HPL to refine performance. Parameters in HPL.dat can
all affect HPL performance. In addition, the selection of compiler and BLAS Math
library may also significantly affect performance. The new HPL.dat files may be
placed in /opt/opa/src/mpi_apps/hpl-config. Use config_hpl to select
them and copy them to all nodes in the run. Alternately, scpall may be used to
copy the file to all nodes. Refer to Intel® Omni-Path Fabric Suite FastFabric
Command Line Interface Reference Guide for more information on scpall.
Intel® Omni-Path Fabric Software
Installation Guide
76
November 2015
Order No.: H76467-1.0
Install Intel® Omni-Path Fabric Suite Fabric Manager GUI—Intel® Omni-Path Fabric
14.0
Install Intel® Omni-Path Fabric Suite Fabric
Manager GUI
This section provides the procedures to install the Fabric Manager GUI on both
Windows* and Linux* platforms. Refer to the following sections:
•
Windows* Installation
•
Linux* Installation on page 81
14.1
Windows* Installation
14.1.1
System Requirements for a Windows* Environment
The minimum system requirements are as follows:
•
14.1.2
Windows* operating system
—
Windows Vista* Service Pack 2 or higher
—
Windows* 7 Service Pack 1 or higher
—
Windows* 8.x
—
Windows Server* 2008 Service Pack 2 or higher
—
Windows Server* 2012
•
x86 or x64 processor architecture
•
Oracle* Java Runtime Environment (JRE) 1.7 or higher
•
Ethernet card/local network access
•
2GB or greater of RAM
•
1280x800 resolution (65K color depth)
Download the Intel® Omni-Path Fabric Suite Fabric Manager
GUI
1.
Go to http://www.intel.com.
2. Click the Download Center link.
3. In the search downloads box enter Fabric Manager GUI.
OR
In the Find By Category section:
November 2015
Order No.: H76467-1.0
a.
Select Network Connectivity.
b.
Select Intel® Omni-Path Architecture.
c.
Select Intel® Omni-Path Fabric Suite Fabric Manager.
Intel® Omni-Path Fabric Software
Installation Guide
77
Intel® Omni-Path Fabric—Install Intel® Omni-Path Fabric Suite Fabric Manager GUI
14.1.3
Use the Installation Wizard to Install the Fabric Manager GUI
1.
Double-click the IntelOPA-FMGUI-windows-x_x_x_x_x.exe file on the
desktop,
where x_x_x_x_x is the version number of the Fabric Manager GUI application
being installed.
2. At the User Account Control prompt: Do you want to allow the following
program to make changes to this computer? Click Yes.
3.
Select the language to be used for the installation screens and then click OK.
•
If this is a first-time installation, the Intel® Omni-Path Fabric Manager GUI
Setup window opens and you see the following:
Intel® Omni-Path Fabric Software
Installation Guide
78
November 2015
Order No.: H76467-1.0
Install Intel® Omni-Path Fabric Suite Fabric Manager GUI—Intel® Omni-Path Fabric
•
If the Fabric Manager GUI is already installed on your system, the installer
prompts you with the following dialog:
Select Yes to continue with the installation; the Fabric Manager GUI
uninstaller will be invoked and then the installation procedure continues with
the following screen:
Select Yes to uninstall the current Fabric Manager GUI. See Windows*
Procedure on page 82 for instructions. After the uninstall is finished, the
following screen appears:
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
79
Intel® Omni-Path Fabric—Install Intel® Omni-Path Fabric Suite Fabric Manager GUI
4.
Click Next .
The License Agreement window appears.
Intel® Omni-Path Fabric Software
Installation Guide
80
November 2015
Order No.: H76467-1.0
Install Intel® Omni-Path Fabric Suite Fabric Manager GUI—Intel® Omni-Path Fabric
5.
Accept the License Agreement by checking the I accept the terms of the
License Agreement checkbox and then click Next .
The Choose Install Location window appears.
Note: Intel recommends that you use the default file location.
6. Click Install.
The Installing window appears briefly.
When the installation is complete, the Completing Fabric Manager GUI Setup
window appears.
7. Click Finish.
The Fabric Manager GUI is installed.
14.2
Linux* Installation
14.2.1
System Requirements for a Linux* Environment
The minimum system requirements for the Linux* installation are as follows:
•
14.2.2
Linux* operating system
—
Red Hat* Enterprise Linux* 5.5 or higher
—
SUSE* Linux* Enterprise Server 10 or higher
•
x86 or x64 processor architecture
•
Oracle* Java* Runtime Environment (JRE) 1.7 or higher
•
X Window System
•
Ethernet card/local network access
•
2GB or greater of RAM
•
1280x800 resolution (65K color depth)
Download the Intel® Omni-Path Fabric Suite Fabric Manager
GUI
1.
Go to http://www.intel.com.
2.
Click the Download Center link.
3.
In the search downloads box enter Fabric Manager GUI.
OR
In the Find By Category section:
14.2.3
a.
Select Network Connectivity.
b.
Select Intel® Omni-Path Architecture.
c.
Select Intel® Omni-Path Fabric Suite Fabric Manager.
Use RPM to Install the Fabric Manager GUI
1.
November 2015
Order No.: H76467-1.0
Log in to the server where the Fabric Manager GUI will be installed, using a user
with root access.
Intel® Omni-Path Fabric Software
Installation Guide
81
Intel® Omni-Path Fabric—Install Intel® Omni-Path Fabric Suite Fabric Manager GUI
2.
Open a Terminal window in X Windows.
3.
Change directories to the directory where you downloaded the .rpm file.
4.
Enter the following command:
rpm -iv IntelOPA-FMGUI-linux-x_x_x_x_x.noarch.rpm
where x_x_x_x_x is the version number of the Fabric Manager GUI application
being installed.
5.
Wait for the successful completion of the command.
The Fabric Manager GUI is installed.
14.3
Start the Fabric Manager GUI Application
14.3.1
Windows* Procedure
The following procedure provides the steps to start the Intel® Omni-Path Fabric Suite
Fabric Manager GUI application.
1. From the Windows* Start menu, select All Programs
2.
Select Intel>Omni-Path>Fabric Manager GUI
The Fabric Manager GUI application starts.
14.3.2
Linux* Procedure
The following procedure starts the Intel® Omni-Path Fabric Suite Fabric Manager GUI
application from a terminal window on X Window.
1.
Open a Terminal window in X Windows.
2.
Type the following command:
fmgui
The Fabric Manager GUI application starts.
Alternatively, the following steps start the Fabric Manager GUI application from the K
Desktop Environment (KDE). .
1.
From the K Menu, select Applications.
2.
Select the Fabric folder.
3. Select Fabric Manager GUI.
The Fabric Manager GUI application starts.
14.4
Configure Startup Options
Note:
Intel recommends that you accept the default settings.
Refer to the Intel® Omni-Path Fabric Suite Fabric Manager GUI Online Help for
procedures to set user preferences.
Intel® Omni-Path Fabric Software
Installation Guide
82
November 2015
Order No.: H76467-1.0
Install Intel® Omni-Path Fabric Suite Fabric Manager GUI—Intel® Omni-Path Fabric
14.5
Uninstall the Intel® Omni-Path Fabric Suite Fabric
Manager GUI
The Fabric Manager GUI must be closed for the uninstall to be successful. The
uninstall program will not warn you if the application is open. Warnings may be
received at the end of the uninstall process stating that certain files have not been
removed or it might just fail.
14.5.1
Windows* Procedure
1.
From the Windows* Start menu, select Control Panel.
2. Under the Programs category, select Uninstall a program.
3. In the program list, select Intel Fabric Manager GUI.
4. Click the Uninstall button at the top of the list.
5. Follow the instructions on the uninstall window.
The Fabric Manager GUI is uninstalled.
14.5.2
Linux* Procedure
1.
Log in to the server where Fabric Manager GUI is installed using a user with root
access.
2.
Open a Terminal window in X Windows.
3.
Type the following command:
# rpm -e IntelOPA-FMGUI-linux-x_x_x_x_x.noarch.rpm
where x_x_x_x_x is the version number of the Fabric Manager GUI application
being uninstalled.
4. Wait for the successful completion of the command.
The Fabric Manager GUI is uninstalled.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
83
Intel® Omni-Path Fabric—Installation Verification and Additional Settings
15.0
Installation Verification and Additional Settings
This section provides instructions for verifying that the software has been properly
installed, the Intel® Omni-Path Fabric drivers are loaded, and that the fabric is active
and ready to use. Information on the Intel HFIs and Performance tuning is also
provided.
15.1
LED Link and Data Indicators
For information on the LEDs function as link and data indicators refer to Intel® OmniPath Fabric Switches Hardware Installation Guide and/or Intel® Omni-Path Host Fabric
Interface Installation Guide.
15.2
Adapter and Other Settings
The following settings can be adjusted for better performance.
Note:
15.3
This section is only applicable to clusters using Intel HFIs.
•
On the switch, use an OPA MTU of 8k bytes, if available, with the Intel®
Omni-Path Host Fabric Interface. 8K MTU is enabled in the Intel® Omni-Path
driver by default. To change this setting for the driver, refer to the Intel® OmniPath Fabric Host Software User Guide.
•
Use a PCIe Max Read Request size of at least 512 bytes with the Intel®
Omni-Path Host Fabric Interface. The Intel® Omni-Path Host Fabric Interface
can support sizes from 128 bytes to 4096 bytes in powers of two. This value is
typically set in the BIOS.
•
Use a PCIe MaxPayload size of 256, where available, with the Intel®
Omni-Path Host Fabric Interface. The Intel® Omni-Path Host Fabric Interface
can support 128, 256, or 512 bytes. This value is typically set by the BIOS as the
minimum value supported both by the PCIe card and the PCIe root complex.
•
Make sure that write combining is enabled. The x86 Page Attribute Table
(PAT) mechanism that allocates Write Combining (WC) mappings for the PIO
buffers has been added and is now the default. If PAT is unavailable, or PAT
initialization fails for some reason, the code will generate a message in the log and
fall back to the MTRR mechanism. Refer to the Intel® Omni-Path Fabric Host
Software User Guide.
ARP Neighbor Table Setup for Large Clusters
The ARP neighbor table may overflow and produce a neighbor table overflow message
to /var/log/messages along with other effects such as ping failing. To resolve this
issue, increase the threshold level for the network devices as follows:
Intel® Omni-Path Fabric Software
Installation Guide
84
November 2015
Order No.: H76467-1.0
Installation Verification and Additional Settings—Intel® Omni-Path Fabric
1.
Check the present threshold level 1 by entering the command:
cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
It will return a value, such as 128, 256, or 512.
2. Increase the value to the next level. For example, if the value is 128, then set the
thresh1 value to 256, , set thresh2 to 512, and set thresh3 to 1024. The following
is an example of the syntax to be used if the value is 128:
echo 256 > /proc/sys/net/ipv4/neigh/default/gc_thresh1
echo 512 > /proc/sys/net/ipv4/neigh/default/gc_thresh2
echo 1024 > /proc/sys/net/ipv4/neigh/default/gc_thresh3
This will stop the error messages that were received in the log file.
15.4
SM Loop Test
The SM Looptest is a diagnostic test facility in the Fabric Manager. As part of this test,
the Subnet Manager stress tests inter-switch links (ISLs) by continuously passing
traffic through them. Other tools, like FastFabric, can be used to monitor the links for
signal integrity issues or other errors. The advantage of the Looptest is that it
provides a guaranteed way to test all of the ISLs in the fabric, without the need for a
large number of end hosts or applications. For information on the SM Loop Test and
how to use the test, refer to the Intel® Omni-Path Fabric Suite Fabric Manager User
Guide.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
85
Intel® Omni-Path Fabric—Configuration
16.0
Configuration
Using advanced fabric features properly requires consistent configuration of multiple
components in the fabric, as well as proper execution of jobs and applications.
This chapter summarizes the interdependencies of some of the advanced features. It
can be used as a reminder and checklist to allow you to take full advantage of the
features.
The configuration subjects discussed in this section are:
16.1
•
Virtual Fabrics on page 86
•
Congestion Analysis on page 89
•
Adaptive Routing on page 90
•
Adaptive Routing, Switch Configuration on page 90
•
Dispersive Routing on page 90
•
Distributed SA Provider on page 91
Virtual Fabrics
Virtual Fabrics (vFabrics) includes both security and quality of service (QOS)
capabilities. vFabrics are configured within the Fabric Manager, either by directly
editing the opafm.xml configuration file or through the Fabric Manager GUI. The
configuration of the Fabric Manager must be consistent with the capabilities of the
hardware and the usage of the fabric by the applications.
16.1.1
Virtual Fabrics, Switch Configuration
On the switches, the number of virtual lanes (VLs) and the maximum MTU must be
configured. The recommended way to accomplish this is through FastFabric, during
initial installation or reconfiguration of the fabric. See Install the Intel® Omni-Path
Fabric Software on page 20. If required, the VL and MTU configuration can also be
directly configured through the switch CLI. See the ismChassisSetMtu CLI
command in the Intel® Omni-Path Fabric Switches Command Line Interface Reference
Guide.
For Fat Tree topologies, 1 VL is used per QOS level. 1 QOS level is used per vFabric
with QOS enabled (plus 1 more VL if there are additional vFabrics without QOS
enabled).
16.1.2
Virtual Fabrics, Fabric Manager Configuration
The Fabric Manager sample configuration has a variety of sample Virtual Fabrics
(vFabrics) that can be easily enabled. See the Virtual Fabrics section of the Intel®
Omni-Path Fabric Suite Fabric Manager User Guide. The following table displays some
useful combinations.
Intel® Omni-Path Fabric Software
Installation Guide
86
November 2015
Order No.: H76467-1.0
Configuration—Intel® Omni-Path Fabric
Table 1.
Sample Fabric Manager vFabric Combinations
Number of
vFabrics
Typical Combinations
Result
1
Default
All traffic together.
2
Compute, AllOthersWithSA
Separate MPI from everything else.
2
Admin, AllOthers
Separate FM/FastFabric from everything else.
2
PSM_Compute_Control,
AllOthersWithSA
Separate PSM control messages from everything
else.
3
Networking, Compute,
AllOthersWithSA
Separate Networking (and IPoIB based file systems),
MPI, and everything else.
3
Networking, Compute, Admin
Separate Networking, MPI, and Admin. Assumes no
other applications use fabric.
3
PSM_Compute_Control,
PSM_Compute_Data,
AllOthersWithSA
Separate PSM control messages from data and
everything else.
4
Networking, Compute, AllOthers,
Admin
Separate Networking, MPI, Fabric Manager/
FastFabric, and everything else.
4
Networking, PSM_Compute_Control,
AllOthers, Admin
Separate Networking, PSM Control, Fabric Manager/
FastFabric, and everything else.
4
Networking, PSM_Compute_Control,
PSM_Compute_Data,
AllOthersWithSA
Separate PSM control and data, networking, and
everything else.
4
Networking, PSM_Compute_Control,
PSM_Compute_Data, Admin
Separate PSM control and data, networking, and FM/
FastFabric. Assumes no other applications use the
fabric.
Note:
The vFabrics Default, Admin, and AllOthersWithSA are mutually exclusive; only one
can be enabled at a time.
Note:
Intel does not recommended using Default when configuring more than 1 Virtual
Fabric. Either AllOthersWithSA or Admin is recommended.
16.1.3
Virtual Fabrics, Fabric Host Software Configuration
When using Virtual Fabrics in conjunction with Intel HFIs and Performance Scaled
Messaging (PSM) with PathRecord query enabled (MPI over PSM Configuration Using
PathRecord Query on page 92), Intel recommends that you enable the Distributed
SA Provider (DSAP) for autostart on all the compute nodes. This will simplify the
operation of MPI jobs using PSM.
Enabling Distributed SA Provider
Enable the Distributed SA Provider (DSAP) for autostart on all the compute nodes. You
can use any of the following methods:
•
Select to Enable OFED IBACM (ibacm) in the Intel OPA Autostart Menu when
installing, upgrading, or running the opaconfig command.
•
Add -E dist_sa to the FF_INSTALL_OPTIONS and FF_UPGRADE_OPTIONS in
opafastfabric.conf, and using FastFabric to install or upgrade all the compute
nodes.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
87
Intel® Omni-Path Fabric—Configuration
•
Use the opacmdall command or any other distributed shell to perform a
opaconfig -E dist_sa operation on all nodes.
Refer to Intel® Omni-Path Fabric Host Software User Guide for more information about
the Distributed SA Provider.
16.1.4
Virtual Fabrics, Application, and ULP Configuration
Virtual Fabrics operates using IBTA compliant mechanisms. One of the key
requirements of Virtual Fabrics is that applications make SA PathRecord queries to
obtain critical address information, such as PKey and SL.
When using Virtual Fabrics, applications must be properly assigned to the right
vFabric. For applications that use PathRecord queries or use IB Multicast Groups, the
Fabric Manager will use the ServiceID or MGID respectively to look up the application
and select an appropriate vFabric to use in response to the query.
MPI over PSM Configuration
Unlike Verbs MPIs, PSM provides a simple and consolidated way to assign jobs to
Virtual Fabrics. For MPI jobs using PSM, this is most easily accomplished by using Path
Record queries and the Distributed SA Provider.
Using PathRecord Query
Refer to the Intel® Omni-Path Fabric Host Software User Guide for how to specify
PSM_PATH_REC=opp and how to configure that variable globally on the compute
nodes. If multiple vFabrics are configured for PSM, with different sets of ServiceIDs in
each, MPI jobs can be assigned to a specific ServiceID through PSM_IB_SERVICE_ID.
For more information on this parameter, refer to the Intel® Omni-Path Fabric Host
Software User Guide.
When using FastFabric to perform the opahostadmin mpiperf or opahostadmin
mpiperdeviation verification steps, the PSM_PATH_REC option must be specified in
opafastfabric.conf, in the FF_MPI_ENV setting. See Intel® Omni-Path Fabric
Suite FastFabric User Guide Configuration Files section for the oapfastfabric.conf
file. Alternatively, it can be specified in the /opt/opa/src/mpi_apps/
ofed.*.param files
When using /opt/opa/src/mpi_apps/run_* scripts to run sample MPI applications
and benchmarks, the /opt/opa/src/mpi_apps/ofed.*.param files must be
edited to uncomment the PSM_PATH_REC setting.
Directly Specifying vFabric or PKey and SL
When not using PathRecord query, vFabric addressing information must be directly
supplied to the MPI job using mpirun command line options or environment variables.
For more information on mpirun and environmental variables, refer to Intel® OmniPath Fabric Host Software User Guide.
When using FastFabric or the /opt/opa/src/mpi_apps/run_* scripts to run
sample MPI applications and benchmarks, the /opt/opa/src/mpi_apps/
ofed.*.param files must be edited.
Intel® Omni-Path Fabric Software
Installation Guide
88
November 2015
Order No.: H76467-1.0
Configuration—Intel® Omni-Path Fabric
MPI over Verbs Configuration
For historical reasons, most MPI implementations over Verbs do not interact with the
SA, and instead hand construct address information using hardcoded PKey, SL, and
other information. When using vFabrics with such MPIs, the vFabric addressing
information must be directly supplied to the MPI job using mpirun command line
options or environment variables. For more information on mpirun and environmental
variables, refer to Intel® Omni-Path Fabric Host Software User Guide.
When using FastFabric or the /opt/opa/src/mpi_apps/run_* scripts to run
sample MPI applications and benchmarks, the /opt/opa/src/mpi_apps/
ofed.*.param files must be edited.
IPoIB Configuration
IPoIB will automatically use the first enabled vFabric in the FM configuration file. This
will typically be Networking, Default, AllOthers, or AllOthersWithSA. The sample FM
configuration file is set up for typical IPoIB operation, with the most common option is
listed first.
If you need to create more than one IPoIB vFabric, then you need to configure
additional pKeys into IPoIB. Refer to Intel® Omni-Path Fabric Host Software User
Guide for more information about using SL and PKeys.
Other Applications and ULPs
Most applications will not require any special operational changes. This is because,
with the exception of MPI, all other applications tend to use IBTA standard approaches
for connection establishment. The Fabric Manager sample configuration file has
application definitions for many popular applications and ULPs. Additional applications
can be added as needed, or the Default, AllOthers, or AllOthersWithSA vFabric
definitions can be used.
Note:
Some applications make use of IPoIB for address resolutions. These applications may
end up using the same vFabric as IPoIB.
16.1.5
Virtual Fabrics, Fabric Manager GUI Configuration
The Fabric Manager GUI may be used to configure the Virtual Fabrics for the Fabric
Manager. Refer to the Intel® Omni-Path Fabric Suite Fabric Manager GUI Online Help
for more information.
16.2
Congestion Analysis
Intel FastFabric provides Congestion Analysis capabilities for statistics gathering.
These tools centralize the statistics monitoring in the PM. For proper operation of
these tools, no other applications can be directly clearing the PMA counters in the
fabric.
16.2.1
Congestion Analysis, Switch Configuration
Intel recommends using FastFabric to configure the switches in the fabric during the
initial installation or reconfiguration of the fabric. Refer to Configure Intel Chassis or
Configure Externally Managed Switches for configuration procedures.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
89
Intel® Omni-Path Fabric—Configuration
16.2.2
Congestion Analysis, Fabric Manager Configuration
Enable PM monitoring by configuring the Pm.SweepInterval parameter. Refer to
Configure Chassis Fabric Manager (FM) on page 37 and the Intel® Omni-Path Fabric
Suite Fabric Manager User Guide for more information.
16.2.3
Congestion Analysis, Management Node Configuration
The Management Node configuration is accomplished by installing the IFS software
and the FastFabric Toolkit. The opatop, opapaquery, and opareport tools will all
detect and access the PM within the Fabric Manager. These tools, and related tools
such as opaallanalysis, opafabricanalysis, opaextractperf, are all safe to
use.
Avoid using tools that directly clear the counters, such as opareport --pmadirect.
Refer to the Intel® Omni-Path Fabric Suite FastFabric Command Line Interface
Reference Guide for more details.
16.3
Adaptive Routing
Adaptive Routing allows the Intel switches to dynamically adjust the fabric routes in
real time. The adjustments occur in response to observed traffic patterns and fabric
congestion. This permits more efficient fabric operation.
16.4
Adaptive Routing, Switch Configuration
Make sure the version of switch firmware being used supports Adaptive Routing. Refer
to the appropriate release notes for more information:
•
Intel® Omni-Path Fabric Managed Switches Release Notes
•
Intel® Omni-Path Fabric Externally-Managed Switches Release Notes
Adaptive Routing, Fabric Manager Configuration
Adaptive routing is configured completely in the Fabric Manager. Refer to the Intel®
Omni-Path Fabric Suite Fabric Manager User Guide for information on configuring
adaptive routing.
Note:
Adaptive Routing dynamic adjustments can cause packets from a particular source to
arrive at the destination out of order, which may affect applications that utilize
ordered traffic.
16.5
Dispersive Routing
Dispersive Routing allows MPIs using Intel PSM technology to spread their traffic
across multiple routes. This allows more of the fabric routes to be used by each HFI,
statistically improves the efficiency of the fabric, and reduces potential congestion and
bottlenecks.
Intel® Omni-Path Fabric Software
Installation Guide
90
November 2015
Order No.: H76467-1.0
Configuration—Intel® Omni-Path Fabric
16.5.1
Dispersive Routing, Fabric Manager Configuration
Use of Dispersive Routing requires the Fabric Manager to configure more than one
route to each HFI. This is accomplished by enabling the LMC capabilities of the Intel®
Omni-Path architecture. To do this, set the Lmc configuration to one or more. This will
cause up to 2LMC alternate routes to be available per HFI port.
Within the Fabric Manager, the PathSelection option can affect how many paths are
reported. Intel recommends that you use the default value of Minimal. For more
information about LMC and Fabric Resiliency, refer to Intel® Omni-Path Fabric Suite
Fabric Manager User Guide.
16.5.2
Dispersive Routing, PSM Configuration
By default, PSM will detect and use the multiple LIDs for dispersive routing. Additional
controls over the routes used are possible through:
16.6
•
Using Distributed SA Provider and PathRecord queries in PSM - Refer to
Distributed SA Provider on page 91.
•
Configuring PSM_PATH_SELECTION - Refer to Intel® Omni-Path Fabric Host
Software User Guide.
Distributed SA Provider
The Distributed SA Provider (DSAP) allows for a highly scalable way for applications to
query the Fabric Manager’s Subnet Administrator. This can help to improve flexibility
of the fabric operation and configuration, especially when using Virtual Fabrics.
The implementation involves dist_sa processes running on each compute node. The
processes synchronize key PathRecord information with the Fabric Manager so it is
always up to date and immediately available for rapid application startup.
16.6.1
Distributed SA Provider, Fabric Manager Configuration
Refer to the Intel® Omni-Path Fabric Suite Fabric Manager User Guide for more
information on configuring Distributed SA Provider in Fabric Manager.
16.6.2
Distributed SA, Fabric Host Software Configuration
Enable the Distributed SA Provider (DSAP) for autostart on all the compute nodes.
Refer to Enabling Distributed SA Provider on page 87 for information on enabling
Distributed SA Provider.
For more information on Distributed SA Provider, refer to the Intel® Omni-Path Fabric
Host Software User Guide.
16.6.3
Distributed SA Provider, Application and ULP Configuration
Distributed SA Provider is integrated with Intel PSM for use by MPI jobs. The
default /etc/sysconfig/opa/dist_sa.conf file matches the default opafm.xml
configuration with regard to IB ServiceIDs used for PSM.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
91
Intel® Omni-Path Fabric—Configuration
MPI over PSM Configuration Using PathRecord Query
Refer to the Intel® Omni-Path Fabric Host Software User Guide for information how to
specify PSM_PATH_REC=opp, and how to configure that variable globally on the
compute nodes.
When using FastFabric to perform the opahostadmin, mpiperf, or opahostadmin
mpiperdeviation verification steps, the PSM_PATH_REC option must be specified in
the opafastfabric.conf file in the FF_MPI_ENV setting. Refer to Intel® Omni-Path
Fabric Suite FastFabric User Guide Configuration Files section for information on
opafastfabric.conf . Alternatively it can be specified in the /opt/opa/src/
mpi_apps/ofed.*.param files.
When using /opt/opa/src/mpi_apps/run_* scripts to run sample MPI applications
and benchmarks, the /opt/opa/src/mpi_apps/ofed.*.param files must be
edited to uncomment the PSM_PATH_REC setting.
Other Applications and ULPs
At this time, no other applications will take advantage of the Distributed SA Provider.
Other applications will continue to operate as they have in the past. Those which are
IBTA compliant will interact directly with the centralized SA.
Intel® Omni-Path Fabric Software
Installation Guide
92
November 2015
Order No.: H76467-1.0
Upgrade the Management Node—Intel® Omni-Path Fabric
17.0
Upgrade the Management Node
This procedure provides discussion and step-by-step directions to upgrade an Fabric
Management Node from a previous Intel® Omni-Path Fabric Suite (IFS) software
version to the latest IFS software version.
17.1
Preinstallation
Prior to upgrading to IFS software, ensure the following items have been performed:
•
Review the Release Notes for a list of compatible software.
•
Uninstall all versions of 3rd party IB stacks.
•
Back up the following configuration files in case the upgrade fails:
—
/etc/sysconfig/opafm.xml
—
/etc/sysconfig/opa/*
—
/var/opt/opa/analysis/baseline/*
—
Refer To the OS documentation for a list of any other OS-specific files that
should be included in any backups.
Note:
When managing a cluster where the IPoIB settings on the compute nodes are
incompatible with the Fabric Management Node (for example when a 4K MTU is used
on the compute nodes and a 2K MTU is used on the management nodes), Intel
recommends that you not run IPoIB on the Fabric Management Nodes.
17.2
Intel® Omni-Path Fabric Suite Upgrade
Upgrading the IFS software on a Fabric Management Node with an existing IFS
software is accomplished using the IntelOPA-IFS.DISTRO.VERSION.tgz package
file.
1. Download and extract the installation packages following the Download and
Extract Installation Packages section.
2. Change directory to IntelOPA-IFS.DISTRO.VERSION directory.
cd IntelOPA-IFS.DISTRO.VERSION
3.
Start the TUI installation.
./INSTALL
The Intel OPA VERSION Software main menu is displayed.
Intel OPA VERSION Software
1)
2)
3)
4)
November 2015
Order No.: H76467-1.0
Install/Uninstall Software
Reconfigure OFED IP over IB
Reconfigure Driver Autostart
Generate Supporting Information for Problem Report
Intel® Omni-Path Fabric Software
Installation Guide
93
Intel® Omni-Path Fabric—Upgrade the Management Node
6) FastFabric (Host/Chassis/Switch Setup/Admin)
X) Exit
4. Select 1) Install/Uninstall Software.
Screen 1 of 2 of the Intel OPA Install Menu is displayed.
Intel OPA Install (VERSION release) Menu
Please Select Install Action (screen
0) OFED OPA Stack
[
Upgrade
1) OFED IBACM
[
Upgrade
2) Intel HFI Components[
Upgrade
3) OPA Tools
[
Upgrade
4) OFED OPA Development[
Upgrade
5) FastFabric
[
Upgrade
6) OFED IP over IB
[
Upgrade
7) OPA FM
[
Upgrade
8) MVAPICH2 (hfi,gcc) [
Upgrade
9) MVAPICH2 (hfi,Intel)[
Upgrade
a) MVAPICH2 (hfi,PGI) [
Upgrade
b) OpenMPI (hfi,gcc)
[
Upgrade
c) OpenMPI (hfi,Intel) [
Upgrade
d) OpenMPI (hfi,PGI)
[
Upgrade
N)
P)
R)
X)
Next Screen
Perform the selected actions
Re-Install All
Return to Previous Menu (or ESC)
1 of 2):
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
I) Install All
U) Uninstall All
5. Review the items to be upgraded; the default value is in brackets (Upgrade or up
To Date). To change a value, type the alphanumeric character associated with the
item.
6.
Press N to go to the next screen.
Screen 2 of 2 of the Intel OPA Install Menu is displayed.
Intel OPA Install (VERSION release) Menu
Please Select Install Action (screen 2 of 2):
0) GASNet (hfi,gcc)
[
Upgrade
][Available]
1) OpenSHMEM (hfi,gcc) [
Upgrade
][Available]
2) MVAPICH2(verbs,gcc) [
Upgrade
][Available]
3) OpenMPI (verbs,gcc) [
Upgrade
][Available]
4) MPI Source
[
Upgrade
][Available]
5) OFED Debug Info
[Don't Install][Available]
N)
P)
R)
X)
7.
Next Screen
Perform the selected actions
Re-Install All
Return to Previous Menu (or ESC)
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
I) Install All
U) Uninstall All
Review the items to be upgraded; the default value is in brackets (Upgrade, Up To
Date, or Don't Install).
To change a value, type the alphanumeric character associated with the item.
8.
Press P to perform the selected actions from the two screens.
9. For each system prompt, select the default by pressing Enter.
Intel® Omni-Path Fabric Software
Installation Guide
94
November 2015
Order No.: H76467-1.0
Upgrade the Management Node—Intel® Omni-Path Fabric
The Intel OPA Autostart Menu is displayed.
Intel OPA Autostart (VERSION release) Menu
Please Select Autostart Option:
0) OFED OPA Stack ()
1) OFED IBACM (ibacm)
2) Intel HFI Components ()
3) OFED IP over IB ()
4) OPA FM (opafm)
P) Perform the autostart changes
S) Autostart All
X) Return to Previous Menu (or ESC)
[Enable ]
[Disable]
[Enable ]
[Enable ]
[Enable ]
R) Autostart None
10. Review the items for Autostart; the default value is in brackets (Enable or
Disable). To change a value, type the alphanumeric character associated with the
item.
Intel recommends leaving all of the Autostart selections as default.
11. Press P to perform the selected actions from the screen.
The system prompts:
Hit any key to continue...
12. Press any key to proceed with the installation.
The system prompts:
A System Reboot is recommended to activate the software changes
Hit any key to continue...
13. Press any key to complete the installation.
The installation completes and returns to the main menu.
14. Press X to exit.
15. Reboot the server.
reboot
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
95
Intel® Omni-Path Fabric—Upgrade the Fabric Software
18.0
Upgrade the Fabric Software
Upgrade Intel® Omni-Path Fabric Software
If an existing fabric needs to have the Intel® Omni-Path Fabric software upgraded,
follow this procedure:
Note:
On each Fabric Management Node, perform an upgrade installation of the IFS software
using the procedure documented in the Upgrade the Management Node on page 93
section.
1. (All) Perform steps 1 and 2 in the Install Host Software on the Remaining Hosts
Using the FastFabric TUI on page 54.
2.
Select the following items to be performed in the menu:
•
(optional) The 1) Edit Config and Select/Edit Hosts Files option
will permit the opafastfabric.conf and hosts files to be edited.
•
The 5) Install/Upgrade OPA Software option will upgrade the Intel®
Omni-Path Fabric software on all the selected hosts. By default, it will look in
the current directory for the FF_PRODUCT.$FF_PRODUCT_VERSION.tgz file.
If it is not found in the current directory, you will be prompted to supply a
directory name where this file can be found.
Note: The upgrade installation will update any existing OFED Delta software,
and is only valid for hosts that already have a previous version of OFED
Delta software installed.
3.
Press P to perform the upgrade.
Then
If
1) Edit Config and Select/Edit Hosts Files was selected
Continue with the next step.
1) Edit Config and Select/Edit Hosts Files was not selected
Skip to Step 6.
4.
Press any key for the following system prompt:
About to: vi /etc/sysconfig/opa/opafastfabric.conf
nHit any key to continue (or ESC to abort)...
The opafastfabric.conf file is opened in the editor, review all the settings.
See Intel® Omni-Path Fabric Suite FastFabric User Guide Configuration Files
section for more information about the opafastfabric.conf file.
5. When finished updating the opafastfabric.conf file, quit out of the editor.
The system prompts:
You should select a file which OMITS this host
Select Host File to Use/Edit [/etc/sysconfig/opa/hosts]:
Intel® Omni-Path Fabric Software
Installation Guide
96
November 2015
Order No.: H76467-1.0
Upgrade the Fabric Software—Intel® Omni-Path Fabric
6.
Select a hosts list file that lists all the hosts except the Fabric Management nodes.
If necessary, create a new file at this time, potentially based on the
existing /etc/sysconfig/opa/hosts file.
Note: Do not list any of the Fabric Management Nodes in the host file (for
example, the nodes that have FastFabric installed).
Note: The file may list the Management Network or IPoIB hostnames for the
selected hosts
The host file is opened in the editor, review all the information. See Intel® OmniPath Fabric Suite FastFabric User Guide Configuration Files section for more
information about the host file.
7.
When finished updating the hosts file, quit out of the editor.
The system prompts:
Do you want to edit/review/change the files? [y]:
8. Press N to continue.
The system prompts:
Performing Host Setup: Install/Upgrade OPA Software
Hit any key to continue...
9.
Press any key to continue.
The system displays prompts that require your response as it is going through the
installation
10. Select the default by pressing Enter.
The Intel OPA Autostart Menu appears.
Intel OPA Autostart (VERSION release) Menu
Please Select Autostart Option:
0) OFED OPA Stack ()
1) OFED IBACM (ibacm)
2) Intel HFI Components ()
3) WFR Lite driver ()
4) OFED IP over IB ()
P) Perform the autostart changes
S) Autostart All
X) Return to Previous Menu (or ESC)
[Enable ]
[Disable]
[Enable ]
[Enable ]
[Enable ]
R) Autostart None
11. Press P to proceed.
When the selected hosts have completed rebooting, the FastFabric OPA Host
Setup Menu is displayed. The upgrade is complete.
If any hosts fail to be updated, use the View opahostadmin result files
option to review the result files from the update. Refer to the Intel® Omni-Path
Fabric Suite FastFabric Command Line Interface Reference Guide for more details.
Note: When using the Intel® Omni-Path Fabric software, FastFabric may be used
to upgrade the Intel® Omni-Path Fabric software stack (IntelOPABasic.DISTRO.VERSION.tgz)on the remaining hosts. on the remaining
hosts.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
97
Intel® Omni-Path Fabric—Upgrade the Fabric Software
12. (Linux) If any other setup operations need to be performed on all hosts, use the
Run a command on all hosts menu option. This option executes a the
specified Linux* shell command (or sequence of commands separated by
semicolons) against all selected hosts.
Note: Check the relevant release notes for the new Intel® Omni-Path Fabric Host
Software release being installed for any such additional required steps.
13. (Linux) Reboot Hosts will reboot all the selected hosts and ensure they go
down and come back up properly (as verified through ping over the management
network). When the hosts come back up, they will be running the installed Intel®
Omni-Path Fabric Host Software.
14. Repeat the verification steps for the fabric as discussed in Install the Intel® OmniPath Fabric Software on page 20.
Intel® Omni-Path Fabric Software
Installation Guide
98
November 2015
Order No.: H76467-1.0
Upgrade from IntelOPA-Basic to IntelOPA-IFS—Intel® Omni-Path Fabric
19.0
Upgrade from IntelOPA-Basic to IntelOPA-IFS
If you upgrade the Intel® Omni-Path Software (Intel® OPA Software) from
IntelOPA-Basic to IntelOPA-IFS, FastFabric and OPA_FM software are
installed. This section provides step-by-step instructions for the upgrade, using the
Install TUI menus. This installation procedure presumes that you are logged in and
have completed all pre-installation requirements.
1. Download and extract the IntelOPA-IFS.DISTRO.VERSION.tgz package file as
described in Download and Extract Installation Packages.
2.
Change to the IntelOPA-IFS.DISTRO.VERSION directory with the command:
cd IntelOPA-IFS.DISTRO.VERSION
3. Start the Install TUI:
./INSTALL
The Intel OPA VERSION Software main menu appears.
Intel OPA VERSION Software
1)
2)
3)
4)
6)
Install/Uninstall Software
Reconfigure OFED IP over IB
Reconfigure Driver Autostart
Generate Supporting Information for Problem Report
FastFabric (Host/Chassis/Switch Setup/Admin)
X) Exit
4. Select 1) Install/Uninstall Software.
Screen 1 of 2 of the Intel OPA Install Menu appears.
Intel OPA Install (VERSION release) Menu
Please Select Install Action (screen
0) OFED OPA Stack
[ Up To Date
1) OFED IBACM
[ Up To Date
2) Intel HFI Components[ Up To Date
3) OPA Tools
[ Up To Date
4) OFED OPA Development[ Up To Date
5) FastFabric
[
Install
6) OFED IP over IB
[ Up To Date
7) OPA FM
[
Install
8) MVAPICH2 (hfi,gcc) [ Up To Date
9) MVAPICH2 (hfi,Intel)[ Up To Date
a) MVAPICH2 (hfi,PGI) [ Up To Date
b) OpenMPI (hfi,gcc)
[ Up To Date
c) OpenMPI (hfi,Intel) [ Up To Date
d) OpenMPI (hfi,PGI)
[ Up To Date
1 of 2):
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
][Available]
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
VERSION
N) Next Screen
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
99
Intel® Omni-Path Fabric—Upgrade from IntelOPA-Basic to IntelOPA-IFS
P) Perform the selected actions
R) Re-Install All
X) Return to Previous Menu (or ESC)
I) Install All
U) Uninstall All
5. Review the items to ensure that FastFabric and OPA_FM show Install and all
other selections show Up To Date.
6. Press P to perform the FastFabric and OPA_FM installation.
7. Follow the system prompts. For each prompt, select the default by pressing
Enter.
The Intel OPA Autostart Menu appears.
Intel OPA Autostart (VERSION release) Menu
Please Select Autostart Option:
0) OFED OPA Stack ()
1) OFED IBACM (ibacm)
2) Intel HFI Components ()
3) OFED IP over IB ()
4) OPA FM (opafm)
P) Perform the autostart changes
S) Autostart All
X) Return to Previous Menu (or ESC)
8.
[Enable ]
[Disable]
[Enable ]
[Enable ]
[Enable ]
R) Autostart None
Review the items for Autostart; the default value is in brackets (Enable or
Disable). To change a value, type the alphanumeric character associated with
the item.
Intel recommends leaving all of the Autostart selections set to the default values.
9.
Press P to perform the selected actions from the screen.
The system prompts:
Hit any key to continue...
10. Press any key to proceed with the installation.
The system prompts:
Firmware Update unnecessary
Hit any key to continue...
11. Press any key to proceed with the installation.
The system prompts:
A System Reboot is recommended to activate the software changes
Hit any key to continue...
12. Press any key to proceed with the installation.
The installation completes and returns to the main menu.
13. Press X to exit.
14. Reboot the server.
Intel® Omni-Path Fabric Software
Installation Guide
100
November 2015
Order No.: H76467-1.0
Multi-Subnet Fabrics—Intel® Omni-Path Fabric
Appendix A Multi-Subnet Fabrics
Intel® Omni-Path Fabric Suite FastFabric supports management of both single-subnet
fabric and multi-subnet fabrics.
When operating a multi-subnet fabric, a subnet manager (SM) is required for each
subnet. An SM may be run within switches within each subnet, or a host-based SM
may be run. A host-based SM can manage multiple subnets (assuming the host server
is connected to more than one subnet).
A number of combinations are possible for multi-subnet fabrics:
1. All subnets are completely independent (except for any interconnecting
routers): If a separate FastFabric node is being used per subnet, and servers are
not installed in more than one subnet, the individual subnets can be treated
completely separately. In this case, follow all the previous FastFabric instructions
for each fabric.
2.
The subnets are primarily independent: If the only components common to
more than one subnet are the FastFabric nodes (and possibly SM nodes) and no
servers are installed in more than one subnet, refer to the following instructions
for Primarily Independent Subnets.
3. The subnets are overlapping: If multiple components are common to more
than one subnet, such as FastFabric node(s), servers, etc., refer to the following
instructions for Overlapping Subnets on page 103.
A.1
Primarily Independent Subnets
If the FastFabric node (and possible SM nodes) is the only common server between
subnets, FastFabric may be used to assist in server installation and fabric operation.
Follow the installation instructions outlined in Install the Omni-Path Fabric Software
with the following adjustments:
From Design of the Fabric, design the cabling such that the FastFabric node will be
connected to each subnet it will manage. The FastFabric node must also have a
management network path to all the nodes in all the subnets that it will manage. As
part of the design, consider where routes between subnets are wanted between
routers, IPoIB routers, etc.
Design of the Fabric can be performed as per the instructions. When installing the IFS
software on the Fabric Management Node, IPoIB will need to be configured such that
each subnet is an independent IPoIB network interface, typically with different IP
subnets. Refer to the Intel® Omni-Path Fabric Host Software User Guide for more
information on configuring IPoIB.
Note:
When managing a cluster where the IPoIB settings on the compute nodes are
incompatible with the Fabric Management Node (e.g., when a 4K MTU is used on the
compute nodes and a 2K MTU is used on the Fabric Management Node), Intel
recommends that you not run IPoIB on the Fabric Management Node(s).
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
101
Intel® Omni-Path Fabric—Multi-Subnet Fabrics
Configure Intel® Omni-Path Chassis on page 30 can be performed as per the
instructions. When creating the chassis file, list all Intel internally-managed switches
in all subnets. If required, additional files may also be created per subnet that list only
the Intel chassis in each subnet. When editing the ports file, list all the Fabric
Management Node ports that access the managed fabrics. If required, additional files
may be created per subnet that list only the Fabric Management Node port connected
to the given managed fabric.
Configure Chassis Fabric Manager (FM) on page 37 can be performed as per the
instructions. At least one subnet manager is required per subnet. Refer to the Intel®
Omni-Path Fabric Suite Fabric Manager User Guide for more information on how to
configure a host SM node to manage more than one subnet.
Configure Intel® Omni-Path Chassis on page 30 can be performed as per the
instructions. When editing the ports file, list all the Fabric Management Node ports
that access the managed fabrics. If required, additional files may be created per
subnet that list only the Fabric Management Node port connected to the given
managed fabric. If required the switches file may specify a hfi:port per switch.
However, if hfi:port is not specified, all the hfi:port specified in the ports file will be
searched to locate the given OPA Switch’s Node GUID.
Install Intel Omni-Path Fabric Host Software on the Remaining Servers can be
performed as per the instructions. When creating the hosts file, list the hosts in all
subnets except the Fabric management node where FastFabric is being run. If
required, additional files may be created per subnet that list the hosts in each subnet
(except the Fabric Management Node).
Verify Intel Omni-Path Fabric Host Software on the Remaining Servers has the
following adjustments from the instructions.
•
( All): Create the allhosts file as per the instructions. Next, create additional
files per subnet that list all the hosts in each subnet including the Fabric
Management Node. When editing the ports file, list all the Fabric Management
Node ports that access the managed fabrics. If required, additional files may be
created per subnet that list only the Fabric Management Node port connected to
the given managed fabric.
•
(All): Summary of Fabric Components on page 64 can be performed as per the
instructions.
•
(All): Verify hosts pingable, sshable, and active on page 64 can be performed as
per the instructions.
•
(All): Verify OPA Fabric status and topology on page 65 can be performed as per
the instructions.
•
(Host): Verify Hosts see each other on page 65 can be run for each subnet by
using the allhosts files specific to each subnet (i.e., those listing only hosts in a
single subnet).
•
(Host): Verify Hosts ping via IPoIB on page 65 may be run per the instructions.
•
(Host): Check MPI Performance on page 65 can be run for each subnet by using
the allhosts files specific to each subnet (i.e., those listing only the hosts in a
single subnet).
Installation of Additional Fabric Management Nodes can be performed as per the
instructions. When copying FastFabric configuration files to the additional Fabric
Management Nodes, be sure to also copy the additional hosts, chassis, and allhosts
files that were created per subnet.
Intel® Omni-Path Fabric Software
Installation Guide
102
November 2015
Order No.: H76467-1.0
Multi-Subnet Fabrics—Intel® Omni-Path Fabric
Note:
In asymmetrical configurations where the Fabric Management Nodes are not all
connected to the same set of subnets, the files copied to each management node may
need to be slightly different. For example, configuration files for
opafabricanalysis may indicate different port numbers or host files used for
FastFabric, and MPI may need to list different hosts.
Configure and Initialize Health Check Tools can be performed as per the instructions.
Additionally, make sure the /etc/sysconfig/opa/ports file lists each of the Fabric
Management Node local HFIs and ports that are connected to a unique subnet. When
running opareports, opafabricinfo, opafabricanalysis, or
opaallanalysis, the default will be to use the ports file. If required, the -p and t options or the PORTS/PORTS_FILE environment variables may be used to specify
all the HFIs and ports on the Fabric Management Node such that all subnets are
checked. Similarly, the esm_chasssis and chassis files used should list all relevant
Intel chassis in all subnets.
Running High Performance Linpack can be run for each subnet by creating mpi_hosts
files specific to each subnet (i.e., only listing hosts in a single subnet).
Upgrade the Management Node on page 93 can be performed as per the instructions.
A.2
Overlapping Subnets
If multiple components are common between subnets (in addition to the Fabric
Management Nodes), FastFabric may be used to assist in server installation and fabric
operation. Follow the installation instructions outlined in Install the Intel® Omni-Path
Fabric Software on page 20 with the following adjustments:
From Design of the Fabric, design the cabling such that the FastFabric node will be
connected to each subnet it will manage. The FastFabric node must also have a
management network path to all the nodes in all the subnets it will manage. As part of
the design, consider where routes between subnets are required, between routers,
IPoIB routers, etc.
Set Up the Fabric can be performed as per the instructions. When installing the IFS
software on the Fabric Management Node, IPoIB will need to be configured such that
each subnet is an independent IPoIB network interface, typically with different IP
subnets. Refer to the Intel® Omni-Path Fabric Host Software User Guide for more
information on configuring IPoIB.
Note:
When managing a cluster where the IPoIB settings on the compute nodes are
incompatible with the Fabric Management Node (e.g., when a 4K MTU is used on the
compute nodes and a 2K MTU is used on the management nodes), Intel recommends
that you not run IPoIB on the Fabric Management Node(s).
Configure Intel® Omni-Path Chassis on page 30 can be performed as per the
instructions. When creating the chassis file, list all Intel internally-managed switches
in all subnets. If required, additional files may be created per subnet that list only the
Intel chassis in each subnet. When editing the ports file, list all the Fabric
Management Node ports that access the managed fabrics. If required, additional files
may be created per subnet that list only the Fabric Management Node port connected
to the given managed fabric.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
103
Intel® Omni-Path Fabric—Multi-Subnet Fabrics
Configure Chassis Fabric Manager (FM) on page 37 can be performed as per the
instructions. At least one subnet manager is required per subnet. Refer to the Intel®
Omni-Path Fabric Suite Fabric Manager User Guide for more information on how to
configure a host-based SM node to manage more than one subnet.
Configure Intel® Omni-Path Chassis on page 30 can be performed as per the
instructions. When editing the ports file, list all the Fabric Management Node ports
that access the managed fabrics. If required, additional files may be created per
subnet that list only the Fabric Management Node port connected to the given
managed fabric. If required, the switches file may specify a hfi:port per switch.
However, if hfi:port is not specified, all the hfi:port entries specified in the ports file
will be searched to locate the given switch’s node GUID.
Install Intel Omni-Path Fabric Host Software on the Remaining Servers can be
performed as per the instructions. When creating the hosts file, list all the hosts in all
subnets except the Fabric Management Node where FastFabric is being run. If
required, additional files may also be created per subnet that list the hosts in each
subnet (except the Fabric Management Node).
For hosts that are connected to more than one subnet, IPoIB will need to be
configured such that each subnet is an independent IPoIB network interface, typically
with different IP subnets. Refer To the Intel® Omni-Path Fabric Host Software User
Guide for more information on configuring IPoIB.
Verify Intel Omni-Path Fabric Host Software on the Remaining Servers has the
following adjustments from the instructions.
•
(All): Create the allhosts file per the instructions. Next, create additional files
per subnet that list all the hosts in each subnet including the Fabric Management
Node. When editing the ports file, list all the Fabric Management Node ports which
access the managed fabrics. If required, additional files may also be created per
subnet that list only the Fabric Management Node port connected to the given
managed fabric.
•
(All): Summary of Fabric Components on page 64 can be performed per the
instructions.
•
(All): Verify hosts pingable, sshable, and active on page 64 can be performed as
per the instructions.
•
(All): Verify OPA Fabric status and topology on page 65 can be performed as per
the instructions.
•
(Host): Verify Hosts see each other on page 65 can be run for each subnet by
using the allhosts files specific to each subnet (i.e., those only listing hosts in a
single subnet).
•
(Host): Verify Hosts ping via IPoIB on page 65 may be run per the instructions.
•
(Linux): Refresh ssh Known Hosts on page 65 may be run per the instructions.
•
(Host): Check MPI Performance on page 65 can be run for each subnet by using
the allhosts files specific to each subnet (i.e., those listing only the hosts in a
single subnet). This is currently not available for OFED.
Installation of Additional Fabric Management Nodes can be performed as per the
instructions. When copying FastFabric configuration files to the additional Fabric
Management nodes, be sure to also copy the additional hosts, chassis, and allhosts
files created per subnet.
Intel® Omni-Path Fabric Software
Installation Guide
104
November 2015
Order No.: H76467-1.0
Multi-Subnet Fabrics—Intel® Omni-Path Fabric
Note:
In asymmetrical configurations, where the Fabric Management Nodes are not all
connected to the same set of subnets, the files copied to each management node may
need to be slightly different. For example, configuration files for
opafabricanalysis indicating different port numbers or host files used for
FastFabric and MPI may need to list different hosts.
Configure and Initialize Health Check Tools can be performed per the instructions. In
addition, make sure the /etc/sysconfig/opa/ports file lists the Fabric
Management Node local HFIs and ports that are connected to a unique subnet. When
running opareport, opafabricinfo, opafabricanalysis, or opaallanalysis,
the default is to use the ports file. If required, the -p and -t options or the PORTS/
PORTS_FILE environment variable may be used to specify all the HFIs and ports on
the Fabric Management Node, such that all subnets are checked. Similarly, the
esm_chasssis and chassis files used should list all relevant Intel chassis in all
subnets.
Running High Performance Linpack can be run for each subnet by
creating ,mpi_hosts files specific to each subnet (i.e., only listing hosts in a single
subnet).
Upgrade the Management Node on page 93 can be performed per the instructions.
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
105
Intel® Omni-Path Fabric—Client/Server Configuration to Boot Over Fabric
Appendix B Client/Server Configuration to Boot Over
Fabric
This section describes a possible configuration for a PXE server, and how to use Intel®
Omni-Path Boot to download a kernel from that server.
Note:
These steps were compiled using RHEL* 7. Server setup, such as installed packages,
could change based on the server OS. For a full description of RHEL* 7 network
configuration, see https://access.redhat.com/documentation/en-US/
Red_Hat_Enterprise_Linux/7/html/Networking_Guide/part-Servers.html.
B.1
Server Configuration
Intel® Omni-Path Boot expects a standard PXE server with a few specific
requirements. Those requirements are specific to the Intel® Omni-Path Fabric Suite
Fabric Manager installation, the PXELINUX configuration file, and the kernel being
downloaded remotely.
B.1.1
Configuring IFS
Refer to Intel® Omni-Path Fabric Suite documentation for the full description of how to
install Intel® Omni-Path Fabric Suite on the server.
IFS contains the Intel® Omni-Path Host Fabric Interface driver and the required IP
over IB software stack. It also optionally installs a Fabric Manager (FM). The FM
should be installed for the Intel® Omni-Path Boot client to be fully initialized.
B.1.2
Installing a DHCP Server
Follow these steps to install a DHCP server.
1. Install the DHCP package:
yum install dhcp
2.
Add the DHCP configuration file.
Example /etc/dhcp/dhcpd.conf:
log-facility local7;
option domain-name=”localhost”;
default-lease-time 600;
max-lease-time 1800;
authoritative;
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.221 192.168.1.230;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.1.255;
option routers 192.168.1.220;
Intel® Omni-Path Fabric Software
Installation Guide
106
November 2015
Order No.: H76467-1.0
Client/Server Configuration to Boot Over Fabric—Intel® Omni-Path Fabric
}
filename “pxelinux/bootx64.efi”;
3. Start the DHCP server:
service dhcp start
B.1.3
Installing a TFTP Server
Follow these steps to install a TFTP server.
1. Install the TFTP package:
yum install tftp-server
2. Install the XINETD package:
yum install xinetd
3.
Add the TFTP configuration file.
Example /etc/xinetd.d/tftp:
service tftp
{
socket_type
protocol
wait
user
server
server_args
disable
per_source
cps
}
=
=
=
=
=
=
=
=
=
dgram
udp
yes
root
/usr/sbin/in.tftpd
-s /var/lib/tftpboot
no
11
100 2
4. Start the XINETD server:
service xinetd start
B.1.4
Installing a PXE Server
The PXE Server is contained within a directory structure indicated by the
server_args argument of the TFTP configuration file.
Follow these steps to install a PXE server.
1.
Create the base directory structure:
mkdir -p /var/lib/tftpboot
2. Add the PXELINUX configuration file:
The PXELINUX configuration file is critical to the overall Intel® Omni-Path Boot.
The client expects to have a label of "OPA" with a kernel and optional initrd. If
these are not provided, Intel® Omni-Path Boot will fail.
Example /var/lib/tftpboot/pxelinux/pxelinux.cfg/default:
menu INCLUDE pxelinux.cfg/graphics.cfg
DEFAULT vesamenu.c32
NOESCAPE 1
ALLOWOPTIONS 0
boot label in /var/lib/tftpboot/pxelinux
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
107
Intel® Omni-Path Fabric—Client/Server Configuration to Boot Over Fabric
LABEL Karmac
MENU LABEL default
MENU DEFAULT
KERNEL bootx64.efi
LABEL OPA
MENU LABEL Omni-Path Boot
MENU DEFAULT
KERNEL rhel7/vmlinuz
INITRD rhel7/initramfs.img
The rhel7/vmlinuz and rhel7/initramfs.img files are a bootable kernel and
initramfs stored, in this example, in /var/lib/tftpboot/pxelinux/rhel7/.
Note: The command line arguments passed to the PXE server kernel, in this case
rhel7/vmlinuz, should be embedded in that kernel. Intel® Omni-Path
Boot does not pass command line arguments to downloaded kernels.
Note: The downloaded kernel, in this case rhel7/vmlinuz, should be built as an
EFI_STUB kernel, as it is executed in the EFI environment.
B.2
Intel® Omni-Path Boot Client
First, the Intel® Omni-Path Boot EFI application must be flashed to the Intel® OmniPath Host Fabric Interface (HFI) PCIe add-in card. IN Intel® Xeon Phi™ F series and
the 6th generation Intel® Core™ processor F series, this file should be provided as part
of the UEFI BIOS release, and this section can be skipped.
Note:
The HFI1 UEFI package is currently being integrated in IFS. After this integration is
complete, it will be available as an RPM. Until that time, it will be released as a tarball.
Please adjust the steps below accordingly.
1.
Download and install the HFI1 UEFI RPM, which provides two files,
HfiPcieGen3.rom and fabric_boot.efi:
yum install hfi1_uefi
2. Flash the HFI Gen3 UEFI driver to the first 128k of the HFI EEPROM:
hfi1_eprom -w -p 0 HfiPcieGen3.rom
3. Flash the Intel® Omni-Path Boot EFI application to the beginning of the second
block in the HFI EEPROM:
hfi1_eprom -w -p 1 fabric_boot.efi
4.
Reboot the system.
5.
In the UEFI BIOS boot options, select UEFI Network.
Intel® Omni-Path Fabric Software
Installation Guide
108
November 2015
Order No.: H76467-1.0
IFS Software Installation Checklist—Intel® Omni-Path Fabric
Appendix C IFS Software Installation Checklist
The following sections provide a checklist to aid in tracking the steps as they are
completed for Fabric Setup, Installation and Verification. Check off each step as its
performed. Refer to Install the Omni-Path Fabric Software for a more detailed
explanation of each step.
Pre-Installation
C.1
Step
Description
Complete
Intel®
1.
Ensure that hardware is installed, cabled, and powered. Refer to the
Hardware Installation Guide.
Omni-Path Fabric Switches
2.
Ensure that an HFI is installed in each server. Refer to the Intel® Omni-Path Host Fabric Interface Installation
Guide.
3.
The hardware configuration should be reviewed to ensure everything was installed and installed properly,
according to the plan. Refer to the local hardware configuration plan.
4.
Ensure that the required Operating System is installed on each server with the following options:
• Root user command prompt ends in “# “ or “$".
Note: There must be a space after “# “ or “$".
• TCL and Expect packages installed on all Fabric Management Nodes.
Refer to the Intel® Omni-Path Fabric Software Release Notes for supported Operating Systems.
5.
Ensure the capability of remote login as root enabled.
• SSH server enabled
• All servers configured with the same root password
6.
Ensure that there is a TCP/IP Host Name Resolution
• If using /etc/hosts update the /etc/hosts file on the Fabric Management Node
•
•
If using DNS, all Management Network and IPoIB hostnames added to DNS
/etc/resolv.conf file configured on Fabric Management Node.
7.
Ensure that an NTP server is setup.
8.
Proceed to the installation of the software using the installation checklist for your type of Installation:
• Install the Fabric Software on a Cluster Checklist
C.2
Install the Intel® Omni-Path Fabric Software on a Cluster
Description
Step
1.
Complete the Pre-Installation Requirements. Refer to Pre-Installationchecklist.
2.
Download and extract the Fabric Software. Refer to Download and Extract
Installation Packages
3.
Install Intel® Omni-Path Fabric software. Refer to Install the Omni-Path Fabric
Software
4.
Configure and verify the Intel chassis. Refer to Configure Intel Chassis
Complete
continued...
November 2015
Order No.: H76467-1.0
Intel® Omni-Path Fabric Software
Installation Guide
109
Intel® Omni-Path Fabric—IFS Software Installation Checklist
Step
Description
Complete
Intel®
5.
Install and configure the Fabric Manager. Refer to Install
Suite Fabric Manager GUI on page 77
6.
Configure the firmware on the externally managed Intel switches. Refer to
Configure Externally Managed Switches
7.
Install Intel® Omni-Path Software on the remaining servers. Refer to Install Intel
Omni-Path Fabric Host Software on the Remaining Servers
8.
Verifying Intel® Omni-Path Software on the remaining servers. Refer to Verify
Intel Omni-Path Fabric Host Software on the Remaining Servers
9.
Install Intel® Omni-Path Fabric Suite on additional Fabric Management Nodes.
Refer to Installation of Additional Fabric Management Nodes
10.
Configure and initialize health check tools. Refer to Configure and Initialize Health
Check Tools
11.
Run the High Performance Linpack application. Refer to Running High Performance
Linpack
Intel® Omni-Path Fabric Software
Installation Guide
110
Omni-Path Fabric
November 2015
Order No.: H76467-1.0
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising