PathScale InfiniPath Interconnect Installation Guide - 1.3

PathScale InfiniPath Interconnect Installation Guide - 1.3
PathScale™ InfiniPath™ Interconnect
Installation Guide
PathScale, Inc.
Version 1.3
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
Copyright (c) QLogic Corp. 2006. All rights reserved.
Copyright (c) PathScale 2004, 2005, 2006. All rights reserved.
QLogic, the QLogic logo, PathScale, InfiniPath, and EKOPath are trademarks of QLogic. in the
United States and other countries. All other trademarks are the property of their respective owners.
In accordance with the terms of their valid QLogic customer agreements, customers are permitted
to make electronic and paper copies of this document for their own exclusive use.
All other forms of reproduction, redistribution, or modification is prohibited without the prior
express written permission of QLogic Corp.
Document Number: 3-071305-05
Last modified: June 9, 2006
Revision #
Date
Change
1.0
August 2005
Initial version
1.1
November 2005
Added new supported kernels and distributions info. New diagrams. Info on
OpenIB. Info on Rocks.
1.2
February 2006
Formatting changes. Added numbering style to headers. Added document
number. Updated copyright information.
1.3 Beta 1
April 2006
All references to OpenIB are now
OpenFabrics. Merged former OpenIB
Installation Guide with this Guide.
1.3
June 2006
Easier installation instructions.
Updated technical information.PE-880
Illustrations.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
2
Table of Contents
PREFACE
About this Guide
6
CHAPTER 1 Hardware Installation
9
1.1: Overview
1.2: Hardware installation requirements
1.2.1: Hardware
1.2.1.1: Form Factors
1.2.2: Switches and cabling
1.2.3: Software
1.2.4: Configuring the BIOS
1.3: Safety with electricity
1.4: Unpacking information
1.4.1: Verify package contents
1.4.2: List of the package contents
1.4.3: Unpacking the PathScale InfiniPath Adapter
1.5: Hardware installation
1.5.1: Hardware installation for PE-880 with PCIe riser
1.5.2: Hardware installation for PE-880 without riser
1.5.3: Hardware installation for HT-465 with HTX riser
1.5.4: Hardware installation for HT-465 without HTX riser
1.6: Cabling the adapter to the InfiniBand switch
1.6.1: Standard InfiniBand copper cabling
1.6.2: Optical fibre option
1.7: Completing the installation
CHAPTER 2 Software Installation
26
2.1: Software installation overview
2.2: Software installation requirements
2.2.1: Linux environment
2.2.2: Before installing InfiniPath and OpenFabrics
2.2.3: Kernel patches
2.2.4: Types of nodes in a cluster environment
2.3: InfiniPath Software RPMs
PathScale InfiniPath Interconnect Installation Guide
9
9
9
10
11
11
11
12
12
12
13
14
15
15
18
20
22
24
24
24
24
Version 1.3
27
28
28
28
29
29
30
3
Table of Contents
2.4: OpenFabrics for PathScale RPMs
2.5: Downloading and installing InfiniPath and OpenFabrics software
2.5.1: Downloading the RPMs
2.5.2: Installing the software
2.6: Managing and installing software using Rocks
2.6.1: Installing Rocks and PathScale RPMs
2.6.2: Installing Rocks and PathScale RPMs-Details
2.6.3: Further information on Rocks and kickstart
2.7: InfiniPath and OpenFabrics driver overview
2.8: Configuring the InfiniPath drivers
2.8.1: Configuring the ipath_core driver
2.8.2: Configuring the ipath_ether network interface
2.8.2.1: ipath_ether configuration on Fedora and RHEL4
2.8.2.2: ipath_ether configuration on SUSE 9.3, SUSE 10.0, and SLES 10 (Beta)
2.9: OpenFabrics configuration and startup
2.9.1: Modify /etc/sysconfig/infinipath
2.9.2: Configuring the IPoIB network interface
2.9.3: Enabling and restarting
2.9.4: Unloading OpenFabrics modules
2.10: OpenSM
2.11: Testing the drivers
2.12: Recompiling the drivers
2.13: Further information on configuring and loading drivers
2.14: Compiler support
2.15: Switch configuration and monitoring
2.16: LED link and data indicators
2.17: Customer acceptance utility
2.18: Removing software packages
2.18.1: Downgrading
2.19: Installing newer drivers from other distributions
2.20: Installation troubleshooting
APPENDIX A RPM Descriptions
53
A.1: InfiniPath RPMs
A.1.1: Different nodes may use different RPMs
A.2: OpenFabrics RPMs
PathScale InfiniPath Interconnect Installation Guide
30
31
31
31
33
33
34
37
37
38
38
39
39
40
43
43
44
45
46
46
47
47
48
48
48
48
49
50
51
51
52
Version 1.3
53
53
54
4
Chapter :
APPENDIX B Regulatory Information
58
Index
PathScale InfiniPath Interconnect Installation Guide
59
Version 1.3
5
About this Guide
PREFACE
The PathScale InfiniPath Interconnect Installation Guide contains complete instructions for
installing the PathScale InfiniPath Interconnect hardware adapters and the InfiniPath and
OpenFabrics software. The adapters covered in this guide are the PathScale InfiniPath PCI
Express™ adapter, the PE-880, and the HT-460/HT-465 HTX™ adapters. This chapter
describes the contents, intended audience, and organization of the PathScale InfiniPath
Interconnect Installation Guide, and defines the conventions used to convey instructions
and noteworthy information.
What’s new in this release
QLogic Corp. acquired PathScale in April 2006. In this 1.3 release, product names and
website locations will still refer to PathScale. In a future release, product names, internal
program and output message names, and website locations will transition to QLogic.
This version of InfiniPath provides support for PathScale’s new PCI Express HCA, the
InfiniPath PE-880, expanded operating system support, and support for the latest OpenFabrics software stack. The PE-880 is supported on Opteron and Intel x86_64 systems with
PCIe x8 or x16 slots. This software will support all PathScale InfiniPath HCAs, including
the HT-460/HT-465, which leverages HTX™. Multiple InfiniPath cards per node are supported. A single software installation works for all the cards.
Included in this InfiniPath 1.3 software release are the following components:
•
•
•
•
•
InfiniPath HCA driver
InfiniPath MPI
Standard networking over Ethernet emulation (TCP/IP, etc.)
InfiniPath subnet management agent and associated utilities
OpenFabrics for InfiniPath, including IPoIB and OpenSM
The OpenFabrics for PathScale RPMs as described in this documentation are compatible
with the InfiniPath Version 1.3 release. The earlier OpenIB for PathScale RPMs (Versions
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
6
Preface
0.1.0 and 0.2.0) for the InfiniPath 1.1 and 1.2 releases are not compatible with this InfiniPath Version 1.3 release.
More details about the hardware and software are found in Chapter 1 “Hardware Installation” and Chapter 2 “Software Installation”.
Who should read this guide
This installation guide is intended for cluster administrators responsible for installing the
InfiniPath PE-880 on a AMD Opteron or Intel® EM64T processor-based system, or the
InfiniPath HT-460/HT-465 product on an AMD Opteron™ machine cluster. More detailed
installation infromation and instructions for administration of the InfiniPath cluster are
provided in the PathScale InfiniPath Interconnect User’s Guide.
This guide assumes you are familiar with cluster networking and with the specific hardware to which your processors are connected. Prior to installing the InfiniPath Adapter,
you should have basic knowledge of your host and target operating systems, and working
knowledge of message passing concepts.
This document does not contain all the information you need in order to use basic Linux
commands or to perform all system administration tasks. For this information, see the
software documentation that you received with your system.
How this guide is organized
The PathScale InfiniPath Interconnect Installation Guide is organized into these sections:
•
Chapter 1 “Hardware Installation”, instructions for installing the InfiniPath PE-880,
and HT-460 or HT-465 hardware
• Chapter 2 “Software Installation”, instructions for installing the PathScale InfiniPath
and OpenFabrics software
• Appendix A “RPM Descriptions”
• Appendix B “Regulatory Information”
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
7
Chapter :
Conventions used in this guide
This Guide uses these typographical conventions:
Convention
Meaning
command
Fixed-space font is used for literal items such as commands, functions, programs, files and pathnames, and program output;
variable
Italic fixed-space font is used for variable names in programs and
command lines.
concept
Italic font is used for emphasis, concepts, and publication titles.
user input
Bold fixed-space font is used for literal items in commands or
constructs that you type in.
$
Indicates a command line prompt.
#
Indicates a command line prompt as root when using bash or sh.
[]
Brackets enclose optional elements of a command or program
construct.
...
Ellipses indicate that a preceding element can be repeated.
>
Right caret identifies the cascading path of menu commands used
in a procedure.
1.1
The current version number of the software is included in the
RPM names and within this documentation.
NOTE:
Indicates important information.
Accessing documentation
The InfiniPath product documentation includes:
• The PathScale InfiniPath Interconnect Installation Guide
• The PathScale InfiniPath Interconnect User’s Guide
• Release Notes
The Troubleshooting Appendix for installation, InfiniPath and OpenFabrics administration, and MPI issues is located in the PathScale InfiniPath Interconnect User’s Guide.The
guides, release notes, man pages and other InfinPath documentation are also available on
the PathScale web site:
http://www.pathscale.com/docs/infinipath/1.3/docs-1.3.html
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
8
CHAPTER 1
Hardware Installation
This chapter lists the requirements and provides instructions for installing the PathScale
InfiniPath Interconnect adapters. Instructions are included for the PathScale InfiniPath
PCI Express Adapter and PCIe riser card, PE-880, and InfiniPath HT-460 or HT-465
adapter hardware and HTX riser card. These components will also be referred to as the
adapter and the riser card in the remainder of this Guide.
1.1 Overview
The adapter is a low-latency, high-bandwidth, high message rate cluster interconnect for
InfiniBand. The PE-880 adapter plugs into a standard PCI Express x8 or x16 slot on AMD
Opteron or Intel EMT64 processor-based systems. The HT-460 or HT-465 attaches
directly to AMD Opteron processor-based servers through a standard HyperTransport
HTX slot on an Opteron motherboard.
The InfiniPath interconnect is InfiniBand 4X, with a raw data rate of 10Gb/s (data rate of
8Gb/s).
OpenFabrics for PathScale is interoperable with other vendors’ InfiniBand HCAs running
compatible OpenFabrics releases.
1.2 Hardware installation requirements
This section lists hardware and software environment requirements for installing the
InfiniPath PE-880, the InfiniPath HT-460 or HT-465.
1.2.1 Hardware
The PE-880 is designed to work on both AMD Opteron and Intel EM64T-based motherboards with PCI Express x8 or x16 slots. The HT-460/HT-465 adapter is designed to work
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
9
Chapter 1: Hardware Installation
on AMD Opteron-based motherboards. For the most up-to-date listing of supported
motherboards (and operating systems), please refer to our support website:
http://www.pathscale.com/support.html
The PE-880 is installed in motherboards that have PCI Express x8 or x16 slots. Any 64-bit
EM64T or Opteron motherboard with PCIe support should work. Installation of the PE880 in a 1U or 2U chassis requires the use of the riser card. See Figure 1-3 for an illustration of a PCI Express slot in a typical motherboard.
The HT-460 or HT-465 adapters are installed in motherboards that support HTX cards.
See Figure 1-7 for an illustration of an HTX slot for a typical Opteron motherboard. Installation of the InfiniPath adapter in a 1U or 2U chassis requires the use of the riser card. The
currently supported motherboards that support HTX cards are:
• Iwill DK8-HTX, a dual socket Opteron motherboard
• Supermicro H8DCE-HT, a dual socket Opteron motherboard
• Supermicro H8QC81E, a quad socket Opteron motherboard
The motherboard vendor is the optimal source for information on the layout and use of
HyperTransport-enabled expansion slots on supported motherboards.
A standard InfiniBand copper cable up to a length of twenty meters is required. See 1.6.1
“Standard InfiniBand copper cabling” for more information. Optical fibre with optical
adapters such as the Emcore is also supported. See the section 1.6.2 “Optical fibre option”.
1.2.1.1 Form Factors
The PE-880 is the model number for the adapter card that ships in the PCIe low profile
short form factor. This card can be used with either a full height or low profile faceplate.
The HT-460 is the model number for the adapter card that ships in the HTX full height
short form factor. The HTX low profile form factor is referred to as the HT-465. It is the
same as the HT-460, except for its more compact size. In either case, the adapter is backward and forward compatible for the motherboards in which it is supported. Unless otherwise stated, the adapter is referred to as the HT-465 in this documentation.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
10
Chapter 1: Hardware Installation
When the HT-460 or HT-465 adapter is installed with the riser card, it may prevent some
or all of the other PCI expansion slots from being used. This depends upon the form factor
of the adapter and motherboard.
NOTE: Run ipath_control -i to see information on which form factor adapter is
installed. The PCIe half height short form factor is referred to as the PE-880, the HTX
full height short form factor is referred to as the HT-460, and the HTX low profile
form factor is referred to as the HT-465. The file /sys/bus/pci/drivers/
ipath_core/00/boardversion contains the same information. For more information, see C.9.9 “ipath_control” in the Troubleshooting Appendix in the PathScale
InfiniPath Interconnect User’s Guide.
For the most up-to-date listing of InfiniPath PCIe and HTX Adapter model numbers and
the motherboards in which they are supported, please go to our support website:
http://www.pathscale.com/support.html
1.2.2 Switches and cabling
InfiniPath utilizes standard, off-the-shelf InfiniBand 4X switches and cabling. There are
two options for Subnet Management in your cluster:
• Use the Subnet Manager on one or more managed switches supplied with your Infiniband switches.
• Use OpenSM if OpenFabrics is enabled.
More information on tested and supported switches is listed in the PathScale InfiniPath
Interconnect User’s Guide in Chapter 1 “InfiniPath interconnect overview”. Cabling instructions are found in in this Guide in 1.6 “Cabling the adapter to the InfiniBand switch”.
1.2.3 Software
For each machine in the cluster obtain and install Linux kernel software.
See 2.2 “Software installation requirements” for information on supported distributions,
associated kernels and other software requirements..
NOTE: Before installing the InfiniPath software, complete the hardware installation as
described in this chapter.
1.2.4 Configuring the BIOS
A properly configured BIOS is required. The BIOS on the motherboard contains code
needed to bootstrap the system when it is powered up. It also contains certain parameters
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
11
Chapter 1: Hardware Installation
characterizing the system. These parameters may include date and time, configuration settings, and information about the installed hardware.
There are currently two issues concerning BIOS setting that you need to be aware of. The
first is that MTRR mapping needs to be set to “Discrete”, and ACPI needs to be enabled.
These are discussed in greater detail in Appendix C “BIOS settings” in the PathScale InfiniPath Interconnect User’s Guide.
You can check and adjust these BIOS settings using the BIOS Setup Utility. For specific
instructions on how to do this, follow the hardware documentation that came with your
system.
1.3 Safety with electricity
Observe these guidelines and safety precautions when working around computer hardware
and electrical equipment:
1.
Locate the power source shutoff for the computer room or lab where you are working.
This is where you will turn OFF the power in the event of an emergency or accident.
Never assume that power has been disconnected for a circuit. Always check first.
2. Don’t wear loose clothing. Fasten your tie or scarf, remove jewelry, and roll up your
sleeves. Wear safety glasses when working under any conditions that might be hazardous to your eyes.
3. Shut down and disconnect the system’s power supply from AC service before you begin
work, to insure that standby power is not active. Note that many motherboards and
power supplies maintain standby power at all times. Inserting or removing components
while standby is active can damage them.
4. Use normal precautions to prevent electrostatic discharge, which can damage integrated
circuits.
1.4 Unpacking information
This section provides instructions for safely unpacking and handling the InfiniPath
adapter. To avoid damaging the adapter card, always take normal precautions to avoid electrostatic discharge.
1.4.1 Verify package contents
The InfiniPath adapter system should arrive in good condition. Before unpacking, check
for any obvious damage to the packaging. If you find any obvious damage to the packaging
or to the contents, please notify your reseller immediately.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
12
Chapter 1: Hardware Installation
1.4.2 List of the package contents
The package contents for the PE-880 adapter are:
•
InfiniPath PE-880
Standard PCIe risers can be used, typically supplied by your system or motherboard vendor. The contents are illustrated in Figure 1-1 below.
The package contents for the HT-465 adapter are:
•
•
InfiniPath HT-465
HTX riser card for use in 1U or 2U chassis
The contents are illustrated in Figure 1-2 below.
The PE-800 and the HT-400 are the PathScale InfiniPath Interconnect ASICs, which are
the central components of the interconnect. The PE-800 is shown in Figure 1-1, and the
HT-400 is shown in Figure 1-2.
PCI Express riser card. Not supplied; shown for reference.
PCI Express
edge connectors
InfiniBand
connector
PE-800 ASIC
FIGURE 1-1.
Face plate
InfiniPath PE-880 card with riser, top view
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
13
Chapter 1: Hardware Installation
HTX riser card
HTX edge
InfiniBand
connector
connectors
HT-400 ASIC
HT-465 Low Profile card
Face plate
PathScale
InfiniBand
connector
Face plate
PathScale
HT-460 Full Height
Short card
FIGURE 1-2.
InfiniPath HT-460/HT-465 Full and Low Profile cards with riser, top view
1.4.3 Unpacking the PathScale InfiniPath Adapter
When unpacking, ground yourself before removing the InfiniPath Adapter from the antistatic bag.
1.
Grasping the InfiniPath Adapter by its face plate, pull the adapter out of the anti-static
bag. Handle the adapter only by its edges or the IB connector. Do not allow the InfiniPath Adapter or any adapter card components to touch any metal parts.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
14
Chapter 1: Hardware Installation
2.
After checking for visual damage, store the InfiniPath Adapter and the riser card in the
anti-static bag until you are ready to install them.
1.5 Hardware installation
This section contains hardware installation instructions for all of the adapters. Instructions
for the PE-880 PCI Express (PCIe) adapter are given first, followed by instructions for the
HT-465 adapter.
1.5.1 Hardware installation for PE-880 with PCIe riser
Most installations will be in 1U and 2U chassis, using the PCI Express riser card. This
results in a horizontal installation of the PE-880. This type of installation will be described
first. Installation in a 3U chassis will be described in the next section.
Installation of InfiniPath PE-880 in 1U or 2U chassis requires installation with a PCI
Express riser card.
NOTE: A taller riser card can be used if necessary. The PE-880 can connect to any of the
standard compatible PCIe riser cards.
To install the InfiniPath adapter with a PCI Express riser card:
1.
2.
3.
4.
5.
The BIOS should be already be configured properly by the motherboard manufacturer.
However if any additional BIOS configuration is required, it will usually need to be
done before installing the InfiniPath adapter. See 1.2.4 “Configuring the BIOS”.
Shut down the power supply to the system into which you’ll be installing the InfiniPath
adapter.
Take precautions to avoid damage to the cards by grounding yourself or touching the
metal chassis to discharge static electricity before handling them.
Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with
your system.
Locate the PCIe slot on your motherboard. Note that the PCIe slot has two separate sections, with the smaller slot opening located towards the front. See Figure 1-3, below.
These two sections correspond to the shorter and longer connector edges of the adapter
and riser.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
15
Chapter 1: Hardware Installation
PCIe slot in
typical motherboard
FIGURE 1-3.
PCIe slot in a typical motherboard
6.
Determine if a blanking panel is installed in your chassis. If it is, remove it so that the
InfiniBand connector will be accessible. Refer to your system vendor instructions for
how to remove the blanking panel.
7. Remove the InfiniPath PE-880 from the anti-static bag.
8. Locate the face plate on the connector edge of the card.
9. Connect the InfiniPath adapter and PCIe riser card together, forming the assembly that
you’ll insert into your motherboard. To do this, first visually line up the card slot connector edge with the edge connector of the PCIe riser card.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
16
Chapter 1: Hardware Installation
PCIe riser card
InfiniPath adapter
Face plate
LEDs
InfiniBand connector
FIGURE 1-4.
InfiniPath PE-880 assembly with riser card
10. Holding the InfiniPath adapter by its edges, carefully insert the card slot connector into
the PCIe riser card edge connector, as show in Figure 1-4 above. The result is a combined L-shaped assembly of the PCIe riser card and InfiniPath Adapter. This assembly
is what you’ll insert into the PCIe slot on the motherboard in the next step.
11. Turn the assembly so that the riser card connector edge is facing the PCIe slot on the
mother board, and the face plate is toward the front of the chassis.
12. Holding this assembly above the motherboard at about a 45 degree angle, slowly lower
it so that the InfiniBand connector on the face plate clears the blanking panel opening
of the chassis from the inside. Slowly align the connector edge of the riser card with the
motherboard’s PCIe slot. The short section of the connector must align with the short
section of the slot.
13. Insert the riser assembly into the motherboard’s PCIe slot, ensuring good contact. The
InfiniPath Adapter should now be parallel to the motherboard and about one inch
above it.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
17
Chapter 1: Hardware Installation
FIGURE 1-5.
Assembled PE-880 with riser
14. Secure the face plate to the chassis.
The InfiniPath adapter has a screw hole on the side
of the face plate which can be attached to the chassis with a retention screw. The securing method may vary depending upon the chassis manufacturer. Refer to the system
documentation for information about mounting details such as mounting holes, screws
to secure the card, or other brackets.
The InfiniPath PE-880 with PCIe riser card is now installed. Next, install the cables as
described in 1.6 “Cabling the adapter to the InfiniBand switch”. Then test your installation
by powering up and verifying link status. See 1.7 “Completing the installation”.
1.5.2 Hardware installation for PE-880 without riser
The installation of InfiniPath PE-880 without a PCIe riser card requires a 3U or larger
chassis. The card slot connectors on the PE-880 fit into the PCIe slot in a vertical installation.
To install the InfiniPath Adapter without the PCIe riser card:
1.
The BIOS should be already be configured properly by the motherboard manufacturer.
However if any additional BIOS configuration is required, it will usually need to be
done before installing the InfiniPath adapter. See 1.2.4 “Configuring the BIOS”.
2. Shut down the power supply to the system into which you’ll be installing the InfiniPath
Adapter.
3. Take precautions to avoid damage to the cards by grounding yourself or touching the
metal chassis to discharge static electricity before handling them.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
18
Chapter 1: Hardware Installation
4.
5.
6.
7.
8.
If you are installing the InfiniPath Adapter into a covered system, you will first need to
remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with
your system.
Locate the PCIe slot on your motherboard. See Figure 1-3, above.
Remove the InfiniPath Adapter from the anti-static bag.
Visually line up the card slot connector with the motherboard’s HTX slot. The short
section of the connector must align with the short section of the slot.
Holding the InfiniPath Adapter by its edges, carefully insert the card slot connector into
the motherboard’s PCIe slot, ensuring good contact. This results in the InfiniPath
Adapter being positioned perpendicular to the motherboard.
FIGURE 1-6.
9.
PE-880 without riser installed a 3U chassis
Secure the face plate to the chassis. The InfiniPath adapter has a screw hole on the side
of the face plate which can be attached to the chassis with a retention screw. The securing method may vary depending upon the chassis manufacturer. Refer to the system
documentation for information about mounting details such as mounting holes, and
screws to secure the card, or other brackets.
Next, install the cables. as described in 1.6 “Cabling the adapter to the InfiniBand switch”.
Then test your installation by powering up the system . See 1.7 “Completing the installation”.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
19
Chapter 1: Hardware Installation
1.5.3 Hardware installation for HT-465 with HTX riser
Most installations will be in 1U and 2U chassis, using the HTX riser card. This results in a
horizontal installation of the HT-465. This type of installation will be described first.
Installation in a 3U chassis will be described in the next section.
Installation of InfiniPath HT-465 in 1U or 2U chassis requires installation with an HTX
riser card.
NOTE: The illustrations in this section are shown for the Full Height Short form factor.
Installation of the HTX Low Profile form factor will follow the same steps.
To install the InfiniPath adapter with an HTX riser card:
1.
2.
3.
4.
5.
The BIOS should be already be configured properly by the motherboard manufacturer.
However if any additional BIOS configuration is required, it will usually need to be
done before installing the InfiniPath adapter. See 1.2.4 “Configuring the BIOS”.
Shut down the power supply to the system into which you’ll be installing the InfiniPath
adapter.
Take precautions to avoid damage to the cards by grounding yourself or touching the
metal chassis to discharge static electricity before handling them.
Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with
your system.
Locate the HTX slot on your motherboard. Note that the HTX slot has two separate
connectors, corresponding to the connector edges of the adapter. See figure below.
HTX slot in a typical Opteron motherboard
FIGURE 1-7.
PathScale InfiniPath Interconnect Installation Guide
HTX Slot
Version 1.3
20
Chapter 1: Hardware Installation
6.
Determine if a blanking panel is installed in your chassis. If it is, remove it so that the
InfiniBand connector will be accessible. Refer to your system vendor instructions for
how to remove the blanking panel.
7. Remove the InfiniPath HT-465 from the anti-static bag.
8. Locate the face plate on the connector edge of the card.
9. Connect the InfiniPath adapter and HTX riser card together, forming the assembly that
you’ll insert into your motherboard. To do this, first visually line up the card slot connector edge with the edge connector of the HTX riser card.
HTX riser card
Face plate
InfiniPath adapter
LEDs
InfiniBand connector
FIGURE 1-8.
InfiniPath adapter with riser card.
10. Holding the InfiniPath adapter by its edges, carefully insert the card slot connector into
the HTX riser card edge connector, as show in Figure 3 above. The result is a combined
L-shaped assembly of the HTX riser card and InfiniPath Adapter. This assembly is what
you’ll insert into the HTX slot on the motherboard in the next step.
11. Turn the assembly so that the riser card connector edge is facing the HTX slot on the
mother board, and the face plate is toward the front of the chassis.
12. Holding this HT assembly above the motherboard at about a 45 degree angle, slowly
lower it so that the InfiniBand connector on the face plate clears the blanking panel
opening of the chassis from the inside. Slowly align the connector edge of the HTX
riser card with the motherboard’s HTX slot. The HT riser and HTX slot should line up
perfectly at this point.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
21
Chapter 1: Hardware Installation
13. Insert the HT riser assembly into the motherboard’s HTX slot, ensuring good contact.
The InfiniPath Adapter should now be parallel to the motherboard and about one inch
above it.
FIGURE 1-9.
Assembled HT-465 with riser
14. Secure the face plate to the chassis.
The InfiniPath adapter has a screw hole on the side
of the face plate which can be attached to the chassis with a retention screw. The securing method may vary depending upon the chassis manufacturer. Refer to the system
documentation for information about mounting details such as mounting holes, screws
to secure the card, or other brackets.
The InfiniPath HT-465 with HTX riser card is now installed. Next, install the cables as
described in 1.6 “Cabling the adapter to the InfiniBand switch”. Then test your installation
by powering up and verifying link status. See 1.7 “Completing the installation”.
1.5.4 Hardware installation for HT-465 without HTX riser
The installation of InfiniPath HT-465 without an HTX riser card requires a 3U or larger
chassis. The card slot connectors on the HT-465 fit into the HTX slot in a vertical installation.
To install the InfiniPath Adapter without the HTX riser card:
1.
The BIOS should be already be configured properly by the motherboard manufacturer.
However if any additional BIOS configuration is required, it will usually need to be
done before installing the InfiniPath adapter. See 1.2.4 “Configuring the BIOS”.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
22
Chapter 1: Hardware Installation
2.
3.
4.
5.
6.
7.
8.
Shut down the power supply to the system into which you’ll be installing the InfiniPath
Adapter.
Take precautions to avoid damage to the cards by grounding yourself or touching the
metal chassis to discharge static electricity before handling them.
If you are installing the InfiniPath Adapter into a covered system, you will first need to
remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with
your system.
Locate the HTX slot on your motherboard. See Figure 1-7, above.
Remove the InfiniPath Adapter from the anti-static bag.
Visually line up the card slot connector with the motherboard’s HTX slot.
Holding the InfiniPath Adapter by its edges, carefully insert the card slot connector into
the motherboard HTX slot ensuring good contact. This results in the InfiniPath
Adapter being positioned perpendicular to the motherboard.
FIGURE 1-10.
9.
HT-465 without riser installed a 3U chassis
Secure the face plate to the chassis. The InfiniPath adapter has a screw hole on the side
of the face plate which can be attached to the chassis with a retention screw. The securing method may vary depending upon the chassis manufacturer. Refer to the system
documentation for information about mounting details such as mounting holes, and
screws to secure the card, or other brackets.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
23
Chapter 1: Hardware Installation
Next, install the cables. as described in 1.6 “Cabling the adapter to the InfiniBand switch”.
Then test your installation by powering up the system . See 1.7 “Completing the installation”.
1.6 Cabling the adapter to the InfiniBand switch
1.6.1 Standard InfiniBand copper cabling
The cable installation uses a standard InfiniBand cable. Any InfiniBand cable that has
been qualified by the vendor should work. The longest IB cable we have currently qualified
is twenty meters.
To install the InfiniBand cables:
1.
2.
3.
4.
5.
Check that you have removed the protector plugs from the cable connector ends.
Different vendor cables might have different latch mechanisms. Determine if your cable
has a spring-loaded latch mechanism. If your cable is spring-loaded, grasp the metal
shell and pull on the plastic latch to release the cable. To insert, push and the cable snaps
into place. You will hear a short “click” sound from the cable connector when it snaps in.
If your cable latch mechanism is not spring-loaded, simply push on the metal case, then
push the plastic latch to lock the cable in place.
The InfiniBand cables are symmetric; either end can be plugged into the switch. Connect the InfiniBand cable to the connector on the InfiniPath PE-880 or HT-465.
Depress the side latches of the cable when connecting. (On some cables this latch is
located at the top of the cable connector.) Make sure the lanyard handle on the cable
connector is slid forward toward the card connector until fully engaged.
Connect the other end of the cable to the InfiniBand switch.
1.6.2 Optical fibre option
The PathScale InfiniPath adapter also supports connection to the switch by means of optical fibres through optical media converters such as the Emcore QT2400. Not all switches
support these types of convertors. For more information on the Emcore convertor, contact www.emcore.com.
1.7 Completing the installation
To complete the hardware installation:
1.
Complete any other installation steps for other components.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
24
Chapter 1: Hardware Installation
2.
Replace the cover plate and back panel.
3. Verify that the power cable is properly connected.
4. Turn on the power supply, and boot the system normally.
5. Watch the LED indicators. The LEDs will flash only once, briefly, at power-on. The
LEDs are functional only after the InfiniPath software has been installed, the driver has
been loaded, and the system is connected to an InfiniBand switch. To use the LEDs to
check state, see 2.16 “LED link and data indicators”.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
25
Software Installation
CHAPTER 2
This chapter provides instructions for installing the PathScale InfiniPath and OpenFabrics
for PathScale software.
The InfiniPath software includes drivers, protocol libraries, PathScale’s implementation of
the MPI message passing standard, and example programs, including benchmarks.
The software provided with the PathScale InfiniPath Interconnect product consists of:
• PathScale MPI
• Protocol Libraries
• Layered Ethernet driver
• Layered OpenFabrics driver
• Library, protocol, and other support for OpenFabrics
• Subnet Management Agent
• InfiniPath Driver
• Basic tests, benchmarks, and utilities
OpenFabrics for PathScale kernel module support is now built and installed as part of the
InfiniPath RPM install. The OpenFabrics user commands, libraries, and utilities are also
provided in this are part of the OpenFabrics RC4 (Release Candidate 4) RPMs. PathScale’s
OpenFabrics layered driver, ib_ipath, appears as a completely standard OpenFabrics
driver and is OpenFabrics compliant.
The OpenFabrics components of the software is intended for developers who are interested
in using OpenFabrics. As this is an early release, you may find that some features have not
yet been fully implemented. Support for the various libraries, interfaces, protocols, and
programs is currently under development. This release provides support for the following
OpenFabrics Upper Layer Protocols and utilities:
•
•
•
•
IPoIB (TCP/IP networking)
SDP (Sockets Direct Protocol)
OpenSM
UD (Unreliable Datagram)
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
26
Chapter 2: Software Installation
• RC (Reliable Connection)
• UC (Unreliable Connection)
• SRQ (Shared Receive Queue)
Future releases will provide support for:
• uDAPL/kDAPL (user Direct Access Provider Library/kernel Direct Access Provider
Library)
• SRP (SCSI RDMA Protocol)
• iSER (iSCSI Extensions for RDMA)
Further information, including the use of MPI, is found in the companion document,
PathScale InfiniPath Interconnect User’s Guide. For convenience, some information appears
in both documents.
2.1 Software installation overview
InfiniPath and OpenFabrics software installation involves these steps in the following
order:
1.
2.
3.
4.
5.
6.
Make sure that the InfiniPath PE-880 or HT-465 installation has been completed
according to the instructions in Chapter 1 “Hardware installation”.
Verify that the Linux kernel software is installed on each node in your cluster. The
required kernels and supported Linux distributions for both InfiniPath and OpenFabrics are defined below under 2.2 “Software installation requirements”.
For each release, download the InfiniPath/Openfabrics software from the PathScale
web site to a local server directory, and from there install the appropriate packages on
each cluster node as described under 2.3 “InfiniPath Software RPMs”, 2.4 “OpenFabrics
for PathScale RPMs”, and 2.5 “Downloading and installing InfiniPath and OpenFabrics
software”. Configure the InfiniPath and OpenFabrics drivers as described under 2.8
“Configuring the InfiniPath drivers”.
Check that all InfiniBand switches are configured as described under 2.15 “Switch configuration and monitoring”.
Perform the recommended health checks as described under 2.17 “Customer acceptance utility”.
After installing the InfiniPath and OpenFabrics for PathScale refer to the PathScale
InfiniPath Interconnect User’s Guide for more information on using the InfiniPath and
OpenFabrics products.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
27
Chapter 2: Software Installation
2.2 Software installation requirements
Information on supported distributions and kernels and on setting up your environment
are given below.
2.2.1 Linux environment
The currently supported distributions and associated Linux kernel versions for InfiniPath
and OpenFabrics for PathScale are listed in the following table. The kernels are the ones
that shipped with the distributions, unless otherwise noted.
TABLE 2-1. InfiniPath/OpenFabrics supported distributions and kernels
Distribution
InfiniPath/OpenFabrics
supported kernels
Fedora Core 3
2.6.12
Fedora Core 4
2.6.12, 2.6.13, 2.6.14, 2.6.15, 2.6.16
Red Hat Enterprise Linux 4 (RHEL4)
2.6.9 (U2/U3)
CentOS 4.0-4.3 (Rocks 4.0-4.3)
2.6.9
SUSE Linux 9.3
2.6.11
SUSE Linux 10.0
2.6.13
SLES 10 (Release Candidate)
2.6.16
Future releases will be supported on more distributions and kernels, including Fedora Core
5.
2.2.2 Before installing InfiniPath and OpenFabrics
There are a few things to keep in mind when setting up the environment:
• It is possible to have a cluster running with different kernel versions. However, PathScale recommends and supports clusters where all nodes run equivalent software.
• Different distributions require different versions of the InfiniPath software distribution,
as described below under 2.3 “InfiniPath Software RPMs”.
• Among the many optional packages that each distribution offers, the InfiniPath software requires openssh and openssh-server and, if the MPD job launcher is to be
used, python. These must be on every node. Note that in the SUSE and SLES 10 distributions openssh-server is a part of the openssh package.
• The runtime and build environments should be the same. Compatibility between executables built on different distributions cannot be guaranteed.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
28
Chapter 2: Software Installation
• You need to install sysfsutils for your distribution before installing the OpenFabrics for PathScale RPMs, as there are dependencies. Check your distribution’s documentation for information about sysfsutils. This package is called udev on SUSE 9.3.
2.2.3 Kernel patches
Some kernels, such as some versions of Fedora Core 4 (2.6.16), have CONFIG_PCI_MSI=n
as the default. If the InfiniPath driver is being compiled on a machine without
CONFIG_PCI_MSI=y configured, you will get a compilation error. This default may also
be introduced with updates to other Linux distributions or local configuration changes.
A change was made in the kernel.org 2.6.12 kernel that can cause an InfiniPath driver runtime error with the PE-880. This change is found in most linux distributions with 2.6.12 2.6.16 kernels, including Fedora Core 3, Fedora Core 4, and SUSE Linux 10.0. Affected systems are thoses that contain the AMD8131 PCI bridge. Such systems may experience a
problem with MSI (Message Signaled Interrupt) that impairs the operation of the InfiniPath PE-880 adapter.
PathScale-built kernels and patches for these problems are available for download on the
PathScale website. Please go to:
http://www.pathscale.com/infinipath_support/infinipath_index.html
Follow the links to the download pages. This topic is also covered in C.4.2 “pci_msi_quirk”
in Appendix C “Troubleshooting” in the User’s Guide.
2.2.4 Types of nodes in a cluster environment
In a cluster environment different nodes may be used for different functions, such as
launching jobs, for software development, or running jobs. We define these nodes in the
following way:
• Front end node. This node will launch jobs. It is referred to as the front end node in this
documentation.
• Compute node. These are the nodes that are used to run jobs.
• A development or build node. These are the machines on which examples or benchmarks can be compiled.
Any machine can serve any combination of these three purposes, but a typical cluster has
many compute nodes and only one or just a few front end nodes. The number of nodes
used for development will vary. The terms defined above will be used in the rest of the
documentation.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
29
Chapter 2: Software Installation
2.3 InfiniPath Software RPMs
Linux distributions of InfiniPath software are installed from binary RPMs. RPM is a Linux
packaging and installation tool used by Red Hat, SUSE, and CentOS.
Each set of RPMs uses a build identifier xxx and a distribution identifier yyy. The RPM
distribution identifiers are shown below with the associated operating systems.
TABLE 2-2. RPMs to use for each node in a cluster
Distribution
identifier
Used on
fc3
Fedora Core 3, Fedora Core 4, Red Hat Enterprise Linux 4 (rhel4), CentOS 4.0-4.3 (Rocks 4.0-4.3)
suse9.3
SUSE 9.3, SUSE 10.0, SLES 10 (RC)
The individual RPMs are listed and described in Appendix A “RPM Descriptions”. To
generate a list of InfiniPath software package contents on each RPM run:
$ rpm -qlp rpm_file_name
2.4 OpenFabrics for PathScale RPMs
The OpenFabrics kernel module support is now part of the InfiniPath RPMs. However, in
order to use OpenFabrics, you need to install these user libraries, utilities and other program RPMs.
Each set of the OpenFabrics for PathScaleRPMs uses a distribution identifier yyy. The
identifiers for this release are fc4 and suse10.0. Note that these identifiers are different
than those for the InfiniPath RPMs.
The RPM distribution identifiers are shown below with the associated operating systems.
TABLE 2-3. RPMs to use for each node in a cluster
Distribution
identifier
Used on
fc4
Fedora Core 4
suse10.0
SUSE 10.0, SLES 10 (RC)
NOTE: The earlier OpenIB for PathScale RPMs (Versions 0.1.0 and 0.2.0) for the InfiniPath
1.1 and 1.2 releases are not compatible with this InfiniPath Version 1.3 release. If you
have installed these RPMs, you must remove or upgrade them prior to using the
InfiniPath Version 1.3 release. The OpenFabrics for PathScale RPMs as described in
this documentation are compatible with the InfiniPath Version 1.3 release.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
30
Chapter 2: Software Installation
To generate a list of software package contents on each OpenFabrics RPM run:
$ rpm -qlp rpm_file_name
2.5 Downloading and installing InfiniPath and
OpenFabrics software
In this section we assume that the correct Linux kernel and a supported distribution have
been installed on every node. Although PathScale recommends that all RPMs are installed
on all nodes, some are optional depending on which type of node is being used. To see
which RPMS are required or optional for each type of node, according to its function as a
compute node, front end node, development machine, or subnet manager, see Appendix A
“RPM Descriptions”.
2.5.1 Downloading the RPMs
There are multiple interdependent RPM packages that make up the InfiniPath and OpenFabrics software. The RPMs can be downloaded from:
http://www.pathscale.com/infinipath_support/downloads-1.3.html
NOTE: The OpenFabrics RPMs are also contained in this tar file.
The RPMs are packaged into a single tar file for download. Follow the instructions for
downloading on the web page. Download it to any convenient directory. Then:
$ tar xvf InfiniPath-1.3-yyy-rpm-set.tar
where yyy is the distribution identifier, fc3 or suse9.3. The tar command will create a
directory called InfiniPath-1.3-yyy-rpm-set and place the RPMs there.
NOTE: A specific download directory is not required. The install process will create and
install in the correct directories. The locations of the install directories are given in
the PathScale InfiniPath Interconnect User’s Guide in the Chapter 2 “Installed layout”
section.
2.5.2 Installing the software
Before installing, please take note of the following:
• Make sure that all previously existing OpenFabrics RPMs (including the previous
OpenIB for PathScale Beta 1 and 2 RPMs) are uninstalled. See 2.18 “Removing software
packages” for more information on uninstalling.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
31
Chapter 2: Software Installation
• It is recommended that all RPMs are installed on all nodes. For information about
selective installation, see Appendix A “RPM Descriptions”.
The RPMs need to be available on each node on which they will be used. One way to do
this is to copy the RPMs to a directory on each node that will need them. Another way is to
put the RPMs in a directory accessible (e.g., via NFS ) to every node. After making sure the
RPMs are on each node, login as root and, for the InfiniPath and MPI RPMs, run the command:
# rpm -Uvh infinipath*.rpm mpi*.rpm Development/mpi-devel*.rpm
For the OpenFabrics RPMs run the command:
# rpm -Uvh lib*.rpm opensm*.rpm srp*.rpm Development/lib*.rpm
Development/opensm*.rpm
Note that all development files now reside in the Development subdirectory.
This method above is the simplest way to install: parallel command starters can be used for
installation on multiple nodes, but this is beyond the scope of this documentation.
You can also run the the rpm command with the --verify option to check if there are
missing files for the RPMs:
$ rpm -a --verify ’mpi*’ ’infinipath*’ ’Development/mpi-devel*’
This will check to see if there are missing files for those RPMs.
In the case of the OpenFabrics RPMs, the verification command is slightly different. This is
because these RPMs have many different prefixes:
$ rpm --verify rpm_name_pre
where rpm_name_pre is the descriptive name that precedes the version and repository
identifiers in an RPM. For example:
$ rpm --verify libibverbs
would verify the contents of :
libibverbs-1.0-1.rc4.yyy.x86_64.rpm
Use ipath_checkout to flag any missing RPMs. See 2.17 “Customer acceptance utility”
for more information.
NOTE: Installation of the infinipath driver RPM (infinipath-kernel-1.3-xxx-yyy) builds kernel modules for the currently running kernel version. These infinipath modules will
work with only that kernel. If a different kernel is booted, the infinipath driver RPM
must be re-installed.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
32
Chapter 2: Software Installation
An alternate method to manage installation is given below in 2.6 “Managing and installing
software using Rocks”.
2.6 Managing and installing software using Rocks
Rocks is a distribution designed for managing clusters from the San Diego Supercomputer
center (SDSC).
Rocks is a way to manage the kickstart automated installation method that was created by
Red Hat. By using the Rocks conventions, the installation process can be automated for
clusters of any size. A Roll is an extension to the Rocks base distribution that supports different cluster types or provides extra functionality.
We extend the normal Rocks compute node appliance .xml file by adding two functions:
one to install the PathScale compilers, and an install script that loads the drivers after kickstart reboots the machine.
This section provides an overview on one way of building a Rocks cluster using the recommend rolls, and a sample xml file that describes the contents of a kickstart config file. By
reading this and following the instructions on the Rocks web site, you can then install the
PathScale RPMs on the required cluster nodes.
NOTE: There are many ways to use Rocks to manage clusters. Familiarizing oneself first with
kickstart and then Rocks is advisable before using this method to install the InfiniPath RPMs.
2.6.1 Installing Rocks and PathScale RPMs
The following instructions are for building a Rocks 4.0.0 cluster with the 2.6.9 Linux kernel. It will install both the PathScale EKO compilers and InfiniPath. These instructions are
guidelines only; you will need to refer to material on the Rocks web site to complete an
installation. If you want to use later versions of Rocks, these instructions will serve as general procedural steps.
The steps are summarized first, then more details are given in the corresponding numbered steps in the next section.
1.
2.
3.
4.
5.
Download the required rolls from the Rocks web site at http://www.rocksclusters.org/
Build the front end node with these rolls.
Download and install the config and kickstart RPMs onto the front end node.
Download and install the PathScale RPMs onto the front end node.
Create a directory that contains the config, kickstart, and PathScale RPMs on the front
end node.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
33
Chapter 2: Software Installation
6.
Create the xml file /home/install/site-profiles/4.0.0/nodes/extend-compute.xml
7. Use the customized xml file to complete the installation. This file describes the contents of the kickstart config file to install the RPMs from the front end node onto all the
other nodes. As this is a Rocks-specific process, you are referred to the Rocks web site
for futher step-by-step instructions.
2.6.2 Installing Rocks and PathScale RPMs-Details
This section fills in the installation details for the numbered steps listed above.
1.
Download the required rolls from the Rocks web site. Go to:
http://www.rocksclusters.org/ftp-site/
Follow links to the following CDs:
Boot Roll: Rocks 4.0.0 X86_64 Compute ISO
(Area51+Base+Ganglia_HPC+Java+Kernel+SGE)
OS Roll - Disk 1
OS Roll - Disk 2
You will also need:
updates-4.0.0-0.x86_64.disk1.iso
2.
Build the front end node with the above CDs. For more details see the Rocks installation documentation on the Rocks web site.
3. Download the config and kickstart RPMs onto the front end node. These can also be
found at http://www.rocksclusters.org/ftp-site/
rocks-config-4.0.0-1.x86_64.rpm
rocks-config-frontend-4.0.0-1.x86_64.rpm
roll-kernel-kickstart-4.0.0-1.noarch.rpm
4.
Install these RPMs listed above in step 3 on the front end node. For more details see the
Rocks installation documentation on the Rocks web site. Then copy the RPMs to
/home/install/contrib/4.0.0/x86_64/RPMS
NOTE: This next step gives an example that presents solutions for two common issues Rocks
users might encounter: installing both the InfiniPath RPMs and the PathScale compiler for a run-time environment. However, it is not a requirement to install the compiler.
5.
Download and install the PathScale 2.4 compiler and InfiniPath 1.3 RPMs on the front
end node; this example includes both for convenience. Then copy the RPMs to the
same directory as in step 4.
The directory /home/install/contrib/4.0.0/x86_64/RPMS should now contain the files in this list. Note that just the prefix of the RPM names for infinipath, mpi,
and the compiler are included; the full name will vary from release to release. See your
current release for the complete names.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
34
Chapter 2: Software Installation
infinipathinfinipath-kernelinfinipath-docinfinipath-libsmpi-benchmarkmpi-develmpi-docmpi-frontendmpi-libspathscale-basepathscale-compilers-libspathscale-pathdbpathscale-sub-clientrocks-config-4.0.0-1.x86_64.rpm
rocks-config-frontend-4.0.0-1.x86_64.rpm
roll-kernel-kickstart-4.0.0-1.noarch.rpm
*kernel-smp-2.6.9-current_version.src.rpm
*kernel-smp-2.6.9-current_version.x86_64.rpm
*kernel-smp-devel-2.6.9-current_version.x86_64.rpm
*The kernel RPMs are built using the how-to instructions on the rocksclusters.org web
site, and added into this file in the same manner as for the PathScale, config, and kickstart RPMs mentioned in the steps above. Check for the appropriate
current_version of 2.6.9.
6. Create the file:
/home/install/site-profiles/4.0.0/nodes/extend-compute.xml
Use the following contents:
<?xml version="1.0" standalone="no"?>
<kickstart>
<description>
</description>
<changelog>
</changelog>
<-- EKOPath Compilers -->
<package>pathscale-sub-client</package>
<package>pathscale-base</package>
<package>pathscale-compilers-libs</package>
<-- InfiniPath Drivers -->
<package>mpi-doc</package>
<package>mpi-frontend</package>
<package>mpi-libs</package>
<package>mpi-benchmark</package>
<package>mpi-devel</package>
<post>
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
35
Chapter 2: Software Installation
<-- Here we fix up Grub boot behavior -->
<-- add serial console on tty1 -->
sed -e ’s/kernel<.*>$/kernel\1 console=tty1 console=ttyS1,9600n8/’ \
/boot/grub/grub-orig.conf > /tmp/grub-orig.conf
<-- set serial console speed -->
sed -e ’s/hiddenmenu/serial --unit=0 --speed=9600 --word=8 \
--parity=no --stop=1\nterminal serial console/g’ \
/tmp/grub-orig.conf > /tmp/grub-orig2.conf
<-- remove tsc -->
sed -e ’s/tsc//g’ /tmp/grub-orig2.conf > /tmp/grub-orig3.conf
<-- remove noacpi, needed to make InfiniPath happy -->
sed -e ’s/pci=noacpi//g’ /tmp/grub-orig3.conf > /tmp/grub-orig4.conf
mv /tmp/grub-orig4.conf /boot/grub/grub-orig.conf
<!-- add getty for console to inittab at all runlevels -->
<file name="/etc/inittab" mode="append">
s1:2345:respawn:/sbin/agetty -L 9600 ttyS1 vt100
</file>
<-- Remove something for the kernel upgrade -->
rm -f /etc/rc.d/rocksconfig.d/pre-09-prep-kernel-source
<-- Reinstall our infinipath driver RPM -->
<!-- This adds the MPD disable-->
<file name="/etc/rc.d/rocksconfig.d/post-97-infinipath" mode="create
perms="a+rx">
#!/bin/sh
rpm -ivh --force \
/home/install/contrib/4.0.0/x86_64/RPMS/infinipath*.rpm
/etc/init.d/infinipath start
rm /etc/rc.d/rocksconfig.d/post-97-infinipath
</file>
<-- Add PathScale compiler library path to ld.so -->
<file name="/etc/ld.so.conf" mode="append">
/opt/pathscale/lib/2.4/
</file>
</post>
</kickstart>
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
36
Chapter 2: Software Installation
The important thing to note in this file is that the installation of the InfiniPath drivers is
done in the <post> section, as it is a "live" install. This file can be used as a guideline: it
may be cut and pasted, and then modified to suit your needs.
7. The completion of the installation is done using this xml file. Instructions for completing the install process are given under the Documentation link on the Rocks web site
at:
http://www.rocksclusters.org/Rocks/
2.6.3 Further information on Rocks and kickstart
Checking the Rocks website for updates is recommended. Extensive documentation on
installing Rocks and custom rolls is on the Rocks website. For more information on Rocks,
visit:
http://www.rocksclusters.org/
To find more information on Red Hat Enterprise Linux 4, and on using kickstart, see:
http://www.redhat.com/
2.7 InfiniPath and OpenFabrics driver overview
The ipath_core driver is responsible for InfiniPath adapter initialization, handling
interrupts for errors, ipath_ether networking, and OpenFabrics for PathScale, mediating
access to the interconnect for user programs, and handling memory mapping. It also provides support to layered drivers such as ipath_ether , ib_ipath, and support programs
such as the Subnet Management Agent (SMA). This diagram shows the relationship
between the InfiniPath and OpenFabrics software. Not all components are shown.
TCP/IP
ipath_ether
uses ipath_sma
for SMA
OpenFabrics
components
IPoIB
uses ib_mad for SMA
OpenSM
ib_ipath
InfiniPath
components
ipath_core
FIGURE 2-1.
Relationship between InfiniPath and OpenFabrics software
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
37
Chapter 2: Software Installation
As shown, ipath_core is the base driver for both the InfiniPath and OpenFabrics software components.
If you want to enable TCP-IP networking for running Ethernet traffic over the InfiniPath
link, you can configure the optional ipath_ether network interface files.
NOTE: It is not necessary to configure the ipath_ether driver to run MPI jobs.
Kernel module support for OpenFabrics is now built and installed as a part of the InfiniPath install. Use of OpenFabrics is optional. When using OpenFabrics you can configure
the IPoIB network interface.
However, In this release, it is not possible to successfully use the OpenFabrics IPoIB driver
and the InfiniPath ipath_ether driver simultaneously, because each driver requires a
different program for SMA services. The ipath_ether driver requires that the
ipath_sma program be used for SMA services, while the IPoIB driver requires that the
OpenFabrics ib_mad module be used for SMA services.
OpenSM is also an optional component of OpenFabrics.
2.8 Configuring the InfiniPath drivers
This section provides information on configuration for the InfiniPath drivers.
2.8.1 Configuring the ipath_core driver
The primary configuration file for the ipath_core driver, ipath_ether Ethernet
driver, the OpenFabrics for PathScale module ib_ipath, and the Subnet Management
Agent (SMA) and associated daemons is:
/etc/sysconfig/infinipath
This is where options to the driver, OpenFabrics for PathScale module, and the SMA are
provided. The INFINIPATH_NOSMA, INFINIPATH_NOMUX, and INFINIPATH_NODUMP
variables in /etc/sysconfig/infinipath are used to modify SMA behavior. The variable INFINIPATH_OPENIB is used to start OpenFabrics and is also used to modify SMA
behavior when using IPoIB. See 2.9.4 “Unloading OpenFabrics modules” for more information.
The device files are:
/dev/ipath
/dev/ipath_sma
NOTE: If you are upgrading, your existing configuration file will not be overwritten.
/dev/ipath0, /dev/ipath1, ...
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
38
Chapter 2: Software Installation
The numbered device files allow access to a specific InfiniPath unit.
Normally this configuration file is set up correctly at installation and the driver(s) are
loaded automatically during system boot once the RPMs have been installed. Assuming
that all the InfiniPath and OpenFabrics sofware has been installed, the default settings
upon startup will be:
•
•
•
•
•
InfiniPath ipath_core is enabled
InfiniPath user mode SMA is started (ipath_mux and ipath_sma)
InfiniPath ipath_ether is not running until configured
OpenFabrics ib_ipath and IPoIB are not running until configured
OpenSM is not enabled on startup. Enable it on the nodes(s) where it will be used as
subnet manager. OpenFabrics must be configured for it to work.
In the following sections, specific additional instructions for enabling different drivers and
modifying behaviors are presented.
2.8.2 Configuring the ipath_ether network interface
These instructions are for enabling TCP-IP networking over the InfiniPath link. To enable
IPoIB networking, see the section 2.9.2 “Configuring the IPoIB network interface”.
You must create a network device configuration file for the layered Ethernet device on the
InfiniPath HT-465 or PE-880. This configuration file will resemble the configuration files
for the other Ethernet devices on the nodes. Typically on Opteron servers there are two
Ethernet devices present, numbered as 0 (eth0) and 1 (eth1). This examples assumes we
create a third device, eth2 .
NOTE: When multiple InfiniPath chips are present, the configuration for eth3, eth4, and so
on follow the same format as for adding eth2 in the examples below.
Two slightly different procedures are given below for the ipath configuration; one for
Fedora and one for SUSE or SLES 10 (Beta).
Many of the entries that are used in the configuration directions below are explained in the
file sysconfig.txt. To familiarize yourself with these, please see:
/usr/share/doc/initscripts-*/sysconfig.txt
2.8.2.1 ipath_ether configuration on Fedora and RHEL4
These instructions are for the Fedora Core 3, Fedora Core 4, Fedora Core 5 and Red Hat Enterprise Linux 4 distributions.
1.
Check for the number of Ethernet drivers you currently have by either one of the two
following commands :
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
39
Chapter 2: Software Installation
$
ifconfig -a
$
ls /sys/class/net
As mentioned above we assume that two Ethernet devices (numbered 0 and 1) are already
present.
2.
Edit the file /etc/modprobe.conf (as root) by adding the following line:
alias eth2 ipath_ether
3.
The following file then needs to be created or edited if it already exists. You need to be
root to create or edit it.
/etc/sysconfig/network-scripts/ifcfg-eth2
If you are using DHCP (dynamic host configuration protocol), add the following lines
to ifcfg-eth2:
# PathScale Interconnect Ethernet
DEVICE=eth2
ONBOOT=yes
BOOTPROTO=dhcp
If you are using static IP addresses, use the following lines instead, substituting your
own IP address for the sample one given here.The normal matching netmask is shown.
# PathScale Interconnect Ethernet
DEVICE=eth2
BOOTPROTO=static
ONBOOT=YES
IPADDR=192.168.5.101
#Substitute your IP address here
NETMASK="255.255.255.0"
#Normal matching netmask
TYPE=Ethernet
This will cause the ipath_ether Ethernet driver to be loaded and configured during
system startup. To check your configuration, and make the ipath_ether Ethernet
driver available immediately, use the command (as root):
#
4.
/sbin/ifup eth2
Check whether the Ethernet driver has been loaded with:
$ lsmod | grep ipath_ether
5.
Verify that the driver is up with:
$ ifconfig -a
2.8.2.2 ipath_ether configuration on SUSE 9.3, SUSE 10.0, and SLES 10
(Beta)
These configuration steps will cause the ipath_ether network interfaces to be automatically
configured when you next reboot the system. The examples are shown assuming two
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
40
Chapter 2: Software Installation
onboard Ethernet interfaces are present (eth0 and eth1), and only one InfiniPath interface.
If there are 3 Ethernet interfaces, start with "eth3", rather than "eth2". If you have more
than one InfiniPath Ethernet interface to configure, add 1 for each, (eth2, eth3, etc.) and in
step 2, add one to the unit number, so replace .../00/guid with /01/guid for the second InfiniPath interface, and so on.
Step 3 is applicable only to SUSE 10; it is required because SUSE 10 uses a newer version of
the udev subsystem.
NOTE: The MAC address (media access control address) is a unique identifier attached to
most forms of networking equipment. Step 2 below determines the MAC address to
use, and will be referred to as $MAC in the subsequent steps. $MAC must be replaced
in each case with the string printed in step 2.
The following steps must all be executed as the root user.
1.
Be sure that the ipath_ether module is loaded:
# lsmod | grep -q ipath_ether || modprobe ipath_ether
2.
Determine the MAC address that will be used:
# sed ’s/^\(..:..:..\):..:../\1/’ \
/sys/bus/pci/drivers/ipath_core/00/guid
The output should appear similar to this (6 hex digit pairs, separated by colons):
00:11:75:04:e0:11
The GUID can also be returned by running:
$
ipath_control -i
$Id: PathScale Release1.3Beta2 $ $Date: 2006-05-02-02:17 $
00: Version: Driver 2.0, InfiniPath_HT-460, InfiniPath1 3.2,
PCI 2, SW Compat 2
00: Status: 0xf1 Initted SMA Present IB_link_up IB_configured
00: LID=0x5e MLID=0xc000 GUID=00:11:75:00:00:04:e0:11 Serial:
1125250053
Note that removing the middle two 00:00 octets from the GUID in the above output
will form the MAC address
NOTE: If either step 1 or step 2 fails in some fashion, the problem must be found and corrected before continuing. Verify that the rpms are installed correctly, and that infinipath has correctly been started. If problems continue, run ipathbug-helper and
report the results to your reseller or InfiniPath support organization.
3.
Skip to Step 4 if you are using SUSE 9.3. This step is only done on SUSE 10 or SLES 10
(Beta) systems. Edit the file:
/etc/udev/rules.d/30-net_persistent_names.rules
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
41
Chapter 2: Software Installation
(If this file does not exist, this step should be skipped.)
Check each of the lines starting with SUBSYSTEM=, to find the highest numbered interface. (For standard motherboards, the highest numbered interface will typically be 1.)
Add a new line at the end of the file, with the interface name at the end of the line having a number 1 larger than the highest numbered interface already present. If "eth1"
was highest, use "eth2" (as used in this example). The new line to add will look like
this:
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="$MAC",
IMPORT="/sbin/ rename_netiface %k eth2"
This will appear as a single line in the file. $MAC is replaced by the string from step 2
above.
4. Create the network module file:
/etc/sysconfig/hardware/hwcfg-eth-id-$MAC
with the following lines:
MODULE=ipath_ether
STARTMODE=auto
This will cause the ipath_ether Ethernet driver to be loaded and configured during
system startup.
NOTE: Choose step 5 if you are using DHCP, or step 6 if you are using static IP addresses.
5.
If you are using DHCP (dynamically assigned IP addresses), create the network configuration file:
/etc/sysconfig/network/ifcfg-eth2
with the contents:
STARTMODE=onboot
BOOTPROTO=dhcp
NAME=’InfiniPath Network Card’
_nm_name=eth-id-$MAC
Proceed to step 7.
6. If you are NOT using DHCP (you are using static IP addresses), create the network configuration file:
/etc/sysconfig/network/ifcfg-eth2
with the following contents:
STARTMODE=onboot
BOOTPROTO=static
NAME=’InfiniPath Network Card’
NETWORK=192.168.5.0
NETMASK=255.255.255.0
BROADCAST=192.168.5.255
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
42
Chapter 2: Software Installation
IPADDR=192.168.5.211
_nm_name=eth-id-$MAC
Make sure that you substitute your own IP address for the sample IPADDR shown here.
The BROADCAST, NETMASK, and NETWORK lines need to match for your network.
7. To verify that the configuration files are correct, you will normally now be able to run
the commands:
# ifup eth2
# ifconfig eth2
In some cases, however, it may be necessary to reboot the system before the configuration
changes will work.
2.9 OpenFabrics configuration and startup
Startup of OpenFabrics follows these steps:
•
•
•
•
Modify the /etc/sysconfig/infinipath file
Configure the IPoIB network interface
Shut down ipath_ether,if running, before restarting
Restart InfiniPath
These steps are explained in greater detail below. Use of OpenSM will be covered in a later
section.
NOTE: The following instructions work for both FC4 and SUSE 10.0, and SLES 10 (Beta)
2.9.1 Modify /etc/sysconfig/infinipath
Control of the OpenFabrics modules is through the same file as for the InfiniPath software.
To start OpenFabrics modify this file
/etc/sysconfig/infinipath
by uncommenting this line:
INFINIPATH_OPENIB=1
By setting this value to 1, this also sets INFINIPATH_NOMUX, INFINIPATH_NOSMA,
and INFINIPATH_NODUMP. This means that the user mode SMA ipath_mux and
ipath_sma will not be used, and therefore ipath_ether can not be used, as it requires
these for SMA services. This will enable IPoIB, and the OpenFabrics ib_mad module will
be used for SMA services, once InfiniPath has been restarted.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
43
Chapter 2: Software Installation
Note that the following config file is modified during a fresh install:
/etc/modprobe.conf ( kernel module loader configuration)
It is not modified if the packages are removed.
2.9.2 Configuring the IPoIB network interface
At this time, automatic configuration of networking over OpenFabrics using the IPoIB
mechanism is not reliable, so instructions are given here to manually configure your OpenFabrics IPoIB network interface. This example assumes that you are using sh or bash as
your shell, and that all required InfiniPath and PathScale OpenFabrics RPMs are installed,
and your startup scripts have been run, either manually or at system boot.
For this example, we assume that your IPoIB network is 10.1.17.0 (one of the networks
reserved for private use, and thus not routable on the internet), with a /8 host portion, and
therefore requires that the netmask be specified.
This example assumes that no hosts files exist, and that the host being configured has the
IP address 10.1.17.3, and that DHCP is not being used. Type the following commands (as
root):
# ifconfig ib0 10.1.17.3 netmask 0xffffff00
To verify the configuration, type:
# ifconfig ib0
The output from this command should be similar to this:
ib0
Link encap:InfiniBand HWaddr
00:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:10.1.17.3 Bcast:10.1.17.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:128
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Next, type:
# ping -c 2 -b 10.1.17.255
The output of the ping command should be similar to that below, with a line for each host
already configured and connected:
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
44
Chapter 2: Software Installation
WARNING: pinging broadcast address
PING 10.1.17.255 (10.1.17.255) 517(84) bytes of data.
174 bytes from 10.1.17.3: icmp_seq=0 ttl=174 time=0.022 ms
64 bytes from 10.1.17.1: icmp_seq=0 ttl=64 time=0.070 ms (DUP!)
64 bytes from 10.1.17.7: icmp_seq=0 ttl=64 time=0.073 ms (DUP!)
2.9.3 Enabling and restarting
After installing the OpenFabrics RPMs, enabling the kernel modules, and configuring the
IPoIB network interface as mentioned above, you must restart infinipath on each of the
nodes on which you installed.
A standard script is provided to start, stop, or restart the driver and associated daemons.
You must be root to run it. General usage is as follows:
# /etc/init.d/infinipath [start|stop|restart]
This method will not reboot the system. The following set of commands shows how this
script can be used. You can omit the commands to start/stop opensm if you are not running it on that node.
# /etc/init.d/opensm stop
# /etc/init.d/infinipath restart
# /etc/init.d/opensm start
In the case where the ipath_ether interface is already configured, then you will need to
shutdown the ipath_ether interface manually before restarting. For example, if
ipath_ether is configured as eth2, use this sequence of commands:
# ifdown eth2
# /etc/init.d/infinipath restart
NOTE: Restarting InfiniPath will terminate any InfiniPath MPI processes, as well as any
OpenFabrics processes that are running at the time. Processes using networking over
ipath_ether will return errors.
An alternative method is to reboot on each compute node on which you installed. To
reboot, type the following (as root):
# /sbin/reboot
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
45
Chapter 2: Software Installation
2.9.4 Unloading OpenFabrics modules
To unload the OpenFabrics modules, comment out this line in /etc/sysconfig/
infinipath.
#INFINIPATH_OPENIB=1
and restart InfiniPath by running (as root):
# /etc/init.d/infinipath restart
If you want to restart ipath_ether (assuming it was previously configured) type:
# ifup eth2
The ipath_mux and ipath_sma programs will be restarted, and the ipath_ether link
is brought back up.
2.10 OpenSM
OpenSM is an optional component of the OpenFabrics project that provides a subnet manager for InfiniBand networks. This package can be installed on all machines, but only
needs to be enabled on the machine in your cluster that is going to act as a subnet manager.
You do not need to use OpenSM if any of your InfiniBand switches provide a subnet manager.
It requires that OpenFabrics is configured, as decribed in 2.9 “OpenFabrics configuration
and startup”. After installing the opensm package, OpenSM is configured to be off on the
next machine reboot. You can use the chkconfig command (as root) to enable it:
# chkconfig opensm on
You can start opensm without rebooting your machine as follows:
# /etc/init.d/opensm start
and you can stop it again like this:
# /etc/init.d/opensm stop
If you wish to pass any arguments to the OpenSM program, modify the file:
/etc/sysconfig/opensm
and add the arguments to the "OPTIONS" variable. Here is an example:
# Use the UPDN algorithm instead of the Min Hop algorithm.
OPTIONS="-u"
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
46
Chapter 2: Software Installation
2.11 Testing the drivers
Normally, the driver(s) are loaded automatically during system boot once the RPMs have
been installed. However, there may be times when the state of the drivers needs to be controlled manually for testing purposes. The following commands can be used to check or
configure state.
To check the configuration state, use the command:
$ chkconfig --list infinipath
To enable the driver, use the command (as root):
# chkconfig infinipath on 2345
To disable the driver on the next system boot, use the command (as root):
# chkconfig infinipath off
NOTE: This does not stop and unload the driver, if it is already loaded.
A standard script is provided to start, stop, or restart the driver and associated daemons.
To use this script, type (as root):
# /etc/init.d/infinipath [start|stop|restart]
To prevent all use of InfiniPath and unload the driver:
# /etc/init.d/infinipath stop
NOTE: This will not prevent the driver from loading on the next reboot.
If you need to find which InfiniPath and OpenFabrics modules are running, try the following command:
$ lsmod | egrep ’ipath_|ib_|rdma_|findex’
2.12 Recompiling the drivers
If you upgrade the kernel then you must reboot the system. Then you either have to reinstall the InfiniPath software or recompile the kernel modules (drivers). To recompile the
kernel modules, first reboot, then type the following (as root):
# cd /usr/src/pathscale/drivers
# ./make-install.sh
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
47
Chapter 2: Software Installation
2.13 Further information on configuring and loading
drivers
See the modprobe(8), modprobe.conf(5), and lsmod(8), man pages for more
information. Also see the file /usr/share/doc/initscripts-*/sysconfig.txt for
more general information on configuration files.
These topics are also covered in more detail in the PathScale InfiniPath Interconnect User’s
Guide, in Chapter 2 “InfiniPath Cluster Administration”.
2.14 Compiler support
Presently, in addition to the PathScale EKOPath Compiler Suite, PathScale MPI supports a
number of other compilers. These include PGI 5.2 and 6.0, Intel 9.0, the GNU gcc 3.3.x,
3.4.x, and 4.0.x compiler suites and gfortran.
The PathScale EKOPath Compiler Suite is not yet supported on Fedora Core 4, Fedora
Core 5, SUSE 10, or SLES 10 RC, all systems that have the GNU 4.x compilers and compiler
environment (header files and libraries).
However, it is possible to cross-compile on a distribution that supports the PathScale compiler, such as FC3 or SUSE 9.3, and then run on one of these other distributions that uses
the GNU 4.x compiler and environment. For more information on cross-compilation, see
3.4.4 “Cross-compilation issues”in the PathScale InfiniPath Interconnect User’s Guide.
In addition, gfortran is not currently supported on Fedora Core 3.
Please check the PathScale website for updated information on supported compilers.
2.15 Switch configuration and monitoring
Follow the vendor documentation for installing and configuring your switches. A list of
supported and tested switches is in the PathScale InfiniPath Interconnect User’s Guide.
2.16 LED link and data indicators
The LEDs function as link and data indicators once the InfiniPath software has been
installed, the driver has been loaded, and the fabric is being actively managed by a Subnet
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
48
Chapter 2: Software Installation
Manager. The following table shows the possible states of the LEDs. The green LED will
normally illuminate first. The normal state is Green On, Amber On.
TABLE 2-4.
LED
On state
Indicates Activity
Off state
Indicates possible problems
Green
Signal detected.
Software not installed or started.
Ready to talk to an SM to bring
link fully up.
Loss of signal. Check cabling.
Link configured.
SM may be missing.
Properly connected and ready to
receive data packets and link
packets.
Link may not be configured.
Amber
Switch not powered up.
Check the connection.
2.17 Customer acceptance utility
ipath_checkout is a bash script used to verify that the installation is correct and that all
the nodes of the network are functioning and mutually connected by the InfiniPath fabric.
It is to be run on a front end node, and requires specification of a hosts file:
$ ipath_checkout [options] hostsfile
where hostsfile designates a file listing the hostnames of the nodes of the cluster, one
hostname per line. The format of hostsfile is as follows:
hostname1
hostname2
...
ipath_checkout performs the following seven tests on the cluster:
1. ping all nodes to verify all are reachable from the frontend.
2. ssh to each node to verify correct configuration of ssh.
3.
4.
5.
6.
7.
Gather and analyze system configuration from nodes.
Gather and analyze RPMs installed on nodes.
Verify InfiniPath hardware and software status and configuration.
Verify ability to mpirun jobs on nodes.
Run bandwidth and latency test on every pair of nodes and analyze results.
The possible options to ipath_checkout are:
-h, --help
Displays help messages giving defined usage.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
49
Chapter 2: Software Installation
-v, --verbose
-vv, --vverbose
-vvv, --vvverbose
These specify three successively higher levels of detail in reporting results of tests. So,
there are four levels of detail in all, including the case of where none these options are
given.
-c, --continue
When not specified, the test terminates when any test fails. When specified, the tests
continue after a failure, with failing nodes excluded from subsequent tests.
--workdir=DIR
Use DIR to hold intermediate files created while running tests. DIR must not already
exist.
-k, --keep
Keep intermediate files that were created while performing tests and compiling reports.
Results will be saved in a directory named pathscale_* or the directory name given
to --workdir.
--skip=LIST
Skip the tests in LIST (e.g. --skip=2457 will skip tests 2, 4, 5, and 7)
In most cases of failure, the script suggests recommended actions. Please see the
ipath_checkout man page for further information and updates. Also refer to Appendix C “Troubleshooting” in the PathScale InfiniPath Interconnect User’s Guide.
2.18 Removing software packages
Instructions for uninstalling or downgrading InfiniPath and OpenFabrics software is given
below.
To uninstall the InfiniPath software packages on any node, using a bash shell, type the
command (as root):
# rpm -e $(rpm -qa ’mpi*’ ’infinipath’)
This will uninstall the InfiniPath software RPMs on that node.
To uninstall the OpenFabrics software packages on any node, using a bash shell, type the
command (as root):
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
50
Chapter 2: Software Installation
# rpm -e rpm_name_pre
where rpm_name_pre is the descriptive name that precedes the version and repository
identifiers in an RPM. For example:
# rpm -e libibverbs
would uninstall libibverbs-1.0-1_100.77_yyy_psc.x86_64.rpm on that node.
For both InfiniPath and OpenFabrics it is recommended that you remove all the packages
at the same time.
2.18.1 Downgrading
If you want to downgrade, please remove both the InfiniPath and OpenFabrics RPMs, then
install the older bits. We have found that rpm flags like "--oldpackage" will not generate
a correct downgrade.
2.19 Installing newer drivers from other distributions
In this release, the InfiniPath RPM was split into infinipath and infinipath-kernel. The driver source now resides in infinipath-kernel. This means that newer
drivers can be installed as they become available. Those who wish to install newer drivers,
for example, from OFED (Open Fabrics Enterprise Distribution), should be able to do so.
However, some extra steps need to be taken in order to install properly.
1.
Install all InfiniPath RPMs, including infinipath-kernel. infinipath-kernel
installs into:
/lib/modules/$(uname -r)/updates
This should not affect any other installed InfiniPath or OpenFabrics drivers.
2. Reload the InfiniPath and, if used, OpenFabrics modules to verify that the installation
works. If using the OpenFabrics modules, first modify this file:
/etc/sysconfig/infinipath
Uncomment this line:
INFINIPATH_OPENIB=1
Reload all modules by using this command (as root):
# /etc/init.d/infinipath restart
3.
Run ipath_checkout or other OpenFabrics test program to verify that the InfiniPath
card(s) work properly.
4. Unload the InfiniPath (and OpenFabrics, if in use) modules with the command:
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
51
Chapter 2: Software Installation
# /etc/init.d/infinipath stop
5.
Remove the InfiniPath kernel components with the command:
$ rpm -e infinipath-kernel --nodeps
The option --nodeps is required because the other InfiniPath RPMs depend on
infinipath-kernel.
6. Verify that no InfiniPath or OpenFabrics modules are present in the /lib/modules/
$(uname -r)/updates directory.
7. If not yet installed, install the InfiniPath and OpenFabrics modules from your alternate
set of RPMs.
8. Reload all modules by using this command (as root):
# /etc/init.d/infinipath start
An alternate mechanism can be used, if provided as part of your alternate installation.
9. Run ipath_checkout or other OpenFabrics test programs to verify that your InfiniPath card(s) work correctly.
This information is repeated in Appendix C “Troubleshooting” in the PathScale InfiniPath
Interconnect User’s Guide.
2.20 Installation troubleshooting
If you have any difficulties installing the hardware or software, please refer to Appendix C
“Troubleshooting” in the PathScale InfiniPath Interconnect User’s Guide. This Troubleshooting section contains information on problems related to installation, system administration, and PathScale MPI issues.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
52
APPENDIX A
RPM Descriptions
The following sections show more detailed descriptions of the RPMs for InfiniPath and
OpenFabrics.
A.1 InfiniPath RPMs
For ease of installation, it is recommended that all RPMs are installed on all nodes. However, some RPMs are optional. Since cluster nodes may be used for different functions, it is
possible to selectively install on the node which performs a certain function. If you want to
selectively install the RPMs, see the following tables for a comparison of required and
optional packages.
A.1.1 Different nodes may use different RPMs
In a cluster environment different nodes may be used for different functions, such as
launching jobs, software development, or running jobs. We define these nodes in the following way:
• Front end node. This node will launch jobs. It is referred to as the front end node in this
documentation.
• Compute node. These are the nodes that are used to run jobs.
• A development or build node. These are the machines on which examples or benchmarks can be compiled.
Any machine can serve any combination of these three purposes, but a typical cluster has
many compute nodes and only one or just a few front end nodes. The number of nodes
used for development will vary. Although PathScale recommends installing all RPMs on all
nodes, not all InfiniPath software is required on all nodes. See Table A-1 for information
on installation of software RPMs on specific types of nodes.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
53
Chapter :
TABLE A-1. Optional or required InfiniPath RPMs for each type of node
RPM
Front end
Compute
Development
infinipath-kernel-1.3-xxx_yyy_x86_64.rpm
Optional
Required
Optional
Optional
Required
Optional
Optional
Optional
Optional
Optional
Required
Optional
Optional
Required*
Optional
Optional
Optional
Required
Optional
Optional
Optional
Required
Required**
Optional
Optional
Required
Required
InfiniPath drivers,OpenFabrics kernel modules
infinipath-1.3-xxx_yyy_x86_64.rpm
Utilities and source code
InfiniPath configuration files
infinipath-doc-1.3-xxx_yyy_noarch.rpm
InfiniPath man pages and other documents
infinipath-libs-1.3-xxx_yyy_i386.rpm
InfiniPath protocol shared libraries for 32-bit and
64-bit systems
mpi-benchmark-1.3-xxx_yyy_x86_64.rpm
MPI benchmark binaries
mpi-devel-1.3-xxx_yyy_noarch.rpm
Source code for the MPI development environment, including headers and libs MPI examples
and benchmarks. Use to build the examples or
rebuild the benchmarks.
mpi-doc-1.3-xxx_yyy_noarch.rpm
Man pages for MPI functions and other MPI documents
mpi-frontend-1.3-xxx_yyy_i386.rpm
MPI job launch scripts and binaries, including
mpirun and MPD
mpi-libs-1.3-xxx_yyy_i386.rpm
Shared libraries for MPI
*Required if using ipath_checkout. Otherwise, installation is optional.
**Required on nodes that run mpirun or MPD. Otherwise installation is optional.
A.2 OpenFabrics RPMs
For ease of installation, it is recommended that all the OpenFabrics RPMs are installed on
all nodes. However, some components, such as OpenSM, are optional. If you choose to
selectively install, be sure to install the optional packages on all the cluster nodes.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
54
Chapter :
TABLE A-2. RPMs for OpenFabrics and OpenSM
RPM name
Comment
libdat-1.2.0-1.2.x86_64.rpm
Optional
uDAPL support
Optional
libdat-devel-1.2.0-1.2.x86_64.rpm
Development files for uDAPL support
Optional
libibat-0.9.0-1.rc4.yyy.x86_64.rpm
Along with the OpenFabrics kernel drivers, libibat provides a userspace InfiniBand Address Translation API
libibat-devel-0.9.0-1.rc4.yyy.x86_64.rpm
Development files for the libibat library
Optional
libibat-utils-0.9.0-1.rc4.1.yyy.x86_64.rpm
Optional
Utilities for the libibat library
Optional *
libibcm-0.9.0-1.rc4.yyy.x86_64.rpm
Along with the OpenFabrics kernel drivers, libibcm provides a userspace InfiniBand Connection Managment API
libibcm-devel-0.9.0-1.rc4.yyy.x86_64.rpm
Optional
Development files for the libibcm library
libibcommon-1.0-1.rc4.yyy.x86_64.rpm
Required for OpenSM
Common utility functions for the IB diagnostic and management
tools
libibcommon-devel-1.0-1.rc4.yyy.x86_64.rpm
Optional*
Development files for the libibcommon library
libibmad-1.0-1.rc4.yyy.x86_64.rpm
Low layer IB functions for use by the IB diagnostic and management
programs. These include MAD, SA, SMP, and other basic IB functions.
Required for OpenSM
libibmad-devel-1.0-1.rc4.yyy.x86_64.rpm
Development files for the libibmad library
Optional
libibumad-1.0-1.rc4.yyy.x86_64.rpm
Required for OpenSM
Provides the user MAD library functions which sit on top of the user
MAD modules in the kernel. These are used by the IB diagnostic and
management tools, including OpenSM
libibumad-devel-1.0-1.rc4.yyy.x86_64.rpm
Optional*
Development files for the libibumad library
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
55
Chapter :
TABLE A-2. RPMs for OpenFabrics and OpenSM
RPM name
Comment
libibverbs-1.0-1.rc4.yyy.x86_64.rpm
Required
Library that allows userspace processes to use InfiniBand “verbs” as
described in the InfiniBand Architecture Specification. This includes
dierct hardware access for fast path operations. For this library to be
usefule, a device-specfic plug-in module should also be installed.
libibverbs-devel-1.0-1.rc4.yyy.x86_64.rpm
Optional**
Static libraries and header files for the libibverbs verbs library
libibverbs-utils-1.0-1.rc4.yyy.x86_64.rpm
Required
Useful libibverbs example programs such as ibv_devinfo, which displays information about InfiniBand devices.
libipathverbs-1.0-1.rc4.yyy.x86_64.rpm
Required
Static libraries and header files for the libibverbs verbs library
libipathverbs-devel-1.0-1.rc4.yyy.x86_64.rpm
Optional
Static libraries and header files for the libibverbs verbs
library
Optional
libmthca-1.0-1.rc4.yyy.x86_64.rpm
Provides a device-specific userspace driver for Mellanox HCAs
(MT23108 InfiniHost and MT25208 InfiniHost III Ex) for use with
the libibverbs library
libmthca-devel-1.0-1.rc4.yyy.x86_64.rpm
Optional
Static version of libmthca that may be linked directly to an application, which may be useful for debugging
librdmacm-0.9.0-2.rc4.yyy.x86_64.rpm
Optional
Support for the new connection manager
librdmacm-devel-0.9.0-2.rc4.yyy.x86_64.rpm
Optional
Development files for the new connection manager.
librdmacm-utils-0.9.0-2.rc4.yyy.x86_64.rpm
Optional
Utilities for the new connection manager
Required
libsdp-0.9.0-1.rc4.yyy.x86_64.rpm
Can be LD_PRELOAD-ed to have a sockets application use InfiniBand Sockets Direct Protocol (SDP) instead of TCP, transparently and
without recompiling the application
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
56
Chapter :
TABLE A-2. RPMs for OpenFabrics and OpenSM
RPM name
Comment
openib-diags-1.0-1.rc4.yyy.x86_64.rpm
Useful programs for troubleshooting and checking state.
Optional
opensm-1.2.0-1.rc4.yyy.x86_64.rpm
Required for OpenSM
OpenSM provides an implementation of an InfiniBand Subnet Manager and Administrator. Such a software entity is required to run for
in order to initialize the InfiniBand hardware (at least one per each
InfiniBand subnet).
opensm-devel-1.2.0-1.rc4.yyy.x86_64.rpm
Optional
srptools-0.0.4-2.rc4.yyy.x86_64.rpm
Optional
Support for SRP
*libibcm-utils-*, the utilities for the libibcm library, are no longer installed. Checking
for dependencies on this RPM is advised.
**Only needed if compiling software against the packages.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
57
Regulatory Information
APPENDIX B
United States
This equipment has been tested and found to comply with the limits for a Class A digital
device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy
and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area
is likely to cause harmful interference in which case the user will be required to correct the
interference at his own expense.
Any modification to this unit not expressly approved by PathScale could void the user’s
authority to operate the equipment.
Europe
Warning - This is a Class A product. In a domestic environment this product may cause
radio interference in which case the user may be required to take adequate measures.
Japan
This is a Class A product based on the standard of the Voluntary Control Council For
Interference by Information Technology Equipment (VCCI). If this equipment is used in a
domestic environment, radio disturbance may arise. When such trouble occurs, the user
may be required to take corrective actions.
Canada
This Class A digital apparatus complies with Canadian ICES-003.
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
58
Index
B
BIOS, configuring the, 11
C
cables, qualified, 24
compiler support, 48
configuration
ipath_core, 38
ipath_ether on Fedora and RHEL4, 39
ipath_ether on SUSE and SLES 10, 40
IPoIB, 44
OpenFabrics, 43
OpenSM, 46
D
distributions, supported for InfiniPath and
OpenFabrics, 28
driver testing, 47
OpenSM, 46
R
restarting InfiniPath and OpenFabrics, 45
RPMs
downloading, 31
list of InfiniPath, 53
list of OpenFabrics, 54
S
safety precautions, 12
switch configuration, 48
T
troubleshooting, 52
U
uninstalling, 50
F
front matter
intended audience for this guide, 7
list of InfiniPath documents 8
organization of this guide, 7
typographic conventions used in this guide, 8
url for online documentation 8
I
InfiniPath software, list of, 26
installation
for HTX adapter hardware, 20–24
for PCIe adapter hardware, 15–19
hardware requirements for, 9–11
InfiniPath and OpenFabrics software, 31
overview of software, 27
restarting after, 45
see also RPMs,
software requirements for, 11
using Rocks for, 33–37
verification of, see ipath_checkout,
ipath_checkout 49
ipath_ether, incompatibility with IPoIB, 38
L
LEDs, states indicated by, 48
Linux environment, see distributions, supported for
InfiniPath and OpenFabrics,
O
OpenFabrics
supported libraries,interfaces,protocols,and
programs, 26
PathScale InfiniPath Interconnect Installation Guide
Version 1.3
59
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising