Mellanox UFM Unified Fabric Manager User Manual
Unified Fabric Manager UFM is a network management software that provides a central console for managing InfiniBand fabrics. It offers in-depth fabric visibility and control, enabling multiple isolated application environments on a shared fabric. UFM also provides service-oriented automatic resource provisioning, and helps to quickly resolve fabric problems.
Advertisement
Advertisement
Unified Fabric Manager for
InfiniBand
User Manual
Version 5.2
www.mellanox.com
Mellanox Technologies Confidential
Version 5.2 Revision History
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE
THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT
HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S)
AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT
GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED.
IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT
LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: (408) 970-3400
Fax: (408) 970-3403
Mellanox Technologies, Ltd.
Hakidma 26
Ofer Industrial Park
Yokneam 2069200
Israel www.mellanox.com
Tel: +972 (0)74 723 7200
Fax: +972 (0)4 959 3245
© Copyright 2015. Mellanox Technologies. All Rights Reserved.
Mellanox®, Mellanox logo, BridgeX®, ConnectX®, Connect-IB®, CoolBox®, CORE-Direct®, GPUDirect®, InfiniBridge®,
InfiniHost®, InfiniScale®, Kotura®, Kotura logo, Mellanox Connect. Accelerate. Outperform logo, Mellanox Federal
Systems® Mellanox Open Ethernet®, MetroX®, MLNX-OS®, Open Ethernet logo, PhyX®, ScalableHPC®, SwitchX®,
TestX®, The Generation of Open Ethernet logo, UFM®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are registered trademarks of Mellanox Technologies, Ltd.
CyPU™, ExtendX™, FabricIT™, FPGADirect™, HPC-X™, Mellanox Care™, Mellanox CloudX™, Mellanox NEO™,
Mellanox Open Ethernet™, Mellanox PeerDirect™, Mellanox Virtual Modular Switch™, MetroDX™, NVMeDirect™,
StPU™, Switch-IB™, Unbreakable-Link™ are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
2
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Contents
Enables Multiple Isolated Application Environments on a Shared Fabric .............. 23
Service-Oriented Automatic Resource Provisioning .............................................. 23
3
Mellanox Technologies Confidential
Version 5.2 Revision History
Additional Prerequisites for UFM High Availability (HA) Installation....................... 34
Installing the UFM Server Software as Standalone ................................................ 39
Installing the UFM Server Software for High Availability ........................................ 40
Configuring UFM HA with Additional External UFM-Subnet Managers ................. 48
Running the UFM Server Software in Management Mode ..................................... 51
Running the UFM Software in High Availability Mode ............................................ 52
Running the UFM Software in Monitoring Mode..................................................... 52
Launching a UFM GUI Session for Apple MacBook .............................................. 63
4
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Extensible Object Model (Advanced License Only) ............................................... 78
Viewing, Designing, and Managing Devices Windows ........................................... 83
5
Mellanox Technologies Confidential
Version 5.2 Revision History
Expected Outcome - Assigning a Logical Server ................................................. 114
Expected Outcome - Assigning a Network or Network Interface ......................... 116
Expected Outcome - Assigning a Logical Gateway Group .................................. 119
Setting/Editing Logical Gateway Group Properties .............................................. 119
6
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Connection Congestion Tab (Advanced License Only) ........................................ 150
Running SSH Commands on a Device (Advanced License only) ........................ 159
Setting User-Defined Attributes (Advanced License Only) ................................... 160
Running Saved Commands (Advanced License only) ......................................... 162
Releasing a Computer (Server) from a Logical Server ........................................ 164
7
Mellanox Technologies Confidential
Version 5.2 Revision History
Adding and Removing Ports for a Mirroring Session ........................................... 175
Torus-2QoS and Traffic-Aware Torus Routing ..................................................... 189
Event-Triggered Scripts (Advanced License Only) .............................................. 197
Resetting and Disabling a Specific Non-optimal Link ........................................... 203
Example of Congestion Control Manager Options File ........................................ 204
Example of Lossy Configuration Manager Options File ....................................... 209
8
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Saving a Monitoring Session Template (Advanced Licensed Only) .................... 213
Running a Monitoring Session from a Template (Advanced License Only) ......... 219
9
Mellanox Technologies Confidential
Version 5.2 Revision History
10
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
UFM Server Communication with the UFM GUI Client ........................................ 297
UFM Server Communication with SNMP Trap Managers .................................... 297
Summary of UFM Server Communication with Clients ........................................ 298
UFM Server InfiniBand Communication with the Switch ...................................... 298
UFM Server Communication with Switch Management Software (Optional) ....... 298
Summary of UFM Server Communication with InfiniBand Switches .................... 299
UFM Server InfiniBand Communication with HCAs ............................................. 300
UFM Server Communication with Host Management (Optional) ......................... 300
Summary of UFM Server Communication with InfiniBand Hosts ......................... 301
UFM Server High Availability (HA) Active – Standby Communication ............................... 301
UFM Server HA Active – Standby Communication .............................................. 301
UFM Subnet Manager Default Properties ......................................................... 304
11
Mellanox Technologies Confidential
Version 5.2 Revision History
Adaptive Routing Manager Options File Example ................................................ 373
12
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
List of Figures
13
Mellanox Technologies Confidential
Version 5.2 Revision History
14
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
List of Tables
15
Mellanox Technologies Confidential
Version 5.2 Revision History
16
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Revision History
Table 1: Document Revision History
Release Date
5.2
5.1
5.0
4.9
August 2015
April 2015
Description
•
Added the following section:
•
•
•
Updated the following sections:
•
•
•
•
•
Other Daily Report Configurations
UFM Subnet Manager Default Properties
E.2.2: Utilities Descriptions (ibdiagnet and ibdiagpath)
•
Updated the following sections:
•
Appendix B: UFM Subnet Manager Default Properties
January 2015
•
Added the following sections:
•
•
Updated the following sections:
•
•
•
•
•
•
•
•
•
Table 47: SM Limits Configuration
Other Daily Report Configurations
Mellanox Devices Supported by UFM
Appendix B: UFM Subnet Manager Default Properties
E.2.2: Utilities Descriptions (ibdiagnet and ibdiagpath)
October 2014
•
Added the following sections:
•
•
Daily Report and its subsections
•
•
Updated the following sections:
•
•
•
•
•
•
•
•
•
•
•
Configuring Identification Settings
Setting/Editing QoS (Optional)
Editing Network Interface QoS Properties
Configuring Routing and Quality of Service
Version 5.2
17
Mellanox Technologies Confidential
Version 5.2
18
Release Date
Rev 4.8 August 2014
Rev 4.7
Rev
4.6.1
April 2014
December
2013
Rev 4.5 August 2013
Description
•
•
Table 57: Policy Table Options
Appendix B: UFM Subnet Manager Default Properties
•
F.1: Device Management Feature Support for InfiniBand
•
Removed the following sections:
•
•
•
Application Performance Optimization
Optimizes Performance via End-to-End QoS Management
Traffic Aware Routing Algorithm –TARA
•
Added the following sections:
•
•
Updated the following sections:
•
•
•
User Account Management (Advanced License Only)
•
Updated the following sections:
•
•
•
•
•
UFM with Remote Fabric Collector
Configuring UFM for SR-IOV (Advanced License Only)
Prerequisites for UFM Server Software Installation
•
UFM Subnet Manager Default Properties
•
•
Mellanox Devices Supported by UFM
•
Added the following sections:
Differences Between UFM Licenses
•
Removed the following sections
•
Splitting and Merging Layers
•
Vcollect Client Utility
•
Added the following sections:
•
•
•
UFM with Remote Fabric Collector
UFM Server as a Virtual Appliance
Installing Fabric Collector on a Management Host in the Fabric
Configuring The Fabric Collector
•
•
•
Recovery from Consecutive Failures
Routing Chains and its subsections
•
Updated the following sections:
•
•
•
•
•
•
•
•
Configuring UFM Server for Monitoring History Database
Configuring Monitoring History Database
UFM.MIB file location
Configuring General Settings in gv.cfg
Manage Devices Tabs - IB
Revision History
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Release Date Description
•
•
•
•
•
•
•
•
•
•
•
•
UFM Subnet Manager Default Properties
Commands for InfiniBand Diagnostics
Configuring Site Access Credentials
•
•
Added the following sections:
•
•
•
•
•
•
•
Monitoring History Data Migration
Enhanced Event Management (Advanced License Only)
Isolating a Switch from Routing
Restoring Monitoring History Configuration
Rev 4.0 February 2013
•
Added the following sections:
•
•
•
•
•
•
•
Viewing a Monitoring History Session
Preventing Disk Overutilization
Monitoring History Database System Requirements
Initializing Monitoring History Database
Monitoring History Session Display
•
•
Monitoring History Session Graphical Representation
Monitoring History Window Toolbar
•
Updated the following sections:
•
•
•
Launching a UFM GUI Session [UM3.8 up]
Launching a UFM GUI Session for Apple MacBook
Version 5.2
19
Mellanox Technologies Confidential
Version 5.2 About This Manual
About This Manual
This manual is intended for cluster and data center administrators who are responsible for the deployment, configuration, and day-to-day maintenance of the fabric.
Audience
This manual is intended for cluster and data center administrators who are responsible for the deployment, configuration, and day-to-day maintenance of the fabric.
Document Conventions
The following lists conventions used in this document.
NOTE: Identifies important information that contains helpful suggestions.
CAUTION: Alerts you to the risk of personal injury, system damage, or loss of data.
WARNING: Warns you that failure to take or avoid a specific action might result in personal injury or a malfunction of the hardware or software. Be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents before you work on any equipment.
20
Typography
The following table describes typographical conventions in Mellanox documentation. All terms refer to isolated terms within body text or regular table text unless otherwise mentioned in the Notes column.
Example Notes Term, Construct,
Text Block
File name, pathname
Console session (code)
/opt/ufm/conf/gv.cfg
-> flashClear <CR>
Linux shell prompt #
Mellanox CLI Guest Mode Switch >
Complete sample line or block.
Comprises both input and output.
The code can also be shaded.
The "#"character stands for the
Linux shell prompt.
Mellanox CLI Guest Mode.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Term, Construct,
Text Block
Example
Mellanox CLI admin mode
Switch #
String < > or []
Management GUI label, item name
New Network,
New Environment
Notes
Mellanox CLI admin mode
Strings in < > or [ ] are descriptions of what will actually be shown on the screen, for example, the contents of
<your ip> could be 192.168.1.1
Management GUI labels and item names appear in bold, whether or not the name is explicitly displayed (for example, buttons and icons).
Note the quotes. The text entered does not include the quotes.
Version 5.2
User text entered into
Manager, e.g., to assign as the name of a logical object
"Env1", "Network1"
Related Documentation
For additional information, see the following documents:
•
UFM for InfiniBand Quick Start Guide (LIT-00056)
•
UFM for InfiniBand Release Notes (DOC-00549)
•
UFM for InfiniBand User Manual (DOC-00600)
•
UFM for InfiniBand and Ethernet API Guide (DOC-00563)
•
UFM for InfiniBand and Ethernet API Reference (Advanced License only) (DOC-01077)
•
UFM for InfiniBand SDK Tools and Examples (Advanced License only) (DOC-01009)
•
UFM Multi-Site Portal User Manual (4134)
Mellanox Technologies Confidential
21
Version 5.2
1
Unified Fabric Manager (UFM®) Overview
Unified Fabric Manager (UFM®) Overview
1.1 Scale-Out Your Fabric with Unified Fabric Manager
Mellanox's Unified Fabric Manage (UFM®) is a powerful platform for managing scale-out computing environments. UFM enables data center operators to efficiently monitor and operate the entire fabric, boost application performance and maximize fabric resource utilization.
While other tools are device-oriented and involve manual processes, UFM’s automated and application-centric approach bridges the gap between servers, applications and fabric elements, thus enabling administrators to manage and optimize from the smallest to the largest and most performance-demanding clusters.
1.2 UFM Benefits
•
Central console for fabric management
•
In-depth visibility of traffic behavior, fabric and device health
•
Service-oriented automatic provisioning of fabric resources
•
Automatic coping with changes
•
End-to-end quality of service (QoS) provisioning
•
Quick resolution of fabric problems
•
Enables multiple isolated application environments on a shared fabric
•
Optimizes performance via end-to-end (QoS) management
•
Seamless failover in High Availability mode
•
Open architecture for seamless integration with third-party tools in customer environments
1.2.1 Central Console for Fabric Management
UFM provides all fabric management functions via one central console.
The ability to monitor, troubleshoot, configure and optimize all fabric aspects is available via one interface. UFM’s central dashboard provides a one-view fabric-wide status view.
1.2.2 In-Depth Fabric Visibility and Control
UFM includes an advanced granular monitoring engine that provides real-time access to switch and host data, enabling cluster-wide monitoring of fabric health and performance, real-time identification of fabric-related errors and failures, quick problem resolution via granular threshold-based alerts and a fabric utilization dashboard.
1.2.2.1 Advanced Traffic Analysis
Fabric congestion is difficult to detect when using traditional management tools, resulting in unnoticed congestion and fabric under-utilization. UFM’s unique traffic map quickly
22
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
1.2.3
1.2.4
1.2.5
1.2.6
1.2.7
identifies traffic trends, traffic bottlenecks, and congestion events spreading over the fabric, which enables the administrator to identify and resolve problems promptly and accurately.
Enables Multiple Isolated Application Environments on a Shared Fabric
Consolidating multiple clusters into a single environment with multi-tenant data centers and heterogeneous application landscapes requires specific policies for the different parts of the fabric. UFM enables segmentation of the fabric into isolated partitions, increasing traffic security and application performance.
Service-Oriented Automatic Resource Provisioning
UFM uses a logical fabric model to manage the fabric as a set of business-related entities, such as time critical applications or services. The logical fabric model enables fabric monitoring and performance optimization on the application level rather than just at the individual port or device level. Managing the fabric using the logical fabric model provides improved visibility into fabric performance and potential bottlenecks, improved performance due to application-centric optimizations, quicker troubleshooting and higher fabric utilization.
Quick Resolution of Fabric Problems
UFM provides comprehensive information from switches and hosts, showing errors and traffic issues such as congestion. The information is presented in a concise manner over a unified dashboard and configurable monitoring sessions. The monitored data can be correlated per job and customer, and threshold-based alarms can be set.
Seamless Failover Handling
Failovers are handled seamlessly and are transparent to both the user and the applications running on the fabric, significantly lowering downtime. The seamless failover makes UFM in conjunction with other Mellanox products, a robust, production-ready solution for the most demanding data center environments.
Open Architecture
UFM provides an advanced Web Service interface and CLI that integrate with external management tools. The combination enables data center administrators to consolidate management dashboards while flawlessly sharing information among the various management applications, synchronizing overall resource scheduling, and simplifying provisioning and administration.
Version 5.2
1.3 Main Functionality Modules
The UFM application comprises the following functional modules:
•
Fabric Dashboard
•
Fabric Design & Configuration
•
Fabric Discovery and Physical View
23
Mellanox Technologies Confidential
Version 5.2
1.3.1
1.3.2
1.3.3
1.3.4
1.3.5
1.3.6
Unified Fabric Manager (UFM®) Overview
•
Central Device Management
•
Monitoring
•
Configuration
•
Fabric Health
•
Logging
•
High Availability
Fabric Dashboard
UFM’s central dashboard provides a one-view fabric-wide status view. The dashboard shows fabric utilization status, performance metrics, fabric-wide events, and fabric health alerts.
The dashboard enables you to efficiently monitor the fabric from a single screen, and serves as a starting point for event or metric exploration.
Fabric Design and Configuration
In the Design view you can define the fabric logical model on which UFM bases configuration of the fabric. The correlation of all UFM functionality to the logical model enables simple and quick fabric configuration, easy change management and business oriented analysis as follows:
•
Monitored data and fabric events are correlated to the logical model.
•
Fabric and host configuration is performed according to the logical model.
•
Performance optimization is derived from the logical model.
Fabric Discovery and Physical View
UFM discovers the devices on the fabric and populates the views with the discovered entities. In the physical view of the fabric, you can view the physical fabric topology, model the data center floor, and manage all the physical-oriented events.
Central Device Management
UFM provides the ability to centrally access switches and hosts, and perform maintenance tasks such as remote access, firmware upgrade, shutdown and restart.
Monitoring
UFM includes an advanced granular monitoring engine that provides real time access to switch and server data. Fabric and device health, traffic information and fabric utilization are collected, aggregated and turned into meaningful information.
Configuration
In-depth fabric configuration can be performed from the Configuration module, such as routing algorithm selection and access credentials.
The Event Policy Table, one of the major components of the Configuration view, enables you to define threshold-based alerts on a variety of counters and fabric events. The fabric
24
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
1.3.7
1.3.8
1.3.9
administrator or recipient of the alerts can quickly identify potential errors and failures, and actively act to solve them.
Fabric Health
The fabric health tab contains valuable functions for fabric bring-up and on-going fabric operations. It includes one-click fabric health status reporting, UFM Server reporting, database and logs’ snapshots and more.
Logging
The Logging view enables you to view detailed logs and alarms that are filtered and sorted by category, providing visibility into traffic and device events as well as into UFM server activity history.
High Availability
In the event of a failover, when the primary (active) UFM server goes down or has been disconnected from the fabric, UFM's High Availability (HA) capability allows for a secondary (standby) UFM server to immediately and seamlessly take over fabric management tasks. Failovers are handled seamlessly and are transparent to both the user and the applications running in the fabric. UFM’s High Availability capability, when combined with Mellanox's High Availability switching solutions allows for non-disruptive operation of complex and demanding data center environments.
Version 5.2
Mellanox Technologies Confidential
25
Version 5.2
1.4
Unified Fabric Manager (UFM®) Overview
Fabric Topology with UFM
UFM is a host based solution, providing all management functionality required for managing fabrics.
Figure 1: Fabric Topology with UFM
26
UFM Server is a server on which UFM is installed, and has full visibility over the fabric to manage routing on all devices.
UFM HA Server is a UFM installed server on a secondary server for High Availability deployment.
Managed Switching Devices are fabric switches and gateways that are managed by UFM.
Managed Servers are the compute nodes in the fabric on which the various applications are running. UFM manages all servers connected to the fabric.
UFM Host Agent is an optional component that can be installed on the Managed Servers.
UFM Host Agent provides local host data and host device management functionality.
The UFM Host Agent provides the following functionality:
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
Discovery of IP address, CPU and memory parameters on host
•
Collection of CPU/Memory/Disk performance statistics on host
•
Upgrading HCA Firmware and OFED remotely
•
Creating IP interface on top of the InfiniBand partition
UFM Switch Agent is an embedded component in Mellanox switches which allows IP address discovery on the switch and allows UFM to communicate with the switch.For more
information, see Feature Support Matrix .
Version 5.2
1.5 UFM Software Architecture
The following figure shows the UFM high-level software architecture with the main software components and protocols. Only the main logical functional blocks are displayed and do not necessarily correspond to system processes and threads.
Figure 2: UFM High Level Software Architecture
1.5.1
1.5.2
Graphical User Interface
UFM Graphical User Interface is a Java application in which the application id can be downloaded and started from any supported Web Browser (Web-start). The GUI application uses the standard WebServices API provided by UFM Server.
Client Tier API
Third party clients are managed by the Web Services (WS) API.
27
Mellanox Technologies Confidential
Version 5.2 Unified Fabric Manager (UFM®) Overview
1.5.3
1.5.4
1.5.5
Client Tier SDK Tools (Advanced License Only)
Support for UFM’s API and set of tools that enhance UFM functionality and interoperability with third-party applications are provided as part of UFM with Advanced license.
UFM Server
UFM Server is a central data repository and management server that manages all physical and logical data. UFM Server receives all data from the Device and Network tiers, and invokes Device and Network tier components for management and configuration tasks. UFM
Server uses a database for data persistency. The server is built on the Python twisted framework.
Subnet Manager
Subnet Manager (SM) is the InfiniBand “Routing Engine”, a key component used for fabric bring-up and routing management.
UFM uses the Open Fabric community OpenSM Subnet Manager. UFM uses a plug-in API for runtime management and fabric data export.
1.5.6
1.5.7
1.5.8
1.5.9
Performance Manager
The UFM Performance Manager component collects performance data from the managed fabric devices and sends the data to the UFM Server for fabric-wide analysis and display of the data.
Device Manager
The Device Manager implements the set of common device management tasks on a variety of devices with varying management interfaces. The Device Manager uses SSH protocol and operates native device CLI (command-line interface) commands.
UFM Host Agent
UFM Host Agent is a small footprint software package that supports system parameter discovery, advanced server machine monitoring, and device management functionality. UFM
Host Agent uses a proprietary protocol for communication with the UFM Server.
The UFM Host Agent from previous UFM software releases is not supported by UFM 3.x.
You must install the UFM 3.0 Agent.
UFM Switch Agent
UFM Switch Agent is an integrated part of the switch software. The agent supports system parameter discovery and device management functionality on switches.
1.5.10 Communication Protocols
UFM uses the following communication protocols:
•
User Interface GUI and CLI clients communicate with the UFM server by means of Web
Services carried on SOAP.
•
The UFM server communicates with the Host Agent located on managed devices by proprietary TCP/UDP-based discovery and monitoring protocol and SSH.
28
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
Monitoring data is sent by the Host Agent to the UFM Server Listener by a proprietary
TCP-based protocol. The UFM Server communicates with the Switch Agent located on managed switches by proprietary TCP/UDP-based discovery protocol and SSH.
Version 5.2
Mellanox Technologies Confidential
29
Version 5.2
2 Installation and Initial Configuration
Installation and Initial Configuration
2.1 Overview of Installation and Initial Configuration
UFM™ software includes Server and Agent components. UFM Server software should be installed on a central management node. For optimal performance, and to minimize interference with other applications, it is recommended to use a dedicated server for UFM.
The UFM Agent is an optional component, and should be installed on fabric nodes. The
UFM Agent should not be installed on the Management server.
The following sections provide step-by-step instructions for installing and activating the license file, installing the UFM server software, and installing the UFM Agent.
2.2 UFM System Requirements
The following sections describe UFM system requirements.
2.2.1 UFM Server Requirements
Table 2: UFM Server Requirements
Platform
OS and Kernel
Type and Version
64-bit OS:
•
Red Hat 6.0: 2.6.32_71.el6
•
Red Hat 6.1: 2.6.32_131.0.15.el6
•
Red Hat 6.2: 2.6.32-220.el6
•
Red Hat 6.3: 2.6.32_279
•
Red Hat 6.4: 2.6.32-358
•
Red Hat 6.5: 2.6.32_431
•
Red Hat 6.6
: 2.6.32-504.el6.x86_64
•
•
Red Hat 7.0: 3.10.0-123
•
CentOS 6.0: 2.6.32_71.el6
•
CentOS 6.1 : 2.6.32_131.0.15.el6
•
CentOS 6.2 : 2.6.32_220.el6
•
CentOS 6.3 : 2.6.32_279
•
CentOS 6.4 : 2.6.32-358
•
CentOS 6.5: 2.6.32_431
•
•
•
•
SLES11 SP1: 2.6.32.12-0.7-default
•
SLES11 SP2: 3.0.13_0.27
1
2
30
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Platform
CPU
HCAs
OFED
Type and Version
•
SLES11 SP3: 3.0.76_0.11
•
Oracle Enterprise Linux 6.1: 2.6.32_131.12.1, 2.6.32_131.0.15
•
Oracle Enterprise Linux 6.3: 2.6.32-279 x86_64
•
Mellanox ConnectX®-2 with Firmware 2.9.1200
•
Mellanox ConnectX®-3 with Firmware 2.35.5000
•
Mellanox ConnectX®-3 Pro with Firmware 2.35.5000
•
Mellanox ConnectX®-4 with Firmware 12.12.xxxx
•
Mellanox Connect-IB® with Firmware 10.12.xxxx
•
MLNX_OFED 1.5.3-3.1.0
•
MLNX_OFED 2.0-x.x.x-2.4-x.x.x
•
MLNX_OFED 3.0-1.0.0
Version 5.2
NOTE: UFM does not support systems in which the Network Manager (NetworkManager) service is enabled.
Before installing UFM on Red Hat OS, make sure to disable it the service.
2.2.2
Table 3: UFM Server Resource Requirements per Cluster Size
Fabric Size CPU Requirements
1
Memory
Requirements
Up to 1000 nodes
1000 - 5000 nodes
5000 - 10000 nodes
4-core server
8-core server
16-core server
4 GB
16 GB
32 GB
Above 10000 nodes Consult with Mellanox Support
Disk Space Requirements
Minimum Recommended
20 GB 80 GB
40 GB
80 GB
120 GB
160 GB
1. CPU requirements refer to resources consumed by UFM. You can also dedicate a subset of cores on a multicore server, for example, 4 cores for
UFM on a 16-core server.
UFM Agent Requirements
Table 4: UFM Agent Requirements
Platform
OS
CPU
HCAs
OFED
Type and Version
RedHat 6.0 – 6.5
CentOS 6.0 – 6.5
SLES 11 SP1 – SP3
Scientific Linux (SL) 6 x86_64
ConnectX DDR/QDR/FDR, Firmware 2.8.0600 and later
OFED 1.5.3-1 and later
31
Mellanox Technologies Confidential
Version 5.2
2.2.3
Installation and Initial Configuration
Platform
Disk Size
Type and Version
Required: 10 MB
UFM GUI Client Requirements
Table 5: UFM GUI Client Requirements
Platform
OS
Browser
Memory
Type and Version
Any host running either JRE1.6.X or JRE 1.7.X (Official Oracle
JVM)
*UFM GUI might malfunction if both JRE 1.6 and 1.7 are installed.
Internet Explorer, Firefox, Chrome, Opera
Minimum: 2 GB
Recommended: 4 GB
2.2.4 UFM OpenSM Version
Table 6: UFM Open SM Version
Platform
OpenSM
Type and Version
UFM for InfiniBand 5.2 package includes OpenSM version 5.2.
2.3
2.3.1
32
UFM Software Installation Prerequisites
Before installing UFM software, verify the prerequisites for standalone and high availability installation.
Prerequisites for UFM Server Software Installation
Verify that a supported version of Linux is installed on your machine. For details, see UFM
The following table lists the packages that must be installed on your machine (according to system OS) before you install the UFM server software.
Red Hat / CentOS / OEL
httpd mysql-5.0 or MySQL-clientcommunity-5.1 mysql-server-5.0 or MySQL-servercommunity-5.1 php
PyXML net-snmp net-snmp-libs net-snmp-utils
SUSE Linux Enterprise Server (SLES) 11
apache2-prefork
Mysql-client (5.0.67-13.20.1)
Mysql (5.0.67-13.20.1) apache2-mod_php5 pyxml (0.8.4-194.23.38) net-snmp (5.4.2.1-8.2.1)
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Red Hat / CentOS / OEL
mod_ssl python-lxml-1.3.4-1 (or above) infiniband-diags python-xml
Openssl
(Make sure openssl version on RH6.5 is 1.0.1e-16 or higher)
SUSE Linux Enterprise Server (SLES) 11
Apache-worker (2.2.10-2.24.5) python-lxml-1.3.4-1 (or above)
The following RPMs should be taken from the SLES SDK distribution:
• python-pexpect (2.3-23.22)
• python-serial (2.4-1)
• python-twisted (8.0.1-1.62)
• python-twisted-conch (8.1.0-1.22)
• python-zopeinterface (3.3.0-79.48)
• python-iptables (1.4.6-2.2.24.x86_64) infiniband-diags python-xml
Sles11 sp3 (additional pre-required rpms for
Sless11 sp3):
• apache2-mod_php53
• php53
• python-gtk(HA installation)
• libgnutls26(HA Installation)
Openssl
In addition, ensure the following before you begin installation:
•
The computer hostname is not defined as 127.0.0.1 and localhost is defined as 127.0.0.1.
•
The host name must NOT appear on the loopback address line. An example of the loopback address is: 127.0.0.1 localhost.localdomain localhost.
•
Disable the firewall service (/etc/init.d/iptables stop), or ensure that the required ports are open (see the prerequisite script).
•
SELinux is disabled.
•
If you run the Linux prelink utility (http://linux.die.net/man/8/prelink), exclude heartbeat by adding the following line to /etc/prelink.conf:
-b /usr/lib64/heartbeat
(The prelink utility causes heartbeat to function incorrectly.)
•
If more than one fabric is managed by different UFM instances, set up different management network spaces for each fabric (not the same LAN).
•
Uninstall any previously installed Subnet Manager from the UFM server machine.
•
OFED 1.5.x version (Mellanox, or community OFED) is installed prior to installing UFM with ib0 and/or ib1 interface up and running.
•
The default Mellanox OFED installation includes opensm. Remove the Mellanox OFED
opensm before UFM installation by running rpm -e, for example: rpm -e opensm-3.3.9.MLNX_20111006_e52d5fc-0.1
33
Mellanox Technologies Confidential
Version 5.2
2.3.2
Installation and Initial Configuration
By default, ib0 and eth0 are configured as primary access points for the UFM management. If different management and/or InfiniBand interfaces (including bond interfaces) are used as the primary access points, you should modify the configuration file by running the script /opt/ufm/scripts/change_fabric_config.sh as described in section
Configuring General Settings in gv.cfg
Change the UFM Agent interface to the Ethernet and/or IPoIB interfaces used for communication with UFM Agent: ufma_interfaces = ib0,eth0
•
An IP address is defined for the local Ethernet interface and for the InfiniBand interface
(/etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-ib0 and/or ifcfg-ib1).
•
When standalone UFM is used in “Fabric Collector Remote” mode, it might not be equipped with Mellanox HCA, since it does not require in-band access. to the monitored fabric.
Additional Prerequisites for UFM High Availability (HA) Installation
•
Reliable and high capacity out-of-band IP connectivity between the UFM Primary and
Secondary servers (1 Gb Ethernet is recommended). This connectivity is used for heartbeat messages and DRBD synchronization.
•
Format two identical servers with dedicated disk partitions for UFM replication. Since the
UFM configuration file is replicated to the standby server, both master and standby servers must have the same interfaces.
•
Allocate exactly the same size partition on both servers (master and slave) for the
replicated data. See UFM Server Requirements for the recommended partition size.
Partitions should not be mounted and must be zeroed (file system should not be installed on the partitions). For disk partitioning, see the Linux user manual (man fdisk).
•
Uninstall existing DRBD and Heartbeat services. UFM installation replaces existing
DRBD and Heartbeat configuration files.
•
We recommend that you establish a passwordless SSH (via /root/.ssh/authorized_keys file) between the two servers before installation.
•
In fabrics consisting of multiple tiers of switches, it is recommended that the management ports (ib0) of the primary and secondary UFM server be connected to different fabric switches on the same tier (the outermost edge in CLOS 5 designs).
This is because by default, UFM manages the IB fabric via ib0, port 1 of the HCA.
Failure or disconnect of ib0, the IB management port, causes a failure condition in UFM resulting in HA failover.
When the management ports (ib0) of the primary and secondary UFM server are connected to the same switch, a failure of this switch will result in a disconnect of both
UFMs from the fabric and therefore UFM will not be able to manage the fabric.
NOTES:
34
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Subnet Manager is running over native InfiniBand layer, therefore bonding the IpoIB interfaces will not provide high availability. For additional information, please refer to
section UFM Failover to Another Port
The UFM installation includes the InfiniBand Performance Management module
(IBPM). This module is responsible for reporting performance information back to
UFM and upper layer applications. When available, this process is offloaded to the nonmanagement port (default ib1) of the UFM server. Failure or disconnect of the nonmanagement port (ib1) on the primary UFM server will not cause UFM to failover. By default, the UFM Health Monitoring process is configured to try to restart the IBPM.
For more information, see UFM Health Configuration
2.4
2.5
2.5.1
UFM Installation Steps
To install the UFM software
1.
Download the UFM software and license file
2.
Install the UFM server software and activate the license file
3.
4.
5.
Downloading the UFM Software and License File
Before you obtain a license for the UFM software, prepare a list of servers with the MAC address of each server on which you plan to install the UFM software. These MAC addresses are requested during the licensing procedure.
Obtaining the License
UFM is licensed per managed device according to the UFM license agreement.
When you purchase UFM, you will receive an email with instructions on how to obtain your product license. A valid UFM license is a prerequisite for installation and operation of UFM.
UFM licenses are per managed node and are aggregative. If you install an additional license, the system adds the previous node number and the new node number, and manages the sum of the nodes. For example, if you install a license for 10 managed nodes and an additional license for 15 nodes, UFM will be licensed for up to 25 managed nodes.
To obtain the license
1. Go to Mellanox’s Licensing and Download Portal http://license1.mellanox.com/LMManage/login.aspx
and log in as specified in the licensing email you received.
•
If you did not receive your Mellanox Licensing and Download Portal login information, contact your product reseller.
•
If you purchased UFM directly from Mellanox and you did not receive the login information, contact [email protected]
.
2. Click the Licenses tab. The list of software product serial licenses you own is displayed as well as software product license information and status.
35
Mellanox Technologies Confidential
Version 5.2
2.5.2
Installation and Initial Configuration
3. Select the serial number of the product license you want to activate.
4. In the MAC Address field, enter the MAC address of the delegated license-registered host. If applicable, in the HA MAC Address field, enter your High Availability (HA) server MAC address.
If you have more than one NIC installed on a UFM Server, use any of the MAC addresses.
5. Click Generate License to create the license key file for the software.
6. Click the license key file (.lic file) to download the file and save it on your local computer.
If you replace your NIC or UFM server, repeat the process of generating the license to set new MAC addresses. You can only regenerate a license two times. To regenerate the license after that, contact Mellanox Sales Administration at [email protected]
.
Downloading the UFM Software
This software download process applies to software updates as well as for first time installation.
If you own the UFM Media Kit and this is your first time installation, skip this section.
To download the UFM software
1. Click the Downloads tab, select the relevant version of the software and click the zip file to download.
2. In the Download file window, click the Download link.
3. Save the file on your local drive.
4. Click Close.
2.6
36
Installing the UFM Server Software
The default UFM installation directory is /opt/ufm.
UFM Server installation options are:
•
Standalone
•
High Availability (HA)
To prevent interference with other user software running on the UFM Server, UFM uses a separate instance of MySQL with a non-default data location and access interfaces.
The following processes might be interrupted during the installation process:
• httpd
• dhcpd
• mysqld
After installation:
•
Activate the software license.
•
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
•
NOTE: Before you run UFM, ensure that all ports used by the UFM server for internal and
external communication are open and available. For the list of ports, see Used Ports
2.6.1 Installed Packages
The following table lists the packages that are installed by the UFM installation for each system OS.
Red Hat / CentOS / OEL
enum-0.4.4-1.noarch.rpm python-IPy-0.70-1.el6.noarch.rpm pyDes-1.3.1-1.noarch.rpm
PyDO-2.0.1-1.noarch.rpm
Not required python-crypto-2.3-3.rhel6.x86_64.rpm
SLES 11 SP1 / SP2/ SP3
enum-0.4.4-1.noarch.rpm
IPy-0.70-1.noarch.rpm pyDes-1.3.1-1.noarch.rpm
PyDO-2.0.1-1.noarch.rpm
TwistedSNMP-0.3.13-3.x86_64.rpm pycrypto-2.3-1.x86_64.rpm pyserial-2.5-1.rhel6.noarch.rpm python-fpconst-0.7.3-7.0.cf.el6.noarch.rpm
Not required
Not required
SOAPpy-0.11.6-11.el6.noarch.rpm Not required python-zope-interface-3.6.1-7.0.cf.el6.x86_64.rpm python-zopeinterface-3.3.0-
79.17.x86_64.rpm python-twisted-core-13.2.0-1.0.cf.el6.x86_64.rpm Not required python-twisted-web-13.2.0-1.0.cf.el6.x86_64.rpm Not required
Not required python-twisted-10.1.0-13.3.x86_64.rpm
python-ZSI-2.0-6.el6.noarch.rpm ZSI-2.0-1.noarch.rpm python-4Suite-XML-1.0.2-9.el6.x86_64.rpm pysnmp-se-3.5.2-3.el6.noarch.rpm python-twistedsnmp-0.3.13-3.el6.x86_64.rpm pyinotify-0.9.2-1.noarch.rpm psutil-0.3.0-1.x86_64.rpm python-pyasn1-0.0.13b-2.rhel6.noarch.rpm python-lxml-2.2.3-1.1.el6.x86_64.rpm python-django-1.3.1-1.el6.rf.noarch.rpm python-4Suite-XML-1.0.2-9.x86_64.rpm pysnmp-se-3.5.2-1.noarch.rpm
Not required pyinotify-0.9.2-1.noarch.rpm psutil-0.3.0-1.x86_64.rpm python-pyasn1-0.0.13-2.1.x86_64.rpm python-lxml-2.1.2-1.20.x86_64.rpm python-django-1.3-1.suse11.x86_64.rpm django-piston-0.2.3-1.noarch.rpm python-zope-filesystem-1-5.el6.x86_64.rpm python-zope-interface-3.5.2-2.1.el6.x86_64.rpm python-south-0.7.3-1.el6.noarch.rpm libtasn1-2.3-3.el6_2.1.x86_64.rpm gnutls-2.8.5-4.el6_2.2.x86_64.rpm django-piston-0.2.3-1.noarch.rpm
Not required
Not required python-south-0.7.3-1.suse11.noarch.rpm
Not required
Not required
Mellanox Technologies Confidential
37
Version 5.2 Installation and Initial Configuration
38
Red Hat / CentOS / OEL
freetds-0.64-1.el6.rf.x86_64.rpm pyodbc-2.1.7-1.el6.x86_64.rpm sql_server.pyodbc-1.0-1.noarch.rpm python-twisted-conch-13.2.0-
1.0.cf.el6.x86_64.rpm
Not required
Not required
Not required
Not required
Not required
SLES 11 SP1 / SP2/ SP3
freetds-0.82-9.2.x86_64.rpm python-pyodbc-2.1.8-6.1.x86_64.rpm sql_server.pyodbc-1.0-1.noarch.rpm
Not required egenix-mx-base-2.0.6-1.x86_64.rpm python-distribute-0.6.19-2.2.x86_64.rpm unixODBC-2.2.12-197.25.x86_64.rpm libtdsodbc0-0.82-9.2.x86_64.rpm ipcalc-0.40-10.noarch.rpm
On a system with SLES 11 SP1-SP3, the Apache Web server configuration is changed by the installation process to include the following modules:
• php5
• proxy
• proxy_http
• headers
The following additional packages are installed by the UFM HA installation process for each type of system OS.
Red Hat / CentOS / OEL libnet-1.1.5-1.el6.x86_64.rpm pils-2.1.3-1.el6.x86_64.rpm stonith-2.1.3-1.el6.x86_64.rpm heartbeat-2.1.3-1.el6.x86_64.rpm
Not required heartbeat-2.1.3-19.1.x86_64.rpm heartbeat-core-2.1.3-19.1.x86_64.rpm
Not required drbd-utils-8.3.11-1.el6.x86_64.rpm heartbeat-resources-2.1.3-19.1.x86_64.rpm drbd-utils-8.3.10-1.x86_64.rpm drbd-bash-completion-8.3.11-1.el6.x86_64.rpm drbd-bash-completion-8.3.10-1.x86_64.rpm drbd-heartbeat-8.3.11-1.el6.x86_64.rpm drbd-pacemaker-8.3.11-1.el6.x86_64.rpm drbd-udev-8.3.11-1.el6.x86_64.rpm drbd-km-2.6.32_131.0.15.el6.x86_64-8.3.11-
1.el6.x86_64.rpm drbd-xen-8.3.11-1.el6.x86_64.rpm
SLES 11 SP1 / SP2 /SP3
Not required
Not required
Not required drbd-heartbeat-8.3.10-1.x86_64.rpm drbd-pacemaker-8.3.10-1.x86_64.rpm drbd-udev-8.3.10-1.x86_64.rpm drbd-km-2.6.32.12_0.7_default-8.3.10-1.x86_64.rpm drbd-xen-8.3.10-1.x86_64.rpm drbd-8.3.10-1.x86_64.rpm drbd-8.3.11-1.el6.x86_64.rpm
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
NOTE: Non-standard kernels are compiled with DRBD-8.3.14.
Version 5.2
2.6.2 Installing the UFM Server Software as Standalone
NOTE: If you start to install UFM on a server with a non-supported kernel version, an error message appears with instructions for continuing the installation. For more information, see
Installing UFM on a Non-Supported Kernel Version
To install the UFM server software as standalone for InfiniBand
1. Create a temporary directory (for example /tmp/ufm).
2. Open the UFM software zip file that you downloaded. The zip file contains the following installation files for:
•
Red Hat 6 / CentOS 6 / OEL 6: ufm-5.2.0-XXX.el6.x86_64.tgz
•
SLES 11 SP1-SP3: ufm-5.2.0-XXX.sles.x86_64.tgz
3. Extract the installation file for your system's OS to the temporary directory that you created.
4. From within the temporary directory, run the following command as root:
./install.sh -o ib
The UFM software is installed. You can now remove the temporary directory.
2.6.2.1 Installing UFM on a Non-Supported Kernel Version
To install UFM on a server with a non-supported kernel version, you must compile the kernel dependent modules (DRBD, MFT) on the specific kernel. The UFM installation package includes the sources and the relevant compilation utility.
When an unsupported kernel version is detected during UFM installation, the following error message appears:
UFM installation is missing kernel-dependent <list of RPMs> for local kernel
<kernel-number>.
Copy $running_dir/ufm-rpm-src.tar.gz file to a machine with C compiler. Open the file and run the "compile_rpm.sh -o <output dir> [OPTIONS]" command.
Copy the created tar file back to <output dir> directory and run install again.
To proceed with UFM installation:
1. Copy the ufm-rpm-src.tar.gz file from your UFM install directory to a machine with a C compiler. The kernel-dependent modules can be compiled on the UFM host, if the host has the C compiler and compilation environment installed. Make sure kernel-devel rpm is installed on host where installation is performed
2. Extract the file: tar xzvf ufm-rpm-src.tar.gz
3. Run the compile_rpm.sh script the file and run the relevant command:
For standalone:
39
Mellanox Technologies Confidential
Version 5.2
2.6.3
Installation and Initial Configuration compile_rpm.sh -o <output dir> -m
For High Availability: compile_rpm.sh -o <output dir> -d -m
The compilation creates a tar file in the specified <output dir> directory.
4. After the compilation completes, copy the created tar file back to the UFM install directory. It is recommended to save the compiled kernel-dependent RPMs for future installations.
5. Run the UFM installation command again and continue installation for standalone or HA, as required.
Installing the UFM Server Software for High Availability
NOTE: If you start to install UFM on a server with a non-supported kernel version, an error message appears with instructions for continuing the installation. For more information, see
Installing UFM on a Non-Supported Kernel Version
40
If high availability (HA) is required, install UFM for HA on a server that is designated to be the master. The installation process replicates the UFM configuration file (gv.cfg) on the server that is designated to be the standby.
If failover occurs, UFM and SM are started on the standby server. The new master SM uses the same configuration files since they are replicated. (InfiniBand Subnet Manager (SM) failover is not used.)
You can also run UFM in HA mode with additional UFM Subnet Managers:
1. Install UFM for HA as described below.
2. On the additional servers, install the UFM server software as Standalone
3.
Configure UFM to run in HA mode with additional UFM Subnet Managers
Before you begin installation, ensure that you have configured the required network
connectivity as described in Additional Prerequisites for UFM High Availability (HA)
To install the UFM server software for High Availability for InfiniBand
1. Create a temporary directory (for example /tmp/ufm).
2. Open the UFM software zip file that you downloaded. The zip file contains the following installation files for:
•
Red Hat 6 / CentOS 6 / OEL 6: ufm-5.2.0-XXX.el6.x86_64.tgz
•
SLES 11 SP1- SP3: ufm-5.1.0-XXX.sles.x86_64.tgz
3. Extract the installation file for your system's OS to the temporary directory that you created.
4. On the master server, run the following command as root:
./install.sh -o ib -m <ip:drbd-partition[:hostname]> -s <ip:drbdpartition[:hostname]> -c <virtual-ip> [-p <interface>] [ -n <netmask> ] [
-b <broadcast-ip> ] [ -u <heartbeat udp port> ] [ -t <drbd tcp port> ]
The parameters in square brackets are optional. The following table describes installation options.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
-n
-b
-u
-t
After installation, you can remove the temporary directory.
Option
-o
-m
-s
-c
-p
Description
(Installation only) Install mode, InfiniBand or Ethernet.
Master node parameters including IP address partition for drbd and optional hostname.
Standby node parameters including IP address partition for drbd and optional hostname.
Common IP address. GUI clients should connect to this address in HA mode.
This address is migrated between master and standby upon failover.
Interface for Common IP address alias-interface.
Mandatory if Common IP address does not match management IP subnet.
Netmask for the common IP address.
Broadcast address for the common IP address.
UDP port for heartbeat (default 694).
TCP port for drbd (default 8888).
Version 5.2
Note: For options -m and -s, use a reliable and high-capacity out-of-band management network (1 Gb Ethernet is recommended). Using inband IPoIB will cause the HA split-brain condition if there is an InfiniBand network failure.
2.6.4
High Availability Installation Examples
./install.sh -o ib -m 192.168.0.1:/dev/sda5:master -s
192.168.0.2:/dev/sda6:standby –c 192.168.0.3 -n 255.255.255.0 -b
192.168.0.255
./install.sh -o ib -m 192.168.0.1:/dev/sda5 -s 192.168.0.2:/dev/sda6 -c
192.168.0.3 172.30.20.33 –p eth2
For more information, see High Availability
Performing Unattended Installation
You can install UFM using an install script that defines all installation parameters. This means that when you run the script, UFM is installed without further prompts.
To perform unattended installation
1. Extract the UFM tar file to a temporary directory (for example /tmp/ufm).
2. Open the automatic_install.conf configuration file.
3. Configure installation parameters as described in the table.
Table 7: Installation Parameters
Parameter
install_mode
Description
The type of fabric that UFM will manage - ib, eth.
ib is the default.
41
Mellanox Technologies Confidential
Version 5.2 Installation and Initial Configuration
42
Parameter
install_method server_mode ignore_prerequisites_warn license_dir
Description
install (default) - Installs UFM as a new installation. If there is an existing UFM installation, it is removed prior to the new installation.
upgrade - If there is an existing installation, it is upgraded to the new installation version, otherwise UFM is installed as a new installation.
The operation mode of the UFM server - standalone,
high_availability. The default is standalone.
Whether to ignore warnings about missing prerequisites - no, yes.
The default is no, which means that the installation stops if there are warnings.
Remove the comment (#) to change the setting.
The directory in which to place the UFM licenses. When
install_method is install, the licenses are copied to the configured directory after installation completes.
Remove the comment (#) to configure the setting, otherwise you must copy the license files manually after installation for UFM to work.
The following parameters are ignored when server_mode is not high_availability.
Table 8: Parameters for High Availability Installation
Parameter
master_node standby_node common_ip netmask broadcast_ip heartbeat_port drbd_port
Description
Mandatory - remove the comment (#) to set this parameter.
The address of the master node in the form IP:partition:hostname, for example, 192.168.0.1:/dev/sda5:master
Mandatory - remove the comment (#) to set this parameter.
The address of the standby node in the form
IP:partition:hostname, for example,
192.168.0.2:/dev/sda6:standby
Mandatory - remove the comment (#) to set this parameter.
The common IP address for GUI clients to connect to.
Optional - remove the comment (#) if required.
Netmask for the common IP address.
Optional - remove the comment (#) if required.
Broadcast address for the common IP address.
Optional - remove the comment (#) if required.
UDP port for heartbeat (default 694).
Optional - remove the comment (#) if required.
TCP port for DRBD (default 8888).
Example Installation Configuration File
# UFM installation configuration file
# Configure installation parameters in this file, as required, and then run to install or upgrade UFM.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
# Mandatory: The fabric type to manage.
# Options: eth, ib install_mode = ib
# Mandatory: Install new or upgrade existing?
# Options: install, upgrade
# install: Remove existing installation, then install UFM.
# If no existing installation, just install UFM.
# upgrade: Upgrade existing installation.
# If no existing installation, just install UFM. install_method = install
# Mandatory: The operation mode of the UFM server.
# Options: standalone, high_availability server_mode = standalone
# Optional: Ignore prerequisites warnings? When set to "no", installation stops on warnings.
# Options: yes, no
# Remove comment if required (otherwise installation stops on warnings).
#ignore_prerequisites_warn = yes
# Optional: Directory that holds the UFM licenses.
# The license files are copied to UFM after installation completes.
# This value is ignored if install_method is not "install".
# Remove comment if required - otherwise licenses are not
# copied and you must copy the license files manually after installation in order for UFM to work.
#license_dir = /
# High availability installation
# The following attributes are ignored if server_mode is not
# "high_availability".
# Mandatory for high availability. Remove comments when server_mode=high_availability.
# Master node
#master_node = IP:partition:hostname
# For example:
#192.168.0.1:/dev/sda5:master
#Standby node
#standby_node = IP:partition:hostname
# For example:
#192.168.0.2:/dev/sda6:standby
# Common IP address for GUI clients to connect to.
#common_ip = 192.168.0.3
# Optional for high availability. Remove comments if required
# Netmask for the common IP address
#netmask = 255.255.255.0
# Broadcast address for the common IP address
#broadcast_ip = 192.168.0.255
# UDP port for heartbeat (default 694)
#heartbeat_port = 694
# TCP port for DRBD (default 8888)
#drbd_port = 8888
Version 5.2
43
Mellanox Technologies Confidential
Version 5.2
2.6.5
Installation and Initial Configuration
Activating the Software License
To activate the software license
1. Before starting the UFM software, copy your license file(s) downloaded from Mellanox’s
Licensing and Downloading Site (volt-ufm-<serial-number>.lic) to the master server under the /opt/ufm/files/licenses directory. We recommend that you back up the license file(s).
In High Availability mode, the license files are replicated to the standby machine automatically. Your software is now activated.
2. Run the UFM software as described in the following sections.
2.7
2.7.1
44
Initial Configuration
After installing the UFM server software, before running UFM, perform the following:
•
Configure General Settings in gv.cfg
•
•
•
Activate and Enable Congestion Control Manager (Advanced License Only)
•
Activate and Enable Lossy Configuration Manager (Advanced License Only)
•
Configure UFM Health Monitoring Settings
Configuring General Settings in gv.cfg
Configure general settings in the /opt/ufm/conf/gv.cfg file. When running UFM in HA mode, the gv.cfg file is replicated to the standby server.
•
Fabric interfaces
This parameter should not be edited directly in the gv.cfg file. To change the interface, use the command below and only when the UFM server is inactive/stopped.
/opt/ufm/scripts/change_fabric_config.sh –i <interface>
For HA installation, the command should be run on the host with DRBD primary state.
If you have configured bonding for failover to another port, use the bond name. For more
information, see UFM Failover to Another Port
•
Running UFM in Monitoring mode monitoring_mode = yes
For more information, see Running the UFM Software in Monitoring Mode
•
Running UFM SM only (UFM HA with additional SMs)
For more information, see Configuring UFM HA with Additional Subnet Managers
•
Add a subnet manager plug-in (for example, ccmgr) to the event_plugin_name option:
# Event plugin name(s) event_plugin_name osmufmpi ccmgr
•
Add the plug-in options file to the event_plugin_options option:
# Options string that would be passed to the plugin(s)
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2 event_plugin_options ccmgr --conf_file /opt/ufm/files/conf/cc-mgr.conf
These plug-in parameters are copied to the opensm.conf file in Management mode only.
•
SNMP settings (UFM Advanced)
/opt/ufm/scripts/update_gv_cfg.sh --section "Notifications" --key
"snmp_listeners" --values "host1, host2:166"
For more information, see SNMP Settings (UFM Advanced only)
The following show examples of configuration settings in the gv.cfg file:
•
Polling interval for Fabric Dashboard information ui_polling_interval = 30
For more information, see Configuring the Fabric Dashboard Polling Interval
•
[Optional] UFM Server local IP address resolution (by default, the UFM resolves the address by gethostip). UFM GUI should have access to this address. ws_address = <specific IP address>
•
HTTP/HTTPS Port Configuration
# WebServices Protocol (http/https) and Port ws_port = 8088 ws_protocol = http
For more information, see HTTP/HTTPS Port Configuration
•
Connection (port and protocol) between the UFM server and the APACHE server ws_protocol = <http or https> ws_port = <port number>
For more information, see Launching a UFM GUI Session.
•
SNMP get-community string for switches (fabric wide or per switch)
# default snmp access point for all devices
[SNMP] port = 161 gcommunity = public
•
Configuring Monitoring History
[MonitoringHistory]
# history_report_timeout = 600
NOTE: Other [Monitoringhistory] parameters are updated implicitly via the configuration script and should NOT be modified in the gv.cfg file directly.
•
Enhanced Event Management (Alarmed Devices Group)
[Server] auto_remove_from_alerted = yes
For more information, see Enhanced Event Management.
•
Log verbosity
[Logging]
# optional logging levels
#CRITICAL, ERROR, WARNING, INFO, DEBUG level = INFO
45
Mellanox Technologies Confidential
Version 5.2 Installation and Initial Configuration
For more information, see UFM Log
•
Settings for saving port counters to a CSV file
[CSV] write_interval = 60 ext_ports_only = no
For more information, see Saving the Port Counters to a CSV File
•
Max number of CSV files (UFM Advanced)
[CSV] max_files = 1
For more information, see Saving Periodic Snapshots of the Fabric (Advanced License
NOTE: The access credentials that are defined in the following sections of the gv.cfg file are used only for initialization:
•
SSH_Server
•
SSH_Switch
•
TELNET
•
IPMI
•
SNMP
To modify these access credentials, use the UFM GUI. For more information, see
Configuring Site Access Credentials
2.7.2
2.7.3
2.7.4
46
FDR10 Support
FDR10 support is disabled by default in UFM.
To enable FDR10 support
1. Modify the FDR10 parameter value in the
/opt/ufm/files/conf/
opensm/opensm.conf file.
•
0 - Do not use FDR10 (MLNX ExtendedPortInfo MADs are disabled)
•
1 - Enable FDR10 when supported
•
2 - Disable FDR10 when supported
2. Set the MLX_EPI parameter to true (MLX_EPI=true) in the
/opt/ufm/files/conf/
infiniband-diags/ibdiag.conf file.
Quality of Service (QoS) Support
Quality of Service (QoS) is set as False by default.
To enable it and benefit from its capabilities, set it to True in the
/opt/ufm/files/conf/opensm/opensm.conf
file.
UFM Failover to Another Port
You can configure UFM to fail over the UFM subnet manager (SM) to another InfiniBand port on the UFM Server that is connected to the fabric. When failure is detected on an
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
InfiniBand port or link, failover occurs without stopping the UFM Server or other related
UFM services, such as mysql, http, DRDB, and so on.
When the UFM Server is connected by two or more InfiniBand ports to the fabric, you can configure UFM Subnet Manager failover to one of the other ports. When failure is detected on an InfiniBand port or link, failover occurs without stopping the UFM Server or other related UFM services, such as mysql, http, DRDB, and so on. This failover process prevents failure in a standalone setup, and preempts failover in a High Availability setup, thereby saving downtime and recovery.
Figure 3: Network Configuration for Failover to IB Port
Version 5.2
NOTE: UFM SM failover is not relevant for Monitoring mode, because in this mode, UFM must be connected to the fabric over ib0 only.
To enable UFM failover to another port:
•
Configure bonding between the InfiniBand interfaces to be used for SM failover. In an
HA setup, the UFM active server and the UFM standby server can be connected differently; but the bond name must be the same on both servers.
•
Set the value of fabric_interface to the bond name. using the
/opt/ufm/scripts/
change_fabric_config.sh command as described in Configuring
. If ufma_interface is configured for IPoIB, set it to the
bond name as well. These changes will take effect only after a UFM restart. For example, if bond0 is configured on the ib0 and ib1 interfaces, in gv.cfg, set the parameter fabric_interface
to bond0.
•
If IPoIB is used for UFM Agent, add bond to the ufma_interfaces list as well.
When failure is detected on an InfiniBand port or link, UFM initiates the give-up operation that is defined in the Health configuration file for OpenSM failure. By default:
•
UFM discovers the other ports in the specified bond and fails over to the first interface that is up (SM failover)
•
If no interface is up:
47
Mellanox Technologies Confidential
Version 5.2
2.7.5
Installation and Initial Configuration
•
In an HA setup, UFM initiates UFM failover.
•
In a standalone setup, UFM does nothing.
If the failed link becomes active again, UFM will select this link for the SM only after SM restart.
Configuring UFM HA with Additional External UFM-Subnet Managers
You can run UFM in HA mode with additional external UFM Subnet Managers. This mode:
•
Provides additional Subnet Managers for failover.
•
Enables UFM upgrade without fabric downtime.
While the main UFM Server is running, it synchronizes the configuration files on all the external UFM-SMs. If the main UFM Server fails (or stops for maintenance operations) an
External SM takes mastership and manages the fabric until the main UFM Server resume operations.
The External UFM-SM is responsible for identifying a situation in which it does not receive configuration updates while the main UFM-SM is still active. In this case, one of the following occurs:
•
The priority of the SM is reduced to 0 (default) or
•
The SM is stopped if configured: stop_disconnected_sm = yes (see configuration section).
48
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Figure 4: UFM HA with Additional External UFM-Subnet Managers
Version 5.2
NOTE: Configuration files should be modified only on the main UFM Server and only while the main UFM Server is operational.
2.7.5.1 UFM HA with Additional External UFMs Installation Prerequisites
Before you install the UFM HA with Additional External UFMs, ensure that the following requirements are met:
•
Provide a list of remote UFM- SM’s – IP addresses: (one IP per line)
/opt/ufm/files/conf/ external_sm.conf
•
Define an ssh trust mode between UFM hosts and hosts that running remote UFM-SM. password-less ssh between UFM HA and every external UFM-SM host (2 x N) according to the /opt/ufm/files/conf/external_sm.conf file
2.7.5.2 Installing UFM with External UFM-SMs
The main UFM Server can be installed in Standalone
or High Availability mode. External
UFM-SM requires installation of the entire UFM package in the Standalone mode. For
installing External UFM-SM see Installing the UFM Server Software as Standalone . All
External UFM-SM must have the same version as the main UFM-SM.
2.7.5.3 Configuring UFM HA on the Main UFM
The following are several configuration settings changed when configuring UFM HA on the
Main UFM:
•
Set management mode
In the /opt/ufm/files/conf/gv.cfg (on the primary UFM) set the management_mode to a mode that allows other SM. UFM will continue to print a warning if another SM is running in the fabric. Is is crucial to change the
49
Mellanox Technologies Confidential
Version 5.2 Installation and Initial Configuration management_mode
when UFM is stopped since this setting effects the start / stop behavior. management_mode = allow_other_sm
•
List of External UFM-SMs
In the/opt/ufm/files/conf/ external_sm.conf file (on the primary UFM) add the IP addresses of all the External UFM-SMs. IP addresses of UFM HA hosts such as the examples below should not appear in this file.
192.168.10.11
192.168.10.12
192.168.10.13
•
Parameters to be overwritten when opensm.conf is copied to the External SM
In the /opt/ufm/files/conf/opensm.conf.sync_mask file (on the primary
UFM) the parameters below will be overwritten once the opensm.com file is copied to the External SM. log_flags 0x03 sm_priority 14 sminfo_polling_timeout 30000 polling_retry_number 6
Modifying the values of guid and sm_priority is forbidden.
•
Configuration of External SM behavior
In the/opt/ufm/files/conf/sm_sync.conf file (on the primary UFM) set the stop_disconnected_sm file as shown below to handle disconnection state (A state in which the Remote UFM-SM does not receive configuration updates while the main
UFM-SM is still active) stop_disconnected_sm = no
•
If set to "no" (default): The external SM is not stopped even when not synchronized, but the SM priority reduced.
•
If set to yes: The SM process is stopped and is resumed only after the new configuration files is received.
Time interval to check and synchronize the configuration conf_update_time = 60
2.7.5.4 Configuring UFM HA on External UFM SMs
To configure the UFM HA on External UFM SMs set the running mode as SM only on each additional UFM Server in the /opt/ufm/files/conf/gv.cfg (on the external UFM-
SM) file: management_mode = sm_only
2.7.5.5 Running the UFM Software with External UFM-SM
To run the UFM Software with External UFM-SM:
1. Run the main UFM Server according to the operating mode ( standalone
.
50
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
2. Once all the External UFM-SMs are synchronized, start each External UFM-SM by invoking /etc/init.d/ufmd start
The main UFM-SM must run with priority 15 and it must be the only SM with priority 15.
•
If another SM with priority 15 is found during the startup, the UFM will not start
•
If another SM with a lower priority is found during startup, a warning message with the current master SM details will be printed and the main UFM-SM will start and take mastership
The External UFM-SM runs with priority 14 or moved to priority 0 if in the disconnected state.
2.7.5.6 Stopping ufmd & ufmha
The safe_stop command forces synchronization of all external UFM-SM configuration, changes the local SM priority to 12 and waits for other remote UFM-SM to take over before stopping the ufmd. If an error is detected during safe_stop, an error message is displayed describing the error and the stop procedure is canceled.
It is recommended using the safe_stop instead of stop to prevent unexpected loss in the fabric.
•
/etc/init.d/ufmd safe_stop
- in the Standalone mode
•
/etc/init.d/ufmha safe_stop
- in HA mode
Version 5.2
2.8
2.8.1
Running the UFM Server Software
Before you run UFM
•
Perform initial configuration.
•
Ensure that all ports used by the UFM server for internal and external communication are
open and available. For the list of ports, see Used Ports .
You can run the UFM server software in the following modes:
•
Management
•
High Availability
CAUTION: In Management or High Availability mode, ensure that all Subnet
Managers in the fabric are disabled before running UFM. Any remaining active Subnet
Managers will prevent UFM from running.
•
Monitoring
•
Monitoring mode with failover to an external SM
Running the UFM Server Software in Management Mode
After installing, run the UFM Server by invoking:
/etc/init.d/ufmd start
Log files are located under /opt/ufm/files/log (the links to log files are in /opt/ufm/log).
51
Mellanox Technologies Confidential
Version 5.2
2.8.2 Running the UFM Software in High Availability Mode
On the Master server, run the UFM Server by invoking:
/etc/init.d/ufmha start
Installation and Initial Configuration
NOTE: The opt/ufm/files directory is unmounted during this operation, therefore, ensure that you run this script outside of the opt/ufm/files directory.
2.8.3
You can specify additional command options for the ufmha service.
Table 9: ufmha Command Options
Command
start stop safe_stop shutdown failover takeover status reset health_start health_stop health_restart help
Description
Starts the heartbeat on both servers.
Stops the heartbeat on both servers (used for configuration mode).
This option is applicable to allow_other_sm mode only.
Stops the heartbeat on both servers but fails if the remote UFM-SM is not ready.
Stops the heartbeat on both servers and unmounts configuration directory.
Initiates failover (change mastership from local server to remote server).
Initiates takeover (change mastership from remote server to local server).
Shows current HA status.
Cleans the database and restarts the UFM server.
For advanced users only.
Starts the UFM Health module.
For advanced users only.
Stops the UFM Health module.
For advanced users only.
Restarts the UFM Health module.
Displays this help.
Running the UFM Software in Monitoring Mode
Run UFM in Monitoring mode while running concurrent instances of Subnet Manager on
Mellanox switches. Monitoring and event management capabilities are enabled in this mode.
UFM non-monitoring features such as provisioning and performance optimization are disabled in this mode.
For information about running Monitoring mode with failover to an external SM, see
Running UFM in Monitoring Mode with Failover to an External SM.
The following table describes whether features are enabled or disabled in Monitoring mode.
52
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Table 10: Features Enabled/Disabled in Monitoring Mode
Feature
Fabric Discovery
Topology Map
Fabric Dashboard
Fabric Monitoring
Alerts and Thresholds (inc. SNMP traps)
Fabric Logical Model
Subnet Manager and plugins
Subnet Manager Configuration
Automatic Fabric Partitioning
Central Device Management
Quality of Service
Failover (High Availability mode)
Traffic Aware Routing Algorithm
Fabric Collective Accelerator (FCA)
Device Management
Integration with Schedulers
Unhealthy Ports
Disabled
Disabled
Disabled
Disabled
Disabled
Disabled
Disabled
Disabled
Enabled/Disabled in Monitoring
Mode
Enabled
Enabled
Enabled
Enabled
Enabled
Enabled
Disabled
Disabled
Disabled
In Monitoring mode, UFM periodically discovers the fabric and updates the topology maps and database.
For Monitoring mode, connect UFM to the fabric using port ib0 only. The fabric must have a subnet manager (SM) running on it (on another UFM, HBSM, or switch SM).
NOTE: When UFM is running in Monitoring mode, the internal OpenSM is not sensitive to changes in OpenSM configuration (opensm.conf).
NOTE: When running in Monitoring mode, the following parameters are automatically overwritten in the /opt/ufm/files/conf/opensm/opensm_mon.conf file on startup: event_plugin_name osmufmpi event_plugin_options --vendinfo -m 0
Any other configuration is not valid for Monitoring mode.
To run in Monitoring mode
1. In the /opt/ufm/conf/gv.cfg configuration file:
53
Mellanox Technologies Confidential
Version 5.2 Installation and Initial Configuration
•
Set monitoring_mode to yes
•
If required, change mon_mode_discovery_period (the default is 60 seconds)
•
Set reset_mode to no_reset
We recommend this setting when running multiple instances of UFM so that each port
counter is not reset by different UFM instances. For more information, see Resetting
2. Restart the UFM Server.
The Running mode is set to Monitoring, and the frequency of fabric discovery is updated according to the setting of mon_mode_discovery_period.
3. Go to the About window to verify that the Running mode is Monitoring.
54
Table 11: Monitoring Mode Parameters
Parameter
monitoring_mode mon_mode_discovery_peri od
Description
When set to Yes, UFM runs in
Monitoring mode.
When changing this setting, restart UFM for the change to take effect.
Determines fabric timeout polling interval in seconds.
When changing this setting, restart UFM for the change to take effect.
Values
No (default), Yes
60 (default) 30 - 300
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
2.8.4
To change running mode from Monitoring to Management
1. Set the monitoring_mode parameter to No in the configuration file.
2. Shut down all other active SMs in the fabric.
3. Restart the UFM server.
4. The running mode is set to Management. You can verify the running mode in the About window.
UFM with Remote Fabric Collector
UFM with remote Fabric Collector enables the user to run the UFM Server on a Virtual
Machine (VM) or on a host not directly connected to the fabric. It collects and sends fabric data to a specified UFM Server via HTTP/HTTPS.
Fabric Collector can be used by everyone that requires fabric monitoring only, on existing, running clusters without installing UFM Server.
UFM supports redundancy of fabric collectors. It is possible to run multiple fabric collectors on different hosts in the fabric. UFM selects one of the fabric collectors and replaces it with another one if it gets disconnected for any reason.
Figure 5: UFM Fabric Collector Architecture
Version 5.2
2.8.4.1 UFM Server as a Virtual Appliance
A UFM virtual appliance (OVA file) can be downloaded from Mellanox website.
This virtual appliance can be used to monitor an InfiniBand fabric in conjunction with a
Fabric Collector installed on a host in the fabric. The OVA file was created using vmware tools and was tested with WMware ESXi version 5.0.0 and VMware player version 6.0.1.
The virtual appliance requires a UFM license similar to UFM installed on a physical server.
55
Mellanox Technologies Confidential
Version 5.2 Installation and Initial Configuration
To activate the UFM virtual appliance:
1. Deploy the virtual appliance on a VMware hypervisor or player and start it.
2. Log in using the console with username “root” and password “ufmvm”.
3. Configuring the Fabric Collector as described in section Configuring The Fabric
Installing Fabric Collector on a Management Host in the Fabric
Fabric Collector is part of the MLNX_OFED package but not installed by default. It can be installed either automatically upon MLNX_OFED installation, by enabling the “--withfabric-collector”
option, or manually by using the RPM utility (rpm -ivh
<fabric collector rpm> ). The installation path is:
/opt/mellanox/fabric_collector
.
For older installations of MLNX_OFED, you can download a supplemental package of
Fabric Collector from the Mellanox License Portal site as well as a VMware image based on
RH 6.4 which includes UFM.
NOTE: The UFM Server should be installed in a standalone mode on any supported physical or virtual host. For UFM installation, please refer to section 2.6.2. In “Fabric
Collector Remote” mode in the UFM User Manual.
NOTE: When standalone UFM is used in “Fabric Collector Remote” mode, it might not be equipped with Mellanox HCA, since does not require in-band access to the monitored fabric.
2.8.4.2 Configuring the Fabric Collector
To configure the Fabric Collector:
1. [On UFM Server or the virtual appliance] Get the UFM Server Management IP address. ip a s <interface>
2. [On a Remote Host] Open the Fabric Collector configuration file
(/etc/fabric_collector/fb_collector.conf).
3. [On a Remote Host] Replace the "host" attribute with the IP provided in Step 1: host = <IP address>
4. [On a Remote Host] Replace the “site_name” attribute with a unique name to distinguish between different fabrics. site_name = <unique name>
5. [On UFM Server or the virtual appliance] Open the UFM configuration file
(opt/ufm/files/conf/gv.cfg).
6. [On UFM Server or the virtual appliance] Change the following attributes: fabric_interface = lo fabric_collector_mode = remote monitoring_mode = yes
7. [On a Remote Host] Start the fabric collector service.
/etc/init.d/fbcollectord start
56
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
8. [On UFM Server or the virtual appliance] Start UFM.
/etc/init.d/ufmd start
NOTE: The UFM Server waits for fabric initial data from the Fabric Collector and therefore, startup may take a while. If the UFM Server does not receive the initial fabric data, it exits after 5 minutes.
Version 5.2
2.8.4.3 Important Notes
•
UFM Server boot takes longer when there is no Fabric Collector running on the fabric because the UFM Server waits up to 5 minutes to receive data from the Fabric Collector
2.8.5 HTTP/HTTPS Port Configuration
After installation, you can configure the web server to communicate in the secure protocol
HTTP/S. For further information, please refer to the Launching a UFM GUI Session [UM3.4 up] section.
Port 8088 is an internal port that is used by the UFM server (a port that is not exposed to the user by the Apache Web Server). Apache web server listens on port 80 and forwards the incoming traffic to the local port 8088. Port 8088 is configurable, port 80 is not.
To configure using HTTP/S protocol instead of the default HTTP, add the following to the configuration file at /opt/ufm/conf/gv.cfg::
# WebServices Protocol (http/https) and Port ws_port = 8088 ws_protocol = https
UFM installation configures HTTPS protocol in the webserver as follows:
•
Configures listening on port 443
•
Configures default virtual host
•
Creates/uses local certificates
2.8.6 Setting up the GUI Client Machine
Ensure that the Sun Java Web Start Application is installed on the computer running the
GUI. To download directly from java.sun.com, go to the JAVA download page at:
http://java.sun.com/javase/downloads/index.jsp
Before using the UFM GUI, configure GUI setup as follows:
•
Configure Screen Resolution
•
Configure the Browser for GUI Optimization when Using a Proxy Server
•
Configure JRE and clear the Java cache
2.8.6.1 Screen Resolution
The UFM GUI is best viewed at a resolution format of at least 1024 by 768 pixels. The
Device Manager was especially designed to work at this resolution.
All browsers in Linux and Windows are supported by the UFM GUI.
57
Mellanox Technologies Confidential
Version 5.2 Installation and Initial Configuration
2.8.6.2 Browser Configuration - GUI Optimization when Using a Proxy Server
When the proxy option is enabled, configure your browser to bypass the proxy server to access the switch. Set the IP of the switch in the Proxy Settings window, as shown in the following example. Each browser has its own configuration. Check how to set the proxy bypass in your specific browser.
To configure your browser to bypass the proxy server
1. From the Internet Explorer Main Menu, go to Tools>Internet Options>Connections.
2. Click LAN settings
3. In the LAN Settings window, check the following boxes: Automatically detect settings and Bypass proxy server for local addresses.
4. Click Advanced.
5. In the Proxy Settings window Exceptions field, enter the IP of the switch to bypass.
6. Click OK on each window. Your settings are saved.
The following example shows how to bypass the proxy settings in Internet Explorer.
Figure 6: Example of Proxy Bypass Settings (IE)
2.8.6.3 Configuring JRE
The JRE installation package is available on the Mellanox support site and the product CD supplied with the switch. Install Java and configure Java to ensure that your cache remains clear for optimal GUI performance.
To configure JRE:
1. On your computer, install JRE version 1.6.x.
2. Locate the JRE application on your computer (for example C:\Program
Files\Java\jre1.6.XXX)
58
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
3. Use the command line to access the bin folder and execute javaws -viewer. The
Java Application Cache Viewer window is displayed.
For example: C:\Program Files\Java\jre1.6.0_03\bin>javaws – viewer
4. The Java Cache Viewer Window is displayed. Close it.
5. In the Java Control Panel, select the General tab and then click Settings.
Version 5.2
The Temporary Files Settings window opens.
6. In the Disk Space area, set the amount of disk space for storing temporary files to the lowest possible settings by moving the slider to the minimum (1 MB), as shown.
7. Click Delete Files.
Mellanox Technologies Confidential
59
Version 5.2 Installation and Initial Configuration
8. Select all check boxes and click OK.
9. In the Temporary Files Settings window, click OK to close the window.
10. In the Java Control Panel window, in the General tab, click Network Settings.
11. Select the Direct Connection option and click OK.
12. In the Java Control Panel, select the Advanced tab.
13. Open Security and then click Mixed code (sandboxed vs trusted) security verification.
14. Check the Enable -hide warning and run with protections option.
60
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
2.8.7 Launching a UFM GUI Session
Before accessing the UFM GUI:
•
If required, you can change the configuration of the connection (port and protocol) between the UFM server and the APACHE server in the file gv.cfg:
•
ws_protocol = http or https
•
Setting the parameter ws_protocol to http allows unsecured access
• setting the parameter ws_protocol to https denies unsecured access. If it is set to https, and the user tries to log in using http, the following error message is displayed upon logging into the GUI:
•
ws_port = port number
Mellanox Technologies Confidential
61
Version 5.2
To launch a UFM GUI session
1. Launch the GUI by entering the following URL in your browser:
http://<UFM_server_IP>/ufmui
https://<UFM_server_IP>/ufmui
Installation and Initial Configuration
2. On the UFM Welcome Page, click either Launch Unified Fabric Manager or
Download GUI Launcher (.jar file).
•
If you choose Download GUI Launcher to activate UFMUI.jar locally, please follow the instructions below to launch UFM GUI. a. Create a shortcut of the .jar file. Right click the file --> Create shortcut. b. Open the new .jar file properties. Right click the new file --> Properties. c. Change the Target path to: "C:\Program Files\Java\<Java version>\bin\javaw.exe" jar UFMUI.jar -s <IP address>. d. Double click the .jar file to log into the GUI.
62
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
If you choose Launch Unified Fabric Manager, please insert your credentials in the
Login Window and start using the GUI.
Version 5.2
2.8.8
3. In the Login Window, enter your User Name and your predefined user Password and click OK.
Once you have entered your user name and password, the main window opens, showing
the UFM Dashboard. For more information, see Fabric Dashboard for OnScreen Status
Launching a UFM GUI Session for Apple MacBook
Before accessing the UFM GUI:
•
If required, you can change the configuration of the connection (port and protocol) between the UFM server and the APACHE server in the file gv.cfg:
•
ws_protocol = http or https
•
Setting the parameter ws_protocol to http allows unsecured access
Mellanox Technologies Confidential
63
Version 5.2 Installation and Initial Configuration
• setting the parameter ws_protocol to https denies unsecured access. If it is set to https, and the user tries to log in using http, the following error message is displayed upon logging into the GUI:
•
ws_port = port number
To launch a UFM GUI session
1. Launch the GUI by entering the following URL in your browser:
http://<UFM_server_IP>/ufmui
2.
https://<UFM_server_IP>/ufmui
64
3. On the UFM Welcome Page, click either Launch Unified Fabric Manager or
Download GUI Launcher (.jar file).
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
•
If you choose Download GUI Launcher to activate UFMUI.jar locally, please follow one of the methods below to launch UFM GUI.
•
Method A i. Open the download folder. ii. Right click the .jar file --> Open with Jar Launcher./ Click the .jar file to launch the application.
•
Method B i. Click the Spotlight to search for the Jar Bundler.
(If not installed, the Jar Bundler can be downloaded from: http://sourceforge.net/projects/jarbundler/) ii. Launch the Jar Bundler. iii. Click Choose to select the .jar file.
Mellanox Technologies Confidential
65
Version 5.2 iv. Search for the downloaded UFMUI.jar file.
Installation and Initial Configuration v. Click Choose. vi. Type "-s <IP address>" into the Arguments to Main: field.
66
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual vii.Click Create Application.... and choose the folder in which to create your application.
Version 5.2
•
If you choose Launch Unified Fabric Manager, please insert your credentials in the
Login Window and start using the GUI.
4. In the Login Window, enter your User Name and your predefined user Password and click OK.
67
Mellanox Technologies Confidential
Version 5.2 Installation and Initial Configuration
2.8.9
Once you have entered your user name and password, the main window opens, showing
the UFM Dashboard. For more information, see Fabric Dashboard for OnScreen Status
User Authentication
UFM User Authentication is based on standard Apache User Authentication. Each Web
Service client application must authenticate against the UFM server to gain access to the system.
The UFM software comes with one predefined user:
•
Username: admin
•
Password: 123456
You can change the definition, add, or delete users. For more information, see Managing
2.8.10 Licensing
UFM enables functionality based on the license that was purchased and installed. This license determines the functionality and the size of the fabric that the product can manage.
For further details on the difference between UFM Licenses, please refer to Appendix I.
If several licenses are installed on the server, UFM uses only the highest type of license and takes into consideration the managed devices limits on it, regardless of any other licenses that may exist on the server. For example: If you have a UFM Advanced license for one thousand managed nodes installed on the server, any evaluation licenses or other standard
UFM licenses will be disregarded.
To view License information
1. After you have installed and activated your software, you can view your licenses in the
GUI by clicking the About icon ( ) in the main window.
68
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
2. To view the key information, click the relevant product key in the User Product Keys list.
The details for the specific Key are displayed on the right side under Key Details.
General Information (at the top of the About window) includes:
•
UFM software version and build
•
Client version: GUI version
•
Product Functionality: Level of functionality enabled for the end-user.
Product Functionality is updated only after startup. If you replace the UFM license, UFM continues to work in the previous mode until the UFM server is restarted.
Key detail information is listed per Product Key and includes:
•
Customer number: Customer number provided by Mellanox
•
Serial Number: UFM serial number provided by Mellanox
•
License Type: Permanent or evaluation
•
Expiration Date: License expiration date (limited), unlimited License (nothing is written)
•
Functionality: UFM or UFM Advanced
2.8.10.1 Removing a License
You can remove a license by deleting the license file from /opt/ufm/files/licenses.
69
Mellanox Technologies Confidential
Version 5.2
2.9
2.9.1
Installation and Initial Configuration
Installing and Running the UFM Agent Software
The UFM agent is an optional component that you can install on the fabric nodes to collect local node information and perform local management tasks.
UFM Agent Installation Prerequisites
Before you install the UFM agent, ensure that the following packages are installed on your system:
• net-tools
• coreutils
• procps
• sysstat
• sed
• util-linux
• grep
• gawk
•
OpenIPMI-tools
•
Iproute
• ip-bonding
Most of these packages are installed by the default LINUX installation. The UFM agent installation script terminates the installation if packages are missing. However, package names might vary between Linux distributions.
NOTE: TCP/UDP port 6306 is used for communication with the UFM server. This value might be overwritten (see installation option ufma-ufm-mcast-port).
2.9.2 Installing the UFM Agent Software
The default UFM installation directory is /opt/ufm.
To install the UFM Agent software:
1. Create a temporary directory (for example /tmp/ufm).
2. Extract ufma-3.0.0-XXX-<OS>-x86_64.tgz to the temporary directory (where <OS> is
redhat or suse).
3. Change directory to /tmp/ufm/ufma-3.0.0-XXX-<OS>-x86_64.
4. From the temporary directory, run the following command as root: ./install.sh -ufma-mcast-ifaces
"ethxx,ibxx". You can use command options to override default installation parameters.
5. After installation you can remove the temporary directory.
70
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
2.9.2.1 UFMA Installation Parameters
Table 12: General Installation Parameters
Parameter
--autostart
Default
y
--start-ipmi
--scriptsonly y
Notes
Override this if you do not want to start the
UFMA process automatically.
Override this option if you do not use IPMI in your fabric.
Use this option when you install UFMA on the
UFM server.
Table 13: UFMA Parameters
Parameter
--ufma-cpu-affinity
--ufma-ufm-mcast-addr
--ufma-ufm-mcast-port
--ufma-mcast-ifaces
--ufma-dir
Default
None
224.0.23.172
6306
Notes
CPU core number, starting from zero.
This address MUST be configured in
UFM.
This port MUST be configured in
UFM.
N/A, mandatory parameter You can bind UFMA listener to specific interfaces; for example, eth0, ib3
.
/opt/ufma Use only if /opt/ is mounted as readonly.
Version 5.2
2.9.3 Running the UFM Agent Software
After installing the UFM Agent software, the UFM Agent runs automatically. The UFM
Agent reports messages to /var/log/ufma.log.
•
To check the status, invoke:
/etc/init.d/ufmad status
•
To set the verbosity of the log level, use the UFMA_LOG_LEVEL environment variable.
Valid values are from 0 (low) to 6 (high).
To activate the new setting, restart the agent using:
/etc/init.d/ufmad restart
For troubleshooting, send the log files to Mellanox support.
Mellanox Technologies Confidential
71
Version 5.2
2.9.4
Installation and Initial Configuration
UFM SNMP Agent
UFM supports system MIBs, and answers SNMP queries for the following parameters:
Parameter
sysDescr sysObjectID sysContact
Value
UFM Server 5.2
1.3.6.1.4.1.5206.1.200 www.mellanox.com
Default get community string is “public”, default set community string is “private”.
2.10 Working with Fabric Collective Accelerator (FCA)
When you install both UFM and FCA, you must copy the FCA license to the UFM server so that FCA can work with UFM.
2.11 Upgrading UFM Software
After UFM installation, UFM detects existing UFM versions previously installed on the machine and prompts you to run a clean install of the new version or to upgrade. We recommend backing up the UFM configuration before upgrading the UFM as specified in
section UFM Database and Configuration Snapshot
NOTE: UFM upgrade is supported from UFM v3.5 and above.
2.11.1 Standalone Server Upgrade
You can upgrade the UFM standalone server software for InfiniBand from version 2.3.
To upgrade the UFM server software
1. Create a temporary directory (for example /tmp/ufm).
2. Open the UFM software zip file that you downloaded. The zip file contains the following installation files for:
•
Red Hat 6 / CentOS 6 / OEL 6: ufm-5.1.0-XXX.el6.x86_64.tgz
•
SLES 11 SP1-SP3: ufm-5.1.0-XXX.sles.x86_64.tgz
3. Extract the installation file for your system's OS to the temporary directory that you created.
4. Stop the UFM server by running: /etc/init.d/ufmd stop
5. From within the temporary directory, run the following command as root:
./upgrade.sh
6. Select one of the following at the prompt:
•
Upgrade from the previous version (preserves existing UFM data and configuration).
•
Re-install. In this case, all previous data and configuration will be lost.
72
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
7. Restart the UFM server by running: /etc/init.d/ufmd start.
After the upgrade:
•
Remove the temporary directory.
•
Clear the Java cache on the GUI client machine. See Configuring JRE
2.11.2 High Availability Upgrade
You can upgrade the UFM server HA software for InfiniBand from version 2.3. The upgrade is performed on the master server.
To upgrade the UFM Server software
1. Create a temporary directory (for example /tmp/ufm).
2. Open the UFM software zip file that you downloaded. The zip file contains the following installation files for:
•
Red Hat 6 / CentOS 6 / OEL 6: ufm-5.1.0-XXX.el6.x86_64.tgz
•
SLES 11 SP1-SP3: ufm-5.1.0-XXX.sles.x86_64.tgz
3. Extract the installation file for your system's OS to the temporary directory that you created.
4. Stop the UFM server by running: /etc/init.d/ufmha stop
5. From within the temporary directory, run the following command as root:
./upgrade.sh
6. Run the UFM server by running: /etc/init.d/ufmha start
After the upgrade:
•
Remove the temporary directory.
•
Clear the Java cache on the GUI client machine. See Configuring JRE
2.11.3 UFM Agent Upgrade
The UFM Agent is a stateless application. There is no dedicated upgrade procedure. The
Agent installation process detects whether there is a previous version of the agent and automatically removes it. For information about installing a new version of the UFM Agent,
see Installing the UFM Agent Software
2.11.4 Remote UFM-SM Upgrade
1. Stop UFM "/etc/init.d/ufmha safe_stop". safe_stop stops the UFM safely, it synchronizes all the configuration to External UFM-SM, change Local SM priority to 13 and wait for other External UFM-SM to take the mastership before stopping. If any error detected during safe_stop the stop procedure is canceled.
2. Upgrade the UFM following the HA upgrade procedure.
3. Start and validate UFM HA.
Version 5.2
2.12 Uninstalling UFM
UFM Server and UFM Agent can be uninstalled by running an uninstall script as described in the following sections.
Mellanox Technologies Confidential
73
Version 5.2 Installation and Initial Configuration
2.12.1 Uninstalling the UFM Server
To uninstall the UFM Server:
1. Go to /opt/ufm.
2. Run ./uninstall.sh.
Child interfaces are not deleted.
To delete primary interfaces, restart /etc/init.d/openibd.
2.12.2 Uninstalling the UFM Agent
To uninstall the UFM Agent:
1. Go to /opt/ufma/bin.
2. Run ./uninstall.sh.
Child interfaces are not deleted.
To delete primary interfaces, delete all the ib0.8xxx from network-scripts directory, then restart /etc/init.d/openibd.
74
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
3 Getting Familiar with UFM's Logical Model
3.1
3.1.1
3.1.2
Overview of Logical Model
UFM enables the fabric administrator to manage the fabric based on business-oriented requirements, as opposed to managing the fabric based on device-oriented and port-oriented management systems.
The business-centric capability is provided by UFM’s logical model which treats the physical fabric topology as an abstraction. You can define groups of fabric resources. For example, servers represent a certain application, a job running on the fabric, or a reserved computing resource pool for specific customers of a multi-tenant fabric.
All UFM functionality is associated with the logical model. For example, monitored data and fabric events are correlated to the logical groups, fabric and host configuration is performed according to the model, and performance optimization is derived from the logical model.
UFM Model Basics
The fabric managed by UFM consists of a set of physical and logical objects including their connections. The Object Model has a hierarchical object-oriented tree structure with objects as the tree elements. Each object defines an abstraction for physical or logical fabric elements.
Physical Model
The Physical Model represents the physical resources and connectivity topology of the
Network. UFM enables discovery, monitoring and configuration of the managed physical objects.
Table 14: Physical Objects
Icon Name
Port Object
Module Object
Link Object
Description
Represents the external physical port on switch or on Host Channel Adapter (HCA). A port is identified by its number. UFM provides
InfiniBand standard management and monitoring capabilities on the port level.
Represents the Field Removable Unit, Line card, and Network card on switch or HCA on host. For Mellanox Switches, Line and
Network Cards are modeled as modules.
Represents the physical connection between two active ports.
Version 5.2
Computer Object Represents the computer (host) connected to the Fabric. The UFM Agent installed on the host provides extended monitoring and management capabilities. Hosts without agents
75
Mellanox Technologies Confidential
3.1.3
Version 5.2
76
Icon Name
Switch Object
Description
are limited to InfiniBand standard management and monitoring capabilities.
Represents the switch chassis in the Fabric. A
Switch object is created for every Mellanox
Switch. Switches of other vendors are represented as InfiniBand Switches and limited by InfiniBand standard management and monitoring capabilities.
Rack Object Represents the arbitrary group of switches or computers. When linked devices are shown as a group, the link is shown between the group and the peer object.
Logical Model
The logical model consists of groups of resources that represent business requirements and is independent of hardware, technology and location. The following list summarizes each of the key UFM logical object types. The logical object types are presented in a typical hierarchical order used by a Fabric administrator to design and configure the data center.
Table 15: Logical Objects
Icon Name
Environment Object
Logical Gateway Group
Object
Logical Server Cluster
Object
Compute Object
Local Network Object
Global Network Object
Description
In UFM terminology, a logical environment represents an organization or application requiring computing services.
The Logical Gateway Group configures partitioning and QoS to a certain gateway.
Additionally, it provides the user the flexibility to choose which networks are assigned to each gateway.
The Logical Server (or Cluster) represents the group of HCAs selected or allocated for the specific job.
The compute object represents a Device in the Logical Server Group
Defines an IP network segment and its attributes, which consist of an IP address sub-range (from within the total IP address range defined by the organization) Network policies/configuration, QoS, and preferences.
Multiple Logical Servers can be connected to each other by attaching them to the same
Network. The connection is provided by the logical network interface object (see next item).
Getting Familiar with UFM's Logical Model
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Icon Name
Network Interface (NIC)
Object
Flow Object
Description
Network objects, can be global or local.
Local Network objects are located inside an environment object, thus they are able to connect between logical servers within the same environment.
Global Network objects are located
outside the Environment objects, thus they are able to connect between different
Environments.
Global and local Networks have the same connectivity functionality.
A logical Network Interface object is the logical connection between a Logical
Server object and a logical Network object.
Connecting a Logical Server to a logical network creates a Network Interface including IP range information, partitioning and QoS.
As a result of connection activity, UFM configures the relevant parameters on core switches and on each of the hosts allocated to the logical server object.
Note: Information for each host is inherited from the network object, but most of it can be modified on the interface level.
A Flow object defines specific traffic from the source logical Network Interface to the destination logical Network Interface. The traffic is defined by a set of filter parameters, such as, source/destination
L2/L3 addresses, L2 partitioning, and upper layer ports and protocols.
Quality of Service parameters can be specified on the Flow level to provide very granular QoS definition.
UFM may have multiple users with different access rights.
Version 5.2
Mellanox Technologies Confidential
77
Version 5.2 Getting Familiar with UFM's Logical Model
The following figure contains a simple example of the way logical objects are used to define a data center. The figure shows a data center Environment comprising one Local Network, one Global Network, and two Logical Server objects.
Figure 7: Logical Model
3.1.4 Extensible Object Model (Advanced License Only)
UFM enables you to define additional attributes for the following objects:
•
Environment
•
Logical Server
•
Network
•
Network Interface
•
Flow
•
Device
You can use the user-defined attributes to provide additional valuable information for managing your fabric, and as part of integration projects with other third-party tools.
You configure user-defined attributes in the Common Tasks pane in the View, Manage
Devices, and Design windows. For information, see Setting User-Defined Attributes
78
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
3.2 Object Types
This section describes the nature of the UFM object types in more depth and how they are used for designing logical data centers.
Logical Environment Object
Version 5.2
The logical environment object defines an environment, which can represent any of the following: a logical data center, a logical department data center, or a typical multi-tier application environment. A single logical environment consists of an association of logical servers, networks, and assigned resources.
You can create as many logical environments as needed and assign real resources only to the ones that are used at any given moment, or shift resources from one to the other, to match the application requirements.
Tasks can be conducted on all elements of an environment, thus simplifying administration, allowing task delegation and requiring fewer manual operations.
Logical Server Object
The purpose of a logical server is to define one or more services, such as a Web services application, an Oracle database, or a simulation cluster application. For each logical server, you can define its services and attributes, such as the network resources it uses, and other dependencies and properties. A logical server can also represent a cluster.
A logical server object represents a fabric service that spans all the cluster nodes allowing administrators to configure or monitor a cluster application/service. A service such as a Webserver cluster or a database cluster may span multiple physical nodes, and can be managed in
UFM as one entity, thus simplifying configuration and administration.
Mellanox Technologies Confidential
79
Version 5.2 Getting Familiar with UFM's Logical Model
Figure 8: UFM Logical Server Internal Hierarchy
A logical server can be connected to a network, which allows the server to communicate with other servers or resources on that network. When defining this connection using UFM, an element called the logical network interface object is created automatically. The object, which represents a Network Interface Card (NIC), holds the specified properties of the connection between the logical server and the logical network Interface. The properties consist mainly of the IP address base and range, (assigned automatically by UFM if not indicated otherwise). Network attributes, such as network mask and default gateway, are not specified per interface, and need to be defined only once when configuring the Network object.
Logical Network Object
80
A logical network object represents an isolated data network, corresponding to network
VLANs, partitions or VPNs. Each of the objects has a base IP address, a network mask, and a default gateway from which it can connect to other networks. Network objects may have additional attributes that define its Quality of Service (QoS), security, or the name of IP resolution mechanisms.
There are several types of network objects:
•
Global Network represents a network that can connect resources from different environments, for example, an organizational LAN.
•
Local Network represents a network that can only connect servers within an individual environment, for example, a backend network connecting several application tiers.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
•
Management Network (Reserved) is the default UFM Network and does not appear as a separate icon. Each server is connected to the Management Network by default. The
Management Network is secured by a unique mechanism that prevents servers from seeing each other, allowing them to see only the UFM server (or other nodes that were manually assigned as full members of the management Network).
Use the network object to define Network properties just once, and then associate to them as many logical servers as required. The association provides the server with all the attributes it needs, such as the relevant IP ranges from which addresses can be allocated, masks, and gateways. You no longer need to manually type the attributes every time, and you can maintain the attributes centrally, reducing administration overhead.
For each Network object, the IP address and the address mask represent a contiguous range of IP addresses.
A group of Logical Servers all connected to the same Network object can communicate with each other only when the network and the interface are defined as a full membership. In this way, UFM connects Logical Servers, which would not be able to access each other without a full membership connection.
In the same way, a Global Network object connects Logical Servers from different logical environments. While Logical Servers exist within one specific logical environment, Global
Network objects are situated outside the confines of environment objects, enabling them to connect between Logical Servers from different environments. This allows Logical Servers from different environments to communicate with each other.
Logical Network Interface Object
A logical network interface object is the logical connection between a logical server object and a logical network object.
Linking a logical server to a logical network object enables the server to communicate, both within its own environment and, in the case of global networks, with other environments.
You define this association using one of the UFM windows (Network Interface Properties), to automatically create an interface element for the logical server; the Logical Network
Interface (NIC) object. The interface object contains the connection properties required for the logical server. The logical server inherits these properties from the network object, unless they were overridden on the network interface.
The logical network interface object is displayed in UFM as the connecting line between a logical server object and the associated logical data center object.
Each logical server is associated with a default management network, and you can associate a logical server with as many additional network objects as needed.
Flow Object
81
Mellanox Technologies Confidential
Version 5.2
3.2.1
Getting Familiar with UFM's Logical Model
The Flow object defines specific traffic from the source logical Network Interface to the destination logical Network Interface. The traffic is defined by a set of filter parameters, such as source/destination L2/L3 addresses, L2 partitioning, upper layer ports and protocols, and so on.
Defining flows enables you to filter traffic according to the filter parameters and define
Quality of Service on specific traffic patterns. When Quality of Service parameters are not defined for a specific flow, the flow will inherit the parameters from the source logical
Network Interface object.
For each new logical Network Interface object, a default flow is created. The default flow models general traffic from the source logical Network Interface to all other Network
Interfaces connected to the same Network, without any additional filtering.
If the flow definition does not include a destination logical Network Interface, the flow models traffic from the source logical Network Interface to all other Network Interfaces connected to the same Network. Any other filter parameters can be applied to this flow.
Network Memberships
UFM implements a mechanism of memberships to control communications between nodes throughout the Network, providing enhanced security and isolation in the fabric.
Membership can be configured on the network and the network interface levels, while it is inherited to the logical servers associated with the network objects. Membership is applied and enforced per compute node.
•
Full Membership:A server can be a full member in a network, allowing it to communicate with all other network members.
A typical implementation of this configuration is an MPI job in which all members need full communication amongst themselves.
•
Partial Membership: As a partial member in a Network, a server can only communicate with full members.
An administrator may define shared resources such as storage or management nodes as full members, while defining standard servers as limited members, which blocks access by the standard servers.
3.3
3.3.1
Getting Familiar with the UFM GUI
The UFM Software has several main GUI views.
Main Tabs/Categories/Navigator Buttons
The following table describes the main UFM windows and categories:
Table 16: Navigator Tabs
Tab Icon Description
Click to get a top view of the Fabric status (summary).
82
Mellanox Technologies Confidential
3.3.2
3.3.3
Unified Fabric Manager for InfiniBand User Manual
Tab Icon Description
Click to design and build Environments and Logical Servers. In this view you can also attach the Logical Servers to Networks and allocate physical resources to Logical Servers.
Click to view the Fabric from the following perspectives:
Environments, Logical servers, Networks, and Sites. This view also enables you to perform some minor tasks.
Click to view, drill-down (ports), analyze (traffic and alarms), and configure all Fabric devices (physical elements). This information is presented in table format.
Click to configure the Fabric, including SM, Routing Table, Event
Management, User Management, Site Access Credentials, and Nonoptimal Links.
Click to establish monitoring sessions with managed devices.
During the session, the UFM server polls the devices for the requested information.
The UFM Monitoring engine enables you to initiate multiple sessions for any combination of Health, Traffic, and Performance parameters.
Data can be monitored on switches and fabric nodes and can be aggregated on groups of objects.
Click to run and view fabric reports, UFM reports, and system logs.
You can also back up UFM configuration files, and compare actual and planned fabric topology.
Dashboard Window
The Dashboard Window shows a top view of the Fabric status and provides a quick view of the health of the Fabric in the following smaller dashboard windows:
•
Top 10 Bandwidth and Congestion dashboards
•
Resource Utilization and Status
•
Traffic Map
•
Top 10 Logical Server dashboard showing the bandwidth consumers
•
Top 10 alarmed nodes
Dashboards can be dragged and dropped to allow simultaneous viewing of the dashboards and other views.
For more information, see Fabric Dashboard for OnScreen Status Monitoring
Viewing, Designing, and Managing Devices Windows
•
The View window displays the physical as well as logical entities in your Fabric.
•
The Design window allows you to design your Fabric.
Version 5.2
83
Mellanox Technologies Confidential
Version 5.2 Getting Familiar with UFM's Logical Model
•
The Manage Devices window allows you to manage the hardware elements of your fabric switches and servers.
These windows have a common format. For more information, see Fabric Design
3.3.3.1 Left Pane – Navigation Tree
Most windows have a left panel with the Fabric components displayed in hierarchical tree format. In the Design window, the Fabric objects are icons that you can drag into the map to design the Fabric.
3.3.3.2 Main Pane – Fabric View
The main pane shows the fabric in design format, map view or in table format. Global
Network or Logical Server contents are displayed in the Main pane area.
You can change the polling interval for updating the fabric topology display in the main
pane. For more information, see Configuring the Polling Interval for the Fabric Topology
3.3.3.3 Right Pane – Internal Structure (Top), Detailed Information (Middle) and
Common Tasks (Bottom)
The right pane in the View and Manage Devices windows enables you to:
•
View the Internal Structure of a selected object.
The Internal Structure pane shows the contents of an object selected in the Main pane
(Design area, Map view and Tables) or when applicable, in the Left Tree.
•
View the Properties of a selected object.
The Properties pane shows the contents of an object selected in the Main pane (Design area, Map view and Tables) or when applicable, in the Left Tree.
You can also use the Properties option to set or edit the properties of an object.
Properties are available:
•
From the Properties section in the right pane
•
Under the Common Tasks Properties link
To view object properties, you can also:
•
Right-click a component icon in the Map view or in the Left Tree.
•
Double-click the objects under the View tab (only).
•
Perform relevant actions on selected objects from options in the Common Tasks pane.
In many cases, you can perform tasks on selected single or multiple objects from the right
3.3.3.4 Bottom Pane – Event Table
The bottom pane shows the fabric events in real-time. You can adjust pane size to provide more screen space for other panes, as required.
84
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
3.3.4
3.3.5
3.3.6
3.3.7
Config Window
The Config window consists of several sections in which to configure the Fabric.
Each section in the Config window contains a title, a short overview, real time state of some fields, and a link into the relevant configuration panel.
Monitor Window
The Monitor window adopts a tri-pane format when a monitoring session is initiated. The left pane contains the list of saved monitoring templates, the main pane displays the monitoring session and the right pane includes the color legend and object information.
Health Window
The Health window consists of several sections in which to run and view reports and logs for the fabric and UFM. You can also back up UFM configuration files from this window.
General UFM Window Icons
The following icons appear in several views in UFM.
Table 17: Main Window Icons
Icon Name
Expand
Collapse
About
Interval
Function
Displays the Events Table by clicking the
Expand button (located in the lower section of the window).
Hides the Events Table by clicking the
Collapse button (located in the lower section of the window).
Displays the Server and Client Version numbers, Copyright and Licensing information.
Sets the polling interval for refreshing the fabric topology display views in the current session. For more information, see
Configuring the Polling Interval for the Fabric
Version 5.2
3.3.7.1 Right-Click/Context Sensitive Menu
You can use right-click menus in all views. Menu options change according to the objects you click.
3.3.7.2 Expanding and Collapsing Objects
For objects with child nodes, UFM displays an icon to the left of the object icon. To expand the object and reveal the children assigned to it, click the icon once. The siblings are displayed under the parent object. Click the icon again one time to collapse the parent object
(hide the siblings).
85
Mellanox Technologies Confidential
Version 5.2 Getting Familiar with UFM's Logical Model
3.3.7.3 Tooltips
Tooltips show information for specific objects or NICs (connecting lines). Display tooltips by positioning the cursor over the icon or line. Tooltips can also be displayed for toolbar items.
3.3.7.4 Moving Icons
Click and drag icons to move the object icons to the desired location in the Design and View windows.
To select multiple icons, click and then drag the mouse over the group you want to select, or press Ctrl and select the desired icons.
Icon locations can be reset by clicking the Hierarchical layout button on the toolbar.
3.3.7.5 Dragging and Dropping
You can drag and drop a table, a dashboard, or area by positioning the cursor on the blue header (at the top) and releasing at the desired location. The drag and drop feature is useful if, for example, you wish to work in another view and still see the information in the dashboard/table/area you dragged.
The cursor changes, as shown, when the specified area or table can be dragged and dropped.
3.3.8
86
The following windows and areas can be dragged and dropped:
•
All dashboards
•
Managed Devices: all tables
•
Monitor: Table Sessions (only)
•
Events Table
•
Port Counters Graph
General Table Features
The following icons appear in many tables throughout UFM.
Table 18: Table Features
Icon Feature
Hide and
Display
Save as CSV
Function
Hides and displays the filter bar or search bar.
Save (export) the current table locally as a .csv file.
Choose
Column
Opens a dialog box to select which columns to display.
To change the order of columns in a table
•
Drag and drop the column heading to the required position.
To filter information in a table
1. If the Filter bar is not displayed, click the Display button.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
2. From the drop-down menu, select the desired filter.
3. In the blank field, enter the search string. The table filters incrementally as you type your entry.
For example, to view all the events with the ID 352, enter 352 in the field. To filter according to date, enter the desired date in the blank field.
4. Click Clear to clear the current search.
Version 5.2
Mellanox Technologies Confidential
87
Version 5.2
4
Fabric Dashboard for On Screen Status Monitoring
Fabric Dashboard for On Screen Status
Monitoring
4.1 Overview of Fabric Dashboard
The Dashboard Window provides a summary of the fabric's status, including traffic, resources, jobs, and errors.
Figure 9: Fabric Dashboard View
4.2
The Fabric Dashboard view contains six dashboards, which provide real-time information about the fabric.
Floating Dashboards
Each dashboard can be transformed into a floating panel allowing you to keep the selected dashboard on the top of the screen and view the information in that dashboard while navigating to other categories (Design, View, Manage Devices, Config, Monitor, Logs).
To turn a dashboard into a floating pane
1. Place the mouse cursor over the titles.
The cursor changes appearance in to the 'move' icon, indicating that you can move the pane.
2. Click the title and drag the panel to the desired position. The following figure shows a floating pane:
88
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
4.3
3. Click the pin icon in the main dashboard. The floating pane returns to its original place.
Top N Dashboards
The Top N dashboards show the top N objects listed according to specific topics, such as bandwidth, congestion, alarms, and traffic. The following table details the options available in the Top N dashboard.
Table 19: Top Nodes by Bandwidth
Options
Bar View
List View
Drop-Down Menu
Monitoring Attributes
Description
Shows the top N objects as a bar graph or a pie chart.
Parameters
Bar graph
Pie chart
Shows the top N objects as list.
In the List View, you can select a node and right-click to display the device menu from which you can perform devicespecific tasks. For more information, see
Double-click the node to display to the relevant table in the Managed Devices view.
Select the N number of top nodes for which you would like to see the top bandwidth.
Select the attribute for monitoring.
5, 10, 15, 20
89
Mellanox Technologies Confidential
Version 5.2
4.4
Fabric Dashboard for On Screen Status Monitoring
Parameters Options
For example:
Description
Jumps to the location where you will be able to get more data according to the specific dashboard.
Top N Nodes by Bandwidth
The Top N Nodes by Rx or Tx Bandwidth panel shows the top nodes (servers) that are transmitting or receiving the most bandwidth per second. These nodes are classified topdown according the defined Transmit (Tx) or Receive (Rx) bandwidth (MB/sec Rate).
Bandwidth is measured as a rate in bytes/sec.
•
Transmitted (Tx) bandwidth is measured by N server ports in MB/sec
•
Received (Rx) bandwidth is measured by N server ports in MB/sec
N can be 5, 10, 15, or 20.
Figure 10: Top Nodes by Bandwidth - Bar
90
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Figure 11: Top Nodes by Bandwidth - List
Version 5.2
4.5
For general information about Top N dashboards, see Top N Dashboards
Fabric Resource Utilization
The Fabric Resource Utilization dashboard shows a general view of the fabric resource utilization and the health of the fabric (alarms) according to:
•
Servers – Total/Free/Allocated:
•
Total: Number of servers that exist in the fabric (this includes free and allocated servers, but not UFM servers)
•
Free: Number of servers in the fabric that are not allocated to any Logical Server
•
Allocated: Number of servers that are allocated to a Logical Server
•
Switches – Total
•
Total: Number of switches that exist in the fabric
•
Object number by Alarms – Normal/Critical/Minor/Warning
Mellanox Technologies Confidential
91
Version 5.2 Fabric Dashboard for On Screen Status Monitoring
•
Shows the total number of alarmed objects according to the most critical severity. For example, if a Server X has 2 minor alarms, 1 warning, and 2 critical alarms, the
Fabric Resource Utilization dashboard will show one server in the critical row.
Figure 12: Fabric Resource Utilization
92
The following table describes the options available in the dashboard.
Table 20: Fabric Resource Utilization Options
Icon Name
Free
Description
Number of servers that are not allocated (free).
Allocated Number of servers that are allocated to the Logical Servers.
Normal
Critical
Minor
Warning
Information/notification provided to the user of Normal operating state or a Normal system event.
Note: The number represents the number of objects affected by this severity (servers and switches).
The system or a system component fails to operate.
Note: The number represents the number of objects affected by this severity (servers and switches).
A problem in the fabric, but does not prevent its operation.
Note: The number represents the number of objects affected by this severity (servers and switches).
A low priority problem in the fabric, but does not prevent its operation. A warning is asserted when an event exceeds a predefined threshold.
Note: The number represents the number of objects affected by this severity (servers and switches).
You can define event severity and filter events in the Config view Events table.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
4.6 Traffic Map
The Traffic Map dashboard shows the normal traffic versus congested traffic distributed on switch tiers. This view, together with the Top N Congestion dashboard, give you a full status of the traffic congestion of the fabric.
Version 5.2
Four double bars represent the transmitted bandwidth (normalized transmit data) and normalized congested bandwidth (CBW), both measured in bytes/sec with minimum, average, and maximum bandwidth values.
The percentage of total theoretical bandwidth (TBW) is calculated based on the underlying
InfiniBand technology (SDR, DDR, QDR, FDR or EDR). The speed can be viewed when checking the ports.
•
The vertical axis shows the following:
•
Bandwidth (BW) is represented by a green bar and is measured in %
•
Congested Bandwidth (CBW) is represented by a red bar and is measured in %
•
Minimum, average, and maximum bandwidth are represented in each bar by a subset color
•
The horizontal axis represents the tiers.
The bottom of the dashboard represents the tier-related transmitted traffic, which is divided into four segments by measurement ports:
•
Tier 1 – represents the traffic injected by all adapters
•
Tier 2 – represents the traffic sent from the edge switches to the core of the fabric (in case of a single Director switch, this tier indicates traffic utilization inside the Director between the line and fabric boards)
•
Tier 3 – represents the traffic sent from the core to the edge switches
•
Tier 4 – represents the traffic sent from the edge switch to the adapters
NOTE: The illustrations at the bottom of the tiers show a four-tier topology:
Server [tier 1] Switch [tier 2] Director Switch [tier 3] Switch [tier 4] Server.
93
Mellanox Technologies Confidential
Version 5.2
4.6.1
Fabric Dashboard for On Screen Status Monitoring
Bandwidth Bars
The Bandwidth graph shows how traffic is traversing the fabric and how traffic is being transmitted between the servers. For example, the following considerations could be evaluated:
•
The size of the difference between Max Bandwidth and Min Bandwidth.
•
The traffic that is flowing in the middle tiers and whether it would be more efficient to move the traffic to the edges to save the uplinks.
Bandwidth levels are measured in percentages, as shown:
4.6.2 Congestion Bars
The Congestion graph shows where congestion starts. For example, the following considerations could be evaluated:
•
If congestion is in the first or second tier, there is probably a routing problem
•
If there is no red bar, it means that there is no congestion or no routing problems
Congestion levels are measured in percentages, as shown:
4.7
94
Top N Congested Nodes by Rx/Tx Bandwidth
The Top N Congested Nodes by Rx or Tx Bandwidth dashboard shows the top congested nodes, classified top-down according to the defined Transmit (Tx) or Receive (Rx) bandwidth.
Bandwidth is measured as congestion bandwidth rate (CBW) by percentage.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
For Tx, congestion is measured by N HCA ports.
•
For Rx, congestion is measured by N switch ports connected to HCAs.
N can be 5, 10, 15, or 20.
Figure 13: Top N Congested Nodes by Bandwidth - Bar View
Version 5.2
Figure 14: Top N Congested Nodes by Bandwidth - List View
The following table describes the options available on this dashboard.
Mellanox Technologies Confidential
95
Version 5.2 Fabric Dashboard for On Screen Status Monitoring
Table 21: Top N Congested Nodes by Rx/Tx Bandwidth
Options
Bar View
List View
Drop-Down Menu
RCBW
TCBW
More…
Description
Shows the top N congested nodes as a bar graph.
X axis shows the rate as a percentage.
Y axis shows the congested Node (server) name.
Shows the top N congested nodes as a list.
Each congested node is shown in a row with the name of the node and its picture. It also shows the bandwidth rate.
Default: 10 nodes
Receive Congested Bandwidth (percentage)
Transmit Congested Bandwidth (percentage)
Jumps to the Traffic table in the Managed Devices view.
4.8 Top N Logical Servers by Rx/Tx Traffic
The Top N Logical Servers by Rx/Tx Traffic dashboard shows the top Logical Servers classified top-down in a pie shape representation according to bandwidth distribution by N
Logical Servers.
Options: Transmit bandwidth (Tx) or Receive bandwidth (Rx).
N can be 5, 10, 15, or 20.
Figure 15: Top N Logical Servers by Traffic - Pie View
96
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Figure 16: Top N Logical Servers by Traffic - List View
Version 5.2
4.9
The following table describes the options available on the dashboard.
Table 22: Top Logical Server by Bandwidth
Options
Pie view
List view
Drop-Down Menu
RxBW
TxBW
More…
Description
Shows the top N Logical Servers as a pie chart, with bandwidth in
MB/sec as a percentage for each Logical Server.
Shows the top N Logical Servers as a list.
Each Logical Server is shown in a row with the name of the node and bandwidth rate in MBps.
Default: 10 Logical Servers
Receive Bandwidth (MB/sec)
Transmit Bandwidth (MB/sec)
Jumps to the Traffic table in the Managed Devices view.
Top N Alarmed Nodes
The Top Alarmed Nodes panel shows the top nodes with alarms classified top-down.
Alarmed nodes are measured according to:
•
Severity - only the top nodes, in order of severity:
•
Critical
•
Minor
•
Warning
•
Normal
•
Alarm - numbers
97
Mellanox Technologies Confidential
Version 5.2 Fabric Dashboard for On Screen Status Monitoring
N can be 5, 10, 15, or 20.
Figure 17: Top Alarm Nodes - Bar View
Figure 18: Top Alarm Nodes - List View
Figure 19: Top N Alarmed Nodes - Bar View
98
The following table describes the options available on the dashboard.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Table 23: Top N Alarmed Nodes
Options
Bar view
List view
Drop down menu
More…
Description
Shows the top N alarmed nodes as a bar graph.
X axis shows the number of alarms.
Y axis shows the names of the alarmed nodes (servers)
Shows the top N alarmed nodes as a list.
Each alarmed node is shown in a row with the name of the node and the number of alarms.
Default: 10 alarmed nodes
Jumps to the Alarms table in the Managed Devices view.
Version 5.2
4.9.1 Viewing Alarmed Nodes
You can see the alarmed nodes in the Views window in graphic format. You can also see the alarmed nodes in the Manage Devices window in table format, as shown in the following figures.
Mellanox Technologies Confidential
99
Version 5.2 Fabric Dashboard for On Screen Status Monitoring
100
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
5 Fabric Design
5.1
5.2
Overview of Fabric Design
UFM’s management infrastructure enables fabric monitoring and performance optimization on the application-service level, rather than just at the individual port or device level.
The Design window allows you to define your fabric using higher level abstractions, such as network and virtual servers. Specifying the fabric as an abstraction layer allows you to view alerts, or to understand your performance behavior in terms of applications or services. The abstraction layer design also allows UFM to more optimally allocate fabric resources to serve the application/service needs. Using the relevant property window in the Design
window, you can add or modify the following objects:
•
Environment
•
Logical Server
•
Logical Server Group
•
Local Network
•
Global Network
•
Network Interface
•
Flow
Design Concept
When designing your fabric, it is recommended to create the objects in the following order:
1. Create an environment.
2. Create a network.
3. Create logical servers.
4. Connect logical server to a network with a logical server interface.
5. Assign compute nodes and memory using the Logical Server wizard.
6. Create flows between logical network interfaces.
7. Configure QoS for networks, interfaces, and flows.
Version 5.2
Mellanox Technologies Confidential
101
Version 5.2
The following figure represents the design concept.
Figure 20: UFM Design Concept
Fabric Design
5.3 Designing the Fabric
The Design view allows you to:
•
Manage the fabric according to specific needs (for example: business needs).
•
Enable Fabric partitioning and setting QoS policy.
•
Automate configuration and change management.
102
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
5.3.1 Design Window
Click the Design tab and the Design window opens.
The following figure shows an example of the Design window within a selected environment. The environment contains two Logical Servers connected to two different
Networks.
Figure 21: Design Window in Selected Environment
Version 5.2
Table 24: Design Window Features
Callout
1
2
3
4
5
6
7
8
Description
Main Tabs / Categories / Navigator
Drop-down Menu
Choose the top view (all environments) or a specific environment.
Left pane component bar
Drag and Drop the icons.
Main area / Drop target area /Map view
Internal structure
Shows contents of the selected object in the map.
Properties of the selected object in the map.
Common tasks
Icons to view properties and perform actions on the selected object.
Expand icon to view the Events and Port counters.
5.3.1.1 Design Window Toolbar
Use the Design window toolbar buttons to design and view your fabric.
Figure 22: Design Window Toolbar
The Toolbar buttons are described in the following table:
Mellanox Technologies Confidential
103
Version 5.2 Fabric Design
Table 25: Design Window Toolbar Icons
Icon Name
Network View
Function
Moves one level up
Use to create flows.
Pan Moves a group of objects across the window.
Select Selects a specific object.
Zoom in Rectangle Magnifies the selected rectangular area. Click-and-drag the mouse pointer to create a zoom rectangle.
Magnifies the area. Zoom in
Zoom out Reduces the view (when in zoom in state).
Fit to view Reduces the view to fit to the window’s main area. When in zoom in state, you can make the picture fit into the map view.
5.3.2 Component Drag and Drop Bar
The icons in the Component Drag and Drop bar represent the Network components. All these icons can be dragged and dropped into the Map view area (main pane). The function of each component bar icon is described in the following table:
Table 26: Function of Component Bar Icons
Icon Description
For dragging a new environment into the Map view.
For dragging a new global network into the Map view.
For dragging a new local network into the Map view.
For dragging a new logical server group into the Map view.
For dragging a new logical gateway group into the Map view.
104
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Icon Description
For clicking and drawing a new NIC connector between objects into the
Map view and connecting a logical server with a network.
Version 5.2
5.3.3 Internal Structure Pane
The Internal Structure pane shows the real-time state of your fabric design as a graphic map.
The contents of the Internal Structure pane vary according to the component selected in the
Map view. When a Logical Server is selected, the Internal Structure pane appears as follows:
Figure 23: Internal Structure Pane
5.4 Creating an Environment
To create a new Environment
1. In the Component area (left pane), select the Environment icon ( the main area.
) and drag it into
Mellanox Technologies Confidential
105
Version 5.2 Fabric Design
5.5
2. In the Name field, enter a name for your new Environment.
3. Optional: In the Description field, enter a description for your new Environment.
4. Optional: In the Event Script field, enter the name of the script to be called on an environment event.
5. Click Apply or OK. Your new Environment is created. You can see it in the main window, and by clicking the Environments drop-down menu in the left panel.
6. Create Environment objects such as a new network. Double-click an Environment to add
Local Networks, Logical Servers, and interfaces. See Creating a Network
After the objects are created, you can edit them or delete them by right-clicking the object and selecting the appropriate option. Each change is saved in the model in real-time.
Creating a Network
This section describes how to create a new global or local Network in the Design window.
To create a global or local network
1. In the Design tab, ensure that you are within the Environment or Global Network in which you want to create a Network. Note that in UFM default state, no Networks appear in the main area.
2. In the Component area (left pane), select the Global Network icon ( ) or the
106
Local Network icon ( ) and drag it into the Map area, or double-click an
Environment or Global Network to add a local network.
3. Enter the parameters described in the Network Parameters table that follows the procedure.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
4. (Optional) In the QoS tab, set QoS parameters for the network. See Setting/Editing QoS
5. (Optional) In the IP Services tab, set IP Services parameters for the network. See
Setting/Editing IP Services (Optional)
6. (Optional) In the Filter tab, you can set auxiliary parameters to filter services and hosts.
7. Click Apply or OK. Your new Network is created. You can see it in the main window and by clicking the Environments drop-down menu on the left panel.
When creating a Network or Network Interface, the following SM files are edited as a result of Network and Network Interface configuration:
•
partitions.conf: The partitions.conf file is changed when a Network is created, deleted, or modified.
•
qos-policy.conf: The qos-policy.conf file is changed when a Network, Logical Server, or
Network Interface is created, deleted or updated.
After you create a Network, you can attach it to a logical server, which will create a network interface that will enable you to use partitions and configure QoS on logical server members.
107
Mellanox Technologies Confidential
Version 5.2 Fabric Design
Double-click an Environment or Global Network to add Logical Servers and interfaces. For
information about creating a new Logical Server, see Creating a Logical Server Group
Table 27: Network Parameters
Field
Environment
Name
Description
Pkey
Event Script
IP Configuration
Network Mask
Default Gateway
Description
Read only. The name of the Environment to which the Network belongs.
Enter the name for your new Network.
Optional: enter a description for your new Network.
Enter the Pkey of the network that you want to define (in hexadecimal format).
If the Pkey is not specified during the network definition process (in the
Network Configuration window or by using the Logical Server wizard),
UFM will select the best available Pkey for the Network.
Select the default membership.
•
Limited/partial: Members with limited membership cannot communicate with other members with limited membership, but communication is allowed between every other combination of membership types.
•
Full (default): Members with full membership may communicate with all hosts (members) within the network/partition.
The name of the user-defined script to be called on a Network-related event.
[Optional] Enter the Network IP subnet.
NOTE: Using class A network addresses (255.0.0.0) for Logical
Networks may cause high memory consumption.
Enter the Network mask.
Enter the Network default gateway.
NOTE: For those who use Mellanox gateway systems, for proper system functionality, disable the automatic partitioning by changing the attribute gateway_port_partitioning = none
in the
/opt/ufm/files/conf/gv.cfg
configuration. Restart UFM for the change to take effect.
5.5.1 Setting/Editing QoS (Optional)
UFM allows fabric traffic prioritization by providing four pre-defined Service Levels (SL).
Each SL defines different queuing priority of the traffic in the fabric. The SL is configured centrally and is applied to all fabric ports. Prioritization occurs when traffic with different SL levels is competing for bandwidth on the same port at the same time.
NOTE: To benefit from QoS capabilities in the fabric, please set the qos in the
/opt/ufm/files/conf/opensm/opensm.conf
to TRUE.
108
QoS is provisioned to the SM via the partitions.conf and qos-policy.conf configuration files. You cannot remove or manually modify these files.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
QoS parameters are associated with network interfaces.
The UFM software defines the attributes in the qos-policy.conf file. When each port group is associated with logical server members, a QoS-level is associated with the QoS parameters set, and the matching rule represents the network interface object. The MTU limit is defined for the network object.
A partition is specified by the network. The QoS is generally defined by the network but can be overwritten by the network interface for the specific logical server, providing a more granular definition of QoS for the specific Logical Server.
NOTE: Before setting QoS, ensure that you have selected the preferred algorithm. If you do not pre-select the algorithm, UFM automatically applies QoS settings to the default algorithm: Minhop. For more information about configuring the algorithm, see
To set QoS Parameters:
1. In the New Network dialog box, or in the Network Properties dialog box (when editing), select the QoS tab.
2. Edit the QoS parameters as described in the table that follows.
3. Click Apply or OK to save your changes.
The QoS settings are applied to the QoS service that you selected as an algorithm, or to the default algorithm (Minhop).
Table 28: QoS Properties
QoS Field
Average Load
MTU Limit
Description
The average traffic load to the typical destination per Logical Interface in
MB per second.
The Maximum Transmission Unit (number of bytes) is defined for Network object.
109
Mellanox Technologies Confidential
Version 5.2
5.5.2
QoS Field
Service Level
Rate Limit
Fabric Design
Description
Select a predefined service level:
•
SL0 (Default)-Strict High Priority queue in which the traffic will always be served first in the fabric. The SL is suitable for low-latency and/or high-priority applications.
•
SL1-High priority queue, 70% bandwidth.
•
SL2-Medium priority queue, 20% bandwidth.
•
SL3-Low priority queue, 10% bandwidth.
Rate Limit in Mbits per second. This value is converted to a standard
InfiniBand enumerator (rate_limit, which has fixed values), and provisioned to the SM via the partitions.conf and qos-policy.conf files.
NOTE: Following an update of the QoS Properties of Management Network (default pkey), an opensm restart is required for the changes to take effect.
Setting/Editing IP Services (Optional)
UFM enables you to specify one of the following IP distribution (configuration) methods:
•
Static – The UFM Agent creates a new interface with static IP addresses.
•
External – UFM does not create an interface on hosts. Host configuration is user-defined.
To set IP Services Parameters:
1. In the New Network dialog box, or in the Network Properties dialog box (when editing), select the IP Services tab.
110
2. Select the IP Configuration Method.
3. Click Apply or OK to save your changes.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
5.5.3 Deleting a Network
To delete a Network:
1. Select the object.
2. Do one of the following:
•
Press the Delete key on your keyboard.
•
Click Network from Common Tasks (right panel).
A warning appears asking you if you want to delete the Network.
3. Click Yes to confirm.
5.6 Creating a Logical Server Group
The Logical Server object allows you to define a logical server or cluster, allocate resources
(both logical and physical), and add network interfaces to connect logical servers to the network (partitioning). The resources automatically allocated by UFM inherit the properties of the network in which they reside. Specific resources may be allocated manually.
When creating a logical server group:
•
The UFM server machine cannot be defined as a logical server resource.
•
UFM does not allow the UFM Server to be part of central device management actions, such as reboot, shutdown, and software upgrade.
•
Double-click an environment or global network to add logical servers and interfaces. In
UFM default state, no networks appear in the main area.
To create a Logical Server Group:
1. Select the Design tab.
Version 5.2
2. In the Component area (left pane), click the Logical Server Group icon ( and drag it in the main area.
)
Mellanox Technologies Confidential
111
Version 5.2 Fabric Design
3. In the Name and Description window of the New Logical Server Wizard:
•
In the Name and Description fields, enter a name for the new Logical Server and an optional description.
•
Optional: In the Event Script field, enter the name of the user-defined script to be called on a Network-related event.
4. Click Next.
112
5. In the Allocation Method window of the wizard, select the allocation method:
•
Assign Systems Manually – Select to manually assign the physical servers that will be part of the Logical Server. Go to step 6.
•
Assign Systems Automatically – Select to enable UFM to assign the servers that will be part of the Logical Server.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
In the Units field, specify the number of physical servers to be allocated to Logical
Servers, then click Filter.
Version 5.2
Optional: Enter the minimum server requirements and constraints for all servers assigned automatically, then click OK or Apply.
•
CPU Type - device CPU type (any, ia32, ia64, i386, PPC, x86_64).
•
# of CPUs - number of CPUs.
•
Speed - required CPU speed (in MHz).
•
Min Memory - minimum memory required (in MB).
6. Go to step 8.
7. Click Next. The Assigning Computers window of the wizard appears.
Select any unassigned computers you would like to assign to the new Logical Server.
You can select and assign more than one computer if you are defining a Logical Server
Group.
You can skip the selection and create an empty Logical Server. The physical servers can be added at a later stage.
8. Click Next.
113
Mellanox Technologies Confidential
Version 5.2 Fabric Design
9. In the Network Interfaces window of the wizard:
•
Optional: Create and add a new global or local network by clicking or .
•
To select a network from the Available Networks list, use the Down Arrow button
to connect (add) a server to the selected Network.
•
To remove a server from the selected Network, select a Network from the Network
Interfaces list and use the Up Arrow button .
•
Click the Edit button and edit the Network Interface general and QoS parameters. For
more information, see Setting/Editing Network Interface Properties
10. Click Finish. Your new logical sever is created. You can see it in the main window and by clicking the Environments the drop-down menu in the left panel.
CAUTION: When creating a Network interface via the Logical Server wizard, make sure that the IP Address range is greater than the number of ports in the Logical Server.
You cannot modify the parameter after the Network Interface is created.
You can modify this parameter only when the new Network Interface is created. If you do not use the wizard to create a Logical Server, this parameter is calculated automatically by UFM and cannot be changed.
If you need to increase the number of Logical Server members, delete the Network
Interface and create a new one after new members are allocated.
5.6.1
114
Expected Outcome - Assigning a Logical Server
When the user assigns a Logical Server to a Network, the following actions take place:
•
Logical:
•
All hosts are configured for the specific partition and QoS, which are represented by the network. A Logical Server represents the group of hosts.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
5.6.2
5.6.3
•
Physical:
•
The InfiniBand partition (PKey) and QoS parameters are provisioned to every port of the Logical Server host by the SM.
•
When a partition is defined on the host’s port, the IP interface is created on top of the partition and is implemented by the agent.
Setting/Editing Logical Server Properties
After a Logical Server is created, you can edit it.
To edit the Logical Server
1. Select a logical server and do one of the following:
•
Double-click the logical server.
•
Right-click the logical server and select the Properties option.
•
Select Properties from Common Tasks.
2. Do any of the following:
•
Edit the description field (Name and Description).
•
Add and/or remove devices.
•
Assign Networks to Network Interfaces or remove them.
•
Create a new Network.
For more information, see Tasks on Devices and Ports
Deleting a Logical Server
To delete a Logical Server
1. Select the object.
2. Do one of the following:
•
Click the Delete key on your keyboard.
•
Click Delete Logical Server from Common Tasks (right panel).
A warning appears asking you if you want to delete the Logical Server.
3. Click Yes to confirm.
Version 5.2
5.7 Creating a New Network Interface
A network interface can be created using one of following:
•
Use the Logical Server Wizard under Interfaces and attach it to the Logical Server(s).
See Creating a Logical Server Group
•
Double-click an environment or global network to add network interfaces.
•
Drag the New Interface icon from the Component area, as described in the following procedure.
115
Mellanox Technologies Confidential
Version 5.2
To create a Network Interface
1. Select the Design tab.
In the UFM default state no Networks appear in the main area.
Fabric Design
5.7.1
5.7.2
2. In the Component area (left pane), click the New Interface icon ( ).
3. In the main area, click the desired Network, then select the Logical Server for which to create the interface. (You can also first click the Logical Server and then the Network.)
A line is drawn between the two and the new interface is created.
Expected Outcome - Assigning a Network or Network Interface
When creating a Network or Network Interface, the following SM files are edited as a result of Network and Network Interface configuration:
•
partitions.conf: The partitions.conf file is changed when a Network is created, deleted, or modified.
•
qos-policy.conf: The qos-policy.conf file is changed when a Network, Logical Server, or
Network Interface is created, deleted or updated.
NOTE: If the pkey is not specified during Network definition process (Network
Configuration window or Logical Server Wizard), UFM will select the best available pkey for the Network.
If the NIC QoS properties are not set manually, the NIC will inherit all QoS properties from the Network.
Setting/Editing Network Interface Properties
To edit Network Interface properties
1. Select a Network Interface and do one of the following:
•
Double-click the network interface.
•
Right-click the network interface and select the Properties option.
•
Select Properties from Common Tasks.
2. Fill in, edit, or view the parameters listed in the following table.
116
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
5.7.3
3. Click Apply or OK to save your changes. New properties are applied.
Table 29: Network Interface (NIC) Properties Parameters
Option
Name
Description
Network
Membership
Event Script
Description
Shows the name of the two objects on either side of the interface (readonly field)
Optional: Enter the desired description for the Network Interface.
Shows the name of the Network to which the interface belong (read-only field)
Select the default membership:
•
Limited/partial - Members with limited membership cannot communicate with other members with limited membership but communication is allowed between every other combination of membership types.
•
Full - Members with full membership can communicate with all hosts
(members) within the Network/partition.
Name of user-defined script called on a Network Interface related event
Editing Network Interface QoS Properties
Network Interface QoS properties are inherited from the Network QoS properties. A different QoS can be provided for a specific Logical Server connected to the Network.
Select the preferred routing algorithm before setting QoS properties. If you do not pre-select the algorithm, UFM will automatically apply QoS settings to the default algorithm, Minhop.
To set Network Interface QoS Parameters:
1. Selected the preferred algorithm.
2. In the Network Interface dialog box, select the QoS tab.
117
Mellanox Technologies Confidential
Version 5.2 Fabric Design
5.7.4
3. Edit the QoS parameters as required. For more information about QoS parameters, see
Setting/Editing QoS (Optional)
4. Click Apply or OK to save your changes. The QoS settings are applied to the QoS service that you selected as an algorithm, or to the default algorithm (Minhop).
Deleting a Network Interface
To delete a Network Interface:
1. Select the link.
2. Do one of the following:
•
Click the Delete key on your keyboard.
•
Click Delete Network Interface from Common Tasks (right panel).
A warning appears asking you if you want to delete the Logical Server.
3. Click Yes to confirm.
5.8
118
Creating a New Logical Gateway Group
A Logical Gateway Group can be created using one of following:
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
Double-click an environment or the logical gateway group to add gateway groups.
•
Drag the Logical Gateway Group icon from the Component area, as described in the following procedure.
To create a Logical Gateway Group
1. Select the Design tab.
In the UFM default state no Gateway Groups appear in the main area.
Version 5.2
5.8.1
5.8.2
2. In the Component area (left pane), click the New Interface icon ( ).
3. In the main area, click the desired Gateway, then select the Logical Server for which to create the interface. (You can also first click the Logical Server and then the Gateway.)
A line is drawn between the two and the new interface is created.
Expected Outcome - Assigning a Logical Gateway Group
When creating a Gateway or a Gateway Group, the following SM files are edited as a result of Network and Network Interface configuration:
•
partitions.conf: The partitions.conf file is changed when a Network is created, deleted, or modified.
•
qos-policy.conf: The qos-policy.conf file is changed when a Network, Logical Server, or
Network Interface is created, deleted or updated.
NOTE: If the pkey is not specified during Network definition process (Network
Configuration window or Logical Server Wizard), UFM will select the best available pkey for the Network.
If the NIC QoS properties are not set manually, the NIC will inherit all QoS properties from the Network.
Setting/Editing Logical Gateway Group Properties
To edit Logical Gateway Group properties
1. Select a Logical Gateway Group and do one of the following:
•
Double-click the Logical Gateway Group.
•
Right-click the Logical Gateway Group and select the Properties option.
•
Select Properties from Common Tasks.
2. Fill in, edit, or view the parameters listed in the following table.
119
Mellanox Technologies Confidential
Version 5.2 Fabric Design
5.8.3
5.9
120
3. Click Apply or OK to save your changes. New properties are applied.
Table 30: Logical Gateway Group Properties Parameters
Option
Name
Description
Environment
OS Type
Icon
URL
Event Script
Description
Shows the name of the two objects on either side of the interface (readonly field)
Optional: Enter the desired description for the Network Interface.
Shows the name of the Environment
Shows the operating system
Name of user-defined script called on a Network Interface related event
Deleting a Logical Gateway Group
To delete a Logical Gateway Group:
1. Select the link.
2. Do one of the following:
•
Click the Delete key on your keyboard.
•
Click Delete Logical Gateway Group from Common Tasks (right panel).
A warning appears asking you if you want to delete the Logical Gateway Group.
3. Click Yes to confirm.
Configuring Flows (Advanced License Only)
A flow defines traffic from a source network interface to its destination. You can create, modify, and delete flows.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
5.9.1 Creating a Flow
You define a flow from the source network interface.
To create a flow
1. In the Network Interface dialog box, choose the Flows tab.
Version 5.2
5.9.2
2. In the toolbar, click the Add Flow ( ) button.
3. Set parameters as described in the next section.
Setting/Editing Flow Properties
You can modify existing flows from the source network interface.
To set flow properties
1. In the Network Interface dialog box, choose the Flows tab and double-click the flow to edit.
Mellanox Technologies Confidential
121
Version 5.2 Fabric Design
122
2. Set the general Flow properties, as described in the table that follows.
3. Click OK or Apply.
Table 31: General Flow Properties
Parameter
Name
Description
Protocol
Service Port
Destination
Description
Unique flow name.
(Optional) – a user-defined description for the flow.
The protocol used by the flow traffic:
•
Any
•
RDS
•
SDP
•
ISER
The source and destination ports for the selected protocol. A value of
0 indicates any port.
•
Select ANY or a specific Logical Server Group connected to the same Network.
•
When ANY is selected, the flow is defined only by the source network interface.
•
When the Destination and Source are identical, this defines internal traffic.
•
The destination environment can be changed only if the flow passes through the global network.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
5.9.3 Editing Flow QoS Properties
By default, Flow QoS parameters are inherited from the Network Interface QoS settings.
However, you can define the QoS parameters for a specific Flow that override the Network
Interface QoS settings.
To set Flow QoS properties
1. In the Flow Properties dialog box, select the QoS tab.
Version 5.2
5.9.4
You can specify two sets of QoS parameters: Source to Destination and Destination to
Source. When a flow connects two different Logical Server or VM Groups, different QoS parameters will be applied to traffic from Source to Destination and from Destination to
Source. When Destination is ANY, or when Destination is the same as Source, the
Destination to Source QoS settings are not relevant.
For more information about QoS parameters, see Setting/Editing QoS (Optional)
2. Click OK or Apply.
Deleting a Flow
To delete a Flow:
1. Select the Flow in Network Interface Flow Tab.
2. Do one of the following:
•
Press the keyboard Delete key.
•
Click Delete Network Interface from Common Tasks (right panel).
123
Mellanox Technologies Confidential
Version 5.2
A warning appears asking you if you want to delete the Flow.
3. Click Yes to confirm.
Fabric Design
5.10 Partitioning
Partitioning enforces isolation of the fabric. When you create a Network or Network
Interface by using the Logical Server Wizard or by using a link from the component area
(left panel), a partition is created.
When using the link from the component area, you can only create a single partition.
The Logical Server Wizard allows you to create multiple partitions and to define the pkeys for each one of these partitions. If you do not specify the PKey, UFM will apply a default
PKey.
The Logical Network in UFM is associated with the partition. When the Logical Server is connected to a Network and a Network Interface is created, devices (members of the Logical
Server) are added to the specified partition.
The default partition is created on all managed devices. Devices that are running an SM, all switches, and the 10GbE gateway are added to the default partition with full membership. By default, all the HCA ports are also added to the default partition with FULL membership.
Partitioning is provisioned to the Subnet Manager via the partitions.conf configuration file, which cannot be removed or modified.
NOTE: For those who use Mellanox gateway systems, for proper system functionality, disable the automatic partitioning by changing the attribute gateway_port_partitioning = none
in the
/opt/ufm/files/conf/gv.cfg
configuration. Restart UFM for the change to take effect.
If required, you can add an extension to the partitions.conf file that is generated by UFM.
You can edit the file, /opt/ufm/files/conf/partitions.conf.user_ext, and the content of this extension file will be added to the partitions.conf file. Files synchronization is done by UFM on every logical model change. However, it can also be triggered manually by running the
/opt/ufm/scripts/sync_partitions_conf.sh script. The script merges the
/opt/ufm/files/conf/partitions.conf.user_ext file into the
/opt/ufm/files/conf/opensm/partitions.conf file and starts the heavy sweep on the Subnet
Manager.
NOTE: The maximum length of the line in the partitions.conf file is 4096 characters.
However, to enable long PKeys, it is possible to split the pkey membership to multiple lines:
IOPartition=0x4, ipoib, sl=0, defmember=full : <port-guid1> , <port-guid2> ;
IOPartition=0x4, ipoib, sl=0, defmember=full : <port-guid3> , <port-guid4> ;
124
The partitions.conf.user_ext uses the same format as the partitions.conf file. See SM
Partitions.conf File Format for the format of the partitions.conf file.
For example, to add server ports to PKey 4:
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
IOPartition=0x4, ipoib, sl=0, defmember=full : 0x8f10001072a41;
CAUTION: The content of the extension file,
/opt/ufm/files/conf/partitions.conf.user_ext, is not validated, and might cause a misconfiguration of the Subnet Manager or it might misfunction. It is recommended to avoid using the extension file. If the extension option is used, it is strongly recommended to check the opensm logs and check the PKeyTable for specified ports using the command /opt/ufm/opensm/sbin/smpquery pkeys -G <GUID>
<port> .
5.11 InfiniBand Bonding on a Managed Host
Bonding between two ports of the same HCA is fully supported when the UFM Agent is running on the host. The UFM Agent learns the bond interface configuration that was created manually, and can then configure partitioning, meaning a p-key-based interface, on top of the bond interface.
Example:
1. Create bonding on ib0 and ib1 devices.
Define bondx as the ufm_mcast_ifaces in the ufma.ini file: ufm_mcast_ifaces=eth0,bondx
2. Create the logical network.
3. Create the logical server and allocate the host.
4. Connect the logical server to the logical network.
Two ib devices with pkey=1 are created immediately after connecting between the logical network and the logical server group.
5. UFM Agent automatically creates bonding of ib0.XXXX and ib1.XXXX.
When the UFM Agent is not installed, UFM learns and presents bond slave interfaces as separate interfaces, i.e. bonding is not indicated. You can manually bond the interfaces after creating the logical network.
Mellanox Technologies Confidential
125
Version 5.2
6 Fabric Discovery and Views
6.1
6.2
Fabric Discovery and Views
Overview of Fabric Discovery and Views
The View window shows the fabric, its topology, the fabric elements and properties. UFM performs automatic fabric discovery, and displays the fabric elements as well as the connectivity between the elements. In the View window you can see how the fabric is organized and the elements making up the fabric (e.g., switches and servers). You can also drill down into specific fabric elements to view the properties of the element.
In addition, you can group physical elements of the fabric into racks to match the physical arrangement of the fabric.
The following views are available in the View window from the sub-selection trees:
•
Environments – sum of Logical Servers.
•
Logical Servers – entails all the computers.
•
Networks - includes global and local networks.
•
Sites - includes all devices and groups.
•
Devices – All switches and servers in the fabric. The ports for each device are listed in the Internal Structure area.
•
Gateways – gateways and bridges
•
Groups – racks and other groups; lists all the devices.
Configuring Polling Interval for the Fabric Topology Display
The default polling interval for updating the UFM fabric topology display is 30 seconds. The polling interval is displayed in the Interval drop-down box in the top-right corner of the main window.
126
You can change the polling interval for the current session at any time in the main window.
You can also change the default polling interval in the gv.cfg configuration file.
For large fabrics, it is recommended to increase the polling interval.
To change the polling interval for the current session
•
In the main UFM window Interval drop-down box, select the required polling interval.
Polling interval options are 15, 30, 45, 60, 90, 120, 180, 240, 300 seconds.
To change the default polling interval
•
In the /opt/ufm/conf/gv.cfg file, set the ui_polling_interval command.
The default is 30 seconds. Options are 15, 30, 45, 60, 90, 120, 180, 240, 300 seconds.
The new default interval takes effect the next time that you begin a UFM GUI session.
Example: ui_polling_interval = 30
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
6.3 Accessing the View Window
Whenever you open or refresh the View window, UFM performs full fabric discovery.
To access the View window
Version 5.2
1. Click the View tab .
2. Select the object on the left panel. The contents of the object are displayed in the View window, as shown.
D
E
F
Callout
A
B
C
Section
Devices Label
Fabric Tree
Map View
Description
Shows devices by Name or IP Address or GUID.
Hierarchy includes:
•
Environments (contains Logical Servers)
•
Networks
•
Sites (contains switches, gateways, servers, racks)
Displays the managed fabric as a map:
• server. indicates the device with the UFM
Map and Table tabs
Internal Structure
Properties
•
indicates the device with the standby UFM server.
•
indicates the device with the Subnet
Manager.
•
indicates a device with the UFM Agent installed.
•
indicates that there are alarms for the device.
Displays the contents of the selected object.
Displays the properties of the selected object
127
Mellanox Technologies Confidential
Version 5.2
6.3.1
Callout
G
Section
Common Tasks
Fabric Discovery and Views
Description
Provides a menu of tasks that can be performed on the selected object.
View Window Toolbar
Use the View window toolbar buttons to view your fabric.
Figure 24: Design Window Toolbar
128
The Toolbar buttons are described in the following table:
Table 32: View Window Toolbar Icons
Icon Name
Reload Host Names
Function
Refreshes host names and disables Edit mode.
Clear All
Layout
Clears the Map View and hides all the environments. To display them back, click the Reload Host Names icon.
Select a Layout from the drop-down menu:
Grid Layout
Uniform length edges
Hierarchical Layout
Loads the user layout from the server
Saves the user layout
Imports a layout from a local file
Pan
Select
Zoom in Rectangle
Zoom in
Zoom out
Fit to view
New Rack
Exports a layout to a local file
Moves a group of objects across the window.
Selects a specific object.
Magnifies the selected rectangular area. Click-and-drag the mouse pointer to create a zoom rectangle.
Magnifies the area.
Reduces the view (when in zoom in state).
Reduces the view to fit to the window’s main area. When in zoom in state, you can make the picture fit into the map view.
Creates a new rack/group.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Icon Name
Expand/Collapse
Internal Links
Hide Links
Search
Collapse/Expand all
Servers
Function
Expands and collapses racks.
Shows/hides internal rack links.
Hides/shows rack’s links.
Best used for large fabrics.
Searches for a devise in the fabric.
Collapses all the grouped servers.
NOTE: For optimal usage it is recommended to group hosts into Racks in case of large fabrics (hundreds of nodes or more). Racks can automatically be created per switch and containing all the hosts connected to this switch by running the arrange_in_racks.py
script (part of the UFM SDK).
Doing so, decreases dramatically the number of graph nodes displayed. For example, if you group 500 hosts into a rack, instead of calculating position of 500 graph nodes, the layout only have one position to calculate.
Version 5.2
6.4
6.5
Mapping View Default Layout
The previous saved layout is set as the default layout when opening the GUI. The layout can be overridden at any given time. Before saving a layout for the first time, the default layout is determined by the number of devices (hosts + switches) in the fabric.
•
For small fabrics (<=SMALL_FABRIC_MAX_BOUNDARY* nodes), the default layout is
`Uniform Length Edges`
•
For large fabrics (>SMALL_FABRIC_MAX_BOUNDARY* nodes), the default layout is
`Grid`
* SMALL_FABRIC_MAX_BOUNDARY
parameter is configurable at
/opt/ufm/conf/gui.properties
(default = 324).
Edit Mode in View Default Layout
UFM provides a new mode called Edit in the View default layout. It enables the user to isolate and work on a specific node in the fabric. Edit mode does not respond to any topology changes, all the changes will be displayed once exiting the mode.
Once entering the Edit mode, the user can right click the device and perform the following actions:
129
Mellanox Technologies Confidential
Version 5.2
•
Show the device in the Map view
Fabric Discovery and Views
•
Show the devices neighbors in the Map view
130
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
Hide the device from the Map view
Version 5.2
6.6
•
Clear All devices ( )
To exit the Edit mode, click the Reload Host Names button ( ).
Enhanced Event Management (Advanced License Only)
You may group your devices into the following groups:
•
Alarmed Devices
The device is automatically added to the Alarmed Devices group. Devices cannot be manually added to the group, only removed from it. The group is permanent, it cannot be created or deleted by the user.
•
If the auto_remove_from_alerted parameter is set to 'yes', all the devices are auto-removed from the group once the alarm is no longer alive
•
If the auto_remove_from_alerted parameter is set to 'no', the devices are not auto-removed from the group. You must remove them manually from the group after restart
Once entering the Alarmed Devices group, the user can right click the device and perform the following actions:
•
Remove From Alarmed Group
•
Suppress Notifications
•
Suppressed Devices
Devices in this group will not generate events (alarms). In the Suppressed Devices group you may manually add or remove devices. The group is permanent, cannot be created or
131
Mellanox Technologies Confidential
Version 5.2 Fabric Discovery and Views deleted by the user. Once a device in this group, it will be marked by a special icon in
6.7
GUI .
If the device was alarmed before it was suppressed, the alarm remains valid even while it is in the Suppressed Devices group. However, the alarms generated during the period that the device was in the group will not be shown after removing the device from the group even if they are still alive .
Once entering the Suppressed Devices group, the user can right click the device and perform the following actions:
•
Activate Notifications
Host Grouping
Host Grouping enables the user to create a cleaner view of the fabric by grouping together hosts connected to the same switch bur not in the rack. This feature is not available when working in Editing Mode.
The following are the Host Grouping capabilities:
•
Expand – Right click on the grouped hosts to expand them
•
Collapse – An action performed on a switch, causing an addition of its hosts to exist the group or create of new ones as needed
132
•
Expand/Collapse all (on the toolbar)
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
Switch to hyperlink – Reveals the port to port connections between each of the parenting
Version 5.2
6.8
6.8.1
6.8.2
switches and the hosts
•
Search – A device searched even if it is a part of the hosted group
•
Save/export/import the new layout
•
Tree Device Selection – Contains the selected computer
•
The Host Grouping indicates the hosts count and their severity. The severity displayed is the worst severity of the hosts in the group.
Features of the View window
The View window includes both logical and physical hierarchies of the fabric.
•
The left fabric tree shows each of the fabric objects by its hierarchy (Environments,
Networks, and Sites).
•
The main area shows the contents of the object selected on the tree selection, either in graphical format (Map view) or in tabular format (Table view).
•
The right side contains three dynamic panels: Internal Structure, Properties, and
Common Tasks. When a new object is selected on the map, the contents of all three dynamic panels are updated according to the newly selected object.
•
This window also enables you to add devices to a rack, remove devices from the rack, or view the rack’s contents (by double-clicking the rack).
Additional Tasks
You can perform the following additional tasks:
•
View the object’s properties by double-clicking it.
•
Add a device to a rack by right-clicking a device and choosing a rack from the menu or by clicking a rack from the toolbar. You may also delete racks.
•
Start an SSH session by clicking the SSH icon on the toolbar.
•
Edit objects (by double-clicking a task from Common Tasks) or delete them (by pressing the Delete key when a task is selected). Each update is saved in the model in real-time.
For more information, see Tasks on Devices and Ports
Devices
Each object has its own graphical representation. If the device is managed (a small screen icon appears over the device):
•
Each view (Table view and Map view) allows you to see the device’s status (event icons).
•
In Map view, you may also see the connections (represented by a line) between each of the devices and between devices and racks.
133
Mellanox Technologies Confidential
Version 5.2
6.8.3
6.8.4
Fabric Discovery and Views
•
You can select multiple devices by holding the Ctrl key down and clicking the devices.
Note that in Table view you can quickly add multiple devices to a rack, just by selecting the rows.
Map View
The Map view shows a representation in graphical format of the contents of the object selected.
Table View
The Table view allows you to view the selected object detailed name, status, description, and specific parameters.
Figure 25: Table Display in View Window
6.8.5
To filter the Table view according to a specific parameter:
1. Select the parameter from the drop-down menu.
2. Enter the desired value in the search field.
3. Click Clear to clear the search.
View Options
At the top of the left panel, there are several View options. Display the devices by name, by address, by GUID, or without labels at all.
The left tree, the map view, and the Internal Structure apply the relevant labels to the selected devices.
134
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
6.8.5.1 Viewing Rack Contents
After you have created a rack and added devices to it, you may view the rack’s contents by double-clicking it, as shown in the following figure:
Figure 26: Viewing the Rack’s Contents
Version 5.2
You can double-click the open rack or click the collapse icon on the toolbar to "close" the rack.
For information on creating racks, see Rack Grouping
Mellanox Technologies Confidential
135
Version 5.2
7 Managing Devices
7.1
7.1.1
7.2
Managing Devices
Overview of Managing Devices
The UFM Manage Devices window contains special views for centralized device management.
The Manage Devices window allows you to obtain information and perform group operations on devices. Information includes port status, alarms, traffic, and errors.
You may perform a wide variety of tasks or actions on a single device or multiple devices by using the right-click menu and Common Tasks panel, including configuring a single device or multiple devices, establishing an SSH session to the device, rebooting the device, shutting down the device, and adding the device to a logical server or removing it from the Logical
Server. You may also add devices to a rack, clear alarms, and perform firmware and software upgrades. Note that these tasks are available throughout the UFM application.
Device Management Agent
The Device Management Agent provides advanced management capabilities that depend on the host as a Network end point, including switches, servers, and gateways.
The Agent provides a set of management services and performs Network device IP discovery, learns the configuration, executes commands, and handles events.
The UFM server must use SSH-based communication to query host configuration service.
Device Management Agent and OFED use the 'hostname -s' command to set the hostname in the fabric node description. The hostname must be resolved at the DNS level or added to the
/etc/hosts. See the following open bug in RedHat: https://bugzilla.redhat.com/show_bug.cgi?id=319981
Accessing the Manage Devices Window
The Manage Devices view allows you to manage the switches and hosts (hardware)
To access the Manage Device window:
7.2.1
•
Click the Manage Devices tab .
Manage Devices Window
The Manage Devices window is different from the View window since the Manage Devices window shows all the data pertaining to physical devices in tabular format.
The left fabric tree and the right panels are the same as in the View category. The main area
(tab) shows the physical devices. The other tabs provide more information about the devices, including port status, alarms, traffic, and errors.
136
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Note that this window focuses on viewing and managing the physical components. Some logical features are available.
Figure 27: Manage Devices Window
Version 5.2
7.2.1.1 Manage Devices Window Display
The Manage Devices window shows you the devices contained within a specific object
(mostly hardware-related). This window displays in tabular form, and allows you to perform a set of tasks for each device or for multiple devices (if applicable).
•
Left Fabric tree is a hierarchical representation of the fabric and is identical to the one described in the View window. However, when selecting a logical object, for example logical devices, you can see the devices (computers) that are contained in that object.
•
Main area contains the lists of selectable physical devices or ports in tabular form. For example, if you select a specific environment, you will see the specific devices that are allocated to their logical servers. Each tab in this area displays additional device information.
•
Right panel allows you to perform the following tasks:
•
View the Internal Structure of a selected device.
•
Set or edit the Properties of a selected device.
•
Perform a set of actions (if applicable) by using the Common Tasks pane.
You can select multiple objects on which to perform actions.
7.2.1.2 Selecting Ports and Devices
The Manage Devices Window displays all the information in tabular form, allowing multiple selection of devices. When selecting a single device, multiple devices or ports, the following are displayed:
•
Left Fabric tree shows the device selection highlighted in blue.
137
Mellanox Technologies Confidential
Version 5.2 Managing Devices
•
Main area shows the device(s) in table format, the ports for the device (port tab), the status of the specific device(s) (alarms tab), the traffic that the ports are generating (traffic tab), and the source of errors on each port.
•
Internal Structure shows the list of ports and port status for the device. If selecting multiple devices, the internal structure shows the list of selected devices or ports.
•
Properties pane shows the total of selected devices or ports.
•
Common Tasks are the actions that can be performed for the selected device(s) or ports.
The Common Tasks allow you to perform variable operations, depending on the selected device. You may also perform tasks per device from the right-click menu.
•
CSV button: The CSV button on the Filter Bar allows you to save all the current configurations per table.
•
Filter Toolbar: The filter allows you to search specific information for the specific device. The following section provides an example.
7.2.1.3 Using the Table Filter
The following example shows a filter search. The search is case sensitive and incremental.
As you type the search string, the table entries are dynamically filtered.
Figure 28: Filter Search (Example)
You can perform the following actions in the table.
•
Assigning Server IP: If the server IP is 0.0.0.0, you can set the IP SSH access point for that device (Right-click the device row and select Configure from the right-click menu.
Then click Access Credentials.). Select IP from the drop-down menu and type 0.0.0.0 to filter devices.
•
Logical Servers: If you want to add devices to a Logical Server, select Logical Server from the drop-down menu and type a Logical Server Name. Check the Logical Server column. You can quickly identify which devices are allocated to which Logical Server.
•
Rack: If you want to find devices in a specific Rack, select Rack from the drop-down menu and type the Rack’s name. Check the Rack column. You can quickly view which devices belong to the Rack.
7.3
138
Manage Devices Window Categories (Tabs)
The Manage Devices window contains a tab for each of the following categories:
•
•
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
7.3.1
•
•
•
•
•
Devices Tab
The Devices tab lists all the physical hardware components for the selected site, server, or switch.
Figure 29: Devices Tab
Version 5.2
The information is displayed in tabular form and includes the device State, ID, Name, Type,
IP address, the Vendor, the CPU type, RAM, to which Rack it belongs, the FW Ver
(firmware versions), Temperature, Agent, and to which Logical Server it belongs.
7.3.1.1 Tasks in the Devices Tab
When selecting a single device, you can see its properties and add it to a rack.
When selecting multiple devices, you can add them to a rack. The multiple selection is useful if you have many devices on which you would like to perform identical tasks.
For more information, see Tasks on Devices and Ports
If there is an alarm, you can also view the alarm on the device. To see details of the alarm,
right-click the Alarm and select View Alarm. For more information, see Alarms Tab
Mellanox Technologies Confidential
139
Version 5.2
7.3.2 Ports Tab
The Ports tab lists in table format the port(s) for the selected device(s).
The following figure shows the table in the Ports tab.
Figure 30: Ports Tab
Managing Devices
The Ports information tab display includes the State of the port, the Node (device) to which the port belongs, the Port number, the Supported link width, Supported Speed, the Active
Width, the Peer Node (the node that is on the other side of the link), the Peer Port (the port that is on the other side of the link.
7.3.2.1 Showing Active Ports
By default, the Ports table displays information for all ports.
To display only the active ports (those that currently generate traffic), in the toolbar, click the gray Active Ports icon . The icon turns green , and the table displays only the active and administratively disabled ports (default setting). Administratively disabled ports are indicated by an icon.
To display all ports, click the green Active Ports icon. The icon turns gray again.
7.3.2.2 Tasks in the Ports Tab
You can perform tasks on a single port, namely viewing the properties, perform a monitoring session on a port, view the peer port, and show the links connected to the selected port and displays the link’s internal structure. You can select and reset multiple ports simultaneously.
For more information, see Tasks on Devices and Ports
7.3.2.3 Port Counters
The port counters feature at the bottom of the UFM window allows you to obtain statistics of the port state.
7.3.2.3.1 Viewing and Setting Port Counters
The Port Counters Graph shows the port traffic transfer in real-time.
140
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
To view Port Counters:
1. In UFM, ensure that you are either in the View or Manage Devices Tab.
2. Select a device in the left tree or the map view.
3. In the Internal Structure or the Table view, click a port.
4. From the right-click menu, select Start Monitoring.
The Start Monitoring option on a port is also available in the internal structure and under Common Tasks (right panel). The Port Counters tab is enabled at the bottom of the UFM window, showing the port monitoring session.
Figure 31: Port Counters
Version 5.2
The following port traffic data is shown:
Table 33: Port Traffic Data
Port traffic
Normalized Xmit Data
Normalized Dropped Bandwidth
Description
Percentage of port bandwidth utilization
Percentage of dropped bandwidth in time units relative to the port bandwidth
Table 34: Event Table Icons
Icon Name
Start
Stop
Function
Restarts the Port Monitoring Session
Stops the Port Monitoring Session
141
Mellanox Technologies Confidential
Version 5.2 Managing Devices
A comprehensive list of Port counters is available in Supported Port Counters and Events
7.3.2.3.2 Saving the Port Counters to a CSV File
The UFM allows you to save the events as local CSV (Excel) files which are used for:
•
Importing to Excel and other systems for presentation
•
Perform advanced analysis based on data in the files
•
Send them to support for troubleshooting
To save the port counters to a local CSV file from the GUI
1. In a monitoring session, click to save the port counters to a CSV file. A message prompts you to save the file at the desired location.
2. In the File Name field, enter a name for your CSV file.
3. On your local computer, browse to a location and click Save. Your CSV file is saved.
To configure the UFM to automatically save the port counters to a CSV file on the
UFM Server
•
In the [CSV] section of the /opt/ufm/conf/gv.cfg configuration file, set the parameters described in the following table.
Parameter
write_interval ext_ports_only
Description
The period for polling the port counters.
The default, 0, disables saving the CSV files.
The files are saved in
/opt/ufm/files/csv/
in the following naming convention:
PortCounters--<time>.CSV, where date format is ddmmyy, and the time format is hhmmss.
Suppresses information about internal ports. Set to Yes to transmit only external port information.
Values
0 (default),
30-3600 seconds
No (default), Yes
7.3.2.3.3 Saving Periodic Snapshots of the Fabric (Advanced License Only)
UFM enables you to save multiple PortCounters CSV files for historical analysis. UFM will save multiple files on the UFM Server according to a configuration parameter in UFM’s configuration file.
To configure the maximum number of CSV files to save
•
In the /opt/ufm/conf/gv.cfg configuration file, set the max_files parameter as described in the following table:
Table 35: Port Traffic Write Interval
Parameter
max_files
Description
The maximum number of files to be saved in /opt/ufm/files/csv/ folder.
Values
1 (default), 1 - 100
142
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
NOTE: After you install the UFM Advanced license, the settings for maintaining historical port counter data are enabled.
7.3.2.3.4 Viewing CSV File Information
To view the CSV file information:
1. Go to http://<UFM server ip>/PortCounters.CSV to view the CSV file.
The CSV file is displayed. See Supported Port Counters and Events for detailed
information on port counters.
Version 5.2
2. Go to the last row to view the Port Counter summaries.
Mellanox Technologies Confidential
143
Version 5.2 Managing Devices
7.3.2.3.5 64-Bit and 32-Bit Counters
Whenever possible (QDR devices and up, ConnectX-2 HCA), UFM polls 64-bit (extended) counters. You do not need to manage counter reset policy for 64-bit counters.
32-bit counters are polled only when 64-bit counters are not available, for example, data counters on pre-QDR switches, old HCAs, Error counters, and so on. In these cases, you can
reset 32-bit counters according to UFM reset policies. For more information, see Port
32-Bit Port Counters Reset Policy
UFM resets physical 32-bit port counters according to a configurable port counter reset policy. You can set the reset_mode parameter in the /opt/ufm/conf/gv.cfg configuration file to one of the following policies:
•
Reset on Every Poll
•
Reset on Threshold
•
No Reset
When you select the policy, consider the polling rate, fabric traffic patterns, and the type of counter. The following table describes the effects of the port reset policy on the Data, IB
Error, and Xmit Wait counters.
Table 36: Effects of Port Counter Reset Policy on Counters
reset_mode Value
reset_every_poll
(default) reset_on_threshold no_reset
Data Counters
(Xmit/Rcv Data, Rcv
Packets)
Resets after every poll.
Resets when last measured value exceeds the limit specified in
counter_reset_thres
hold parameter of gv.cfg
.
IB Error Counters
Resets when last measured value exceeds threshold specified by the
counter_reset_thres
hold parameter of gv.cfg
.
Resets when last measured value exceeds the limit specified in
counter_reset_thres
hold parameter of gv.cfg
.
Counters are not reset by the UFM. User is solely responsible for counter resets in the fabric
Counters are not reset by the UFM. User is solely responsible for counter resets in the fabric
Xmit Wait Counter
Resets when last measured value exceeds threshold specified by the
counter_reset_thresh
old parameter of gv.cfg
.
Resets when last measured value exceeds the limit specified in
counter_reset_thresh
old parameter of gv.cfg
.
Counters are not reset by the UFM. User is solely responsible for counter resets in the fabric
For more information about other parameters affecting reset policy, see Resetting Physical
144
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
CAUTION: Only the default counter reset policy and polling rate settings ensure that the counters remain within limit. Other settings must be selected according to the user requirements in order to prevent counter saturation.
7.3.2.3.6 Resetting Physical Port Counters
To reset 32-bit physical port counters
•
In the /opt/ufm/conf/gv.cfg configuration file set counter_reset_policy,
counter_reset_threshold, and ibpm_max_polling_rate parameters according to the following table:
Table 37: Port Counter Reset Policy Parameters
Parameter
counter_reset_policy counter_reset_threshold
Ibpm_max_polling_rate
Description
Determines if and when the counters will be reset by UFM.
Values
• reset_every_poll (default)
• reset_on_threshold
• no_reset
25-100 percent The percent of the maximum counter value. When the counter reaches the defined threshold,
UFM resets the counter on the next poll when
counter_reset_policy is set to reset_on_threshold.
The maximum polling interval in seconds for checking the counters.
5 (default), 5-30
NOTE: In UFM Views and Monitoring Sessions, the values displayed are UFM cumulative values, regardless of the physical port value.
7.3.2.3.7 Resetting Cumulative Port Counters
UFM allows you to reset the cumulative port counters.
To reset cumulative port counters:
•
In the Monitoring Session View and Manage Devices View, press the Cumulative
Counter Reset button , as shown in the following figure.
Version 5.2
Mellanox Technologies Confidential
145
Version 5.2
7.3.3 Unhealthy Tab
The Unhealthy tab shows all the unhealthy nodes in the fabric.
Managing Devices
After the Subnet Manager examines the behavior of subnet nodes (switches and hosts) and discovers that a node is “unhealthy” according to the conditions specified below, the node is displayed in the Unhealthy Ports tab. Once a node is declared as “unhealthy” the Subnet
Manager can either ignore, report, isolate or disable the node. The user is provided the ability to control the actions performed and the phenomena that declares a node “unhealthy”.
Moreover, the user has the ability to “clear” nodes that previously marked as “unhealthy”.
The information is displayed in tabular form and includes the device State, Peer Node, Peer
Port, Peer GUID, Peer LID, Condition, Status Time, Unhealthy Node, Unhealthy Port,
Unhealthy GUID.
NOTE: The feature requires OpenSM parameter hm_unhealthy_ports_checks to be set to
TRUE (default).
NOTE: This feature is not available in the "Monitoring Only Mode".
146
The following are the conditions that would declare a node as “unhealthy”:
•
Reboot - If a node was rebooted more than 10 times during last 900 seconds
•
Flapping - If several links of the node found in Initializing state in 5 out of 10 previous sweeps
•
Unresponsive - A port that does not respond to one of the SMPs and the MAD status is
TIMEOUT in 5 out of 7 previous SM sweeps
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
7.3.4
•
Noisy Node - If a node sends traps 129, 130 or 131 more than 250 traps with interval of less than 60 seconds between each two traps
•
Seterr - If a node respond with bad status upon SET SMPs (PortInfo, SwitchInfo, VLArb,
SL2VL or Pkeys)
•
Illegal - If illegal MAD fields are discovered after a check for MADs/fields during receive_process
•
Manual - Upon user request mark the node as unhealthy/healthy
•
Link Level Retransmission (LLR) – Activated when retransmission-per-second counter exceeds its threshold
All conditions except LLR generate Unhealthy port event, LLR generates a High Data retransmission event.
To clear a node from the Unhealthy Tab:
1. Go to the Manage Devices menu.
2. Go to the Unhealthy Ports tab
3. Choose the desired node, right click it and mark it as healthy.
To mark a node as permanently healthy:
1. Open the /opt/ufm/files/conf/health-policy.conf.user_ext file.
2. Enter the node and the port information and set it as "Healthy".
3. Run the /opt/ufm/scripts/sync_hm_port_health_policy_conf.sh script.
Alarms Tab
The Alarms tab shows all the alarms for the selected device(s) (per port).
Figure 32: Alarms Tab
Version 5.2
An alarm is the specific object status deriving from event(s) that occurred on the object or object components (objects are Ports, Devices, Logical Servers). For example, when a specific error counter increment exceeds the defined threshold, an alarm is defined on the
Port, Device, or Logical Server that contains the Device.
Alarms are shown on ports according to their Severity, the Source of the alarm (Device
GUID), the Alarm ID, the alarm Description, the Date and Time of the alarm, the cause of the alarm (Reason), and the alarm Count. They can also be filtered according to the parameters.
147
Mellanox Technologies Confidential
Version 5.2 Managing Devices
Alarms are classified according to severity: Normal, Warning, Minor, and Critical.
The following table describes the alarm states in UFM. In the map and table view, an alarmed device shows the following states: Info (Normal), Warning, Minor, and Critical.
Table 38: Alarm States
Icon Name
Info
(Normal)
Critical
Minor
Description
Information/notification displayed during normal operating state or a normal system event.
Critical means that the operation of the system or a system component fails.
Minor reflects a problem in the fabric with no failure.
Warning Warning reflects a low priority problem in the fabric with no failure.
A warning is asserted when an event exceeds a pre-defined threshold.
You can define event severity and filter events via the GUI (Config view, Event
Management section). For more information, see Configuring Event Management
Alarms are reflected on objects throughout UFM. The object shows the highest severity alarm of any alarms that occurred on a dependent device or port. An alarm shows the object severity level. An event includes physical error detection and the impact on the logical configuration in the fabric.
7.3.4.1 Tasks and Common Tasks
After you have located an alarm, by using the right-click menu or from Common Tasks, you can perform the following actions:
Table 39: Alarm Task Icons
Icon Description
View alarm information for the device. The detailed Row view box showing the alarm information opens.
[Toolbar] Clear all alarms associated with the device.
[Right pane component bar] Clear the selected alarms.
Alarms are cleared from UFM after the pre-defined lifetime defined in the Event Policy
Table. The default lifetime for most alarms is 300 sec. For more information, see
Alarms are cleared from the UFM after they are fixed manually.
For more information, see Tasks on Devices and Ports
See Supported Port Counters and Events for the list of events and alarms, descriptions,
values and thresholds.
148
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
7.3.5 Traffic Tab
The Traffic tab allows you to view the traffic and congestion counters on the fabric port.
Figure 33: Traffic Tab
Version 5.2
To view the traffic on ports
1. On the left tree, select a device for which you would like to check the traffic.
2. On the toolbar, set the time polling interval in seconds (possible values in seconds: 30,
60, 120, and 300)
3. On the toolbar, click Start to start (or restart) checking the traffic on the selected device(s) (per port). The Traffic table shows the traffic per port in real-time.
4. Click Stop to stop polling the traffic on the selected device. If desired, you may restart again by clicking Start .
7.3.5.1 Tasks in the Traffic Tab
You may perform tasks on a single port, namely viewing the properties, perform a monitoring session on a port, view the peer port, and show the links connected to the selected port and displays the link’s internal structure. You can select and reset multiple ports simultaneously.
For more information, see Tasks on Devices and Ports
Mellanox Technologies Confidential
149
Version 5.2
7.3.6 Errors Tab
The Errors tab allows you to view the error counters on the fabric port.
Figure 34: Errors Tab
Managing Devices
To view the errors on ports
1. On the left tree, select a device for which you would like to check the error.
2. On the toolbar, set the time polling interval in seconds (possible values in seconds: 30,
60, 120, and 300).
3. On the toolbar, click Start to start (or restart) checking the errors in real-time for the selected device(s) (per port). The Error table lists the errors and the traffic per port.
4. Click Stop to stop polling for errors on the selected device. If desired, you may restart again by clicking Start .
7.3.6.1 Tasks in the Errors Tab
You can perform tasks on a single port, namely view the properties, perform a monitoring session on a port, view the peer port, show the links connected to the selected port, and display the link’s internal structure. You can select and reset multiple ports simultaneously.
For more information, see Tasks on Devices and Ports
7.3.7 Connection Congestion Tab (Advanced License Only)
The Connection Congestion tab enables you to view, compare, and analyze congestion on specific switch ports, caused by different flows (by Source and
Destination).
150
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
7.3.8
The Connection Congestion table displays Congestion Control Monitoring data from
InfiniScale IV switches.
NOTE: This feature is enabled by InfiniBand Switch Software, version 3.1.
Each switch sends information for up to 10 top congested flows. Non-congested flows, for which there are no congestion hits, are not displayed.
The table provides the following information:
Table Column
Switch
Port
Chip GUID
Chip Port
Source Device
Source LID
Destination Device
Destination LID
Service Level
Flow Hits (%)
Port Hits (%)
Description
Name or IP of InfiniScale IV Switch or Director.
External port of InfiniScale IV Switch or Director. Internal ports are represented by “Internal”.
GUID of reporting InfiniScale IV Chip.
Local port number of reporting InfiniScale IV Chip.
Host name or IP of the source device.
Source Host LID.
Host name or IP of the destination device.
Destination Host LID.
The service level of the specified flow.
Percent of hit events for a specific flow in a specific port.
Percent of hit events on a specific port in the chip.
Accessing Tasks For a Device or Object
You can access a task for a device or for an object from the Design, View, and Manage
Devices windows, by performing any of the following actions:
•
Right-click a device (or the object) and select the task from the menu (if applicable).
151
Mellanox Technologies Confidential
Version 5.2 Managing Devices
•
In the Common Tasks (right pane) select a device (or object) and choose the task to perform. The tasks can be performed on multiple devices by clicking the devices and holding down the Ctrl key on your keyboard.
•
In the Internal Structure Tasks, right-click a device (or object) and choose the task to perform. This is performed per device.
•
In the Devices tab table, right-click a device (row) and choose the desired task to perform.
•
In the Left tree, right-click the object or device and select the task from the menu.
7.3.8.1 Additional Tasks Available from the View window:
•
Map tab - Right-click the Device icon and select the desired task from the menu (if applicable). You can select multiple devices by clicking the icons and holding down the
Ctrl key.
•
Table tab - Right-click the Device row or select multiple rows (multiple devices) and choose the desired task from the menu (if applicable).
7.4
7.4.1
Tasks on Devices and Ports
You can access tasks to be performed on an object by right-clicking the object.
Common tasks can be performed on a single object, or on multiple objects, and can be accessed from the Common Tasks pane on the right of the window:
•
•
Some of the common tasks work only when the server has the UFM Agent running. For
more information on the UFM Agent and how to install it, see Installation and Initial
The following topics describe the tasks that can be performed on devices and ports.
Devices - Common Tasks
The following topic lists the Common Tasks available for devices in the Design window, and from the Devices tab in the Manage Devices window.
Icon Common Task
Configure
Edit User
Attributes
Monitor
Add Group
Description
Enables you to view the device configuration, assign the object to a rack, select the owner (Logical Server), and define Access
Credentials. See Configuring Device Properties
You can also run SSH commands on the device from this task
2
.
Enables you to add and edit attributes for the object. See Setting
Enables you to define a monitoring session. See Creating a
Monitoring Session or Template
Enables you to add the object to a group. See Adding Devices to a
152
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
7.4.2
Icon Common Task
Show Groups
Description
Enables you to view all groups associated with an object, and
remove the object from groups. See Removing Devices from a
Enables you to add or release servers from logical servers. Add or Release from Logical
Server (Servers only)
View Alarms Shows the alarms for the selected object.
Clear Alarms Clears the alarms associated with the selected object.
SSH
Run Command
Launch
Reboot
Opens an SSH session
1
.
Runs a previously defined command on the device
2
.
Launches the URL defined on the device.
Reboots the selected object
3
.
Shutdown Shuts down the selected object
3
.
Software Update Perform a software upgrade
2
. See Upgrading the Server Software
Firmware
Upgrade
Performs a firmware upgrade
2
1. UFM needs the Host IP address. If the UFM Agent is not running on a Host, the IP address is not discovered. However, the IP address can be manually specified via the
Configure task.
2. Available only when the UFM Agent is running on a Host.
3. Can be performed over SSH or via IPMI. For SSH, see note (1). For The IPMI option, the IP address can be manually specified via the
Configure task.
Configuring Device Properties
To set the device properties:
1. From the left Fabric tree, select a device.
2. Right-click the selected device or go to Common Tasks (right panel).
From the menu, select Configure. The Device Configuration dialog box opens displaying the device parameters, the device CPU information (read-only), and the rack
Version 5.2
Mellanox Technologies Confidential
153
Version 5.2 Managing Devices information (if applicable). For information about the parameters, see the table that follows.
154
3. For information about changing the rack assignment, see Rack and General Device
4. For information about setting Device Access Credentials, see Editing Access Credentials
5. For information about setting device name and IP manually, see Configuring
6. Click Apply or OK to save your changes.
Table 40: Device Configuration Parameters
Field
Name
ID
IP Address
Type
Description
Description
Displays the name of the device.
The device ID.
The IP address of the device.
The device type.
The user-defined device description.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Field
Vendor
Temperature
Severity
URL
Event Script
Hardware
Rack
Owner
Description
The device vendor.
The temperature threshold for the device.
The status of the device: Normal (Info), Warning, Minor, Severe.
(Advanced License only) The URL defined on the device. This URL can be
launched by the Launch button. See Defining a URL to Launch
The script to be called on a device event.
Displays the following device information:
•
CPU Type
•
Number of CPUs installed in the device
•
CPU Speed
•
RAM
Rack assignment status. Allows you to set or change the rack assignment.
The Logical Server to which the device belongs.
Version 5.2
Mellanox Technologies Confidential
155
Version 5.2
To see history of events related to a specific device:
•
Open the History tab.
Managing Devices
7.4.2.1 Editing Device Access Credentials
These credentials are set to allow remote accessibility to your device.
To edit your device access credentials;
1. Open the Device Configuration dialog box (see Configuring Device Properties)
2. Click Access Credentials. The Device Access Credentials dialog box opens.
156
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
3. Select the relevant preferred protocol tab:
•
The SSH tab (shown in the previous figure) allows you to define the SSH parameters to open an SSH session to your device.
•
The IPMI tab allows you to set the IPMI parameter session to your device for remote power control.
4. Click the Enabled box to enable the protocol.
5. Click Close to save your changes.
Table 41: Device Access Credential Parameters
Field
User
Password
Confirmation
Manual IP
Port
Timeout
Description
Fill in or edit the computer user name.
Enter the device password.
Enter the device password a second time to confirm.
Enter the device IP address.
Enter the port number.
157
Mellanox Technologies Confidential
Version 5.2
7.4.2.2 Configuring Identification Settings
You can manually define the name and IP on a device to be used if the device is not discovered or does not have a UFM Agent.
To define device name and IP manually
1. Open the Device Configuration dialog box (see Configuring Device Properties ).
2. Click Identification Settings. The Identification Settings dialog box opens.
Managing Devices
3. In the Manual fields, enter the device name and IP.
4. If you want the manual definitions to be used, select the corresponding Manual radio button.
5. Click OK or Apply to save settings.
After setting the InfiniScale IV ® and SwitchX ™ based switches’ IP address manually,
UFM will first validate the new IP before setting it. For further information, please refer to
section Switch Identity Validation
7.4.2.3 Switch Identity Validation
Identification settings for InfiniScale IV ® and SwitchX ™ based switches are set only after an identity validation procedure is run by UFM to ensure that the new settings are correct, and avoid any chance of UFM collecting information from the wrong device.
To validate the information collected from a device:
1. Connect to the device according to the new identification settings.
Note: If the connection cannot be established, the validation procedure fails.
2. Retrieve its identity (System GUID).
3. Compare the retrieved identity to the UFM model’s records to determine whether the validation succeeded or not.
158
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Figure 35: Switch Connectivity Failure
Version 5.2
Figure 36: Identity Mismatch
7.4.3 Running SSH Commands on a Device (Advanced License only)
When a device has the UFM host agent installed, you can run SSH commands on the device from the UFM GUI, and view the displayed output.
To run a command on a device
1. From the left Fabric tree, select a device.
2. Right-click the selected device or go to Common Tasks (right panel).
From the menu, select Configure.
3. In the Device Configuration dialog box, select the Commands tab. (The Commands tab is enabled only with Advanced License when the Host Agent is installed on the device.)
159
Mellanox Technologies Confidential
Version 5.2 Managing Devices
7.4.4
4. In the Input pane, enter SSH commandS with all parameters.
5. Click Execute Command.
The output is displayed in the Output pane.
6. To clear the panes, click .
Setting User-Defined Attributes (Advanced License Only)
You can create and modify additional user-defined attributes for some UFM objects.
To set user-defined attributes
1. From the left Fabric tree, select a device.
2. Right-click the selected device or go to Common Tasks (right panel).
3. From the menu, select Edit User Attributes.
160
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
4. To add a new attribute, in the toolbar, click .
For information about:
•
Defining SSH commands as user-defined attributes, see Defining and Saving Device
•
Defining a URL to launch, see Defining a URL to Launch
5. To edit an entry, double-click the entry.
6. Enter the attribute name and value.
7. Click OK or Apply.
7.4.4.1 Defining and Saving Device Commands (Advanced License only)
When a device has the UFM host agent installed, you can define SSH commands and save them as user-defined attributes. You can then run the saved device commands from the UFM
GUI. Use this feature when you run the same command repeatedly, and you do not need to view the output.
For information about running saved commands, see Running Saved Commands
To define and save SSH commands for a selected device
1. In the Common Tasks pane, click Edit User Attributes. For more information, see
Setting User-Defined Attributes
2. To define an SSH command, click .
3. In the Name field, enter cmd_<name>, where <name> is a unique identifier for the command.
4. In the Value field, enter the command or script name with parameters.
161
Mellanox Technologies Confidential
Version 5.2 Managing Devices
In the following example, the attribute name is cmd_Test, and the value is a python script with a single parameter value, param1.
5. Click OK.
7.4.4.2 Defining a URL to Launch (Advanced License only)
You can define a URL to be launched by the Launch button. The URL is saved as a userdefined attribute.
You can define the URL in the General tab of the Device Configuration Properties dialog
box (see Configuring Device Properties ) or from the Edit User Attributes option. Whenever
you edit the URL definition, the update is reflected in both dialog boxes.
To define a URL from Edit User Attributes
1. In the Common Tasks pane, click Edit User Attributes. For more information, see
Setting User-Defined Attributes
2. To define a URL attribute, click .
3. In the Name field, enter URL.
4. In the Value field, enter the destination address to launch.
7.4.5
5. Click OK.
Running Saved Commands (Advanced License only)
After defining SSH commands as user-defined attributes for a device, you can run the saved commands from the UFM GUI whenever required.
To run saved commands on a selected device
1. In the Common Tasks pane click Run Commands.
162
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
7.4.6
2. Select the command to run. The command name is the suffix of the user-defined attribute name. For example, if you defined a user attribute for an SSH command as cmd_Test, the name in the Run Commands window appears as Test.
3. Click Run.
OR
1. Right-click the device and select Run Command > command_name.
Adding Servers to a Logical Server
The functionality of adding/releasing servers to/from a logical server, is only applicable to unmanaged hosts (servers).
Logical server actions are asynchronous. It can take a few moments until the Logical Server is operational (RUNNING). If the logical server is not ready to operate, the following error message appears.
To add a computer to a Logical Server:
1. Select a single or multiple hosts.
2. From the right-click menu or from Common Tasks, select Add to Logical Server if you want to add this server to a Logical Server. The Add to Logical Server dialog box displays a list of available logical servers.
Mellanox Technologies Confidential
163
Version 5.2 Managing Devices
7.4.7
3. Select the logical server to which to add your server.
4. Click Apply or OK to save your changes. UFM displays a confirmation message that the
Host was successfully added.
Releasing a Computer (Server) from a Logical Server
To release a computer from a Logical Server
1. Select a host/server.
2. From the right-click menu or from Common Tasks, select Remove from Logical Server if you want to release this server from a Logical Server. The dialog box displays a list of available Logical Servers. A warning appears prompting for confirmation.
3. Click OK to confirm. The server is now released from the Logical Server to which it belonged.
7.4.8 Rack and General Device Grouping
You can group devices in racks or in any general device group. To group devices, create a rack or general group, and add devices to it. You can view the contents of a rack or general device group, and add or remove devices.
You can change a device's rack assignment in the Device Configuration dialog box.
7.4.8.1 Adding Devices to a Group
To add devices to a new group
1. In any of the views, select a single or multiple devices.
2. Do one of the following:
•
Right-click the selected device(s) and select Add to Group > New.
164
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
In the Common Tasks (right panel), select Add to Group.
Version 5.2
3. Click to add a new group.
4. Enter a Name and Description for the new group, and select the Type of group.
5. Click Apply or OK to save your changes.
The new group is created containing the selected devices.
To add devices to an existing group
1. In any of the views, select a single or multiple devices.
2. Right-click the selected device(s) and select Add to Group > Groupname. or
In the Common Tasks (right panel): e. Select Add to Group f. Select the type of group - RACK or GROUP. g. Select the group to which you want to add the device(s). h. Click Apply or OK to save your changes.
165
Mellanox Technologies Confidential
Version 5.2 Managing Devices
7.4.8.2 Removing Devices from a Group
To remove devices from a group
1. In any of the views, select the device(s) to remove.
2. Do one of the following:
•
Right-click the selected device(s) and select Show Groups.
•
In the Common Tasks (right panel), select Show Groups.
The dialog box displays all groups associated with the selected devices, including racks,
VMMs and so on.
7.4.9
3. Select the group from which you want to remove the device(s). Use the incremental search field to filter for the required group or groups.
4. Click Apply or OK to save your changes.
Opening an SSH Session to the Server
The functionality of opening an SSH session, is applicable to both managed and unmanaged hosts (servers).
To open an SSH session:
1. Make sure that the SSH parameters to your server are defined. For more information, see
2. Click SSH to open an SSH session for the selected device.
7.4.10 Upgrading Server Software and Firmware
Software/Firmware Upgrade via FTP
Software and firmware upgrade over FTP is enabled by the UFM Agent. UFM invokes the
Software/Firmware Upgrade procedure locally on switches or on hosts. The procedure copies the new software/firmware file from the defined storage location and performs the operation on the device. UFM sends the set of attributes required for performing the software/firmware upgrade to the agent.
The attributes are:
•
File Transfer Protocol – default FTP
•
The Software/Firmware upgrade on InfiniScale III ASIC-based switches supports FTP protocol for transmitting files to the local machine.
166
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
•
The Software/Firmware upgrade on InfiniScale IV-based switches and hosts supports
TFTP and protocols for transmitting files to the local machine.
•
IP Address of file-storage server
•
Path to the software/firmware image location
The software/firmware image files should be placed according to the required structure
under the defined image storage location. See Directory Structure for Software or
•
File-storage server access credentials (User/Password)
In-Band Firmware Upgrade
You can perform in-band firmware upgrades for externally managed switches and HCAs.
This upgrade procedure does not require the UFM Agent or IP connectivity, but it does
require current PSID recognition. See PSID and Firmware Version In-Band Discovery . This
feature requires that the Mellanox Firmware Toolkit (MFT), which is included in the UFM package, is installed on the UFM server. UFM uses flint from the MFT for in-band firmware burning.
Before upgrading, you must create the firmware repository on the UFM server under the directory /opt/ufm/files/userdata/fw/. The subdirectory should be created for each PSID and one firmware image should be placed under it. For example:
/opt/ufm/files/userdata/fw/
MT_0D80110009 fw-ConnectX2-rel-2_9_1000-MHQH29B-XTR_A1.bin
MT_0F90110002 fw-IS4-rel-7_4_2040-MIS5023Q_A1-A5.bin
7.4.10.1 Directory Structure for Software or Firmware Upgrade over FTP
Before performing a software or firmware upgrade, you must create the following directory structure for the upgrade image. The path to the <ftp user home>/<path>/ directory should be specified in the Upgrade dialog box.
<ftp user home>/<path>/
InfiniScale3 - For anafa based switches Software/Firmware upgrade images voltaire_fw_images.tar – firmware image file ibswmpr-<s/w version>.tar – software image file
InfiniScale4 - For InfiniScale IV based switches Software/Firmware upgrade images firmware_2036_4036.tar – Firmware image file upgrade_2036_4036.tgz – Software image file
OFED /* For host SW upgrade*/
OFED-<OS label>.tar.bz2
<PSID>* – For host FW upgrade fw_update.img
The <PSID> value is extracted from mstflint command: mstflint -d <device> q
The device is extracted from the lspci command.
For example:
# lspci
06:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex
# # mstflint -d 06:00.0 q | grep PSID
PSID: VLT0040010001
167
Mellanox Technologies Confidential
Version 5.2 Managing Devices
7.4.10.2 PSID and Firmware Version In-Band Discovery
The device PSID and device firmware version are required for in-band firmware upgrade and for the correct functioning of Subnet Manager plugins, such as Congestion Control Manager and Lossy Configuration Management. For most devices, UFM discovers this information and displays it in the Device Properties pane.
The PSID and the firmware version are discovered by the Vendor-specific MAD. To disable the vendor-specific discovery, change the mode of discovery to 0 as shown below.
In the gv.cfg configuration file change the mode of discovery in the following line from: event_plugin_options = --vendinfo –m 2 to event_plugin_options = --vendinfo –m 0
Available Mode (-m) values are:
•
0 - Disabled - do not collect vendinfo.
•
1 - Not supported - this might cause the subnet Manager to crash.
•
2 - Enabled - collect all vendinfo using General Info MAD - this is the default.
In some firmware versions, the information above is currently not available.
7.4.10.3 Upgrading the Server Software
The functionality to update the server software is applicable only for hosts (servers) with the
UFM Agent.
To upgrade the software:
1. Select a device.
2. From Common Tasks or the right-click menu, select Software Update.
168
3. Enter the parameters listed in the following table.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
4. Click Apply or OK to save your changes.
Table 42: Upgrade Configuration Parameters
Parameter
Protocol
IP
Path
User
Password
Description
Update is performed via FTP protocol.
Enter the host IP.
Enter the parent directory of the FTP directory structure for the Upgrade image. The path should not be an absolute path and should not contain the first slash (/) or trailer slash.
Name of the host user name.
Enter the host password.
Version 5.2
7.4.10.4 Upgrading the Firmware
You can upgrade firmware over FTP for hosts and switches that are running the UFM Agent, or you can perform an in-band upgrade for externally managed switches and HCAs.
Before you begin the upgrade ensure that the new firmware version is in the correct location.
For more information, see Upgrading Server Software and Firmware Upgrade
To upgrade the firmware:
1. Select a host or server.
2. From Common Tasks or the right-click menu, select Firmware Upgrade.
3. Select the type of upgrade to perform, In Band or Over FTP.
4. For upgrade over FTP, enter the parameters listed in the following table.
5. Click Apply or OK to save your changes.
Note: The firmware upgrade takes effect only after the host or externally managed switch is restarted.
169
Mellanox Technologies Confidential
Version 5.2 Managing Devices
Table 43: Upgrade Configuration Parameters
Parameter
IP
Path
User
Password
Description
Enter the Host IP.
Enter the parent directory of the FTP directory structure for the
Upgrade image. The path should not be an absolute path and should not contain the first slash (/) or trailer slash.
Name of the host user name.
Enter the host password.
7.4.11 Rebooting a Server
Rebooting a server is only applicable to unmanaged hosts (servers).
To reboot a server:
1. Select a host/server.
2. From Common Tasks or the right-click menu, select Reboot . A confirmation box appears.
3. Click Yes to confirm. The server reboots.
7.4.12 Shutting Down a Server
Shutting down a server is only applicable to unmanaged hosts (servers) with IPMI configured on the hosts.
To shut down a server:
1. Select a host/server.
2. From Common Tasks or the right-click menu, select Shutdown . A confirmation box appears.
3. Click Yes to confirm. The server shuts down and the Server ID disappears from UFM.
7.4.13 Ports - Common Tasks
The following table lists the Common Tasks available for ports.
Icon Common Task
Properties
Description
Enables you to set or edit the selected port properties.
Disable Enables you to disable the selected port.
Start
Monitoring
Reset
Opens traffic and congestion monitoring session in the bottom pane.
For more information, see Monitoring the Fabric
Enables you to reset the selected port.
Goto Peer
Attach to
Enables you to detect and go to the peer port on the other side of the link.
Selects the links connected to the selected port and displays the link’s internal structure.
170
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Icon Common Task Description
Port Mirroring Enables you to configure a mirroring session.
7.4.14 Viewing Port Properties
To view ports properties
1. Select a host/server.
2. From the port table or the Internal Structure, select a port.
3. From Common tasks (right pane) or the right-click menu, select Properties .
Version 5.2
The Port Configuration dialog box displays the current port settings.
Table 44: Port Configuration Parameters
Parameter
Port
State
Width
MTU
Speed
Unique ID
Peer Port
Description
Port name (GUID)
Status of the port (info, minor, warning, severe)
Shows the link width
Shows the Maximum Transmission Unit (2048, 4096)
Shows the link speed (1x, 4x)
GUID
The peer port on the other end of this link.
171
Mellanox Technologies Confidential
Version 5.2 Managing Devices
7.4.15 Disabling/Enabling a Port
To disable or enable a port
1. Select a host/server.
2. From the port table or the Internal Structure, select a port.
3. From Common Tasks (right pane) or the right-click menu, select Disable
Enable , as required. A confirmation message appears.
or
4. Click Yes to confirm. The port will be disabled or enabled. (The UFM display might take a few seconds until it reflects the change.)
7.4.16 Setting a Port Monitoring Session
To set a port monitoring session
1. Select a host/server.
2. From the port table or the Internal Structure, select a port.
3. From Common Tasks (right pane) or the right-click menu, select Start Monitoring
. The Port Counters tab at the bottom of the window (bottom pane) becomes enabled,
showing the port monitoring session. For more information, see Monitoring the Fabric
4. Click Yes to confirm. The port will be disabled or enabled. (The UFM display might take a few seconds until it reflects the change.)
5. Click Start to start or restart the port monitoring session.
6. Click Stop to stop the port monitoring session.
7.4.17 Resetting a Port
To reset a port
1. Select a host/server.
2. From the port table or the Internal Structure, select a port.
3. From Common Tasks (right pane) or the right-click menu, select Reset Port .
A confirmation message appears.
4. Click Yes to confirm. The port is reset. (The UFM display might take a few seconds until it reflects the change.)
7.4.18 Going to the Peer Port
To go to a peer port
1. Select a host/server.
2. From the port table or the Internal Structure, select a port.
3. From Common Tasks (right pane) or the right-click menu, select Goto Peer
UFM displays the port at the other side of the link in the Ports Tab.
.
172
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
7.4.19 Attaching to a Peer Port
To attach to a peer port
1. Select a host/server.
2. From the port table or the Internal Structure, select a port.
3. From Common Tasks (right pane) or the right-click menu, select Attach To .
UFM selects the links connected to the selected port and displays link’s internal structure.
Version 5.2
7.5 Port Mirroring (Advanced License Only)
You can configure the mirroring of ingress and egress traffic of a specific port on any QDR switch. UFM implements remote mirroring, meaning that the mirrored packets are encapsulated and sent to the target device.
NOTE: Mirroring is supported only on InfiniScale IV Switches 4036, 4036E, 4200,
4700 – from Switch SW Version 3.1 and later.
You can configure a single mirroring session on a device. The mirroring session definition is applied to all InfiniScale IV chips of the selected device.
NOTE: To record mirrored traffic, you must run a traffic analysis tool on the target node (for example, the IBDump tool). For more information, contact Mellanox support at [email protected]
.
7.5.1 Adding and Editing a Mirroring Session
To configure a mirroring session
1. On a QDR switch, open the Device Configuration dialog, and click Mirroring Settings (
).
Mellanox Technologies Confidential
173
Version 5.2
The Mirroring Session dialog box appears:
Managing Devices
2. To select a target device for mirroring: a. Click the button by the Target Device field. b. Select a mirroring target from the list of hosts, and click OK.
3. Set other mirroring session parameters, as described in the following table.
174
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Table 45: Mirroring Session Parameters
Parameter
Monitored Device
Target Device
Packet Size
Service Level
Description
The device whose traffic is monitored.
The device that receives the mirrored traffic.
The number of bytes to be taken from the original packet and sent.
The service Level for the mirrored traffic.
Version 5.2
7.5.2 Adding and Removing Ports for a Mirroring Session
After creating a mirroring session, you can add the ports whose traffic will be sent to the mirroring target.
To add or remove a port for a mirroring session
1. In the tree pane, right-click on the required port from the switch side, and select Port
Mirroring.
2. To add the port, in the Port Mirroring dialog box, check Enable mirroring and then check the traffic to mirror, Received (Ingress) and/or Transmitted (Egress).
7.5.3
3. To remove the port, in the Port Mirroring dialog box, clear the Enable mirroring check box.
Deleting a Mirroring Session
You can delete a mirroring session.
To delete a mirroring session
1. On the mirrored switch, open the Device Configuration dialog box.
2. Click the Delete ( ) button by the Mirroring Session link.
175
Mellanox Technologies Confidential
Version 5.2
8
Configuring and Optimizing the Fabric
Configuring and Optimizing the Fabric
8.1 Overview of the Configuration Window
The Configuration Window is divided as follows:
•
The Event Management enables you to define for each event the severity, Time to Live
(TTL), threshold, report to Log, and report to the GUI. With the Advanced License you can also define event script and send by SNMP trap.
•
The Access Credentials enables you configure the default credentials for servers and switches (SSH, IPMI, and so on).
•
The Users Table enables you to configure users and their authentication. For more
information, see Managing Users
•
The Subnet Manager is a centralized entity that discovers, configures and manages the fabric.
•
The Routing Table enables you to configure a specific routing algorithm.
•
Non-optimal links enables you to set the default behavior for the ports and define the immediate action required based on the settings.
Each section in the Configuration window contains a title, a short overview, real time state of some fields, and a link into the relevant configuration panel.
Figure 37: Configuring the Fabric
8.2
176
InfiniBand Subnet Manager
Unified Fabric Manager is a management platform that uses OpenSM routing engine. UFM
Subnet Manager is developed within the context of an open source environment.
UFM Subnet Manager is InfiniBand-compliant and serves as a Subnet Manager and an administration tool that runs on top of OpenIB.
The general fabric configuration comprises the following features:
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
8.2.1
•
SM Properties
•
Routing
•
Quality of Service (QoS)
•
Event Policy
The following are UFM add-ons, enhancing OpenSM:
•
Dynamic configuration updates - no need to restart OpenSM for the updates to take effect
•
MC root spreading
•
Multicast configuration - updates change only the MFT (Multicast Forwarding Table) block in the switches.
•
New routing algorithm to avoid temporary MC loops and packet drops during MFT distribution time.
The SM is not always installed on the same device as the UFM server. The location of the
SM is indicated in:
•
The View window Map tab. indicates the device with the SM.
•
The Subnet Manager section in the main Config window.
SM Default Files provides the list of paths where the default SM files are stored.
UFM Subnet Manager Default Properties provides a list of the available SM properties.
Configuring the Subnet Manager
The UFM Subnet Manager (SM) is a centralized entity running on the server which discovers and configures all the InfiniBand fabric devices to enable traffic flow throughout the fabric. You can view and configure Subnet Manager parameters in the SM Configuration dialog box.
To access SM configuration:
1. In the main Config window, in the Subnet Manager section, click Subnet Manager
Configuration.
2. Select the relevant tab according to the required configuration:
•
•
•
SM Lossy Manager Configuration
•
SM SL to VL Mapping Configuration
•
•
•
•
•
Version 5.2
177
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
NOTE: To benefit from QoS capabilities in the fabric, please set the qos in the
/opt/ufm/files/conf/opensm/opensm.conf
to TRUE.
8.2.1.1 SM Keys Configuration
The SM Keys tab enables you to view and set the Subnet Manager Keys.
Figure 38: SM Keys Configuration
178
To configure SM Keys
1. Set the fields as described in the following table.
2. Click Apply or OK.
Table 46: SM Keys Configuration
Field
MKey
Description
A field that allows you to view or edit the
M_Key value sent to all ports to qualify all the set (PortInfo). Authentication is performed by the management entity at the destination port and is achieved by comparing the key contained in the SMP with the key (the M_Key Management key) residing at the destination port.
Subnet Prefix An identifier of the subnet. The subnet prefix is used as the most significant 64 bit of the GID of each InfiniBand node in the subnet.
MKey Lease Period A field that allows you to view or edit the lease period used for the M_Key on this subnet in [sec]
SA Key Shows the SM_Key value to qualify the receive SA queries as 'trusted'.
Default
0x0000000000000000
0xfe80000000000000
0
0x0000000000000001
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Field
SM Key
LMC
No Partition
Enforcement
Description
Read-only field that displays the Key of the Subnet Manager (SM).
Defines the LID Mask Control value for the SM. Possible values are 0 to 7.
LID Mask Control (LMC) allows you to assign more than one LID per port.
NOTE: Changes to the LMC parameter require a UFM restart.
Disables partition enforcement by switches.
Default
0x0000000000000001
0
Disabled
Version 5.2
8.2.1.2 SM Limits Configuration
The SM Limits tab enables you to view and set the Subnet Manager Limits.
Figure 39: SM Limits Configuration
To configure SM Limits
1. Set the fields as described in the following table.
2. Click Apply or OK.
Table 47: SM Limits Configuration
Field
Packet Life Time
Subnet Timeout
Description
A field that allows you to view and/or edit the code of maximum lifetime a packet in a switch.
The actual time is 4.096 usec *
2^<packet_life_time>.
The value 0x14 disables this mechanism
A field that allows you to view and/or edit the subnet_timeout code that will be set for all the ports. The actual timeout is 4.096usec *
2^<subnet_timeout>
Default
0x12
18
179
Mellanox Technologies Confidential
Version 5.2
180
Field
Maximal Operational
VL
Head of Queue Life
Time
Leaf Head of Queue
Life Time
VL Stall Count
Leaf VL Stall Count
Force Link Speed En
Force Link Speed Ext
Configuring and Optimizing the Fabric
Description
A field that allows you to view and/or edit the limit of the maximal operational VLs:
•
0: NO_CHANGE
•
1: VL0 1
•
2: VL0_VL1
•
3: VL0_VL3
•
4: VL0_VL7
•
5: VL0_VL14
A field that allows you to view and/or edit the code of maximal time a packet can wait at the head of transmission queue. The actual time is
4.096usec * 2^<head of queue lifetime>
The value 0x14 disables this mechanism
A field that allows you to view and/or edit the maximum time a packet can wait at the head of queue on a switch port connected to a CA or gateway port
A field that allows you to view the number of sequential packets dropped that cause the port to enter the VLStalled state. The result of setting this value to zero is undefined.
This field allows you to view the number of sequential packets dropped that cause the port to enter the VLStalled state. This value is for switch ports driving a CA or gateway port. The result of setting the parameter to zero is undefined.
A parameter that allows you to modify the
PortInfo:LinkSpeedEnabled field on switch ports.
If 0, do not modify.
Values are:
1: 2.5 Gbps
3: 2.5 or 5.0 Gbps
5: 2.5 or 10.0 Gbps
7: 2.5 or 5.0 or 10.0 Gbps
2,4,6,8-14 Reserved
15: set to PortInfo:LinkSpeedSupported
Important Note: For a fabric with DDR 10 Gb
Ethernet Gateways (sRB-20210G) and 4036E version lower than 3.7.0.896, this parameter should be set to None (0) – auto negotiate link speed according to INI settings.
A parameter that allows you to modify the
PortInfo:LinkSpeedExtEnabled field on switch ports .
Default
3
0x12
0x10
0x07
0x07
15
By default, UFM sets the enabled link speed equal to the supported link speed.
31: set to
PortInfo:LinkSpeed
ExtSupported
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Field
Local Physical Error
Threshold
Overrun Errors
Threshold
Description
If 0, don't modify.
PortInfo:LinkSpeedExtEnabled on port
Otherwise, use value for
PortInfo:LinkSpeedExtEnabled on port.
Values are (MgtWG RefID #4722):
1: 14.0625 Gbps (FDR)
2: 25.78125 Gbps (EDR)
3: 14.0625 Gbps or 25.78125 Gbps (FDR or
EDR)
30: Disable extended link speeds
31: set to PortInfo:LinkSpeedExtSupported
Default
A field that allows you to view and/or edit the threshold of local phy errors for sending Trap
129
A field that allows you to view and/or edit the threshold of credit overrun errors for sending
Trap 130
Default
0x08
0x08
8.2.1.3 SM Lossy Manager Configuration
The SM Lossy tab enables you to modify Lossy Configuration Manager options after Lossy
Configuration has been enabled.
Figure 40: SM Lossy Configuration
Version 5.2
For more information, see Managing Lossy Configuration
Mellanox Technologies Confidential
181
Version 5.2 Configuring and Optimizing the Fabric
8.2.1.4 SM SL to VL Mapping Configuration
The SM SL2VL tab presents the SL (service level) to VL (virtual lane) mappings and the configured Lossy Management. You cannot change these configurations in this tab.
Figure 41: SM SL to VL Mappings
8.2.1.5 SM Sweep Configuration
The SW Sweep tab enables you to view and/or set the Subnet Manager Sweep
Configuration parameters.
Figure 42: SM Sweep Configuration
To configure SM Sweep
1. Set the fields as described in the following table.
2. Click Apply or OK.
182
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Table 48: SM Sweep Configuration
Field/Box
Sweep Interval
Reassign LIDs
Force Heavy Sweep
Sweep on Trap
Description
A field that allows you to view and/or edit the number of seconds between subnet sweeps (0 disables it)
If enabled, causes all LIDs to be reassigned.
If enabled, forces every sweep to be a heavy sweep.
If enabled, every trap will cause a heavy sweep.
Default
10
Disabled
Disabled
Enabled
8.2.1.6 SM Handover Configuration
The SM Handover tab enables you to view and/or set the Subnet Manager Handover
Configuration parameters.
Figure 43: SM Handover Configuration
Version 5.2
To configure SM Handover
1. Set the fields as described in the following table.
2. Click Apply or OK.
Table 49: SM Handover Configuration
Field/Box
SM Priority
Polling Timeout
Description
A field that shows the SM priority used for determining the master. Range is 0 (lowest priority) to 15 (highest). Note: Currently, these settings may not be changed.
A field that shows the timeout in [sec] between two polls of active master SM.
Default/
Range
15
Range=10000
183
Mellanox Technologies Confidential
Version 5.2
Field/Box Description
Polling Retries
Honor GUID to LID File If enabled, honor the guid2lid file when coming out of standby state, if the file exists and is valid.
Ignore other SMs
Number of failing polls of remote SM that declares it "not operational".
If enabled, other SMs on the subnet are ignored.
Configuring and Optimizing the Fabric
Default/
Range
4
Disabled
Disabled
8.2.1.7 SM Threading Configuration
The SM Threading tab enables you to view and/or set the Subnet Manager Timing and
Threading Configuration parameters.
Figure 44: SM Threading Configuration
184
To configure SM Threading
1. Set the fields as described in the following table.
2. Click Apply or OK.
Table 50: SM Threading Configuration
Field/Box
Max Wire SMPs
Transaction Timeout
Max Message FIFO
Timeout
Single Thread
Description
A field that shows the maximum number of
SMPs sent in parallel.
A field that shows the maximum time in
[msec] allowed for a transaction to complete.
A field that shows the maximum time in
[msec] a message can stay in the incoming message queue.
When enabled, a single thread is used for handling SA queries.
Default/
Range
4
200
10000
Disabled
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
8.2.1.8 SM Logging Configuration
The SM Logging tab enables you to view and/or set the Subnet Manager Logging
Configuration parameters.
Figure 45: SM Logging Configuration
Version 5.2
To configure SM Logging:
1. Set the fields as described in the following table.
2. Click Apply or OK.
Table 51: SM Logging Configuration
Field/Box
Log File
Log Max Size
Dump Files Directory
Accumulate Log File
Force Log Flush
Log Levels
Description
Path of the Log file to be used.
Default/
Range
/opt/ufm/files/log/opensm.lo
g
A field that allows you to view and/or edit the size limit of the log file in MB. If overrun, the log is restarted.
4096
/opt/ufm/files/log The directory that holds the SM dump file.
If enabled, the log accumulates over multiple SM sessions.
Force flush to the log file for each log message
Available log levels: Error, Info,
Verbose, Debug, Funcs, Frames,
Routing, and Sys.
Enabled
Disabled
Error and Info
185
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
8.2.1.9 SM Additional Settings
The Misc tab enables you to view and/or set additional Subnet Manager Configuration parameters.
Figure 46: SM Misc Configuration
8.3
To configure SM Miscellaneous Parameters
1. Set the fields as described in the following table.
2. Click Apply or OK.
Table 52: SM Misc Configuration
Field/Box
Node Names Map
File
SA database File
No client reregistration
Disable Multicast
Description
A field that allows you to view and/or set the node name map for mapping nodes to more descriptive node descriptions.
SA database file name
If enabled, disables client re-registration.
Default/
Range
None
None
Disabled
Disabled
Exit on Fatal Event
If enabled, the SM disables multicast support and no multicast routing is performed.
If enabled, the SM exits on fatal initialization issues.
Enabled
Configuring Routing, Fabric Partitioning, and Quality of
Service
UFM achieves maximum performance with latency critical tasks by implementing traffic isolation, which minimizes cross-application interference and by prioritizing traffic to ensure critical applications get the optimal service levels.
186
Mellanox Technologies Confidential
8.3.1
Unified Fabric Manager for InfiniBand User Manual
UFM Routing Algorithms
UFM also supports the following routing engines.
•
MINHOP Algorithm - based on the minimum hops to each node where the path length is optimized (i.e., shortest path available).
•
UPDN Unicast routing algorithm - also based on the minimum hops to each node but it is constrained to ranking rules. Select this algorithm if the subnet is not a pure Fat Tree topology and deadlock may occur due to a credit loops in the subnet.
•
DNUP Unicast routing algorithm - similar to UPDN, but allows routing in fabrics which have some channel adapter (CA) nodes attached closer to the roots than some switch nodes.
•
File-Based Algorithm (FILE) - The FILE routing engine loads the LFTs from the specified file, with no reaction to real topology changes.
•
Fat Tree Unicast routing algorithm - Algorithm that optimizes routing for congestionfree "shift" communication pattern.
Select Fat Tree algorithm if a subnet is a symmetrical or almost symmetrical fat-tree. The
Fat Tree also optimizes K-ary-N-Trees by handling non-constant K in cases where leafs
(CAs) are not fully staffed, and the algorithm also handles any Constant Bisectional
Bandwidth (CBB) ratio. As with the UPDN routing algorithm, Fat Tree routing is constrained to ranking rules.
•
Quasi Fat Tree Routing Algorithm - PQFT routing engine is a closed formula algorithm for two flavors of fat trees.
•
Parallel Ports Generalized Fat Tree (PGFT)
•
Quasi Fat Tree (QFT).
PGFT topology may use parallel links between switches at adjacent levels, while QFT uses parallel links between adjacent switches in different sub-trees. The main motivation for that is the need for a topology that is not just optimized for a single large job but also for smaller concurrent jobs.
•
Layered Shortest Path (LASH) unicast routing algorithm - uses InfiniBand virtual layers (SL) to provide deadlock-free shortest path routing while also distributing the paths between layers. LASH is an alternative deadlock-free topology-agnostic routing algorithm to the non-minimum UPDN algorithm, avoiding the use of a potentially congested root node.
•
Dimension Order Routing (DOR) Unicast routing algorithm - based on the Min Hop algorithm, but avoids port equalization, except for redundant links between the same two switches. The DOR algorithm provides deadlock-free routes for hypercubes, when the fabric is cabled as a hypercube and for meshes when cabled as a mesh.
•
Torus-2QoS routing algorithm - designed for large-scale 2D/3D torus fabrics. In addition, you can configure Torus-2QoS routing to be traffic aware, and thus optimized
for neighbor-based traffic. For more information, see Torus-2QoS and Traffic-Aware
•
Routing Engine Chain (Chain) – An algorithm that allows configuring different routing engines on different parts of the IB fabric.
Version 5.2
187
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
•
Adaptive Routing (AR) – enables the switch to select the output port based on the port's
load. For more information see Appendix J:Adaptive Routing .
8.3.1.1 Configuring Routing and Quality of Service
The Routing Configuration Table sets (enables) the preferred routing algorithm supported by the UFM software as well as routing priorities.
UFM supports the following algorithms:
•
Shortest path (MINHOP) - This is the default algorithm
•
Up-down (UPDN)
•
Down-up (DNUP)
•
File-based (FILE)
•
FAT Tree (FTREE)
•
Layered Shortest Path (LASH)
•
Dimension Order Routing (DOR)
•
Torus-2QoS - For more information about configuring Torus-2QoS, see Torus-2QoS and
•
Routing Engine Chain (CHAIN)
•
PQFT (PQFT)
To set Routing Configuration
1. In the Configure tab, under Network Management section, click the Routing link.
188
2. Under Available, select the available Routing Engine.
3. Click the arrow icons to use or remove an algorithm and move up and down the list. The
Algorithm at the top of the list has the highest priority and will be chosen as the Active
Routing engine. If the settings for this algorithm are not successful, UFM takes the next available algorithm, and then repeats until the algorithm settings are successful. For more information, see the Routing Configuration Icons table following this procedure.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
4. Check Use Unicast Cache.
When enabled, Unicast routing cache prevents routing recalculation (which is a heavy task in a large cluster) when no topology change was detected during the heavy sweep, or when the topology change does not require new routing calculation (for example, when one or more CAs/RTRs/leaf switches going down, or one or more of these nodes coming back after being down).
5. Examine the Routing Details.
•
There is a tooltip showing you the full path of where the file is located:
8.3.2
•
The Routing details files are optional. If needed, the Routing Details files must be created manually in /opt/ufm/files/conf/.
For more information, see the Routing Details table following this procedure.
6. Click Apply and OK.
The selected Algorithm becomes the Active Routing Engine.
Table 53: Routing Configuration Icons
Icon Description
Click this button to add an algorithm to be used.
Click this button to remove a used algorithm.
Click this icon to set the priority and move the algorithm up the list.
Click this button to move the algorithm down the list.
Table 54: Routing Details
Field/Box
LID Matrix
LFT File
Root GUID File
Compute Nodes File
GUID Routing Order
File
Description
File holding the LID matrix dump configuration.
File holding the LFT routing configuration.
File holding the root node GUIDS (for fat-tree or Up/Down)
File holding GUIDs of compute nodes for fat-tree routing algorithm.
File holding the routing order GUIDs (for MinHop and Up/Down)
Torus-2QoS and Traffic-Aware Torus Routing
Torus-2QoS is a routing algorithm designed for large-scale 2D/3D torus fabrics. The torus-
2QoS routing engine can provide the following functionality on a 2D/3D torus:
•
Routing that is free of credit loops.
•
Two levels of Quality of Service (QoS), assuming switches and channel adapters support eight data VLs.
189
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
•
The ability to route around a single failed switch, and/or multiple failed links, without introducing credit loops, or changing path SL values.
•
Very short run times, with good scaling properties as fabric size increases.
In addition, UFM enables you to configure Torus-2QoS routing to be traffic aware. This enhancement optimizes the distribution of paths in the fabric topology when the traffic in the fabric occurs only between neighboring nodes.
When the fabric contains switches connected with multiple parallel links, routes are distributed in a round-robin fashion across these links. The default is to visit HCA ports in increasing port order on destination switches.
The following example shows routing optimized for neighbor-based traffic:
•
Communication to nodes 7 and 10 is via link A
•
Communication to nodes 8 and 11 is via link B
•
Communication to nodes 9 and 12 is via link C
Each node receives its own link; no congestion occurs.
The same concept is applied to horizontal traffic.
8.3.2.1 Configuring Torus-2QoS and Traffic-Aware Torus Routing
The following steps are required to enable and configure Torus-2QoS and traffic-aware
Torus routing in UFM.
1. Set Torus-2QoS general parameters in the Torus-2QoS configuration file in
/opt/ufm/files/conf/torus.2QoS.conf. For details, see Setting Torus-2QoS Parameters .
190
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
For traffic-aware Torus routing, set additional port-group parameters in the Torus-2QoS
configuration file. For details, see Setting Traffic-Aware Torus Routing Parameters
2. In the UFM GUI: a. Click the Config tab. b. In the Routing Table section, click the Routing Table link. c. Choose Torus-2QoS as the routing algorithm to be used.
Version 5.2
See Configuring Routing and Quality of Service for details.
8.3.2.2 Setting Torus-2QoS Parameters
Define general Torus-2QoS routing options in the /opt/ufm/files/conf/torus-2qos.conf configuration file.
Table 55: General Torus Routing Options
Option
[torus|mesh] x_radix[m|M|t|T] y_radix[m|M|t|T] z_radix[m|M|t|T] p_link sw0_GUID sw1_GUID yp_link sw0_GUID sw1_GUID zp_link sw0_GUID sw1_GUID xm_link sw0_GUID sw1_GUID ym_link sw0_GUID sw1_GUID zm_link sw0_GUID sw1_GUID x_dateline position y_dateline position z_dateline position
Description
Configures the topology that torus-2QoS will try to construct The first keyword in the configuration must be
torus or mesh.
A 2D topology can be configured by specifying one of
x_radix, y_radix, or z_radix as 1. An individual dimension can be configured as mesh (open) or torus (looped) by suffixing its radix specification with one of m, M, t, or T.
These keywords are used to seed the torus/mesh topology.
For example, xp_link 0x2000 0x2001 specifies that a link from the switch with node GUID 0x2000 to the switch with node GUID 0x2001 would point in the positive x direction, while xm_link 0x2000 0x2001 specifies that a link from the switch with node GUID 0x2000 to the switch with node GUID 0x2001 would point in the negative x direction. All the link keywords for a given seed must specify the same "from" switch.
The dateline keywords configure dateline positions that do not change relative to physical switch locations. This
191
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
Option
next_seed portgroup_max_ports max_ports
Description
configuration ensures that path SL values do not change under any conditions for which it can still route the fabric.
The dateline for a torus dimension is always between the switch with coordinate 0 and the switch with coordinate radix-1 for that dimension. By default, the common switch in a torus seed is taken as the origin of the coordinate system used to describe switch location. The position parameter for a dateline keyword moves the origin (and hence the dateline) the specified amount relative to the common switch in a torus seed.
Specifies that the following link and dateline keywords apply to a new seed specification.
This keyword specifies the maximum number of parallel inter-switch links, and the maximum number of host ports per switch that torus-2QoS can accommodate. The default value is 16. Torus-2QoS will log an error message during topology discovery if this parameter value should be increased. When this keyword appears multiple times, the last instance prevails.
For more information about parameters see the configuration file and the Mellanox OFED
User Manual.
8.3.2.3 Setting Traffic-Aware Torus Routing Parameters
For traffic-aware Torus routing, the additional port-order options in the Torus-2QoS configuration file determine the order in which HCA ports on the destination switch are selected. The configured order affects the distribution of routes across the links between ports.
In the example file in the following section, port order for axis x is configured: port_order_x 7 10 8 11 9 28 25 12 26 27 29 30
If there are two outgoing ports, P1 and P2, on the outgoing switch:
•
PI will communicate with HCA ports 7, 8, 9, 25, 26, 29
•
P2 will communicate with HCA ports 10, 11, 28, 12, 27, 30
Table 56: Traffic-Aware Torus Routing Options
Option
port_order_x p1 p2 p3 ... port_order_y p1 p2 p3 ... port_order_z p1 p2 p3 ...
Description
These keywords represent the order in which HCA ports on a destination switch are selected when computing routes.
You can change the order that HCA ports are selected for each axis, x, y, z.
Duplicate values in the list are ignored.
8.3.2.4 Example of Torus-2QoS Configuration File
# Look for a 2D (since x radix is one) 4x5 torus. torus 1 4 5
192
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
# y is radix-4 torus dimension, need both
# ym_link and yp_link configuration. yp_link 0x200000 0x200005 # sw @ y=0,z=0 -> sw @ y=1,z=0 ym_link 0x200000 0x20000f # sw @ y=0,z=0 -> sw @ y=3,z=0
# z is not radix-4 torus dimension, only need one of
# zm_link or zp_link configuration. zp_link 0x200000 0x200001 # sw @ y=0,z=0 -> sw @ y=0,z=1 next_seed yp_link 0x20000b 0x200010 # sw @ y=2,z=1 -> sw @ y=3,z=1 ym_link 0x20000b 0x200006 # sw @ y=2,z=1 -> sw @ y=1,z=1 zp_link 0x20000b 0x20000c # sw @ y=2,z=1 -> sw @ y=2,z=2 y_dateline -2 # Move the dateline for this seed z_dateline -1 # back to its original position.
# If OpenSM failover is configured, for maximum resiliency
# one instance should run on a host attached to a switch
# from the first seed, and another instance should run
# on a host attached to a switch from the second seed.
# Both instances should use this torus-2QoS.conf to ensure
# path SL values do not change in the event of SM failover.
# port_order defines the order on which the ports would be
# chosen for routing per each axis. port_order_x 7 10 8 11 9 28 25 12 26 27 29 30 port_order_y 9 10 7 11 9 12 25 28 30 29 27 26 port_order_z 11 8 7 10 9 25 12 28 26 29 30 27
8.4 Configuring Event Management
The Event Management allows you to define how fabric events can trigger alarms for effective troubleshooting and fabric maintenance.
The Event Management Policy Table allows you to:
•
Set the thresholds
•
Send specific event by email
•
Define whether an event message should be added to the events log
•
Set alarm lifetime (before it is automatically cleared by UFM)
The default lifetime for most alarms is set to 5 minutes (300 seconds).
To set Event Management
1. In the Config window, click Event Management. The Policy Table appears.
Version 5.2
193
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
2. Click on the drop-down button to search for the event name.
194
The Policy Table keeps a database of all events and attributes.
Events are reported by setting the following parameters:
Table 57: Policy Table Options
Option
GUI
Alarm
Log File
Threshold
TTL
Description/Instructions
When selected the corresponding events will be sent a list of recipients
according to Configuring Email-on-Events
When selected, the corresponding events are displayed in the Events tab in the GUI.
Select the Alarm option to trigger an alarm for a specific event. When selected, the alarms will appear in the Managed Devices/Alarms tab. See
Manage Devices Window Categories (Tabs) and
Select the Log File option if you would like the selected event to be reported in a log file.
An alarm will be triggered when the number of times an event occurs is greater than or equal to the threshold, or when the traffic/error rate exceeds the threshold.
For example: when PortXmit Discards is set to 5 and the counter value grows by 5 units or more between two sequential reads, an alarm is generated.
TTL (Alarm Time to Live) sets the time during which the alarm on the event is visible on UFM GUI. TTL is defined in seconds.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
8.4.1
Option
Severity
Description/Instructions
CAUTION: Setting the TTL to 0 makes the alarm permanent, meaning that the alarm does not disappear from the GUI until cleared manually.
From the drop-down list, set the event Severity according to the following: Info, Warning, Minor, and Critical.
Table 58: Additional Policy Table Options for Advanced License
Option
Script
SNMP
Description/Instructions
When selected, UFM invokes the Event script, if specified in the properties for the event source object, and the general script. For more
information, see Event-Triggered Scripts
The UFM Server will send events to third-party clients by means of
SNMP traps.
Select the event SNMP check box option to enable the system to send an
SNMP trap for the specific event. The SNMP trap will be sent to the port defined in Configuration file located under: /opt/ufm/conf/gv.cfg. For
more information, see SNMP Settings
See Introduction to Alarms for an overview.
See Supported Port Counters and Events for detailed information on port counters and
events.
Configuring Email-on-Events
UFM enables you to configure each event to be sent by email to a list of pre-defined recipients. Every 5 minutes (configurable) UFM will collect all “Mail” selected events and send them to the list of pre-defined recipients. By default, the maximum number of events which can be sent in a single email is 100 (configurable, should be in the range of 1 - 1000)
The order of events in the email body can be set as desired. The available options are: order by severity or order by time (by default: order by severity)
To change email-on-events setting:
1. Edit the /opt/ufm/conf/gv.cfg file.
2. Go to section “[Events]” and set the relevant parameters:
•
sending_interval (default = 5) - Time interval for keeping events (minimum 10 seconds, maximum 24 hours)
•
sending_interval_unit (default = minute) - Optional units: minute, second, hour
•
cyclic_buffer (default=false) - If the cyclic buffer is set to true, older events will be dropped, otherwise newer events will be dropped (if reaches max count)
•
max_events (default = 100) - Maximum number of events to be sent in one mail
(buffer size), should be in the range of 1 – 1000
•
group_by_severity (default = true) - Group events in mail by severity or by time
Version 5.2
195
Mellanox Technologies Confidential
Version 5.2
To receive the email-on-events:
•
Configure SMTP settings via Event Management
a. Go to the Cofig Tab. b. Select “Configure SMTP” at the Event Management panel
Configuring and Optimizing the Fabric
c. Configure the SMTP settings. See section Configuring the SMTP Server .
•
Configure the recipient list for receiving email-on-events
a. Go to the Cofig Tab. b. Select “Configure Recipients” at the Event Management panel.
196
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Once selected, the following window is presented:
Version 5.2
8.4.2
c. In the Recipients List window: i. Enter a valid recipient (a valid email address) at the text field. ii. Click the green plus sign to add the recipient. iii. Click Apply. The chosen recipient will be added to recipients list. iv. Click OK
Once the SMTP and the Recipient list are configured, the selected events (“Mail” checkbox), are send to the recipients, after no longer than the configured interval (by default: 5 minutes) from the time of first event occurrence.
Event-Triggered Scripts (Advanced License Only)
UFM enables you to customize event handling. You can specify whether to run a script for a specific event in the Event Policy Table.
When the Script option is checked for a specific event, the following scripts are triggered when that event occurs:
•
The generic events script
•
The script corresponding to the alerted object (Optional)
The object-related script runs first (if the event occurred on the corresponding object), then the generic script runs.
The following parameters are passed to the script:
•
Event ID
•
Event Category
•
Timestamp
197
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
•
Source Object Full name, e.g. env1.ls1
•
Event Severity
•
Event Description
•
User-defined parameters, for more information, see Setting User-Defined Attributes
Generic Script
The generic script is located in /opt/ufm/files/scripts/ufm_event.sh. You can add to and modify the script according to the required functionality.
NOTE: Please make sure script is executable. If not, please run: chmod +x /opt/ufm/files/scripts/ufm_event.sh
The generic script runs each time an event with the Script option enabled occurs.
Object-Specific Script
An object-specific script can be any Linux executable file. Scripts must be located in the
/opt/ufm/files/scripts/ directory. You should specify the name of the script in the object properties.
NOTE: Please make sure scripts are executable. If not please, run: chmod +x script_file_name
for all the defined scripts.
An object-specific script runs only when an event with the Script option enabled occurs on the specific object.
You can define a single event script per object for the following objects:
•
Environment
•
Logical Server
•
Network
•
Network interface
•
Device
198
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
8.5 Configuring the SMTP Server
SMTP configuration is required to set both the Daily Report and the Email-on-Events
features (see section Configuring Email-on-Events ).
1. In the SMTP Configuration window, enter the following information:
Version 5.2
•
SMTP Server: the IP or host name of the SMTP server.
Examples:
• if mail service is installed ,localhost is a valid value for this field, but usually it cannot send mails outside the local domain.
• smtp.gmail.com
•
SMTP Port (Default value is 25)
•
Use Authentication. By Default this field is unchecked. If checked, you must supply a username and password in the respective fields
•
Use SSL (Default value is false – not using SSL)
•
Sender Name (The name that will be displayed in the email header)
•
Sender Address (A valid email address that will be displayed in the email header)
2. Click the "Apply" button. All configuration of the SMTP server will be saved in the UFM
Database.
199
Mellanox Technologies Confidential
Version 5.2
8.6
Configuring and Optimizing the Fabric
SNMP Settings (Advanced License Only)
When UFM is running in Advanced mode, the GUI Policy Table shows the SNMP traps.
You can then modify and save an SNMP Trap flag for each event. When UFM runs in Basic mode, the SNMP Trap flags are not shown. (SNMP settings are enabled only after installation of the UFM Advanced license.)
UFM sends SNMP Trap using version SNMPV2 to the default port 162.
To set the SNMP properties
•
Run script
/opt/ufm/scripts/change_fabric_config.sh -a "snmp_listeners=<IP Address
1>[:<port 1>][,<IP Address 2>[:<port 2>]…]"
Example
/opt/ufm/scripts/change_fabric_config.sh -a
"snmp_listeners=host1,host2:166"
8.6.1 MIB Interface Description
NMP Trap OID
•
UFM Traps extend the volsnmpTraps object from the UFM MIB { enterprises
5206.200.12 }
•
The TrapOID will be as follows: volsnmpTraps.< UFM Event ID>.
For example, if a trap is generated for a specific event (331,"Node is Down"), it will be defined as volUFMNodeIdDown ::= { volsnmpTraps 331 }.
The UFM MIB file is located at /opt/ufm/files/conf/MELLANOX-UFM-TRAP-MIB.mib.
See the UFM MIB file for details on the SNMP trap structure.
8.7 Configuring Site Access Credentials
You can configure default access parameters for remote administration via the following protocols:
•
SSH
•
IPMI
•
SNMP
•
Telnet
Default credentials are applicable to all switches and servers in the fabric.
NOTE: The default SSH (CLI) switch credentials match the Grid Director series switch.
To change the credentials for IS5030/IS5035 edit the [SSH_Switch] section in the gv.cfg
file.
To set Site Access Credentials:
1. In the Config window, click Access Credentials.
200
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
8.8
2. Select the tab for the required protocol.
3. Define access parameters for the remote user as described in the following table.
4. Click Apply or OK.
Table 59: Site Access Credential Parameters
Parameter
User
Password
Port
Timeout
Description
The name of the user allowed remote access.
Enter the user password, then re-enter to confirm.
Each communication protocol has a default port for connection. You can modify the port number, if required.
Each communication protocol has a default timeout, i.e. the maximum time, in seconds, to wait for a response from the peer. You can modify the timeout, if required.
Configuring Non-optimal Links
A non-optimal Link is a link between two ports that is configured to operate at a certain speed and width, but is operating at a lower rate. The Non-optimal links feature helps you identify potential link failures and reduce fabric inefficiencies.
Non-optimal links can be any of the following:
•
EDR links that operate in FDR,QDR,DDR or SDR mode
•
FDR links that operate in QDR,DDR or SDR mode
•
QDR links that operate in DDR or SDR mode
Mellanox Technologies Confidential
201
Version 5.2 Configuring and Optimizing the Fabric
•
DDR links that operate in SDR mode
•
12X links that operate in 4X or 1X mode
•
4X links that operate in 1X mode
NOTE: Causes for Non-Optimal Links
A link reduction occurs between a port in the InfiniBand fabric (either HCA ports or switch ports) and an invalid peer port. That is the port at the other side of the link fails due to malfunctioning, that could be caused by the link being unresponsive to MADS, or configuration error.
8.8.1 Configuring all Non-Optimal Links in the Fabric
The non-optimal Links window allows you to set the preferred action for Non-optimal Links.
To set Non-optimal Links:
1. Select the Config tab.
2. In the non-optimal links area click non-optimal links.
202
3. From the drop-down menu, select the action for Non-optimal Links behavior. The Drop
Down menu defines the default behavior. Options are: Ignore (default), Disable, and
Reset.
•
Select Disable to disable the Non-optimal Link.
•
Select Ignore to ignore the Non-optimal Links.
•
Select Reset to reset the Non-optimal Links
4. Click Reset to clear all non-optimal links in the fabric (immediate action) The button allows you to do alternative action on-demand.
5. Click Disable all non-optimal links in the fabric (immediate action). The button allows you to do alternative action on-demand.
6. Click Apply and OK.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
NOTE: The Status label at the top of the box shows the current link behavior.
Version 5.2
8.8.2
8.9
Resetting and Disabling a Specific Non-optimal Link
In order to configure a single Non-optimal Link, disable the faulty port.
To disable the faulty port
1. Select the Manage tab then the Ports tab.
2. On the Left Tree, click Sites, drill-down and select the device that requires port disconnection.
3. In the Main pane, right-click the faulty port and select:
•
Reset to reset the selected port and reset the Non-optimal Link. When prompted, confirm by clicking OK.
•
Disable to disable the selected port and clear the Non-optimal Link. When prompted, confirm by clicking OK.
Note that these operations may take a few seconds before they are reflected in the UFM.
Multicast Optimization (Advanced License Only)
The Multicast (MC) optimization routing algorithm:
•
Avoids temporary MC loops and packet drops during Multicast Forwarding Table (MFT) distribution time.
•
Optimizes the MC tree distribution and keeps the configuration robust.
•
Prevents application downtime, and reduces load and congestion.
Multicast optimization results in:
•
Multicast tree stability and reduced switch reconfiguration time.
•
Distribution of roots among all available switches for balanced traffic and reduced congestion.
•
MC tree path optimization for a more balanced load on the fabric and improved performance.
To set Multicast optimization
•
In the /opt/ufm/files/conf/opensm/opensm.conf file, set the multicast_optimization
parameter to TRUE. The default is FALSE.
8.10 Managing Congestion Control
The Congestion Control mechanism controls traffic entry into a network and attempts to avoid oversubscription of any of the processing or link capabilities of the intermediate nodes and networks. Congestion Control also reduces resources by reducing the rate of sending packets.
You can use Congestion Control Manager for devices based on InfiniScale IV and later.
203
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
8.10.1 Activating the Congestion Control Manager
You configure activation of the Congestion Control Manager in the /opt/ufm/conf/gv.cfg configuration file. It is recommended to configure congestion control before you run UFM, since changes to congestion control configuration require a UFM restart.
Note: In general, the default Congestion Control Manager configuration will control congestion in the fabric. For more information about Congestion Control Manager options, see the Mellanox OFED User Manual.
To activate the Congestion Control Manager
In /opt/ufm/conf/gv.cfg/ change the following sections as shown:
•
Add ccmgr to the event_plugin_name option:
# Event plugin name(s) event_plugin_name = osmufmpi ccmgr
•
Add the Congestion Control Manager options file to the event_plugin_options option:
# Options string that would be passed to the plugin(s) event_plugin_options = ccmgr --conf_file /opt/ufm/files/conf/opensm/ccmgr.conf
By default, congestion control is enabled in the Congestion Control Manager options file
(enable TRUE).
If UFM is running, restart UFM.
To disable Congestion Control
In /opt/ufm/files/conf/opensm/cc-mgr.conf, change: enable TRUE to enable FALSE
If UFM is running, restart UFM to update configuration file settings.
To deactivate Congestion Control
1. Disable congestion control in the Congestion Control Manager options file.
2. In //opt/ufm/conf/gv.cfg remove ccmgr from event_plugin_name.
3. Restart UFM.
8.10.2 Example of Congestion Control Manager Options File
The following is an example of the /opt/ufm/files/conf/opensm/cc-mgr.conf file.
##################################
# General Options
# Enable/Disable Congestion Control mechanism on fabric nodes
# Values: < TRUE | FALSE > enable TRUE
# Congestion Control Key (64-bit value) cc_key 0
# Indicate number of nodes.
# CC Table values are calculated based on this number.
# Values: [0-48K]
204
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
# Default: 0 (base CCT calculation on the current subnet size) num_hosts 0
##################################
# Switch Options
# Indicates how aggressive congestion marking should be.
# Values: [0-0xf] (0 - no packet marking, 0xf - very aggressive) threshold 0xf
# The value that provides the mean number of packets
# between marking eligible packets with a FECN.
# Values: [0-0xffff] marking_rate 0xa
# Any packet less than this size [bytes] will not be marked with FECN.
# Values: [0-0x3fc0] packet_size 0x200
##################################
# CA Options
# Specifies congestion control attribute for this port
# Values: 0 - QP based congestion control,
# 1 - SL/Port based congestion control port_control 0
# An array of sixteen bits, one for each SL.
# Each bit indicates whether or not the corresponding
# SL entry is to be modified. ca_control_map 0xffff
# Sets the CC Table Index (CCTI) increase.
# Values: [0-127]
# 0: no CCTI increase - this port will not react upon receiving BECNs ccti_increase 1
# Sets the congestion log event trigger threshold.
# Values: [1-126] trigger_threshold 2
# Sets the CC Table Index (CCTI) minimum.
# Values: [0-127] ccti_min 0
# Sets all the CC Table entries to a specified value (first entry will remain 0).
# Values: <comma-separated list>
# Up to 128 (CCT size) items in the list.
# If less than 128 items provided, rest of the CCT will be filled with the last item's value.
# 0: CCT calculation based on number of nodes. cct 0
# Sets for all SL's the given CCTI timer.
# Values: [0-0xffff]
# 0: CCTI timer calculation based on number of nodes. ccti_timer 0
####################################
# CC MGR Options
# When number of errors exceeds 'max_errors' of send/receive errors or
# timeouts in less than 'error_window' seconds, CC MGR will abort and
# will allow OpenSM to proceed.
# Values for both options: [0-0xffff]
# max_errors = 0: zero tolerance - abort configuration on first error.
# error_window = 0: mechanism disabled - no error checking.
Version 5.2
205
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric max_errors 5 error_window 5
# CC MGR will collect statistics from all nodes every cc_statistics_cycle
[seconds]
# Values: [0-3600]
# 0: no statistics will be collected. cc_statistics_cycle 20
8.11 Managing Lossy Configuration (Advanced License Only)
InfiniBand by default is a lossless network. This is achieved by credit-based communication between neighbor ports. When a port has no remaining credit, additional packets for that port wait on an outgoing queue on the sender port. In some scenarios this can unnecessarily slow down services.
In these cases, it can be preferable to enable "lossy" behavior (dropping packets) for specific nodes to optimize overall application performance.
There are three types of lossy configurations:
•
Lossy-Ingress [in]
Allow dropping of packets sent from switch to host.
•
Lossy-Egress [out]
Allow dropping of packets sent from host to switch.
•
Lossy-Bi-Di [bidi]
Allow dropping of packets in both directions.
Lossy Configuration Manager enables and configures lossy parameters on fabric switches. It scans all the fabric nodes, identifies the switches that support Lossy Configuration, and configures the Lossy functionality on the switch or host HCA according to the specified configuration file.
Lossy Configuration Manager is a Subnet Manager plug-in, and is installed as part of UFM installation. By default, Mellanox InfiniScale IV switches and ConnectX are set with lossy configuration.
This feature is available on:
•
InfiniScale IV switches, FW version 7.4.0 and later
•
ConnectX HCA, FW version 2.8.600 and later
Note: Switches with firmware versions that do not support Lossy Configuration will slow down the Lossy Configuration Manager because of timeouts on the Lossy-related queries to these switches. For smooth operation, use switches that support Lossy Configuration.
8.11.1 Activating Lossy Configuration Manager
Lossy Configuration Manager activation is configured in the /opt/ufm/conf/gv.cfg file and then it reads the configuration file with the lossy configuration required for the fabric. The
Lossy Configuration Manager is enabled and disabled in the configuration file. By default,
Lossy Configuration is disabled. To activate lossy configuration, enable it before you run
UFM.
206
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
To activate the Lossy Configuration Manager plugin
In /opt/ufm/conf/gv.cfg change the following sections as shown:
•
Add lossymgr to the event_plugin_name option:
# Event plugin name(s) event_plugin_name = osmufmpi lossymgr
•
Add the Lossy Configuration Manager options file to the event_plugin_options option:
# Options string that would be passed to the plugin(s) event_plugin_options = --lossy_mgr [-r <retries-number>] [-t <timeout>] f /opt/ufm/files/conf/opensm/lossymgr.conf where: retries-number
is the maximum number of retries for a query. timeout
is the timeout in seconds between retries.
To enable Lossy Configuration
In /opt/ufm/files/conf/lossymgr.conf, change: mode=disable to mode=enable
If UFM is running, restart UFM to update configuration file settings.
After enabling lossy configuration, you can configure lossy configuration options in the
UFM GUI, or in the lossy configuration file. For information about all Lossy Configuration
options, see Lossy Configuration Manager Options
To disable Lossy Configuration
In /opt/ufm/files/conf/lossymgr.conf, change: mode=enable to mode=disable
If UFM is running, restart UFM to update configuration file settings.
Note: The Lossy configuration is cleared from the node on rebooting the node.
To deactivate Lossy Configuration
1. Disable Lossy configuration in the Lossy configuration options file.
2. In /opt/ufm/conf/gv.cfg remove lossymgr from event_plugin_name.
3. Restart UFM.
8.11.2 Lossy Configuration Manager Options
Enable Lossy Configuration and change Lossy Configuration Manager options in the
/opt/ufm/files/conf/lossymgr.conf configuration file:
•
The Lossy configuration file is case sensitive.
Version 5.2
207
Mellanox Technologies Confidential
Version 5.2 Configuring and Optimizing the Fabric
•
Unless stated otherwise, Lossy configuration file settings are updated every Lossy
Configuration Manager cycle, which in turn is executed at every heavy sweep (subnet up) of the Subnet Manager.
After enabling lossy configuration, you can also configure lossy configuration options in the
Subnet Manager configuration dialog box in the UFM GUI. For more information, see SM
To update all settings, restart UFM.
Table 60: Lossy Configuration Manager Options
SM Configuration (UFM
GUI)
Lossy Configuration
File Option
supported_devids = devid1, devid2, ...
Description
mode = <enable | disable>
[vl_mode = all:
[bidi|off], [bidi|off],
[...]]
[vl_mode = leaf:
[in|out|bidi|off],
[in|out|bidi|off], [...]]
By default, InfiniScale IV switches and
ConnectX are set with lossy configuration.
You can specify a list of additional device IDs to be set with lossy configuration. If a device in the list does not support lossy configuration, an error message is written to the SM log file.
The supported_devids configuration is set only on SM startup and changes take effect only after SM restart.
Enable or disable Lossy Configuration
Manager operation.
When changing from enable to disable, all lossy configuration is removed from all fabric nodes, including from down ports. New nodes with lossy configuration added to the fabric after disabling the manager will not be reconfigured.
Default: disable
Set the lossy configuration on all the lossy supported fabric nodes.
The required lossy configuration for each VL is given in the ordered list after
all:
<vl0 configuration>, <vl1 configuration>, ...
Lossy configuration for vl_mode=all can be bi-directional (bidi) or off.
The lossy configuration for all the VLs that are not in the list is off.
Set the lossy configuration only on lossy supported edge nodes.
The required lossy configuration for each VL is given in the ordered list after
leaf:
208
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
SM Configuration (UFM
GUI)
Lossy Configuration
File Option
Description
<vl0 configuration>, <vl1 configuration>, ...
Lossy configuration edge nodes can be
in, out, bidi, or off.
The lossy configuration for all the VLs not in the list is off.
If vl_mode = all is also specified, the lossy configuration for edge nodes is determined by the leaf configuration.
Version 5.2
8.11.3 Example of Lossy Configuration Manager Options File
# list of additional device ids that support lossy configuration
# InfiniScale IV switches and ConnectX are supported by default
# this list is read only on startup supported_devids = 0x1234,0x634a
# operation mode
# mode = [enable | disable] mode = enable
# per vl configuration
# if vl is not present its mode is off
# vl_mode scope: vl0_mode ,vl1_mode,vl2_mode,.vl14_mode
#[vl_mode = all: [in|out|bidi,off], [in|out|bidi,off],
#[vl_mode = leaf: [in|out|bidi,off], [in|out|bidi,off], [in|out|bidi,off] , vl_mode = all: off,off,off,bidi vl_mode = leaf: off,off,in,bidi
8.12 Configuring UFM for SR-IOV (Advanced License Only)
Single-root I/O virtualization (SR-IOV) enables a PCI Express (PCIe) device to appear to be multiple separate physical PCIe devices.
UFM is ready to work with SR-IOV devices by default. You can fine-tune the configuration via the Subnet Manager configuration.
To modify UFM configuration for SRIOV devices
•
In the /opt/ufm/files/conf/opensm.conf file, modify the options as described in the following table.
Argument Value
sm_assign_guid_func
• uniq_count
• base_port (default) sm_assigned_guid
Description
Alias GUID assignment algorithm for
Virtualized environment
Any value between 0 and ff Set the SM assigned GUID byte when sm_assign_guid_func configuration option is set to uniq_count. The GUID is formed from OpenFabrics OUI followed by 40 bits xy 00 ab cd ef where:
• xy is the SM assigned GUID byte
209
Mellanox Technologies Confidential
Version 5.2
Argument Value
Configuring and Optimizing the Fabric
Description
• ab cd ef is an SM autogenerated 24 bits
The configured SM assigned GUID byte must be subnet unique.
Default is 0
Allow both full and limited membership on the same partition allow_both_pkeys
•
TRUE
•
FALSE (default )
8.13 Isolating a Switch from Routing
UFM has the ability to isolate particular switches from routing in order to perform maintenance of the switches with minimal interruption to the existing traffic in the fabric.
The configuration is done via UFM Subnet Manager as follows:
To isolate a switch:
1. Create a file that includes either the Node GUIDs or System GUID of the switches under maintenance.
2. Set the file name of the parameter held_back_sw_file in the
/opt/ufm/files/conf/opensm.conf
file the same as the file created in Step 1.
3. Run kill -s HUP `pidof opensm`
Once OpenSM completes rerouting, the traffic does not go through the ports of isolated switches.
To attach the switch to the routing:
1. Remove the GUID of the switch from the list of isolated switches defined in Step 1 of the isolation process.
2. Run kill -s HUP `pidof opensm`
Once OpenSM completes rerouting, the traffic will go through the switch.
210
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
9 Monitoring the Fabric
9.1
9.1.1
Overview of Monitoring the Fabric
UFM monitoring allows tracking Network bandwidth, congestion, errors, latency, host resource utilization and traffic.
The UFM offers the following monitoring features:
•
Real-time monitoring views
•
Alerts notification for error threshold exceeded
•
Multiple Objects: diverse monitoring views
•
Multiple attributes monitoring
•
SUM, AVG, MIN, MAX, RAW functions for monitored attributes
•
Intelligent Counters: provide error and congestion counters
•
Congestion Map: switch tiers-based congestion map
•
Various graphical presentations: Line and Bar Graphs, Pie Charts, Histograms, Table
views, Top N views
•
InfiniBand port-based error counters
•
InfiniBand congestion XmitWait counter-based congestion measurement
•
InfiniBand port-based bandwidth data
The UFM shows performance monitoring data in the following views:
•
Dashboard Window
•
View Window
•
Monitor Window
•
Port Counters tab (at the bottom of the UFM screen)
Objects and Attributes
The monitoring model is based on an Object-Attribute Matrix, where Object is a model object to be monitored and Attribute is a monitored parameter.
A Monitored Object can be virtually any physical or logical object of the Object Model, including environment, logical server, network, site, device, and port.
A Monitored Attribute can be RAW or Complex.
•
A RAW Attribute is a simple attribute to be monitored, e.g., SymbolErrorCounter. You may set RAW monitoring attributes (counters) according to the following categories: bandwidth, congestion, errors, latency, host resource utilization, traffic, and application and protocol-related attributes. Each category contains a set of counters.
•
A Complex attribute is a function of raw attributes, e.g., SymbolErrorCounter +
LinkErrRecoveryCounter + LinkDownedCounter.
Version 5.2
211
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
9.1.2 Introduction to Alarms
UFM allows you to identify any problem including ports and device connectivity using events and alarms. Problems can be detected both prior to running applications and during standard operation.
Device events are categorized by the different generator entities, namely the SM (Subnet
Manager), PM (Performance Monitor), and the Device Manager.
Various tools are available to perform fabric diagnostics.
For information on configuring events and alarms in the Policy Table, see Configuring Event
9.1.2.1 Events
Events are reported by means of:
•
Logging. For more information, see UFM Logging
•
Event Table at the bottom of the UFM window
•
Event Severity is listed as follows: Info, Warning, Minor, and Critical, as defined in the Policy Table
•
Events and alarms are shown as they occur in real-time
You can save a snapshot of the current events in CSV format from the Event table.
9.1.2.2 Alarms
•
Events trigger alarms (except for "normal" events. i.e., info events) when they exceed a pre-defined threshold. The threshold can be changed in the Events Management Table.
•
The Policy Table maps events to alarms, and alarms are mapped to categories.
•
Throughout UFM, you can view the current alarms reflected on the devices, according to the highest severity. The alarm appears in the fabric tree, Manage Devices category
(tables), View category (maps and tables), and in the Internal Structure. The Manage
Device tool allows you to analyze the health of each port and device in the fabric.
•
UFM allows you to drill down on the problem object, and narrow down the problem port and go to its peer port by using the Goto Peer and Attached to... Drilling down to the object enables you to perform diagnostics on both sides of the link.
•
Alarms are automatically cleared after the Time to live (TTL) of the Event Management has passed. (TTL is set in the Events Management Table).
•
Alarms can be manually cleared by right-clicking a specific alarm or on a specific alarmed object.
For more information on Event Management and the Policy Table, see Configuring Event
9.1.2.3 Thresholds
•
You can configure thresholds for each event in the policy table.
•
Events are issued when a threshold is exceeded.
212
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
Alarms are generated to reflect the severity of events.
•
Alarm severity is propagated to the higher level objects. In other words, an alarm on an object will reflect the most severe alarm level occurring on a dependent object, device, or port.
Version 5.2
9.2
9.2.1
9.2.2
Monitor Window
The Monitor view enables you to establish monitoring sessions with managed devices.
During the session, the UFM server polls the devices for the requested information.
You can initiate multiple monitoring sessions for any combination of health, traffic, and performance parameters. Data can be monitored on switches and fabric nodes, and can be aggregated on groups of objects.
You can save monitoring templates to re-use monitoring session parameters.
Various graphical representations are available to show the information: Table, Graph, Bar, and Pie Chart.
Saving a Monitoring Session Template (Advanced Licensed Only)
You can save the parameters of a monitoring session to a template.
To save a monitoring session as a template
1. Create a monitoring session. See Creating a Monitoring Session
2. To save the monitoring session as a template before running the session, click Save.
3. To save monitoring session parameters to a template while the session is running, in the toolbar, click the Save as Template ( ) button.
For information about running a monitoring session from a template, see Running a
Monitoring Session from a Template
Accessing the Monitor Window
The Monitor view allows you to create sessions to track the health of devices or ports according to their attributes.
To access the Monitor Window:
•
Click the Monitor tab .
The Performance Monitoring window opens.
213
Mellanox Technologies Confidential
Version 5.2
Figure 47: Performance Monitoring View
Monitoring the Fabric
9.2.2.1 Performance Monitoring Sections
The Performance Monitoring window splits into two sections:
•
The left pane contains a list of Monitoring templates
•
The main result panel.
You can create monitoring sessions for one or several nodes. You can select up to four monitoring attributes from the devices.
9.2.3 Creating a Monitoring Session
To create a Monitoring Session
1. Click Create New Session. The Monitoring Wizard opens displaying the Objects for
Monitoring pane.
214
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
2. From the drop-down list box, select the object type to be monitored:
•
Device (Computers and switches)
•
Site
•
Logical Server (applies to the Logical Servers Filter)
•
Computer
•
Switch
•
Gateway
The corresponding objects are displayed in the Available Choices list.
To further filter the list, enter a search string in the search field above the list. The list is filtered incrementally as you enter characters.
3. Move the objects to be monitored to the Your Choices list.
4. Click Next. The Attributes for Monitoring pane appears.
Mellanox Technologies Confidential
215
Version 5.2 Monitoring the Fabric
216
5. Select attributes for monitoring. For graphical display modes, you can select up to a total of 4 attributes.
The available tabs depend on the selected object type. For more information about the
attributes, see Monitoring Network Attributes
6. Click Next. The Visual Definitions pane appears.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
7. Set parameters as described in the table that follows this procedure. (To set the Period
and Interval parameters for Monitoring History, see Monitoring History )
8. To run the session, click Run. For information about viewing a monitoring session, see
Table 61: Monitoring Visual Definition Parameters
Field
Session Name
Function
Object Type
View Type
Procedure
Enter the desired name for the monitoring session.
Select the preferred statistical format:
•
RAW - Provides the whole statistical table of monitored attributes
•
AVG - Provides the average of all objects (according to the selected monitored attributes)
•
MIN - Minimum value of all objects (according to the selected monitored attributes)
•
MAX - Maximum value of all objects (according to the selected monitored attributes)
•
SUM - Provides the sum of all objects (according to the selected monitored attributes)
Select the object for which you would like to run a monitoring session.
Select the preferred graphical monitoring representation formats:
Graph, Table, Bar, Histogram, Pie Chart.
The representations show the object distribution relative to the measured attribute.
9.2.3.1 Monitoring Network Attributes
You can select up to four attributes per monitoring session from the Attributes tabs.
217
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
Attributes tabs for InfiniBand include:
•
Errors
Twelve basic InfiniBand Port counters are used for Network error indication.
•
Bandwidth
•
Tx Rate in MB/sec
•
Tx Rate Packet/sec
•
Rx Rate in MB/sec
•
Rx Rate Packet/sec
•
Congestion Rate (CBW)
See Supported Port Counters and Events for detailed information on these attributes.
The following tables describe the attributes for the CPU Data, Disk, and Memory tabs.
Table 62: CPU Data
Parameter
CPU Usage
CPU Sys
CPU Inter
CPU Ctx Switch
Description
Percentage of time the CPU was busy during the current interval averaged across all CPUs, which is the total percentage of time of
CPU in one of the following: system, user, nice, irq, soft-irq and
steal. Note that this does NOT include I/O wait time.
Percentage of time the CPU was executing in system mode during the current interval, including system, irq, soft-irq and steal modes.
To determine the amount of time spent as a user, subtract the amount of time in the system modes from the total CPU field.
Total number of interrupts/sec
Total number of context switches/sec
Table 63: Disk
Parameter
Disk KB Read
Disk Read
Disk Read Size
Disk KB Write
Disk Write
Disk Write Size
Description
KB read/sec
Number of reads/sec
Average read size in KB. This field is included only if --iosize is specified
KB written/sec
Number of writes/sec
Average write size in KB. This field is included only if --iosize is specified
Table 64: Memory
Parameter
Memory Free
Memory Buffer
Description
Total free memory
Memory used as system buffers
218
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Parameter
Memory Cache
Memory Inactive
Memory Slab
Memory Map
Description
Also commonly known as the file system buffer cache, since buffered I/O uses this memory to cache the data
Inactive memory.
Total memory allocated to slabs
Total mapped memory, which includes AnonPages
9.2.4
9.2.5
Version 5.2
Running a Monitoring Session from a Template (Advanced License
Only)
You can run a monitoring session from a template. You can change the parameters in the template and run the session without saving the changes, or you can save the changes to the the existing or a new template, and then run.
To run a monitoring session from a template
•
In the Monitoring Templates pane, right-click the template and select Run.
To change template parameters and run the monitoring session
1. In the Monitoring Templates pane, right-click the template and select Properties. The
Session Monitoring wizard opens displaying the current parameter settings.
3. When you have made all the changes:
•
To run the modified session without saving changes, in the last step of the wizard, click Run.
•
To save changes to the current template, click Save.
•
To save changes to a new template, click Save As, and enter a name for the template.
Monitoring Session Display
When you run a monitoring session, a new session tab is displayed in the Performance
Monitoring window.
The Monitoring window splits into three panes:
•
The main pane displays tabs with the monitoring sessions
•
The monitored Data pane
•
The monitored objects pane
You can perform tasks on an object by right-clicking the selected monitored object. For more
information, see Tasks on Devices and Ports
The Monitoring window offers a variety of possible monitoring views. After you have created a session, you have the following options:
•
To stop and restart the session,
•
To show/hide attributes,
•
To split or merge monitoring layers,
219
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
•
To change the background color.
The following sections provide more details on the various representations in the Main pane, as well as details on Monitoring Data and Objects.
The following is an example of a graphical representation of a monitoring session:
Figure 48: Monitoring Session - Graphical Representation
9.2.5.1 Monitoring Window Toolbar
The following icons appear in the Monitor window.
Table 65: Monitor Window Icons
Icon
Start
Name Description
Restarts a Monitoring session.
Stop Stops a Monitoring session.
Save Saves monitoring session parameters in a new or existing template.
Exports monitoring session information to a CSV file. Export
Reset Resets cumulative counters.
9.2.5.2 Monitoring Session in Graph Format
After you create a session, you can track the status of the selected nodes attributes by viewing the Graphical Representation of each monitored session, as shown in the following examples.
Graphical Representation
220
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
The following figure shows an example of a Monitoring session in Graph format and RAW function settings, showing all the functions.
Figure 49: Monitoring Session - Graph Format
Version 5.2
9.2.5.2.1 Monitored Data
The Monitored Data area shows the monitored object and its attributes. There is a color legend for the specific attributes, so that you can easily retrieve and track the monitored object's attribute in the main pane.
Figure 50: Monitored Data
Table 66: Color Legend Parameters
Option
Color
Description
Assigned color for the monitored object according to the specific attribute (assigned by default by UFM)
221
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
Option
Object name
Attribute
Value
Description
Name of the object selected for monitoring
Attribute of the object to be monitored
Actual value of the attribute activity
9.2.5.2.2 Monitored Objects
The Monitored Objects area shows the list of objects that were selected for monitoring.
Figure 51: Monitored Objects
Click the object for which you would like to obtain more data. From the Managed Devices pane, you can view detailed information on the device, as well as drill down to the source of the problem.
9.2.5.2.3 Removing Attributes from a Session
To remove attributes from a session
•
On the toolbar, go to Attributes. From the menu, select the attribute to be removed.
The attribute is removed from the graphical representation, which is useful when there is no activity for a selected attribute.
9.2.5.3 Monitoring Session in Table Format
The following figure shows an example of a Monitoring session in Table format and RAW function settings, showing all the functions. The Table representation shows a row per node.
Figure 52: Monitoring Session - Table Format
9.2.5.3.1 Getting More Information
To get more information for an object and troubleshoot potential problems
1. Double-click the Monitored Object (row).
222
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
2. Click the object for which you would like to obtain more data.
The action takes you to Managed Devices, allowing you to view detailed information on the device, and to drill-down to find the source of the problem.
•
Filter: You may filter the table according to object or attribute.
•
Sorting the Table: You may sort the table by clicking the table titles.
•
Tasks: You may perform tasks on each object (if applicable) by right-clicking the object’s row and selecting the preferred task.
For more information, see Tasks on Devices and Ports
9.2.5.4 Monitoring Session in Bar Format
The following figure shows an example of Monitoring session in bar format and RAW function settings, showing all the functions:
Figure 53: Monitoring Session - Bar Format
Mellanox Technologies Confidential
223
Version 5.2 Monitoring the Fabric
9.2.5.4.1 Session Bar Format
You may split and merge the session, as well as remove attributes, just like in the Graphical representation.
9.2.5.5 Monitoring Session in Histogram Format
If you select the Histogram format when creating your session, the monitored parameters will appear according to the selected function. The following example shows a Monitoring session in RAW format (Histogram representation), showing all the functions.
Figure 54: Monitoring Session - Histogram Format
You may split and merge the session as well as remove attributes, as in the Graphical representation.
224
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
9.2.5.6 Monitoring Session in Pie Chart Format
The following figure shows an example of Monitoring session in Pie Chart format.
Figure 55: Monitoring Session - Pie Chart Format
Version 5.2
You may split and merge the session as well as remove attributes, just like in the Graphical representation.
9.2.6 Monitoring History (Advanced License Only)
The Monitoring History features enable historical analysis of fabric health and performance.
The following are the methods of saving historical information in UFM:
•
Saving Active Monitoring Sessions on the UFM Client Machine
•
Saving periodical snapshots of the fabric into CSV Files on the UFM Server
•
Saving and retrieving fabric wide information in a database format
9.2.6.1 Saving Active Monitoring Sessions on the UFM Client Machine
This feature enables the user to save an active monitoring session on the client machine, in a
CSV format, for future analysis.
It provides the highest flexibility and granularity, where each device can be selected and a frequency of down to 2 seconds can be selected.
To monitor historical data:
1. Create a monitoring session. See Creating a Monitoring Session
225
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
2. In the Visual Definition parameters window, define the monitoring period and sampling interval.
3. Run the monitoring session.
4. Click Save to CSV ( ) to save monitored information to a file.
Table 67: Monitoring History Parameters
Field
Period
Interval
Procedure
Sets the display buffer size as a function of the selected monitoring period.
Sets the attributes monitoring sampling interval in seconds (default:
5).
To retrieve the monitored session:
•
Open the CSV file from the directory where it was saved using an external application such as Excel.
9.2.6.2 Saving Periodic Snapshots of the Fabric (Advanced License Only)
UFM enables you to save multiple PortCounters CSV files for historical analysis according to a configuration parameter in UFM’s configuration file.
To configure the maximum number of CSV files to save:
1. Open the /opt/ufm/conf/gv.cfg configuration file
2. Set the max_files parameter as described in the following table.
Table 68: Port Traffic Write Interval
Parameter
max_files
Description
The maximum number of files to be saved in the
/opt/ufm/files/csv/
folder.
Values
1 (default), 1 -
100
226
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
NOTE: After you install the UFM Advanced license, the settings for maintaining historical port counter data are enabled.
To retrieve the monitored session:
•
Copy / open the CSV file from the above directory on the UFM Server to an external application such as Excel.
9.2.6.3 Monitoring History Database
The Monitoring History Database feature performs the following:
•
Saves port counters for the entire fabric in a database format. Local database and remote database modes are supported
•
Allows user to retrieve the information based on a flexible filter (e.g .devices, time)
•
Plots the information graphically on UFM GUI
9.2.6.3.1 Monitoring History Database System Requirements
Monitoring History database and processing is resource consuming. The minimal requirements for UFM server running Monitoring History are:
Fabric size
Up to 1000 nodes
1000 - 5000 nodes
RAM*
24GB
32GB
5000 - 10000 nodes 64GB
Recommended Dedicated partition size
300 GB
500 GB
800 GB
Above 10000 nodes Consult with Mellanox Support
Minimal Monitoring
History Save Frequency
30 sec
60 sec
120 sec
* Monitoring History Engine configuration will cause a run-time error for server with lower available RAM.
The monitoring history period that can be saved in the above disk spaces depends on the saving interval, and can be calculated using the below formula in a cluster with the following parameters:
•
The approximate disk utilization for one port statistics record is 512 bytes
•
The cluster size is P ports (including internal ports of modular devices)
•
The available disk size is D GB
•
The history save frequency is S sec
•
Number of days to keep is N days
512 * P * [ 3600 / S ] * 24 * N / 10^9 should be less than 0.7 * D
For example, if a cluster of 5000 ports and if the monitoring saving interval is 60 seconds and the disk size is 500 GB, the daily data size will be is 3.7 GB.
Version 5.2
227
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
9.2.6.3.2 Configuring Monitoring History Database
To initialize Monitoring History Database:
1. Install UFM.
2. Initialize Monitoring History. Run the script:
/opt/ufm/scripts/monitoring_history_configure.sh
3. Restart UFM.
Restart is required for updates to take effect.
NOTE: UFM HA servers' disks should be partitioned identically, although the Monitoring
History DB is not replicated.
UFM Supports 2 modes of operations:
•
Local Database on the UFM Server
•
Remote Database on a Different Server
Configuring a Local Database on the UFM Server
The Monitoring History database and the files are stored on the UFM server, hence the partition should be prepared accordingly. If the UFM is configured in a High Availability mode, Monitoring History partition should be created on both, the Master and the Standby servers.
Prior to configuring it, the following prerequisite must be implemented:
•
Clean, unmount partition (dedicated to the DB history only)
•
Formatted ext3 type
• dd if=/dev/zero of=<partition_name> bs=1k count=10
•
Run the script (-- local) (<DIR> -> partition name)
The srcipt's synopsis: monitoring_history_configure.sh --init --local --partition <DIR> --db-ip
<IP> --num-of-days-to-keep <days> [--db-name <name>] [--db-port <port>]
Table 69: Local Server Command Options
Parameter
--local
--partition
--db-ip
--db-name
--db-port
--num-of-days-to-keep*
Description
Creates the Monitoring History DB on a local server.
Monitoring History DB partition.
Database server IP address.
[Optional Parameter] Monitoring History DB name, default MonitoringHistory.
[Optional Parameter] SQL server port, default 3308.
The number of days the history will be stored, configurable. The number should not be greater than 3650
(10 years).
228
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
*The following is an example of how to calculate the length of historical data (--num-ofdays-to-keep ) in a cluster with the following parameters:
•
The approximate disk utilization for one port statistics record is 512 bytes
•
The cluster size is P ports (including internal ports of modular devices)
•
The available disk size is D GB
•
The history save frequency is S sec
•
Number of days to keep is N days
512 * P * [ 3600 / S ] * 24 * N / 10^9 should be less than 0.7 * D
For example, in a cluster of 5000 ports, with a disk of 500GB, and a saving interval of 60 seconds, the daily data size will be 3.7 GB, and the data history can be kept for 100 days.
If the Local Monitoring History Database server is running in UFM HA environment, the database is not replicated neither fails over from the primary server to the secondary. The parameter --db-ip must be defined in order for the database to continue to record the data upon UFM failover. The --db-ip parameter has to be set to the physical IP address of the primary server. In this case after failover, UFM will continue to record the data to the primary server and query the data from the primary server even if the failover was triggered by either the InfiniBand connectivity or the UFM software malfunction. If failover was triggered by a complete primary server malfunction or loss, the data will be lost.
Version 5.2
NOTE: Monitoring History Database data must be backed up as it is not replicated to the secondary UFM in HA setup. However it is still available even after UFM failover.
Configuring a Remote Database on a Different Server
The Monitoring History database is installed on an external server not on the UFM server.
Prior to configuring it, the following prerequisite must be implemented:
•
Create a DB on the remote host (mysql or mssql) and set the following:
•
DB name
•
Create a user for the DB with full permissions and a password
•
Run the script (-- remote) (<DIR> -> partition name)
The srcipt's synopsis:
229
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric monitoring_history_configure.sh --init --remote --partition <DIR> --dbengine <mysql or mssql> --db-name <name> --db-ip <IP> --db-user <user> -
-db-password <password> --db-port <port> [--num-of-days-to-keep <days>]
Table 70: Remote Server Command Options
Parameter
--remote
--partition
--db-engine
--db-name
--db-ip
--db-user
--db-password
--db-port
--num-of-days-to-keep
Description
Creates the Monitoring History DB on a remote server.
Monitoring History DB partition.
Database engine type: mysql or mssql
Monitoring History DB name.
Remote DB server IP address.
Remote DB server username.
Remote DB server password.
SQL server port, default 3308.
The number of days the history will be stored, configurable. The number should not be greater than 3650
(10 years).
NOTE: Once the servers are configured, restart the UFM server after the installation is completed for changes to take effect. By default, after the restart the feature is enabled.
The following are examples of remote mysql server:
•
On the remote host mysql> CREATE USER 'mysql'@'r-typhoon9.mtr.labs.mlnx' IDENTIFIED BY
"mysql";
Query OK, 0 rows affected (0.04 sec) mysql> GRANT ALL PRIVILEGES ON *.* TO 'mysql'@'r-typhoon9.mtr.labs.mlnx';
Query OK, 0 rows affected (0.02 sec) mysql> create database MonitoringHistory;
Query OK, 1 row affected (0.01 sec)
•
On the UFM host monitoring_history_configure.sh --init --remote --partition /dev/sda6 -db-engine mysql --db-name MonitoringHistory --db-ip 172.30.3.201 --dbuser mysql --db-password mysql --db-port 3306 --num-of-days-to-keep 30
NOTE: UFM High-Availability mechanism is not synchronizing the UFM Monitoring
History Database.
For monitoring history high availability, it is recommended to use the external database mode and address issues like RAID, Backup, etc. via the database native tools.
230
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
9.2.6.3.3 Configuring UFM Server for Monitoring History Database
UFM Server is configured via the file gv.cfg, section [MonitoringHistory]. The parameters of this section are modified by monitoring_history_configure.sh and should not be modified manually, except for the parameter history_report_timeout .
• history_report_timeout
defines a timeout of report retrieval in seconds
Monitoring History can be initialized several times on the local or the remote DB with the same partition. This can be achieved by using the existing configured Monitoring History partition. Run monitoring_history_configure.sh –init without the – partition option. The data of the existing Monitoring History partition is not harmed.
To disable Monitoring History Database Feature (It stops saving the monitoring data in the DB and disabling History reports feature but does not delete the DB):
•
Disable the Monitoring History Database Feature: monitoring_history_configure.sh --disable
This will update the configuration file gv.cfg.
To enable Monitoring History Database Feature:
•
Enable the Monitoring History Database Feature: monitoring_history_configure.sh --enable
This will update the configuration file gv.cfg.
To change configuration of Monitoring History Database:
•
Change the Monitoring History Database, run: monitoring_history_configure.sh --conf [--num-of-days-to-keep <days>] [-table-size <hours>] [--mode <RW|RO>] [--save-interval <seconds>] [--db-ip
<IP>] [--db-port <port>] [--db-user <user>] [--db-password <password>] [-
-db-name <name>]
• for UFM SA run: /etc/init.d/ufmd model_restart
• for UFM HA run: /etc/init.d/ufmd hamode health_restart
Parameter
--num-of-days-to-keep
--table-size
--mode
Description
The number of days the history will be stored, configurable. The number should not be greater than 3650
(10 years).
The Monitoring History data stored separated by tables. – table-size defines data size in hours stored in one table.
The permissions read/write MH data.
Valid Values:
•
RW - read-write mode, meaning UFM is writing and retrieving data from MH Engine.
•
RO - read-only model, meaning UFM is not writing any new data to MH Engine but can read data.
The default value is RW.
Version 5.2
231
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
Parameter
--save-interval
--db-ip
--db-port
--db-user
--db-password
--db-name
Description
Defines the frequency of default monitoring session history snapshots in seconds, must be a multiple of dashboard_interval value.
Monitoring History SQL server ip address.
Monitoring History SQL server port
Monitoring History SQL user name
Monitoring History SQL password
Monitoring History SQL server database name
To clean the Monitoring History Database:
1. Stop the UFM server.
2. Delete all Monitoring History data: monitoring_history_configure.sh –clear
3. Enable the Monitoring History Database: monitoring_history_configure.sh --enable
4. Start UFM.
9.2.6.3.4 Monitoring History Session Display
When you run a monitoring History session, a new session tab is displayed in the Monitor
Screen.
The Monitoring window splits into three panes:
•
The main pane displays tabs with the monitoring sessions
•
The monitored Data pane
•
The monitored objects pane
You can perform tasks on an object by right-clicking the selected monitored object.
Viewing a Monitoring History Session
To view a Monitoring History Session
1. Click View History Session. The Monitoring Wizard opens displaying the Objects for
Monitoring pane.
232
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
2. Move the objects to be monitored to the Selected Devices list.
•
Presenting your statistics in a Graphical View, limits you to up to 40 ports as the display of the statistics beyond that number becomes too crowded and unreadable.
If you do choose more than 40 ports, the following warning will be displayed.
•
Exporting statistics to a CSV file supports up to 500 devices.
3. Click Next. The Attributes for Monitoring pane appears.
NOTE: The grayed out objects represent fabric nodes (switches or servers) which were connected to the fabric and thus exist in the monitoring history database, but are not currently connected to the fabric.
233
Mellanox Technologies Confidential
Version 5.2
_
Monitoring the Fabric
1. Select attributes for monitoring. For graphical display modes, you can select up to a total of 4 attributes.
2. Click Next. The Visual Definitions pane appears.
234
3. Set parameters as described in the table that follows this procedure.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
4. To run the session, click Run. For information about viewing a monitoring session, see
Table 71: Monitoring Visual Definition Parameters
Field
Session Name
Function
Interval
Procedure
Enter the desired name for the monitoring session.
This function cannot be selected as the information is drawn automatically from the previous created session.
Select the desired Interval size to display your graphical monitoring statistics.
If the requested history session comprises too much data, the following warning is displayed:
Version 5.2
Object Type
View Type
This function cannot be selected as the information is drawn automatically from the previous created session.
Select the preferred graphical monitoring representation formats:
Line, Bar, Histogram, (Table and Pie Chart are grayed out as
The representations show the object distribution relative to the measured attribute.
The following is an example of a graphical representation of a monitoring history session:
Figure 56: Monitoring Session - Graphical Representation
Monitoring History Window Toolbar
The following icons appear in the Monitor History window.
Table 72: Monitor Window Icons
Icon
Name
Export
Description
Exports monitoring session information to a CSV file.
235
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
Icon
Name
Attributes
Zoom In
Zoom Out
Fit To View
Description
Enables the user to select the attributes to view.
Zooms in an enables the user a more focused view on the chosen data.
When zooming in, you can drill down the graph by clicking CTRL and drag the graph to the left or to the right with the mouse.
Zooms in an enables the user a more distant view on the chosen data.
Zooms to the original scale.
9.2.6.4 Preventing Disk Overutilization
In case of Local database server configuration, the UFM Health service checks the actual data size in the database. If the actual data size exceeds the defined threshold, then UFM
Health will remove the portion of the oldest data, regardless of the value set in the --numof-days-to-keep
parameter. The behavior is defined in the configuration file
/opt/ufm/files/conf/UFMHealthConfiguration.xml
. Default threshold value is 70%. It is not recommended to increase the threshold because the same partition is used for the Database output temporary files. By default, the oldest 10% of the saved days will be removed. UFM Health service needs to be restarted after the configuration is modified.
<Test Name="MonitoringHistoryPurgeIfFull" NumOfRetriesBeforeGiveup="1"
RetryTimeoutInSeconds="30">
<TestOperation Name="MHFreeDiskTest">
<Parameters>
<Parameter Name="ThresholdInPercents" Value="70"/>
<Parameter Name="DirectoryName" Value="/opt/ufm/history"/>
</Parameters>
</TestOperation>
<CorrectiveOperation Name="MonitoringHistoryPurgeDB2">
<Parameters>
<Parameter Name="PercentsToDelete" Value="10"/>
</Parameters>
</CorrectiveOperation>
<GiveupOperation Name="CleanDisk">
<Parameters>
<Parameter Name="CleanIfNotUsedInLastNumOfHours" Value="0.5"/>
<Parameter Name="DirectoryName" Value="/opt/ufm/history/files/*"/>
</Parameters>
</GiveupOperation>
</Test>
NOTE: The physical disk utilization will not get lower, but the database will re-use the freed space inside the tablespace files. To shrunk the tablespace file, database downtime and maintenance period should be defined.
9.2.6.5 Monitoring History Data Migration
Monitoring History Data Migration is available only if Monitoring History feature was enabled in UFM v4.0. Monitoring History database schema in UFM v4.5 is divided into number of tables according to the tableLength and the historyLength to allow an easy and
236
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2 efficient managing and accessing the data. After upgrading to UFM v4.5, Monitoring History mechanism will be able collect and report the new monitoring data, and the old history data will be accessible once the Monitoring History migration is completed.
Monitoring History migration must run on the primary UFM server. In case of UFM server failover during migration, the migration procedure will continue to run on the same server where it started. Monitoring History migration can run Offline, in parallel with new
Monitoring History data collection. It unloads old monitored data to buffer and loads it back to the new schema tables. This procedure requires partition with free space equal to the size of the migration data. Migration can use Monitoring History partition as buffer or any other partition accessible from the UFM server.
To run Monitoring History migration:
•
Migrate the new Monitoring History database schema.
/opt/ufm/scripts/monitoring_history_4_5_migrate_data.sh --run --mode
<bulk | lazy> [--days <days>] [--buffer <location>]
Table 73: Monitoring History Migration Parameters
Parameter
--mode bulk
--mode lazy
--days <days>
--buffer <location>
Description
Unload old data to migration buffer, drop old data table, load migration data. This option can be used when partition dedicated to Monitoring History does not have enough disk space but exists other partition that can be used as migration buffer.
Unload old data chunk to migration buffer, load migration chunk to new schema, do it until whole data migrated, drop old data table. This option can be used when partition dedicated to Monitoring History has enough disk space.
Number of days required to migration, default maximum available to migration old history data.
Partition, used during migration in purpose temporary store old history data, default Monitoring History partition.
To delete old monitored data:
•
Free the occupied disk space by not migrating old monitored data.
/opt/ufm/scripts/monitoring_history_4_5_migrate_data.sh --delete
9.3 Alerts and Thresholds
UFM keeps track of counters reported by the ports and uses them to report the port state, derive bandwidth and packet count on each port, and trigger events and alarms according to
Event Management settings (in the Config tab).
An event that occurs in the system is sent to the event table or as a log to a log file as defined in the Event Management where policy is set per event. Logs are events that are saved within the system and sent to the log file and reported to the UFM GUI.
An event can be associated with an alarm. Alarms may occur when events exceed a predefined threshold. Alarms are enabled by default in the Event Management and can be changed as required. The UFM shows the alarmed device, according to the alarm type.
237
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
UFM provides fabric alerts on the following:
•
Communication error
•
Fabric configuration & notification
•
Hardware
The following table details the possible alarm states and their representation in UFM. In the map and table view, an alarmed device shows the state of the alarm: Info (Normal),
Warning, Minor, Critical.
Table 74: Alarm States
Icon Name Description
Info (normal) Information/Notification of normal operating state or a normal system event.
Warning Warning reflects a low priority problem in the fabric but does not prevent its operation. A warning is asserted when an event exceeds a pre-defined threshold.
Minor
Critical
Minor reflects a problem in the fabric but does not stop the operation of the fabric.
Critical means that the system or a system component fabric operation has failed.
You can define event severity and filter events via the GUI. For information on configuring
events and alarms in the Policy Table, see Configuring Event Management
9.4
9.5
Causes of Alarms
Possible causes of an alarm on logical servers:
•
One or more of the physical servers are down or disconnected (Logical Server lacks resources)
•
There is an alarm on one or more physical servers
Possible causes of alarm on port or device:
•
One or more of the physical errors on the port
Events Table
Events information enables faster location of the fault, and faster stabilization of the fabric.
The Events Table is displayed at the bottom of the main pane, and provides information on the events in the system that are defined in the Event Management. The Events Table is available in all the UFM main views, and is updated or refreshed every 30 seconds.
Events are set according to severity as defined in the Event Management. For information
on configuring events and alarms in the Policy Table, see Configuring Event Management
See Supported Port Counters and Events for the comprehensive list of events, descriptions,
and thresholds.
238
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
To view the Events Table
•
In the Event tab, click at the lower right hand side of the window.
The Events table is displayed, showing the list of events with the Event ID, Severity,
Source, Date/Time, Category, and Event Description.
Version 5.2
The parameters listed in the Events table are as follows:
Table 75: Events Table Parameters
Parameter
Event ID
Severity
Source
Date/Time
Category
Description
Displays a system-generated ID that identifies the event. See
Events for a list of the events
reported in the system.
Displays the severity of the event/alarm.
Displays the source of the event, pointing to the Grid, Node/Host
Name, the Board, or the Port.
Values
Severity Types: Normal,
Warning, Minor, Critical
Values: object which can be a physical or logical element
The name of the object types which may be any of the following: Grid, Node, Port,
Logical Server,
Environment, and Site.
Displays the date and time of the event/alarm occurrence.
Displays the event category.
Categories are defined as follows:
•
Unknown (the server may not have been able to relate the event to any of the existing categories)
•
Hardware
•
Traffic
•
Traffic Configuration
•
Traffic Notification
•
Fabric Topology
•
Fabric Configuration
•
Fabric Notification
•
Communication Error
•
Module Status
Default date format example:
Mar 18 14:42:02
See Supported Port Counters and Events for the
comprehensive list of events, descriptions, and thresholds.
239
Mellanox Technologies Confidential
Version 5.2
9.5.1
9.5.2
Monitoring the Fabric
Parameter
Description
Description
•
Environment Notification
•
UFM Server
Describes the event
Values
Rearranging the Events Table
You can drag and drop the Events table by positioning the cursor on the blue header (at the top of the Events Table) and releasing at the desired location.
You can rearrange columns positions within the Events table by dragging and dropping a selected column in the new desired position.
Filtering Events
To filter events
1. Click to show the Event Filter.
9.5.3
2. From the Event drop-down menu, select the desired filter: Event ID, Severity, Source,
Date/Time, Category, and Event Description.
3. In the blank field, enter the search parameter. The Event table automatically filters according to your entry.
For example, to view all the events with the ID 352, enter 352 in the field. To filter according to date, enter the desired date in the blank field.
4. Click Clear to clear your search.
Viewing Additional Event Data
To find out more specific details pertaining to a specific event
1. In the Event table, right-click the event listing.
A menu appears providing the following options:
Icon Option
Properties
Description
Shows the details of the Event, including Event ID, Severity,
Source, Date/Time, Category, and Event Description.
240
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Icon Option
Next
Previous
Refresh
Clear
Go to Map
Go to Node
Description
To view the properties of the next event
To view the properties of the previous event
To refresh the data
Clears the specific event
When an event is related to a specific device (when the source shows the device GUID), goes to the relevant map under the View Window, and selects the related node.
Note: Only available for devices and ports.
When an event is related to a specific device (when the source shows the device GUID), goes to the devices table in the Manage Devices window and shows the node information.
This information allows you to explore the cause of the event.
Note: Only available for devices and ports.
Version 5.2
9.5.4 Saving Events to a CSV File
The UFM allows you to save the events as local CSV (Excel) files which are used for:
•
Importing to Excel and other systems for presentation
•
Perform advanced analysis based on data in the files
•
Send them to support for troubleshooting
To save the events to a CSV file
1. Click to save the port counters to a CSV file. A message prompts you to save the file at the desired location.
2. In the File Name field, enter a name for your CSV file.
3. On your local computer, browse to a location and click Save. Your CSV file is saved.
To view CSV file information
•
Go to the location where you saved your file and double-click your CSV file.
Mellanox Technologies Confidential
241
Version 5.2 Monitoring the Fabric
The CSV file opens showing Event ID, Severity, Source, Date/Time, Category, and
Event Description. For more information about Events Table parameters, see Events
9.6
242
More on Device and System Events
This section provides an overview of all generated events.
UFM provides vast event capabilities, including logs and alerts. In addition, it provides the capability to aggregate events in UFM server, to store them in the model and to display them for client polling by means of Web Services.
The following categories of events are organized according to event source:
•
Subnet Manager events – events issued by the SM
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
9.6.1
9.6.2
9.6.3
Version 5.2
•
Monitoring events – events issued by Performance Monitor
•
Device events – events issued by Network devices
•
UFM Server events – events issued by UFM server
The Events table in Supported Port Counters and Events provides the event summary.
Subnet Manager Events
Subnet Manager events reflect configuration changes and errors on the physical fabric level and are detected and issued by the Subnet Manager.
Subnet Manager events are marked as "SM Core" or "UFM Plug-in".
The following events are generated by the Subnet Manager.
•
InfiniBand Traps – native InfiniBand traps defined in the InfiniBand specification
The InfiniBand traps are sent "as is" by SM Core to UFM plug-in. The UFM Plug-in translates the InfiniBand traps into UFM server event format and forwards the traps to the
UFM server via the SOAP trap channel.
•
Error Events – various errors detected by SM are reflected in the SM Log mechanism
The events are sent by SM Core to UFM plug-in with:
•
Event ID
•
Event Message Text
•
Event Severity
The plug-in reformats and forwards the events to the UFM server.
Monitoring Events
The Monitoring Manager sends performance and error monitoring-related events to the
UFM server.
Monitoring events are categorized as follows:
•
Traffic Error Thresholds -threshold exceeded
•
Congestion notification
•
Complex condition occurrence defined as a function of raw monitored attributes
Monitoring events are described in Supported Port Counters and Events
Device Events
The Central Device Management includes a vast number of events initiated by different types of devices: switches, hosts and gateways.
•
The Mellanox Device Management switch software (VDM) and 10G gateway software generate a number of device-related events that are called “legacy” events. See “Switch”
and “Gateway” events in Supported Port Counters and Events
•
The UFM server is defined as a permanent SNMP manager in all discovered switch devices. The VDM events are sent to the UFM server where they are reformatted, stored
243
Mellanox Technologies Confidential
Version 5.2
9.6.4
Monitoring the Fabric in the model, logged and forwarded to the UFM GUI client (The GUI client periodically polls for events via the regular SOAP channel).
•
The UFM server reformats “legacy” SNMP-based events by creating a monolithic text message from MIB values and appropriate event text.
Device events are listed as VDM or CDM in the "Source" column of the Events table in
Supported Port Counters and Events
UFM Server Events
The UFM Server generates its own events based on changes in any part of the domain the server is managing. The events are reflected in the Object Model. The trigger for these occurrences could be the user’s definition or physical occurrences in the managed fabric.
UFM server events are listed in the Object Model according to Event class. The Event class contains the following attributes:
•
UID – running 64-bit number
•
Type ID – "LS_STATE_CHANGED"
•
Type brief description – "Logical Server state changed"
•
Message Text – "Changed state from allocated to running"
•
Severity: Info/Warning/Error/FatalError
•
Refresh type – special attribute for GUI indication: No/RefreshAll/RefreshElement
•
Object the event relates to – "Env1.LS1"
Each Event is logged into the Event Log File located in /opt/ufm/log directory of the
UFM server. The Log file is saved every day for 5 days and then rotated.
The UFM Server events are marked by UFM Server in the Source column of the Events
counters in the Supported Port Counters and Events Appendix.
The GUI client gets the events by polling periodically the UFM server for new events. This ensures that there are no missed events sent as traps and blocked by the firewall or dropped due to Network issues.
9.7 UFM Server Health Monitoring
The UFM Server Health Monitoring module is a standalone module that monitors UFM resources and processes according to the settings in the
/opt/ufm/files/conf/UFMHealthConfiguration.xml file.
For example:
•
Each monitored resource or process has its own failure condition (number of retries and/or timeout), which you can configure.
•
If a test fails, UFM will perform a corrective operation, if defined for the process, for example, to restart the process. You can change the configured corrective operation. If the corrective operation is set to "None", after the defined number of failures, the give-up operation is performed.
244
Mellanox Technologies Confidential
9.7.1
Unified Fabric Manager for InfiniBand User Manual Version 5.2
•
If a test reaches the configured threshold for the number of retries, the health monitoring initiates the give-up operation defined for the process, for example, UFM failover or stop.
•
By default, events and alarms are sent when a process fails, and they are also recorded in the internal log file.
Each process runs according to its own defined schedule, which you can change in the configuration file.
Changes to the configuration file take effect only after a UFM Server restart. (It is possible to kill and run in background the process nohup python
/opt/ufm/ufmhealth/UfmHealthRunner.pyo &.)
You can also use the configuration file to improve disk space management by configuring:
•
How often to purge MySQL binary log files.
•
When to delete compressed UFM log files (according to free disk space).
The settings in the /opt/ufm/files/conf/UFMHealthConfiguration.xml file are also used to generate the UFM Health Report. For more information about UFM Health reports, see
The following section describes the configuration file options for UFM server monitoring.
UFM Health Configuration
The UFM health configuration file contains three sections:
•
Supported Operations - This section describes all the operations that can be used in tests, and their parameters.
•
Supported Tests - This section describes all the tests. Each test includes:
•
The main test operation.
•
A corrective operation, if the main operation fails.
•
A give-up operation, if the main operation continues to fail after the corrective operation and defined number of retries.
The number of retries and timeout is also configured for each test operation.
•
Test Schedule - This section lists the tests in the order in which they are performed and their configured frequency.
The following table describes the default settings in the
/opt/ufm/files/conf/UFMHealthConfiguration.xml file for each test. The tests are listed in the order in which they are performed in the default configuration file.
You might need to modify the default values depending on the size of your fabric.
For example, in a large fabric, the SM might not be responsive for sminfo for a long time; therefore, it is recommended to increase the values for timeout and number of retries for
SMResponseTest.
Recommended configurations for SMResponseTest are:
•
For a fabric with 5000 nodes:
•
Number of retries = 12
245
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
•
Frequency = 10
•
For a fabric with 10000 nodes:
•
Number of retries = 12
•
Frequency = 20
Test Name /
Description
CpuUsageTest
Checks total CPU utilization.
Test Operation
CPUTest
Tests that overall
CPU usage does not exceed 80%
(this percentage is configurable).
Corrective
Operation
(if Test Operation fails)
None
If UFM Event
Burst Management is enabled, it is automatically initiated when the test operation fails
AvailableDiskSpace
Test
Checks available disk space.
CheckIBFabricInte
rface
Checks state of active fabric interface.
CheckIBFabricInte
rfaceStandby
(HA only) Checks state of fabric interface on standby.
MemoryTest
Checks total memory usage.
FreeDiskTest
Tests that disk space usage for
/opt/ufm does not exceed 90% (this percentage is configurable).
CleanDisk
Delete compressed
UFM log files under /opt/ufm.
IBInterfaceTest
Tests that active fabric interface is up.
BringUpIBFabric
Interface
Bring up the fabric interface.
IBInterfaceTest
OnStandby
Tests that fabric interface on standby is up.
None
None
SMProcessTest
Checks status of the
OpenSM service.
MemoryUsageT
est
Tests that memory usage does not exceed
90% (this percentage is configurable).
SMRunningTes
t
Tests that the
SM process is running.
RestartProcess
Restart the SM process
No. Retries /
Give-up
Operation
1 Retry
None
3 Retries
None
3 Retries
SMOrUFMFail overOrDoNothi
ng
1 Retry
None
1 Retry
None
1 Retry
UFMFailoverO
rDoNothing
Test
Frequency
1 minute
1 hour
35 seconds
1 minute
1 minute
10 seconds
246
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Test Name /
Description
Test Operation Corrective
Operation
(if Test Operation fails)
None SMResponseTest
Checks responsiveness of
SM (when SM process is running).
IbpmTest
Checks status of the
IBPM (Performance
Manager) service.
ModelMainTest
Checks status of the main UFM service
SMTest
Tests SM responsiveness by sending the sminfo query to
SM.
ProcessIsRunni
ngTest
Tests that the
IBPM service is running.
ProcessIsRunni
ngTest
Tests that the
UFM service is running.
RestartProcess
Restart the IBPM service
RestartProcess
Restart the UFM service
HttpdTest
Checks status of the httpd service.
MySqlTest
Checks status of the
MySql service.
CleanMySql
Purges MySql Logs
UFMServerVersion
Test
Checks UFM software version and build.
UFMServerLicense
Test
Checks UFM License information.
UFMServerHAConf
igurationTest
(HA only) Checks the configuration on master and standby.
ProcessIsRunni
ngTest
Tests that the httpd service is running.
ConnectToMyS
ql
Tests that the
MySql service is running.
AlwaysFailTest
Fails the test in order to perform the corrective action.
UfmVersionTes
t
Returns UFM software version information.
RestartProcess
Restart the httpd service
None
PurgeMySqlLogs
Purge all MySql
Logs on each test.
None
UfmLicenseTest
Returns UFM
License information.
None
UfmHAConfigu
rationTest
Returns information about the master
None
No. Retries /
Give-up
Operation
9 Retries
UFMFailoverO
rDoNothing
3 Retries
None
3 Retries
UFMFailoverO
rDoNothing
3 Retries
None
1 Retry
UFMFailoverO
rDoNothing
1 Retry
None
1 Retry
None
1 Retry
None
1 Retry
None
Test
Frequency
10 seconds
1 minute
20 seconds
20 seconds
20 seconds
24 hours
24 hours
24 hours
24 hours
Version 5.2
247
Mellanox Technologies Confidential
Version 5.2 Monitoring the Fabric
Test
Frequency
Test Name /
Description
Test Operation Corrective
Operation
(if Test Operation fails)
UFMMemoryTest
Checks available
UFM memory.
UFMCpuUsageTest
Checks UFM CPU utilization.
CheckDrbdTcpCon nectionPerformance
Test (HA only)
Checks the tcp connection between master and standby and standby
UFM servers.
UfmMemoryUs
ageTest
Tests that UFM memory usage does not exceed
80% (this percentage is configurable).
CPUTest
Tests that UFM
CPU usage does not exceed 60%
(this percentage is configurable).
TcpConnection
PerformanceTe
st
Tests that bandwidth is greater than 100
Mb/sec and latency is less than 70 usec
(configurable).
None
None
None
No. Retries /
Give-up
Operation
1 Retry
None
1 Retry
None
2 Retry
None
1 minute
1 minute
10 minute
NOTE: The Supported Operations section of the configuration file includes additional optional operations that can be used as corrective operations or give-up operations.
9.7.1.1 UFM Core Files Tracking
To receive a notification every time OpenSM or ibpm creates a core dump, please refer to the list of all current core dumps of OpenSM and ibpm in the UFM health report.
To receive core dump notifications:
1. Create the directory that will store the core dumps.
/tmp/cores
2. Set the naming convention for core dump file. The name must include the directory created in the step above.
The convention we recommend is: echo "/tmp/cores/%t.core.%e.%p.%h" > /proc/sys/kernel/core_pattern
248
Mellanox Technologies Confidential
9.7.2
9.7.3
Unified Fabric Manager for InfiniBand User Manual Version 5.2
3. Make sure core dumps directory setting is persistent between reboots. Add to the
/etc/sysctl.confkernel.core_pattern
file the following:
/tmp/cores/%t.core.%e.%p.%h
4. Configure the core file size to be unlimited. ulimit -c unlimited
5. (Only on UFM HA master) Update the UFM configuration file gv.cfg to enable core dump tracking. track_core_dumps = yes
Example of Health Configuration
The default configuration for the overall memory test in the
opt/ufm/files/conf/UFMHealthConfiguration.xml file is:
<Test Name="MemoryTest" NumOfRetriesBeforeGiveup="3"
RetryTimeoutInSeconds="10">
<TestOperation Name="MemoryUsageTest">
<Parameters>
<Parameter Name="ThresholdInPercents" Value="90"/>
</Parameters>
</TestOperation>
<CorrectiveOperation Name="None"/>
<GiveupOperation Name="None"/>
</Test>
This configuration tests the available memory. If memory usage exceeds 90%, the test is repeated up to 3 times at 10 second intervals, or until memory usage drops to below 90%. No corrective action is taken and no action is taken after 3 retries.
To test with a usage threshold of 80%, and to initiate UFM failover or stop UFM after three retries, change the configuration to:
<Test Name="MemoryTest" NumOfRetriesBeforeGiveup="3"
RetryTimeoutInSeconds="10">
<TestOperation Name="MemoryUsageTest">
<Parameters>
<Parameter Name="ThresholdInPercents" Value="80"/>
</Parameters>
</TestOperation>
<CorrectiveOperation Name="None"/>
<GiveupOperation Name="UFMFailoverOrStop"/>
</Test>
Event Burst Management
UFM event burst management can lower the overall CPU usage following an event burst by suppressing events. Event burst management is configured in the gv.cfg configuration file.
When the overall CPU usage exceeds the threshold configured by the CpuUsageTest in the
/opt/ufm/files/conf/UFMHealthConfiguration.xml file, a High CPU Utilization event occurs.
This event initiates the UFM event burst management, which:
•
Suppresses events. The default level of suppression enables critical events only.
•
If, after a specified period of time (30 seconds, by default), no further High CPU
Utilization event occurs, the UFM server enables all events.
To modify Event burst management configuration, change the following parameters in the gv. cfg file:
249
Mellanox Technologies Confidential
Version 5.2
9.7.4
Monitoring the Fabric
# The events' level in case events are suppressed (the possible levels are disable_all_events, enable_critical_events, and enable_all_events)
# The entire feature can be turned off using the level "enable_all_events" suppress_events_level = enable_critical_events
# The amount of time in seconds which events are suppressed suppress_events_timeout = 30
Recovery from Consecutive Failures
UFM Server Health Monitor might restart or trigger a failover in order to recover from specific failures. In case a re-start or failover fails, UFM Server Health Monitor tries the operation again. Upon a number of consecutive failure attempts to restart or failover, UFM
Server Health Monitor stops trying to restart Model Main and allows OpenSM to run without intervention. The behavior maximum number of consecutive restart attempts is defined in the configuration file /opt/ufm/files/conf/UFMHealthConfiguration.xml:
<Parameter Name="RestartAttempts" Value="8"/>
<Failover MaxAllowedAttempts="6"/>
9.8 InfiniBand Performance Manager
The InfiniBand Performance Manager (IBPM) is responsible for reporting performance information to UFM and upper layer applications. IBPM performs InfiniBand counter collection using Management Datagrams (MADs) and reports the collected data to the UFM server (default port 8005).
•
When UFM is connected to the fabric by multiple links, this process is offloaded to the non-management port of the UFM server.
In this setup, the management network should be configured in full membership mode, by setting default_membership = full in gv.cfg. Do not change the management network configuration in the UFM GUI.
•
When UFM is connected to the fabric by a single port, IBPM uses the same port as
OpenSM. It is recommended to connect UFM to the fabric by two links to avoid collisions.
Failure or disconnect of the the non-management port (ib1) on the primary UFM server does not cause a UFM failover. By default, the UFM Health Monitoring process is configured to
try to restart the IBPM. For more information, see UFM Health Configuration
250
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
10 Managing UFM and Fabric Health
10.1 Overview of the Health Window
The Health window provides a central place for fabric information, such as, inventory summary, detecting non-optimal links, fabric errors, BER issues, SM status, and so on. From the Health window you can run and view reports and logs to monitor and analyze UFM and fabric health. These reports contain measurable metrics, and are available in HTML format output for managerial reports.
Figure 57: Health Window
Version 5.2
To access the Health window
•
Click the Health tab .
10.2 Managing Health Reports
UFM Health reports run a series of checks on the fabric or on the UFM server. When you generate a UFM health report or fabric health report, it is automatically displayed through the UFM GUI. You can also display the last-generated report in HTML format in a browser window.
When you display a report in the UFM GUI, you can:
•
Print the displayed report.
•
Save the entire report or sections of the report to a file.
Each check that is run for a report triggers a corresponding fabric health event or UFM server event. Events are also triggered when a check starts and ends. You can configure when an event triggers an alarm in the Event Management window. For more information,
see Configuring Event Management
251
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
10.2.1 Running a Fabric Health Report
You can configure which fabric checks to run and include in the fabric health report.
To run a fabric health report:
1. Select the Health tab.
2. In the Fabric Health area click Run Now. The Fabric Health Report dialog box appears.
252
3. Set report configuration parameters and click OK. UFM generates and displays the report.
For more information about report checks and parameters, see the following table.
Table 76: Fabric Health Report Checks
Check
Duplicate /Zero LID check
Duplicated Node
Description
SM Check
Description
Lists all ports with same LID or zero LID value.
Lists all nodes with same node description.
Does not include switches with the same description.
Checks that:
•
There is one and only one active (master)
Subnet Manager in the fabric.
To run, select:
LIDs Check
Default: Selected
Duplicated Node
Description
Default: Selected
SM Configuration
Check
Default: Selected
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Check
Bad Links Check
Link width
Link speed
Firmware Check
Eye open check
Cable information
UFM Alarms
Description
•
The master is selected according to highest priority and lowest port GUID.
The report lists all SMs in the fabric with their attributes.
Performs a full-fabric discovery and reports
“non-responsive” ports with their path.
To run, select:
Checks if link width is optimally used.
•
When a width is selected, the report lists the active links that do not meet the optimum for the selection.
•
When no width is selected (All), the test checks whether the enabled width on both sides of the link equals the configured maximum (confirms that auto-negotiation was successful).
Checks if link speed is optimally used.
•
When a speed is selected, the report lists the active links that do not meet the optimum for the selection.
•
When no speed is selected (All), the test checks whether the enabled speed on both sides of the link equals the configured maximum (confirms that auto-negotiation was successful).
Checks for firmware inconsistencies. For each device model in the fabric, the test finds the latest installed version of the firmware and reports devices with older versions.
(For QDR only) Lists Eye-Opener information for each link.
When minimum and maximum port bounds are specified, the report lists the links with eye size outside of the specified bounds.
Reports cable information as stored in
EPROM on each port: cable vendor, type, length and serial number.
Lists all open alarms in UFM.
Non-optimal Links
Check
Default: Selected
None-optimal Speed
And Width
Default: Selected
Link Width: The default is ALL.
None-optimal Speed
And Width
Default: Selected
Link Speed: The default is ALL.
Firmware Version
Check
Default: Selected
Eye Open Check
Default: Selected
Minimum and
Maximum port
bound: By default no bounds are defined.
Cable type check &
cable diagnostics
Default: NOT selected because this test might take a long time to complete (40 msec per port)
UFM Alarms
Default: Selected
Version 5.2
253
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
10.2.2 Viewing Fabric Health Reports
When you generate a report, it is automatically displayed in the UFM GUI. At any point in your UFM session, you can view the last generated fabric health report in the UFM GUI or in HTML format.
The report displays:
•
A report summary table of the errors and warnings generated by the report.
•
A fabric summary of the devices and ports in the fabric.
•
Details of the results of each check run by the report.
To view the last Fabric Health report
1. Select the Health tab.
2. In the Fabric Health area:
•
To view the report through the UFM GUI, click Last Report.
•
To view the report in HTML format in a browser, click HTML.
3. In the GUI report display, expand each section of the report to see the report details.
For detailed information about each fabric health check, see Running a Fabric Health Report
The following shows an example of a fabric health report summary.
Figure 58: Fabric Health Report Summary
10.2.3 Running a UFM Health Report
To run a UFM health report:
1. Select the Health tab.
2. In the UFM Health area click Run Now. UFM generates and displays the report.
254
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
For detailed information about each UFM health check, see Viewing UFM Health Reports
The UFM health checks that are performed are configured in the UFM health configuration file. For more information about UFM health check configuration, see UFM Server
Monitoring.
10.2.4 Viewing UFM Health Reports
When you generate a report, it is automatically displayed in the UFM GUI. At any point in your UFM session, you can view the last generated UFM health report in the UFM GUI or in
HTML format.
To view the last UFM Health report
1. Select the Health tab.
2. In the UFM Health area:
•
To view the report through the UFM GUI, click Last Report.
•
To view the report in HTML format in a browser, click HTML.
3. In the GUI report display, expand each section of the report to see the report details.
The following shows an example of a UFM health report summary.
Figure 59: UFM Health Report
Version 5.2
The following tables describe the checks included in the report.
Table 77: UFM Health Report Checks
UFM Configuration
Check
Release Number
Description
UFM software version and build.
License Type Type of license, permanent or evaluation.
License Customer Number The customer number provided by Mellanox.
Mellanox Technologies Confidential
255
Version 5.2
License UID
License Expiration Date
License Functionality
License Devices Limit
Running Mode
Managing UFM and Fabric Health
The UFM serial number provided by Mellanox.
License expiration date for limited licenses.
Level of functionality enabled for the end-user, standard or advanced.
The maximum number of devices that UFM is licensed to manage.
UFM running mode, Standalone or High Availability (HA).
When UFM is in HA mode, additional information is displayed for the master and standby servers.
256
UFM Processing
Check
OpenSM ibpm
ModelMain httpd
MySql
Memory Monitoring
Check
Total memory usage
UFM memory usage
CPU Monitoring
Check
Total CPU Capacity
CPUs Number
Total CPU utilization
UFM CPU utilization
Disk Monitoring
Check
Disk <diskname>
Fabric Interface
Check
Description
Status of the OpenSM service.
Status of the ibpm (Performance Manager) service.
Status of the main UFM service.
Status of the httpd service.
Status of the MySql service.
Description
Percentage of total memory usage.
Percentage of UFM memory usage
Description
Percentage of CPU capacity available
Number of CPUs
Percentage of total CPU utilization.
Percentage of UFM CPU utilization.
Description
Percentage of disk usage.
Description
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Fabric Interface Name and state of fabric interface.
10.3 Fabric Topology Comparison
You can check that the InfiniBand fabric discovered by UFM matches your planned fabric setup.
UFM compares the current fabric topology with the topology in your topology setup file and reports any differences between them.
Use a TOPO file to define the fabric topology. The topology (TOPO) file describes the IB connectivity and systems included in the network. For more details, see the Linux man page for ibdm-topo-file.
At any time during your UFM session, you can view the last generated report through the
UFM GUI or in HTML format in a browser window.
To perform topology comparison
1. Select the Health tab.
2. In the Topology Compare area click Run Now. The LoadTopology File dialog box appears.
3. Select the required topology setup file and click OK.
UFM compares topologies and displays the results.
To view the last Topology Comparison report
1. Select the Health tab.
2. In the Topology Compare area:
•
To view the report through the UFM GUI, click Last Report.
•
To view the report in HTML format in a browser, click HTML.
The following shows an example of a topology comparison report summary.
Figure 60: Topology Comparison Report
Version 5.2
10.4 UFM Database and Configuration Snapshot
You can export and save UFM database information and configuration files in a predefined location. In this way you can create a full snapshot before upgrading.
257
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
By default, the snapshot includes the database with logical model information, and the UFM configuration files. You can also save troubleshooting information, so that you can send all information required for debugging to Mellanox support. The additional troubleshooting information includes system snapshot files and UFM log files.
To perform a snapshot
1. Select the Health tab.
2. In the UFM Snapshot area: a. Select the Include Troubleshooting Information check box to save information for debugging. b. Click Run Now.
UFM creates the snapshot and saves the data to the predefined location.
10.4.1 Changing Snapshot File Location
By default, the snapshot files are stored under /opt/ufm/backup. You can change the location of the snapshot files in the gv.cfg configuration file in the backup folder location section.
For example:
#backup folder location backup_folder=/opt/ufm/backup
10.5 UFM Logging
UFM logging records Fabric events and actions that can serve to identify Fabric issues and assist in troubleshooting. The logs record the following classes of events or activities:
•
Fabric events - Activity, errors and threshold based events, for physical devices and
Logical objects
•
Subnet Manager (SM) events - Activities and notifications generated by the Subnet
Manager
•
UFM events and actions - The UFM Server component log
10.5.1 Viewing Logs
To view Logs
1. Click the Health tab.
2. In the UFM Logs area, click Show Logs.
3. Select a log on the left panel. The Logs window displays the log according to your selection. An example of each log type is shown in the following sections.
10.5.1.1 Logs Window
The Logs window contains two areas:
•
The left log list
•
The main result panel
By selecting a specific log, the log data is displayed in the main area.
258
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
In the Logs window, you can:
•
Export the data into a local text file.
•
Refresh the data.
•
Search for a specific value.
•
Limit the display to a specific number of lines.
10.5.2 Event Logs
Event Logs show history of fabric events detected and initiated by the UFM server. The timestamp and severity of an event is indicated as well as the cause of the event and additional relevant information.
The Event log is kept on the UFM server in the /opt/ufm/log/events.log file.
See Supported Port Counters and Events for a comprehensive list of Events.
10.5.2.1 Example of an Event Log
The following is an example printout of an Event log:
2009-05-23 23:45:59 Port(0008f1040041222e.0008f1040041222e_17) :
PortRcvConstraintErrors counter threshold was exceeded. Threshold is 5, received value is 14. (Minor)
2009-05-23 23:45:59 Port(0008f10400405366.0008f104003f66d3_19) :
PortRcvConstraintErrors counter threshold was exceeded. Threshold is 5, received value is 7. (Minor)
2009-05-23 23:47:54 Site(Grid.default) :Mcast group deleted: ff12601b80010000, 00000016 (Warning)
2009-05-23 23:47:54 Site(Grid.default) :Mcast group deleted: ff12601b80010000, 00000002 (Warning)
2009-05-24 09:08:07 Switch(default.0008f1040041222e) :
Node action port 0008f1040041222e_17 action disable success (Info)
2009-05-24 09:08:07 Switch(default.0008f1040041222e) :
2009-05-24 09:08:07 Site(Grid.default) :GID Address Out Service:prefix fe80000000000000,guid 0002c9030002c44d (Warning)
2009-05-24 09:08:23 Site(Grid.default) :Site configuration changes-
0002c9030002c44b node is down (Warning)
2009-05-24 09:10:28 Switch(default.0008f1040041222e) :Node action port
0008f1040041222e_17 action enable success (Info)
2009-05-24 09:10:31 Site(Grid.default) :GID Address In Service:prefix fe80000000000000,guid 0002c9030002c44d (Info)
The following figure is the Event Log window in UFM.
Figure 61: Event Log
Version 5.2
259
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
10.5.3 SM Log
The SM Log shows messages of the Subnet Manager and communication plug-in.
The log verbosity is defined by selecting the Log Levels in the SM Logging configuration
windows. For more information, see Configuring the Subnet Manager
The following figure shows the SM Log window in UFM.
Figure 62: SM Log
10.5.4 UFM Logs
UFM Log is a general log of UFM Server. The log saves a history of user actions, events, polling results and other server activity.
Log verbosity is defined on start-up in the configuration file /opt/ufm/conf/gv.cfg:
[Logging]
# optional logging levels
#CRITICAL, ERROR, WARNING, INFO, DEBUG level = INFO
The default verbosity level is INFO.
The following figure shows the UFM Log window, as it is displayed in UFM.
260
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Figure 63: UFM Log
Version 5.2
10.5.5 Searching Logs
You can search the log file to instantly find specific information.
To search logs
1. Enter the desired string in the Find field. You may select a specific string within the log.
Copy and paste the string into the Find field (Ctrl-C + Ctrl-V).
•
Throughout the log file, text strings matching the search string are highlighted in yellow.
•
The current cursor position is highlighted in green.
•
The actual matches for your search are displayed in the field right next to the search field.
2. Enable the Match case box to refine your search.
3. Use the Previous ( ) or Next ( ) arrows to find the previous or next occurrence of the string in the text.
4. Click the Wrap Search box so that the search goes back to the beginning after it has reached the end of the search window.
5. At the bottom of the left panel, in the Lines field, enter the number of lines to be displayed in the Log file. The total amount can be up to 10000 lines.
The following window shows a search example.
261
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
Figure 64: Log Search – Example
10.5.6 Fabric Collector Logs
Fabric Collector provides the following log files:
•
/var/log/fb_collector.log - on the host
•
/opt/ufm/files/log/fb_collector_manager.log - on the UFM Server
Log verbosity is defined in the configuration file /opt/ufm/conf/gv.cfg on the UFM
Server:
[FBCollectorManager]
# optional logging levels
#CRITICAL, ERROR, WARNING, INFO, DEBUG log_level = WARNING
The default verbosity level is WARNING.
Log verbosity is defined in the configuration file
/etc/fabric_collector/fb_collector.conf on the Host:
[client]
# Optional logging levels: CRITICAL, ERROR, WARNING, INFO, DEBUG log_level = WARNING
The default verbosity level is WARNING.
10.5.7 Syslog
The Syslog feature allows the user to redirect each of the UFM log files into one centralized remote logging server.
To configure this feature within UFM, edit the gv.cfg file:
[Logging]
# Optional logging levels: CRITICAL, ERROR, WARNING, INFO, DEBUG level = WARNING sm_level = WARNING webserv_level = INFO resource_manager_level = WARNING
262
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual event_log_level = INFO
# addition option to print content of packet received from ibpm (possible values: yes/no, default=no) dump_ibpm_packet = no
#syslog configuration
#local syslog: /dev/log
#remote syslog: host:port syslog_addr=/dev/log
#main ufm log syslog=false sm_syslog=false event_syslog=false
[UnhealthyPorts] log_level = INFO syslog=false
[DailyReport] log_level = INFO syslog=false
To configure the syslog address/dev/log for a local machine, host:port or a remote machine:
NOTE: In case of a remote machine configuration, the host should be configured accordingly.
Version 5.2
1. configure loggers to work with syslog (true/false) a. Logging.syslog: main ufm logger b. Logging. sm_syslog: sm client logger c. Logging.event_syslog: events logger d. UnhealthyPorts.syslog: unhealthy ports logger e. DailyReport.syslog: daily reports logger
To read from syslog:
For a local machine: /var/log/messages
For a remote machine: use syslog UI if applicable, or ssh to the remote machine and fetch
/var/log/messages
10.6 Daily Report
The Daily Report feature collects, analyzes, and reports the most significant issues of the fabric in the last 24 hours (from 00:00 to 24:00). The reports present statistical information such as Summary of Traffic, Congestions and UFM events that occurred during the last 24 hours. These statistics are sent to a pre-defined recipients list on a daily basis (see configuration settings). It is also possible to specify a non 24 hour range, by changing UFM
configuration file - see section 10.6.4.510.6.4.5
The following are the formats of the Daily Report:
•
Interactive - Opened via the browser. The charts are displayed in SVG format. This format can be accessed from UFM’s GUI and is also sent by mail as attachment.
263
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
•
Static – Opened via mail client (Outlook, Gmail, Hotmail, etc). The charts are displayed in PNG format.
10.6.1 The Report Content
10.6.1.1 Sidebar
The Sidebar includes general information regarding the fabric, such as: the site name, number of switches and hosts in the fabric, and the dates on which the report was generated.
Navigation between the charts can be done via the menu charts on the sidebar.
10.6.1.2 Daily Report Highlights
The top of the report shows highlight activities of the network, such as: the host with the most events, the most congested host and switch, and top sender host. To see the related chart of each highlight, click the corresponding icon in the "Link to chart" column.
264
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
10.6.1.3 Available Charts
10.6.1.3.1 Events by Severity
Events by Severity displays in a graphical view the distribution of all the UFM events that occurred during each hour. Events are separated into the following severity levels: Critical,
Minor, and Warning.
Version 5.2
NOTE: Hovering over the bars in the interactive report displays the amount of events per hour.
265
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
10.6.1.3.2 Normalized Traffic and Congestion
Normalized Traffic and Congestion displays in a graphical view the normalized traffic and congestions of the fabric. This graph displays the accumulated data for the Senders in the fabric (not including switches).
Congestion normalization is based on the number of delayed packets (packets that wait in the queue) and bandwidth loss.
The graph displays the percentage of the traffic utilization in green and the percentage of the congestion in red.
NOTE: Hovering over the bars in the interactive report displays the percentage of the traffic/congestion per hour.
10.6.1.3.3 Hosts Utilization Distribution
Hosts Utilization Distribution displays in a graphical view the groups of hosts, where each host belongs to a specific group according to its utilization status.
To see the hosts in each group, click on the pie chart (at the interactive report).
The utilization groups are:
•
Very low - up to 20% utilized
•
Low - 20-40% utilized
•
Moderate - 40-60% utilized
•
High - 60-80% utilized
•
Very high – 80-100% utilized
266
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
NOTE: Hovering over the slices in the interactive report displays the percentage of hosts in this group.
10.6.1.3.4 Most Active Events
Most Active Events displays in a graphical view the most active events, ordered by the number of occurrences during the last 24 hours.
Version 5.2
NOTE: Hovering over the bars in the interactive report displays the number of occurrences for each active event, and hovering on each event’s name displays a tooltip with the event’s description.
267
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
10.6.1.3.5 Top Senders
Top Senders displays in a graphical view the normalized traffic and congestions of the top sender hosts. Congestion normalization is based on the number of the delayed packets
(packets that wait in queue) and bandwidth loss.
The graph displays the percentage of the traffic utilization in green and the percentage of the congestion in red.
NOTE: Hovering over the bars in the interactive report displays the percentage of the traffic/congestion for a selected host.
268
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
10.6.1.3.6 Hosts with Most Events
Hosts with Most Events displays in a graphical view the hosts with the most events. Events are separated into the following severity levels: Critical, Minor, and Warning.
Version 5.2
NOTE: Hovering over the bars in the interactive report displays the amount of events per severity for a selected host.
Mellanox Technologies Confidential
269
Version 5.2 Managing UFM and Fabric Health
10.6.1.3.7 Hosts with Most Critical Events
Hosts with Most Critical Events displays in a graphical view the hosts with the most critical events.
NOTE: Hovering over the bars in the interactive report displays the amount of critical events for a selected host.
270
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
10.6.1.3.8 Most Congested Hosts
Most Congested Hosts displays in a graphical view the normalized congestions of the most congested hosts. Congestion normalization is based on the number of the delayed packets
(packets that wait in queue) and bandwidth loss.
Version 5.2
NOTE: Hovering over the bars in the interactive report displays the percentage of the congestion for a selected host.
Mellanox Technologies Confidential
271
Version 5.2 Managing UFM and Fabric Health
10.6.1.3.9 Hosts with Most Link Down Events
Hosts with Most Link Down Events displays in a graphical view the list of the hosts with the most link down events during the last 24 hours.
NOTE: Hovering over the bars in the interactive report displays the amount of linkdown events for a selected host.
272
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
10.6.1.3.10 Switches with Most Events
Switches with Most Events displays in a graphical view the switches with the most events.
Events are separated into the following severity levels: Critical, Minor, and Warning.
Version 5.2
NOTE: Hovering over the bars in the interactive report displays the amount of events per severity for a selected switch.
Mellanox Technologies Confidential
273
Version 5.2 Managing UFM and Fabric Health
10.6.1.3.11 Switches with Most Critical Events
Switches with Most Critical Events displays in a graphical view the switches with the most critical events.
NOTE: Hovering over the bars in the interactive report displays the amount of critical events for a selected switch.
274
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
10.6.1.3.12 Most Congested Switches
Most Congested Switches displays in a graphical view the normalized congestions of the most congested switches. Congestion normalization is based on the number of delayed packets (packets that wait in queue) and bandwidth loss.
Version 5.2
NOTE: Hovering over the bars in the interactive report displays the percentage of the congestion for a selected switch.
Mellanox Technologies Confidential
275
Version 5.2 Managing UFM and Fabric Health
10.6.1.3.13 Switches with Most Link Down Events
Switches with Most Link Down Events displays in a graphical view the list of the switches with the most link down events during the last 24 hours.
NOTE: Hovering over the bars in the interactive report displays the amount of linkdown events for a selected switch.
NOTE: Clicking on the “help” icon in the upper right corner of each chart, in the interactive report , will display a short description of the chart.
Clicking on the “home” icon in the upper right corner of each chart, in the interactive report , will move the display to the beginning of the report.
276
NOTE: On charts: “Events by Severity”, “Hosts with Most Events”, and “Switches with Most Events”, if the maximum value in the Y-axis is less than 5, an “m” unit will appear and stand for “milli”.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
NOTE: For all charts, if the value is higher than 1000 in the Y-axis, a “k” unit will appear and stand for “killo”.
Version 5.2
10.6.2 Daily Report View in the UFM GUI
To view the Interactive report in the UFM GUI:
1. Go to the Health tab.
2. Under the Daily Report section click on the View Reports link. A list of all the saved reports will be displayed.
3. Select a report of your choice and click View. The selected report will be opened in your default browser.
277
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
10.6.3 The email Report Format
All recipients in the recipients list (see section Configuring the Recipients List for the Daily
) will receive a daily email report. The email body includes the highlights and the
charts in a .png format.
The Daily Report includes the following attached files:
•
daily-report.html - A copy of the interactive report that can also be viewed from UFM’s
GUI
•
sender_hosts_bw_utilization.csv – Contains a list of all the host names in the appropriate distribution column (0-20/20-40/40-60/60-80/80-100). For further
information, see section Hosts Utilization Distribution
•
critical_event.log – Contains a list of all the critical events in the previous day. (This file will be added as an attachment only in case critical events exist in the UFM event.log of the last 24 hours.)
•
unhealthy_ports.csv – Contains a list of unhealthy ports in the fabric. (This file will be added as an attachment only in case unhealthy ports were detected by UFM during the last 24 hours.)
•
Fabric_Health.html – A copy of the fabric health report. For further information on
UFM fabric health report please refer to section Running a Fabric Health Report
278
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
10.6.4 Configuring UFM Daily Report
10.6.4.1 Configuring the SMTP Parameters for the Daily Report
To receive the Daily Report by email, you must set the SMTP configuration.
To configure the SMTP server for Daily Report:
1. Go to the Health tab.
Version 5.2
2. Under the Daily Report section click on the Configure SMTP link.
3. Configure the SMTP settings. See section Configuring the SMTP Server .
10.6.4.2 Configuring the Recipients List for the Daily Report
1. Go to the Health tab.
2. Under the Daily Report section click on the Configure Recipients link. A list of all the saved recipients will be displayed.
279
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
3. In the Recipients List window: a. Enter a valid recipient (a valid email address) at the text field. b. Click the green plus sign to add the recipient. c. Click Apply. The chosen recipient will be added to recipients list. d. Click OK
10.6.4.3 Activating and Deactivating the Daily Report
Daily Report is activated via the /opt/ufm/conf/gv.cfg file. The feature is activated by default.
To activate the Daily Report:
1. Open the /opt/ufm/conf/gv.cfg file.
2. Find the DailyReport section.
3. Set the daily_report_enabled option to true. daily_report_enabled = true
To deactivate the Daily Report:
1. Open the /opt/ufm/conf/gv.cfg file.
2. Find the DailyReport section.
3. Set the daily_report_enabled option to false. daily_report_enabled = false
10.6.4.4 Saving Daily Reports
UFM saves the interactive Daily Reports under the
/opt/ufm/files/reports/Daily directory. Each report will be saved under a
280
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual directory with its respective date. For example, report for Sept. 28 th
, 2014 will be located under: /opt/ufm/files/reports/Daily/2014-09-28/.
By default, the maximum number of reports that will be saved is 365 (one per day)
To configure the maximum number of reports to save:
1. Open the /opt/ufm/conf/gv.cfg file.
2. Find the DailyReport section.
3. Set the max_reports option to the desired value. A count of 0 (zero) means no copies are retained. (default and max is 365).
4. Restart UFM.
Version 5.2
Mellanox Technologies Confidential
281
Version 5.2 Managing UFM and Fabric Health
10.6.4.5 Other Daily Report Configurations
All the Daily Report configuration parameters are located in the "DailyReport" section in gv.cfg configuration file.
The following are additional Daily Report configurations options:
•
top_x option specifies the number of results in the "Top X" charts. Max number can be
20. (Default value is 10). top_x value will be applied to all charts existing in the Daily
Report
•
mail_send_interval option specifies the epoch in minutes after midnight that the report can be emailed. By default, if UFM was down during midnight, and was restarted after
1:00, the report of the previous day will be generated and saved, but will not be emailed.
This can be changed by editing the mail_send_interval. (default value is 60 minutes, meaning that the report will be send only between 00:00 to 1:00)
•
log_level option specifies the Daily Report log verbosity. Default value is INFO.
(Optional values: INFO, WARNING and ERROR)
•
attach_fabric_health_report option indicates whether or not to add the fabric health report as attachment to the mail. Default value is true. (Optional values: true or false)
•
fabric_health_report_timeout specifies the max time in seconds, to wait for fabric health report generation. Default value is 900 seconds (15 minutes).
In case of large fabrics, fabric health report might take longer than the default 15 minutes.
User can enlarge the timeout for fabric health report to complete.
• max_attached_file_size specifies the maximum file size in Bytes for each email attachment that can be sent. Default value is 2 Megabytes.
If the size of a certain file has exceeded this value, the file will not be sent as an attachment in the Daily Report mail.
[DailyReport]
# top_x specifies the number of results per each top x chart.
# max number can be 20.(default is 10) top_x=10
# max_reports specifies the number of reports to save.
# A count of 0 (zero) means no copies are retained.(default and max is
365) max_reports = 365
#time interval in minutes after midnight
#when passed mail will not be sent mail_send_interval=60 log_level = INFO daily_report_enabled = true attach_fabric_health_report = true fabric_health_report_timeout = 900
# max attached file size in bytes, default is 2M (2097152 Bytes) max_attached_file_size = 2097152
• max_attached_file_size specifies the maximum file size in Bytes for each email attachment that can be sent. Default value is 2 Megabytes.
• The start_hour and end_hour options enable selecting a sub-range of the day, during which, the relevant report data will be collected. Since by default this option is configured to collect data from the last 24 hours, the default start_hour is set to 0 (or 00), and the default end_hour is set to 24.
282
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
If these options are configured to different values, the generated report will include data from the specified interval only. The start_hour values range is 00 to 23, and the end_hour values range is 00 to 24. The specified end_hour must be greater than the specified start_hour. If, for example, the start_hour is configured to 08, and the end_hour is configured to 10, the generated report will include data collected between 08:00-10:00
(excluding 10:00).
Version 5.2
10.6.5 Changing Site Name
Site name can be edited via the UFM navigation tree.
To change the site name:
1. In the UFM navigation tree, select the site and right-click on it.
2. From the drop-down menu, select the “Configure Site” option. The following window will be displayed:
3. In the “Configure Site” window, insert the desired name and click “Apply”.
4. Close the window by clicking “OK”. The new site name will be displayed.
283
Mellanox Technologies Confidential
Version 5.2 Managing UFM and Fabric Health
284
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
11 Managing Users
11.1 User Authentication
UFM User Authentication is based on standard Apache User Authentication. Each Web
Service client application must authenticate against the UFM Server to gain access to the system. UFM implements any kind of third-party authentication supported by the Apache
Web Server.
The system default user is the admin user, who has read and write privileges.
All user management tasks can be performed using the UFM GUI.
11.2 Creating User Accounts
You can create and delete user accounts. All users have system administration privileges.
(To create and manage users with other privileges, an Advanced license is required.)
To create a user account
1. In the Config window, click the Users link. The User Management dialog box displays existing users.
2. To add a new user, right-click in the User table, and select Create New User.
Version 5.2
3. Enter the user name and password, and confirm the password.
4. Click OK or Apply.
11.3 User Account Management (Advanced License Only)
The default user (admin) has System Administration rights. A user with system
Administration rights can manage other users' accounts, including creation, deletion, and modification of accounts.
A UFM user can belong to one of the following groups:
•
Tenant Users can view and monitor allocated hosts and logical environments (logical servers, logical networks and interfaces).
For more details regarding user definition, please refer to Managing Users
285
Mellanox Technologies Confidential
Version 5.2 Managing Users
•
Monitoring Only – Users can see the fabric configuration, open monitoring sessions, define monitoring templates, and export monitoring data to CSV files.
•
Fabric Operator – Users can perform all operations allowed to Monitoring group users, and can also configure fabric, modify the fabric design, define logical objects, and allocate resources. Fabric Operator group users cannot create, delete or modify environments or global networks.
•
Fabric Administrator – Users can perform all operations allowed to Fabric Operator group users, and can also create, delete, and modify environments and global networks.
Users in this group cannot manage other users' accounts.
•
System Administrator – Users can perform all operations allowed to Fabric
Administration group, and can also manage other users' accounts.
To configure user accounts
1. In the Config window, click the Users link. The User Management dialog box displays existing users.
286
2. To add a new user, right-click in the User table, and select Create New User.
3. To edit a user's account information, double-click the user name.
4. Enter the user name and password, and select the user's authorization group.
5. Click OK or Apply.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
12 Multiple Tenants
Multiple tenants feature enables the fabric manager to divide and isolate the fabric into multiple segments and assign each tenant to one or more segments.
Each user can:
•
Access, view, and monitor only assigned own fabric resources.
•
View and monitor assigned hosts and logical environments (servers, network and interfaces.
•
Define and run monitoring sessions on assigned hosts and their ports.
•
Be notified on every UFM event related to one of assigned resources - hosts, ports and logical entities (logical server, networks, interfaces).
•
Set his own user password
Version 5.2
12.1 Adding Tenant Users
To add a new tenant user in UFM:
Step 1: Login to UFM GUI - Requires system administrator user rights.
Step 2: Go to Config Tab user panel.
Step 3: Click on the Users Table link.
Step 4: Click on the “+” icon.
Mellanox Technologies Confidential
287
Version 5.2
Step 5: Fill-in the User, Password, and Confirm Password fields.
Step 6: Select the option Tenant form the Group field and press OK.
288
NOTE: You may repeat this operation to add more tenant users.
Mellanox Technologies Confidential
Multiple Tenants
Unified Fabric Manager for InfiniBand User Manual
12.2 Assigning Resources to Tenant Users
System administrator should have all resources, such as logical networks and logical servers, defined and ready for assignment. For more details regarding the Logical Model, please refer
Version 5.2
Logical servers (and their contained physical hosts) will be eventually associated with tenant users through assigning logical networks to them.
12.2.1 Assigning Logical Networks to Tenant Users
To assign logical networks to a tenant user:
Step 1: System administrator should right click on the specific logical network.
Step 2: Select the “Associate With Tenant” option.
Step 3: Select the specific tenant user and press ‘OK’.
289
Mellanox Technologies Confidential
Version 5.2 Multiple Tenants
CAUTION: LogicalServer can not be connected to networks associated with different
Tenant users.
CAUTION: Logical networks can be assigned only to a single tenant user even though each tenant user can be assigned more than one logical network.
12.3 Tenant User - View, Capabilities and Actions
Once a tenant user is added and assigned logical networks, he may login to UFM GUI and view his related resources (hosts, logical networks, and logical server).
12.3.1 Design View
Only a Tenant user assigned logical entities will be presented at the Design View.
Figure 65: Design View Tab
290
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
12.3.2 Manage Devices
Only physical hosts related to the tenant user’s assigned networks will be presented at the
Manage Devices View.
Figure 66: Manage Devices Tab
Version 5.2
12.3.3 Monitoring
Tenant user can monitor only assigned logical servers and related hosts.
Figure 67: Objects for Monitoring Tab
A Tenant user can:
•
Define up to 4 monitoring sessions (Configurable)
•
View only his monitoring sessions (similar to any other UFM user)
291
Mellanox Technologies Confidential
Version 5.2 High Availability
•
Create his own monitoring templates and use only them (similar to any other UFM user)
For further details regarding Monitoring Sessions, please refer to Creating a Monitoring
12.3.4 Events
A Tenant user may receive the following events:
•
Events related to his logical networks and logical servers
•
Events on hosts and ports related to his assigned logical networks
Figure 68: Tenant Events
12.4 Tenant User Limitations
•
A system administrator can define up to 200 tenant users
•
No more than 20 tenant users may connect to UFM GUI simultaneously
13 High Availability
13.1 Overview of High Availability
UFM provides High Availability (HA) mechanisms to allow smooth fabric operation even if the UFM server fails or the connection between the UFM server and the rest of the fabric is not operating optimally.
UFM High Availability requires two distinct servers to run UFM software: one server is initially configured as the UFM active server and the other is configured as the UFM standby server. As a result, when the UFM active server fails or communication to the UFM active server ceases functioning, the UFM standby server takes over and becomes the new UFM active server. After such a failover, it is possible to repair the “old active UFM server” and bring it online as a new “UFM standby server”.
NOTE: Throughout this document, the following terms are used interchangeably:
Master - Active
Standby – Slave
292
UFM recovery relies on three mechanisms:
•
UFM Database replication (from active to standby server)
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
•
UFM Keep Alive (heartbeat) mechanism
•
UFM server failover
For information about installing and running the UFM software for High Availability, see
Installation and Initial Configuration
13.1.1 HA-Related Events
When the UFM server fails over to the UFM standby server, a UFM Failover event is generated.
13.1.2 HA-Related Considerations
We recommend that you locate the active and standby UFM servers in different sections of the fabric, so a single failure of an edge switch or a line card will not disconnect both UFM servers from the fabric.
We recommend that you bring up the failed UFM server as quickly as possible, to enable the fabric to sustain a possible secondary failure of the new active UFM server.
CAUTION: A secondary failover (from the "new" active server to the "newly" brought up standby server) will succeed only after the UFM database's initial replication as the
"new standby server" has been completed. UFM can sustain a second failover only a few minutes after the new UFM standby server is up and running.
This time depends on the size of the replicated partition and link speed (between the active and standby servers).
Version 5.2
13.2 High Availability Functionality
The high availability capability is based on standard Linux packages – heartbeat and drbd.
Both heartbeat and drbd are installed on the master and slave nodes:
• drbd
synchronizes a replicated partition between the two servers (but the partition itself
/dev/drbd0 is visible only on the master node).
/opt/ufm/files
is mounted on the drbd device and all data under this directory
(partition) is replicated.
•
Heartbeat is responsible for starting UFM on master node and stopping it on the slave.
Heartbeat sends "keep alive" messages between the two servers, and when the master fails, the slave assumes mastership.
NOTE: For high availability, use a reliable and high-capacity out-of-band management network (1 Gb Ethernet is recommended). Using inband IPoIB will cause the HA split-brain condition if there is an InfiniBand network failure.
A virtual IP (VIP) address is an IP address that is not connected to a specific computer or
Network Interface card (NIC) on a computer. Incoming packets are sent to the VIP address, but all packets travel through real Network Interfaces.
The VIP address belongs to the master node; failover of the system will result in failover of the virtual IP to the second node as well. When using UFM with HA, it is essential to always use the virtual IP instead of the server’s IP to assure UFM operation on the master server.
293
Mellanox Technologies Confidential
Version 5.2 High Availability
Always use the virtual IP instead of the server’s IP to assure connection to the UFM master server.
The failover will not happen if the standby server is not ready to take the mastership. UFM
Health periodically checks the readiness of the standby server for the following:
• management network connectivity
•
DRBD state
• disk space availability
• if the server is connected to the same InfiniBand cluster
• if the management InfiniBand port is Active and the IPoIB interface is UP and
RUNNING
If any of the condition above is not met, UFM Health will send a critical event. It is strongly recommended to repair the standby server as soon as possible to prevent risk of cluster malfunction.
13.3 Restoring High Availability on a Failed Server
Use the following procedure to replace the passive (standby) UFM server, without removing the installation and configuration from the active (master) UFM server.
When you replace the failed server, you must also generate and install a new license with a new MAC address.
It is strongly recommended to use the same IP address and partition on the new standby server as were used on the old standby server. This enables the restore operation to be performed without interruption of the active UFM. Otherwise, the restore procedure will interrupt the master UFM server.
To restore the standby UFM Server
On the new standby UFM server:
1. Format the server with dedicated disk partitions for UFM replication, identical to the master UFM server.
2. Create a temporary installation directory (for example /tmp/ufm).
3. From the UFM software zip file, extract the installation file for your system's OS (for example ufm5.2.0-XXX.el6.x86_64.tgz) to the temporary installation directory that you created.
4. Go to the installation directory, for example, cd /tmp/ufm/ufm-5.1.0-
XXX.el6.x86_64
5. Install the UFM software on the standby server.
./install.sh -o ib -r
When the installation completes, perform the following on the master UFM server:
1. Create a temporary installation directory (for example /tmp/ufm).
2. From the UFM software zip file, extract the installation file for your system's OS (for example ufm5.2.0-XXX.el6.x86_64.tgz) to the temporary installation directory that you created.
294
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
3. Go to the installation directory, for example, cd /tmp/ufm/ufm-5.2.0-XXX.el6.x86_64.
4. Extract the prereq_general.tar.gz file. tar xvf prereq_general.tar.gz
5. Create a restore directory. mkdir ha_restore
6. Go to the created restore directory. cd ha_restore
7. Extract the files from the archive. tar zxvf ../ha_files.tar.gz
8. Run the restore procedure:
./restore_ha.sh
Usage: restore_ha.sh -m <ip:drbd-disk[:hostname]> -s <ip:drbd-disk [:hostname]>
-c <virtual-ip> [ -n <netmask> ] [-b <broadcast-ip>] [ -u <heartbeat udp port> ] [ -t <drbd tcp port> ]
-n
-b
-u
-t
Option
-o
-m
-s
-c
-p
Description
(Installation only) Install mode, InfiniBand or Ethernet.
Master node parameters including IP address partition for drbd and optional hostname.
Standby node parameters including IP address partition for drbd and optional hostname.
Common IP address. GUI clients should connect to this address in HA mode.
This address is migrated between master and standby upon failover.
Interface for Common IP address alias-interface.
Mandatory if Common IP address does not match management IP subnet.
Netmask for the common IP address.
Broadcast address for the common IP address.
UDP port for heartbeat (default 694).
TCP port for drbd (default 8888).
Note: For options -m and -s, use a reliable and high-capacity out-of-band management network (1 Gb Ethernet is recommended). Using inband IPoIB will cause the HA split-brain condition if there is an InfiniBand network failure.
Examples restore_ha.sh -m 192.168.0.1:/dev/sda5:master -s
192.168.0.2:/dev/sda6:standby -c 192.168.0.3 -n 255.255.255.0 -b
192.168.0.255 restore_ha.sh -m 192.168.0.1:/dev/sda5 -s 192.168.0.2:/dev/sda6 -c
192.168.0.3
295
Mellanox Technologies Confidential
Version 5.2 High Availability
CAUTION:
•
We recommend that you perform a UFM failover from the Active to the Standby server to verify UFM functionality on the second server after installation. After successful verification, you may revert to the original active server. This helps verify the functionality of the standby server installation and failover configuration.
•
We recommend that you periodically backup the UFM configuration for disaster recovery. To perform a backup, copy all the content of the /opt/ufm/files folder to the backup location.
13.4 Restoring Monitoring History Configuration
In case of history data loss, you can restore your Monitoring History configuration as follow:
•
If Monitoring History feature is configured locally, and the failed server is the Monitoring
History main server, the entire collected history is lost and you must reconfigure the
Monitoring History feature:
/opt/ufm/scripts/monitoring_history_configure.sh --init --local -partition <DIR> --db-ip <IP> --num-of-days-to-keep <days> [--db-name
<name>] [--db-port <port>]
If the feature was configured remotely or the failed server is not the main Monitoring
History server , please follow the second option below
•
If the failed server is not the main Monitoring History server, the collected data can be recovered by running the following command on the new standby server:
/opt/ufm/scripts/monitoring_history_configure.sh --init --secondary -partition <DIR> --db-engine <mysql|mssql> --db-ip <IP> --db-port <port>
296
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
14 UFM Communication Requirements
This chapter describes how the UFM server communicates with InfiniBand fabric components.
14.1 UFM Server Communication with Clients
The UFM Server communicates with clients over IP. The UFM Server can belong to a separate IP network, which can also be behind the firewall.
Figure 69: UFM Server Communication with Clients
Version 5.2
14.1.1 UFM Server Communication with the UFM GUI Client
Communication between the UFM Server and the UFM GUI client is HTTP(s) based. The only requirement is that TCP port 80 (443) must not be blocked.
14.1.2 UFM Server Communication with SNMP Trap Managers
The UFM Server can send SNMP traps to configured SNMP Trap Manager(s). By default, the traps are sent to the standard UDP port 162. However, the user can configure the destination port. If the specified port is blocked, UFM Server traps will not reach their destination.
297
Mellanox Technologies Confidential
Version 5.2 UFM Communication Requirements
14.1.3 Summary of UFM Server Communication with Clients
Affected Service Network
GUI Client Out-of-band management*
SNMP Trap Notification Out-of-band management*
Address / Service /
Port
HTTP / 80
HTTPS / 443
UDP / 162
(configurable)
Direction
Bi-directional
UFM Server to SNMP
Manager
*If the client machine is connected to the IB fabric, IPoIB can also be used.
14.2 UFM Server Communication with InfiniBand Switches
Figure 70: UFM Server Communication with InfiniBand Switches
14.2.1 UFM Server InfiniBand Communication with the Switch
The UFM Server must be connected directly to the InfiniBand fabric (via an InfiniBand switch). The UFM Server sends the standard InfiniBand Management Datagrams (MAD) to the switch and receives InfiniBand traps in response.
14.2.2 UFM Server Communication with Switch Management Software
(Optional)
The UFM Server auto-negotiates with the switch management software on Mellanox Grid
Director switches. The communication is bound to the switch Ethernet management port.
The UFM Server sends a multicast notification to MCast address 224.0.23.172, port 6306
(configurable). The switch management replies to UFM (via port 6306) with a unicast
298
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual message that contains the switch GUID and IP address. After auto-negotiation, the UFM
Server and switch management use XML-based messaging.
The following Device Management tasks are dependent on successful communication as described above:
•
Switch IP discovery
•
FRU Discovery (PSU, FAN, status, temperature)
•
Software and firmware upgrades
The UFM Server manages IB Switch Devices over SNMP (default port 161 - configurable) and / or SSH (default port 22 - configurable).
14.2.3 Summary of UFM Server Communication with InfiniBand Switches
Affected Service
InfiniBand Management /
Monitoring
Switch IP Address
Discovery (auto-negotiation with switch management software)
Network
InfiniBand
Out-of-band management
Switch Chassis Management
/ Monitoring
Out-of-band management
Address / Service / Port
Management Datagrams
Multicast 224.0.23.172,
TCP / 6306 (configurable)
TCP / UDP / 6306
(configurable)
SNMP / 161 (configurable)
SSH / 22 (configurable)
Direction
Bi-directional
Multicast: UFM
Server to switch
TCP: Bidirectional
Bi-directional
Version 5.2
Mellanox Technologies Confidential
299
Version 5.2 UFM Communication Requirements
14.3 UFM Server Communication with InfiniBand Hosts
Figure 71: UFM Server Communication with InfiniBand Hosts
14.3.1 UFM Server InfiniBand Communication with HCAs
The UFM Server must be connected directly to the InfiniBand fabric. The UFM Server sends the standard InfiniBand Management Datagrams (MADs) to the Host Card Adapters (HCAs) and receives InfiniBand traps.
14.3.2 UFM Server Communication with Host Management (Optional)
The UFM Server auto-negotiates with the UFM Agent on a Host. The UFM Host Agent can be bound to the management Ethernet port or to an IPoIB interface (configurable). The UFM
Server sends a multicast notification to MCast address 224.0.23.172, port 6306
(configurable). The UFM Agent replies to UFM (port 6306) with a unicast message that contains the host GUID and IP address. After auto-negotiation, the UFM Server and UFM
Agent use XML-based messaging.
The following Device Management tasks are dependent on successful communication as described above:
•
Host IP discovery
•
Host resource discovery and monitoring: CPU, memory, disk
•
Software and firmware upgrades
Note: UFM 3.6 supports in-band HCA FW upgrade. This requires enabling FW version and PSID discovery over vendor-specific MADs. for more information, see the UFM
User Manual.
300
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
The UFM Server connects to the hosts over SSH (default port 22 - configurable) with root credentials, which are located in the UFM Server database.
14.3.3 Summary of UFM Server Communication with InfiniBand Hosts
Affected Service
InfiniBand Management /
Monitoring
Network
InfiniBand
Host IP Address Discovery
(auto-negotiation with UFM
Host Agent)
Out-of-band management or
IPoIB
Host OS Management /
Monitoring
Out-of-band management or
IPoIB
Address / Service / Port Direction
Management Datagrams Bi-directional
Multicast 224.0.23.172,
TCP / 6306
(configurable)
TCP / UDP / 6306
(configurable)
SSH / 22 (configurable)
Multicast: UFM
Server to UFM
Agent
TCP: Bidirectional
Bi-directional
Version 5.2
14.4 UFM Server High Availability (HA) Active – Standby
Communication
Figure 72: UFM Server HA Active – Standby Communication
14.4.1 UFM Server HA Active – Standby Communication
UFM Active – Standby communication enables two services: heartbeat and DRBD.
•
heartbeat is used for auto-negotiation and keep-alive messaging between active and standby servers. heartbeat uses port 694 (udp).
Mellanox Technologies Confidential
301
Version 5.2 UFM Communication Requirements
•
DRBD is used for low-level data (disk) synchronization between active and standby servers. DRBD uses port 8888 (tcp)
Affected Service Network Direction
UFM HA heartbeat
Address / Service /
Port
UDP / 694 Bi-directional
UFM HA DRBD
Out-of-band management*
Out-of-band management*
TCP / 8888 Bi-directional
*An IPoIB network can be used for HA, but this is not recommended, since any InfiniBand failure might cause split brain and lack of synchronization between the active and standby servers.
302
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Appendix A: SM Default Files
The SM default files are located under the following paths:
•
Default SM configuration file - /opt/ufm/files/conf/opensm/opensm.conf
•
Default node name map file - /opt/ufm/files/conf/opensm/ib-node-namemap
•
Default partition configuration file -
/opt/ufm/files/conf/opensm/partitions.conf
•
Default QOS policy configuration file - /opt/ufm/files/conf/opensm/qospolicy.conf
•
Default prefix routes file - /opt/ufm/files/conf/opensm/prefixroutes.conf
•
Default SM log - /opt/ufm/files/log/opensm.log
Version 5.2
Mellanox Technologies Confidential
303
Version 5.2 UFM Subnet Manager Default Properties
Appendix B: UFM Subnet Manager Default
Properties
The following table provides a comprehensive list of UFM SM default properties.
Table 78: UFM SM Default Properties
Category Property
Keys
Limits
M_Key
M_Key Lease
Period
SM_Key
SA_Key
Packet Life
Time
Config File
Attribute
m_key m_key_lease_p eriod sm_key sa_key packet_life_ti me
Default
0x000000000
0000000
0
0x000000000
0000001
0x000000000
0000001
Subnet Prefix subnet_prefix 0xfe8000000
0000000
LMC lmc 0
Partition enforcement part_enforce
•
Out
•
In
•
Both
(default- outbound and inbound enforceme nt enabled)
MKEY lookup m_key_lookup FALSE
RO
RW
0x12
Mode/
Field
RW
RW
RO
RO
RW
RW
Description
M_Key value sent to all ports -used to qualify the set(PortInfo)
The lease period used for the M_Key on the subnet in [sec]
SM_Key value of the SM used for SM authentication
SM_Key value to qualify rcv SA queries as 'trusted'
Subnet prefix used on the subnet
0xfe80000000000000
The LMC value used on the subnet: 0-7
Changes to the LMC parameter require a UFM restart.
Partition enforcement type (for switches)
RW
If FALSE, SM will not try to determine the m_key of unknown ports.
The maximum lifetime of a packet in a switch.
The actual time is
4.096usec *
2^<packet_life_time>
The value 0x14 disables the mechanism
304
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Category Property
Limits
VL Stall
Count
Leaf VL Stall
Count
Head Of
Queue Life time
Leaf Head Of
Queue Life time
Config File
Attribute
Default
vl_stall_count 0x07 leaf_vl_stall_c ount head_of_queue
_lifetime leaf_head_of_q ueue_lifetime
0x07
0x12
0x10
Maximal
Operational
VL
Force Link
Speed max_op_vls 3 force_link_spe ed
15
(Do NOT change)
Version 5.2
Mode/
Field
RO
RO
RW
RW
RW
Description
The number of sequential packets dropped that cause the port to enter the
VL Stalled state. The result of setting the count to zero is undefined.
The number of sequential packets dropped that causes the port to enter theleaf VL Stalled state.
The count is for switch ports driving a CA or gateway port. The result of setting the count to zero is undefined.
The maximum time a packet can wait at the head of the transmission queue. The actual time is
4.096usec *
2^<head_of_queue_lifeti me>
The value 0x14 disables the mechanism
The maximum time a packet can wait at the head of queue on a switch port connected to a CA or gateway port.
Limit of the maximum operational VLs
RO Force PortInfo:
LinkSpeedEnabled on switch ports.
If 0, do not modify.
Values are:
1: 2.5 Gbps
3: 2.5 or 5.0 Gbps
5: 2.5 or 10.0 Gbps
7: 2.5 or 5.0 or 10.0 Gbps
2,4,6,8-14 Reserved
15: set to PortInfo:
LinkSpeedSupported
305
Mellanox Technologies Confidential
Version 5.2 UFM Subnet Manager Default Properties
Category Property
Limits
Sweep
Handover
Subnet
Timeout
Local PHY
Error
Threshold
Reassign Lids
Force Heavy
Sweep
Sweep On trap sweep_on_trap TRUE
Alternative
Route
Calculation
Fabric
Rediscovery
SM Priority
Config File
Attribute
subnet_timeout local_phy_erro rs_threshold max_alt_dr_pa th_retries
0x08
Overrun
Errors
Threshold overrun_errors
_threshold
0x08
Sweep Interval sweep_interval 10 reassign_lids force_heavy_s weep_window max_seq_redis c sm_priority
Default
18 (1second)
FALSE
(disabled)
-1
(enabled)
4
2
15
Mode/
Field
Description
RW
RW
RW
RW
RW
RW
RW
RW
RW
RO
For a fabric with DDR 10
Gb Ethernet Gateways
(sRB-20210G) and 4036E version lower than
3.7.0.896, this parameter should be set to None (0)
– auto negotiate link speed according to INI settings.
The subnet_timeout code that will be set for all the ports.
The actual timeout is
4.096usec *
2^<subnet_timeout>
Threshold of local phy errors for sending Trap
129
Threshold of credit overrun errors for sending
Trap 130
The time in seconds between subnet sweeps
(Disabled if 0)
If TRUE (enabled), all
LIDs are reassigned
Forces heavy sweep after number of light sweeps
(-1 disables this option and 0 will cause every sweep to be heavy)
If TRUE every trap will cause a heavy sweep
Maximum number of attempts to find an alternative direct route towards unresponsive ports
Max Failed Sequential
Discovery Loops
SM (enabled). The priority used for deciding which is the master.
Range is 0 (lowest priority) to 15 (highest)
306
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Category Property
Threadin g
Logging
Ignore Other
SMs
Polling
Timeout
Polling Retries
Honor GUIDto-LID File
Max Wire
SMPs
Transaction
Timeout
Max Message
FIFO Timeout honor_guid2lid
_file max_wire_smp s transaction_tim eout
FALSE
(disabled)
8
200
Single Thread single_thread FALSE
Routing
Threads
Routing
Threads Per
Core
Log File
Config File
Attribute
ignore_other_s m sminfo_polling
_timeout polling_retry_n umber max_msg_fifo
_timeout routing_thread s_num max_threads_p er_core log_file
Default
FALSE
(disabled)
10
4
10000
(disabled)
0
0
/opt/ufm/files/ log/opensm.lo
g
Mode/
Field
RO
RO
RO
RO
RW
RO
RO
RO
RW
RW
RO
Description
If TRUE other SMs on the subnet should be ignored
Timeout in seconds between two active master SM polls
Number of failing remote
SM polls that declares it non-operational
If TRUE, honor the guid2lid file when coming out of standby state, if the guid2lid file exists and is valid
Maximum number of
SMPs sent in parallel
The maximum time in
[msec] allowed for a transaction to complete
Maximum time in [msec] a message can stay in the incoming message queue
Use a single thread for handling SA queries
Number of threads to be used for parallel minhop/updn calculations.
If 0, number of threads will be equal to number of processors.
Max number of threads that are allowed to run on the same processor during parallel computing.
If 0, threads assignment per processor is up to operating system initial assignment.
Path of Log file to be used
307
Mellanox Technologies Confidential
Version 5.2 UFM Subnet Manager Default Properties
Category Property
Log Flags
Config File
Attribute
log_flags
Default Mode/
Field
Error and Info
0x03
RW
RO Force Log
Flush force_log_flus h
FALSE
(disabled)
Log Max Size log_max_size 4096 RW
Description
The log flags, or debug level being used.
Force flush of the log file after each log message
Misc
Accumulate
Log File
Dump Files
Directory
Syslog log
Node Names
Map File accum_log_file dump_files_dir syslog_log node_name_m ap_name
TRUE
(enabled)
/opt/ufm/files/ log
0x0
Null
RO
RO
RW
RW
RO
Limit the size of the log file in MB. If overrun, log is restarted
If TRUE, will accumulate the log over multiple
OpenSM sessions
The directory to hold the file SM dumps (for multicast forwarding tables for example). The file is used collects information.
Sets a verbosity of messages to be printed in syslog
Node name map for mapping node's to more descriptive node descriptions
SA database file name SA database
File
No Clients
Reregistration
Exit On Fatal
Event sa_db_file no_clients_rere g exit_on_fatal
Null
FALSE
(disabled)
TRUE
(enabled)
RO
RO
If TRUE, disables client reregistration
Switch
Isolation From
Routing
Multicast Disable
Multicast
Multicast
Group
Parameters held_back_sw_ file disable_multic ast default_mcg_ mtu
Null
FALSE
(disabled)
0
RW
RO
RW
If TRUE (enabled), the
SM exits for fatal initialization issues
File that contains GUIDs of switches isolated from routing
If TRUE, OpenSM should disable multicast support and no multicast routing is performed
Default MC group MTU for dynamic group creation. 0 disables this feature, otherwise, the value is a valid IB encoded MTU
308
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Category Property
Multicast
Multicast
Enable incremental multcast routing
Multicast
MC root file
QoS
Unhealth y Ports
Routing
Multicast
Group
Parameters
Settings
Enabling
Unhealthy
Ports
Configuration file
Unhealthy actions
MADs validation
Randomizatio n
Config File
Attribute
default_mcg_ra te enable_inc_mc
_routing hm_ports_healt h_policy_file hm_sw_manua l_action validate_smp scatter_ports
Default
0
FALSE mc_roots_file qos null hm_unhealthy_ ports_checks
FALSE
(disabled)
*From UFM v3.7 and on
TRUE null
TRUE
8
Mode/
Field
RW
RW
Description
Default MC group rate for dynamic group creation. 0 disables this feature, otherwise, the value is a valid IB encoded rate
Enable incremental multcast routing
RW
RW
RW
RW no_discover RW
RW
RW
Specify predefined MC groups root guids
If FALSE (disabled),
UFM will not apply QoS settings
Enables Unhealthy Ports configuration
Specifies configuration file for health policy
Specifies what to do with switch ports which were manually added to health policy file
If set to TRUE, opensm will ignore nodes sending non-spec compliant
MADs. When set to
FALSE, opensm will log the warning in the opensm log file about non-compliant node
Assigns ports in a random order instead of roundrobin. If 0, the feature is disabled, otherwise use the value as a random seed.
Applicable to the
MINHOP/UPDN routing algorithms
309
Mellanox Technologies Confidential
Version 5.2
310
UFM Subnet Manager Default Properties
Category Property
Randomizatio n
Config File
Attribute
Default
guid_routing_o rder_no_scatter
TRUE
Unicast
Routing
Caching
GUID
Ordering
During
Routing use_ucast_cach e guid_routing_o rder_file
TRUE
NULL
Torus Routing
Routing
Chains torus_config /opt/ufm/files/ conf/opensm/t orus-
2QoS.con
NULL pgrp_policy_fil e topo_policy_fil e
NULL rch_policy_file NULL
RW
RW
RW
RW
RW
RW
Mode/
Field
RO
Description
Do not use scatter for ports defined in guid_routing_order file
Use unicast routing cache for routing computation time improvement
The file holding guid routing order of particular guids (for MinHop,
Up/Down)
Torus-2QoS configuration file name
Incremental
Multicast
Routing (IMR)
MC Global root max_topologie s_per_sw enable_inc_mc
_routing
1
TRUE
0x000000000
0000000 (for both)
RO
RW
RW
The file holding the port groups policy
The file holding the topology policy
The file holding the routing chains policy
Defines maximal number of topologies to which a single switch may be assigned during routing engine chain configuration.
If TRUE, MC nodes will be added to the MC tree incrementally. When set to FALSE, the tree will be recalculated per eachg change.
Primary and Secondary global mc root guid
Events
Scatter ports
Event
Subscription
Handling
Event
Subscription
Handling mc_primary_ro ot_guid/mc_se condary_root_ guid use_scatter_for
_switch_lid drop_subscr_o n_report_fail drop_event_su bscriptions
FALSE
FALSE
TRUE
RW
RW
RW
Use scatter when routing to the switch’s LIDs
Drop subscription on report failure (o13-17.2.1)
Drop event subscriptions
(InformInfo and
ServiceRecords) on port removal and SM coming out of STANDBY
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Category Property
Virtualiza tion
Virtualization enabled
Maximum ports in virtualization process
Config File
Attribute
virt_enabled virt_max_ports
_in_process
Default
0 (Ignored)
4
Version 5.2
Mode/
Field
RW
RW
Description
Enables/disables virtualization support
Sets a number of ports to be handled on each virtualization process cycle.
Mellanox Technologies Confidential
311
Version 5.2 SM Partitions.conf File Format
Appendix C: SM Partitions.conf File Format
This appendix presents the content and format of the OpenSM partitions.conf file.
OpenSM Partition configuration
===============================
The default partition will be created by OpenSM unconditionally even when partition configuration file does not exist or cannot be accessed.
The default partition has P_Key value 0x7fff. OpenSM's port will always have full membership in default partition. All other end ports will have full membership if the partition configuration file is not found or cannot be accessed, or limited membership if the file exists and can be accessed but there is no rule for the Default partition.
Effectively, this amounts to the same as if one of the following rules below appear in the partition configuration file:
In the case of no rule for the Default partition:
Default=0x7fff : ALL=limited, SELF=full ;
In the case of no partition configuration file or file cannot be accessed:
Default=0x7fff : ALL=full ;
File Format
===========
Comments:
Line content followed after \'#\' character is comment and ignored by parser.
General file format:
<Partition Definition>:[<newline>]<Partition Properties>;
Partition Definition:
[PartitionName][=PKey][,ipoib_bc_flags][,defmember=full|limited]
PartitionName - string, will be used with logging. When omitted
empty string will be used.
PKey - P_Key value for this partition. Only low 15 bits will
be used. When omitted will be autogenerated.
ipoib_bc_flags - used to indicate/specify IPoIB capability of this partition.
defmember=full|limited - specifies default membership for port guid
list. Default is limited.
ipoib_bc_flags:
ipoib_flag|[mgroup_flag]*
ipoib_flag - indicates that this partition may be used for IPoIB, as
a result the IPoIB broadcast group will be created with
the flags given, if any.
Partition Properties:
[<Port list>|<MCast Group>]* | <Port list>
Port list:
<Port Specifier>[,<Port Specifier>]
Port Specifier:
<PortGUID>[=[full|limited]]
312
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
PortGUID - GUID of partition member EndPort. Hexadecimal
numbers should start from 0x, decimal numbers
are accepted too.
full or limited - indicates full or limited membership for this
port. When omitted (or unrecognized) limited
membership is assumed.
MCast Group:
mgid=gid[,mgroup_flag]*<newline>
- gid specified is verified to be a Multicast address
IP groups are verified to match the rate and mtu of the
broadcast group. The P_Key bits of the mgid for IP
groups are verified to either match the P_Key specified
in by "Partition Definition" or if they are 0x0000 the
P_Key will be copied into those bits.
mgroup_flag:
rate=<val> - specifies rate for this MC group
(default is 3 (10GBps))
mtu=<val> - specifies MTU for this MC group
(default is 4 (2048))
sl=<val> - specifies SL for this MC group
(default is 0)
scope=<val> - specifies scope for this MC group
(default is 2 (link local)). Multiple scope settings
are permitted for a partition.
NOTE: This overwrites the scope nibble of the specified
mgid. Furthermore specifying multiple scope
settings will result in multiple MC groups
being created.
qkey=<val> - specifies the Q_Key for this MC group
(default: 0x0b1b for IP groups, 0 for other groups)
WARNING: changing this for the broadcast group may
break IPoIB on client nodes!!!
tclass=<val> - specifies tclass for this MC group
(default is 0)
FlowLabel=<val> - specifies FlowLabel for this MC group
(default is 0)
newline: '\n'
Note that values for rate, mtu, and scope, for both partitions and multicast groups, should be specified as defined in the IBTA specification (for example, mtu=4 for 2048).
There are several useful keywords for PortGUID definition:
- 'ALL' means all end ports in this subnet.
- 'ALL_CAS' means all Channel Adapter end ports in this subnet.
- 'ALL_SWITCHES' means all Switch end ports in this subnet.
- 'ALL_ROUTERS' means all Router end ports in this subnet.
- 'SELF' means subnet manager's port.
Empty list means no ports in this partition.
Notes:
-----
White space is permitted between delimiters ('=', ',',':',';').
PartitionName does not need to be unique, PKey does need to be unique.
313
Mellanox Technologies Confidential
Version 5.2 SM Partitions.conf File Format
If PKey is repeated then those partition configurations will be merged and first PartitionName will be used (see also next note).
It is possible to split partition configuration in more than one definition, but then PKey should be explicitly specified (otherwise different PKey values will be generated for those definitions).
Examples:
--------
Default=0x7fff : ALL, SELF=full ;
Default=0x7fff : ALL, ALL_SWITCHES=full, SELF=full ;
NewPartition , ipoib : 0x123456=full, 0x3456789034=limited, 0x2134af2306 ;
YetAnotherOne = 0x300 : SELF=full ;
YetAnotherOne = 0x300 : ALL=limited ;
ShareIO = 0x80 , defmember=full : 0x123451, 0x123452;
# 0x123453, 0x123454 will be limited
ShareIO = 0x80 : 0x123453, 0x123454, 0x123455=full;
# 0x123456, 0x123457 will be limited
ShareIO = 0x80 : defmember=limited : 0x123456, 0x123457, 0x123458=full;
ShareIO = 0x80 , defmember=full : 0x123459, 0x12345a;
ShareIO = 0x80 , defmember=full : 0x12345b, 0x12345c=limited, 0x12345d;
# multicast groups added to default
Default=0x7fff,ipoib:
mgid=ff12:401b::0707,sl=1 # random IPv4 group
mgid=ff12:601b::16 # MLDv2-capable routers
mgid=ff12:401b::16 # IGMP
mgid=ff12:601b::2 # All routers
mgid=ff12::1,sl=1,Q_Key=0xDEADBEEF,rate=3,mtu=2 # random group
ALL=full;
Note:
----
The following rule is equivalent to how OpenSM used to run prior to the partition manager:
Default=0x7fff,ipoib:ALL=full;
314
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Appendix D: Supported Port Counters and Events
Port counters and events are available in the following views:
•
Events and Port Counters area, at the bottom of the UFM window
•
Error window (Error tab),in the Manage Devices tab
•
In the New Monitoring Session window, in the Monitor Tab, when clicking Create
New Session
•
Event Log in the Log tab (click Show Event Log)
Version 5.2
D.1 InfiniBand Port Counters
The following tables list and describe the port counters and events currently supported:
•
InfiniBand Port Counters
•
Calculated Port Counters
Table 79: InfiniBand Port Counters
Counter
Xmit Data (in bytes)
Rcv Data (in bytes)
Xmit Packets
Rcv Packets
Rcv Errors
Xmit Discards
Description
Total number of data octets, divided by 4, transmitted on all VLs from the port, including all octets between (and not including) the start of packet delimiter and the VCRC, and may include packets containing errors. All link packets are excluded. Results are reported as a multiple of four octets.
Total number of data octets, divided by 4, received on all VLs at the port.
All octets between (and not including) the start of packet delimiter and the VCRC are excluded, and may include packets containing errors.
All link packets are excluded. When the received packet length exceeds the maximum allowed packet length specified in C7-45:, the counter may include all data octets exceeding this limit. Results are reported as a multiple of four octets.
Total number of packets transmitted on all VLs from the port, including packets with errors, and excluding link packets.
Total number of packets, including packets containing errors and excluding link packets, received from all VLs on the port.
Total number of packets containing errors that were received on the port including:
•
Local physical errors (ICRC, VCRC, LPCRC, and all physical errors that cause entry into the BAD PACKET or BAD PACKET
DISCARD states of the packet receiver state machine)
•
Malformed data packet errors (LVer, length, VL)
•
Malformed link packet errors (operand, length, VL)
•
Packets discarded due to buffer overrun (overflow)
Total number of outbound packets discarded by the port when the port is down or congested for the following reasons:
•
Output port is not in the active state
315
Mellanox Technologies Confidential
Version 5.2 Supported Port Counters and Events
316
Counter
Symbol Errors
Link Error Recovery
Link Error Downed
Local Integrity Error
Description
•
Packet length has exceeded NeighborMTU
•
Switch Lifetime Limit exceeded
•
Switch HOQ Lifetime Limit exceeded, including packets discarded while in VLStalled State.
Total number of minor link errors detected on one or more physical lanes.
Total number of times the Port Training state machine has successfully completed the link error recovery process.
Total number of times the Port Training state machine has failed the link error recovery process and downed the link.
The number of times that the count of local physical errors exceeded the threshold specified by LocalPhyErrors;
Rcv Remote Physical Error Total number of packets marked with the EBP delimiter received on the port.
Xmit Constraint Error Total number of packets not transmitted from the switch physical port for the following reasons:
•
FilterRawOutbound is true and packet is raw
•
PartitionEnforcementOutbound is true and packet fails partition key check or IP version check
Rcv Constraint Error Total number of packets received on the switch physical port that are discarded for the following reasons:
•
FilterRawInbound is true and packet is raw
•
PartitionEnforcementInbound is true and packet fails partition key check or IP version check
Excess Buffer Overrun
Error
Rcv Switch Relay Error
VL15 Dropped
XmitWait
The number of times that OverrunErrors consecutive flow control update periods occurred, each having at least one overrun error
Total number of packets received on the port that were discarded when they could not be forwarded by the switch relay for the following reasons:
•
DLID mapping
•
VL mapping
•
Looping (output port = input port)
Number of incoming VL15 packets dropped because of resource limitations (e.g., lack of buffers) in the port
The number of ticks during which the port selected by PortSelect had data to transmit but no data was sent during the entire tick because of insufficient credits or of lack of arbitration.
Table 80: InfiniBand Calculated Port Counters
Counter
Normalized XmitData
Normalized Congested
Bandwidth
Description
Effective port bandwidth utilization in %
XmitData incremental/ Link Capacity
Amount of bandwidth that was suppressed due to congestion
(XmitWait incremental/ Time) * Link Capacity
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Counter
Normalized XmitWait
Description
Separate counters are used for Tier 4 ports and for the rest of the ports.
Congestion in relation to packets transmitted over the link
XmitWait incremental / XmitPackets incremental
This event is calculated only for the port directly connected to receiving hosts.
Separate counters are used for Tier 4 ports and for the rest of the ports.
Version 5.2
D.2 Supported Traps and Events
Device events are listed as VDM or CDM in the Source column of the Events table in the
UFM GUI. For information about defining event policy, see Configuring Event Management
Description / Message
116 Port Xmit
Discards
1 1 Minor 200 300 Port Communication
Error
Total number of outbound packets discarded by the port when the port is down or congested.
Reasons include:
•
Output port is not in the active state
•
Packet length exceeded
NeighborMTU
•
Switch Lifetime
Limit exceeded
•
Switch HOQ
Lifetime Limit exceeded
•
Packets discarded while in
VLStalled State
317
Mellanox Technologies Confidential
Version 5.2 Supported Port Counters and Events
Description / Message
117 Port Xmit
Constraint
Errors
1 1
120 Excessive
121
Buffer
Overrun
Errors
VL15
Dropped
1
1
1
1
Minor 200 300 Port Communication
Error
Total number of packets not transmitted from the switch physical port for the following reasons:
•
FilterRawOutbou nd is true and packet is raw
•
PartitionEnforce mentOutbound is true and packet fails partition key check or IP version check
Minor 100 300 Port Communication
Error
The number of times that OverrunErrors consecutive flow control update periods occurred, each having at least one overrun error.
Message:
ExcessiveBufferOver runErrors counter threshold exceeded.
Threshold is %d, received value is %d.
Minor 50 300 Port Communication
Error
Number of incoming
VL15 packets dropped due to resource limitations
(e.g., lack of buffers) in the port.
Message:
VL15Dropped counter threshold exceeded. Threshold is %d, received value is %d.
318
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Description / Message
118 Port
Receive
Constraint
Errors
1 1 Minor 200 300 Port Communication
Error
Total number of packets received on the switch physical port that are discarded for the following reasons:
•
FilterRawInboun d is true and packet is raw
•
PartitionEnforce mentInbound is true and packet fails partition key check or IP version check
Info 1 300 Port Communication
Error
System GUID is changed for the specific LID
145 System
Image
GUID changed
115 Port
Receive
Switch
Relay
Errors
1 0
1 1 Minor 9999 300 Port Fabric
Configuration
Total number of packets received on the port that were discarded because they could not be forwarded by the switch relay.
Reasons for this include:
•
DLID mapping
•
VL mapping
•
Looping (output port = input port)
Mellanox Technologies Confidential
319
Version 5.2 Supported Port Counters and Events
Description / Message
256 Bad
M_Key
257 Bad
P_Key
1 0 Minor 1
1 0 Minor 1
300 Port Fabric
Configuration
300 Port Fabric
Configuration
Found bad M_Key.
Check your HCA driver or partition settings.
SM Trap.
Management Key
(M_Key): Enforces the control of a master subnet manager.
Administered by the subnet manager and used in certain subnet management packets.
Message: Bad
M_Key: port1(lid
%(lid)d, #%(portn)d)
%(pkey)08x, port2(lid%(lid2)d
#%(portn2)d)
Found a bad P_Key.
Check your partitioning settings.
SM Trap. Partition
Key (P_Key):
Enforces membership.
Administered through the subnet manager by the partition manager
(PM).
Message: Bad
P_Key: port1(lid
%(lid)d, #%(portn)d)
%(pkey)08x, port2(lid%(lid2)d
#%(portn2)d)
320
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Description / Message
258 Bad
Q_Key
1 0 Minor 1
259 Bad
P_Key
Switch
External
Port
1 0 Critica l
1
64 GID
Address
In Service
65 GID
Address
Out of
Service
66 New
MCast
Group
Created
1 0
1 0
1 0
Info
Warni ng
Info
1
1
1
300 Port Fabric
Configuration
300 Port Fabric
Configuration
300
300
Port
Port
Fabric
Notification
Fabric
Notification
Found bad Q_Key.
Security error.
SM Trap. Queue Key
(Q_Key): Enforces access rights for reliable and unreliable datagram service (RAW datagram service type not included).
Message: Bad
Q_Key: port1(lid
%(lid)d, #%(portn)d)
%(pkey)08x, port2(lid%(lid2)d
#%(portn2)d)
Found a bad P_Key.
Check your partitioning settings.
SM Trap. Partition
Key (P_Key):
Enforces membership.
Administered through the subnet manager by the partition manager
(PM).
Message: Bad P_Key switch external port: port1(lid %(lid)d,
#%(portn)d)
%(pkey)08x, port2(lid%(lid2)d
#%(portn2)d)
New GID is connected to the
Fabric
Existing GID is disconnected from the Fabric
300 Port Fabric
Notification
New Multicast Group is created in SM
321
Mellanox Technologies Confidential
Version 5.2 Supported Port Counters and Events
Description / Message
322
67 MCast
Group
Deleted
328 Link is
Up
1 0 Info
1 0 Info
1
1
328 Link is
Down
1 0 Warni ng
1
144 Capability
Mask
Modified
602 UFM
Server
Failover
391 Switch
Module
Removed
0 0
1 1
1 0
Info
Critica l
Info
1
1
1
331 Node is
Down
332 Node is
Up
1
1
0
0
370 Gateway
Ethernet
Link State
Changed
1 0
1 0 371 Gateway
Reregister
Event
Received
372 Number of
Gateways is
Changed
373 Gateway will be
Rebooted
1
1
0
0
Warni ng
Info
1
1
Warni ng
1
Warni ng
Warni ng
Warni ng
1
1
1
300
0
0
0
0
Port Fabric
Notification
Link Fabric
Topology
Link Fabric
Topology
300 Port Fabric
Notification
Site Fabric
Notification
Swit ch
Fabric
Notification
0 Site Fabric
Topology
300 Site Fabric
Topology
0 Gate way
Gateway
0 Gate way
Gateway
0 Gate way
Gateway
0 Gate way
Gateway
Multicast Group is removed from SM.
Event is sent upon discovery of a new link
Event is sent when exiting link is removed
Capability Mask of the specific LID is modified
Failover in UFM
Server (in HA mode)
Module (line card,
FAN or PS) is removed from the switch
Node is disconnected or down
Node is connected or up
Gateway Ethernet
Physical link has changed state
10GbE Gateway received a re-register event from the SM.
Change in the number of 10GbE
Gateways has been detected
10GbE Gateway is about to reboot
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Description / Message
374 Gateway
Reloading
Finished
110 Symbol
Error
1 0
1 1
Info
Warni ng
1 0 Gate way
Gateway
200 300 Port Hardware
111 Link Error
Recovery
1 1 Minor 1 300 Port Hardware
112 Link
Downed
1 1 Critica l
1 300 Port Hardware
113 Port
Receive
Errors
1 1 Minor 5 300 Port Hardware
10GbE Gateway has finished reloading.
Total number of minor link errors detected on one or more physical lanes
Total number of times the Port
Training state machine has successfully completed the link error recovery process
Total number of times the Port
Training state machine has failed the link error recovery process and downed the link.
Total number of packets containing an error that were received on a port.
These errors include:
•
Local physical errors (CRC,
VCRC, FCCRC and all physical errors that cause entry into the
BAD PACKET or BAD
PACKET
DISCARD states of the packet receiver state machine)
•
Data packet errors
•
Link packet errors
•
Packets discarded due to buffer overrun
323
Mellanox Technologies Confidential
Version 5.2 Supported Port Counters and Events
Description / Message
114 Port
Receive
Remote
Physical
Errors
119 Local
Link
Integrity
Errors
1 0
1 1
Minor
Minor
5
5
300
300
Port
Port
Hardware
Hardware
122 Congested
Bandwidt h (%)
Threshold
Reached
1 1 Minor 10 300 Port Hardware
Total number of packets marked with the EBP delimiter received on the port
The number of times that the frequency of packets containing local physical errors has exceeded
LocalPhyErrors.
Message:
LocalLinkIntegrityEr rors counter threshold exceeded.
Threshold is %d, received value is %d
Percent of Congested
Bandwidth has exceeded defined threshold.
Note: a different threshold can be set specifically for Tier 4 ports.
4X link operates as
1X link
131 Nonoptimal link width
(1X instead of
4X)
132 Nonoptimal link width
(1X or 4X instead of
12X)
1 1
1 1
Minor 1
Minor 1
0 Port Hardware
0 Port Hardware 12X links operates as
4X or 1X link
324
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Description / Message
701 Nonoptimal
Link
Speed
1 1 Minor 1 0 Port Hardware
140 Excessive
Buffer
Overrun
Threshold
Reached
1 0 Minor 1 300 Port Hardware
141 Flow
Control
Update
Watchdog
Timer
Expired
1 0 Warni ng
1 300 Port Hardware
392 Module
Temperat ure
Threshold
Reached
1 0 Info 40 0 Mod ule
Hardware
DDR link operates as
SDR or
QRD link operates as
DDR or QDR or
EDR link operates as
FDR,QDR,DDR or
SDR or
FDR link operates as
QDR,DDR or SDR
SM Trap. This error is detected when the number of consecutive flow control update periods with at least one overrun error in each period exceeds the OverrunErrors threshold given in the
PortInfo attribute.
Message: Excessive
Buffer Overrun
Threshold is reached: lid %(lid)d, port
#%(portn)d
SM Trap. The error indicates a failure of the flow control machine at the other end of the link. If the timer expires without receiving an update, a flow control update error has occurred.
Message: Flow
Control Update watchdog timer has expired: lid %(lid)d, port #%(portn)d
Temperature detected by module sensor is too high, has exceeded the defined threshold.
325
Mellanox Technologies Confidential
Version 5.2 Supported Port Counters and Events
Description / Message
326
350 Environm ent Added
1 0 Info 1
351 Environm ent
Removed
306 Logical
Server
Added
1 0
1 0
Info
Info
1
1
307 Logical
Server
Removed
1 0 Info 1
352 Network
Added
1 0 Info 1
353 Network
Removed
1 0 Info 1
340 Network
Interface
Added
1 0 Info 1
341 Network
Interface
Removed
1 0 Info 1
313 Compute
Resource
Allocated
1 0 Info 1
312 Compute
Resource
Released
1 0 Info 1
317 Logical
Server
Compute
Resource is Up
1 1 Warni ng
1
0
0
Env Logical Model New Logical
Environment is created
Env Logical Model Logical Environment is deleted
0
0
0
0
0
0
0
0
0
Logi cal
Serv er
Logi cal
Serv er
Logi cal
Serv er
Logi cal
Serv er
Logi cal
Serv er
Logi cal
Serv er
Logi cal
Serv er
Net wor k
Net wor k
Logical Model New Logical Server or Logical Servers
Group is created
Logical Model Logical Server or
Logical Servers
Group is deleted
Logical Model New Network is created
Logical Model Network is deleted
Logical Model New Network
Interface is created
Logical Model Network Interface is deleted
Logical Model A resource is allocated to the
Logical Server
Logical Model A resource is released from the
Logical Server
Logical Model An allocated resource is Down or
Disconnected
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Description / Message
316 Logical
Server
Compute
Resource is Down
301 Logical
Server
State
Changed
302 Logical
Server
State
Change
Failed
1 1
1 0
1 0
Critica l
Info
1
1
Minor 1
308 Logical
Server
Resources
Allocated
314 Logical
Server
Additiona l
Resources
Allocated
315 Logical
Server
Resources
Released
336 Port
Action
Succeede d
1 0
1 0
1 0
1 0
Info
Info
Info
Info
1
1
1
1
0 Logi cal
Serv er
Logical Model An allocated resources is Up or
Connected back
0
0
0
0
Logi cal
Serv er
Logi cal
Serv er
Logical Model Logical Server state is changed
Logi cal
Serv er
Logi cal
Serv er
Logical Model Logical Server has failed to change the state.
RM (Resource
Manager) Event.
Indicates error in
Logical Server state change. This error might be caused by any error condition related to the Logical
Server resources allocation.
Message: Logical
Server changed state from %s to %s
Logical Model New resources are allocated to the
Logical Server
Logical Model Additional resources are allocated to the
Logical Server
0
0
Logi cal
Serv er
Logical Model
Port Maintenance
Resources were released from the
Logical Server
Port Management
Action (reset, disable) succeeded
327
Mellanox Technologies Confidential
Version 5.2 Supported Port Counters and Events
Description / Message
337 Port
Action
Failed
338 Device
Action
Succeede d
339 Device
Action
Failed
385 Switch
FW
Upgrade
Started
386 Switch
SW
Upgrade
Started
381 Switch
Upgrade
Failed
388 Host FW
Upgrade
Started
389 Host SW
Upgrade
Started
383 Host
Upgrade
Failed
502 Device
Upgrade
Finished
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
Minor 1
Info
Info
Info
Info
Info
Info
Info
Info
1
Minor 1
1
1
1
1
1
1
1
0 Port Maintenance
0 Port Maintenance
Port Management
Action (reset, disable) failed
Device Management
Action succeeded
0 Port Maintenance
0 Swit ch
Maintenance
Device Management
Action failed
Switch FW Upgrade process has started
0 Swit ch
Maintenance Switch SW Upgrade process has started
0 Swit ch
Maintenance
0
0
Com pute r
Com pute r
0 Com pute r
300 Devi ce
Maintenance
Maintenance
Maintenance
Maintenance
Switch SW or FW
Upgrade process failed
Host FW Upgrade process has started
Host SW Upgrade process has started
Host SW or FW
Upgrade process failed
Device SW or FW
Upgrade has finished
328
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Appendix E: Diagnostic Utilities
E.1
Version 5.2
Commands for InfiniBand Diagnostics
Table 81: InfiniBand Diagnostics Commands
Command
ibstat ibstatus ibnetdiscover ibaddr ibroute ibtracert ibping ibsysstat sminfo smpdump smpquery perfquery ibcheckport ibchecknode ibcheckerrs ibchecknet ibswitches ibhosts ibnodes ibportstate ibcheckwidth ibcheckportwidth
Description
Shows the host adapters status
Similar to ibstat but implemented as a script
Scans the topology
Shows the LID range and default GID of the target (default is the local port)
Displays unicast and multicast forwarding tables of the switches.
Displays unicast or multicast route from source to destination.
Uses vendor MADs to validate connectivity between InfiniBand nod es. On exit, (IP) ping-like output is shown.
Obtains basic information for the specific node which may be remote. This information includes: hostname, CPUs, memory utilization.
Queries the SMInfo attribute on a node.
A general purpose SMP utility which gets SM attributes from a specified SMA. The result is dumped in hex by default.
Enables a basic subset of standard SMP queries including the follow ing: node info, node description, switch info, port info.
Fields are displayed in human readable format.
Dumps (and optionally clears) the performance counters of the destination port (including error counters).
Performs basic tests on the specified port.
Performs basic tests on the specified node
Checks if the error counters of the port/node have exceeded some predefined thresholds.
Performs port/node/errors check on the subnet. ibnetdiscover output can be used as in input topology.
Scans the net or uses existing net topology file and lists all switches.
Scans the net or uses existing net topology file and lists all hosts.
Scans the net or uses existing net topology file and lists all nodes.
Gets the logical and physical port states of an InfiniBand port or disables or enables the port (only on a switch).
Performs port width check on the subnet.
This command is used to find ports with 1x link width.
Performs 1x port width check on specified port.
329
Mellanox Technologies Confidential
Version 5.2
Command
ibcheckstate ibcheckportstate ibcheckerrors ibclearerrors ibdiscover.pl
saquery ibdiagnet ibnetsplit ibclearcounters
Diagnostic Utilities
Description
Performs a port state check on the subnet (as well as a physical port state). This command is used to find ports that are not in LinkUp physical port state and that are not in active port state.
Performs a port state check on the specified port (as well as a physical port state).
Performs an error check on the subnet. This command is used to find ports with error counters (Performance Management Agent
(PMA) PortCounters) beyond the indicated thresholds.
Clears all error counters on the subnet.
Clears all port counters on the subnet.
Takes the ibnetdiscover output and a map file and produces a topology file (local node GUID and port connected to remote node
GUID and port).
Issues SA queries. ibdiagnet scans the fabric using directed route packets and extracts all the available information regarding its connectivity and devices.
Automatically groups hosts and creates scripts that can be run in order to split the network into sub-networks each containing one group of hosts.
NOTE: This list may not include all the current updates. For a comprehensive list of the latest tools, see the infiniband-diags subdirectory under /opt/ufm/opensm/sbin/.
E.2
E.2.1
Diagnostic Tools
Model of operation: All utilities use direct MAD access to operate. Operations that require
QP 0
mads only, may use direct routed mads, and therefore may work even in subnets that are not configured. Almost all utilities can operate without accessing the SM, unless GUID to lid translation is required.
Dependencies
Most utilities depend on libibmad and libibumad.
All utilities depend on the ib_umad kernel module.
Multiple port/Multiple CA support:
When no InfiniBand device or port is specified (as shown in the following example for
"Local umad parameters"), the libibumad library selects the port to use by the following criteria:
1. The first port that is ACTIVE.
2. If not found, the first port that is UP (physical link up).
330
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
If a port and/or CA name is specified, the libibumad library attempts to fulfill the user’s request, and will fail if it is not possible.
For example: ibaddr # use the 'best port' ibaddr -C mthca1 # pick the best port from mthca1 only. ibaddr -P 2 # use the second (active/up) port from the first available IB device. ibaddr -C mthca0 -P 2 # use the specified port only.
Common Options & Flags
Most diagnostics take the following flags. The exact list of supported flags per utility can be found in the usage message and can be shown using util_name -h syntax.
# Debugging flags
-d raise the IB debugging level. May be used several times (-ddd or -d -d
-d).
-e show umad send receive errors (timeouts and others)
-h show the usage message
-v increase the application verbosity level.
May be used several times (-vv or -v -v -v)
-V show the internal version info.
# Addressing flags
-D use directed path address arguments.
The path is a comma separated list of out ports.
Examples:
"0" # self port
"0,1,2,1,4" # out via port 1, then 2, ...
-G use GUID address arguments.
In most cases, it is the Port GUID.
Examples:
"0x08f1040023"
-s <smlid> use 'smlid' as the target lid for SA queries.
# Local umad parameters:
-C <ca_name> use the specified ca_name.
-P <ca_port> use the specified ca_port.
-t <timeout_ms> override the default timeout for the
solicited mads.
CLI notation: all utilities use the POSIX style notation, meaning that all options (flags) must precede all arguments (parameters).
Version 5.2
E.2.2 Utilities Descriptions
ibstatus
A script that displays basic information obtained from the local InfiniBand driver. Output includes LID, SMLID, port state, link width active, and port physical state.
Syntax ibstatus [-h] [devname[:port]]
Examples:
331
Mellanox Technologies Confidential
Version 5.2 Diagnostic Utilities ibstatus # display status of all IB ports ibstatus mthca1 # status of mthca1 ports ibstatus mthca1:1 mthca0:2 # show status of specified ports
See also: ibstat ibstat
Similar to the ibstatus utility but implemented as a binary and not as a script. Includes options to list CAs and/or ports.
Syntax ibstat [-d(ebug) -l(ist_of_cas) -p(ort_list) -s(hort)] <ca_name> [portnum]
Examples: ibstat # display status of all IB ports ibstat mthca1 # status of mthca1 ports ibstat mthca1 2 # show status of specified ports ibstat -p mthca0 # list the port guids of mthca0 ibstat –l # list all CA names
See also: ibstatus ibroute
Uses SMPs to display the forwarding tables (unicast (LinearForwardingTable or LFT) or multicast (MulticastForwardingTable or MFT)) for the specified switch LID and the optional lid (mlid) range. The default range is all valid entries in the range 1...FDBTop.
Syntax ibroute [options] <switch_addr> [<startlid> [<endlid>]]
Non standard flags:
-a show all lids in range, even invalid entries.
-n do not try to resolve destinations.
-M show multicast forwarding tables. In this case the range
parameters are specifying mlid range. node-name-map node name map file
Examples: ibroute 2 # dump all valid entries of switch lid 2 ibroute 2 15 # dump entries in the range 15...FDBTop. ibroute -a 2 10 20 # dump all entries in the range 10..20 ibroute -n 2 # simple format ibroute -M 2 # show multicast tables
See also: ibtracert ibtracert
Uses SMPs to trace the path from a source GID/LID to a destination GID/LID. Each hop along the path is displayed until the destination is reached or a hop does not respond. By using the -m option, multicast path tracing can be performed between source and destination nodes.
Syntax ibtracert [options] <src-addr> <dest-addr>
332
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Non standard flags:
-n simple format; don't show additional information.
-m <mlid> show the multicast trace of the specified mlid.
-f <force> force node-name-map node name map file
Examples: ibtracert 2 23 # show trace between lid 2 and 23 ibtracert -m 0xc000 3 5 # show multicast trace between lid 3 and 5 for mcast lid 0xc000. smpquery
Enables a basic subset of standard SMP queries including the following node info, node description, switch info, port info. Fields are displayed in human readable format.
Syntax smpquery [options] <op> <dest_addr> [op_params]
Currently supported operations and their parameters: nodeinfo <addr> nodedesc <addr> portinfo <addr> [<portnum>] # default port is zero switchinfo <addr> pkeys <addr> [<portnum>] sl2vl <addr> [<portnum>] vlarb <addr> [<portnum>]
GUIDInfo (GI) <addr>
MlnxExtPortInfo (MEPI) <addr> [<portnum>]
Combined (-c) : use Combined route address argument node-name-map : node name map file extended (-x) : use extended speeds
Examples: smpquery nodeinfo 2 # show nodeinfo for lid 2 smpquery portinfo 2 5 # show portinfo for lid 2 port 5 smpdump
A general purpose SMP utility that gets SM attributes from a specified SMA. The result is dumped in hex by default.
Syntax smpdump [options] <dest_addr> <attr> [mod]
Non standard flags:
-s show output as string
Examples: smpdump -D 0,1,2 0x15 2 # port info, port 2 smpdump 3 0x15 2 # port info, lid 3 port 2 ibaddr
Can be used to show the LID and GID addresses of the specified port or the local port by default. This utility can be used as simple address resolver.
Syntax
Version 5.2
333
Mellanox Technologies Confidential
Version 5.2 Diagnostic Utilities ibaddr [options] [<dest_addr>]
Non standard flags: gid_show (-g) : show gid address only lid_show (-l) : show lid range only
Lid_show (-L) : show lid range (in decimal) only
Examples: ibaddr # show local address ibaddr 2 # show address of the specified port lid ibaddr -G 0x8f1040023 # show address of the specified port guid sminfo
Issues and dumps the output of an sminfo query in human readable format. The target SM is the one listed in the local port info or the SM specified by the optional SM LID or by the
SM direct routed path.
CAUTION: Using sminfo for any purpose other than a simple query might result in a malfunction of the target SM.
Syntax sminfo [options] <sm_lid|sm_dr_path> [sminfo_modifier]
Non standard flags:
-s <state> # use the specified state in sminfo mad
-p <priority> # use the specified priority in sminfo mad
-a <activity> # use the specified activity in sminfo mad
Examples: sminfo # show sminfo of SM listed in local portinfo sminfo 2 # query SM on port lid 2 perfquery
Uses PerfMgt GMPs to obtain the PortCounters (basic performance and error counters) from the Performance Management Agent (PMA) at the node specified. Optionally show aggregated counters for all ports of node. Also, optionally, reset after read, or only reset counters.
Syntax perfquery [options] [<lid|guid> [[port] [reset_mask]]]
Non standard flags:
-a
-r
-R
Extended (-x)
Xmtsl (-X)
Rcvsl ,( -S)
Xmtdisc (-D) rcverr, (-E) extended_speeds (-T)
Shows aggregated counters for all ports of the destination lid.
Resets counters after read.
Resets only counters.
Shows extended port counters
Shows Xmt SL port counters
Shows Rcv SL port counters
Shows Xmt Discard Details
Shows Rcv Error Details
Shows port extended speeds counters
334
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual oprcvcounters flowctlcounters vloppackets vlopdata vlxmitflowctlerrors vlxmitcounters swportvlcong rcvcc slrcvfecn slrcvbecn xmitcc vlxmittimecc smplctl (-c) loop_ports (-l)
Shows Rcv Counters per Op code
Shows flow control counters
Shows packets received per Op code per VL
Shows data received per Op code per
VL
Shows flow control update errors per
VL
Shows ticks waiting to transmit counters per VL
Shows sw port VL congestion
Shows Rcv congestion control counters
Shows SL Rcv FECN counters
Shows SL Rcv BECN counters
Shows Xmit congestion control counters
Shows VL Xmit Time congestion control counters
Shows samples control
Iterates through each port
Examples: perfquery # read local port's performance counters perfquery 32 1 # read performance counters from lid 32, port 1 perfquery -a 32 # read from lid 32 aggregated performance counters perfquery -r 32 1 # read performance counters from lid 32 port 1 and reset perfquery -R 32 1 # reset performance counters of lid 32 port 1 only perfquery -R -a 32 # reset performance counters of all lid 32 ports perfquery -R 32 2 0xf000 # reset only non-error counters of lid 32 port 2
Version 5.2 ibping
Uses vendor mads to validate connectivity between InfiniBand nodes. On exit, (IP) ping like output is show. ibping is run as client/server. The default is to run as client. Note also that a default ping server is implemented within the kernel.
Syntax ibping [options] <dest lid|guid>
Non standard flags:
-c <count> stop after count packets
-f flood destination: send packets back to back w/o delay
-o <oui> use specified OUI number to multiplex vendor MADs
-S start in server mode (do not return) ibnetdiscover
Performs InfiniBand subnet discovery and outputs a human readable topology file. GUIDs, node types, and port numbers are displayed as well as port LIDs and node descriptions. All nodes (and links) are displayed (full topology). This utility can also be used to list the current connected nodes. The output is printed to the standard output unless a topology file is specified.
Syntax ibnetdiscover [options] [<topology-filename>]
Non standard flags: l Lists connected nodes
335
Mellanox Technologies Confidential
Version 5.2
H
S g full (-f) show (-s)
Router_list (-R) node-name-map cache load-cache diff diffcheck ports : (-p) max_hops (-m) outstanding_smps (-o)
Diagnostic Utilities
Lists connected HCAs
Lists connected switches
Groups
Shows full information (ports' speed and width, vlcap)
Shows more information
Lists connected routers
Nodes name map file filename to cache ibnetdiscover data to filename of ibnetdiscover cache to load filename of ibnetdiscover cache to diff
Specifies checks to execute for -diff
Obtains a ports report
Reports max hops discovered by the library
Specifies the number of outstanding
SMP's which should be issued during the scan ibhosts
Traces the InfiniBand subnet topology or uses an already saved topology file to extract the
CA nodes.
Syntax ibhosts [-h] [<topology-file>]
Dependencies: ibnetdiscover, ibnetdiscover format ibswitches
Traces the InfiniBand subnet topology or uses an already saved topology file to extract the
InfiniBand switches.
Syntax ibswitches [-h] [<topology-file>]
Dependencies: ibnetdiscover, ibnetdiscover format ibchecknet
Uses a full topology file that was created by ibnetdiscover, scans the Network to validate the connectivity and reports errors (from port counters).
Syntax ibchecknet [-h] [<topology-file>]
Dependencies: ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs ibcheckport
Checks the connectivity and performs some simple sanity checks for the specified port. Port address is lid unless -G option is used to specify a GUID address.
336
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Syntax ibcheckport [-h] [-G] <lid|guid> <port_number>
Example: ibcheckport 2 3 # check lid 2 port 3
Dependencies: smpquery, smpquery output format, ibaddr ibchecknode
Checks connectivity and performs some simple sanity checks for the specified node. Port address is lid unless -G option is used to specify a GUID address.
Syntax ibchecknode [-h] [-G] <lid|guid>
Example: ibchecknode 2 # check node via lid 2
Dependencies: smpquery, smpquery output format, ibaddr ibcheckerrs
Checks specified port (or node) and reports errors that surpassed their predefined threshold.
Port address is LID unless the -G option is used to specify a GUID address. The predefined thresholds can be dumped using the -s option and a user defined threshold_file (using the same format as the dump) can be specified using the -t <file> option.
Syntax ibcheckerrs [-h] [-G] [-t <threshold_file>] [-s(how_thresholds)] <lid|guid>
[<port>]
Examples: ibcheckerrs 2 # check aggregated node counter for lid 2 ibcheckerrs 2 4 # check port counters for lid 2 port 4 ibcheckerrs -t xxx 2 # check node using xxx threshold file
Dependencies: perfquery, perfquery output format, ibaddr
Version 5.2 ibportstate
Enables the port state and port physical state of an InfiniBand port to be queried or a switch port to be disabled or enabled.
Syntax ibportstate [-d(ebug) -e(rr_show) -v(erbose) -D(irect) -G(uid) -s smlid -
V(ersion) -C ca_name -P ca_port -t timeout_ms] <dest dr_path|lid|guid>
<portnum> [<op>]
Supported ops: enable, disable, query, on, off, reset, speed, espeed, fdr10, width, down, arm, active, vls, mtu, lid, smlid, lmc, mkey, mkeylease, mkeyprot
Examples: ibportstate 3 1 disable # by lid ibportstate -G 0x2C9000100D051 1 enable # by guid ibportstate -D 0 1 # by direct route ibcheckwidth
337
Mellanox Technologies Confidential
Version 5.2 Diagnostic Utilities
Uses a full topology file created by ibnetdiscover, scans the Network to validate the active link widths and reports any 1x links.
Syntax ibcheckwidth [-h] [<topology-file>]
Dependencies: ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportwidth ibcheckportwidth
Checks the connectivity and checks the specified port for 1x link width. Port address is LID unless the -G option is used to specify a GUID address.
Syntax ibcheckportwidth [-h] [-G] <lid|guid> <port>
Example: ibcheckportwidth 2 3 # check lid 2 port 3
Dependencies: smpquery, smpquery output format, ibaddr ibcheckstate
Uses a full topology file created by ibnetdiscover, scans the Network to validate the port state and port physical state, and reports any ports which have a port state other than
Active or a port physical state other than LinkUp.
Syntax ibcheckstate [-h] [<topology-file>]
Dependencies: ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportstate ibcheckportstate
Checks the connectivity and checks the specified port for proper port state (Active) and port physical state (LinkUp). Port address is LID unless the -G option is used to specify a GUID address.
Syntax ibcheckportstate [-h] [-G] <lid|guid> <port_number>
Example: ibcheckportstate 2 3 # check lid 2 port 3
Dependencies: smpquery, smpquery output format, ibaddr ibcheckerrors
Uses a full topology file that was created by ibnetdiscover, scans the Network to validate the connectivity and reports errors (from port counters).
Syntax ibnetcheckerrors [-h] [<topology-file>]
338
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Dependencies: ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs
Version 5.2 ibdiscover.pl
Uses a topology file created by ibnetdiscover and a discover.map file which the
Network administrator creates which indicates the nodes to be expected and a ibdiscover.topo file which is the expected connectivity and produces a new connectivity file (discover.topo.new) and outputs the changes to stdout. The
Network administrator can choose to replace the "old" topo file with the new one or perform certain changes.
Syntax
The syntax of the ibdiscover.map file is:
<nodeGUID>|port|"Text for node"|<NodeDescription from ibnetdiscover format>
Example:
8f10400410015|8|"ISR 2004"|# SW-6IB4 Voltaire port 0 lid 5
8f10403960558|2|"HCA 1"|# MT23108 InfiniHost Mellanox
Technologies
The syntax of the old and new topo files (ibdiscover.topo and ibdiscover.topo.new) are:
<LocalPort>|<LocalNodeGUID>|<RemotePort>|<RemoteNodeGUID>
Example:
10|5442ba00003080|1|8f10400410015
These topo files are produced by the ibdiscover.pl tool.
ibnetdiscover | ibdiscover.pl
Dependencies: ibnetdiscover, ibnetdiscover format ibnodes
Uses the current InfiniBand subnet topology or an already saved topology file and extracts the InfiniBand nodes (CAs and switches).
Syntax ibnodes [<topology-file>]
Dependencies: ibnetdiscover, ibnetdiscover format ibclearerrors
Clears the PMA error counters in PortCounters by either walking the InfiniBand subnet topology or using an already saved topology file.
Syntax ibclearerrors [-h] [<topology-file>]
Dependencies: ibnetdiscover, ibnetdiscover format, perfquery ibclearcounters
339
Mellanox Technologies Confidential
Version 5.2 Diagnostic Utilities
Clears the PMA port counters by either walking the InfiniBand subnet topology or using an already saved topology file.
Syntax ibclearcounters [-h] [<topology-file>]
Dependencies: ibnetdiscover, ibnetdiscover format, perfquery saquery
Issues SA queries.
Syntax saquery [-h -d -P -N -L -G -s -g][<name>]
Queries node records by default. d
P
N
L (-L)
G (-G)
S (-S)
G (-g)
L (-l)
O (-O) m( -m) x (-x) c (-c)
S (-S)
I (-I) list (-D) src-to-dst (<src:dst>) sgid-to-dgid (<sgid-dgid>) node-name-map smkey <val> slid <lid> dlid <lid> mild <lid> sgid <gid> dgid <gid> gid <gid> mgid <gid>
Reversible", 'r', 1, NULL"
Enables debugging
Gets PathRecord info
Gets NodeRecord info
Returns just the Lid of the name specified
Returns just the Guid of the name specified
Returns the PortInfoRecords with isSM capability mask bit on
Gets multicast group info
Returns the unique Lid of the name specified
Returns name for the Lid specified
Gets multicast member info (if multicast group specified, list member GIDs only for group specified for example 'saquery -m 0xC000')
Gets LinkRecord info"
Gets the SA's class port info
Gets ServiceRecord info
Gets InformInfoRecord (subscription) info the node desc of the CA's
Gets a PathRecord for <src:dst> where src and dst are either node names or LIDs
Gets a PathRecord for <sgid-dgid> where sgid and dgid are addresses in
IPv6 format
Specifies a node name map file
SA SM_Key value for the query. If non-numeric value (like 'x') is specified then saquery will prompt for a value. Default (when not specified here or in ibdiag.conf) is to use SM_Key == 0 (or
\"untrusted\")
Source LID (PathRecord)
Destination LID (PathRecord)
Multicast LID (MCMemberRecord)
Source GID (IPv6 format)
(PathRecord)
Destination GID (IPv6 format)
(PathRecord)
Port GID (MCMemberRecord)
Multicast GID (MCMemberRecord)
Reversible path (PathRecord)
340
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual numb_path ", 'n', 1, NULL" pkey: P_Key (PathRecord,
MCMemberRecord). qos_class (-Q) sl mtu : (-M) rate (-R) pkt_lifetime qkey (-q) (PathRecord,
MCMemberRecord). tclass (-T) flow_label : (-F) hop_limit : (-H) scope join_state (-J) proxy_join (-X) service_id
Number of paths (PathRecord)
QoS Class (PathRecord)
Service level (PathRecord,
MCMemberRecord)
MTU and selector (PathRecord,
MCMemberRecord)
Rate and selector (PathRecord,
MCMemberRecord)
Packet lifetime and selector
(PathRecord, MCMemberRecord)
If non-numeric value (like 'x') is specified then saquery will prompt for a value.
Traffic Class (PathRecord,
MCMemberRecord)
Flow Label (PathRecord,
MCMemberRecord)
Hop limit (PathRecord,
MCMemberRecord)
Scope (MCMemberRecord)
Join state (MCMemberRecord)
Proxy join (MCMemberRecord)
ServiceID (PathRecord)
Dependencies: OpenSM libvendor, OpenSM libopensm, libibumad
Version 5.2 ibsysstat
Uses vendor mads to validate connectivity between InfiniBand nodes and obtain other information about the InfiniBand node. ibsysstat is run as client/server. The default is to run as client.
Syntax ibsysstat [options] <dest lid|guid> [<op>]
Non standard flags:
Current supported operations:
ping - verify connectivity to server (default)
host - obtain host information from server
cpu - obtain cpu information from server
-o <oui> use specified OUI number to multiplex vendor mads
-S start in server mode (do not return) ibnetsplit
Automatically groups hosts and creates scripts that can be run in order to split the network into sub-networks containing one group of hosts.
Syntax
•
Group: ibnetsplit [-v][-h][-g grp-file] -s <.lst|.net|.topo> <-r head-ports|-d max-dist>
•
Split: ibnetsplit [-v][-h][-g grp-file] -s <.lst|.net|.topo> -o out-dir
•
Cobined:
341
Mellanox Technologies Confidential
Version 5.2 Diagnostic Utilities ibnetsplit [-v][-h][-g grp-file] -s <.lst|.net|.topo> <-r head-ports|-d max-dist> -o out-dir
Usage
•
Grouping:
The grouping is performed if the -r or -d options are provided.
•
If the -r is provided with a file containing group head ports, the algorithm examines the hosts distance from the set of node ports provided in the head-ports file (these are expected to be the ports running standby SM's).
•
If the -d is provided with a maximum distance of the hosts in each group, the algorithm partition the hosts by that distance.
NOTE: This method of analyzation may not be suitable for some topologies.
The results of the identified groups are printed into the file defined by the -g option
(default ibnetsplit.groups) and can be manually edited. For groups where the head port is a switch, the group file uses the FIRST host port as the port to run the isolation script from.
•
Splitting:
•
If the -o flag is included, this algorithm analyzes the MinHop table of the topology and identifies the set of links and switches that may potentially be used for routing each group ports. The cross-switch links between switches of the group to other switches are declared as split-links and the commands to turn them off using Directed
Routes from the original Group Head ports are written into the out-dir provided by the
-o flag.
Both stages require a subnet definition file to be provided by the -s flag. The supported formats for subnet definition are:
•
*.net - for ibnetdiscover
•
*.lst - for opensm-subnet.lst or ibiagnet.lst
•
*.topo - for a topology file
HEAD PORTS FILE
This file is provided by the user and defines the ports by which grouping of the other host ports is defined.
Format:
Each line should contain either the name or the GUID of a single port. For switches the port number shall be 0.
<node-name>/P<port-num>|<PGUID>
GROUPS FILE
This file is generated by the program if the head-ports file is provided to it. Alternatively it can be provided (or edited) by the user if different grouping is desired. The generated script for isolating or connecting the group should be run from the first node in each group.
342
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Format:
Each line may be either:
GROUP: <group name>
<node-name>/P<port-num>|<PGUID>
Version 5.2 ibdiagnet ibdiagnet scans the fabric using directed route packets and extracts all the available information regarding its connectivity and devices.
It then produces the following files in the output directory (see below):
•
"ibdiagnet2.log" - A log file with detailed information.
•
"ibdiagnet2.db_csv" - A dump of the internal tool database.
•
"ibdiagnet2.lst" - A list of all the nodes, ports and links in the fabric.
•
"ibdiagnet2.pm" - A dump of all the nodes PM counters.
•
"ibdiagnet2.mlnx_cntrs" - A dump of all the nodes Mellanox diagnostic counters.
•
"ibdiagnet2.net_dump" - A dump of all the links and their features.
•
"ibdiagnet2.pkey" - A list of all pkeys found in the fabric.
•
"ibdiagnet2.aguid" - A list of all alias GUIDs found in the fabric.
•
"ibdiagnet2.sm" - A dump of all the SM (state and priority) in the fabric.
•
"ibdiagnet2.fdbs" - A dump of unicast forwarding tables of the fabric switches.
•
"ibdiagnet2.mcfdbs" - A dump of multicast forwarding tables of the fabric switches.
•
"ibdiagnet2.slvl" - A dump of SLVL tables of the fabric switches.
•
"ibdiagnet2.nodes_info" - A dump of all the nodes vendor specific general information for nodes who supports it.
•
"ibdiagnet2.plft" - A dump of Private LFT Mapping of the fabric switches.
•
"ibdiagnet2.ar" - A dump of Adaptive Routing configuration of the fabric switches.
•
"ibdiagnet2.vl2vl" - A dump of VL to VL configuration of the fabric switches.
Load plugins from:
/tmp/ibutils2/share/ibdiagnet2.1.1/plugins/
You can specify additional paths to be looked in with "IBDIAGNET_PLUGINS_PATH" env variable.
Plugin Name Result Comment libibdiagnet_cable_diag_plugin-2.1.1 Succeeded Plugin loaded libibdiagnet_phy_diag_plugin-2.1.1 Succeeded Plugin loaded
Syntax
[-i|--device <dev-name>] [-p|--port <port-num>]
[-g|--guid <GUID in hex>] [--skip <stage>]
[--skip_plugin <library name>] [--sc]
[--scr] [--pc] [-P|--counter <<PM>=<value>>]
[--pm_pause_time <seconds>] [--ber_test]
343
Mellanox Technologies Confidential
Version 5.2
[--ber_thresh <value>] [--llr_active_cell <64|128>]
[--extended_speeds <dev-type>] [--pm_per_lane]
[--ls <2.5|5|10|14|25|FDR10|EDR20>]
[--lw <1x|4x|8x|12x>] [--screen_num_errs <num>]
[--smp_window <num>] [--gmp_window <num>]
[--max_hops <max-hops>] [--read_capability <file name>]
[--write_capability <file name>]
[--back_compat_db <version.sub_version>]
[-V|--version] [-h|--help] [-H|--deep_help]
[--virtual] [--mads_timeout <mads-timeout>]
[--mads_retries <mads-retries>] [-m|--map <map-file>]
[--vlr <file>] [-r|--routing] [--r_opt <[vs,][mcast,]>]
[--sa_dump <file>] [-u|--fat_tree]
[--scope <file.guid>] [--exclude_scope <file.guid>]
[-w|--write_topo_file <file name>]
[-t|--topo_file <file>] [--out_ibnl_dir <directory>]
[-o|--output_path <directory>]
Cable Diagnostic (Plugin)
[--get_cable_info] [--cable_info_disconnected]
Phy Diagnostic (Plugin)
[--get_phy_info] [--reset_phy_info]
Diagnostic Utilities
344
Options
-i|--device <dev-name> : Specifies the name of the device of the port
used to connect to the IB fabric (in case
of multiple devices on he local system).
-p|--port <port-num> : Specifies the local device's port number
used to connect to the IB fabric.
-g|--guid <GUID in hex> : Specifies the local port GUID value of the
port used to connect to the IB fabric. If
GUID given is 0 than ibdiagnet displays
a list of possible port GUIDs and waits
for user input.
--skip <stage> : Skip the executions of the given stage.
Applicable skip stages (vs_cap_smp
vs_cap_gmp | links | pm |
speed_width_check | all).
--skip_plugin <library name> : Skip the load of the given library name.
Applicable skip plugins:
(libibdiagnet_cable_diag_plugin-2.1.1 |
libibdiagnet_phy_diag_plugin-2.1.1).
--sc : Provides a report of Mellanox counters
--scr : Reset all the Mellanox counters (if –sc
option selected).
--pc : Reset all the fabric PM counters.
-P|--counter <<PM>=<value>> : If any of the provided PM is greater then
its provided value than print it.
--pm_pause_time <seconds> : Specifies the seconds to wait between
first counters sample and second counters
sample. If seconds given is 0 than no
second counters sample will be done.
(default=1).
--ber_test :Provides a BER test for each port.
Calculate BER for each port and check no
BER value has exceeds the BER threshold.
(default threshold="10^-12").
--ber_thresh <value> :Specifies the threshold value for the
BER test. The reciprocal number of the
BER should be provided. Example: for
10^-12 than value need to be
1000000000000 or 0xe8d4a51000
(10^12).If threshold given is 0 than all
BER values for all ports will be
reported.
--llr_active_cell <64|128> : Specifies the LLR active cell size
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
for BER test, when LLR is active in the
fabric.
--extended_speeds <dev-type> : Collect and test port extended speeds
counters. dev-type: (sw | all).
--pm_per_lane : List all counters per lane (when
available).
--ls <2.5|5|10|14|25|FDR10|EDR20> : Specifies the expected link speed.
--lw <1x|4x|8x|12x> : Specifies the expected link width.
--screen_num_errs <num> : Specifies the threshold for printing
errors to screen. (default=5).
--smp_window <num> : Max smp MADs on wire. (default=8).
--gmp_window <num> : Max gmp MADs on wire. (default=128).
--max_hops <max-hops> : Specifies the maximum hops for the
discovery process. (default=64).
--read_capability <file name> : Specifies capability masks
configuration file, giving capability
mask configuration for the fabric.
ibdiagnet will use this mapping for
Vendor Specific MADs sending.
--write_capability <file name> : Write out an example file for
capability masks configuration,
and also the default capability
masks for some devices.
--back_compat_db <version.sub_version> : Show ports section in
"ibdiagnet2.db_csv" according to
given version. Default version 2.0.
-V|--version : Prints the version of the tool.
-h|--help : Prints help information (without
plugins help if exists).
-H|--deep_help : Prints deep help information
(including plugins help).
--virtual : Discover VPorts during discovery
stage.
--mads_timeout <mads-timeout> : Specifies the timeout (in
milliseconds) for sent and received
mads. (default=500).
--mads_retries <mads-retries> : Specifies the number of retreis for
every timeout mad. (default=2).
-m|--map <map-file> : Specifies mapping file, that maps
node guid to name
(format: 0x[0-9a-fA-F]+ "name").
Maping file can also be specified by
Environment variable
"IBUTILS_NODE_NAME_MAP_FILE_PATH".
--src_lid <src-lid> : source lid
--dest_lid <dest-lid> : destination lid
--dr_path <dr-path> : direct route path
-o|--output_path <directory> : Specifies the directory where the
Output files will be placed.
(default="/var/tmp/ibdiagpath/").
Cable Diagnostic (Plugin)
--get_cable_info : Indicates to query all QSFP cables
for cable information. Cable
information will be stored
in "ibdiagnet2.cables".
--cable_info_disconnected : Get cable info on disconnected
ports.
Phy Diagnostic (Plugin)
--get_phy_info : Indicates to query all ports for phy
information.
--reset_phy_info : Indicates to clear all ports phy
information.
345
Mellanox Technologies Confidential
Version 5.2 Diagnostic Utilities ibdiagpath ibdiagpath scans the fabric using directed route packets and extracts all the available information regarding its connectivity and devices. It then produces the following files in the output directory (see below):
•
"ibdiagnet2.log" - A log file with detailed information.
•
"ibdiagnet2.db_csv" - A dump of the internal tool database.
•
"ibdiagnet2.lst" - A list of all the nodes, ports and links in the fabric.
•
"ibdiagnet2.pm" - A dump of all the nodes PM counters.
•
"ibdiagnet2.mlnx_cntrs" - A dump of all the nodes Mellanox diagnostic counters.
•
"ibdiagnet2.net_dump" - A dump of all the links and their features.
Cable Diagnostic (Plugin):
This plugin performs cable diagnostic. It can collect cable info (vendor, PN, OUI etc..) on each valid QSFP cable, if specified.
It produces the following files in the output directory (see below):
•
"ibdiagnet2.cables" - In case specified to collect cable info, this file will contain all collected cable info.
Phy Diagnostic (Plugin)
This plugin performs phy diagnostic.
Load Plugins from:
/tmp/ibutils2/share/ibdiagnet2.1.1/plugins/
You can specify additional paths to be looked in with "IBDIAGNET_PLUGINS_PATH" env variableLoad plugins from:
Plugin Name Result Comment libibdiagnet_cable_diag_plugin-2.1.1 Succeeded Plugin loaded libibdiagnet_phy_diag_plugin-2.1.1 Succeeded Plugin loaded
Syntax
[-i|--device <dev-name>] [-p|--port <port-num>]
[-g|--guid <GUID in hex>] [--skip <stage>]
[--skip_plugin <library name>] [--sc]
[--scr] [--pc] [-P|--counter <<PM>=<value>>]
[--pm_pause_time <seconds>] [--ber_test]
[--ber_thresh <value>] [--llr_active_cell <64|128>]
[--extended_speeds <dev-type>] [--pm_per_lane]
[--ls <2.5|5|10|14|25|FDR10|EDR20>]
[--lw <1x|4x|8x|12x>] [--screen_num_errs <num>]
[--smp_window <num>] [--gmp_window <num>]
[--max_hops <max-hops>] [--read_capability <file name>]
[--write_capability <file name>]
[--back_compat_db <version.sub_version>]
346
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
[-V|--version] [-h|--help] [-H|--deep_help]
[--virtual] [--mads_timeout <mads-timeout>]
[--mads_retries <mads-retries>] [-m|--map <map-file>]
[--src_lid <src-lid>] [--dest_lid <dest-lid>]
[--dr_path <dr-path>] [-o|--output_path <directory>]
Cable Diagnostic (Plugin)
[--get_cable_info] [--cable_info_disconnected]
Phy Diagnostic (Plugin)
[--get_phy_info] [--reset_phy_info]
Options
Version 5.2
Mellanox Technologies Confidential
347
Version 5.2 Diagnostic Utilities
-i|--device <dev-name>
-p|--port <port-num>
-g|--guid <GUID in hex>
--skip <stage>
:Specifies the name of the device of the port used to connect to the IB fabric (in case of multiple devices on the local system).
:Specifies the local device's port number used to connect to the IB fabric.
:Specifies the local port GUID value of the port used to connect to the IB fabric. If GUID given is
0 than ibdiagnet displays a list of possible port
GUIDs and waits for user input.
:Skip the executions of the given stage. Applicable skip stages: (vs_cap_smp | vs_cap_gmp | links | pm
| speed_width_check | all).
--skip_plugin <library name> :Skip the load of the given library name.
Applicable skip plugins:(libibdiagnet_cable_diag_plugin-2.1.1 | libibdiagnet_phy_diag_plugin-2.1.1).
--sc :Provides a report of Mellanox counters
--scr :Reset all the Mellanox counters (if -sc option selected).
:Reset all the fabric PM counters. --pc
-P|--counter <<PM>=<value>>
--pm_pause_time <seconds>
:If any of the provided PM is greater then its provided value than print it.
:Specifies the seconds to wait between first counters sample and second counters sample. If seconds given is 0 than no second counters sample will be done. (default=1).
--ber_test
--ber_thresh <value>
:Provides a BER test for each port. Calculate BER for each port and check no BER value has exceeds the BER threshold.(default threshold="10^-12").
:Specifies the threshold value for the BER test.
The reciprocal number of the BER should be provided. Example: for 10^-12 than value need to be
1000000000000 or 0xe8d4a51000(10^12).If threshold given is 0 than all BER values for all ports will be reported.
--llr_active_cell <64|128> :Specifies the LLR active cell size for BER test, when LLR is active in the fabric.
--extended_speeds <dev-type> :Collect and test port extended speeds counters. dev-type: (sw | all).
--pm_per_lane
:List all counters per lane
(when available).
:Specifies the expected link speed. --ls
<2.5|5|10|14|25|FDR10|EDR20>
--lw <1x|4x|8x|12x>
--screen_num_errs <num>
:Specifies the expected link width.
:Specifies the threshold for printing errors to screen. (default=5).
:Max smp MADs on wire. (default=8). --smp_window <num>
--gmp_window <num>
--max_hops <max-hops>
:Max gmp MADs on wire. (default=128).
:Specifies the maximum hops for the discovery process.(default=64).
348
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
--read_capability <file name> :Specifies capability masks configuration file, giving capability mask configuration for the fabric. ibdiagnet will use this mapping for Vendor
Specific MADs sending.
--write_capability <file name>
:Write out an example file for capability masks configuration, and also the default capability masks for some devices.
--back_compat_db
<version.sub_version>
-V|--version
-h|--help
-H|--deep_help
:Show ports section in "ibdiagnet2.db_csv" according to given version. Default version 2.0.
:Prints the version of the tool.
:Prints help information (without plugins help if exists).
:Prints deep help information (including plugins help).
--virtual :Discover VPorts during discovery stage.
--mads_timeout <mads-timeout> :Specifies the timeout (in milliseconds) for sent and received mads.(default=500).
--mads_retries <mads-retries> :Specifies the number of retries for every timeout mad.(default=2).
-m|--map <map-file> :Specifies mapping file, that maps node guid to name (format: 0x[0-9a-fA-F]+ "name"). Mapping file can also be specified by environment variable
"IBUTILS_NODE_NAME_MAP_FILE_PATH".
--src_lid <src-lid>
--dest_lid <dest-lid>
:source lid destination lid
--dr_path <dr-path> :direct route path
-o|--output_path <directory> :Specifies the directory where the output files will be placed. (default="/var/tmp/ibdiagpath/").
Cable Diagnostic (Plugin)
--get_cable_info :Indicates to query all QSFP cables for cable information. Cable information will be stored in
"ibdiagnet2.cables".
--cable_info_disconnected
Phy Diagnostic (Plugin)
--get_phy_info
--reset_phy_info
:Get cable info on disconnected ports.
:Indicates to query all ports for phy information.
:Indicates to clear all ports phy information.
-i|--device <dev-name>
-p|--port <port-num>
-g|--guid <GUID in hex>
Specifies the name of the device of the port used to connect to the IB fabric (in case of multiple devices on the local system).
Specifies the local device's port number used to connect to the IB fabric.
Specifies the local port GUID value of the port used to
Version 5.2
349
Mellanox Technologies Confidential
Version 5.2
-i|--device <dev-name>
--skip <stage>
--skip_plugin <library name>
--sc
--scr
--pc
-P|--counter <<PM>=<value>>
--pm_pause_time <seconds>
--ber_test
--ber_thresh <value>
--llr_active_cell <64|128>
350
Mellanox Technologies Confidential
Diagnostic Utilities
Specifies the name of the device of the port used to connect to the IB fabric (in case of multiple devices on the local system). connect to the IB fabric. If
GUID given is 0 than ibdiagnet displays a list of possible port GUIDs and waits for user input.
Skips the executions of the given stage. Applicable skip stages: (vs_cap_smp | vs_cap_gmp | links | pm | speed_width_check | all).
Skips the load of the given library name. Applicable skip plugins:
(libibdiagnet_cable_diag_plug in | libibdiagnet_cable_diag_plugi n-2.1.1).
Provides a report of Mellanox counters
Reset all the Mellanox counters (if -sc option selected).
Resets all the fabric PM counters.
If any of the provided PM is greater then its provided value than print it.
Specifies the seconds to wait between first counters sample and second counters sample.
If seconds given is 0 than no second counters sample will be done(default=1).
Provides a BER test for each port.
Calculates BER for each port and check no BER value has exceeds the BER threshold.
(default threshold="10^-12").
Specifies the threshold value for the BER test. The reciprocal number of the BER should be provided. Example: for 10^-12 than value need to be 1000000000000 or
0xe8d4a51000 (10^12).If threshold given is 0 than all
BER values for all ports will be reported.
Specifies the LLR active cell size for BER test, when LLR is active in the fabric.
Unified Fabric Manager for InfiniBand User Manual
-i|--device <dev-name>
Version 5.2
--extended_speeds <dev-type>
--pm_per_lane
--ls<2.5|5|10|14|25|FDR10|EDR20>
--lw <1x|4x|8x|12x>
--screen_num_errs <num>
Specifies the name of the device of the port used to connect to the IB fabric (in case of multiple devices on the local system).
Collect and test port extended speeds counters. dev-type: (sw | all).
List all counters per lane
(when available)
Specifies the expected link speed.
Specifies the expected link width.
--smp_window <num>
--gmp_window <num>
--max_hops <max-hops>
--read_capability <file name>
--write_capability <file name>
-V|--version
-h|--help
-H|--deep_help
--virtual
Specifies the threshold for printing errors to screen.
(default=5).
Max smp MADs on wire.
(default=8)
Max gmp MADs on wire.
(default=128)
Specifies the maximum hops for the discovery process(default=64).
Specifies capability masks configuration file, giving capability mask configuration for the fabric. ibdiagnet will use this mapping for
Vendor Specific MADs sending.
Write out an example file for capability masks configuration, and also the default capability masks for some devices.
Prints the version of the tool.
Prints help information
(without plugins help if exists).
Prints deep help information
(including plugins help).
351
Mellanox Technologies Confidential
Version 5.2 Diagnostic Utilities
-i|--device <dev-name> Specifies the name of the device of the port used to connect to the IB fabric (in case of multiple devices on the local system).
Discover VPorts during discovery stage.
--mads_timeout <mads-timeout>
Specifies the timeout (in milliseconds) for sent and
--mads_retries <mads-retries>
-m|--map <map-file> received mads(default=500).
Specifies the number of retreis for every timeout mad
(default=2).
Specifies mapping file, that maps node guid to name
(format: 0x[0-9a-fA-F]+
"name").Maping file can also be specified by environment variable
"IBUTILS_NODE_NAME_MAP_FILE_P
--src_lid <src-lid>
--dest_lid <dest-lid>
--dr_path <dr-path>
-o|--output_path <directory>
Cable Diagnostic (Plugin)
--get_cable_info
--cable_info_disconnected
Phy Diagnostic (Plugin)
--get_phy_info
ATH". source lid destination lid direct route path
Specifies the directory where the output files will be placed
(default="/var/tmp/ibdiagpath
/").
Indicates to query all QSFP cables for cable information.
Cable information will be stored in
"ibdiagnet2.cables".
Get cable info on disconnected ports.
Indicates to query all ports for phy information. phy information.
352
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual Version 5.2
Mellanox Technologies Confidential
353
Version 5.2 Device Management Feature Support
Appendix F: Device Management Feature Support
This section describes supported device management features.
F.1 Device Management Feature Support for InfiniBand
The following table describes the management features available on supported devices.
Feature
Discovery
IB L2
Discovery
Advanced
Discovery (IP, hostname,
Hosts: CPU, memory, FW version)
Yes
Yes
Yes
Yes
Ethernet access
Management interface
Provisioning/
Configuration
IB Partitioning
(pkey)
QoS: SL (SM configuration)
QoS: Rate
Limit (SM configuration)
Interface/VIF
Configuration
(IP, hostname, mtu, Bonding)
Yes
Yes
N/A
N/A
N/A
Yes
Yes
Yes
Yes
N/A
Yes
No
Yes
N/A
Yes
Yes
Yes
Yes Yes Yes Yes
Yes No No No
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
No No
Yes
Yes
Yes
Yes
Yes
Yes
Device
Monitoring
Yes
Yes with
UF
M
Host
Age nt
Yes Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
N/A N/A No N/A Yes with
UF
M
Host
Age nt
No
354
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Feature
Version 5.2
Device
Resources:
CPU, Memory,
Disk
No Yes No No No No No
Yes No No No
Yes Yes Yes Yes
Yes with
UF
M
Host
Age nt
No
No No Get device alerts
(Temperature,
PS, Fan)
L1 (Physical
Port) –
Monitoring
L2-3
(Interface/VIF)
–Monitoring
Yes
Yes
No
Yes
Yes
No
No
Yes Yes Yes
No
Congestion
Monitoring per port (enables congestion map)
Congestion
Monitoring per flow
(Advanced
Package)
No
No
Device
Management
Add/remove to/from Rack
Yes
Add/remove to/from Logical
Server
N/A
Yes View/clear
Alarms
SSH terminal to device
Yes
Yes
Yes
Yes
N/A
Yes
Yes
Yes
No
Yes
Yes
Yes
N/A
No
Yes
No
Yes
N/A
Yes
Yes
No No No
Yes Yes Yes
No No No
Yes Yes Yes
N/A
Yes
No
N/A
Yes
N/A
Yes
No No
Yes with
UF
M
Host
Age nt
No
Yes Yes
No
Yes Yes
Yes Yes
Yes
Yes
No
Yes
Yes
355
Mellanox Technologies Confidential
Version 5.2
Feature
Device Management Feature Support
356
Power On
Reboot
Shutdown
No
No
No
No
No
No
No
No
No
No
Yes
(SX360
6 only)
No
No
No
No
No No
No No
No No
Port
Enable/Disable
Firmware
Upgrade (HCA
& switch)
No
No
Yes
Yes
Yes
No
Yes Yes Yes Yes
Yes
(Upon
SW upgrade
–
SX6036 only)
No
No
No No
Yes No No
Yes No
Inband
Firmware
Upgrade (over
InfiniBand connection)
Software
Upgrade
(OFED & switch)
No
No
Protocols
Communicatio n UFM Server
– Device
IB/SN
MP
No
Yes
No
No Yes
(SX360
6 only)
No
IB/UDP
/SSH
IB IB/HTT
P/
SSH
IB
No No
IB IB
Yes Yes
Yes with
UF
M
Host
Age nt
IB,
SSH
,
IPM
I,
UD
P
No
IB
1. For a full list of supported IS5000 switches, see Supported IS5000 Switches
2. QoS Rate Limit (SM configuration): On ConnectX HCAs-only, for hosts.
3. XmitWait counter monitoring requires ConnectX HCAs with firmware version 2.6 and
Yes with
IPM
I
Yes with
IPM
I
No
No
Yes with
IPM
I
No
Yes Yes
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual above.
4. This feature requires that the IP address is configured.
F.2 Mellanox Devices Supported by UFM
Table 82: Mellanox Externally Managed Devices Supported by UFM
Platform
FDR switches
QDR Switches
Device
SX6025
SX6015
SX6005
IS5022/5023/5024/5025
Firmware Version
9.2.6100
9.2.6100
9.2.6100
7.4.3000
Table 83: Mellanox Managed Devices Supported by UFM
Platform
FDR Switches
Long-Haul Systems
QDR switches
InfiniBand - Ethernet
Gateways
Device
SX6012
SX6018
SX6036
SX6506
SX6512
SX6518
SX6536
SX1012
MTX6100
MTX6240
MTX6280
Grid Director™ 4036
Grid Director™ 4036E
Grid Director™ 4200
Grid Director™ 4700
IS5030/IS5035
IS5100/IS5200/IS5300/IS5600
FDR- SX6036G
QDR- BX5020
OS Version
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
MLNX-OS 3.3.4402
3.9.1
3.9.1
3.9.1
3.9.1
FabricIT 1.1.3000
FabricIT 1.1.3000
MLNX-OS 3.3.4402
FabricIT BXM 2.1.3554
Version 5.2
NOTE: UFM already supports future EDR IB switches (EDR Switch detection and displaying of EDR links between switches).
357
Mellanox Technologies Confidential
Version 5.2
Appendix G: Used Ports
The following is the list of ports used by the UFM Server for internal and external communication:
Port
80(tcp), 443(tcp)
694(udp)
3307(tcp)
2222(tcp)
8000(udp)
6306(udp)
8005(udp)
8088(tcp)
8089(tcp)
3308 (tcp)
8888(tcp)
15800(tcp)
8081(tcp), 8082(tcp)
Purpose
Used by WS clients (Apache Web Server)
Used by Heartbeat – communication between UFM Primary and
Standby server
Used for internal UFM Server communication with MySQL process
Used for SSH debug console
Used for Multicast requests – communication with legacy UFM Agents on Mellanox Grid Director DDR switches
Used for Multicast requests – communication with latest UFM Agents
Used as UFM monitoring listening port
Used for communication between the Java client and the Web Server
Used for internal communication between UFM server and
MonitoirngHistoryEngine
Used for communication between MonitoirngHistoryEngine and
MonitoringHistory mysql server.
Used by DRBD – communication between UFM Primary and Standby server
Used for communication with legacy UFM Agents on Mellanox Grid
Director DDR switches
Used for internal communication with Subnet Manager
Used Ports
358
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Appendix H: Routing Chains
The routing chains feature is offering a solution that enables one to configure different parts of the fabric and define a different routing engine to route each of them. The routings are done in a sequence (hence the name "chains") and any node in the fabric that is configured in more than one part is left with the last routing engine updated for it.
H.1 Configuring Routing Chains
The configuration for the routing chains feature consists of the following steps:
1. Define the port groups.
2. Define topologies based on previously defined port groups.
3. Define configuration files for each routing engine.
4. Define routing engine chains over defined topologies.
H.1.1 Defining Port Groups
The basic idea behind the port groups is the ability to divide the fabric into sub-groups and give each group an identifier that can be used to relate to all nodes in this group. The port groups are used to define the participants in each of the routing algorithms.
H.1.1.1 Defining a Port Group Policy File
In order to define a port group policy file, set the parameter ‘pgrp_policy_file’ in the opensm configuration file, as follows:
/opt/ufm/files/conf/opensm/port_groups_policy_file.conf
H.1.1.2 Configuring a Port Group Policy
The port groups policy file details the port groups in the fabric. The policy file should be composed of one or more paragraphs that define a group. Each paragraph should begin with the line ‘port-group’ and end with the line ‘end-port-group’.
For example: port-group
…port group qualifiers… end-port-group
H.1.1.3 Port Group Qualifiers
NOTE: Unlike the port group’s begining and ending which do not require a colon, all qualifiers must end with a colon (‘:’). Also – a colon is a predefined mark that must not be used inside qualifier values. An inclusion of a colon in the name or the use of a port group, will result in the policy’s failure.
Version 5.2
359
Mellanox Technologies Confidential
Version 5.2 Routing Chains
Table 83: Port Group Qualifiers
Parameter
name use
Description
Each group must have a name. Without a name qualifier, the policy fails.
‘use’ is an optional qualifier that one can define in order to describe the usage of this port group (if undefined, an empty string is used as a default).
Example
name: grp1 use: first port group
H.1.1.4 Rule Qualifiers
There are several qualifiers used to describe a rule that determines which ports will be added
to the group. Each port group may contain one or more rules of the rule qualifiers in Table
84 (at least one rule shall be defined for each port group).
Table 84: Rule Qualifiers
Parameter
guid list
Description
Comma separated list of guids to include in the group.
If no specific physical ports were configured, all physical ports of the guid are chosen. However, for each guid, one can detail specific physical ports to be included in the group. This can be done using the following syntax:
•
Specify a specific port in a guid to be chosen port-guid: 0x283@3
•
Specify a specific list of ports in a guid to be chosen port-guid: 0x286@1/5/7
•
Specify a specific range of ports in a guid to be chosen port-guid: 0x289@2-5
•
Specify a list of specific ports and ports ranges in a guid to be chosen port-guid: 0x289@2-5/7/9-13/18
•
Complex rule port-guid: 0x283@5-8/12/14, 0x286,
0x289/6/8/12 port guid range It is possible to configure a range of guids to be chosen to the group. However, while using the range qualifier, it is impossible to detail specific physical ports.
Note: A list of ranges cannot be specified.
The below example is invalid and will cause the policy to fail: port-guid-range: 0x283-0x289, 0x290-
0x295
Example
port-guid: 0x283,
0x286, 0x289 port-guid-range:
0x283-0x289
360
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Parameter
port name port regexp union rule subtract rule
Description
One can configure a list of hostnames as a rule. Hosts with a node description that is built out of these hostnames will be chosen.
Since the node description contains the network card index as well, one might also specify a network card index and a physical port to be chosen. For example, the given configuration will cause only physical port
2 of a host with the node description ‘kuku
HCA-1’ to be chosen. port
and hca_idx parameters are optional. If the port is unspecified, all physical ports are chosen. If hca_idx is unspecified, all card numbers are chosen.
Specifying a hostname is mandatory.
One can configure a list of hostname/port/hca_idx
sets in the same qualifier as follows: port-name: hostname=kuku; port=2; hca_idx=1 , hostname=host1; port=3, hostname=host2
Note: port-name qualifier is not relevant for switches, but for HCA’s only.
Example
port-name: hostname=kuku; port=2; hca_idx=1
One can define a regular expression so that only nodes with a matching node description will be chosen to the group port-regexp: SW.*
It is possible to specify one physical port to be chosen for matching nodes (there is no option to define a list or a range of ports).
The given example will cause only nodes that match physical port 3 to be added to the group. port-regexp: SW.*:3
It is possible to define a rule that unites two different port groups. This means that all ports from both groups will be included in the united group. union-rule: grp1, grp2
One can define a rule that subtracts one port group from another. The given rule, for example, will cause all the ports which are a part of grp1, but not included in grp2, to be chosen.
In subtraction (unlike union), the order does matter, since the purpose is to subtract the second group from the first one.
There is no option to define more than two groups for union/subtraction. However, one can unite/subtract groups which are a union or a subtraction themselves, as shown in the port groups policy file example. subtract-rule: grp1, grp2
Version 5.2
361
Mellanox Technologies Confidential
Version 5.2 Routing Chains
H.1.1.5 Predefined Port Groups
There are 3 predefined port groups that are available for use, yet cannot be defined in the policy file (if a group in the policy is configured with the name of one of these predefined groups, the policy fails) –
•
ALL – a group that includes all nodes in the fabric
•
ALL_SWITCHES – a group that includes all switches in the fabric.
•
ALL_CAS – a group that includes all HCA’s in the fabric.
H.1.1.6 Port Groups Policy Examples
port-group name: grp3 use: Subtract of groups grp1 and grp2 subtract-rule: grp1, grp2 end-port-group port-group name: grp1 port-guid: 0x281, 0x282, 0x283 end-port-group port-group name: grp2 port-guid-range: 0x282-0x286 port-name: hostname=server1 port=1 end-port-group port-group name: grp4 port-name: hostname=kika port=1 hca_idx=1 end-port-group port-group name: grp3 union-rule: grp3, grp4 end-port-group
H.1.2 Defining a Topologies Policy File
In order to define a port group policy file, set the parameter ‘topo_policy_file’ in the opensm configuration file.
/opt/ufm/files/conf/opensm/topo_policy_file.conf
H.1.2.1 Configuring a Topology Policy
The topologies policy file details a list of topologies. The policy file should be composed of one or more paragraphs which define a topology. Each paragraph should begin with the line
‘topology’
and end with the line ‘end-topology’.
For example: topology
…topology qualifiers… end-topology
362
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
H.1.2.2 Topology Qualifiers
NOTE: Unlike topology and end-topology which do not require a colon, all qualifiers must end with a colon (‘:’). Also – a colon is a predefined mark that must not be used inside qualifier values. An inclusion of a column in the qualifier values will result in the policy’s failure.
Version 5.2
All topology qualifiers are mandatory. Absence of any of the below qualifiers will cause the policy parsing to fail.
Table 85: Topology Qualifiers
Parameter
id sw-grp hca-grp
Description
Topology ID.
Legal Values – any positive value.
Must be unique.
Name of the port group that includes all switches and switch ports to be used in this topology.
Name of the port group that includes all HCA’s to be used in this topology.
Example
id: 1 sw-grp: some_switches hca-grp: some_hosts
H.1.3 Configuration File per Routing Engine
Each engine in the routing chain can be provided by its own configuration file. Routing engine configuration file is the fraction of parameters defined in the main opensm configuration file.
Some rules should be applied when defining a particular configuration file for a routing engine:
•
Parameters that are not specified in specific routing engine configuration file are inherited from the main opensm configuration file.
•
The following configuration parameters are taking effect only in the main opensm configuration file:
• qos and qos_* settings like (vl_arb, sl2vl, etc.)
• lmc
• routing_engine
H.1.3.1 Defining a Routing Chain Policy File
In order to define a port group policy file, set the parameter ‘rch_policy_file’ in the opensm configuration file, as follows:
/opt/ufm/files/conf/opensm/routing_chains_policy.conf
H.1.3.2 First Routing Engine in the Chain
The first unicast engine in a routing chain must include all switches and HCA’s in the fabric
(topology id must be 0). The path-bit parameter value is path-bit 0 and it cannot be changed.
363
Mellanox Technologies Confidential
Version 5.2 Routing Chains
H.1.3.3 Configuring a Routing Chains Policy
The routing chains policy file details the routing engines (and their fallback engines) used for the fabric’s routing. The policy file should be composed of one or more paragraphs which defines an engine (or a fallback engine). Each paragraph should begin with the line
‘unicast-step’ and end with the line ‘end-unicast-step’.
For example: unicast-step
…routing engine qualifiers… end-unicast-step
H.1.3.4 Routing Engine Qualifiers
NOTE: Unlike unicast-step and end-unicast-step which do not require a colon, all qualifiers must end with a colon (‘:’). Also – a colon is a predefined mark that must not be used inside qualifier values. An inclusion of a colon in the qualifier values will result in the policy’s failure.
364
Table 86: Routing Engine Qualifiers
Parameter
id engine use config topology
Description
‘id’ is mandatory. Without an id qualifier for each engine, the policy fails.
•
Legal values – size_t value (0 is illegal).
•
The engines in the policy chain are set according to an ascending id order, so it is highly crucial to verify that the id that is given to the engines match the order in which you would like the engines to be set.
This is a mandatory qualifier that describes the routing algorithm used within this unicast step.
Currently, on the first phase of routing chains, legal values are minhop/ftree/updn.
This is an optional qualifier that enables one to describe the usage of this unicast step. If undefined, an empty string is used as a default.
This is an optional qualifier that enables one to define a separate opensm config file for a specific unicast step. If undefined, all parameters are taken from main opensm configuration file.
Define the topology that this engine uses.
•
Legal value – id of an existing topology that is defined in topologies policy (or zero that represents the entire fabric and not a specific topology).
•
Default value – If unspecified, a routing engine will relate to the entire fabric (as if topology zero was defined).
•
Notice: The first routing engine (the engine with the lowest id) MUST be configured with
Example
is: 1 engine: minhop use: ftree routing for cluster 1 config:
/etc/config/opens m2.cfg
topology: 1
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Parameter
fallback-to path-bit
Description
topology: 0 (entire fabric) or else, the routing chain algorithm will fail.
Example
This is an optional qualifier that enables one to define the current unicast step as a fallback to another unicast step. This can be done by defining the id of the unicast step that this step is a fallback to.
•
If undefined, the current unicast step is not a fallback.
•
If the value of this qualifier is a non-existent engine id, this step will be ignored.
•
A fallback step is meaningless if the step it is a fallback to did not fail.
•
It is impossible to define a fallback to a fallback step (such definition will be ignored)
This is an optional qualifier that enables one to define a specific lid offset to be used by the current unicast step. Setting lmc > 0 in main opensm configuration file is a prerequisite for assigning specific path-bit for the routing engine.
Default value is 0 (if path-bit is not specified)
-
Path-bit: 1
Version 5.2
H.1.3.5 Dump Files per Routing Engine
Each routing engine on the chain will dump its own data files if the appropriate log_flags is set (for instance 0x43).
•
The files that are dumped by each engine are:
• opensm-lid-matrix.dump
• opensm-lfts.dump
• opensm.fdbs
• opensm-subnet.lst
These files should contain the relevant data for each engine topology.
NOTE: sl2vl and mcfdbs files are dumped only once for the entire fabric and NOT by every routing engine.
•
Each engine concatenates its ID and routing algorithm name in its dump files names, as follows:
• opensm-lid-matrix.2.minhop.dump
• opensm.fdbs.3.ftree
• opensm-subnet.4.updn.lst
•
In case that a fallback routing engine is used, both the routing engine that failed and the fallback engine that replaces it, dump their data.
365
Mellanox Technologies Confidential
Version 5.2 Routing Chains
If, for example, engine 2 runs ftree and it has a fallback engine with 3 as its id that runs minhop, one should expect to find 2 sets of dump files, one for each engine:
• opensm-lid-matrix.2.ftree.dump
• opensm-lid-matrix.3.minhop.dump
• opensm.fdbs.2.ftree
• opensm.fdbs.3.munhop
366
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Appendix I: Differences Between UFM Licenses
UFM Licenses include three types:
•
Standard License
•
Advanced License
•
Evaluation License
Both UFM Standard and Advanced are permanent licenses.
UFM Evaluation license, however, is a trial version of the software which offers the user the opportunity to evaluate the product throughout a limited amount of time.
To view the license type of UFM:
Step 4: Click “About” in the main pane.
The license type in the About window is listed under Key Details.
Version 5.2
I.1 UFM Advanced Features
The following features are available only with UFM Advanced License:
•
Monitoring History
•
Monitoring Templates
•
SNMP Traps to 3rd Party Systems
•
Scriptable Alert Actions
•
Multicast Routing Optimizations
•
User Authorization Management
•
Extensible Model
•
Multi-Site Monitoring Portal
•
Software Development Kit
Mellanox Technologies Confidential
367
Version 5.2 Adaptive Routing
Appendix J: Adaptive Routing
Adaptive Routing (AR) enables the switch to select the output port based on the port's load.
AR supports two routing modes:
•
Free AR: No constraints on output port selection.
•
Bounded AR: The switch does not change the output port during the same transmission burst. This mode minimizes the appearance of out-of-order packets.
Adaptive Routing Manager enables and configures Adaptive Routing mechanism on fabric switches. It scans all the fabric switches, deduces which switches support Adaptive Routing and configures the AR functionality on these switches.
Adaptive Routing Manager supports three algorithms: LAG, TREE, and DF_PLUS. The
Adaptive Routing Manager configures AR groups and AR LFTs tables to allow switches to select output port out of an AR group for a specific destination LID.
The configuration of the AR groups depends on the chosen algorithm:
•
LAG: All ports that are linked to the same remote switch are in the same AR group. This algorithm suits any topology with several links between switches and especially 3D torus/mesh, where there are several link in each direction of the X/Y/Z axis.
•
TREE: All ports with minimal hops to destination are in the same AR group. This algorithm suits tree topologies like fat tree, quasi fat tree, parallel links fat tree etc. This algorithm must run together with UPDN routing engine.
•
DF_PLUS: This is an experimental algorithm designed for Dragonfly plus topology.
NOTE: If certain switches do not support AR, they might get timeouts on the AR-related queries and slightly slow down the AR Manager.
Disable AR configuration on switches that do not support AR by using the AR configuration file as explained below.
J.1
J.2
368
Pre-requisites
The minimum AR requirements are:
•
UFM® 5.0 or later
•
MLNX-OS ® 3.4 3050 or later
•
SwitchIB™ firmware v11.0200.0120
•
MLNX _OFED v2.4-x.x.x or later
•
MXM v3.2 or later
Installing the Adaptive Routing
Adaptive Routing Manager is a Subnet Manager plug-in, i.e. it is a shared library
(libarmgr.so) that is dynamically loaded by the Subnet Manager. Adaptive Routing Manager is installed as a part of Mellanox UFM installation.
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
J.3 Running Subnet Manager with Adaptive Routing Manager
Adaptive Routing (AR) Manager can be loaded/unloaded through the gv.cfg file and enabled/disabled through AR manager configuration file.
J.3.1
J.3.2
Version 5.2
Enabling Adaptive Routing
To enable Adaptive Routing:
1. Add
'armgr' to the gv.cfg file of UFM. event_plugin_name = osmufmpi armgr event_plugin_options = armgr --conf_file /opt/ufm/conf/opensm/ar_mgr.conf
2. Disable use_ucast_cache option in /opt/ufm/conf/opensm/opensm.conf. use_ucast_cache FALSE
3. Restart UFM.
Disabling Adaptive Routing
There are two ways to disable Adaptive Routing Manager:
1. Disable it explicitly in the Adaptive Routing configuration file. An HUP signal to
OpenSM is required to apply the configuration.
2. Enable use_ucast_cache option in /opt/ufm/conf/opensm/opensm.conf. use_ucast_cache TRUE
3. Restart UFM.
OR
1. Remove Adaptive Routing definitions from the gv.cfg file of UFM, so that the lines will appear as follows. event_plugin_name = osmufmpi event_plugin_options = (null)
2. Enable use_ucast_cache option in /opt/ufm/conf/opensm/opensm.conf. use_ucast_cache TRUE
3. Restart UFM.
NOTE: Adaptive Routing mechanism is automatically disabled once the switch receives setting of the usual linear routing table (LFT). Therefore, no action is required to clear
Adaptive Routing configuration on the switches if you do not wish to use Adaptive
Routing.
J.4 Querying Adaptive Routing Tables
When Adaptive Routing is active, the content of the usual Linear Forwarding Routing Table on the switch is invalid, thus the standard tools that query LFT (e.g. smpquery, dump_lfts.sh, and others) cannot be used. To query the switch for the content of its Adaptive Routing table, use the 'smparquery' tool that is installed as a part of the UFM package. To see its usage details, run 'smparquery -h'.
369
Mellanox Technologies Confidential
Version 5.2
J.5
Adaptive Routing
Adaptive Routing Manager Options File
The default AR Manager options file is located at /opt/ufm/files/conf/opensm/ar_mgr.conf.
AR Manager options file contains two types of parameters:
1. General options - Options which describe the AR Manager behavior and the AR parameters that will be applied to all the switches in the fabric.
2. Per-switch options - Options which describe specific switch behavior.
Note the following:
•
Adaptive Routing configuration file is case sensitive.
•
You can specify options for nonexisting switch GUID. These options will be ignored until a switch with a matching GUID will be added to the fabric.
•
Adaptive Routing configuration file is parsed every AR Manager cycle, which in turn is executed at every heavy sweep of the Subnet Manager.
•
If the AR Manager fails to parse the options file, default settings for all the options will be used.
J.5.1 General AR Manager Options
Table 87: General AR Manager Options
Option File
ENABLE:
<true|false>
EN_SL_MASK
Description
Enable/disable Adaptive Routing on fabric switches.
Note that if a switch was identified by AR Man- ager as device that does not support AR, AR
Manager will not try to enable AR on this switch. If the firmware of this switch was updated to support the AR, the AR Manager will need to be restarted (by restarting Subnet Man- ager) to allow it to configure the AR on this switch.
This option can be changed on-the-fly.
Bitmask of SLs on which the AR will be enabled (VL if configured VL as SL)
Values
Default: true
< 0x0000 -
0xFFFF >
Default: 0xFFFE
DISABLE_TR_MASK
(experimental)
Bitmask of disabled transport types
# Bit 0= UD
# Bit 1= RC
# Bit 2= UC
# Bit 3= DCT
<0x0 - 0xF>
Default: 0x0
370
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
Option File
AR_ALGORITHM:
< LAG | TREE |
DF_PLUS >
OP_MODE
AR_MODE:
<bounded|free>
AGEING_TIME:
<usec>
MAX_ERRORS:
<N>
ERROR_WINDOW
: <N>
LOG_FILE: <full path>
Description
•
Adaptive Routing algorithm:
•
LAG: Ports groups are created out of
"parallel" links. Links that are connecting the same pair of switches.
•
TREE: All the ports with minimal hops to destination are in the same group. Must run together with UPDN routing engine.
•
DF_PLUS: experimental algorithm designed for Dragonfly plus topology.
Values
-
Operation mode bitmask that controls the algorithm behavior.
Bit 0 - enable spine to spine none DF routing (applicable only if
DF_PLUS algorithm is selected).
Adaptive Routing Mode:
• free: no constraints on output port selection
• bounded: the switch does not change the output port during the same transmission burst. This mode minimizes the appearance of out-of-order packets
This option can be changed on-the-fly.
Default : 0
Default: bounded
Applicable to bounded AR mode only. Specifies the amount of time, without traffic, that must pass before the switch may declare a transmission burst as finished (32-bit value).This option can be changed on-the-fly.
Deprecated
Default: 30
---
LOG_SIZE: <size in
MB>
AR Manager log file:
/opt/ufm/files/log/ar_mgr.log
This option can not be changed on-the-fly.
Default: /var/log/ armgr.log
This option defines maximal AR Manager log file size in MB. The logfile will be truncated and restarted upon reaching this limit.
This option cannot be changed on-the-fly.
0: unlimited log file size.
Default: 5
Version 5.2
J.5.2 Per-switch AR Options
A user can provide per-switch configuration options with the following syntax:
SWITCH <GUID> {
<switch option 1>;
<switch option 2>;
...
371
Mellanox Technologies Confidential
Version 5.2 Adaptive Routing
}
The following are the per-switch options:
Table 88: Per-switch AR Options
Option File
ENABLE:
<true|false>
AGEING_TIME:
<usec>
Description
Allows you to enable/disable the AR on this switch. If the general ENABLE option value is set to 'false', then this per-switch option is ignored.
This option can be changed on the fly.
Applicable to bounded AR mode only. Specifies the amount of time, without traffic, that must pass before the switch may declare a transmission burst as finished (32-bit value). In the pre-switch options file this option refers to the particular switch only
This option can be changed on-the-fly.
Values
Default: true
Default: 30
372
Mellanox Technologies Confidential
Unified Fabric Manager for InfiniBand User Manual
J.5.3 Adaptive Routing Manager Options File Example
# -----
# Adaptive Routing configuration file
# -----
# General configuration options
# Enable / Disable Adaptive Routing on fabric switches
ENABLE: TRUE; # Values: < TRUE | FALSE>
# Adaptive Routing mode
AR_MODE: FREE; #BOUNDED / FREE
# Bitmask of enabled SLs (VL if configured VL as SL)
EN_SL_MASK: 0xFFFE; # Value: < 0x0000 - 0xFFFF >
# Bitmask of disabled transport types
# Bit 0= UD
# Bit 1= RC
# Bit 2= UC
# Bit 3= DCT
DISABLE_TR_MASK: 0x0; # Value: <0x0 - 0xF> Default: 0x0
# Transmission burst ageing time (usec)
AGEING_TIME: 55;
# Specify log file name - This option can not be changed on the fly
LOG_FILE: /opt/ufm/files/log/ar_mgr.log;
# Specify log size - This option can not be changed on the fly
LOG_SIZE: 70; # In MB
AR_ALGORITHM: TREE; # Values: < TREE | DF_PLUS | LAG >
#OP_MODE: 0x1; # enable spine to spine none DF routing
# Switch specific configuration options
#SWITCH 0x11111 {
# ENABLE: true;
# AGEING_TIME: 77;
#}
#SWITCH 0x22222 {
# ENABLE: false;
#}
J.6
Version 5.2
Adaptive Routing Validation with ibdiagnet
Following a successful deployment of adaptive routing configuration by OpenSM to the IB fabric, it is possible to run ibdiagnet to validate routing correctness as follows:
/opt/ufm/opensm/bin/ibdiagnet -r --r_opt=vs
File version: 1 dump_ar: Switch 0xf452140300301100 en_sl: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15
Groups Definition:
0: 17, 18
1: 1, 2, 3, 4, 5, 6, 7, 8
2: 9, 10, 11, 12, 13, 14, 15, 16
3: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
LFT Definition:
PLFT_NUM: 0
LID : static port : AR group
0x0001 : 021 : UNASSIGNED
373
Mellanox Technologies Confidential
Version 5.2
0x0002 : 000 : UNASSIGNED
0x0003 : 018 : 000
0x0004 : 017 : 000
0x0005 : 018 : 000
0x0006 : 017 : 000
0x0007 : 018 : 000
0x0008 : 018 : 000
0x0009 : 017 : 000
0x000a : 018 : 000
0x000b : 017 : 000
0x000c : 010 : 003
0x000d : 008 : 003
0x000e : 015 : 003
Adaptive Routing
374
Mellanox Technologies Confidential
Download
Public link updated
The public link to your chat has been updated. You may view and manage your chat links after registration here.
Document public link
Advertisement
Key features
Centralized Fabric Management
In-depth Fabric Visibility
Automatic Resource Provisioning
Seamless Failover Handling
Open Architecture
Fabric Health Monitoring
Event Management
Quality of Service Control
Security Features
Frequently asked questions
Unified Fabric Manager (UFM) is a software product designed to manage and optimize InfiniBand fabrics. It provides a central console for configuring, monitoring, and troubleshooting InfiniBand networks.
UFM offers several benefits including centralized management, in-depth fabric visibility, service-oriented automatic resource provisioning, quick resolution of fabric problems, seamless failover handling, and an open architecture.
The main functionalities of UFM include a Fabric Dashboard, Fabric Design and Configuration, Fabric Discovery and Physical View, Central Device Management, Monitoring, Configuration, Fabric Health, Logging, and High Availability.
