11

I/O Fabric Management

The chassis can hold up to six I/O modules (IOMs), each of which can be pass-through or switch modules.

The IOMs are classified into three groups—A, B, and C. Each group has two slots—Slot 1 and Slot 2. The slots are designated with letters, from left to right, across the back of the chassis: A1 | B1 | C1 | C2 | B2 | A2. Each server has slots for two mezzanine cards (MCs) to connect to the IOMs. The MC and the corresponding IOM must have the same fabric.

Chassis IO is segregated into 3 discrete data paths by letter: A, B and C.

These paths are described as “FABRICS” and support Ethernet, Fibre

Channel or InfiniBand. These discrete fabric paths are split into 2 IO

“Banks”, bank one and two. Each server IO adapter (Mezzanine Card or

LOM) can have either 2 or 4 ports depending on capability. These ports are split out evenly to IOM banks one and two to allow for redundancy. When you deploy your Ethernet, iSCSI or FibreChannel networks, span their redundant links across banks one and two for maximum availability. We denote the discrete IOM with the fabric Identifier and the Bank number.

Example: “A1” denotes Fabric “A” in bank “1”. “C2” Denotes Fabric “C” in

Bank “2”.

The chassis supports three fabric or protocol types. The IOMs and Mezzanine

Cards in a group must have the same or compatible fabric types.

• Group A IOMS are always connected to the servers' on-board Ethernet adapters; the fabric type of Group A is always Ethernet.

• For Group B, the IOM slots are permanently connected to the first MC

(mezzanine card) slot in each server module.

• For Group C , the IOM slots are permanently connected to the second MC

(mezzanine card) in each server module.

NOTE: In the CMC CLI, IOMs are referred to by the convention, switchn :

A1=switch-1, A2=switch-2, B1=switch-3, B2=switch-4, C1=switch-5, and C2= switch-6.

I/O Fabric Management 361

Fabric Management

Fabric management helps avoid electrical, configuration, or connectivity related problems due to installation of an IOM or MC that has an incompatible fabric type from the chassis' established fabric type. Invalid hardware configurations could cause electric or functional problems to the chassis or its components. Fabric management prevents invalid configurations from powering on.

Figure 11-1 shows the location of IOMs in the chassis. The location of each

IOM is indicated by its group number (A, B, or C).These discrete fabric paths are split into two IO Banks, bank one and two. On the chassis, the IOM slot names are marked A1, A2, B1, B2, C1, and C2.

Figure 11-1. Rear View of a Chassis, Showing the Location of the IOMs

1 2

362

1 Bank 1 (Slots A1, B1, C1)

I/O Fabric Management

2 Bank 2 (Slots A2, B2, C2)

CMC creates entries in both the hardware log and CMC logs for invalid hardware configurations.

For example:

• An Ethernet MC connected to a Fibre Channel IOM is an invalid configuration. However, an Ethernet MC connected to both an Ethernet switch and an Ethernet pass-through IOM installed in the same IOM group is a valid connection.

• A Fibre Channel pass-through IOM and a fibre channel switch IOM in slots B1 and B2 is a valid configuration if the first MCs on all of the servers are also fibre channel. In this case, CMC powers-on the IOMs and the servers. However, certain fibre channel redundancy software may not support this configuration; not all valid configurations are necessarily supported configurations.

NOTE: Fabric verification for server IOMs and MCs is performed only when the chassis is powered on. When the chassis is on standby power, the iDRACs on the server modules remain powered off and thus are unable to report the server's MC fabric type. The MC fabric type may not be reported in the CMC user interface until the iDRAC on the server is powered on. Additionally, if the chassis is powered on, fabric verification is performed when a server or IOM is inserted (optional). If a fabric mismatch is detected, the server or IOM is allowed to power on and the status LED flashes Amber.

Invalid Configurations

There are three types of invalid configurations:

• Invalid MC or LOM configuration, where a newly installed servers’s fabric type is different from the existing IOM fabric

• Invalid IOM-MC configuration, where a newly installed IOM’s fabric type and the resident MC’s fabric types do not match or are incompatible

• Invalid IOM-IOM configuration, where a newly installed IOM has a different or incompatible fabric type from an IOM already installed in its group

I/O Fabric Management 363

Invalid Mezzanine Card (MC) Configuration

An invalid MC configuration occurs when a single server’s LOM or MC is not supported by its corresponding IOM. In this case, all the other servers in the chassis can be running, but the server with the mismatched MC card does not be allowed to power on. The power button on the server flashes Amber to alert a fabric mismatch. For information about CMC and hardware logs, see

"Viewing the Event Logs" on page 398 .

Invalid IOM-Mezzanine Card (MC) Configuration

The mismatched IOM is held in the power-off state. CMC adds an entry to

CMC and hardware logs noting the invalid configuration and specifying the

IOM name. CMC causes the error LED on the offending IOM to blink. If

CMC is configured to send alerts, it sends e-mail and/or SNMP alerts for this

event. For information about CMC and hardware logs, see "Viewing the

Event Logs" on page 398.

Invalid IOM-IOM Configuration

CMC holds the newly installed IOM in powered-off state, causes the IOM’s error LED to blink, and creates entries in CMC and hardware logs about the

mismatch. For information about CMC and hardware logs, see "Viewing the

Event Logs" on page 398.

Fresh Power-up Scenario

When the chassis is plugged in and powered up, the I/O modules have priority over the servers. The first IOM in each group is allowed to power up before the others. At this time, no verification of their fabric types is performed. If there is no IOM on the first slot of a group, the module on the second slot of that group powers up. If both slots have IOMs, the module in the second slot is compared for consistency against the one in the first.

After the IOMs power up, the servers power up, and CMC verifies the servers for fabric consistency.

A pass-through module and switch are allowed in the same group as long as their fabric is identical. Switches and pass-through modules can exist in the same group even if they are manufactured by different vendors.

364 I/O Fabric Management

Monitoring IOM Health

The health status for the IOMs can be viewed in two ways: from the Chassis

Graphics section on the Chassis Status page or the I/O Modules Status page.

The Chassis Graphics page provides a graphical overview of the IOMs installed in the chassis.

To view health status of the IOMs using Chassis Graphics:

1 Log in to the CMC Web interface.

2 The Chassis Status page is displayed. The right section of Chassis

Graphics depicts the rear view of the chassis and contains the health status for the IOMs. IOM health status is indicated by the color of the IOM subgraphic:

• Green - IOM is present, powered on and communicating with CMC; there is no indication of an adverse condition.

• Amber - IOM is present, but may or may not be powered on, or may or may not be communicating with CMC; an adverse condition may exist.

• Gray - IOM is present and not powered on. It is not communicating with CMC and there is no indication of an adverse condition.

3 Use the cursor to hover over an individual IOM subgraphic and a corresponding text hint or screen tip is displayed. The text hint provides additional information on that IOM.

4 The IOM subgraphic is hyperlinked to the corresponding CMC GUI page to provide immediate navigation to the I/O Module Status page associated with that IOM.

To view the health status of all IOMs using the I/O Modules Status page:

1 Log in to the CMC Web interface.

2 Select I/O Modules in the Chassis menu in the system tree.

3 Click the Properties tab.

I/O Fabric Management 365

4 Click the Status subtab. The I/O Modules Status page displays.

Table 11-1. I/O Modules Status Information

Item

Slot

Present

Health

Description

Displays the location of the I/O module in the chassis by group number (A, B, or C) and Bank (1 or 2). IOM Enumeration: A1 , A2 ,

B1 , B2 , C1 , or C2 .

Displays whether the IOM is present ( Yes or No ).

OK Indicates that the IOM is present and communicating with CMC. In the event of a communication failure between CMC and the server, CMC cannot obtain or display health status for the IOM.

Informational Displays information about the IOM when no change in health status (OK, Warning, Severe) has occurred.

Warning Indicates that warning alerts have been issued, and c orrective action must be taken.

If corrective actions are not taken, it could lead to critical or severe failures that can affect the integrity of the IOM.

Examples of conditions causing Warnings: IOM fabric mismatch with the server's mezzanine card fabric; invalid IOM configuration, where the newly installed IOM does not match the existing IOM on the same group.

Severe Indicates that at least one Failure alert has been issued. Severe status represents a system failure on the IOM, and corrective action must be taken immediately .

Examples of conditions causing Severe status:

Failure in IOM detected; IOM was removed.

NOTE: Any change in health is logged to both the hardware and CMC

log. For more information, see "Viewing the Event Logs" on page 398.

366 I/O Fabric Management

Table 11-1. I/O Modules Status Information (continued)

Item

Fabric

Description

Displays the type of fabric for the IOM: Gigabit Ethernet, 10GE

XAUI, 10GE KR, 10GE XAUI KR, FC 4 Gbps, FC 8 Gbps, SAS 3

Gbps, SAS 6 Gbps, Infiniband SDR, Infiniband DDR, Infiniband

QDR, PCIe Bypass Generation 1, PCIe Bypass Generation 2.

NOTE: Knowing the fabric types of the IOMs in your chassis is critical in preventing IOM mismatches within the same group. For information

about I/O fabric, see "I/O Fabric Management" on page 361.

Name

Launch IOM

Management

Console

Displays the IOM product name.

If the button is present for a particular I/O module, clicking it launches the IOM management console for that I/O module in a new browser window or tab.

NOTE: This option is only available for the managed switch I/O modules. It is not available for pass-through I/O modules or unmanaged Infiniband switches.

NOTE: If an I/O Module is inaccessible because it is powered off, its LAN interface is disabled, or the module has not been assigned a valid IP address, the Launch IOM GUI option is not displayed for that I/O Module.

NOTE: You are prompted to log in to I/O module management interface.

NOTE: You can configure the I/O module IP address using CMC

GUI, as described in "Configuring Network Settings for an

Individual IOM" on page 370.

Role When linking I/O modules together, the Role displays the I/O

Module stacking membership. Member means the module is part of a stack set. Master indicates the module is a primary access point.

Power Status Displays the power status of the IOM: On , Off , or N/A (Absent).

Service Tag Displays the service tag for the IOM. The service tag is a unique identifier provided by Dell for support and maintenance.

Any change in health is logged to both the hardware and CMC log.

For more information, see "Viewing the Event Logs" on page 398.

NOTE: Pass-throughs do not have service tags. Only switches have service tags.

I/O Fabric Management 367

Viewing the Health Status of an Individual IOM

The I/O Module Status page (separate from the I/O Modules Status page) provides an overview of an individual IOM.

To view the health status of an individual IOM:

1 Log in to the CMC Web interface.

2 Expand I/O Modules in the system tree. All of the IOMs (1–6) appear in the expanded I/O Modules list.

3 Click the IOM you want to view in the I/O Modules list in the system tree.

4 Click the Status subtab. The I/O Modules Status page displays.

Table 11-2. I/O Module Health Status Information

Item

Location

Name

Present

Health

Description

Displays the location of the IOM in the chassis by group number

(A, B, or C) and slot number (1 or 2). Slot names: A1 , A2 , B1 , B2 ,

C1 , or C2 .

Displays name of the IOM.

Displays whether the IOM is Present or Absent .

OK Indicates that the IOM is present and communicating with CMC. In the event of a communication failure between CMC and the server, CMC cannot obtain or display health status for the IOM.

Informational Displays information about the IOM when no change in health status (OK, Warning, Severe) has occurred.

Examples of conditions causing Informational status: the IOM presence was detected; a user requested IOM power cycle.

368 I/O Fabric Management

Table 11-2. I/O Module Health Status Information (continued)

Item

Power Status

Service Tag

Fabric

Description

Warning Indicates that warning alerts have been issued, and c orrective action must be taken . If corrective actions are not taken, it could lead to critical or severe failures that can affect the integrity of the IOM.

Examples of conditions causing Warnings:

IOM fabric mismatch with the server's mezzanine card fabric; invalid IOM configuration, where the newly installed

IOM does not match the existing IOM on the same group.

Severe Indicates that at least one Failure alert has been issued. Severe status represents a system failure on the IOM, and corrective action must be taken immediately .

Examples of conditions causing Severe status:

Failure in IOM detected; IOM was removed.

NOTE: Any change in health is logged to both the hardware and CMC

log. For information on viewing logs, see "Viewing the Hardware Log"

on page 398 and "Viewing the CMC Log" on page 400.

Displays the power status of the IOM: On , Off , or N/A (Absent).

Displays the service tag for the IOM. The service tag is a unique identifier provided by Dell for support and maintenance.

Displays the type of fabric for the IOM: Gigabit Ethernet, 10GE

XAUI, 10GE KR, 10GE XAUI KR, FC 4 Gbps, FC 8 Gbps, SAS 3

Gbps, SAS 6 Gbps, Infiniband SDR, Infiniband DDR, Infiniband

QDR, PCIe Bypass Generation 1, PCIe Bypass Generation 2.

NOTE: Knowing the fabric types of the IOMs in your chassis is critical in preventing IOM mismatches within the same group. For information

about I/O fabric, see "I/O Fabric Management" on page 361.

I/O Fabric Management 369

Table 11-2. I/O Module Health Status Information (continued)

Item

MAC Address Displays the MAC address for the IOM. The MAC address is a unique address assigned to a device by the hardware vendor as a means for identification.

NOTE: Pass-throughs do not have MAC addresses. Only switches have MAC addresses.

Role

Description

Displays the I/O module stacking membership when modules are linked together:

• Member - the module is part of a stack set

• Master - the module is a primary access point.

Configuring Network Settings for an Individual IOM

The I/O Modules Setup page allows you to specify the network settings for the interface used to manage the IOM. For Ethernet switches, the out-of-band management port (IP address) is what is configured. The in-band management port (that is, VLAN1) is not configured using this interface.

NOTE: To change settings on the I/O Modules Configuration page, you must have

Fabric A Administrator privileges to configure IOMs in Group A; Fabric B

Administrator privileges to configure IOMs in Group B; or Fabric C Administrator privileges to configure IOMs in Group C.

NOTE: For Ethernet switches, the in-band (VLAN1) and out-of-band management IP addresses cannot be the same or on the same network; this results in the out-of-band IP address not being set. Refer to the IOM documentation for the default in-band management IP address.

NOTE: Only those IOMs present in the chassis are displayed.

NOTE: Do not configure I/O module network settings for Ethernet pass-through and

Infiniband switches.

370 I/O Fabric Management

To configure the network settings for an individual IOM:

1 Log in to the CMC Web interface.

2 Click I/O Modules in the system tree. Click the Setup subtab. The

Configure I/O Modules Network Settings page displays.

3 To configure network settings for I/O modules, type/select values for the following properties, and then click Apply .

NOTE: Only IOMs that are powered on can be configured.

NOTE: The IP address set on the IOMs from CMC is not saved to the switch's permanent startup configuration. To save the IP address configuration permanently, you must enter the connect switch-n command, or racadm connect switch -n RACADM command, or use a direct interface to the IOM GUI to save this address to the startup configuration file.

Table 11-3. Configure I/O Module Network Settings

Item

Slot

Name

Power State

Description

Displays the location of the IOM in the chassis by group number

(A, B, or C) and slot number (1 or 2). Slot names: A1, A2, B1, B2,

C1, or C2. (The Slot value cannot be changed.)

Displays the IOM product name. (The IOM name cannot be changed.)

Displays the Power State of the IOM. (The Power State cannot be changed from this page.)

I/O Fabric Management 371

Table 11-3. Configure I/O Module Network Settings (continued)

Item Description

DHCP Enabled Enables the IOM on the chassis to request and obtain an

IP address from the Dynamic Host Configuration Protocol (DHCP) server automatically.

Default: Checked (enabled).

If this option is checked, the IOM retrieves IP configuration

(IP address, subnet mask, and gateway) automatically from a

DHCP server on your network.

NOTE: When this feature is enabled, the IP Address, Gateway, and Subnet Mask property fields (located immediately adjacent following this option) are inactivated, and any previously entered values for these properties are ignored.

If this option is not checked, you must manually enter a valid

IP address, gateway, and subnet mask in the corresponding text fields immediately following this option.

IP Address Specifies the IP address for the IOM network interface.

Subnet Mask Specifies the subnet mask for the IOM network interface.

Gateway Specifies the gateway for the IOM network interface.

372 I/O Fabric Management

Troubleshooting IOM Network Settings

The following list contains troubleshooting items for IOM network settings:

• CMC can read the IP address setting too quickly after a configuration change; it displays 0.0.0.0

after clicking Apply. You must hit the refresh button in order to see if the IP address is set correctly on the switch.

• If an error is made in setting the IP/mask/gateway, the switch does not set the IP address and returns a 0.0.0.0

in all fields. Common errors are:

• Setting the out-of-band IP address to be the same as, or on the same network as, the in-band management IP address.

• Entering an invalid subnet mask.

• Setting the default gateway to an address that is not on a network that is directly connected to the switch.

For more information on IOM network settings, refer to the Dell

PowerConnect M6220 Switch Important Information document and the Dell

PowerConnect 6220 Series Port Aggregator White Paper .

I/O Fabric Management 373

374 I/O Fabric Management