Mellanox Technologies MSX6512-NR Hardware Installation Manual

Add to My manuals
48 Pages

advertisement

Mellanox Technologies MSX6512-NR Hardware Installation Manual | Manualzz

216 Port InfiniBand FDR Switch Platform

Hardware Installation Guide

PN: MSX6512-NR, MSX6512-4R

Rev 1.0

www.mellanox.com

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE

PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE

OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE

CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX

TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX

TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE

DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,

INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,

PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS

INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND

RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies, Ltd.

Beit Mellanox

PO Box 586 Yokneam 20692

Israel

Tel: +972 (0)4 909 7200 ;

+972 (0)74 723 7200

Fax: +972 (0)4 959 3245

2

© Copyright 2012.

Mellanox, Mellanox logo, BridgeX, ConnectX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, PhyX, SwitchX, Virtual Protocol

Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd. FabricIT, MLNX-OS, Unbreakable-Link, UFM and

Unified Fabric Manager are trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Mellanox Technologies

Document Number: 3877

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Table of Contents

Rev 1.0

Chapter 1 Chassis Installation 9

1.1 Environmental and Safety Recommendations

1.2 Chassis Package Contents

1.3 Leaf Package Contents

9

10

10

1.4 Spine Package Contents

1.5 Management Package Contents

11

11

1.6 Physical Installation 11

1.6.1

ESD (ElectroStatic Discharge) Connection 11

1.6.2 Installation Procedure

1.7 Installing the Chassis

1.8 Inserting the Leafs

1.9 Inserting a Management Module

1.10 Inserting a Spine Board

12

18

22

23

24

Chapter 2 Installing the Cable Holder

Chapter 3 Ground Connections

25

28

Chapter 4 Power Connections

4.1 Powering Up the Switch Platform

Chapter 5 InfiniBand QSFP Cable Installation

5.1 Port Connector Interfaces

5.2 Supported Approved Cables

5.3 Cable Power Classes

Chapter 6 Chassis Power Up

6.1 Power Supply, Management Board, and Spine Board Indicator Status at Power ON

29

29

31

31

32

32

33

34

Chapter 7 InfiniBand Fabric Initialization and Management

7.1 Configuring the Switch Management Modules for the First Time

7.2 Rerunning the Wizard

Chapter 8 MLNX-OS Management Software

8.1 Starting an SSH Connection to the Switch

8.2 Starting a WebUI Connection to the Switch

Chapter 9 Resetting the Switch – RST

Chapter 10 TroubleShooting

10.1 Power Supply Unit

10.2 Leaf Board

10.3 Management Module

10.3.1 Yellow Status LED (for the Chassis) on the Management Module is Lit

10.3.2 Yellow LED for the Leaf Fan on the Management Module is Lit

10.3.3 Yellow LED for the Spine Fan on the Management Module is Lit

10.4 Spine Board

10.5 MLNX-OS Software

35

35

40

41

41

41

43

44

45

46

46

46

44

44

45

45

Mellanox Technologies 3

4

Rev 1.0

List of Figures

Figure 1: Shock and Tilt Stickers

Figure 2: Shelf Installation Kit Parts

Figure 3: Inserting the Caged Nuts for the Shelf

Figure 4: Use These Holes for the Rail Slides

Figure 5: Connect Rail Slide to Rack Vertical support

Figure 6: Inserting the Caged Nuts for the Faceplate

Figure 7: Chassis Rails and Rail Slides

Figure 8: Screw the Handles Onto the Chassis

Figure 9: The Rails are Already Connected Onto the Chassis

Figure 10: Chassis on the Shelf

Figure 11: Put on the Rail Slide

Figure 12: Face Plate Mounting Bolt Locations

Figure 13: Move the Rail Slide to the Vertical Support

Figure 14: Ejector Latch

Figure 15: Ejector Latch

Figure 16: Spine Board Insertion

Figure 17: Ejector Latch

Figure 18: Cable Holders

Figure 19: Inserting the Caged Nuts for the Cable Holder

Figure 20: Cable Holder Placement

Figure 21: Ground Connection

Figure 22: Multiple Power Inlets - Electric Caution Notification

Figure 23: Spine Module

Figure 24: Port Numbering

Figure 25: Top and Bottom Ports

Figure 26: Spine Side Panel Display Status Indications

Figure 27: Management Module Status Indications for Normal Operation

Figure 28: Connecting the Harness to the Console

Figure 29: Web UI Login Page

Figure 30: Reset Button

32

34

34

35

42

43

28

30

30

32

24

25

26

27

22

23

23

24

19

20

21

21

17

17

18

19

13

14

16

16

Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

List of Tables

Table 1: Revision History of this User’s Manual

Table 2: Reference Documents and Web Sites

Table 3: Serial Terminal Program Configuration

Table 4: Configuration Wizard Session - IP Configuration by DHCP

Table 5: Configuration Wizard Session - IP Zeroconf Configuration

Table 6: Configuration Wizard Session - Static IP Configuration

6

7

36

36

38

39

Rev 1.0

Mellanox Technologies 5

Rev 1.0

Revision History

Table 1 - Revision History of this User’s Manual

Revision

1.0

Date

July2010 Initial release

Details

6 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

About this Manual

Rev 1.0

This manual provides installation and set-up instructions for the SX6512 QSFP Chassis

InfiniBand Switch Platform.

Intended Audience

This manual is intended for users and system administrators responsible for installing and setting up the switch platform.

The manual assumes familiarity with the InfiniBand

®

architecture specification.

Related Documentation

The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the following:

Table 2 - Reference Documents and Web Sites

InfiniBand Architecture Specification, Vol. 1,

Release 1.2.1

Switch Hardware Errata

The InfiniBand Architecture Specification that is provided by

IBTA

For any possible errata due to hardware issues see the switch support product page. This requires a customer support login.

Mellanox MLNX-OS™ SwitchX® Software WebUI

User’s Manual

WebUI Overview for MLNX-OS software.

Mellanox MLNX-OS™ SwitchX® Software User

Manual

This document contains information regarding configuring and managing Mellanox Technologies' SwitchX Switch Platforms.

MLNX-OS™ Software Command Reference Guide

Command Reference Guide for MLNX-OS listing all of the commands available through MLNX-OS with explanations and examples.

MLNX-OS™ Software Configuration Guide

Configuration Guide for MLNX-OS displaying different configuration scenarios.

Document Conventions

Throughout this manual, the name SX6512 and the terms chassis and switch are used to describe the switch, unless explicitly indicated otherwise.

The following pictures are used throughout this document to indicate information that is important to the user.

This symbol makes recommendations to the user.

Mellanox Technologies 7

Rev 1.0

This symbol indicates information that is helpful to the user.

This symbol indicates a situation that can potentially cause damage to hardware or software.

WARNING! This symbol indicates a situation that can potentially cause personal injury and / or damage to hardware or software.

8 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

1 Chassis Installation

Rev 1.0

This chassis can be installed in standard 19” racks that have depths between 65cm

(25.6in.) and 80cm (31.5in.) between the vertical supports of the rack.

Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical, power, and thermal precautions for rack-mounted equipment. Your chassis comes only with the power supplies and fans pre-installed. The rest of the openings are populated with blanks. All of the leafs, spines, and management modules come shipped in separate packages.

The chassis requires initial configuration to get the chassis and Fabric management up and running through remote management. see “Configuring the Switch Management Modules for the First Time” on page 35.

This unit is intended for installation in a Restricted Access Location. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security.

1.1

Environmental and Safety Recommendations

The following are Mellanox recommendations.

Recommended ambient temperature in the System room is 20 o

± 5 o

C.

Recommended humidity ranges is 40% ± 15% without condensing.

It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference.

It is highly recommended that the installation site building be equipped with a lightning rod.

It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system.

Mellanox Technologies 9

Rev 1.0

Chassis Installation

The system requires a KVA rated UPS system. It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure.

Make sure that the outlets and circuits will not be overloaded. Spread out the load over at least two or three circuits or use a 3 phase circuit. N+N grid redundancy requires 2 independent power supply sources.

1.2

Chassis Package Contents

The chassis package includes:

• 1 chassis with the following modules installed:

4 fans

4 PSUs

If you are not using a mechanical lift to install the chassis,reduce the weight of the chassis by removing all of the power supply units, and fan units, and put aside for reinstallation after the chassis is installed in the rack.

• 1 installation kit

• 1 box containing 4 power cords 250v 15a 2.0M, C14 to C13, USA UL Standard

• 1 cable management kit

• 1 Installation Guide

Before you install your new SX6512 chassis, unpack the system and check to make sure that all the parts have been sent, check this against the parts list see “Installation and Cable Management

Kit Parts” on page 12. Check the container for visible damages that may have occurred during shipping.

If anything is damaged or missing, contact your customer representative immediately.

The rack mounting holes conform to the EIA-310 standard for 19-inch racks. Guarantee proper ventilation, by leaving 8cm (3”) of space to the front and rear of the switch.

This will ensure proper air flow through the chassis. This is crucial for maintaining good airflow at ambient temperature. In particular, route cables such that they do not impede the air into or out of the chassis.

1.3

Leaf Package Contents

The leafs are ordered by the customer and are shipped 4 to a box. The customer will receive as many boxes as needed to fill the order.

10 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

1.4

Spine Package Contents

The spines are shipped 3 to a box. The customer will receive as many boxes as needed to complete a full complement of spines.

Rev 1.0

1.5

Management Package Contents

The package includes:

• all of the management modules ordered by the customer

• 1 RJ45 to DB9 harness for each management module received

If you want to hook up to a USB connector you will need to acquire a DB9 to USB adapter.

1.6

Physical Installation

Warning:

This equipment is very heavy. Safety is the first concern. Make sure that proper manpower and equipment are used for transporting and moving the chassis.

The fully loaded chassis weighs:

120 kg (265 lbs) full configuration

48 KG ( 105 lbs) empty configuration

83.94kg (185lbs.) As shipped configuration

The switch platform uses 10U of rack space in a standard 19” rack, 9U for the chassis and

1U for the shelf. The switch ships from the factory with mounting holes on the spine side only. There are upper brackets to connect the leaf side to the rack near the top of the chassis. The weight of the switch is supported from underneath the unit by the shelf.

This chassis can be installed in standard 19” racks that have between 65cm (25.6in.) and

80cm (31.5in.) between the vertical supports of the rack. Make sure that a fully populated rack including cables will have sufficient air flow for cooling.

Choose a rack which is able to support the mechanical and environmental characteristics of a fully populated switch chassis.

1.6.1

ESD (ElectroStatic Discharge) Connection

Before starting any procedure on the SX6512 switch system do the following.

Step 1.

Step 2.

Put an ESD prevention wrist strap on your wrist, and make sure there is good contact between your body and the strap.

Plug the other end of the wrist strap to a valid ground. Make sure that this is a tight fit.

Mellanox Technologies 11

Rev 1.0

1.6.2 Installation Procedure

1.6.2.1

Requirements

You will need:

• #2 phillips screwdriver

• a grounding lug

• #3 phillips screwdriver

• ground wire to properly ground the chassis

The installation will be much easier with a power screwdriver.

Chassis Installation

It is recommended to use AWG6 or 4mm diameter wire for grounding purposes.

It is recommended to have at least two people for the duration of the installation procedure. Use a mechanical lift to raise this chassis. If not, use enough manpower to ensure the safety and wellbeing of all of the people involved in the installation.

1.6.2.2

Installation and Cable Management Kit Parts

Parts for installing the shelf

• 1 shelf

• 16 caged nuts 8 for the shelf and 8 for the faceplate

• 16 M6 bolts for the caged nuts 8 for the shelf and 8 for the faceplate

• 2 Shelf rail slides

Parts for installing the chassis

• 2 Chassis rail slides

• 8 pan head bolts

• 2 Flat 4 hole metal spacers

• 8 split lock washers

Parts for installing the cable manager

• 1 RH cable holder

• 8 caged nuts M6

• 4 Handles

• 8 Allen head screws

• 1 allen wrench

• 1 LH cable holder

• 8 M6 bolts

You will need 10 U of space in the rack. Nine U for the chassis and one U for the shelf.

1.6.2.3

Container Mishandling

The container has shock and tilt stickers applied. These will turn red if the container has been mishandled or roughly handled. Upon receipt of the container look for and inspect the shock and tilt stickers to confirm that they have not tripped. If one or more are red notify the shipper and Mella-

12 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide nox. This on its own does not indicate damage to the contents. But, be sure to carefully inspect the contents if any of the shock and tilt stickers have tripped.

Figure 1: Shock and Tilt Stickers

Rev 1.0

TIL T WA T CH

W

A R

N I N

G

HANDLE   WITH   CARE

W

A R N

I N

HANDLE   WITH   CARE

Stickers unaffected by shipping

TIL T WA T CH

W

A R

N I N

G

HANDLE   WITH   CARE

W

A R N

I N

HANDLE   WITH   CARE

Stickers showing excessive shock or tilt.

Mellanox Technologies 13

Rev 1.0

Angled

Bracket

M6 X4 Caged nut

M6 X4 bolt

Figure 2: Shelf Installation Kit Parts

Chassis Installation

Shelf

Rail

Rail Slide

14 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

1.6.2.4

Opening the Container

Rev 1.0

Washer

L

R

Rail Slid

Pan head screws

Screws for the handles

Step 1.

Step 2.

Step 3.

Step 4.

Step 5.

Step 6.

Step 7.

Step 8.

Check the shock and tilt stickers on the container to see if it was subjected to excessive shock or tilting. If any of these stickers are tripped contact your

Mellanox representative.

Before starting the procedure, put the ESD strap on and connect it to a valid ground.

Open the crate by opening the latches.

Unscrew the sides of the crate.

Remove and put aside the box containing the cables.

Remove and put aside the box containing the cable management kit.

Remove and put aside the box containing the Installation kit.

Visually inspect the chassis, make sure that:

 there is no visible damage

4 PSUs are installed

 all 4 fans are installed

Step 9.

Remove all protective plastic film from all sides and top of the chassis.

Step 10.

If you are not using a mechanical lift to install the chassis, reduce the weight of the chassis by removing all of the spines, power supply units, fan units, and management modules, and put aside for reinstallation after the chassis is installed in the rack.

Mellanox Technologies 15

Rev 1.0

Chassis Installation

The leafs, spines, and management modules are shipped separately.

1.6.2.5

Installing the Shelf

Step 1.

Step 2.

Place the ESD mat on the floor where you will be working and put on the ESD strap.

Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground.

Clip 4 caged nuts into the holes in the rack you will be using to connect the shelf brackets. Check that both sides of the shelf are at the same level in the rack.

Figure 3: Inserting the Caged Nuts for the Shelf

21

Top of shelf

20

Step 3.

Bottom of shelf

Do not use this hole

19

Clip 4 caged nuts into the holes in the rack you will be using to connect the shelf rail slides. Check that both sides of the shelf will be at the same level in the rack.

Figure 4: Use These Holes for the Rail Slides

21

20

16

19

Step 4.

Using two of the bolts for each rail slide, install the rail slides onto the rack. Check that both sides of the switch, left and right, are the same level in the rack.

Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Figure 5: Connect Rail Slide to Rack Vertical support

Rev 1.0

Step 5.

Step 6.

Step 7.

Step 8.

Step 9.

Place the four bolts for the caged nuts within reach.

Slide the shelf into the rail slides.

Put the shelf into place and screw the bolts into the nuts from step 2

Tighten all of the screws to 9.2 Nm or 81.5 pound inches.

Insert 8 caged nuts for the faceplate in the exact locations shown in

Figure 6,“Inserting the Caged Nuts for the Faceplate”.

Figure 6: Inserting the Caged Nuts for the Faceplate

5

4

3

2

1

7

6

8

9

Place the caged nuts here.

Shelf

Mellanox Technologies 17

Rev 1.0

1.7

Installing the Chassis

Figure 7: Chassis Rails and Rail Slides

Chassis Installation

Rail Slide

L

R

Step 1.

Screw the handles onto the chassis. Use the 8 allen head screws. Two go on the faceplate and two go on the sides of the chassis.

18 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Figure 8: Screw the Handles Onto the Chassis

Handles

L

Rev 1.0

Figure 9: The Rails are Already Connected Onto the Chassis

R

When lifting manually use your legs,

bend your knees

, and

not

your back.

Mellanox Technologies

<40 lbs

<18 kgs

40 - 70 lbs

18 - 32 kgs

70 - 121 lbs

32 - 55 kgs

>121 lbs

>55 kgs

19

Rev 1.0

Step 2.

Step 3.

Step 4.

Chassis Installation

Lift the chassis and slide it onto the shelf. Use a mechanical lift or enough people to safely lift the chassis. A full chassis weighs ~120 kgs an empty chassis weighs ~50 kgs.

Remove the two handles from the sides of the chassis.

Push the chassis into the rack until the faceplate is ~ 20cm (~ 8”) from the vertical support.

Figure 10: Chassis on the Shelf

R

Step 5.

Put the rail slides onto the rails.

20 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Figure 11: Put on the Rail Slide

Rev 1.0

R

R

Step 6.

Step 7.

Slide the chassis further into the rack.

Slide the chassis all the way into the rack until the faceplate is touching the vertical rack support. The caged nuts placed in the last procedure should line up with the holes in the faceplate.

Figure 12: Face Plate Mounting Bolt Locations

Holes in the faceplate for mounting on the rack.

Mellanox

®

L

PS1

PS3

Fan   #1  

S1

S2

S3

S4

S5

S6

1 3 5 7 9 11 13 15 17

UID

2 4 6 8 10 12 14 16 18

1 3

5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

1 3

5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

1 3 5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

1 3 5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

1 3 5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

Fan   #2 Fan   #3 Fan   #4

SX6512

R

PS2

PS4

Step 8.

Step 9.

Screw the 8 screws through the faceplate and into the caged nuts.

Slide the rail slides to the vertical rails on the leaf side.

Mellanox Technologies 21

Rev 1.0

Figure 13: Move the Rail Slide to the Vertical Support

Chassis Installation

R

Step 10.

Place the 4 holed spacer on the outside of the vertical support and screw in the 4 screws for each rail slide.

Step 11.

Remove the two handles from the spine side and save all four handles for future use.

Step 12.

Ground the switch.

Step 13.

Replace all of the spines, power supply units, fan units, and management modules removed at the start of the installation procedure.

Step 14.

Plug in the power cables.

Step 15.

Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation.

Warning:

Any yellow status LEDs is cause for concern and must be dealt with immediately.

It can take up to 5 minutes to boot up, during which time the status LED may indicate flashing green.

Step 16.

You can start connecting all of the cables to the switch.

1.8

Inserting the Leafs

The leaf boards can be installed in any of the available leaf slots.

22 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Figure 14: Ejector Latch

Leaf

Leaf

Push the latch release.

Step 1.

Step 2.

Step 3.

Start with the latch fully open, that is 45 o

to the front panel of the leaf.

Carefully set the leaf board into the side guide rails and slowly slide the board into the chassis.

Catch the hooks onto the vertical bar and push the latch shut.

1.9

Inserting a Management Module

Step 1.

Step 2.

Step 3.

Start with the latches fully open, that is 45 o

to the front panel of the leaf.

Carefully seat the management module into the slot and slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge.

Press the ejectors inward until the locks snap. This indicates that the board is fully seated.

Figure 15: Ejector Latch

Management Module /Spine

Management Module /Spine

Rev 1.0

Push the latch release.

Mellanox Technologies 23

Rev 1.0

1.10 Inserting a Spine Board

Figure 16: Spine Board Insertion

Open the Ejector Latches

Chassis Installation

1 3 5 7 9 11 13 15 17

UID

2 4 6 8 10 12 14 16 18

19

21 23 25 27 29 31

33 35

20 22 24 26 28 30 32 34 36

Step 1.

Step 2.

Step 3.

Step 4.

Step 5.

Step 6.

Start with the latches fully open, that is 45 o

to the front panel of the leaf.

Hold the spine board by the sides with both latches fully open.

Insert the board into its rail. The board should slide in without resistance.

Push the board in until the hooks on the latches are touching the vertical bar.

Now press both latches inward simultaneously. You should feel the latches are hooked into the front panel.

Press both latches applying some force to close them completely. The locking latch will engage.

Figure 17: Ejector Latch

Management Module /Spine

Management Module /Spine

Push the latch release.

24 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

2 Installing the Cable Holder

Now is the time to install the cable holder.

Figure 18: Cable Holders

Right side cable holder

12 X M6 caged nut

12 X M6 bolt for caged nut

Rev 1.0

Left side cable holder

Step 1.

Place the Cable holder next to the rack, on the connector side of the chassis, and identify the holes where the caged nuts were placed.

Mellanox Technologies 25

Rev 1.0

Figure 19: Inserting the Caged Nuts for the Cable Holder

Installing the Cable Holder

Top bracket connects here.

9

8

7

5

4

3

2

1

6

Shelf

Place the caged nuts here.

26

The cable holders should go to the outside of the vertical supports.

Step 2.

Set the bottom of the cable holder at the level of the shelf.

If the cable holder is not set properly the upper bracket will not line up with the cable holder.

Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Figure 20: Cable Holder Placement

Notch for upper bracket washer

L

Set this even with the top of the shelf

Step 3.

Step 4.

Screw the cable holder onto the rack using the screws provided.

Repeat steps 1- 3 for the second cable holder.

Rev 1.0

Mellanox Technologies 27

Rev 1.0

3 Ground Connections

Ground Connections

Make sure to connect the ground post to a valid electrical ground. Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge. A ground wire of AWG 6 or 4mm diameter is recommended for grounding this device. The chassis is concurrently grounded through each of the PSUs. Only connect the PSU cords to properly grounded outlets. Do not rely on the PSU grounds. It is absolutely necessary to connect the grounding post. Make sure the connections are solid and permanent. If you choose to not use the ground screw, make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack. Test the ground using an Ohm meter.

Figure 21: Ground Connection

Mellanox

®

L

PS1

PS3

Fan   #1  

S1

S2

1 3 5 7 9 11 13

15 17

UID

2 4 6 8 10 12 14 16 18

1 3 5 7 9 11 13

15 17

UID

2 4 6 8 10 12 14 16 18

S3

S4

S5

S6

1 3 5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

1 3 5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

1 3 5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

1 3 5 7 9 11 13 15 17

UID 2 4 6 8 10 12 14 16 18

Fan   #2

Ground Post

Fan   #3

19 21 23 25 27 29 31 33 35

20 22 24 26 28 30 32 34 36

19 21 23 25 27 29 31 33 35

20 22 24 26 28 30 32 34 36

19 21 23 25 27 29 31 33 35

20 22 24 26 28 30 32 34 36

19 21 23 25 27 29 31 33 35

20 22 24 26 28 30 32 34 36

19 21 23 25 27 29 31 33 35

20 22 24 26 28 30 32 34 36

19 21 23 25 27 29 31 33 35

20 22 24 26 28 30 32 34 36

Fan   #4

SX6512

R

PS2

PS4

ESD Post

28 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

4 Power Connections

The switch includes integrated hot-swap power supplies which support up to 4 load-sharing 1000W or 1600W supply units. The slots for the power supply units (PSUs) are on the spine side. The left side has odd numbered PSUs and the right side has even numbered

PSUs. Each PSU has a dedicated AC inlet. This design enables the optional use of separate main and backup AC feeds. The input voltage is Auto range, 100-240 VAC, 50Hz or

60Hz. The output voltage for the PSUs is 48V. The power cords are standard 3-wire AC power cords including a safety ground.

Before starting any procedure on the SX6512 switch system put an ESD prevention wrist strap on your wrist and connect to the chassis.

4.1

Powering Up the Switch Platform

Rev 1.0

Make sure that the power cords are compatible with your outlets. Power cords for different countries can be ordered from Mellanox. See the user manual appendix for ordering part numbers for power cords.

Step 1.

Step 2.

Plug in the power cords to the PSUs.

Plug the other end of the power cord into grounded outlets.

Make sure that the outlets and circuits will not be overloaded. Spread out the load over at least two or three circuits or use a 3 phase circuit.

With N+1 PSU redundancy the chassis must be started with a full compliment of possible

PSUs, thereafter it can run on one less than the total number of PSUs. This final PSU is redundant and allows for hot swapping a PSU should one fail. Connecting the PSUs to different AC lines provides AC fail over protection.

N+N - will allow for either of the following 2 scenarios.

• Connecting 2 power supplies to a different power supply grid will create N+N grid redundancy, this is High Availability. Under these conditions should a power supply fail

(an electric company power failure or blackout for example) power grid High Availability will continue to keep the chassis running at full capacity through a secondary or backup power supply grid or generator or battery backup system.

• The chassis can also be configured for N+1 times 2. This is all 4 PSUs are supplied from a single power source usually the commercial electric grid. This enables 2 power supply redundancy.

With N+N optional PSU grid redundancy the chassis can run on 1/2 of the full complement of PSUs. N+N allows the chassis to run on 2 PSUs supplied from one power grid while 2 are connected to a second power grid.

Mellanox Technologies 29

Rev 1.0

.

Power Connections

With power grid A charged with current and power grid B not charged there is only grid redundancy and not PSU redundancy.

The power system will divide the current consumption by the number of working PSUs. Should one of the PSUs fail, the total current consumption will then be divided by the remaining working

PSUs. When the failed PSU is hot swapped the new PSU will ramp up and pass its share of current, so that the total current is always divided by the number of working PSUs.

Figure 22: Multiple Power Inlets - Electric Caution Notification

CAUTION

Risk of electric shock and energy hazard.

The PSUs are all independent.

Disconnect all power supplies to ensure a powered down state inside of the switch platform.

ACHTUNG

Gafahr des elektrischen Schocks.

Entferrnen des Netzsteckers elnes

Netzteils spannungsfrei. Um alle

Einhieten spannungsfrei zu machen sind die Netzstecker aller Netzteile zu entfernen

ATTENTION

Risque de choc et de danger

 e’lectriques. Le de’branchment d’une seule alimentation stabilise’e ne de’branch uniquement qu’un module “Alimentation Stabilise’e”.

Pour isoler completement le module en cause, Il faut de’brancher toutes les alimentations stabilise’es.

Step 3.

Check the Status LEDs on all of the management modules and confirm that all of the

LEDs show status lights consistent with normal operation.

Warning

: Any yellow or red status LEDs on any of the management modules is cause for concern and must be dealt with immediately.

Step 4.

Check that none of the LEDs on the spines are yellow.

It can take up to 5 minutes to boot up the system. Turn off the system if any LEDs remain red for more than 5 minutes.

Step 5.

Check that the leaf status LEDs in the spines are lit.

Figure 23: Spine Module

1 3 5 7 9 11 13 15 17

UID

2 4 6 8 10 12 14 16 18

19 21 23 25 27 29 31 33

35

20 22 24 26 28 30 32 34 36

Spine Status Fan Status

Spine Leaf IB Connection Status

30 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

5 InfiniBand QSFP Cable Installation

The switch uses industry standard QSFP InfiniBand cables which are available from Mellanox Technologies. The standard QSFP cables support full 56+56Gb/s (FDR),

40+40Gb/s (FDR10), 40+40Gb/s (QDR), 20+20Gb/s (DDR) and 10+10Gb/s (SDR) bidirectional wire speed of the switch ports. All InfiniBand QSFP connections are made to the leaf boards. Each leaf has 18 InfiniBand QSFP connectors in two rows, which are numbered 1-18.

Rev 1.0

If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase.

FDR BandWidth is supported only by MLNX QSFP cables.

FDR and FDR10 are only guaranteed to work with approved Mellanox Cables.

All cables can be inserted or removed with the unit powered on. To insert a cable, press the connector onto the port receptacle until the connector is firmly seated. The LED indicator, corresponding to each data port, will light orange when the physical connection is established (that is, when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port). When a logical connection is made the LED will change to green. When data is being transferred the light will blink green. To remove, disengage the lock and slowly pull the connector away from the port receptacle.

Warning

: Take care to not impede the air flow through the ventilation holes next to the

InfiniBand ports. Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack.

5.1

Port Connector Interfaces

The connector side of the switch has 12 leaf boards and each leaf board has 18 QSFP ports. The ports on each leaf board are placed in two rows, 9 ports to a row. The ports are labelled as shown in Figure 24. The bottom row ports are flipped from the top row. See

Figure 25.

Mellanox Technologies 31

Rev 1.0

1

InfiniBand QSFP Cable Installation

3

Figure 24: Port Numbering

5

7

9 11

13

15 17

2 4 6 8 10 12 14 16 18

Figure 25: Top and Bottom Ports

5.2

Supported Approved Cables

For a list of approved cables for this switch see the Mellanox approved cable list.

www.mellanox.com/related-docs/user_manuals/Mellanox_approved_cables.pdf

5.3

Cable Power Classes

Chassis and switches need to be able to dissipate the heat generated by high power I/O cables and modules. The Mellanox SX65xx chassis are rated for cables up to class 2 as per the SFF committee classification ( SFF-8436.PDF

). See http://www.mellanox.com/content/pages.php?pg=cables for the cable class rating of Mellanox cables.

32 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

6 Chassis Power Up

Before starting any procedure on the SX

6512

series chassis system put an ESD prevention wrist strap on your wrist and connect to the SX

6512

series chassis.

With N+1 PSU redundancy the chassis must be started with a full complement of possible

PSUs, thereafter it can run on one less than the total number of PSUs. This final PSU is redundant and allows for hot swapping a PSU should one fail. Connecting the PSUs to different AC lines provides AC failover protection.

The system should continue to run and allow a hot swap of a defective PSU. Should there not be enough power to keep all of the leafs running, MLNX-OS may power down some leafs. If this happens it will be necessary to reboot the chassis once the defective PSU has been replaced.

With 1600W optional PSU supplies the chassis can run on as little as 1/2 of the full complement of PSUs. N+N allows the chassis to run on 2PSUs supplied from one power grid while 2 are connected to a second power grid.

1. Check all FRUs for proper insertion and seating before connecting the AC power cords.

Boards

Power supplies

Leaf fan modules

Spine fan modules

Populate the chassis with the leafs before you power up the chassis.

2. Insert all leafs that you plan to use, in the chassis.

3. Insert all spines in the chassis.

4. Insert thermal blanks in unused leaf slots to maintain balanced air flow.

5. Tighten all leaf and spine mounting screws.

6. Connect the power cords to the PSUs.

7. Connect the power cords to grounded electrical outlets.

With N+1 1000 W PSU redundancy, do not power up the chassis with less than all

PSUs installed.

Rev 1.0

Mellanox Technologies 33

Rev 1.0

Chassis Power Up

6.1

Power Supply, Management Board, and Spine Board Indicator Status at Power ON

It can take up to 5 minutes to boot up the system. Turn off the system if any LEDs remain red for more than 5 minutes.

As the power is turned on, you should observe the following conditions for normal operation:

Step 1.

Step 2.

Step 3.

Step 4.

Power Supply Unit(s) AC OK and DC OK indicators are ON and FAIL indicators are

OFF.

There is a

green

Status LED per spine board, per leaf board, and per management module that indicates power supplies are good.

Spine Board indicators will display status of internal links to the installed leaf boards.

All PHY links to existing leaf Boards should be ON.

Check the Spine LEDs and make sure they coincide with Figure 26.

Figure 26: Spine Side Panel Display Status Indications

STATUS

PSU

STATUS

SPINE

FANS

LEAF

FANS

MNG 1

STATUS

MNG 2

STATUS

Step 5.

Check the Spine LEDs and make sure they coincide with Figure 27.

Figure 27: Management Module Status Indications for Normal Operation

IB

STATUS

I2C

CONSOLE MGT

RST PSU S.FANS

L.FANS

MASTER

MGT MAC:

00:02:C9:11:34:8A

34 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

7 InfiniBand Fabric Initialization and Management

Rev 1.0

Subnet Management and Subnet Administration are standard components defined by the

InfiniBand Specification which implement fabric initialization, discovery, and configuration.

The switch’s switching fabric can be managed by MLNX-OS, as well as any third party Subnet Management software running on an InfiniBand Host that is connected to any switch port. The switch is also compatible with the Open Source Subnet Manager

OpenSM.

The chassis is basically an InfiniBand fabric built out of individual InfiniBand switch devices and appears so to external Subnet Managers. The chassis supports the System

Image GUID feature of the InfiniBand specification which means that software management tools that support this feature can recognize all the internal devices as a single switch system.

7.1

Configuring the Switch Management Modules for the First Time

Step 1.

On SX65xx systems, when having dual management systems, first connect the cable and configure the master management module CPU and only then configure the slave.

By default the master CPU is the top management module. For further information on the master and slave roles, see the MLNX-OS Software UM section “High Availability” for more information.

Initial configuration must be done on all of the management modules.

Connect the host PC RS232 connector to the CONSOLE port of the management module using the supplied cable.

Figure 28: Connecting the Harness to the Console

Connect the host

PC to here

IB

STATUS

I2C

CONSOLE MGT

RST PSU S.FANS

L.FANS

MASTER

M

00

No remote IP connection is available at this stage.

Step 2.

Configure a serial terminal program (for example, HyperTerminal, Minicom, or Tera Term) on your host PC with the settings described in Table 3.

Mellanox Technologies 35

Rev 1.0

InfiniBand Fabric Initialization and Management

Table 3 - Serial Terminal Program Configuration

Parameter Setting

Step 3.

Step 4.

Baud Rate

Data bits

Stop bits

Parity

Flow Control

9600

8

1

None

None

Login (from serial terminal program) as

admin

and use

admin

as password. This starts the Mellanox configuration wizard.

Go through the Mellanox configuration wizard. Table 4 shows an example of a wizard session.

Table 4 - Configuration Wizard Session - IP Configuration by DHCP (Sheet 1 of 2)

Wizard Session Display (Example)

Mellanox configuration wizard

Do you want to use the wizard for initial configuration? yes

Step1: Hostname? [switch-1]

Step 2: Use DHCP on mgmt0 interface? [no]

Comments

You must perform this configuration the first time you operate the switch or after resetting the switch. Type

‘y’ and then press <Enter>.

If you wish to accept the default hostname, then press

<Enter>. Otherwise, type a different hostname and press <Enter>.

Perform this step to obtain an IP address for the switch.

(eth0 is the management port of the switch.)

If you wish the DHCP server to assign the IP address, type ‘yes’ and press <Enter>.

If you type ‘no’ (no DHCP), then you will be asked whether you wish to use the ‘zeroconf’ configuration or not. If you enter ‘yes’ (yes Zeroconf), the session will continue as shown in Table 5.

Step 3: Enable IPv6? [yes] yes

If you enter ‘no’ (no Zeroconf), then you need to enter a

static

IP, and the session will continue as shown in

Table 6.

The management interface will be able to use IPv6 addresses.

Step 4: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] no

This turns on autoconfiguration of the IPv6 addresses.

This is unsuitable for DHCPv6.

36 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Table 4 - Configuration Wizard Session - IP Configuration by DHCP (Sheet 2 of 2)

Wizard Session Display (Example) Comments

Step 5: Admin password (Press <Enter> to leave unchanged)? <new_password>

Step 6: Confirm admin password? <new_password>

To avoid illegal access to the machine, please type a password and then press <Enter>. Then confirm the password by re-entering it.

Note that password characters are

not

printed.

You have entered the following information: The wizard displays a summary of your choices and then asks you to confirm the choices or to re-edit them.

1. Hostname: <switch name>

2. Use DHCP on mgmt0 interface: yes

3. Enable IPv6: yes

4. Enable IPv6 autoconfig (SLAAC) on mgmt0 interface: no

5. Admin password (Enter to leave unchanged):

(CHANGED)

Either press <Enter> to save changes and exit, or enter the configuration step number that you wish to return to.

To change an answer, enter the step number to return to.

Otherwise hit <enter> to save changes and exit.

Note:

To run the command “configuration jump-start” you must be in Config mode.

Choice: <Enter>

Configuration changes saved.

To return to the wizard from the CLI, enter the “configuration jump-start” command from configuration mode. Launching CLI...

<switch name> [>

Rev 1.0

Mellanox Technologies 37

Rev 1.0

InfiniBand Fabric Initialization and Management

Table 5 - Configuration Wizard Session - IP Zeroconf Configuration

Wizard Session Display - IP Zeroconf Configuration (Example)

Mellanox configuration wizard

Do you want to use the wizard for initial configuration? y

Step 1: Hostname? [switch-112126]

Step 2: Use DHCP on mgmt0 interface? [yes] no

Step 3: Use zeroconf on mgmt0 interface? [no] yes

Step 4: Default gateway? [For example:192.168.10.1]

Step 5: Primary DNS server?

Step 6: Domain name?

Step 7: Enable IPv6? [yes] yes

Step 8: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] no

Step 9: Admin password (Enter to leave unchanged)?

Step 9: Confirm admin password? (Enter to leave unchanged)?

You have entered the following information:

1. Hostname: switch-112126

2. Use DHCP on mgmt0 interface: no

3. Use zeroconf on mgmt0 interface? [no] yes

4. Default gateway: 192.168.10.1)

5. Primary DNS server:

6. Domain name:

7. Enable IPv6: yes

8. Enable IPv6 autoconfig (SLAAC) on mgmt0 interface: no

9. Admin password (Enter to leave unchanged): (unchanged)

To change an answer, enter the step number to return to.

Otherwise hit <enter> to save changes and exit.

Choice:

Configuration changes saved.

To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI...

switch-1 >

38 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Table 6 - Configuration Wizard Session - Static IP Configuration

Wizard Session Display - Static IP Configuration (Example)

Mellanox configuration wizard

Do you want to use the wizard for initial configuration? y

Step 1: Hostname? [switch-112126]

Step 2: Use DHCP on mgmt0 interface? [yes] n

Step 3: Use zeroconf on mgmt0 interface? [no]

Step 4: Primary IP address? [for example 192.168.10.4]

Mask length may not be zero if address is not zero (interface eth0)

Step 5: Netmask? [0.0.0.0] 255.255.255.0

Step 6: Default gateway? [for example 192.168.10.1]

Step 7: Primary DNS server?

Step 8: Domain name?

Step 9: Enable IPv6? [yes] yes

Step 10: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] no

Step 11: Admin password (Enter to leave unchanged)?

You have entered the following information:

1. Hostname: switch-112126

2. Use DHCP on mgmt0 interface: no

3. Use zeroconf on mgmt0 interface: no

4. Primary IP address: 192.168.10.4

5. Netmask: 255.255.255.0

6. Default gateway: 192.168.10.1

7. Primary DNS server:

8. Domain name:

9. Enable IPv6: yes

10.Enable IPv6 autoconfig (SLAAC) on mgmt0 interface: no

11.Admin password (Enter to leave unchanged): (unchanged)

To change an answer, enter the step number to return to.

Otherwise hit <enter> to save changes and exit.

Choice:

Configuration changes saved.

To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI...

switch-1

>

Step 5.

Before attempting a remote (for example, SSH) connection to the switch, check the eth0 interface configuration. Specifically, verify the existence of an

IP address. To check the current eth0 configuration, enter the following commands:

sx-43 [standalone: master] > enable sx-43 [standalone: master] # show interfaces mgmt0

Mellanox Technologies

Rev 1.0

39

Rev 1.0

InfiniBand Fabric Initialization and Management

The following is an example of the output:

Interface mgmt0 state

Admin up: yes

Link up: yes

IP address: 192.168.10.43

Netmask: 255.255.255.0

Speed: 1000Mb/s (auto)

Duplex: full (auto)

Interface type: ethernet

Interface source: physical

MTU: 1500

HW address: 00:02:C9:11:2A:AE

Comment:

RX bytes: 1343502058 TX bytes:

313920869

RX packets: 17589211 TX packets: 992717

RX mcast packets: 0 TX discards: 0

RX discards: 0 TX errors: 0

RX errors: 0 TX overruns: 0

RX overruns: 0 TX carrier: 0

RX frame: 0 TX collisions: 0

TX queue len: 1000

Step 6.

On SX65xx systems, configure the Box IP (BIP) centralized location for chassis High

Availability according to the steps in the MLNX-OS Software UM section “Box IP

Centralized Location”.

7.2

Rerunning the Wizard

If you want to rerun the wizard run the following commands:

sx-43 [standalone: master] > enable sx-43 [standalone: master]# configure terminal sx-43 [standalone: master]# configuration jump-start

40 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

8 MLNX-OS Management Software

MLNX-OS is a software based management system that can be run with either a command line interface or with a

WebUI

interface. Chassis management comes with the switch system.

8.1

Starting an SSH Connection to the Switch

Rev 1.0

SSH connection also supports TelNet.

Step 1.

Step 2.

Step 3.

Set up an Ethernet connection between the switch and a local network machine (“the remote machine” henceforth) using a standard RJ-45 connector.

Connect to the remote machine.

Start a remote shell to the switch using the following command:

ssh –l admin <your ip address>

When prompted for a password enter “admin”.

Step 4.

Note: The IP address used above is the same IP address that was assigned to the Mellanox configuration wizard in the “Configuring the Switch Management Modules for the

First Time” section.

Step 5.

You can enter any supported command now.

For a complete reference of commands, please see

Mellanox MLNX-OS Management

Software User’s Manual

.

8.2

Starting a WebUI Connection to the Switch

Step 1.

Step 2.

Set up an Ethernet connection between the switch and a local network machine (“the remote machine” henceforth) using a standard RJ-45 connector.

Start a Web browse – Google Chrome, Internet Explorer 7.0 or Mozilla

Firefox 3.0.

Make sure the screen resolution is set to 1024*768 or higher.

Mellanox Technologies 41

Rev 1.0

Step 3.

Step 4.

MLNX-OS Management Software

Enter the following URL: http://<switch_IP_address>

 where <switch_IP_address> is the IP address of the switch or its DNS name.

You will receive the login window for remote management of the switch.

Figure 29: Web UI Login Page

Step 5.

Use

admin

for both the login (Account) and the password.

42 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

9 Resetting the Switch – RST

The Reset button resets the chassis management module when the button is pushed. When the button is held down for 15 seconds the chassis is reset and the password deleted.

Rev 1.0

CONSOLE

DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button. Sharp objects can cause damage, use a flat object such as a paper clip.

MGT

Figure 30: Reset Button

RST PSU S.FANS

L.FANS

MASTER

MGT

00:02:C9

This button resets the CPU of the management module. A quick push of this button performs this reset. When the reset button is pushed on the master management module this management module is reset becoming the slave and the other management module becomes the master. If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button is pushed. When the button is held down for 15 seconds the management module is reset and the password is deleted. You will then be able to enter without a password and make a new password for the user admin.

Mellanox Technologies 43

Rev 1.0

10 TroubleShooting

10.1 Power Supply Unit

TroubleShooting

If the Power supplies cannot supply enough power the management module may shut down the leafs until the power being supplied is sufficient to run the operating leafs. If the management module does shut down some leafs, it may be necessary to reboot the chassis to get the shut down leafs up and running again.

As each PSU is plugged in, make sure that the green power LEDs on the PSU comes on.

If the AC power LED is off:

Step 1.

Check that the power cable is the correct power cable for your country.

Step 2.

Step 3.

Check that the power cable is plugged into a working outlet.

Check that the power cable has a voltage within the range of 100 - 240 volts AC.

Step 4.

Step 5.

Step 6.

Step 7.

Remove and reinstall the power cable.

Check the circuit breakers to be sure that the breaker has not tripped.

Check that the power cable is good. Replace the power cable.

If the AC power LED is green but the OK power LED is off or the FAIL LED is on –

Replace the PSU.

10.2 Leaf Board

The power LED for the Leaf board is off:

Step 1.

Step 2.

Step 3.

Make sure that all of the PSUs are showing DC OK.

Uninstall and reinstall the Leaf board.

Step 4.

Step 5.

When the Yellow LED is on, this indicates a fault in the board, uninstall and reinstall the Leaf board.

If uninstalling and reinstalling the Leaf board does not work, burn the latest FW on the

Leaf board and uninstall and reinstall the Leaf board.

Replace the Leaf board with a new one.

Should any of the leaf boards shut down due to over temperature, it may be necessary to reboot the chassis.

44

The Physical link LED for the InfiniBand connector does not come on:

Step 1.

Check that both ends of the cable are connected.

Step 2.

Check that the locks on the ends are secured.

Step 3.

Step 4.

Step 5.

Make sure that the latest FW version is installed on both the HCA card and the switch.

If media adapters are used check that the all connections are good, tight, and secure.

Replace the cable.

The Activity indication does not come on:

Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Check that the Subnet Manager has been started.

10.3 Management Module

10.3.1 Yellow Status LED (for the Chassis) on the Management Module is Lit

Step 1.

Step 2.

Check the MLNX-OS management for confirmation and possible explanation of the alert.

Reset the master management module by pushing the rest button. If you have two management modules installed this will convert the master management module to the slave and convert the slave to the master.

If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the management module is removed.

Step 3.

Step 4.

Step 5.

Step 6.

Step 7.

Make sure the S.Fans and L.Fans LEDs are green.

Make sure that the spine and the leafs both have the same version of FW.

Reburn the FW and remove and reinstall the management module.

If you are running the chassis with only one management module, remove and reinstall the management module. Make sure the mating connectors of the unit are free of any dirt and/or obstacles. See Section 5.5 on page 122.

If you are running the chassis with only one management module, replace the management module.

10.3.2 Yellow LED for the Leaf Fan on the Management Module is Lit

Step 1.

Step 2.

Step 3.

Step 4.

Step 5.

Step 6.

Check the MLNX-OS management for confirmation and possible explanation of the alert.

Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked (especially dust over the holes).

If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner.

Determine which fan module is problematic by checking the status LED on each fan module.

Remove and reinstall the problematic fan unit. Make sure the mating connector of the new unit is free of any dirt and/or obstacles. See Section 5.4 on page

117.

Replace the Leaf fan module.

Rev 1.0

Replace defective leaf fan modules as soon as they are identified.

Mellanox Technologies 45

Rev 1.0

TroubleShooting

Should any of the leaf boards shut down due to over temperature, it may be necessary to reboot the chassis.

10.3.3 Yellow LED for the Spine Fan on the Management Module is Lit

Step 1.

Step 2.

Step 3.

Step 4.

Step 5.

Step 6.

Check the MLNX-OS management for confirmation and possible explanation of the alert.

Determine which spine has a defective fan by checking the Fan LEDs on all of the spines.

Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked (especially dust over the holes).

If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner.

Remove and reinstall the fan unit of the spine. Make sure the mating connector of the new unit is free of any dirt and/or obstacles. See Section 5.4 on page 117.

Replace the spine fan module.

Replace defective spine fan modules as soon as they are identified.

10.4 Spine Board

The yellow LED on the Spine board is lit:

Step 1.

Check the MLNX-OS management for confirmation and possible explanation of the alert.

Step 2.

Step 3.

Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked (especially dust over the holes).

If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner.

Step 4.

Step 5.

Step 6.

Step 7.

Remove and reinstall the spine board. Make sure the mating connectors of the unit is free of any dirt and/or obstacles. See Section 5.3 on page 114.

Make sure that the spine and the Leafs both have the same version of FW.

Reburn the FW and remove and reinstall the spine.

Replace the spine board.

10.5 MLNX-OS Software

For a instructions concerning software updates see the Mellanox MLNX-OS™ SwitchX® Software WebUI User’s Manual or the Mellanox MLNX-OS™ SwitchX® Software User Manual.

46 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

For more information and documentation on the SX6512 Series go to: the Mellanox Website http://www.mellanox.com

Rev 1.0

Mellanox Technologies 47

Rev 0.1

USA

Mellanox Technologies Inc.

350 Oakmead Parkway, Suite 100

Sunnyvale, CA 94085

Tel: (408) 970-3400

Fax: (408) 970-3403 www.mellanox.com

E-mail: [email protected]

Map and Directions

Tech Support: 408-916-0055

(8:00AM to 5:00 PM - Pacfic)

Mellanox Technologies, Ltd.

Beit Mellanox

PO Box 586 Yokneam 20692

Israel

Tel: +972 (0)4 909 7200 ;

+972 (0)74 723 7200

Fax: +972 (0)4 959 3245

Tech Support: +972 4 9097200 ext 250

(8:00AM to 5:00 PM - Israel Time)

www.mellanox.com

48 Mellanox Technologies

216 Port InfiniBand FDR Switch Platform

Installation Guide

advertisement

Related manuals

Download PDF

advertisement