AOC-R1UG-IBQ Add-on Card

User’s Guide

Revison 1.0

AOC-R1UG-IBQ Add-on Card User’s Guide

The information in this User’s Manual has been carefully reviewed and is believed to be accurate. The vendor assumes no responsibility for any inaccuracies that may be contained in this document, makes no commitment to update or to keep current the information in this manual, or to notify any person or organization of the updates. Please Note: For the most up-to-date version of this manual, please see

our web site at www.supermicro.com

.

Super Micro Computer, Inc. (“Super micro”) reserves the right to make changes to the product described in this manual at any time and without notice. This product, including software and documentation, is the property of Supermicro and/or its licensors, and is supplied only under a license. Any use or reproduction of this product is not allowed, except as expressly permitted by the terms of said license.

IN NO EVENT WILL SUPERMICRO BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,

SPECULATIVE OR CONSEQUENTIAL DAMAGES ARISING FROM THE USE OR INABILITY TO USE

THIS PRODUCT OR DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH

DAMAGES. IN PARTICULAR, SUPERMICRO SHALL NOT HAVE LIABILITY FOR ANY HARDWARE,

SOFTWARE, OR DATA STORED OR USED WITH THE PRODUCT, INCLUDING THE COSTS OF

REPAIRING, REPLACING, INTEGRATING, INSTALLING OR RECOVERING SUCH HARDWARE,

SOFTWARE, OR DATA.

Any disputes arising between manufacturer and customer shall be governed by the laws of Santa Clara

County in the State of California, USA. The State of California, County of Santa Clara shall be the exclusive venue for the resolution of any such disputes. Super Micro's total liability for all claims will not exceed the price paid for the hardware product.

FCC Statement: This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the manufacturer’s instruction manual, may cause harmful interference with radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case you will be required to correct the interference at your own expense.

California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only to products containing CR (Manganese Dioxide) Lithium coin cells. Perchlorate

Material-special handling may apply. See www.dtsc.ca.gov/hazardouswaste/perchlorate for further details.

WARNING: HANDLING OF LEAD SOLDER MATERIALS USED IN THIS

PRODUCT MAY EXPOSE YOU TO LEAD, A CHEMICAL KNOWN TO THE

STATE OF CALIFORNIA TO CAUSE BIRTH DEFECTS AND OTHER

REPRODUCTIVE HARM.

Manual Revison 1.0

Release Date: June 10, 2010

Unless you request and receive written permission from Super Micro Computer, Inc., you may not copy any part of this document.

Information in this document is subject to change without notice. Other products and companies referred to herein are trademarks or registered trademarks of their respective companies or mark holders.

Copyright © 2010 by Super Micro Computer, Inc.

All rights reserved.

Printed in the United States of America

ii

Table of Contents

Chapter 1 Safety Guidelines

.......................................................... 1-1

1-1 ESD Safety Guidelines

..................................................................... 1-1

1-2 General Safety Guidelines

............................................................... 1-1

1-3 An Important Note to Users

............................................................. 1-1

Chapter 2 Card Components

......................................................... 2-1

2-1 Front Components

............................................................................ 2-1

2-2 Front Connectors, Jumpers and LEDs

.......................................... 2-1

QSFP Port............................................................................................... 2-1

LEDs ....................................................................................................... 2-1

J1 Jumper ............................................................................................... 2-2

J2 Jumper ............................................................................................... 2-2

Chapter 3 Installation

......................................................................... 3-1

3-1 Hardware Installation

........................................................................ 3-1

Before Installation ................................................................................... 3-1

Installation ............................................................................................... 3-1

3-2 Installing Drivers and Firmware

...................................................... 3-2

Linux Drivers ........................................................................................... 3-2

Windows Drivers ..................................................................................... 3-2

Firmware Update Procedures ................................................................. 3-3

iii

AOC-R1UG-IBQ Add-on Card User’s Guide

Notes iv

Introduction

About this Manual

This manual is written for system integrators, PC technicians and knowledgeable PC users who intend to integrate SuperMicro's AOC-R1UG-IBQ add-on card to their system. This add-on card is used to add an InfiniBand port to the H8DGG-QF serverboard.

Product Features

The AOC-R1UG-IBQ add-on card offers the following features:

• Mellanox® ConnectX-2 QDR

• Single QSFP connector

• 40Gb/s per port

• 1us MPI ping latency

• CPU off-load of transport operations

• End-to-end QoS and congestion control

• Hardware-based I/O virtualization

• TCP/UDP/IP stateless off-load

• External optical media adapter and active cable support

• Dimensions: 12.79” (32.48cm) x 8.66” (22cm) (length x width)

NOTE: The AOC-R1UG-IBQ add-on card is designed for the H8DGG-QF

Supermicro motherboard. Please go to www.supermicro.com

for the latest list of supported platforms.

v

AOC-R1UG-IBQ Add-on Card User’s Guide

Operating Systems Supported

The AOC-R1UG-IBQ add-on card supports the following Operating Systems (OS):

• Windows 2003/Windows 2008

• VMware ESX Server 3.5

• SLES

• RedHat

• Fedora

• CCS 2003

An Important Note to Users

All images and layouts shown in this user's guide are based upon the latest revision available at the time of publishing. The card you have received may or may not look exactly the same as the graphics shown in this manual.

Returning Merchandise for Service

A receipt or copy of your invoice marked with the date of purchase is required before any warranty service will be rendered. You can obtain service by calling your vendor for a Returned Merchandise Authorization (RMA) number. When returning to the manufacturer, the RMA number should be prominently displayed on the outside of the shipping carton, and mailed prepaid or hand-carried. Shipping and handling charges will be applied for all orders that must be mailed when service is complete.

For faster service, RMA authorizations may be requested online at: http://www.supermicro.com/support/rma/

Whenever possible, repack the add-on card in the original Supermicro box, using the original packaging materials. If these are no longer available, be sure to pack the add-on card in an anti-static bag and inside the box. Make sure that there is enough packaging material surrounding the add-on card so that it does not become damaged during shipping.

This warranty only covers normal consumer use and does not cover damages incurred in shipping or from failure due to the alteration, misuse, abuse or improper maintenance of products.

During the warranty period, contact your distributor first for any product problems.

vi

Contacting SuperMicro

Headquarters

Address:

Tel:

Fax:

Email:

Web Site:

Super Micro Computer, Inc.

980 Rock Ave.

San Jose, CA 95131 U.S.A.

+1 (408) 503-8000

+1 (408) 503-8008 [email protected] (General Information) [email protected] (Technical Support) www.supermicro.com

Europe

Address:

Tel:

Fax:

Email:

Super Micro Computer B.V.

Het Sterrenbeeld 28, 5215 ML

‘s-Hertogenbosch, The Netherlands

+31 (0) 73-6400390

+31 (0) 73-6416525 [email protected] (General Information) [email protected] (Technical Support) [email protected] (Customer Support)

9

Asia-Pacific

Address:

Tel:

Fax:

Web Site:

Technical Support:

Email:

Tel:

Super Micro Computer, Inc.

4F, No. 232-1, Liancheng Rd.

Chung-Ho 235, Taipei County

Taiwan, R.O.C.

+886-(2) 8226-3990

+886-(2) 8226-3991 www.supermicro.com.tw

[email protected]

+886-2-8228-1366, ext. 132 or 139

vii

AOC-R1UG-IBQ Add-on Card User’s Guide

Notes viii

Chapter 1

Safety Guidelines

To avoid personal injury and property damage, carefully follow all the safety steps listed below when accessing your system or handling the components.

1-1 ESD Safety Guidelines

Electric Static Discharge (ESD) can damage electronic components. To prevent damage to your system, it is important to handle it very carefully. The following measures are generally sufficient to protect your equipment from ESD.

• Use a grounded wrist strap designed to prevent static discharge.

• Touch a grounded metal object before removing a component from the antistatic bag.

• Handle the add-on card by its edges only; do not touch its components, peripheral chips, memory modules or gold contacts.

• When handling chips or modules, avoid touching their pins.

• Put the card and peripherals back into their antistatic bags when not in use.

1-2 General Safety Guidelines

• Always disconnect power cables before installing or removing any components from the computer.

• Disconnect the power cable before installing or removing any cables from the system.

• Make sure that the add-on card is securely and properly installed on the motherboard to prevent damage to the system due to power shortage.

1-3 An Important Note to Users

All images and layouts shown in this user's guide are based upon the latest PCB

Revision available at the time of publishing. The card you have received may or may not look exactly the same as the graphics shown in this manual.

1-1

AOC-R1UG-IBQ Add-on Card User’s Manual

Notes

1-2

Chapter 2

Card Components

2-1 Front Components

Figure 2-1

shows the front components of the AOC-R1UG-IBQ add-on card.

Figure 2-1. AOC-R1UG-IBQ Add-on Card Front Components

4 2

3

1

DESIGNED IN USA

AOC-R1UG-IBQ

REV: 1.00

Table 2-1 lists the add-on card’s front components and their description.

Table 2-1. Front Components

Number Name

1

2

QSFP Connector Port

J1 Jumper

3

4

Description

Port for a single QSFP Connector

Flash ROM Present enable/disable jumper

Mellanox ConnectX-2 QDR chip

Controller chip for the AOC-R1UG-IBQ add-on card

.

J2 Jumper This jumper is used for Engineering Debugging only.

2-2 Front Connectors, Jumpers and LEDs

This section describes the front connectors, jumpers and LEDs found on the

AOC-R1UG-IBQ add-on card.

QSFP Port

The AOC-R1UG-IBQ card contains a single QSFP port. This port allows the

AOC-R1UG-IBQ add-on card to connect to a QSFP passive copper network cable at speeds of up to 40 Gbps using four channels. This connector offers four times the density of traditional SFP ports and support speeds up to 10 Gb/s per channel.

LEDs

The AOC-R1UG-IBQ add-on card contains an LED that is used to display activity as well as link-up of the port. See

Table 2-2 for LED color and blinking definitions.

2-1

AOC-R1UG-IBQ Add-on Card User’s Manual

LED

Activity

Link-up

Color

Blinking

Yellow

Green

Off

Table 2-2. LAN Port LEDs

Definition

Port Activity

Logical Link-up

Physical Link-up

No connection

J1 Jumper

The J1 jumper is used to tell the Mellanox chip that there is a Flash ROM present in the system. If the jumper is closed (enabled) it means that no Flash ROM is present, while open (disabled) means that a Flash ROM is present.

J2 Jumper

The J2 jumper is used for engineer debugging only.

2-2

Chapter 3

Installation

3-1 Hardware Installation

User the directions below to install the AOC-R1UG-IBQ add-on card into your system.

Before Installation

Before Installation, do the following

1.

Power down the system.

2.

Remove the power cord from the wall socket.

3.

Use industry standard anti-static equipment (i.e. gloves or wrist strap) and/or an environment that prevents accidental electrostatic discharge.

4.

Familiarize yourself with the server, motherboard, and/or chassis documentation.

5.

Confirm that your Operating System includes the latest updates and hot-fixes.

Installation

NOTE: This card is designed for the H8DGG-QF serverboard only.

The AOC-R1UG-IBQ add-on card can beinstalled directly into both the SBX connector and PCI-E 2.0 x8 (in a x16) slot onto the H8DGG-QF serverboard. Position the add-on card in both the PCI-E slot and over the SBX connector and gently push down on both sides of the card until it slides into the PCI-E connector and SBX connector together.

Installing the Add-on Card

1.

Remove the server cover and, if necessary, set aside any screws for later use.

2.

Remove the add-on card slot cover. If the case requires a screw, place the screw aside for later use.

3.

Position the add-on card in both the PCI-E slot and over the SBX connector and gently push down on both sides of the card until it slides into the PCI-E connector and SBX connector together.

4.

Secure the add-on card to the chassis. If required, use the screw that you previously removed.

5.

Replace the chassis cover.

6.

Connect the incoming QSFP cable to the QSFP port provided on the card.

3-1

AOC-R1UG-IBQ Add-on Card User’s Manual

7.

Plug the power cord into the wall socket and power up the system.

3-2 Installing Drivers and Firmware

Use the procedures below to install both drivers and firmware for the AOC-R1UG-IBQ add-on card for both Linux and Windows.

Linux Drivers

Use the following procedures for installing AOC-R1UG-IBQ add-on card drivers for the

Linux operating system.

Installing InfiniBand Drivers for the Linux Operating System

1.

From the CDR-NIC LAN driver CD or FTP site, go to the following directory:

M

ELLANOX

Æ I

NFINI

B

AND

Æ L

INUX

.

2.

Choose the InfiniBand Linux driver package file.

3.

Install the driver by the following commands: tar xzvf OFED-<ver>.tgz cd OFED-<ver>

./install.pl

This installs the Linux drivers to your system.

Installing 10G Drivers for the Linux Operating System

1.

From the CDR-NIC LAN driver CD or FTP site, go to the following directory:

M

ELLANOX

Æ C

ONNECT

X_EN

Æ L

INUX

.

2.

Choose the 10G Linux driver package file.

3.

Install the driver by the following commands: tar xzvf mlnx_en-<ver>.tgz cd mlnx_en-<ver>

./install.sh

This installs the 10G drivers to your system.

Windows Drivers

Use the following procedures for installing AOC-R1UG-IBQ add-on card drivers for the

Windows operating system.

Installing InfiniBand Drivers for the Windows Operating System

1.

From the CDR-NIC LAN driver CD or FTP site, go to the following directory:

M

ELLANOX

Æ I

NFINI

B

AND

Æ W

INDOWS

.

2.

Choose the desired InfiniBand Windows driver package file.

3-2

Chapter 3: Installation

3.

Double-click to run and install the driver package file.

Firmware Update Procedures

Use the following procedures for updating the AOC-R1UG-IBQ add-on card firmware for both the Linux and Windows operating systems.

Updating Firmware for the Linux Operating System

1.

From the CDR-NIC LAN driver CD or FTP site, go to the following directory:

M

ELLANOX

Æ F

IRMWARE

_T

OOL

Æ L

INUX

.

2.

Choose the MFT Linux package file and untar the package file.

3.

Install the package by the following command: cd mft-<ver>

./install.sh

4.

Install the firmware by the following commands: mst start mst status *note down the pci_cr0 device name* flint –d <pci_cr0 device name> -i <firmware file name>.bin b

Upadating Firmware for the Windows Operating System

1.

From the CDR-NIC LAN driver CD or FTP site, go to the following directory:

M

ELLANOX

Æ F

IRMWARE

_T

OOL

Æ W

INDOWS

.

2.

Choose the desired MFT Windows package file.

3.

Double-click to run and install the MFT package file.

4.

Open up the DOS Command Prompt and locate the MFT folder.

5.

Install the firmware by the following commands: mst start mst status *note down the pci_cr0 device name* flint –d <pci_cr0 device name> -i <firmware file name>.bin b

3-3

AOC-R1UG-IBQ Add-on Card User’s Manual

Notes

3-4

Disclaimer

The products sold by Supermicro are not intended for and will not be used in life support systems, medical equipment, nuclear facilities or systems, aircraft, aircraft devices, aircraft/emergency communication devices or other critical systems whose failure to perform be reasonably expected to result in significant injury or loss of life or catastrophic property damage. Accordingly, Supermicro disclaims any and all liability, and should buyer use or sell such products for use in such ultra-hazardous applications, it does so entirely at its own risk. Furthermore, buyer agrees to fully indemnify, defend and hold Supermicro harmless for and against any and all claims, demands, actions, litigation, and proceedings of any kind arising out of or related to such ultra-hazardous use or sale.

AOC-R1UG-IBQ Add-on Card User’s Manual

Notes