Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage
Cisco UCS C3X60 M4 Server Node For Cisco UCS
S3260 Storage Server Service Note
This document covers server node installation and replacement of internal server node components.
Cisco UCS C3X60 M4 Software/Firmware Requirements, page 1
Cisco UCS C3X60 M4 Server Node Overview, page 2
Removing or Installing a C3X60 M4 Server Node, page 6
Replacing C3X60 M4 Server Node and I/O Expander Components, page 11
Service Headers on the Server Node Board, page 47
Storage Controller Considerations, page 50
Related Documentation, page 60
Cisco UCS C3X60 M4 Software/Firmware Requirements
The Cisco UCS S3260 system firmware and software requirements for using the Cisco UCS C3X60 M4 server nodes are listed in
Table 1 S3260 System Minimum Levels for C3X60 M4 Server Nodes
Software or Firmware Minimum Version
Server Cisco IMC
Server BIOS
2.0(13)
2.0(13)
Cisco UCS Manager (UCSM-managed system only)
3.1(2)
Cisco Systems, Inc.
www.cisco.com
1
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Cisco UCS C3X60 M4 Server Node Overview
Cisco UCS C3X60 M4 Server Node Overview
Externally Viewable LEDs, page 3
C3X60 M4 Server Node Internal Component Locations, page 4
External Features
Figure 1 Cisco UCS C3X60 M4 Server Node Rear-Panel Features
1 Reset button
2 Server node power button/LED
3 Unit identification button/LED
4 Server node health LED (CR22 on server node board, viewable 3 inches inside mesh screen)
5 KVM cable connector
6 Ejector levers (two)
7 C3X60 M4 server node label (M4 SVRN)
2
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Cisco UCS C3X60 M4 Server Node Overview
Externally Viewable LEDs
Table 2 Cisco UCS C3X60 M4 Server Node Externally Viewable LEDs
LED
Server node power
Definition of States
Off—There is no AC power to the server node.
Unit identification
Amber—The server node is in standby power mode. Power is supplied only to the Cisco IMC.
Green—The server node is in main power mode. Power is supplied to all server node components.
Off—The Identification LED is not in use.
Server node health LED
This LED is on the server node board, but is viewable about 3 inches behind the rear-panel mesh screen.
Blue, blinking—The Identification LED is activated.
Green—The server node is operating normally.
Green, blinking—The server is in standby mode or sleep state.
Amber—The server node is in a degraded condition. Degraded condition is defined as one or more of the following:
—
Power supply redundancy lost (power supply unplugged or failed)
—
SIOC redundancy lost
—
Faulty or mismatched CPUs
—
DIMM failure
—
Failed drive in a RAID configuration
Amber, blinking—The server node is in a critical condition. Critical condition is defined as the following:
—
Boot failure
—
Fatal CPU and/or bus errors detected
—
Fatal uncorrectable memory error detected
—
Both SIOCs failed
—
Both drives in a RAID configuration failed
—
Excessive thermal conditions
3
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Cisco UCS C3X60 M4 Server Node Overview
KVM Cable Connector
This connector allows you to connect a local keyboard, video, and mouse (KVM) cable if you want to perform setup and management tasks locally rather than remotely.
Buttons
Reset button—You can hold this button down for 5 seconds and then release it to restart the server node controller chipset if other methods of restarting do not work.
Server node power button/LED—You can press this button to put the server node in a standby power state or return it to full power instead of shutting down the entire system. See also
Externally Viewable LEDs, page 3
.
Unit identification button/LED—This LED can be activated by pressing the button or by activating it from the software interface. This
helps to locate a specific server node. See also Externally Viewable LEDs, page 3 .
C3X60 M4 Server Node Internal Component Locations
Figure 2 Cisco UCS C3X60 M4 Server Node Internal Components
1 RTC battery CR2032
(accessible with the server node cover on)
2 DIMM sockets (8 sockets for each CPU, 16 total)
3 CPUs and heatsinks (two)
4 Trusted platform module (TPM) socket on server board
5 Cisco storage controller card with supercap power module attached
(software designation SBMezz1)
6 SSD socket location, on server board under controller card
(software designation SBNVMe1)
This socket supports one NVMe PCIe SSD.
4
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Cisco UCS C3X60 M4 Server Node Overview
I/O Expander Internal Component Locations
The C3X60 M4 server node might include an optional I/O expander that attaches to the top of the server node.
Figure 3 I/O Expander Internal Components
1 PCIe card sockets
(software designations IOESlot1 and IOESlot2) x8 electrical lane, 16x physical socket
2 Cisco storage controller card with supercap power module attached (software designation IOEMezz1)
3 NVMe PCIe SSD sockets, two on-board under the RAID controller card
(software designations IOENVMe1 and IOENVMe2)
5
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Removing or Installing a C3X60 M4 Server Node
Removing or Installing a C3X60 M4 Server Node
Shutting Down a C3X60 M4 Server Node, page 6
C3X60 M4 Server Node Population Rules, page 7
Replacing a C3X60 M4 Server Node, page 8
Shutting Down a C3X60 M4 Server Node
You can invoke a graceful shutdown or a hard shutdown of a server node by using either the Cisco Integrated Management Controller (Cisco
IMC) interface, or the power button that is on the face of the server node.
Shutting Down a Server Node By Using the Cisco IMC GUI
To use the Cisco IMC GUI to shut down the server node, follow these steps:
1.
Use a browser and the management IP address of the system to log in to the Cisco IMC GUI.
2.
In the Navigation pane, click the Chassis menu.
3.
In the Chassis menu, click Summary.
4.
In the toolbar above the work pane, click the Host Power link.
The Server Power Management dialog opens. This dialog lists all servers that are present in the system.
5.
In the Server Power Management dialog, select one of the following buttons for the server that you want to shut down:
CAUTION:
To avoid data loss or damage to your operating system, you should always invoke a graceful shutdown of the operating system.
Do not power off a server if any firmware or BIOS updates are in progress.
Shut Down—Performs a graceful shutdown of the operating system.
Power Off—Powers off the chosen server, even if tasks are running on that server.
It is safe to remove the server node from the chassis when the Chassis Status pane shows the Power State as Off for the server node that you are removing.
The physical power button on the server node face also turns amber when it is safe to remove the server node from the chassis.
Shutting Down a Server Node By Using the Power Button on the Server Node
To use the physical server node power button to shut down the server node only, follow these steps:
1.
Check the color of the server node power status LED (see
):
Green—The server node is powered on. Go to step 2.
Amber—the server node is powered off. It is safe to remove the server node from the chassis.
2.
Invoke either a graceful shutdown or a hard shutdown:
CAUTION
: To avoid data loss or damage to your operating system, you should always invoke a graceful shutdown of the operating system.
Do not power off a server if any firmware or BIOS updates are in progress.
Graceful shutdown—Press and release the Power button. The software performs a graceful shutdown of the server node.
Emergency shutdown—Press and hold the Power button for 4 seconds to force the power off the server node.
When the server node power button turns amber, it is safe to remove the server node from the chassis.
6
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Removing or Installing a C3X60 M4 Server Node
C3X60 M4 Server Node Population Rules
Observe the following rules:
Do not mix a C3X60 M4 server node and a C3X60 M3 server node in the same Cisco UCS S3260 system. An M4 server node can be
identified by the “M4 SVRN” label on the rear panel (see Figure 1
).
Single-node systems:
—
Cisco IMC releases earlier than 2.0(13): If your S3260 system has only one server node, it must be installed in bay 1.
—
Cisco IMC releases 2.0(13) and later: If your S3260 system has only one server node, it can be installed in either server bay.
NOTE:
Whichever bay a server node is installed to, it must have a corresponding SIOC. That is, a server node in bay 1 must be paired with a SIOC in SIOC slot 1; a server node in bay 2 must be paired with a SIOC in SIOC bay 2.
7
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Removing or Installing a C3X60 M4 Server Node
Replacing a C3X60 M4 Server Node
The server node is accessed from the rear of the system, so you do not have to pull the system out from the rack.
CAUTION:
Before you replace a server node, export and save the Cisco IMC configuration from the node if you want that same configuration on the new node. You can import the saved configuration to the new replacement node after you install it.
1.
Optional—Export the Cisco IMC configuration from the server node that you are replacing so that you can import it to the replacement
to the next step.
You do not have to power off the chassis in the next step. Replacement with chassis powered on is supported if you shut down the server node before removal.
2.
Shut down the server node by using the software interface or by pressing the node power button, as described in
C3X60 M4 Server Node, page 6 .
3.
Remove the server node from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers (see
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
4.
If the server node has an I/O expander attached, use the procedure in
If there is no I/O expander, continue with the next step.
5.
Install a new server node:
a.
With the two ejector levers open, align the new server node with the empty bay. Note these configuration rules:
—
Cisco IMC releases earlier than 2.0(13): If your S3260 system has only one server node, it must be installed in bay 1
—
Cisco IMC releases 2.0(13) and later: If your S3260 system has only one server node, it can be installed in either server bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
6.
Power on the server node.
7.
Perform initial setup on the new server node to assign an IP address and your other preferred network settings. See Initial System Setup in the Cisco UCS S3260 Storage Server Installation and Service Guide .
8.
IMC Configuration to a Server Node, page 9
.
8
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Removing or Installing a C3X60 M4 Server Node
Exporting Cisco IMC Configuration From a Server Node
This operation can be performed using either the GUI or CLI interface of the Cisco IMC. The example in this procedure uses the CLI commands. For more information see Exporting a Cisco IMC Configuration in the CLI and GUI guides here: Configuration Guides .
1.
Log in to the IP address and CLI interface of the server node that you are replacing.
2.
Enter the following commands as you are prompted:
Server# scope cimc
Server /cimc# scope import-export
Server /cimc/import-export# export-config <protocol> <ip-address> <path-and-filename>
3.
Enter the user name, password, and pass phrase.
This sets the user name password, and pass phrase for the file that you are exporting. The export operation begins after you enter a pass phrase, which can be anything that you choose.
To determine whether the export operation has completed successfully, use the show detail command. To abort the operation, type
CTRL+C.
The following is an example of an export operation. In this example, the TFTP protocol is used to export the configuration to IP address
192.0.2.34, in file /ucs/backups/cimc5.xml.
Server# scope cimc
Server /cimc # scope import-export
Server /cimc/import-export # export-config tftp 192.0.2.34 /ucs/backups/cimc5.xml
Username:xxxx
Password:****
Passphrase:***
Export config started. Please check the status using "show detail".
Server /cimc/import-export # show detail
Import Export:
Operation: EXPORT
Status: COMPLETED
Error Code: 100 (No Error)
Diagnostic Message: NONE
Importing Cisco IMC Configuration to a Server Node
This operation can be performed using either the GUI or CLI interface of the Cisco IMC. The example in this procedure uses the CLI commands. For more information see Importing a Cisco IMC Configuration in the CLI and GUI guides here: Configuration Guides .
1.
SSH into the CLI interface of the new server node.
2.
Enter the following commands as you are prompted:
Server# scope cimc
Server /cimc# scope import-export
Server /cimc/import-export# import-config <protocol> <ip-address> <path-and-filename>
3.
Enter the user name, password, and pass phrase.
This should be the user name, password, and pass phrase that you used during the export operation. The import operation begins after you enter the pass phrase.
9
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Removing or Installing a C3X60 M4 Server Node
The following is an example of an import operation. In this example, the TFTP protocol is used to import the configuration to the server node from IP address 192.0.2.34, from file /ucs/backups/cimc5.xml.
Server# scope cimc
Server /cimc # scope import-export
Server /cimc/import-export # import-config tftp 192.0.2.34 /ucs/backups/cimc5.xml
Username:xxxx
Password:****
Passphrase:***
Export config started. Please check the status using "show detail".
Server /cimc/import-export # show detail
Import Export:
Operation: Import
Status: COMPLETED
Error Code: 100 (No Error)
Diagnostic Message: NONE
10
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing C3X60 M4 Server Node and I/O Expander Components
Removing a C3X60 M4 Server Node or I/O Expander Top Cover, page 12
Removing an I/O Expander From a C3X60 M4 Server Node, page 14
C3X60 M4 Internal Diagnostic LEDs, page 19
Replacing DIMMs Inside the C3X60 M4 Server Node, page 20
Replacing CPUs and Heatsinks Inside the C3X60 M4 Server Node, page 23
Replacing an RTC Battery Inside the Server Node, page 28
Replacing an NVMe SSD Inside the C3X60 M4 Server Node, page 29
Installing a Trusted Platform Module (TPM) Inside the C3X60 M4 Server Node, page 31
Replacing a Storage Controller Card Inside the C3X60 M4 Server Node, page 34
Replacing an I/O Expander, page 37
Replacing an I/O Expander, page 37
Replacing a PCIe Card Inside an I/O Expander, page 41
Replacing a Storage Controller Card Inside the I/O Expander, page 43
Replacing an NVMe SSD Inside the I/O Expander, page 45
11
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Removing a C3X60 M4 Server Node or I/O Expander Top Cover
The optional I/O expander and the server node use the same top cover. If an I/O expander is attached to the top of the server node, the top cover is on the I/O expander, as shown in the side view in
Figure 4 . In this case, there is also an intermediate cover between the server node
and the I/O expander.
Figure 4 Side View, Server Node With Attached I/O Expander
1 Top cover
In this view, the top cover is on the attached I/O expander.
2 Cover securing screws on sides of the top cover
(two on each side)
3 Intermediate cover
In this view, the intermediate cover is attached to the server node.
4 Cover securing screws on sides of the intermediate cover
(two on each side)
NOTE:
You do not have to slide the system out of the rack to remove the server node from the rear of the system.
1.
Shut down the server node by using the software interface or by pressing the node power button, as described in
C3X60 M4 Server Node, page 6 .
2.
Remove the server node from the system (or server node with attached I/O expander, if present):
a.
Grasp the two ejector levers and pinch their latches to release the levers (see
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
3.
Remove the top cover from the server node or the I/O expander (if present).
a.
Remove the four screws that secure the top cover and set them aside. There are two screws on each side of the top cover (see
).
b.
Lift the latch handle to an upright position (see Figure 5 ).
c.
Turn the latch handle 90 degrees to release the lock.
d.
Slide the cover toward the rear (toward the rear-panel buttons) and then lift it from the server node or I/O expander (if present).
12
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
4.
Reinstall the top cover:
a.
Set the cover in place on the server node or I/O expander (if present), offset about one inch toward the rear. Pegs on the inside of the cover must engage the tracks on the server node or I/O expander base.
b.
Push the cover forward until it stops.
c.
Turn the latch handle 90 degrees to close the lock.
d.
Fold the latch handle flat.
e.
Reinstall the four securing screws on the sides of the top cover.
5.
Reinstall a server node (or server node with attached I/O expander, if present):
a.
With the two ejector levers open, align the new server node with the empty bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
6.
Power on the server node.
Figure 5 Cisco UCS C3X60 M4 Server Node or I/O Expander Top Cover
1 Server node or I/O expander top cover
2 Locations of four securing screws on sides of top cover
(not visible in this view)
3 Latch handle (shown in closed, flat position)
13
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Removing an I/O Expander From a C3X60 M4 Server Node
This sections in this topic describes how to remove an I/O expander and intermediate cover from a C3X60 M4 server node so that you can access the components inside the server node.
Disassembling the Server Node/ I/O Expander Assembly, page 14
Reassembling the Server Node/ I/O Expander Assembly, page 16
NOTE:
You do not have to power off the chassis in the next step. Replacement with the chassis powered on is supported if you shut down the server node before removal.
Disassembling the Server Node/ I/O Expander Assembly
1.
Shut down the server node by using the software interface or by pressing the node power button, as described in
C3X60 M4 Server Node, page 6 .
2.
Remove the server node with attached I/O expander from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers (see
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node with attached I/O expander straight out from the system.
3.
Remove the top cover from the I/O expander:
a.
Remove the four screws that secure the top cover and set them aside. There are two screws on each side of the top cover (see
).
b.
Lift the latch handle to an upright position (see Figure 5 ).
c.
Turn the latch handle 90 degrees to release the lock.
d.
Slide the cover toward the rear (toward the rear-panel buttons) and then lift it from the I/O expander.
4.
Remove the I/O expander from the server node:
a.
Remove the five screws that secure the I/O expander to the top of the server node (see Figure 6 ).
Figure 6 I/O Expander Securing Screws (Five)
14
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
b.
Use two small flat-head screwdrivers (1/4-inch or equivalent) to help separate the connector on the underside of the I/O expander from the socket on the server node board.
Insert a screwdriver about 1/2-inch into the “REMOVAL SLOT” that is marked with an arrow on each side of the I/O expander (see
). Then lift up evenly on both screwdrivers at the same time to separate the connectors and lift the I/O expander about 1/2-inch.
c.
Grasp the two handles on the I/O expander board and lift it straight up.
Figure 7 Separating the I/O Expander From the C3X60 M4 Server Node
1 Side view showing removal slot for screwdriver insertion
(one each side of I/O expander)
2 Rear view of server node with I/O expander
5.
Remove the intermediate cover from the server node:
a.
Remove the four screws that secure the intermediate cover and set them aside. There are two screws on each side of the intermediate cover (see
b.
Slide the intermediate cover toward the rear (toward the rear-panel buttons) and then lift it from the server node.
15
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Reassembling the Server Node/ I/O Expander Assembly
1.
Reinstall the intermediate cover to the server node:
a.
Set the intermediate cover in place on the server node, offset about one inch toward the rear. Pegs on the inside of the cover must engage the tracks on the server node base.
b.
Push the cover forward until it stops.
c.
Reinstall the four screws that secure the intermediate cover.
2.
Reinstall the I/O expander to the server node:
CAUTION:
Use caution to align all features of the I/O expander with the intermediate cover and server node before mating the connector on the underside of the expander with the socket on the server board. The connector can be damaged if correct alignment is not used.
a.
Carefully align the I/O expander with the alignment pegs on the top of the intermediate cover (see
).
b.
Set the I/O expander down on the intermediate cover and lower it gently to mate the mezzanine connector and the socket on the server board.
Figure 8 Replacing the I/O Expander to the M4 Server Node
1 Intermediate cover on server node
2 I/O expander mezzanine connector
3 Alignment pegs on intermediate cover
16
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
c.
If a RAID controller is present or an NVMe SSD is present in the right-hand socket (IOENVMe2) of the I/O expander, you must remove them to access the PRESS HERE plate in the next step.
See
Replacing a Storage Controller Card Inside the I/O Expander, page 43
and
Replacing an NVMe SSD Inside the I/O Expander, page 45
to remove them, then return to the next step.
d.
Press down firmly on the plastic plate marked “PRESS HERE” to fully seat the connectors (see
Figure 9 I/O Expander, Showing “PRESS HERE” Plate
e.
If you removed a RAID controller or an NVMe SSD to access the PRESS HERE plate, reinstall them now.
See
Replacing a Storage Controller Card Inside the I/O Expander, page 43
and
Replacing an NVMe SSD Inside the I/O Expander, page 45
.
CAUTION:
Before you reinstall the I/O expander securing screws, you must use the supplied alignment tool (UCSC-C3K-M4IOTOOL) in the next step to ensure alignment of the connectors that plug into the internal chassis backplane. Failure to ensure alignment might damage the sockets on the backplane.
3.
Insert the four pegs of the alignment tool into the holes that are built into the connector side of the server node and I/O expander. Ensure that the alignment tool fits into all four holes and lies flat (see
The alignment tool is shipped with systems that are ordered with an I/O expander. It is also shipped with I/O expander replacement spares. You can order the tool using Cisco PID UCSC-C3K-M4IOTOOL.
17
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Figure 10 Using the I/O Expander Alignment Tool
1 Alignment tool
2 Connector side of server node and I/O expander
3 Alignment tool installed
4.
5.
Remove the alignment tool.
6.
Reinstall the I/O expander top cover:
a.
Set the cover in place on the I/O expander, offset about one inch toward the rear. Pegs on the inside of the cover must engage the tracks on the I/O expander base.
b.
Push the cover forward until it stops.
c.
Turn the latch handle 90 degrees to close the lock.
d.
Fold the latch handle flat.
e.
Reinstall the four securing screws on the sides of the cover (see
).
7.
Install a server node with attached I/O expander to the chassis:
a.
With the two ejector levers open, align the server node and I/O expander with the two empty bays.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
8.
Power on the server node.
18
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
C3X60 M4 Internal Diagnostic LEDs
There are internal diagnostic LEDs on the edge of the server node board. These LEDs can be viewed while the server node is removed from the chassis, up to 30 seconds after AC power is removed.
1.
NOTE:
You do not have to remove the server node cover to view the LEDs on the edge of the board.
2.
system.
A fault LED that lights amber indicates a faulty component.
Figure 11 Cisco UCS C3X60 M4 Server Node Internal Diagnostic LEDs
1 Unit identification button
19
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing DIMMs Inside the C3X60 M4 Server Node
There are 16 DIMM sockets on the server node board.
CAUTION:
DIMMs and their sockets are fragile and must be handled with care to avoid damage during installation.
CAUTION:
Cisco does not support third-party DIMMs. Using non-Cisco DIMMs in the system might result in system problems or damage to the motherboard.
NOTE:
To ensure the best system performance, it is important that you are familiar with memory performance guidelines and population rules before you install or replace the memory.
DIMM Performance Guidelines and Population Rules
DIMM Population Rules, page 21
Memory Mirroring Mode, page 21
Lockstep Channel Mode, page 21
DIMM Sockets
shows the DIMM sockets and how they are numbered on a C3X60 M4 server node board.
A server node has 16 DIMM sockets (8 for each CPU).
Channels are labeled with letters as shown in
For example, channel A = DIMM sockets A1, A2.
Each channel has two DIMM sockets. The blue socket in a channel is always socket 1.
Figure 12 Cisco UCS C3X60 M4 DIMM and CPU Numbering
20
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
DIMM Population Rules
Observe the following guidelines when installing or replacing DIMMs:
For optimal performance, spread DIMMs evenly across both CPUs and all channels.
Populate the DIMM sockets of each CPU identically. Populate the blue DIMM 1 sockets first, then the black DIMM 2 slots. For example, populate the DIMM slots in this order:
1.
A1, E1, B1, F1, C1, G1, D1, H1
2.
A2, E2, B2, F2, C2, G2, D2, H2
Observe the DIMM mixing rules shown in Table 3 .
Table 3 DIMM Mixing Rules
DIMM Parameter
DIMM Capacity:
RDIMM = 16 or 32 GB
DIMM Speed:
2133- or 2400-MHz
DIMM Type:
RDIMMs
DIMMs in the Same Channel
You can mix different capacity DIMMs in the same channel (for example, A1, A2).
DIMMs in the Same Bank
You can mix different capacity DIMMs in the same bank. However, for optimal performance DIMMs in the same bank (for example, A1, B1, C1, D1) should have the same capacity.
You can mix speeds, but DIMMs will run at the speed of the slowest DIMMs/CPUs installed in the channel.
You can mix speeds, but DIMMs will run at the speed of the slowest DIMMs/CPUs installed in the bank.
You cannot mix DIMM types in a channel.
You cannot mix DIMM types in a bank.
Memory Mirroring Mode
When you enable memory mirroring mode, the memory subsystem simultaneously writes identical data to two channels. If a memory read from one of the channels returns incorrect data due to an uncorrectable memory error, the system automatically retrieves the data from the other channel. A transient or soft error in one channel does not affect the mirrored data, and operation continues.
Memory mirroring reduces the amount of memory available to the operating system by 50 percent because only one of the two populated channels provides data.
Lockstep Channel Mode
When you enable lockstep channel mode, each memory access is a 128-bit data access that spans four channels.
Lockstep channel mode requires that all four memory channels on a CPU must be populated identically with regards to size and organization. DIMM socket populations within a channel do not have to be identical but the same DIMM slot location across all four channels must be populated the same.
For example, DIMMs in sockets A1, B1, C1, and D1 must be identical. DIMMs in sockets A2, B2, C2, and D2 must be identical. However, the A1-B1-C1-D1 DIMMs do not have to be identical with the A2-B2-C2-D2 DIMMs.
21
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing DIMMs
1.
Shut down the server node by using the software interface or by pressing the node power button, as described in
C3X60 M4 Server Node, page 6 .
2.
Remove a server node from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers (see
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
3.
Do one of the following to access the component inside the server node:
Server Node or I/O Expander Top Cover, page 12 before continuing with the next step.
If the server node has an I/O expander attached—Remove the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14
before you continue with the next step.
4.
Locate the faulty DIMM and remove it from the socket on the riser by opening the ejector levers at both ends of the DIMM socket.
5.
Install a new DIMM:
NOTE:
a.
Align the new DIMM with the socket on the riser. Use the alignment key in the DIMM socket to correctly orient the DIMM.
b.
Push the DIMM into the socket until it is fully seated and the ejector levers on either side of the socket lock into place.
6.
Do one of the following:
If the server node did not have an I/O expander attached—Reinstall the server node cover as described in
Server Node or I/O Expander Top Cover, page 12 before continuing with the next step.
If the server node had an I/O expander attached—Reinstall the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14
before you continue with the next step.
7.
Install a server node to the chassis:
a.
With the two ejector levers open, align the new server node with the empty bay.
—
Cisco IMC releases earlier than 2.0(13): If your S3260 system has only one server node, it must be installed in bay 1
—
Cisco IMC releases 2.0(13) and later: If your S3260 system has only one server node, it can be installed in either server bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
8.
Power on the server node.
22
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing CPUs and Heatsinks Inside the C3X60 M4 Server Node
Two CPUs are inside each server node. Although CPUs are not spared separately for this server node, you might need to move your CPUs from a faulty server node to a new server node.
CPU Configuration Rules
A server node must have two CPUs to operate. See
Figure 12 for the C3X60 M4 server node CPU numbering.
Replacing CPUs and Heatsinks
CAUTION:
CPUs and their motherboard sockets are fragile and must be handled with care to avoid damaging pins during installation. The
CPUs must be installed with heatsinks and their thermal pads to ensure proper cooling. Failure to install a CPU correctly might result in damage to the system.
NOTE:
This server uses the new independent loading mechanism (ILM) CPU sockets, so no Pick-and-Place tools are required for CPU handling or installation. Always grasp the plastic frame on the CPU when handling.
1.
.
2.
Remove a server node from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers (see Figure 1
).
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
3.
Do one of the following to access the component inside the server node:
If the server node does not have an I/O expander attached—Remove the server node cover as described in
Server Node or I/O Expander Top Cover, page 12
before continuing with the next step.
If the server node has an I/O expander attached—Remove the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14 before you continue with the next step.
4.
Use a Number 2 Phillips-head screwdriver to loosen the four captive screws that secure the heatsink, and then lift it off of the CPU.
NOTE:
Alternate loosening each screw evenly to avoid damaging the heatsink or CPU.
5.
Open the CPU retaining mechanism:
a.
Unclip the first retaining latch labeled with the
.
b.
Open the hinged CPU cover plate.
icon, and then unclip the second retaining latch labeled with the
23
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Figure 13 CPU Socket
1 5
2
3
4
1
CPU retaining latch
2
CPU retaining latch
3
Hinged CPU cover plate
4
5
Hinged CPU seat
Finger-grips on plastic CPU frame
6.
Remove any existing CPU:
a.
.
b.
Grasp the CPU by the finger-grips on its plastic frame and lift it up and out of the hinged CPU seat.
c.
Set the CPU aside on an antistatic surface.
7.
Install a new CPU:
a.
Grasp the new CPU by the finger-grips on its plastic frame and align the tab on the frame that is labeled “ALIGN” with the hinged
b.
Insert the tab on the CPU frame into the seat until it stops and is held firmly.
The line below the word “ALIGN” should be level with the edge of the seat, as shown in
c.
Swing the hinged seat with the CPU down until the CPU frame clicks in place and holds flat in the socket.
d.
Close the hinged CPU cover plate.
e.
Clip down the CPU retaining latch with the
icon, and then clip down the CPU retaining latch with the icon. See
24
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Figure 14 CPU and Socket Alignment Features
2
1
1
SLS mechanism on socket
2
Tab on CPU frame (labeled ALIGN)
8.
Install a heatsink:
CAUTION:
The heatsink must have a new thermal grease on the heatsink-to-CPU surface to ensure proper cooling. New heatsinks have a pre-applied pad of grease. If you are reusing a heatsink, you must remove the old thermal grease and apply grease from a syringe.
a.
Do one of the following:
—
If you are installing a new heatsink, remove the protective film from the pre-applied pad of thermal grease on the bottom of the new heatsink. Then continue with step
—
If you are reusing a heatsink, continue with step
b.
Apply an alcohol-based cleaning solution to the old thermal grease and let it soak for a least 15 seconds.
c.
Wipe all of the old thermal grease off the old heatsink using a soft cloth that will not scratch the heatsink surface.
d.
Apply thermal grease from the syringe that is included with the new CPU to the top of the CPU. Apply about half the syringe
contents to the top of the CPU in the pattern that is shown in Figure 15
.
NOTE:
If you do not have a syringe of thermal grease, you can order a spare (Cisco PID UCS-CPU-GREASE3).
Figure 15 CPU Thermal Grease Application Pattern
25
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
9.
Align the heatsink captive screws with the motherboard standoffs, and then use a Number 2 Phillips-head screwdriver to tighten the captive screws evenly.
CAUTION:
Alternate tightening each screw evenly to avoid damaging the heatsink or CPU.
10.
Do one of the following:
If the server node did not have an I/O expander attached—Reinstall the server node cover as described in
Server Node or I/O Expander Top Cover, page 12 before continuing with the next step.
If the server node had an I/O expander attached—Reinstall the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14
before you continue with the next step.
11.
Install a server node:
a.
With the two ejector levers open, align the new server node with the empty bay.
—
Cisco IMC releases earlier than 2.0(13): If your S3260 system has only one server node, it must be installed in bay 1
—
Cisco IMC releases 2.0(13) and later: If your S3260 system has only one server node, it can be installed in either server bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
12.
Power on the server node.
26
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Additional CPU-Related Parts To Order With RMA Replacement Server Nodes
When a return material authorization (RMA) of the server node or CPU is done on a system, there are additional parts that might not be included with the CPU or motherboard spare bill of materials (BOM). The TAC engineer might need to add the additional parts to the RMA to help ensure a successful replacement.
Scenario 1—You are re-using the existing heatsinks:
—
Heat sink cleaning kit (UCSX-HSCK=)
—
Thermal grease kit for S3260 server nodes (UCS-CPU-GREASE3=)
—
Intel CPU Pick-n-Place tool for EP CPUs (UCS-CPU-EP-PNP=)
Scenario 2—You are replacing the existing heatsinks:
—
Heat sink (UCSB-HS-01-EP=)
—
Heat sink cleaning kit (UCSX-HSCK=)
—
Intel CPU Pick-n-Place tool for EP CPUs (UCS-CPU-EP-PNP=)
A CPU heatsink cleaning kit is good for up to four CPU and heatsink cleanings. The cleaning kit contains two bottles of solution, one to clean the CPU and heatsink of old thermal interface material and the other to prepare the surface of the heatsink.
It is important to clean the old thermal interface material off of the CPU prior to installing the heatsinks. Therefore, when ordering new heatsinks it is still necessary to order the heatsink cleaning kit at a minimum.
27
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing an RTC Battery Inside the Server Node
The real-time clock (RTC) battery retains system settings when the server is disconnected from power. The battery type is CR2032. Cisco supports the industry-standard CR2032 battery, which can be purchased from most electronic stores.
NOTE:
When the RTC battery is removed or it completely loses power, settings that were stored in the BMC of the server node are lost.
You must reconfigure the BMC settings after installing a new battery.
1.
Shut down the server node by using the software interface or by pressing the node power button, as described in
C3X60 M4 Server Node, page 6 .
2.
Remove a server node from the system:
a.
Grasp the two ejector levers on the node and pinch their latches to release the levers.
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
NOTE:
You do not have to remove the server node cover or an I/O expander to access the RTC battery.
3.
Remove the server node RTC battery:
a.
Locate the RTC battery. See
b.
Pull the battery retaining clip away from the battery and pull the battery from the socket.
4.
Install the new RTC battery:
a.
Pull the retaining clip away from the battery socket and insert the battery in the socket.
NOTE:
The flat, positive side of the battery marked “+” should face the retaining clip.
b.
Push the battery into the socket until it is fully seated and the retaining clip clicks over the top of the battery.
5.
Install a server node:
a.
With the two ejector levers open, align the new server node with the empty bay.
—
Cisco IMC releases earlier than 2.0(13): If your S3260 system has only one server node, it must be installed in bay 1
—
Cisco IMC releases 2.0(13) and later: If your S3260 system has only one server node, it can be installed in either server bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
6.
Power on the server node.
7.
Reconfigure the BMC settings for this node.
28
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Figure 16 RTC Battery Inside the C3X60 M4 Server Node
1 Server node RTC battery on board, below edge of cover
Replacing an NVMe SSD Inside the C3X60 M4 Server Node
The SBNVMe1 socket inside each C3X60 M4 server node can support a single NVMe PCIe SSD. The SSD might be under a storage controller card, if one is installed in the server node.
The software designation of the SSD socket inside the server node is SBNVMe1.
NOTE:
2.5-inch form-factor NVMe SSDs are bootable in UEFI mode; legacy booting is not supported.
1.
.
2.
Remove a server node from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers.
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
3.
Do one of the following to access the component inside the server node:
If the server node does not have an I/O expander attached—Remove the server node cover as described in
Server Node or I/O Expander Top Cover, page 12
before continuing with the next step.
If the server node has an I/O expander attached—Remove the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14 before you continue with the next step.
4.
If a storage controller is present in the server node, remove it to provide clearance:
a.
Loosen the four captive thumbscrews that secure the card to the server node (see Figure 19
).
b.
Grasp the card at both ends and lift it evenly to disengage the connector on the underside of the card from the mezzanine socket.
5.
Remove the SSD:
29
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
a.
Remove the single screw that secures the drive to its bracket (see Figure 17
).
b.
Slide the drive to disengage it from its horizontal socket, then lift it from the server node.
6.
Install a new SSD:
a.
Set the drive in its bracket on the server board, then slide it forward to engage its connector with the socket.
b.
Install the single screw that secures the drive to the bracket.
7.
If you removed a storage controller card, reinstall it:
a.
Align the card with bracket over the mezzanine socket and the four standoffs.
b.
Press down on both ends of the card to engage the connector on the underside of the card with the mezzanine socket.
c.
Tighten the four captive thumbscrews that secure the card to the server node.
8.
Do one of the following:
If the server node did not have an I/O expander attached—Reinstall the server node cover as described in
Server Node or I/O Expander Top Cover, page 12 before continuing with the next step.
If the server node had an I/O expander attached—Reinstall the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14
before you continue with the next step.
9.
Install a server node:
a.
With the two ejector levers open, align the new server node with the empty bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
10.
Power on the server node.
Figure 17 SSD Inside the C3X60 M4 Server Node
1 Single securing screw 2 SSD socket SBNVMe1
30
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Installing a Trusted Platform Module (TPM) Inside the C3X60 M4 Server Node
The trusted platform module (TPM) is a small circuit board that attaches to a socket on the server node board.
This section contains the following procedures, which must be followed in this order when installing and enabling a TPM:
1.
Installing the TPM Hardware, page 31
2.
Enabling TPM Support in the BIOS, page 32
3.
Enabling the Intel TXT Feature in the BIOS, page 33
TPM 2.0 Considerations
If there is no existing TPM in the server, you can install TPM 2.0. TPM 2.0 requires Intel v4 code or later.
CAUTION:
If your M4 server node Intel v4 system is currently supported and protected by TPM version 2.0, a potential security exposure might occur if you downgrade the system software and BIOS to an earlier version.
NOTE:
If the TPM 2.0 becomes unresponsive, reboot the server.
Installing the TPM Hardware
NOTE:
For security purposes, the TPM is installed with a one-way screw. It cannot be removed with a standard screwdriver.
1.
.
2.
Remove a server node from the system:
a.
Grasp the two ejector levers on the node and pinch their latches to release the levers.
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
NOTE:
You do not have to remove the server node cover or an I/O expander to access the TPM socket.
3.
Install a TPM:
a.
Locate the TPM socket on the server node board, as shown in Figure 18
.
b.
Align the connector that is on the bottom of the TPM circuit board with the TPM socket. Align the screw hole on the TPM board with the screw hole adjacent to the TPM socket.
c.
Push down evenly on the TPM to seat it in the motherboard socket.
d.
Install the single one-way screw that secures the TPM to the motherboard.
4.
Install a server node:
a.
With the two ejector levers open, align the new server node with the empty bay.
—
Cisco IMC releases earlier than 2.0(13): If your S3260 system has only one server node, it must be installed in bay 1
—
Cisco IMC releases 2.0(13) and later: If your S3260 system has only one server node, it can be installed in either server bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
31
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
5.
Power on the server node.
6.
Continue with Enabling TPM Support in the BIOS, page 32
.
Figure 18 TPM Location Inside the C3X60 M4 Server Node
1 TPM socket location on server board
Enabling TPM Support in the BIOS
NOTE:
After TPM hardware installation, you must enable TPM support in the BIOS:
1.
Enable TPM support:
d.
Watch during bootup for the F2 prompt, and then press F2 to enter BIOS setup.
e.
Log in to the BIOS Setup Utility with your BIOS Administrator password.
f.
On the BIOS Setup Utility window, choose the Advanced tab.
g.
Choose Trusted Computing to open the TPM Security Device Configuration window.
h.
Change TPM SUPPORT to Enabled.
i.
Press F10 to save your settings and reboot the server node.
2.
Verify that TPM support is now enabled:
a.
Watch during bootup for the F2 prompt, and then press F2 to enter BIOS setup.
b.
Log into the BIOS Setup utility with your BIOS Administrator password.
c.
Choose the Advanced tab.
d.
Choose Trusted Computing to open the TPM Security Device Configuration window.
e.
Verify that TPM SUPPORT and TPM State are Enabled.
3.
Continue with Enabling the Intel TXT Feature in the BIOS, page 33
.
32
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Enabling the Intel TXT Feature in the BIOS
Intel Trusted Execution Technology (TXT) provides greater protection for information that is used and stored on the business server. A key aspect of that protection is the provision of an isolated execution environment and associated sections of memory where operations can be conducted on sensitive data, invisibly to the rest of the system. Intel TXT provides for a sealed portion of storage where sensitive data such as encryption keys can be kept, helping to shield them from being compromised during an attack by malicious code.
1.
Reboot the server node and watch for the prompt to press F2.
2.
When prompted, press F2 to enter the BIOS Setup utility.
3.
Verify that the prerequisite BIOS values are enabled:
a.
Choose the Advanced tab.
b.
Choose Intel TXT(LT-SX) Configuration to open the Intel TXT(LT-SX) Hardware Support window.
c.
Verify that the following items are listed as Enabled:
—
VT-d Support (default is Enabled)
—
VT Support (default is Enabled)
—
TPM Support
—
TPM State
If VT-d Support and VT Support are already enabled, skip to
.
If VT-d Support and VT Support are not enabled, continue with the next steps to enable them.
a.
Press Escape to return to the BIOS Setup utility Advanced tab.
b.
On the Advanced tab, choose Processor Configuration to open the Processor Configuration window.
c.
Set Intel (R) VT and Intel (R) VT-d to Enabled.
4.
Enable the Intel Trusted Execution Technology (TXT) feature:
a.
Return to the Intel TXT(LT-SX) Hardware Support window if you are not already there.
b.
Set TXT Support to Enabled.
5.
Press F10 to save your changes and exit the BIOS Setup utility.
33
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing a Storage Controller Card Inside the C3X60 M4 Server Node
The Cisco storage controller card connects to a mezzanine-style socket inside the server node. The supercap power module (SCPM) comes already attached to a new RAID card, so you do not have to remove it separately.
NOTE:
Do not mix different storage controllers in the same system. If the system has two server nodes, they must both contain the same controller.
NOTE:
server node.
1.
Shut down the server node by using the software interface or by pressing the node power button, as described in
C3X60 M4 Server Node, page 6 .
2.
Remove a server node from the system:
a.
Grasp the two ejector levers on the node and pinch their latches to release the levers.
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
3.
Do one of the following to access the component inside the server node:
Server Node or I/O Expander Top Cover, page 12 before continuing with the next step.
If the server node has an I/O expander attached—Remove the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14
before you continue with the next step.
4.
Remove a storage controller card:
a.
Loosen the captive thumbscrews that secure the card to the board (see
for RAID card UCS-C3K-M4RAID or Figure 20
for HBA card UCS-C3K-M4DHBA).
b.
Grasp the card at both ends and lift it evenly to disengage the connector on the underside of the card from the mezzanine socket.
5.
Install a storage controller card:
NOTE
: If you are installing Cisco UCS S3260 Dual Pass-Through Controller (UCS-C3K-M4DHBA), it is supported in an M4 server node only; it is not supported in the I/O expander or in M3 server nodes.
NOTE
: If you are installing Cisco UCS S3260 Dual Pass-Through Controller (UCS-C3K-M4DHBA), it requires chassis part number
68-5286-06 or later. The chassis motherboard in earlier chassis versions does not support this controller. You can determine the chassis part number by looking on the part-number label on the top-front of the chassis, or by using the inventory-all command, as shown in the following example using the Cisco IMC CLI:
Server# scope chassis
Server/chassis# inventory-refresh
Server/chassis# inventory-all
CHS_FRU(ID1)
Board Part Number : 73-16125-03
Product Manufacturer : Cisco Systems Inc
Product Part Number : 68-5286-07
34
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
a.
Align the card over the mezzanine socket and the three standoffs.
b.
Press down on both ends of the card to engage the connector on the underside of the card with the mezzanine socket.
c.
Tighten the captive screws that secure the card to the board (see Figure 19
).
6.
Do one of the following:
Server Node or I/O Expander Top Cover, page 12
before continuing with the next step.
If the server node had an I/O expander attached—Reinstall the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14 before you continue with the next step.
7.
Install a server node:
a.
With the two ejector levers open, align the new server node with the empty bay.
—
Cisco IMC releases earlier than 2.0(13): If your S3260 system has only one server node, it must be installed in bay 1
—
Cisco IMC releases 2.0(13) and later: If your S3260 system has only one server node, it can be installed in either server bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
8.
Power on the server node.
9.
See
Restoring RAID Configuration After Replacing a RAID Controller, page 52
to restore your RAID configuration.
Figure 19 RAID Controller Card (UCS-C3K-M4RAID) Thumbscrews Inside the C3X60 M4 Server Node
35
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Figure 20 HBA Controller Card (UCS-C3K-M4DHBA) Thumbscrews Inside the C3X60 M4 Server Node
36
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing an I/O Expander
The server node with optional I/O expander is accessed from the rear of the system, so you do not have to pull the system out from the rack.
NOTE:
The I/O expander is field-replaceable. However, upgrading a server node to add an I/O expander after-factory is not supported.
NOTE:
You do not have to power off the chassis in the next step. Replacement with the chassis powered on is supported if you shut down the server node before removal.
1.
.
2.
Remove the server node with attached I/O expander from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers (see Figure 1
).
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node with I/O expander straight out from the system.
3.
Remove the top cover from the I/O expander:
a.
Remove the four screws that secure the top cover and set them aside. There are two screws on each side of the top cover (see
b.
Lift the latch handle to an upright position (see
).
c.
Turn the latch handle 90 degrees to release the lock.
d.
Slide the cover toward the rear (toward the rear-panel buttons) and then lift it from the I/O expander.
4.
Remove the I/O expander from the server node:
a.
Remove the five screws that secure the I/O expander to the top of the server node (see
).
Figure 21 I/O Expander Securing Screws (Five)
37
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
b.
Use two small flat-head screwdrivers (1/4-inch or equivalent) to help separate the connector on the underside of the I/O expander from the socket on the server node board.
Insert a screwdriver about 1/2-inch into the “REMOVAL SLOT” that is marked with an arrow on each side of the I/O expander (see
). Then lift up evenly on both screwdrivers at the same time to separate the connectors and lift the I/O expander about
1/2-inch.
c.
Grasp the two handles on the I/O expander board and lift it straight up.
Figure 22 Separating the I/O Expander From the M4 Server Node
1 Side view showing removal slot for screwdriver insertion
(one each side of I/O expander)
2 Rear view of server node with I/O expander
5.
Reinstall the I/O expander to the server node:
CAUTION:
Use caution to align all features of the I/O expander with the server node before mating the connector on the underside of the expander with the socket on the server board. The connector can be damaged if correct alignment is not used.
a.
Carefully align the I/O expander with the alignment pegs on the top of the intermediate cover (see
b.
Set the I/O expander down on the server node intermediate cover and push down gently to mate the connectors.
38
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Figure 23 Replacing the I/O Expander to the M4 Server Node
1 I/O expander top cover
2 I/O expander mezzanine connector
3 Server node intermediate cover
4 Server node intermediate cover screws
(two each side)
5 Alignment pegs on intermediate cover
c.
If a RAID controller is present or an NVMe SSD is present in the right-hand socket (IOENVMe2) of the I/O expander, you must remove them to access the PRESS HERE plate in the next step.
See
Replacing a Storage Controller Card Inside the I/O Expander, page 43
and
Replacing an NVMe SSD Inside the I/O Expander, page 45
to remove them, then return to the next step.
d.
Press down firmly on the plastic plate marked “PRESS HERE” to fully seat the connectors (see
Figure 24 I/O Expander, Showing “PRESS HERE” Plate
e.
If you removed a RAID controller or an NVMe SSD to access the PRESS HERE plate, reinstall them now.
39
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
See Replacing a Storage Controller Card Inside the I/O Expander, page 43 and
Replacing an NVMe SSD Inside the I/O Expander, page 45 .
CAUTION:
Before you reinstall the securing screws, you must use the supplied alignment tool (UCSC-C3K-M4IOTOOL) in the next step to ensure alignment of the connectors that connect to the internal chassis backplane. Failure to ensure alignment might damage the sockets on the backplane.
6.
Insert the four pegs of the alignment tool into the holes that are built into the forward-connector side of the server node and I/O
expander. Ensure that the alignment tool fits into all four holes and lies flat (see Figure 25
).
NOTE:
The alignment tool is shipped with systems that are ordered with an I/O expander. It is also shipped with I/O expander replacement spares. You can order the tool using Cisco PID UCSC-C3K-M4IOTOOL.
Figure 25 Using the I/O Expander Alignment Tool
1 Alignment tool
2 Forward-connector side of server node and I/O expander
3 Alignment tool installed
7.
8.
Remove the alignment tool.
9.
Reinstall the I/O expander cover:
a.
Set the cover in place on the I/O expander, offset about one inch toward the rear. Pegs on the inside of the cover must set into the tracks on the I/O expander base.
b.
Push the cover forward until it stops.
c.
Turn the latch handle 90 degrees to close the lock.
d.
Fold the latch handle flat.
e.
Reinstall the four screws that secure the top cover to the I/O expander.
40
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
10.
Install the server node with I/O expander to the chassis:
a.
With the two ejector levers open, align the server node and I/O expander with the two empty bays.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
11.
Power on the server node.
Replacing a PCIe Card Inside an I/O Expander
The optional I/O expander has two horizontal PCIe sockets.
Table 4 PCIe Expansion Slots in I/O Expander
Slot Number Electrical
Lane Width
Gen-3 x8
Socket Length
PCIe 1 (IOESlot1) x16 connector
PCIe 2 (IOESlot2) Gen-3 x8 x16 connector
1.
This is the supported length because of internal clearance.
2.
This is the size of the rear panel opening.
Supported Card
Length
1
3/4 length
3/4 length
Supported Card
Height
2
1/2 height
1/2 height
NCSI Support
No
No
1.
.
2.
Remove a server node with attached I/O expander from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers (see Figure 1
).
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node with attached I/O expander straight out from the system.
3.
Remove the top cover from the I/O expander:
a.
Remove the four screws that secure the top cover and set them aside. There are two screws on each side of the top cover (see
b.
Lift the latch handle to an upright position (see
).
c.
Turn the latch handle 90 degrees to release the lock.
d.
Slide the cover toward the rear (toward the rear-panel buttons) and then lift it from the I/O expander.
4.
Remove an existing PCIe card (or a filler panel if no card is present):
a.
Release the card-tab retainer. On the inside of the I/O expander, pull the spring-loaded plunger on the card-tab retainer inward and then rotate the card-tab retainer 90 degrees to the open position (see
b.
Slide the PCIe card horizontally to free its edge connector from the socket, and then lift the card out from the I/O expander.
If no card is present, remove the filler panel from the slot.
5.
Install a new PCIe card:
a.
With the card-tab retainer in the open position, set the card in the I/O expander and align its edge connector with the socket.
41
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
b.
Slide the card horizontally to fully engage the edge connector with the socket. The card’s tab should sit flat against the rear-panel opening.
c.
Close the card-tab retainer. Rotate the retainer 90 degrees until it clicks and locks.
6.
Replace the I/O expander cover:
a.
Set the cover in place on the I/O expander, offset about one inch toward the rear. Pegs on the inside of the cover must engage the tracks on the I/O expander base.
b.
Push the cover forward until it stops.
c.
Turn the latch handle 90 degrees to close the lock.
d.
Fold the latch handle flat.
e.
Reinstall the four screws that secure the top cover to the I/O expander.
7.
Install a server node with I/O expander to the chassis:
a.
With the two ejector levers open, align the server node and I/O expander with the two empty bays.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
8.
Power on the server node.
Figure 26 PCIe Card Sockets Inside the I/O Expander
1 PCIe card sockets
(software designations IOESlot1 and IOESlot2)
2 Spring-loaded peg on card-tab retainer
(inside the I/O expander)
3 Card-tab retainer (outside the I/O expander)
42
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing a Storage Controller Card Inside the I/O Expander
The software designation of the controller card socket inside the I/O expander is IOEMezz1.
1.
Shut down and power off the server node by using the software interface or by pressing the node power button, as described in
Down a C3X60 M4 Server Node, page 6 .
2.
Remove a server node with attached I/O expander from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers (see Figure 1
).
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node with attached I/O expander straight out from the system.
3.
Remove the cover from the I/O expander:
a.
Remove the four screws that secure the top cover and set them aside. There are two screws on each side of the top cover (see
b.
Lift the latch handle to an upright position (see
).
c.
Turn the latch handle 90-degrees to release the lock.
d.
Slide the cover toward the rear (toward the rear-panel buttons) and then lift it from the I/O expander.
4.
Remove a Cisco modular RAID controller card:
a.
Loosen the four captive thumbscrews that secure the card to the I/O expander (see
b.
Grasp the card at both ends and lift it evenly to disengage the connector on the underside of the card from the mezzanine socket.
5.
Install a Cisco modular RAID controller card:
a.
Align the card with bracket over the mezzanine socket and the four standoffs.
b.
Press down on both ends of the card to engage the connector on the underside of the card with the mezzanine socket.
c.
Tighten the four captive thumbscrews that secure the card to the server node.
6.
Replace the I/O expander cover:
a.
Set the cover in place on the I/O expander, offset about one inch toward the rear. Pegs on the inside of the cover must set into the tracks on the I/O expander base.
b.
Push the cover forward until it stops.
c.
Turn the latch handle 90-degrees to close the lock.
d.
Fold the latch handle flat.
e.
Reinstall the four screws that secure the top cover to the I/O expander.
7.
Install a server node with attached I/O expander to the chassis:
a.
With the two ejector levers open, align the new server node with the empty bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
8.
Power on the server node.
43
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Figure 27 Storage Controller Card Thumbscrews Inside the I/O Expander
44
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
Replacing an NVMe SSD Inside the I/O Expander
The I/O expander has two sockets for NVMe SSDs.
The software designations of the NVMe PCIe SSD sockets inside the I/O expander are IOENVMe1 and IOENVMe2 (see
1.
Shut down and power off the server node by using the software interface or by pressing the node power button, as described in
Down a C3X60 M4 Server Node, page 6 .
2.
Remove a server node with attached I/O expander from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers (see Figure 1
).
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
3.
Remove the cover from the I/O expander:
a.
Remove the four screws that secure the top cover and set them aside. There are two screws on each side of the top cover (see
b.
Lift the latch handle to an upright position (see
).
c.
Turn the latch handle 90-degrees to release the lock.
d.
Slide the cover toward the rear (toward the rear-panel buttons) and then lift it from the I/O expander.
4.
If a RAID controller is present, remove it from the I/O expander to provide clearance:
a.
Loosen the four captive thumbscrews that secure the card (see Figure 27 ).
b.
Grasp the card at both ends and lift it evenly to disengage the connector on the underside of the card from the mezzanine socket.
5.
Remove the NVMe PCIe SSD:
a.
Remove the single screw that secures the drive to its bracket (see
b.
Slide the drive to disengage it from its horizontal socket, then lift it from the I/O expander.
6.
Install a new NVME PCIe SSD:
a.
Set the drive in its bracket, then slide it forward to engage its connector with the socket.
b.
Install the single screw that secures the drive to the bracket.
7.
If you removed a RAID controller, reinstall it:
a.
Align the card with bracket over the mezzanine socket and the four standoffs.
b.
Press down on both ends of the card to engage the connector on the underside of the card with the mezzanine socket.
c.
Tighten the four captive thumbscrews that secure the card.
8.
Replace the I/O expander cover:
a.
Set the cover in place on the I/O expander, offset about one inch toward the rear. Pegs on the inside of the cover must set into the tracks on the I/O expander base.
b.
Push the cover forward until it stops.
c.
Turn the latch handle 90-degrees to close the lock.
45
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Replacing C3X60 M4 Server Node and I/O Expander Components
d.
Fold the latch handle flat.
e.
Reinstall the four screws that secure the top cover to the I/O expander.
9.
Install a server node with attached I/O expander to the chassis:
a.
With the two ejector levers open, align the new server node with the empty bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
10.
Power on the server node.
Figure 28 NVMe PCIe SSDs Inside the I/O Expander
1 Single securing screw (one each SSD) 2 NVMe PCIe SSD sockets
IOENVMe1 and IOENVMe2
46
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Service Headers on the Server Node Board
Service Headers on the Server Node Board
The server node board includes headers that you can jumper for certain service functions. This section includes the following topics:
Service Header Locations on the C3X60 M4 Server Node Board, page 47
Using the Clear Password Header J64, page 48
Using the Clear CMOS Header P19, page 48
Service Header Locations on the C3X60 M4 Server Node Board
for the locations.
Header J64 = Password clear
Header P19 = CMOS clear
Figure 29 Service Headers on the C3X60 M4 Server Node Board
1
Header J64 = Password clear
2
Header P19 = CMOS clear
47
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Service Headers on the Server Node Board
Using the Clear Password Header J64
You can use a jumper on header J64 to clear the administrator password.
1.
Shut down the server node by using the software interface or by pressing the node power button, as described in
C3X60 M4 Server Node, page 6 .
2.
Remove a server node from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers.
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node straight out from the system.
3.
Do one of the following to access the component inside the server node:
Server Node or I/O Expander Top Cover, page 12 before continuing with the next step.
If the server node has an I/O expander attached—Remove the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14
before you continue with the next step.
4.
Locate header J64 (see
5.
Install a jumper to pins 1 and 2 of the header.
6.
Install the server node:
a.
With the two ejector levers open, align the server node with the empty bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
7.
Power on the server node.
8.
After the server node has fully booted, shut it down again, as described in
Shutting Down a C3X60 M4 Server Node, page 6
.
9.
Remove the server node from the system, and then remove the server node cover.
10.
Remove the jumper from pins 1 and 2.
NOTE:
If you do not remove the jumper, the Cisco IMC clears the password each time that you boot the server node.
11.
Do one of the following:
If the server node did not have an I/O expander attached—Reinstall the server node cover as described in
Server Node or I/O Expander Top Cover, page 12 before continuing with the next step.
If the server node had an I/O expander attached—Reinstall the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14
before you continue with the next step.
12.
Power on the server node.
Using the Clear CMOS Header P19
You can install a jumper to header P19 to clear the CMOS settings.
1.
Shut down the server node by using the software interface or by pressing the node power button, as described in
C3X60 M4 Server Node, page 6 .
48
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Service Headers on the Server Node Board
2.
Remove a server node chassis from the system:
a.
Grasp the two ejector levers and pinch their latches to release the levers.
b.
Rotate both levers to the outside at the same time to evenly disengage the server node from its midplane connectors.
c.
Pull the server node chassis straight out from the system.
3.
Do one of the following to access the component inside the server node:
If the server node does not have an I/O expander attached—Remove the server node cover as described in
Server Node or I/O Expander Top Cover, page 12
before continuing with the next step.
If the server node has an I/O expander attached—Remove the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14 before you continue with the next step.
4.
Locate header P19 (see
5.
Install a jumper to pins 1 and 2 of the header.
6.
Install the server node to the system:
a.
With the two ejector levers open, align the server node with the empty bay.
b.
Push the server node into the bay until it engages with the midplane connectors and is flush with the chassis.
c.
Rotate both ejector levers toward the center until they lay flat and their latches lock into the rear of the server node.
7.
Power on the server node.
8.
9.
Remove the server node from the system, and then remove the server node cover.
10.
Remove the jumper from pins 1 and 2.
NOTE:
If you do not remove the jumper, the Cisco IMC clears the CMOS settings each time that you boot the server node.
11.
Do one of the following:
Server Node or I/O Expander Top Cover, page 12
before continuing with the next step.
If the server node had an I/O expander attached—Reinstall the I/O expander and intermediate cover as described in
Expander From a C3X60 M4 Server Node, page 14 before you continue with the next step.
12.
Power on the server node.
49
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
Storage Controller Considerations
Supported Storage Controllers and Required Cables, page 50
Cisco UCS C3X60 12G SAS RAID Controller Information, page 50
Cisco UCS S3260 12G Dual Pass-Through Controller Information, page 51
Best Practices For Configuring RAID Controllers, page 51
Restoring RAID Configuration After Replacing a RAID Controller, page 52
Embedded Software RAID, page 53
For More Information on Using Storage Controllers, page 60
Supported Storage Controllers and Required Cables
lists the supported storage controller cards for the C3X60 M4 server node.
Table 5 Cisco UCS C3X60 M4 Supported Storage Controller Options
Controller
Cisco UCS C3X60 12G SAS
RAID Controller for M4 Server
Nodes
(with 4 GB cache)
UCS-C3K-M4RAID
Cisco UCS S3260 Dual
Pass-Through Controller
UCS-S3260-DHBA
NOTE
: This controller is supported in an M4 server node only; it is not supported in the I/O expander or M3 server nodes.
NOTE
: This controller requires chassis part number 68-5286-06 or later. The chassis motherboard in earlier chassis versions does not support this controller.
Style
Mezzanine connector
Mezzanine connector
Maximum Drives RAID Levels
60 internal 0, 1, 5, 6, 10, 50, 60
Required Cables
None. Card installs to mezzanine socket inside the server node or
I/O expander.
60 internal Non-RAID None. Card installs to mezzanine socket inside the server node.
Cisco UCS C3X60 12G SAS RAID Controller Information
The Cisco UCS C3X60 12G SAS RAID Controller for M4 server nodes is based on the Broadcom 3316 SAS/SATA, 16-port RAID-on-chip
(RoC).
The controller can be used in JBOD mode (non-RAID) or in MegaRAID hardware RAID mode with a choice of RAID levels 0,1,5,6,10,
50, or 60.
Maximum drives controllable—64 (the server has maximum 60 internal drives)
Maximum drives per span—32
Maximum spans—8
50
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
Cisco UCS S3260 12G Dual Pass-Through Controller Information
The Cisco UCS S3260 Dual Pass-Through Controller is based on the Broadcom 3316 SAS/SATA, 16-port RAID-on-chip (RoC). This controller is divided into two boards: a base board and a power board. The power board supplies power to the base board.
This pass-through controller has the following features:
Dual Broadcom SAS3316-based subsystems.
Dual 8x SAS-3 lanes (12G, 6G, 3G, and SATA), one for each subsystem.
Dual 8x PCIe Gen-3 lanes via the server-board mezzanine connector, one for each subsystem.
Maximum 50 W power consumption.
Best Practices For Configuring RAID Controllers
4K Sector Format Drives, page 51
RAID Card Firmware Compatibility, page 51
Choosing Between RAID 0 and JBOD, page 51
RAID 5/RAID 6 Volume Creation, page 51
Background Operations (BGOPs), page 52
4K Sector Format Drives
Do not configure 4K sector format and 512-byte sector format drives as part of the same RAID volume.
RAID Card Firmware Compatibility
Firmware on the RAID controller must be verified for compatibility with the current Cisco IMC and BIOS versions that are installed on the server. If not compatible, upgrade or downgrade the RAID controller firmware accordingly using the Host Upgrade Utility (HUU) for your firmware release to bring it to a compatible level.
See the HUU guide for your Cisco IMC release for instructions on downloading and using the utility to bring server components to compatible levels: HUU Guides
Choosing Between RAID 0 and JBOD
The RAID controller supports JBOD mode (non-RAID) on physical drives that are in pass-through mode and directly exposed to the OS.
We recommended that you use JBOD mode instead of individual RAID 0 volumes when possible.
RAID 5/RAID 6 Volume Creation
The RAID controller allows you to create a large RAID 5 or 6 volume by including all the drives in the system in a spanned array configuration (RAID 50/RAID 60). Where possible, we recommend you create multiple, smaller RAID 5/6 volumes with fewer drives per
RAID array. This provides redundancy and reduces the operations time for initialization, RAID rebuilds, and other operations.
51
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
Choosing I/O Policy
The I/O policy applies to reads on a specific virtual drive. It does not affect the read-ahead cache. RAID volumes can be configured in two types of I/O policies. These are:
Cached I/O—In this mode, all reads are buffered in cache memory. Cached I/O provides faster processing.
Direct I/O—In this mode, reads are not buffered in cache memory. Data is transferred to the cache and the host concurrently. If the same data block is read again, it comes from cache memory. Direct I/O makes sure that the cache and the host contain the same data.
Although Cached I/O provides faster processing, it is useful only when the RAID volume has a small number of slower drives. With the
C3X60 4-TB SAS drives, Cached I/O has not shown any significant advantage over Direct I/O. Instead, Direct I/O has shown better results over Cached I/O in a majority of I/O patterns. We recommended you use Direct I/O (the default) in all cases.
Background Operations (BGOPs)
The RAID controller conducts different background operations like Consistency Check (CC), Background Initialization (BGI), Rebuild
(RBLD), Volume Expansion & Reconstruction (RLM), and Patrol Real (PR).
While these BGOPS are expected to limit their impact to I/O operations, there have been cases of higher impact during some of the I/O operations such as Format. In these cases, both the I/O operation and the BGOPS may take more time to complete. In such cases, we recommend you limit concurrent BGOPS and other intensive I/O operations where possible.
Restoring RAID Configuration After Replacing a RAID Controller
When you replace a RAID controller, the RAID configuration that is stored in the controller is lost.
To restore your RAID configuration to your new RAID controller, follow these steps.
1.
Replace your RAID controller. See
Replacing a Storage Controller Card Inside the C3X60 M4 Server Node, page 34
.
2.
If this was a full chassis swap, replace all drives into the drive bays, in the same order that they were installed in the old chassis.
3.
Reboot the server node.
4.
Press any key (other than C) to continue when you see the following onscreen prompt:
All of the disks from your previous configuration are gone. If this is an unexpected message, then please power of your system and check your cables to ensure all disks are present.
Press any key to continue, or ‘C’ to load the configuration utility.
5.
Watch the subsequent screens for confirmation that your RAID configuration was imported correctly:
If you see the following message, your configuration was successfully imported. The LSI virtual drive is also listed among the storage devices.
N Virtual Drive(s) found on host adapter.
If you see the following message, your configuration was not imported. In this case, reboot the server node and try the import operation again.
0 Virtual Drive(s) found on host adapter.
52
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
Embedded Software RAID
Launching the LSI Embedded MegaRAID Configuration Utility, page 53
Installing LSI MegaSR Drivers For Windows and Linux, page 53
Each server node includes an embedded MegaRAID controller that can control two rear-panel solid state drives (SSDs) in a RAID 0 or 1 configuration. This embedded software RAID is available only when the server node has the UCS S3260 12G Dual Pass-Through Controller
(UCS-S3260-DHBA) installed. When this HBA is installed, the two rear-panel SSDs can be controlled through software RAID mode or
AHCI mode, when selected in the server BIOS.
NOTE:
Embedded software RAID is not available when the HW RAID Cisco UCS C3X60 12G SAS RAID Controller
(UCS-C3K-M4RAID) is installed. In that case, the rear-panel SSDs are controlled by hardware RAID.
NOTE:
VMware ESX/ESXi or any other virtualized environments are not supported for use with the embedded MegaRAID controller.
Hypervisors such as Hyper-V, Xen, or KVM are also not supported for use with the embedded MegaRAID controller.
NOTE:
The embedded RAID controller in server node 1 can control the upper two rear-panel SSDs; the embedded RAID controller in server node 2 can control the lower two rear-panel SSDs.
Launching the LSI Embedded MegaRAID Configuration Utility
1.
When the server reboots, watch for the prompt to press Ctrl+M.
2.
When you see the prompt, press Ctrl+M to launch the utility.
Installing LSI MegaSR Drivers For Windows and Linux
NOTE:
The required drivers for this controller are already installed and ready to use with the LSI software RAID Configuration Utility.
However, if you will use this controller with Windows or Linux, you must download and install additional drivers for those operating systems.
This section explains how to install the LSI MegaSR drivers for the following supported operating systems:
Microsoft Windows Server
Red Hat Enterprise Linux (RHEL)
SUSE Linux Enterprise Server (SLES)
For the specific supported OS versions, see the Hardware and Software Interoperability Matrix for your server release.
This section contains the following topics:
Downloading the LSI MegaSR Drivers, page 53
Microsoft Windows Driver Installation, page 54
Linux Driver Installation, page 55
Downloading the LSI MegaSR Drivers
The MegaSR drivers are included in the C-Series driver ISO for your server and OS. Download the drivers from Cisco.com.
1.
Find the drivers ISO file download for your server online and download it to a temporary location on your workstation:
a.
See the following URL: http://www.cisco.com/cisco/software/navigator.html
b.
Click Unified Computing and Servers in the middle column.
53
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
c.
Click Cisco UCS C-Series Rack-Mount Standalone Server Software in the right-hand column.
d.
Click your model of server in the right-hand column.
e.
Click Unified Computing System (UCS) Drivers.
f.
Click the release number that you are downloading.
g.
Click Download to download the drivers’ ISO file.
h.
Verify the information on the next page, and click Proceed With Download.
i.
Continue through the subsequent screens to accept the license agreement and then browse to a location where you want to save the drivers’ ISO file.
Microsoft Windows Driver Installation
This section describes how to install the LSI MegaSR driver in a Windows installation.
This section contains the following topics:
Windows Server 2008R2 Driver Installation, page 54
Updating the Windows Driver, page 55
Linux Driver Installation, page 55
Windows Server 2008R2 Driver Installation
The Windows operating system automatically adds the driver to the registry and copies the driver to the appropriate directory.
1.
Create a RAID drive group using the LSI Software RAID Configuration Utility before you install this driver for Windows. Launch this utility by pressing Ctrl+M when
LSI SWRAID
is shown during the BIOS POST.
2.
.
3.
Prepare the drivers on a USB thumb drive:
a.
Burn the ISO image to a disk.
b.
Browse the contents of the drivers folders to the location of the embedded MegaRAID drivers:
/<OS>/Storage/Intel/C600/
c.
Expand the Zip file, which contains the folder with the MegaSR driver files.
d.
Copy the expanded folder to a USB thumb drive.
4.
Start the Windows driver installation using one of the following methods:
To install from local media, connect an external USB DVD drive to the server and then insert the first Windows installation disk into
To install from remote ISO, log in to the server’s Cisco IMC interface and continue with the next step.
5.
Launch a Virtual KVM console window and click the Virtual Media tab.
a.
Click Add Image and browse to select your remote Windows installation ISO file.
b.
Check the check box in the Mapped column for the media that you just added, and then wait for mapping to complete.
6.
Power cycle the server.
7.
Press F6 when you see the F6 prompt during bootup. The Boot Menu window opens.
54
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
8.
On the Boot Manager window, choose the physical disk or virtual DVD and press Enter. The Windows installation begins when the image is booted.
9.
Press Enter when you see the prompt, “Press any key to boot from CD.”
10.
Observe the Windows installation process and respond to prompts in the wizard as required for your preferences and company standards.
11.
When Windows prompts you with “Where do you want to install Windows,” install the drivers for embedded MegaRAID:
a.
Click Load Driver. You are prompted by a Load Driver dialog box to select the driver to be installed.
b.
Connect the USB thumb drive that you prepared in
c.
On the Windows Load Driver dialog that you opened in Step a, click Browse.
d.
Use the dialog box to browse to the location of the drivers folder on the USB thumb drive, and then click OK.
Windows loads the drivers from the folder and when finished, the driver is listed under the prompt, “Select the driver to be installed.”
e.
Click Next to install the drivers.
Updating the Windows Driver
1.
Click Start, point to Settings, and then click Control Panel.
2.
Double-click System, click the Hardware tab, and then click Device Manager. Device Manager starts.
3.
In Device Manager, double-click SCSI and RAID Controllers, right-click the device for which you are installing the driver, and then click Properties.
4.
On the Driver tab, click Update Driver to open the Update Device Driver wizard, and then follow the wizard instructions to update the driver.
Linux Driver Installation
This section explains the steps to install the embedded MegaRAID device driver in a Red Hat Enterprise Linux installation or a SUSE Linux
Enterprise Server installation.
This section contains the following topics:
Obtaining the Driver Image File, page 55
Preparing Physical Installation Disks For Linux, page 56
Installing the Red Hat Linux Driver, page 57
Installing the SUSE Linux Enterprise Server Driver, page 58
Obtaining the Driver Image File
See
dud-[
driver version
].img, which is the boot image for the embedded MegaRAID stack.
NOTE:
The LSI MegaSR drivers that Cisco provides for Red Hat Linux and SUSE Linux are for the original GA versions of those distributions. The drivers do not support updates to those OS kernels.
55
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
Preparing Physical Installation Disks For Linux
This section describes how to prepare physical Linux installation disks from the driver image files, using either the Windows operating system or the Linux operating system.
NOTE:
The driver image is too large for a floppy disk, so use a USB thumb drive instead.
NOTE:
Alternatively, you can mount the dud.img file as a virtual floppy disk, as described in the installation procedures.
Preparing Physical Installation Disks For Linux With the Windows Operating System
Under Windows, you can use the RaWrite floppy image-writer utility to create disk images from image files.
1.
Download the Cisco UCS C-Series drivers ISO, as described in
Downloading the LSI MegaSR Drivers, page 53
and save it to your
Windows system that has a diskette drive.
2.
Extract the dud.img file:
a.
Burn the ISO image to a disc.
b.
Browse the contents of the drivers folders to the location of the embedded MegaRAID drivers:
/<OS>/Storage/Intel/C600/
c.
Expand the Zip file, which contains the folder with the driver files.
3.
Copy the driver update disk image dud-[driver version].img and your file raw write.exe to a directory.
NOTE:
RaWrite is not included in the driver package.
4.
If necessary, use this command to change the filename of the driver update disk to a name with fewer than eight characters: copy
dud-[driver version].img dud.img
5.
Open the DOS Command Prompt and navigate to the directory where raw write.exe is located.
6.
Enter the following command to create the installation diskette: raw write
7.
Press Enter.
You are prompted to enter the name of the boot image file.
8.
Enter: dud.img
9.
Press Enter.
You are prompted for the target disk.
10.
Insert a floppy disk into the server and enter: A:
11.
Press Enter.
12.
Press Enter again to start copying the file to the diskette.
13.
After the command prompt returns and the floppy disk drive LED goes out, remove the disk.
14.
Label the diskette with the image name.
Preparing Installation Disks with a Linux Operating System
Under Red Hat Linux and SUSE Linux, you can use a driver disk utility to create disk images from image files.
56
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
NOTE:
The driver image is too large for a floppy disk, so use a USB thumb drive instead.
1.
Linux system that has a disk drive.
2.
Extract the dud.img file:
a.
Burn the ISO image to a disc.
b.
Browse the contents of the drivers folders to the location of the embedded MegaRAID drivers:
/<OS>/Storage/Intel/C600/
c.
Expand the Zip file, which contains the folder with the driver files.
3.
Copy the driver update disk image dud-[driver version].img to your Linux system.
4.
Insert a blank USB thumb drive into a port on your Linux system.
5.
Create a directory and mount the DUD image to that directory:
mkdir <destination_folder>
mount -oloop <driver_image> <destination_folder>
6.
Copy the contents in the directory to your USB thumb drive.
Installing the Red Hat Linux Driver
For the specific supported OS versions, see the Hardware and Software Interoperability Matrix for your server release.
This section describes the fresh installation of the Red Hat Enterprise Linux device driver on systems with the embedded MegaRAID stack.
1.
Create a RAID drive group using the LSI Software RAID Configuration utility before you install this driver for the OS. Launch this utility by pressing Ctrl-M when
LSI SWRAID
is shown during the BIOS POST.
2.
Prepare the dud.img file using one of the following methods:
of this procedure.
To install from a virtual floppy disk: Download and save the Cisco UCS C-Series drivers’ ISO, as described in
MegaSR Drivers, page 53 . Then continue with the next step.
3.
Extract the dud.img file:
a.
Burn the ISO image to a disc.
b.
Browse the contents of the drivers folders to the location of the embedded MegaRAID drivers:
/<OS>/Storage/Intel/C600/
c.
Copy the dud-<
driver version
>.img file to a temporary location on your workstation.
4.
Start the Linux driver installation using one of the following methods:
To install from local media, connect an external USB DVD drive to the server and then insert the first RHEL installation disk into the
To install from remote ISO, log in to the server’s Cisco IMC interface. Then continue with the next step.
57
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
5.
Launch a Virtual KVM console window and click the Virtual Media tab.
a.
Click Add Image and browse to select your remote RHEL installation ISO file.
b.
Click Add Image again and browse to select your dud.img file.
c.
Check the check boxes in the Mapped column for the media that you just added, then wait for mapping to complete.
6.
Power cycle the server.
7.
Press F6 when you see the F6 prompt during bootup. The Boot Menu window opens.
8.
On the Boot Manager window, select the physical disk or virtual DVD and press Enter.
The RHEL installation begins when the image is booted.
9.
Enter one of the following commands at the boot prompt:
For RHEL 6.x (32- and 64-bit), enter:
linux dd blacklist=isci blacklist=ahci nodmraid noprobe=<atadrive number>
For RHEL 7.x (32- and 64-bit), enter:
linux dd modprobe.blacklist=ahci nodmraid
NOTE:
The noprobe values depend on the number of drives. For example, to install RHEL 6.5 on a RAID 5 configuration with three drives, enter Linux dd blacklist=isci blacklist=ahci nodmraid noprobe=ata1 noprobe=ata2
10.
Press Enter.
The prompt asks whether you have a driver disk.
11.
Use the arrow key to choose Yes, and then press Enter.
12.
Choose fd0 to indicate that you have a floppy disk with the driver on it.
13.
Do one of the following actions:
If you prepared the IMG file on a physical diskette in
, connect an external disk drive to the target server and then insert the disk in
the A:/ drive and press Enter.
If you mapped the IMG file as a virtual floppy in
, choose the location of the virtual floppy.
The installer locates and loads the driver for your device. The following message appears:
Loading megasr driver...
14.
Follow the Red Hat Linux installation procedure to complete the installation.
15.
Reboot the system.
Installing the SUSE Linux Enterprise Server Driver
For the specific supported OS versions, see the Hardware and Software Interoperability Matrix for your server release.
This section describes the installation of the SUSE Linux Enterprise Server driver on a system with the embedded MegaRAID stack.
1.
Create a RAID drive group using the LSI SWRAID Configuration utility before you install this driver for the OS. Launch this utility by pressing Ctrl+M when
LSI SWRAID
is shown during the BIOS POST.
2.
Prepare the dud.img file using one of the following methods:
Then return to
58
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Storage Controller Considerations
MegaSR Drivers, page 53 . Then continue with the next step.
3.
Extract the dud.img file:
a.
Burn the ISO image to a disc.
b.
Browse the contents of the drivers folders to the location of the embedded MegaRAID drivers:
/<OS>/Storage/Intel/C600/
c.
Copy the dud-<
driver version
>.img file to a temporary location on your workstation.
4.
Start the Linux driver installation using one of the following methods:
To install from local media, connect an external USB DVD drive to the server and then insert the first RHEL install disc into the drive.
Skip to
To install from remote ISO, log in to the server’s Cisco IMC interface and continue with the next step.
5.
Launch a Virtual KVM console window and click the Virtual Media tab.
a.
Click Add Image and browse to select your remote RHEL installation ISO file.
b.
Click Add Image again and browse to select your dud.img file.
c.
Check the check box in the Mapped column for the media that you just added, and then wait for mapping to complete.
6.
Power cycle the server.
7.
Press F6 when you see the F6 prompt during bootup. The Boot Menu window opens.
8.
On the Boot Manager window, select the physical disk or virtual DVD and press Enter. The SLES installation begins when the image is booted.
9.
When the first SLES screen appears, choose Installation.
10.
Enter one of the following in the Boot Options field:
For SLES 11 and SLES 11 SP1 (32- and 64-bit), enter: brokenmodules=ahci
For SLES 11 SP2 (32-and 64-bit), enter: brokenmodules=ahci brokenmodules=isci
For SLES 12, enter: brokenmodules=ahci
11.
Press F6 for the driver and choose Yes.
12.
Do one of the following actions:
If you prepared the IMG file on a physical disk in 2.
, insert the USB thumb drive to the target server and then insert the disk in the A:/ drive and press Enter.
If you mapped the IMG file as a virtual floppy in
, choose the location of the virtual floppy.
“Yes” appears under the F6 Driver heading.
13.
Press Enter to choose Installation.
14.
Press OK.
The following message is displayed:
LSI Soft RAID Driver Updates added.
59
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Related Documentation
15.
At the menu, choose the driver update medium and press the Back button.
16.
Continue and complete the installation process by following the prompts in the installation wizard.
For More Information on Using Storage Controllers
The LSI utilities have help documentation for more information about using the utilities.
Full Avago Technologies/LSI documentation is also available:
For hardware SAS MegaRAID— Avago Technologies/LSI 12 Gb/s MegaRAID SAS Software User’s Guide, Rev. F
For embedded software MegaRAID— LSI Embedded MegaRAID Software User Guide
Related Documentation
Cisco UCS S3260 Storage Server Installation and Service Guide
Regulatory Compliance and Safety Information For Cisco UCS S-Series Hardware
60
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Related Documentation
Legal Information
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE
WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO
BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE
FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE
INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE.
IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO
REPRESENTATIVE FOR A COPY.
The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications.
Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense.
The following information is for FCC compliance of Class B devices: This equipment has been tested and found to comply with the limits for a Class B digital device, pursuant to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference in a residential installation. This equipment generates, uses and can radiate radio frequency energy and, if not installed and used in accordance with the instructions, may cause harmful interference to radio communications. However, there is no guarantee that interference will not occur in a particular installation. If the equipment causes interference to radio or television reception, which can be determined by turning the equipment off and on, users are encouraged to try to correct the interference by using one or more of the following measures:
Reorient or relocate the receiving antenna.
Increase the separation between the equipment and receiver.
Connect the equipment into an outlet on a circuit different from that to which the receiver is connected.
Consult the dealer or an experienced radio/TV technician for help.
Modifications to this product not authorized by Cisco could void the FCC approval and negate your authority to operate the product.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley
(UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the
University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS
ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES,
EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE
PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR
INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING
OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers.
Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.
All printed copies and duplicate soft copies are considered un-Controlled copies and the original on-line version should be referred to for latest version.
Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at www.cisco.com/go/offices.
61
Cisco UCS C3X60 M4 Server Node For Cisco UCS S3260 Storage Server Service Note
Related Documentation
62
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Related manuals
advertisement