Oracle Exalogic Elastic Cloud: Datacenter Network Integration

Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
An Oracle White Paper
February 2014
Oracle Exalogic Elastic Cloud:
Datacenter Network Integration
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
Disclaimer
The following is intended to outline our general product direction. It is intended for information
purposes only, and may not be incorporated into any contract. It is not a commitment to deliver
any material, code, or functionality, and should not be relied upon in making purchasing
decisions. The development, release, and timing of any features or functionality described for
Oracle’s products remains at the sole discretion of Oracle.
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
Table of Contents Executive Summary ............................................................................ 1 Exalogic in the Datacenter Network .................................................... 1 Exalogic System I/O ............................................................................ 2 Network Services in Exalogic .......................................................... 2 Ethernet Gateway for Exalogic ........................................................ 3 Integrating Exalogic with the Datacenter LAN ..................................... 4 Datacenters with Existing 10 Gb L2 Infrastructure .......................... 4 Datacenters without an Existing 10 Gb L2 Infrastructure or with Unique
Connectivity Requirements ............................................................. 5 LAN Connectivity and Network Isolation ............................................. 7 Server Connectivity ......................................................................... 7 Conclusion .......................................................................................... 9 Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
Executive Summary
The Oracle Exalogic Elastic Cloud provides a unique I/O subsystem called Exabus. Exabus
employs a converged network fabric to provide all I/O services to the applications running
within an Exalogic system. Applications residing within an Exalogic system can access all
network services provided within the datacenter network through Exabus. An Exalogic system
can be easily integrated into an existing datacenter’s network infrastructure, if the physical
connectivity and network services provided within the network infrastructure are understood
during the deployment planning phase.
This white paper outlines the physical connectivity solutions supported by the Oracle Exalogic
Elastic Cloud and provides an overview of network service delivery through Exabus.
Exalogic in the Datacenter Network
Enterprise
Core
Core Layer
Aggregation Layer
Network Services
• Content Switching
• NAT
• NAM
• IDS/IPS
• Firewall, Cache, SSL offload, …
Access Layer
InfiniBand
10 Gb Ethernet
1 Gb Ethernet
Layer 2 access with Exadata
NIC teaming
Exalogic
Blade server with
integrated switch
Layer 3 Access
Figure 1: The Datacenter Network
From the viewpoint of the datacenter LAN, Exalogic appears as of a collection of 10 Gb
Ethernet attached servers. Connecting Exalogic into the LAN requires that the system be
connected to a layer-2 switch. An Exalogic system may connect to additional Exalogic, Exadata,
SPARC Supercluster or ZFS Storage Appliance systems via InfiniBand, but the primary interface
to the system for client access is through the 10 Gb Ethernet interface.
1
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
Within a hierarchical network model, Exalogic resides within the access layer. Exalogic connects
into the network through layer-2 switches and “higher level” network services such as general
firewall capabilities, content switching, etc., are provided through the aggregation (or
distribution) layer.
Exalogic System I/O
Exalogic utilizes QDR InfiniBand as a foundation for the Exalogic I/O subsystem known as
Exabus. All compute servers and the Exalogic storage devices are redundantly connected to the
Exalogic InfiniBand fabric, which functions as a highly available, high-performance system
backplane for Exalogic. The InfiniBand fabric is constructed of InfiniBand Gateways, which act
both as InfiniBand “leaf” switches and InfiniBand to 10 Gb Ethernet Gateways, as well as a 36port “spine” switch, which enables the expansion of the Exalogic system. This expansion can be
achieved by attaching up to eight “Full” Exalogic, Exadata, SPARC SuperCluster, or ZFS Storage
Appliance systems to the fabric.
Figure 2: Exalogic System Internal Network Connectivity
Network Services in Exalogic
All network services, which are provided in an enterprise data center, can be made available to
applications running within the Exalogic system. For source-destination connections that lie
within the Exabus domain, the Exalogic I/O infrastructure provides hardware-based layer-2
2
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
services for InfiniBand as well as software-based services for Ethernet and IP. Error!
Reference source not found. outlines the network services provided through Exabus.
Table 1 – Network services provided through Exabus
Ethernet Gateway for Exalogic
The Gateway function in Exalogic is provided by the Sun Network QDR InfiniBand Gateway Switch.
The Sun Network QDR InfiniBand Gateway Switch enables hosts on the InfiniBand fabric to share
one or more 10 GbE links to an external LAN and allows the hosts sharing a 10 GbE link to
communicate between each other as well as with nodes on the external LAN as if they all had a
private NIC connected to the external LAN. The Gateway is implemented as a shared NIC
Model and is presented as a multi-address endpoint on both IB and Ethernet fabrics. Hosts on
the IB fabric communicate with nodes on the Ethernet LAN via the Gateway, but with other
hosts on the IB fabric directly. However, logically the hosts sharing an external Ethernet port are
all part of the same L2 Ethernet subnet seen from the external LAN.
Through the InfiniBand Gateways, each OS instance residing within Exalogic can be provisioned
with virtualized Ethernet NICs. An InfiniBand Gateway provides a total of eight physical ports
of 10 Gb Ethernet. Each physical port supports up to 1,024 virtual NICs, each with it’s own
MAC address and (optional) VLAN ID.
The Sun Network QDR InfiniBand Gateway Switch utilizes QSFP connectors to provide both
Ethernet and InfiniBand connectivity. Two QSFP ports on the Gateway (the two upper rightmost ports, as shown in Figure 3) are for Ethernet connectivity. The QSFP connector aggregates
four 10 Gb Ethernet connections to a single connector. The Gateway’s Ethernet ports support
10G Base-SR multi-mode fiber. Within the Exalogic system, the 10G Base-SR QSFP
transceivers are pre-installed in the Gateway Ethernet ports. Passive fiber cables are available in
two formats:
•
“Splitter” cables which provide an MTP/MPO termination on one end of the cable and
four separate male LC-terminated “pig tails” on the other end of the cable. The splitter
cable enables connectivity to four 10 Gb Ethernet switch ports through 10G Base-SR
transceivers.
3
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
•
“Straight” cables which provides MTP/MPO termination on both ends of the cable.
The straight cable enables connectivity to a single port on either a 10 Gb Ethernet
switch that supports QSFP connectors (4x10 GbE) or to a patch panel supporting
MTP-MPO.
4 x 10GbE QSFP Connector
4 x 10GbE QSFP Connector
Name: 0A_Eth_X:
Name: 1A_Eth_X:
Port 0 ! 0A_Eth_0
Port 1 ! 0A_Eth_1
Port 2 ! 0A_Eth_2
Port 3 ! 0A_Eth_3
Port 4 ! 1A_Eth_0
Port 5 ! 1A_Eth_1
Port 6 ! 1A_Eth_2
Port 7 ! 1A_Eth_3
0 1 2 3
NM2-GW Rear Panel - Right Side
Figure 3 - Locating the Ethernet ports on the InfiniBand Gateway
Integrating Exalogic with the Datacenter LAN
As previously stated, Exalogic presents Ethernet NICs to the data center LAN through the
Ethernet ports on the Exalogic InfiniBand Gateways. An Exalogic “full rack” contains four
InfiniBand Gateways and thus can provide up to thirty-two 10 GbE connections to the data
center L2 infrastructure. Exalogic quarter and half rack configurations include only two
InfiniBand Gateways and provide up to sixteen 10 GbE connections.
Datacenters with Existing 10 Gb L2 Infrastructure
In data centers with an existing 10 Gb Ethernet infrastructure, Exalogic can be connected
directly into the L2 infrastructure by connecting “splitter” cables to the InfiniBand Gateways and
attaching 10G Base SR transceivers on the cable’s four LC male terminated “pig tails”. SFP+ SR
transceivers are commonly supported by most switch vendors. It is recommended that the
4
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
switch vendor’s transceiver be utilized on the LC male terminated “pig tails” to ensure
transceiver-switch interoperability.
In order to connect directly to an existing 10 Gb L2 infrastructure, the following must be
supported:
•
•
•
10G Base-SR transceivers for connection to the L2 switch.
Transceivers must support connection to LC male terminated optical cables
Multi-mode fiber cable plant
Figure 4 identifies the components required to connect an Exalogic system into an existing 10
Gb L2 infrastructure. Error! Reference source not found. identifies the component counts
(along with associated part numbers) required to connect an Exalogic “full rack” to an existing
10 Gb L2 infrastructure.
4 x SFP+
Transceiver
MPO-to-4 LC
Fiber Splitter Cable
Customer
10 GbE
L2 Infrastructure
Male LC
Female LC
Figure 4 – Components required to connect Exalogic into an Existing 10 Gb L2 infrastructure
Table 2 Components required to connect an Exalogic "full rack" to an existing 10 Gb L2 infrastructure
Datacenters without an Existing 10 Gb L2 Infrastructure or with Unique
Connectivity Requirements
5
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
Data centers with a L2 infrastructure that does not meet the requirements outlined above should
interface to a 10 Gb L2 switch, which does meet those requirements, such as the Sun Network
10 GbE Switch 72p. Additionally, many third party 10 Gb Ethernet switches can provide this
capability. A pair (for high availability) of Sun Network 10 GbE Switch 72p’s can interface up
to 4 Exalogic “full racks” to a mix of data center network speeds and media types including:
•
•
1 Gb Ethernet
o Copper - 1000 Base-T
o Fiber
§ Single Mode – 1000 Base-Lx
§ Multi Mode – 1000 Base-Sx
10 Gb Ethernet
o Copper – SFP+ Direct Attach (aka 10GSFP+Cu or TwinAx)
o Fiber
§ Single Mode – 10G Base-LR
Figure 5 depicts the components required to connect an Exalogic system into a data center
infrastructure. Table 3 identifies the component counts (along with associated part numbers)
required to connect an Exalogic “full rack” to the data center LAN.
Sun Network 10GbE Switch 72p
MPO-MPO Passive
Fiber Cables
Customer
Ethernet
Infrastructure
Sun Switch to Data Center Ethernet
SFP+
Transceivers
QSFP Transceivers
Exalogic to
Sun Switch
Figure 5 - Components required to connect Exalogic into a data center LAN
6
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
Table 3 Components required to connect an Exalogic "full rack" to the data center LAN
LAN Connectivity and Network Isolation
Server Connectivity
The network interfaces on Exalogic compute nodes utilize active-passive bonding for high
availability. Each server is provisioned with a dual-ported InfiniBand HCA (Host Channel
Adapter). The InfiniBand ports are active-passive bonded through the software stack and each
port is connected to a separate InfiniBand Gateway with the “active” network load distributed
amongst the InfiniBand Gateways in the system. (Note this distribution is statically established
during system configuration.)
7
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
Ethernet traffic is carried over the InfiniBand fabric by encapsulating Ethernet frames within
InfiniBand packets. The Ethernet over InfiniBand Protocol (EoIB) is a network interface
implementation over InfiniBand. EoIB encapsulates Layer 2 datagrams over an InfiniBand
Datagram (UD) transport service. The InfiniBand UD datagrams encapsulates the entire
Ethernet L2 datagram and its payload. The EoIB Protocol also enables routing of packets from
the InfiniBand fabric to a 1 or 10 Gb Ethernet subnet.
Ethernet provisioning is provided by instantiating bonded virtualized NICs (aka VNICs) within
the OS instance. The active-passive bonding of the VNICs follows that of the associated
InfiniBand HCA ports. Each VNIC is associated with a physical port on the InfiniBand Gateway
and can be associated with a VLAN. The Exalogic system is configured such that each VNIC in
a bonded pair is associated with physical port on a different InfiniBand Gateway for high
availability.
Figure 6 depicts the server connections within the Exalogic system and the attachment of the
system to a single LAN. Note each server has active-passive bonded InfiniBand HCA ports and
VNICs. Also note, the connectivity shown in Figure 6 from NM2-GW to external LAN is
specific only to the Exalogic physical (OEL or Solaris) deployment. For Exalogic virtual the
connectivity from NM2-GW to external LAN will be covered in a separate white paper.
Server 1
Server 2
Bond
VLAN X
Bond
VLAN X
VLAN Y
EoIB VNIC
EoIB VNIC
EoIB VNIC
EoIB VNIC
HCA
IB port 1
HCA
IB port 2
HCA
IB port 2
HCA
IB port 1
Active Port
Passive Port
IB
IB
IB
IB
IB
NM2-GW
10G Port:
0A_Eth_X
10G Port:
0A_Eth_X
VLAN X,Y,.....
VLAN X,Y,.....
Ethernet Switch
Ethernet Switch
LAN
Figure 6 - Exalogic Physical (Linux/Solaris) Server Ethernet Connectivity, showing servers provisioned to support multiple VLANs
connected to a single LAN.
Figure 7 depicts Exalogic servers connected to multiple external LANs. Note that a VNIC is
instantiated for each external connection. Also note, the connectivity shown in Figure 7 from
NM2-GW to external LANs is specific only to the Exalogic physical (OEL or Solaris)
8
Oracle White Paper—Oracle Exalogic Elastic Cloud: Datacenter Network Integration
deployment. For Exalogic virtual the connectivity from NM2-GW to external LAN will be
covered in a separate white paper.
Server 1
Bond 1
EoIB
VNIC
Server 1
Bond 2
EoIB
VNIC
EoIB
VNIC
HCA
IB port 1
Bond 1
EoIB
VNIC
EoIB
VNIC
HCA
IB port 2
IB
EoIB
VNIC
HCA
IB port 1
IB
NM2-GW #1
Bond 2
EoIB
VNIC
HCA
IB port 2
IB
IB
NM2-GW #2
IB
10G Port:
0A_Eth_X
10G Port:
1A_Eth_X
Ethernet Switch
Ethernet Switch
LAN #1
EoIB
VNIC
10G Port:
0A_Eth_X
Ethernet Switch
10G Port:
1A_Eth_X
Ethernet Switch
LAN #2
Figure 7 – Exalogic Physical (Linux/Solaris) server Ethernet connectivity, showing servers connected to multiple LANs
Conclusion
Exabus is capable of delivering a rich network environment to applications resident within the
Exalogic Elastic Cloud. That network environment can be dynamically configured to meet the
demands of the application environment. The converged network infrastructure enabled by
Exabus provides a flexible, secure, high performance solution for all application I/O.
With appropriate planning, the Exalogic Elastic Cloud can be seamlessly integrated into a
datacenter’s network infrastructure.
9
Oracle Exalogic Elastic Cloud:
Datacenter Network Integration
February, 2014
Authors: Steve Callahan, Ola Torudbakken
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and
Oracle Corporation
the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
World Headquarters
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
500 Oracle Parkway
fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
Redwood Shores, CA 94065
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
U.S.A.
means, electronic or mechanical, for any purpose, without our prior written permission.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective
owners.
030
THIS PAGE INTENTIONALLY LEFT BLANK
Download PDF