How to Connect Oracle Exadata to 10G Networks Using Oracle

An Oracle White Paper
October 2013
How to Connect Oracle Exadata to 10 G
Networks Using Oracle’s Ethernet Switches
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Introduction ....................................................................................... 1
Exadata Database Machine X3-2 Full Rack Configuration................. 1
Multirack Exadata Database Machine X3-2 Configuration ................. 3
Exadata Database Machine X3-8 Full Rack Configuration................. 5
Multirack Exadata Database Machine X3-8 Configuration ................. 7
High Availability Option for Multirack Exadata Database Machine
X3-8 Configuration............................................................................. 9
How to Configure the Network for Oracle Exadata .......................... 11
Database Servers to Oracle Switch ES1-24 10 G Switch ............. 11
Oracle Switch ES1-24 10 G Switch to Client’s 10 G Network
Using L2 Uplinks.......................................................................... 14
Oracle Switch ES1-24 10 G Switch to Client’s 10 G Network
Using L3 Uplinks.......................................................................... 16
Conclusion ...................................................................................... 19
References .................................................................................. 19
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Introduction
This paper provides instructions for how to connect various Oracle Exadata configurations to an
existing 10 G Ethernet network using Oracle’s Ethernet switches. The paper also includes a description
and sample commands to show how to configure the network ports when connecting Oracle Exadata
database servers to a 10 G network.
Exadata Database Machine X3 is Oracle's next generation Database Machine, combining massive
memory and low-cost disks to deliver the highest performance and petabyte scalability at the lowest
cost. It is the ideal database platform for the varied and unpredictable workloads of cloud computing.
There are two different Oracle Exadata system models, Exadata Database X3-2 and Exadata Database
X3-8. Both systems can be deployed in a single-rack or multiple-rack configuration. This paper
includes a section for each of the configuration options, describing how to connect that configuration
to the client’s 10 G network. For each configuration, there is also a detailed parts list so readers can
easily purchase all the necessary components.
Oracle Exadata systems can also be connected to gigabit Ethernet (GbE) networks, but this topic will
be covered in a separate white paper and is not discussed in this paper.
Exadata Database Machine X3-2 Full Rack Configuration
Exadata Database Machine X3-2 includes eight Oracle Database servers, each with four 10 G Base-T
and two 10 G SFP+ onboard ports. It has two rack units (at location 41U and 42U) available for server
access switches with front-to-back airflow.
The proposed networking configuration utilizes two Oracle Switch ES1-24 switches. Oracle Switch
ES1-24 is a 10 GbE switch that is 1U half-width and has 24-port access, including 20x10 G Base-T
ports with 100Mbps/1/10 Gbps auto-negotiation and four 10 G SFP+ ports. Oracle Switch ES1-24
comes with industry-standard L2 and L3 features, logical link aggregation, server load balancing, and
support for virtualization. The data sheet and product related information for Oracle Switch ES1-24 is
available here.
The four 10 G Base-T ports on the Oracle Database servers can be connected directly to the 10 G
Base-T ports on Oracle Switch ES1-24 switches. There is no need for extra transceivers/cables, thus
reducing the TCO significantly.
1
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Figure 1: 10 GbE networking configuration for Exadata Database Machine X3-2, single full rack
Oracle Switch ES1-24 is a 1U half-wide switch, and two switches are placed in 1U space in 41U. The
switches are available in front-to-back air flow.
Exadata Database Machine X3-2 contains eight Oracle Database servers with four 10 G Base-T ports
each, two of which are connected to each of the Oracle Switch ES1-24 10 G Base-T ports for high
availability (HA). Cat 6 cables up to 55 meters or Cat 6a cables up to 100 meters can be used. The four
10 G SFP+ ports on the Oracle Switch ES1-24 switches are dedicated for uplink connectivity to the
client’s network with a 4:1 oversubscription.
Table 1 shows all the components required for this proposed networking configuration.
TABLE 1: COMPONENTS REQUIRED FOR CONNECTING EXADATA DATABASE MACHINE X3-2, FULL RACK TO 10 GBE NETWORK
QUANTITY
PART NUMBER
2
DESCRIPTION
Oracle Switch ES1-24 10 GbE switch with front-to-rear air flow
CONNECTIONS FOR ORACLE SWITCH ES1-24 TO DATABASE SERVERS
32
Industry standard
Cat 6 or Cat 6a cable to database servers
CONNECTIONS FOR ORACLE SWITCH ES1-24 TO UPLINK
8
Industry standard
SFP+ transceivers, cables for uplinks
2
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Multirack Exadata Database Machine X3-2 Configuration
In a multirack configuration with four Exadata Database Machine X3-2 systems, the 10 GbE
connectivity to the data center LAN is established by deploying two Oracle Switch ES1-24 10 GbE
switches in each Oracle Exadata rack and then connecting those switches to the 10 G network through
Oracle’s Sun Network 10 GbE Switch 72p, which acts as an aggregate switch.
Each Oracle Exadata rack has two Oracle Switch ES1-24 server access switches connected in rack
location 41U. The four onboard 10 G Base-T ports on the Oracle Database servers connect to the
10 G Base-T ports on the Oracle Switch ES1-24. The Sun Network 10 GbE Switch 72p top-of-rack
switch is used as an aggregate switch with the four 10 G SFP+ connections from Oracle Switch
ES1-24 terminating at Sun Network 10 GbE Switch 72p. Sun Network 10 GbE Switch 72p has 16
QSFP (four 10 G) ports and eight 10 G SFP+ ports.
The QSFP ports on Sun Network 10 GbE Switch 72p are connected with the QSFP to 4xSFP+
splitter cables that have QSFP on one end and 4 SFP+ ports on the other end. There are both copper
and optical cabling components available. For copper cables, no transceivers are required and the
QSFP port on Sun Network 10 GbE Switch 72p is connected to direct-attach Twinax QSFP to
4xSFP+ splitter cable, which has QSFP on one end and four SFP+ ports on the other end and comes
in 1, 3, and 5 meters. The QSFP end of the cables connects to Sun Network 10 GbE Switch 72p while
the splitter cable end of the cables is connected to the SFP+ port on Oracle Switch ES1-24.
For optical cables, the QSFP to 4xLC optical splitter cable is connected to the QSFP port on Sun
Network 10 GbE Switch 72p. On Oracle Switch ES1-24, the SFP+ transceiver, either long range (LR)
or short range (SR), is connected to the SFP+ port along with LC-LC terminated fiber optical cable.
3
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Figure 2: 10 GbE networking configuration for Exadata Database Machine X3-2, four full racks
Table 2 shows all the components required for this proposed networking configuration. Either the
copper or optical cables can be used to connect the Oracle Switch ES1-24 switch to the Sun Network
10 GbE Switch 72p switch.
TABLE 2: COMPONENTS REQUIRED FOR CONNECTING FOUR EXADATA DATABASE MACHINE X3-2 FULL RACKS TO
10 GBE NETWORK
QUANTITY
PART NUMBER
8
1
DESCRIPTION
Oracle Switch ES1-24 10 GbE switch with front-to-rear air flow
x2074A-F/x2074A-R
Sun Network 10 GbE Switch 72p
Connections for Oracle Switch ES1-24 Switches
10 GBE – 10 G BASE-T TO DATABASE SERVERS
128
Industry standard
Cat 6 or Cat 6a cables
10 GBE – SFP+ MULTIMODE FIBER (10 G BASE-SR) TO SUN NETWORK 10 GBE SWITCH 72P
32
X2129A-N
SFP+ SR transceiver
4
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
10 GBE – SFP+ MULTIMODE FIBER (10 G BASE-LR) TO SUN NETWORK 10 GBE SWITCH 72P
32
X5562A-Z
SFP+ LR transceiver
Connections for Sun Network 10 GbE Switch 72p
10 GBE – SFP+ TO ORACLE SWITCH ES1-24
8
X2124A
QSFP transceivers
8
X2127A-10 m
MPO-to-4xLC splitter cables, 10 meters
X2127A-20 m
MPO-to-4xLC splitter cables, 20 meters
X2127A-50 m
MPO-to-4xLC splitter cables, 50 meters
10 GBE – SFP+ TWINAX COPPER TO ORACLE SWITCH ES1-24
8
X2125A-1M-N
Twinax QSFP -4xSFP+ splitter, 1 meter
X2125A-3M-N
Twinax QSFP -4xSFP+ splitter, 3 meters
X2125A-5M-N
Twinax QSFP -4xSFP+ splitter, 5 meters
10 GBE – SFP+ TO UPLINKS
8
Customer-supplied
SFP+ transceivers, cables
Exadata Database Machine X3-8 Full Rack Configuration
Exadata Database Machine X3-8 has two 8-core database servers, each with eight 10 G SFP+ ports
and eight 1 G Base-T ports. The proposed networking configuration consists of the Sun Network
10 GbE Switch 72p top-of-rack switch as an aggregate with all the network connections from the
server connecting to the Sun Network 10 GbE Switch 72p switch.
Sun Network 10 GbE Switch 72p has 16 QSFP (four 10 G) ports and eight 10 G SFP+ ports. It is an
ultra-dense, non-blocking, low-latency switch with an industry-standard L2 and L3 feature set. It can
be managed through Oracle Integrated Lights Out Manager (Oracle ILOM). The data sheet and
product-related information for Sun Network 10 GbE Switch 72p are available here.
The QSFP ports on Sun Network 10 GbE Switch 72p are used to connect to eight 10 G SFP+ ports
on both of the database servers using QSFP to 4xSFP+ splitter cables.
Copper and optical cabling components are available. For copper cables, the direct-attach Twinax
QSFPx4 SFP+ splitter cable is connected with the QSFP end to Sun Network 10 GbE Switch 72p and
the splitter cables going to the eight 10 G ports on the database servers. When using an optical cable
component, the QSFP to 4xLC optical splitter cable is connected with the QSFP end on Sun Network
10 GbE Switch 72p. The other end of the splitter cables is connected to the database servers, using
either a long range (LR) or short range (SR) SFP+ transceiver. The 8x10 G SFP ports on Sun Network
10 GbE Switch 72p are used for uplinks. This provides a 2:1 oversubscription to the core.
5
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Figure 3: 10 GbE networking configuration for Exadata Database Machine X3-8, single full rack
Table 3 shows all the components required for this proposed networking configuration. Either the
copper or optical cables can be used to connect Sun Network 10 GbE Switch 72p to the database
servers.
TABLE 3: COMPONENTS REQUIRED FOR CONNECTING EXADATA DATABASE MACHINE X3-8, SINGLE FULL RACK, TO 10 GBE
NETWORK
QUANTITY
PART NUMBER
DESCRIPTION
1
x2074A-F/x2074A-R
Sun Network 10 GbE Switch 72p
Connections for Sun Network 10 GbE Switch 72p
10 GBE – SFP+ MULTIMODE FIBER (10 G BASE-SR) TO DATABASE SERVERS
4
X2124A
QSFP transceivers
6
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
4
16
X2127A-10 m
MPO-to-4xLC splitter cables, 10 meters
X2127A-20 m
MPO-to-4xLC splitter cables, 20 meters
X2127A-50 m
MPO-to-4xLC splitter cables, 50 meters
X2129A-N
SFP+ SR transceiver
10 GBE – SFP+ SINGLE MODE FIBER (10 G BASE-LR) TO DATABASE SERVERS
4
X2124A
QSFP transceivers
4
X2127A-10 m
MPO-to-4xLC splitter cables, 10 meters
X2127A-20 m
MPO-to-4xLC splitter cables, 20 meters
X2127A-50 m
MPO-to-4xLC splitter cables, 50 meters
X2129A-N
SFP+ SR transceiver
16
10 GBE – SFP+ TWINAX COPPER TO DATABASE SERVERS
4
X2125A-1M-N
Twinax QSFP -4xSFP+ splitter, 1 meter
X2125A-3M-N
Twinax QSFP -4xSFP+ splitter, 3 meters
X2125A-5M-N
Twinax QSFP -4xSFP+ splitter, 5 meters
10 GBE – SFP+ TO UPLINKS
8
Customer supplied
SFP+ Transceivers, Cables
Multirack Exadata Database Machine X3-8 Configuration
A multirack configuration with four Exadata Database Machine X3-8 racks can utilize a single Sun
Network 10 GbE Switch 72p for connection to the client’s 10 GbE network. One Sun Network
10 GbE Switch 72p has enough ports to provide 10 GbE network connectivity to all the database
servers in the four racks.
Each Exadata Database Machine X3-8 rack has two database servers with eight cores, eight 10 G
SFP+ ports, and eight 1 G Base-T ports. Sun Network 10 GbE Switch 72p has 16 (four 10 G) QSFP
ports, which are connected to the database servers across all four racks and eight 10 G SFP+ ports
dedicated for uplink with an 8:1 oversubscription to the core.
7
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Figure 4: 10 GbE networking configuration for Exadata Database Machine X3-8, multiple full racks
Given below are all the components required for this proposed networking configuration. Either
copper or optical cables can be used to connect Sun Network 10 GbE Switch 72p to the database
servers.
TABLE 4: COMPONENTS REQUIRED FOR CONNECTING FOUR EXADATA DATABASE MACHINE X3-8 FULL RACKS TO 10 GBE
NETWORK
QUANTITY
PART NUMBER
DESCRIPTION
1
x2074A-F/x2074A-R
Sun Network 10 GbE Switch 72p
Connections for Sun Network 10 GbE Switch 72p
10 GBE – SFP+ MULTIMODE FIBER (10 G BASE-SR) TO DATABASE SERVERS
16
X2124A
QSFP transceivers
16
X2127A-10 m
MPO-to-4xLC splitter cables, 10 meters
X2127A-20 m
MPO-to-4xLC splitter cables, 20 meters
X2127A-50 m
MPO-to-4xLC splitter cables, 50 meters
X2129A-N
SFP+ SR transceiver
64
10 GBE – SFP+ SINGLE MODE FIBER (10 G BASE-LR) TO DATABASE SERVERS
16
X2124A
QSFP transceivers
8
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
16
64
X2127A-10m
MPO-to-4xLC splitter cables, 10 meters
X2127A-20 m
MPO-to-4xLC splitter cables, 20 meters
X2127A-50 m
MPO-to-4xLC splitter cables, 50 meters
X2129A-N
SFP+ SR transceiver
10 GBE – SFP+ TWINAX COPPER TO DATABASE SERVERS
16
X2125A-1M-N
Twinax QSFP -4xSFP+ splitter, 1 meter
X2125A-3M-N
Twinax QSFP -4xSFP+ splitter, 3 meters
X2125A-5M-N
Twinax QSFP -4xSFP+ splitter, 5 meters
10 GBE – SFP+ TO UPLINKS
8
Customer-supplied
SFP+ transceivers, cables
High Availability Option for Multirack Exadata Database Machine
X3-8 Configuration
When high availability is desired, multiple Exadata Database Machine X3-8 racks can be connected via
two Sun Network 10 GbE Switch 72p switches to enable redundant connections to the client’s
10 GbE network. The two Sun Network 10 GbE Switch 72p switches are each connected to all the
database servers and to the 10 GbE network via eight uplink ports.
Each Exadata Database Machine X3-8 rack has two database servers with eight cores, eight 10 G
SFP+ ports, and eight 1 G Base-T ports. Sun Network 10 GbE Switch 72p has 16 (four 10 G) QSFP
ports, which are connected to the database servers across all four racks with four 10 G SFP+ ports
from each database server going to a QSFP port on each Sun Network 10 GbE Switch 72p for high
availability. Sun Network 10 GbE Switch 72p has eight 10 G SFP+ ports dedicated for uplink. This
provides a 4:1 oversubscription to the core.
9
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Figure 5: 10 GbE high-availability networking configuration for Exadata Database Machine X3-8, multiple full racks
Table 5 shows all the components required for this proposed networking configuration. Either the
copper or optical cables can be used to connect Sun Network 10 GbE Switch 72p to the Oracle
Database servers.
TABLE 5: COMPONENTS REQUIRED FOR CONNECTING FOUR EXADATA DATABASE MACHINE X3-8 FULL RACKS TO 10 GBE
NETWORK
QUANTITY
PART NUMBER
DESCRIPTION
2
x2074A-F/x2074A-R
Sun Network 10 GbE Switch 72p
Connections for Sun Network 10 GbE Switch 72p
10 GBE – SFP+ MULTIMODE FIBER (10 G BASE-SR) TO DATABASE SERVERS
16
X2124A
QSFP transceivers
16
X2127A-10 m
MPO-to-4xLC splitter cables, 10 meters
X2127A-20 m
MPO-to-4xLC splitter cables, 20 meters
X2127A-50 m
MPO-to-4xLC splitter cables, 50 meters
X2129A-N
SFP+ SR transceiver
64
10
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
10 GBE – SFP+ SINGLE MODE FIBER (10 G BASE-LR) TO DATABASE SERVERS
16
X2124A
QSFP transceivers
16
X2127A-10 m
MPO-to-4xLC splitter cables, 10 meters
X2127A-20 m
MPO-to-4xLC splitter cables, 20 meters
X2127A-50 m
MPO-to-4xLC splitter cables, 50 meters
X2129A-N
SFP+ SR transceiver
64
10 GBE – SFP+ TWINAX COPPER TO DATABASE SERVERS
16
X2125A-1M-N
Twinax QSFP -4xSFP+ splitter, 1 meter
X2125A-3M-N
Twinax QSFP -4xSFP+ splitter, 3 meters
X2125A-5M-N
Twinax QSFP -4xSFP+ splitter, 5 meters
10 GBE – SFP+ TO UPLINKS
8
Customer-supplied
SFP+ transceivers, cables
How to Configure the Network for Oracle Exadata
This section provides an example showing how to configure the network connections when
connecting an Exadata Database Machine X3-2 system to the client’s 10 G network using the Oracle
Switch ES1-24s switch. This can be used as a guideline for configuring Exadata Database Machine
X3-8 systems using Sun Network 10 GbE Switch 72p. However, note that the logical link aggregation
(LLA) feature is supported only on Oracle Switch ES1-24.
There are two steps in the configuration process. The first step is to configure the ports on the
database servers within Oracle Exadata. The onboard 10 G Base-T ports from the servers are
connected to the 10 G Base-T ports on the Oracle Switch ES1-24 10 G switches. These ports can take
advantage of logical link aggregation (when connecting to an Oracle Switch ES1-24) with active-active
connections, or they can be configured with active-standby connections in an HA setup.
The second step is to configure the uplink ports from Oracle Switch ES1-24 so that the ports are
connected to the client’s 10G network. These ports can be configured using L2 or L3 uplink
connectivity.
Each of these configuration options is covered in the following sections.
Database Servers to Oracle Switch ES1-24 10 G Switch
Consider the topology shown in Figure 6. The Exadata Database Machine X3-2 system has eight
database servers, each with four 10 G Base-T ports. From each server, one of the 10 G Base-T ports is
connected to 10 G Base-T ports on each of the Oracle Switch ES1-24 10 G switches. The ports on the
server can be configured in active-standby mode or active-active mode.
11
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
If LLA is configured on the Oracle Switch ES1-24 switches with an inter-switch link (ISL)
port-channel between them, then the server ports connecting to the Oracle Switch ES1-24 switches
should be configured in active-active mode. This supports rapid failover and increased capacity with
the server-bound traffic being hashed on either the direct link to the server or over the ISL.
On the Oracle Solaris server, the ports are aggregated using “aggr” whereas on the Oracle Linux
server, they are aggregated using “bonding,” as shown below.
[Host-6 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth1 (primary_reselect always)
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:10:e0:22:0f:d9
Slave queue ID: 0
Slave Interface: eth6
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:10:e0:22:0f:da
Slave queue ID: 0
12
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Figure 6: 10 GbE HA networking configuration for Exadata Database Machine X3-2, single full rack—L2 uplinks
In the topology shown in Figure 6, the onboard 10 G Base-T ports on the database server are
connected to port 1 on each of the Oracle Switch ES1-24 switches. VLANs 2 and 3 are configured on
the database server.
Given below is the L2 configuration for the Oracle Switch ES1-24 switches.
First disable GVRP, GMRP and then configure the VLANs 2 and 3.
SEFOS# configure terminal
SEFOS(config)# set gvrp disable
SEFOS(config)# set gmrp disable
SEFOS(config)# set port-channel enable
SEFOS(config)# interface vlan 1
SEFOS(config-if)# shutdown
SEFOS(config-if)# no ip address
SEFOS(config-if)# end
13
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Step 1: Create VLANs 2 and 3 on the Oracle Switch ES1-24 switches.
SEFOS(config)# vlan 2
SEFOS(config-vlan)# ports add extreme-ethernet 0/1
SEFOS(config-vlan)# vlan active
SEFOS(config-vlan)# exit
SEFOS(config)# vlan 3
SEFOS(config-vlan)# ports add extreme-ethernet 0/1
SEFOS(config-vlan)# vlan active
SEFOS(config-vlan)# exit
SEFOS(config-vlan)# end
SEFOS# write startup-config
Step 2: Configure the port connecting to database server as trunk port.
SEFOS# configure terminal
SEFOS(config)# interface range extreme-ethernet 0/1
SEFOS(config-if)# switchport mode trunk
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
Oracle Switch ES1-24 10 G Switch to Client’s 10 G Network Using L2 Uplinks
Consider the topology shown above in Figure 6. The four 10 G SFP+ ports on the Oracle Switch ES124 10 G switches are reserved for uplink connection to the client’s 10 GbE network. For configuring
the uplink ports to the core switches, the guidelines below should be followed. In an HA setup with
two core switches, connect two 10 G ports from Oracle Switch ES1-24 to each of the core switches, as
shown in Figure 6. The uplink ports are aggregated and configured in trunk mode to allow the traffic
for VLANs 2 and 3 to be carried.
Note: If the client’s core switches provide only 1 G connectivity, the setting on the uplink ports should
be “speed 1000.” At 1 G speed, the ports operate in forced-1000-FDX mode; hence, auto-negotiation
MUST be disabled on the client’s core switch ports connecting to the uplink ports on Sun Network
10 GbE Switch 72p.
Step 3: Configure port aggregation for uplink ports. Port-channel 10 aggregates ports 21–22 to core
switch1. Port-channel 11 aggregates ports 23–24 to core switch2.
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 10
SEFOS(config-if)# no shutdown
SEFOS(config)# exit
SEFOS(config)# interface port-channel 11
SEFOS(config-if)# no shutdown
SEFOS(config)# end
14
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Step 4: Add ports to uplink port-channels.
SEFOS# configure terminal
SEFOS(config)# interface range extreme-ethernet 0/21-22
SEFOS(config-if)# no shutdown
SEFOS(config-if)# channel-group 10 mode active
SEFOS(config-if)# exit
SEFOS(config)# interface range extreme-ethernet 0/23-24
SEFOS(config-if)# no shutdown
SEFOS(config-if)# channel-group 11 mode active
SEFOS(config-if)# end
Step 5: Configure uplink port-channels to allow all VLANs.
SEFOS# configure terminal
SEFOS(config)# interface port-channel 10
SEFOS(config-if)# switchport mode trunk
SEFOS(config-if)# no shutdown
SEFOS(config)# exit
SEFOS(config)# interface port-channel 11
SEFOS(config-if)# switchport mode trunk
SEFOS(config-if)# no shutdown
SEFOS(config)# end
Note: In the above topology shown in Figure 6, if the core switches are running vPC, then portchannels 10 and 11 should be configured for LACP mode, and the individual ports of port-channels 10
and 11 should connect to both switches:
Ports 21, 23 to core switch1
Ports 22, 24 to core switch2
For detecting and avoiding loops in the network, configure the MSTP protocol. However, if the core
switch is configured for PVRST (Cisco proprietary protocol), configure PVRST on Oracle Switch
ES1-24, as shown below. Check the spanning tree for each VLAN using the command show
spanning-tree.
Note: While using PVRST, the ports can be either access ports or trunk ports. By default, a trunk port
will become a member of all VLANs configured on the switch. VLAN pruning capability is not
currently supported.
Step 6: Configure L2- PVRST on Oracle Switch ES1-24 such that it is not the root bridge for any
VLANs configured. Doing so makes the core switch become the root, and only intended traffic (not all
core traffic) reaches the Oracle Switch ES1-24 switches.
SEFOS# configure terminal
SEFOS(config)# spanning-tree mode
Spanning Tree enabled protocol is
PVRST is started.
PVRST Module status is changed
SEFOS(config)# spanning-tree vlan
SEFOS(config)# spanning-tree vlan
SEFOS(config)# end
pvrst
MSTP, now MSTP is being shutdown
2 brg-priority 61440
3 brg-priority 61440
15
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Oracle Switch ES1-24 10 G Switch to Client’s 10 G Network Using L3 Uplinks
Consider the topology shown in Figure 7, where the uplink ports from the Oracle Switch ES1-24
switch connecting to the client’s core switch are configured as L3 uplinks. Uplink ports 21 to 24 are
aggregated into port-channel 1 and are members of VLAN 20. VLAN 20 is assigned an IP address that
is in the same subnet as the ports on the core switch connecting to the uplink ports on Oracle Switch
ES1-24. Port 1 on Oracle Switch ES1-24 is connected to the database server and is a member of
VLAN 10. VLAN 10 is assigned an IP address in the same subnet as the database servers in VLAN 10.
Note: The database servers must be set up in active-standby mode since L3-based configurations are
not supported with LLA.
At the least, static routing should be set up on Oracle Switch ES1-24 to reach the default external
gateway. Additionally, if needed, dynamic routing protocols such as RIP and OSPF (in accordance with
the routing protocol defined on the core) could be configured.
The inter-switch link (ISL) between the two switches is established over port 15 and VLAN 99 is
configured as ISL VLAN.
Figure 7: 10 GbE HA networking configuration with virtual router redundancy protocol (VRRP) for Exadata Database Machine X32 single full rack—L3 uplinks
16
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Enable/disable default features and then configure the VLANs:
SEFOS# configure terminal
SEFOS(config)# set gvrp disable
SEFOS(config)# set gmrp disable
SEFOS(config)# set lldp enable
SEFOS(config)# set port-channel enable
SEFOS(config)# ip routing
SEFOS(config)# int vlan 1
SEFOS(config-if)# no ip address
SEFOS(config-if)# shut
SEFOS(config)# exit
Step 1: Configure untagged VLANs between the two Oracle Switch ES1-24 switches as “access
VLAN”.
SEFOS# configure terminal
SEFOS(config)# vlan 99
SEFOS(config-vlan)# ports add extreme-ethernet 0/15 name ISL
SEFOS(config-vlan)# vlan active
SEFOS(config-vlan)# exit
SEFOS(config)# interface extreme-ethernet 0/15
SEFOS(config-if)# switchport access vlan 99
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
Step 2: Configure VLAN 10 as tagged on the Oracle Switch ES1-24 connecting to the database servers.
SEFOS# configure terminal
SEFOS(config)# vlan 10
SEFOS(config-vlan)# ports add extreme-ethernet 0/1
SEFOS(config-vlan)# ports add extreme-ethernet 0/15
SEFOS(config-vlan)# vlan active
SEFOS(config-vlan)# exit
SEFOS(config-vlan)# end
Step 3: Assign L3 addresses to ISL VLAN and VLAN 10.
SEFOS# configure terminal
SEFOS(config)# interface vlan 99
SEFOS(config-if)# ip address 99.99.99.101 255.255.255.0
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS# configure terminal
SEFOS(config)# interface vlan 10
SEFOS(config-if)# ip address 192.168.10.101 255.255.255.0
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
Step 4: Configure VRRP.
SEFOS# configure terminal
SEFOS(config)# router vrrp
SEFOS(config-router)# int vlan 99
17
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
SEFOS(config-router)#
SEFOS(config-router)#
SEFOS(config-router)#
SEFOS(config-router)#
SEFOS(config-router)#
SEFOS(config-router)#
SEFOS(config-router)#
SEFOS(config-router)#
SEFOS(config-router)#
vrrp 1 ipv4 99.99.99.1
vrrp 1 accept-mode enable
vrrp 1 priority 110
exit
int vlan 10
vrrp 1 ipv4 192.168.10.1
vrrp 1 accept-mode enable
vrrp 1 priority 110
exit
Note: About priority, it will be configured only on the primary switch; other switches will have default
priority 100.
Note: The IP addresses for VLANs 10 and 99 will be x.x.x.101 on one switch and x.x.x.102 on the
other.
On the server side, the static route to reach interface VLAN 10 via VRRP address 192.168.10.1 should
be configured.
Step 5: Create the port-channel for the uplink ports and add members to the uplink port-channel.
SEFOS# configure terminal
SEFOS(config)# interface port-channel 1
SEFOS(config-if)# switchport access vlan 20
SEFOS(config-if)# no shutdown
SEFOS(config)# exit
SEFOS(config)# interface extreme-ethernet 0/21-24
SEFOS(config-if)# channel-group mode 1 active
SEFOS(config-if)# no shutdown
SEFOS(config)# end
Step 6: Assign IP addresses to VLAN 20.
SEFOS# configure terminal
SEFOS(config)# interface vlan 20
SEFOS(config-if)# ip address 192.168.20.101 255.255.255.0
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
Step 7: Set the default route to the external gateway reachable through the uplink port-channel 1.
SEFOS# configure terminal
SEFOS(config)#ip route 0.0.0.0 0.0.0.0 192.168.20.1 1
SEFOS(config)# exit
Step 8: Save the configurations.
SEFOS# copy running-config startup-config
18
How to Connect Oracle Exadata to 10G Networks Using Oracle’s Ethernet Switches
Conclusion
There are many variables when connecting and configuring an Oracle Exadata system to a data center
network. This document is an attempt to simplify that process and reduce risk for Oracle customers by
providing detailed instructions for how to use Oracle networking products with Oracle Exadata.
By following the guidelines outlined in this paper, readers can properly set up and configure the
network environment to connect an Oracle Exadata system to their existing 10 G network.
References
For more information, visit the web resources listed in Table 6.
TABLE 6. WEB RESOURCES FOR FURTHER INFORMATION
WEB RESOURCE DESCRIPTION
WEB RESOURCE URL
Oracle Exadata
www.oracle.com/technetwork/server-storage/engineered-systems/exadata/
Oracle's Sun Network 10 GbE Switch 72p
www.oracle.com/us/products/networking/ethernet/top-of-rackswitches/overview/
Oracle Switch ES1-24
www.oracle.com/us/products/networking/ethernet/switch-es1-24/overview/
Sun Ethernet Fabric Operating System CLI Base
http://docs.oracle.com/cd/E19934-01/html/E26513/
Reference Manual
19
How to Connect Oracle Exadata to 10G
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Networks Using Oracle’s Ethernet Switches
October 2013, Version 1.0
This document is provided for information purposes only, and the contents hereof are subject to change without notice. This
Author: Savi Venkatachalapathy
document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in
law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any
Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.
liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This
document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our
prior written permission.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Worldwide Inquiries:
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and
Phone: +1.650.506.7000
are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are
Fax: +1.650.506.7200
trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0113
oracle.com