Solution overview
CPU: 2 x Intel Xeon™ Gold 6136 CPU with 12 cores at 3.0 GHz
Memory: 24x 16 GB at 2666 MHz
MDS SAS controller: 2 x Dell 12 Gbps SAS HBAs
MDS Storage Array: 1 x Dell EMC PowerVault ME4024
> Disks: 12 x 960 GB SAS SSD (active/passive configuration, no DNE) or 24 x 960 GB SAS
SSD (active/Aactive Configuration, DNE)
2.4
Network components
2.4.1
Dell networking H1048-OPF
Intel Omni-Path Architecture (OPA) is an evolution of the Intel True Scale Fabric Cray Aries interconnect and internal Intel IP (12). In contrast to Intel True Scale Fabric edge switches that support 36 ports of InfiniBand
QDR-40Gbps performance, the new Intel Omni-Path fabric edge switches support 48 ports of 100 Gbps performance. The switching latency for True Scale edge switches is 165 ns-175 ns. The switching latency for the 48-port Omni-Path edge switch has been reduced to around 100 ns-110 ns. The Omni-Path host fabric interface (HFI) MPI messaging rate is expected to be around 160 million messages per second (MMPS) with a link bandwidth of 100 Gbps.
2.4.2
Mellanox SB7700
This 36-port Non-blocking Managed InfiniBand EDR 100 Gb/s Switch System provides the highest performing fabric solution in a 1U form factor by delivering up to 7.2 Tb/s of non-blocking bandwidth with 90 ns port-toport latency.
2.4.3
Dell networking S3048-ON
Management traffic typically communicates with the baseboard management controller (BMC) on the compute nodes using IPMI. The management network is used to push images or packages to the compute nodes from the master nodes and for reporting data from client to the master node. Dell EMC Networking
S3048-ON is recommended for management network.
2.5
Reference architecture
This reference architecture is configured with Intel OPA fabric, Dell EMC Ready Solution for HPC NFS
Storage and Dell EMC Ready Solution for HPC Lustre Storage. Dell EMC Ready Solution for HPC NFS
Storage, referred to as Dell NFS Storage Solution- HA configuration is configured for a general usage such as a home directory, while Dell EMC Ready Solution for HPC Lustre Storage servers as a high-performance
scratch storage system. An example solution is illustrated in Figure 3. Although benchmark testing was
performed with 64 C6420s, throughput test results showed that 56 compute nodes are optimal with medium base configuration of Dell EMC Ready Solution for HPC Lustre Storage (9).
11 Reference Architecture of Dell EMC Ready Solution for HPC Life Sciences | Document 309
Solution overview
Figure 3 Dell EMC Ready Solution for HPC Life Sciences with Intel® OPA fabric
Figure 4 illustrates the details of a management network. It requires three Dell Networking S3048-ON
switches. Two switches in Rack 1 (Figure 3) make up the internal management network for deploying,
provisioning, managing, and monitoring the cluster. One of the switches in Rack 2 has 48 ports that are split into multiple untagged virtual LANs to accommodate multiple internal networks.
12 Reference Architecture of Dell EMC Ready Solution for HPC Life Sciences | Document 309
