High Performance High Performance, High Capacity tktl ti

High Performance,
Performance
High Capacity
network
t
k storage
t
solutions
l ti
7th TERENA Storage Meeting
Poznan, Sept.
p 9, 2010
Toine Beckers: tbeckers@ddn.com
tbeckers@ddn com
Auke Kuiper: akuiper@ddn.com
© 2009 DataDirect Networks, Inc. All Rights Reserved.
Agenda
g
• S2A6620: Entry Level Storage
• S2A9900: HPC & Archiving
g
• SFA10K:
SFA10K HPC (Embedded)
(E b dd d)
Company
p y Data at a Glance
•
Fast Growing Data Infrastructure Provider for Companies who
Demand Extreme Performance for their Large Content Files
and Unstructured Data
– Integrated Portfolio of Extreme Storage Platforms, Intuitive
S
Storage
M
Management S
Software
f
and
dC
Consulting
li S
Services
i
– Over 10 Years of Stability and Experience
– Over $150M in Annual Revenue
• Growing, Profitable and Hiring
– Over 250 Petabytes Installed Worldwide
• Clients include XBOX LIVE
LIVE, Slide and Saudi Aramco
– Global Partners include IBM, Sony and Dell
– Expanding Globally, with Established Offices
• Europe,
Europe India
India, Australia
Australia, Asia Pacific and Japan
“DDN could find itself a market leading provider for the Internet computing era in
the same way
a that EMC did in the transactional era and NetApp did in the
distributed era”
DDN = HPC
•
•
•
•
•
DDN provides more
bandwidth to the top500 list
than all other vendors
combined!
8 out of Top10 systems
choose DDN
45 out of Top100
5 systems over 120GB/s
Mix of applications:
– Government/University
– Defense/Intelligence
– Oil Exploration
E l
ti
– Product Design
– Archival, Backup
A Common Building
g Block
Multipath Controller I/O
SS6000 60-Slot
Drive Enclosure
IO Module or
IO
Module or
6620 Controller
SAS
Expander
• Used in SFA, S2A and WOS Products
SAS
Expander
SAS
Expander
SAS
Expander
IO Module or
IO
Module or
6620 Controller
SAS
Expander
SAS
Expander
SAS
Expander
SAS
Expander
• Long Term Investment Protection
•
•
•
•
Non-blocking performance over 2.4GB/s
Fully
y redundant - no single
g p
point of failure
Dual-porting SATA for end-end path failover
Mix SAS, SATA & SSD in one enclosure
15 x Dual‐Port HDDs/SSDs
15 x Dual‐Port HDDs/SSDs
15 x Dual‐Port HDDs/SSDs
15 x Dual‐Port HDDs/SSDs
SS6000 Internals
Completely Fault Tolerant
Experts in Extreme Scaling
Far ahead of the pack
# of Supercomputer CPUs
300,000
Fastest File System: 240GB/s
World’s Fastest Supercomputer
20,000,000
4,000,000,000
Online Users Served
Xbox Live Community
Individual Photos
~25 PBs of Storage
Accelerating
g Accelerators
DDN is the leading provider of
affordable, high-availability
storage for the next generation
of p
particle p
physics
y
research.
DDN Supplied Over 20PB of
Atlas Storage in 2009 Alone
LHC Customer Base
• Tier 0
– CERN-LHCb (1*S2A8500, 30TB)
– CERN-LHCb (1*S2A9900,
(
, 60TB))
• Tier 1
–
–
–
–
–
–
–
–
BIGGRID (13
(13*S2A9900
S2A9900, 6 PB)
FZK-2009 (7*S2A9900, 10PB)
FZK-2010 (3*S2A9900, 6.6PB)
FZK (SFA10K,
(SFA10K 0.6PB)
0 6PB)
TRIUMF (2*DCS9900, 0.6PB)
IN2P3 (7*DCS9550, 1.5PB)
PIC (2*S2A9900,
(2*S2A9900 2
2.4PB)
4PB)
INFN-CNAF (5*S2A9900, 7PB)
• Tier 2
–
–
–
–
–
DESY (2*S2A9900, 1.2PB)
DESY (2*SFA10K, 1.8PB)
NBI ((1*S2A6620,, 60*2TB))
INFN-PISA (1*S2A9900, 250TB)
INFN-PADOVA (1*S2A9900, 240TB)
We Serve Clients that
Others Simply Can’t
• For Companies
p
with Throughput-Intensive,
g p
, Real-Time
Environments, DataDirect Networks Leads the Industry
– Post Production and Broadcast
– Internet
I t
t Media,
M di Real-Time
R l Ti
Web
W b 2.0
20
– High Performance Computing
• For Companies
p
requiring
q
g Scalable Capacity
p
y with Consistent
Performance, no Other Storage Platform Comes Close
– Deep Archival
– Engineering and Life
f Sciences
S
– Oil & Gas
The Pinnacle in Video Storage
g
DDN has delivered solutions to
over 500 of the world’s
l
largest
media
di organizations
i
i
Sample HPC Partners & Customers

S2A9900
Purpose-Built
High Performance Streaming & Archiving Storage Platform
© 2009 DataDirect Networks, Inc. All Rights Reserved.
S2A Design
g Architecture
Low Latency - High Performance,
Performance Silicon
Based Storage Appliance
•Parallel Access For Hosts
•Parallel Access To A Large Number Of Disk Drives
•Quality
Q lit Of S
Service
i
•Scalability
•Drive
Drive Error Recovery In Real Time
•True State Machine Control
– 10 Virtex 4 FPGAs, 16 Intel embedded processors, 8
Data FPGAs
An Implementation of
Parallelism w/ Double Parity
RAID Protection
8 FC-8 and/or
4 IB 4X
Parallel Host Ports
 Double Disk Failure
Protection
 LUNs can span tiers
 All ports access all
storage
t
 Reed-Solomon Code
Implemented in a
Hardware State Machine
2 x 10 SAS
Loops to
Disks
− No penalty for RAID 6!
Tier 2 A
B
C
D
E
F
G
H
P
P
 No loss of performance
on any failure
Tier 3 A
B
C
D
E
F
G
H
P
P
Multi Tier Storage
 Multi-Tier
Support, SSD, SAS,
SATA Disks
RAID 6, 8+2 Byte Stripe
Enclosure 10
P
Enclosure 9
P
Enclosure 8
H
Enclosure 7
G
Enclosure 6
F
Enclosure 5
E
Enclosure 4
D
Enclosure 3
C
Enclosure 2
B
Enclosure 1
Tier 1 A
 Parity Computed On
Writes AND Reads
 Up to 1200 disks total
RAID 0
• 960 formattable disks
Simple,
p , Reliable Configuration
g
Direct Connection and RAID
Striping Provides Maximum
Data Availability
y
Direct cabling avoids daisy chaining
Data is striped across
channels/enclosures
Drive Channels are RAIDed 8+2
Drive Enclosures are RAIDed 8+2
Only DDN Enclosure RAIDing
can withstand the loss of
20% of system enclosures &
drives while delivering full
data availability!!
SATAssure Data Integrity
• The Parallel Data Recovery Engine allows data
reconstruction and integrity checking
– S2A hardware enables SATAssure software to verify
all data read from the disks (no silent data
corruption)
– S2A h
hardware
d
allows
ll
SATA
SATAssure tto send
dh
hosts
t
“fixed” data (data integrity is assured)
– S2A hardware enables SATAssure to correct data on
the disk for future accesses (self-healing array)
– Multiple levels of disk recovery attempted before
failing drives (replace fewer drives)
– S2A controller journaling allows partial rebuilds (less
time in degraded mode)
Data Corruption Error Handling
FPGA
Host Data Striping
A
B
C
D
E
F
G
H
P
S
G
H
P1
P2
Second step corrects error
First step isolates error
A
B
C
D
E
F
Cache
Protocol/PHY
A
B
C
D
E
Incorrect Data
SCSI
FC - or - SAS
F
G
H
P
S
Disks
FPGA
Host Data Striping
B
C
D
E
F
G
H
P
S
XO
OR 2
XOR 1
X
A
The
Thedata
cache
is flushed
has been
to the
repaired
disk and
by the
the FPGA
disks have
usingnow
the parity
correct
Data
Information.
on channel F.
A
B
C
D
E
F
G
H
P
S
Cache
SCSI
FC - or - SAS
A
B
C
D
E
F
Protocol/PHY
G
H
P
S
Disks
Data Center Efficiency
•



•
•
Leading
g Power Efficiency
y
Only 4 x 30A 220V Drops per 600TB
1/4th the components of competing
storage: less power supplies, fans
Additional D-MAID reduces
consumption even further
Truly Green Storage!
S2A D-MAIDTM

Intelligent Power Management

Optimized for Backup/VTL/Archive

Spin Down Tiers of Inactive Drives

12 seconds to spin up

No Application Change
Active
Dynamic MAID*
300TB (300 x 1TB SATA)
7.1 kW
4.5 kW
600TB (600 x 1TB SATA)
13 5 kW
13.5
8 29 kW
8.29
1.2PB (1200 x 1TB SATA)
26.1 kW
15.8 kW
1.2PB Dynamic-MAID Savings
Up to $36,000/yr
* S2A D-MAID Savings results assume 80% data dormancy for online archive, $0.20 kWhr
Worst Case Recovery
• Di
Disks
k can b
become completely
l t l unresponsive
i
to all commands
– Th
The internal
i t
l OS can enter
t a loop
l
th
thatt does
d
nott
enable external commands
– A power cycle always recovers the drive
– S2A 9900 automatically power cycles a drive in
place
• Drives that issue SMART warnings or grow
defects at an increasing rate are copied to
spare drives
S2A9900 Configuration
with 10 Enclosures
8 FC-8 and/or
4 IB 4X
Parallel Host Ports
2 x 10 SAS
Loops to
Disks
Tier 2 A
B
C
D
E
F
G
H
P
P
Tier 3 A
B
C
D
E
F
G
H
P
P
Tier 60
Enclosure 10
P
Enclosure 9
P
Enclosure 8
H
Enclosure 7
G
Enclosure 6
F
Enclosure 5
E
Enclosure 4
D
Enclosure 3
C
Enclosure 2
B
Enclosure 1
Tier 1 A
S2A9900 Configuration
with 5 Enclosures
8 FC-8 and/or
4 IB 4X
Parallel Host Ports
2 x 10 SAS
Loops to
Disks
F
G
H
P
P
Tier 2 A
B
C
D
E
F
G
H
P
P
Tier 3 A
B
C
D
E
F
G
H
P
P
Tier 30
Enclosure 5
E
Enclosure 4
D
Enclosure 3
C
Enclosure 2
B
Enclosure 1
Tier 1 A
S2A9900 Configuration
with 3 Enclosures
8 FC-8 and/or
4 IB 4X
Parallel Host Ports
2 x 10 SAS
Loops to
Disks
D
E
F
G
H
P
P
Tier 2 A
B
C
D
E
F
G
H
P
P
Tier 3 A
B
C
D
E
F
G
H
P
P
Tier 15
Enclosure 3
C
Enclosure 2
B
Enclosure 1
Tier 1 A
Scalability,
y Density
y & Sustained Performance
The World Scalability & Density Leaders


3 Enclosures
16U
5 Enclosures
24U: 1/2 Rack
10 Enclosures
44U: 1 Rack
20 Enclosures
84U: 2 Racks
Up to 150 Drives
Up to 300TB
Up to 2.8 GB/s
Up to 300 Drives
Up to 600TB
Up to 5.6 GB/s
Up to 600 Drives
Up to 1.2PB
Up to 5.6 GB/s
Up to 1,200 Drives
Up to 2.4PB
Up to 5.6 GB/s
Simple Cabling: All Enclosures are direct connected (up to 10 enclosures) to the
S2A Appliances for easy configuration and maximum reliability.
M i
Maximum
A
Availability:
il bilit S2A St
Storage S
Systems
t
can llose up tto 20% off th
the available
il bl
drive enclosures without impacting host performance or data availability.
Extreme Storage Platform
An integrated solution portfolio
File
Sttorage
Parallel File Storage
ExaScaler
GridScaler
Storage
Systems
20K Clients
200GB/s+
1000s of Clients
200GB/s+, NAS
Virtual Tape
Object Storage
xStreamScaler
xStream VTL
WOS
100s SAN/LAN Clients
HSM Capable
1-8 VTL Servers
FC, iSCSI, iSER
40 Billion Objects
4 Replicated Sites
S2A9900
S2A6620
SFA10000
6GB/s
1,200 HDD/SSD in 2 Racks
2GB/s, Up to 350K IOPS
120 HDD/SSD in 8U
10GB/s, Up to 1M IOPS
1,200 HDD/SSD in 2 Racks
SAS
Disks
SAN File Storage
Up to 15K
SATA
Up to 2TB
SLC SSD
32 & 64GB
p to 35K I0PS
Up
S2A6620
Entry Level Mixed Workload Platform
© 2009 DataDirect Networks, Inc. All Rights Reserved.
StorageScaler
Begin – Standalone RAID
SBB RAID M
Module
d l
Upgrade to….
S2A9900
S2A6620
Replace SBB RAID Modules
with SBB SAS Expander
M d l
Modules
Replace SBB RAID Modules
with SBB Storage Server
Modules
S2A6620
Up to 350,000 Cache IOPS, 30,000 Disk IOPS
4 x Active/Active Host Ports: FC4, FC8
Scales to Support 120 Hard Drives in 8U
Up to 2.0 GB/s Performance
Mix SSD, SAS + SATA For Storage Tiering
Up to 11 Systems (660 TB) per Rack
Optimized
Applications
Metadata Storage
Database
Data Warehouse
Animation
Content Service
Remote Backup
Scale-Out
HPC/NAS
RAID 5 and RAID 6 Options
Journaled Fast Drive Rebuild
Active/Active Storage Managers with Failover
Full SATAssure Data Protection
Windows MultiPath Support
Cache Mirroring
SFA
Storage Fusion Architecture
The Next Generation
© 2009 DataDirect Networks, Inc. All Rights Reserved.
2010+ Petaflop
p Systems
y
• LLNL
– 1TB/sec and 30PB (Lustre)
• Argonne
– 500GB/sec and 60PB (GPFS, PVFS)
• ORNL
– 800GB/sec and 30PB (Lustre)
• CEA
– 500GB/sec (Lustre)
• HLRS
– 200-300GB/sec
200 300GB/sec
• LRZ
– 400GB/sec
The Case for Change
g
• CPU technology
gy has moved from increasing
g speeds
p
to
increasing compute cores
– The latest CPUs, with Hyperthreading, can run 8 threads
simultaneously,
i lt
l or 16 th
threads
d ffrom a d
dual-socket
l
k t server!!
• Single computers can now send multiple simultaneous I/O
requests to the storage system
– This looks like transactional or random I/O to the storage
– Lots
L t off file
fil operations
ti
mean lots
l t off file
fil system
t
metadata
t d t operations
ti
• New Storage systems must be able to perform well at both
high throughput (sequential) and high IOPS (transactional)
workloads
SFA10K
Using SFA as Block Storage device
© 2009 DataDirect Networks, Inc. All Rights Reserved.
SFA10000 Features
Highly Parallelized SFA Storage
Processing Engine
16 x 8Gb Fibre Channel Host Ports
or 8 x QDR InfiniBand Host Ports
SFA Interface Virtualization
SFA Interface Virtualization
Highly Parallel
Data I/O Processing,
Management, Integrity
8GB High-Speed
Cache
60Gb/s
Cache Link
Internal SAS
Switching
8GB High-Speed
Cache
Internal SAS
Switching
Massive
M
i
I/O
Back-End
480Gb/s Internal SAS Storage Management Network
1
2
3
4
1
2
3
4
1
m
5
P
RAID 5,6
6
7
8
P
RAID 5,6
Q
RAID 6
1
Up
p to 1200 SAS,, SATA or SSD Drives with full
redundant paths
Q
RAID 6
RAID 5,6
RAID 5,6
56
RAID 1
Active/Active Design
10GB/s Read & Write Speed
1 Million Burst IOPS
300K Random Disk IOPS
16GB Battery-Backed,
Mirrored Cache
RAID Levels 1, 5 and 6
Intelligent Block Striping
SATAssure Data Protection
GUI, SNMP, CLI
16 x FC-8 ports or
8 x QDR-IB ports
SFA10000 Configurations
g
5 Enclosure System
Up
p to 300 Drives
2 BBUs, 28U
10 Enclosure System
Up to 600 Drives
2 BBUs, 48U
20 Enclosure System
Up to 1,200 Drives
2 BBUs, 88U
High Availability Drive Channel & Enclosure RAIDing
SFA10KE
Using SFA as Appliance
© 2009 DataDirect Networks, Inc. All Rights Reserved.
SFA Embedded: The Vision
• Embed Storage Intensive Applications within the
Storage Controller
– Reduce complexity
complexity, infrastructure and
administration
– Reduce cost as well as lower operational cost
– Increase performance for latency sensitive
applications
li ti
Storage Fusion Architecture
Clients/Hosts
LAN
LAN
File/Application
Servers
SAN
A
Synergistic Fusion of
Application Servers,
Interconnects and
Multiple
RAID
Storage
Arrays in One
Simple-to-Manage
Arrays
Integrated Platform
SAN
SFA10000E
p
Couplet
Traditional Client/Server and Storage
Infrastructure
SFA10000E Features
Low Latency Embedded Storage
Application Platform
16 x 10Gb Ethernet Host Ports
or 16 x QDR InfiniBand Host Ports
SFA Interface Virtualization
AP
SFA Interface Virtualization
RP
8GB High8G
i h
Speed
Cache
45GB AP
Memory
RP
AP
8GB High60Gb/s
Cache LinkSpeed Cache
Internal SAS
Switching
45GB AP
Memory
Internal SAS
Switching
240Gb/s Internal SAS Storage Management Network
1
2
3
4
1
2
3
4
1
m
5
P
RAID 5,6
6
7
8
P
RAID 5,6
Active/Active Design
8 Application CPU Cores
90GB of Application RAM
16 x 10Gb Ethernet or
M
Massive
i
I/O
Back-End
16 x QDR InfiniBand Ports
Up to 6GB/s Read & Write Speed
500,000+ Burst IOPS
RAID 5,6
150K Random Disk IOPS
16GB Mirrored Cache
RAID 5,6
56
RAID Levels 1, 5 and 6
RAID 1
Intelligent Block Striping
Up to 600 SAS
SAS, SATA or SSD Drives
Parallel Storage
Processing Engine
Q
RAID 6
Q
RAID 6
1
Up
p to 600 SAS,, SATA or SSD Drives with full redundant
paths
IO Path Acceleration
Storage Fusion Architecture shortens the IO path
f
from
the
h application
li
i
to storage, reducing
d i
latency
l
and increasing IOPS performance.
Example: Implementing Lustre Today
IInfrastructure:
f
• Multiple OSS servers
• 2 Lustre MDS servers
• 1 Lustre MGS server
• Multiple RAID Arrays
f OSTs
for
OST
• RAID Array for MDTs
• SAN Switching
• Multiple Racks of
equipment requiring
power cooling and floor
power,
space
EXAScaler HPC Storage on the
SFA10000E Appliance
SFA10000E with
embedded EXAScaler can
result in a 10 to 1 or
greater reduction in
managed systems.
Storage Fusion
Architecture
not only reduces
complexity it streamlines
complexity,
IO by reducing latency
protocol conversions
and p
Eliminating Application Overhead
Embedded Services Eliminate
Communication Overhead
Communication per traditional SCSI Transfer
4KB I/Os = 10KB of Communication
32KB I/Os Become 20% Less Efficient
cce e ated Through
oug Memory
e o y Copy,
Accelerated
Eliminating SCSI Transfer
SFA Application Platform
IB/10GbE
Application
pp
X
MMAP’d
Hi-Speed
Direct I/O
IB/10GbE
Failover
Application
pp cat o X
Virtualized I/O
Virtualized I/O
KVM Driver
KVM Driver
DDN RAID Stack
Cache
Coherency
& Mirroring
DDN RAID Stack
Linux Kernel
Linux Kernel
SFA Controller
SFA Controller
Scaling Performance with the
SFA10000E Storage Appliance
GridScaler &
ExaScaler Clients
1.2PB
2.4PB
4.8PB
3.6PB
20GB/s
40GB/s
60GB/s
80GB/s
IB or 10Gig-E
GridScaler
& ExScaler
Servers
Embedded
Add additional
SFA Couplets to
y Scale
Linearly
Performance
300-600 3.5” Disk Drives
SFA10000E
Integrate
I
t
t multiple
lti l appliances
li
to scale to over 100’s of GB/s
and 10’s of Petabytes
Multi-Platform Architecture
Block Storage Array
Clustered Filer
SFA10000E
DDN File Storage
EXAScaler
GridScaler
Open Appliance
SFA10000E
Customer Applications
pp
SFA10000
SFA10000E
Block Storage Target
Embedded Storage Server
SFA10000
Block Storage Target
SFA10000
Block Storage Target
Product Evolution
Flexible Deployment Options: 3 System Modalities
© 2010
Thank You
Toine Beckers
tbeckers@ddn.com
Download PDF