US007203135B2
(12) United States Patent
(10) Patent N0.:
Kano et al.
(54)
(45) Date of Patent:
DISK ARRAY SYSTEM AND METHOD FOR
(56)
U'S' PATENT DOCUMENTS
Inventors: Azuma Kano, OdaWara (JP); Takuji
OgaWa, OdaWara (JP); Ikuya
5,423,046 A
6/1995 Nunnelley et a1‘
_
YagisaWa, Tokyo (JP)
(Commued)
(73) Assignee: Hitachi, Ltd., Tokyo (JP)
(*)
Notice:
Apr. 10, 2007
References Cited
CONTROLLING DISK ARRAY SYSTEM
(75)
US 7,203,135 B2
FOREIGN PATENT DOCUMENTS
Subject to any disclaimer, the term of this
EP
0795824
patent is extended or adjusted under 35
U.S.C. 154(b) by 114 days.
9/1997
(Continued)
OTHER PUBLICATIONS
(21)
(22)
Appl' NO': 11/024’644
Filed;
Dec_ 30, 2004
SGI In?niteStorage TP9300S Storage Array, Data Sheet, 2 pp.,
available at: http://WWW.sgi.c0rn/pdfs/3643.pdf.
(65)
Prior Publication Data
Us 2005/0154942 A1
Jul. 14, 2005
(Continued)
Primary ExamineriAli NeyZan'
_
_
(74) Attorney, Agent, or FirmiMattingly, Stanger, Malur &
Related US. Application Data
(63)
(30)
Continuation of application No. 10/766,015, ?led on
Jan. 29, 2004, noW Pat. No. 7,057,981.
Foreign Application Priority Data
Nov. 28, 2003
(51)
Brundidge’ RC
(JP)
(57)
A disk array system having ?rst and second housings and a
controller for controlling the ?rst and second housings. Fiber
........................... .. 2003-400517
Int- ClGHB 17/22
(2006-01)
G06F 11/00
(2006-01)
ABSTRACT
Channel hard disk drives are received in the ?rst housing,
and serial ATA hard disk drives are received in the second
housing. When reading data stored in a serial ATA hard disk
drive in the second housing, the controller reads a plurality
of pieces of data including the data to be read and parity data
(52)
US. Cl. ............................. ..
369/3028; 369/3038;
for the plurality of pieces of data from all the hard disk
369/30.58; 714/6; 714/43; 711/114
drives ofan RAID group to Which the hard disk drive storing
(58)
Field of Classi?cation Search ........... .. 369/3027,
the data to be read belengs- Thus, the controller examines
369/30.58, 30.28, 30.29, 30.38, 30.06, 30.4,
369/3041, 30.43, 34.01, 30.46, 30.09; 714/43,
714/6; 709/214; 711/114,162,11,111;
Whether the plurality of pieces of data including the data to
be read are Written in the hard disk drives With erroneous
contents or not.
710/11, 316
See application ?le for complete search history.
23 Claims, 19 Drawing Sheets
INFORMATION
PROCESSING N300
APPARATUS
l
CHANNH
~500
_,1001
CONVERT
~_ _. )
SWITCH
‘3.31001
SWITCH
901
902
~2102
51
US 7,203,135 B2
Page 2
US. PATENT DOCUMENTS
_
5,475,814 A
_
12/1995 Tomlmltsu
2002/0162057 A1
10/2002 Talagala
2003/0031187 A1
2/2003
2003/0041201 A1
2/2003 Rauscher
Heflernan etal.
5,546,558 A
8/1996 Jaoobson et a1.
2003/004l278 A1
200% Lin
5503903 A
5305788 A
2/1997 Aklzawa et 31'
9/1998 Johnson
2003/0046460 A1
2003/0061491 A1
3/2003 Inoue et al.
3/2003 Jaskiewicz etal
5,812,754 A
9/1998
Luietal.
.
'
5,822,782 A
10,1998 Humlicek et a1‘
2003/0097487 A1
5/2003 R1etZe etal.
5,832,204 A
“H998 Apperley et a1‘
2003/0097504 A1
5/2003 oeela etal.
5,845,319
5,848,282
5,867,736
5,884,098
5,951,691
6,061,750
6,108,748
6,115,797
6,137,679
6,154,850
6,173,360
6,201,692
6219752
12/1998
12/199g
2/1999
3/1999
9/1999
5/2000
8/2000
9/2000
10/2000
11/2000
1/2001
3/2001
Yorimitsu
Kang et a1‘
jantz
Mason, Jr,
Ng etal.
Beardsley et al.
Ofek et al.
Kanda er a1~
Chang
Idleman
Beardsley et a1.
GmTlble et a1~
4/ 2001 S‘?k‘d°
2003/0110330
2003/0115204
2003/0115437
2003/0131291
2003/0135577
2003/0135609
2003/0145167
2003/0149840
2003/0163639
2003/0167439
2003/0172150
2003/0182502
2003/0189811
6/2003
6/2003
6/2003
7/2003
7/2003
7/2003
7/2003
8/2003
8/2003
9/2003
9/2003
9/2003
2,552,345‘; E
Z5881 2151:1321“
2003/0196002 A1 10/2003 Nakayama et al.
632403486 Bl
50001 Ofek et all
2003/0196147 A1
10/2003 Hirata elal.
6,282,602 B1
80001 Blumenau et a1‘
2003/0204671 A1
10/2003 Matsunaml et al.
2003/0212859
2003/0221061
2003/0221077
2003/0231420
2004/0010660
2004/0010662
11/2003
11/2003
11/2003
12/2003
l/2004
V2004
6,351,375
6,400,730
6,449,709
6,484,236
6,484,269
6,502,108
A
A
A
A
A
A
A
A
A
A
B1
B1
B1
B1
B1
B1
B2
B1
B1
6,510,491 B1
6,549,978
6,553,408
6,606,690
6,628,513
6636934
6,640,278
6,651,137
B2
B1
B2
B1
B1
B1
B2
“585% B1
6,684,282 B1
2/2002
6/2002
9/2002
11/2002
11/2002
12/2002
Hsieh et a1‘
Latif
Gates
pujimoto et a1‘
Kopylovitz
Day, 111 et al.
1/2003 Franklin elal.
4/2003
4/2003
8/2003
9/2003
10/2003
10/2003
11/2003
Mansuretal.
Merrell et a1.
Padovano
Gallagher 9t a1~
Linnell
Nolan et a1~
Baek et a1~
12/2003 C011?“ et 31'
V2004 Kofls
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
A1
Fujle etal.
Greenblatt et a1.
Tomita
Morrison et a1.
Weber et al.
Carlson et al.
Tomita
Bolt
Banrn et al.
Talagala et al.
Kennedy
Kleirnan et al.
10/2003 Peeke etal.
Ellls
El-Bataletal.
Ohno etal.
Kano et a1.
Konshak et a1.
Aruga
2004/0024930 A1
2/2004 Nakayama etal.
2004/0034731
2004/0068610
2004/0073747
2004/0078707
2004/0088482
2004/0107325
2004/0111485
2/2004
4/2004
4/2004
4/2004
5/2004
6/2004
6/2004
A1
A1
A1
A1
A1
A1
A1
sivensen
Umberger et a1.
Lu
Apperley etal.
Tanzer et al.
Mori
Mimatsu etal.
2004/0111560 A1
2004/0117517 A1
6/2004 Takase etal.
2004/0117534 A1
20040128627 A1
6/2004 Parry et al.
7/2004 Z
2004/0139260 A1
7/2004 StelnmetZ
6/2004 Beauchamp etal.
6,684,295
6,708,232
6,728,922
6,745,287
6,763,409
B2
B2
B1
B2
B1
l/2004
3/2004
4/2004
6,2004
7/2004
FuJlIIlOtO et a1.
obara etal.
Sundaram et al.
Fujimoto et a1‘
Elliott
2004/0148329 A1
7/2004 Og'rlsawara et a1.
6,763,436
6,772,287
6,772,365
6,792,486
B2
B2
B1
B1
'7/2004
8/2004
g/2004
9/2004
Gabber et a1‘
Uchiyama et a1‘
()bara
Hanan et a1,
2004/0148460
2004/0148461
2004/0153614
2004/0158676
A1
A1
A1
A1
7/2004
7/2004
8/2004
55/2004
StelnmetZ et a1.
Steinmetz
Bitner et a1.
Kasmirsky et a1.
6,801,992
6,831,839
6,834,326
2001/0014956
2001/0019509
2001/0046118
B2
B2
B1
A1
10/2004
12/2004
12/2004
8/2001
Gajjar et al.
Bovell
Wang et al.
Nagata 9t a1~
2004/0162940
2004/0169996
2004/0177218
2004/0177228
A1
A1
A1
A1
8/2004
9/2004
9/2004
9/2004
9/2004
9/2004
Yagisawa et al.
Pauletal.
Meehan et al.
Leonhardt et al.
A1
9/2001 A119 91 a1~
A1
2001/0054136 A1
2001/0056527 A1
2002/0007469 A1
11/2001
12/2001
12/2001
V2002
5885x8332; ii
Z588;
ayfls
2004/0193760 A1
Yamanflshi er 91NFIOIIPYa eta1~
Nlnomlya eta1~
Take“? et 31'
2004/0193791
2004/0199515
2004/0221101
2004/0236908
A1
A1
A1
A1
2004/0243386 A1
Matsunami et al.
Felton etal.
10/2004 Penny etal.
11/2004 Voorhees etal.
11/2004 Suzuki etal.
12/2004 Stolowitz et al.
'
2004/0243699 A1
12/2004 Koclances et al.
2002/0049886 A1
4/2002 Fumya et 31'
2004/0267516 A1
12/2004 Jibbe et al
2002/0062387 A1
5/2002 Yatziv
2004/0268037
2004/0268069
2005/0027900
2005/0097132
2005/0117468
12/2004
12/2004
2/2005
5/2005
6/2005
2002/0062454
2002/0069317
2002/0069334
2002/0l38705
2002/014404g
A1
A1
A1
A1
A1
2002/0144057 A1
2002/0147945 A1
2002/0162048 A1
5,2002
6/2002
6/2002
9/2002
10/2002
Fung
Chow et a1‘
Hsia et al.
Suzuki et a1‘
Bolt
10/2002 Li et a1,
10/2002 Fox etal.
10/2002 Ackaretetal.
'
A1
A1
A1
A1
A1
2005/0120263 A1
2005/0120264 A1
2005/0138154 A1
Buchanan, Jr. et al.
Satoyama etal.
Pertey
Cochran et a1.
Kano et a1.
6/2005 Kano et a1.
6/2005 Kano etal.
6/2005 Seto
US 7,203,135 B2
Page 3
2005/0154942 A1*
7/2005
Kano et al. ................. .. 714/43
FOREIGN PATENT DOCUMENTS
EP
EP
EP
EP
JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
JP
0844561
1001345
1 237 087
1315074
02188835
5150909
9330182
09330184
10301720
2001142650
2001167040
2001337868
2002150746
2002/333954
2002333954
200336146
2004178557
5/1998
5/2000
9/2002
5/2003
7/1990
6/1993
12/1997
12/1997
11/1998
5/2001
6/2001
12/2001
5/2002
11/2002
11/2002
2/2003
6/2004
OTHER PUBLICATIONS
Adaptec FS4500 Fibre to SATA RAID, Data Sheet, 2 pp., available
at:
http:/www.sunstarco.com/PDF%20Files/
Adaptec%20FS4500%20SATA.pdf.
Judd, Ian, “Device Services Interface”, Online, Accredited Stan
dards Committee, X3, Information Processing Systems, Jun. 19,
1996, pp. 1-8.
“Veritas Volume Manager Storage Administrator 3.2, Administra
tor’s Guide”, Online, Veritas Software Corporation, Jul. 2001, pp.
1-184.
EMC 2-Gigabit Disk-Array Enclosure EMC Corporation (DAE2),
FC and ATA Models, Hardware Reference P/N 014003048, Rev
A02.
Serial Attached SCSI and Serial ATA Compatibility, Intel, 2002, pp.
1-8.
ESG Product Brief EMC Mar. 2003, ClARiiON with ATA, pp. 1-2.
EMC CLARiiON Backup Storage Solutions Back-up-to-Disk
Guide with LEGATO Networker Diskbackup Option (DBO), Engi
neering White Paper, Apr. 8, 2003, pp. 1-28.
EMC CLARiiON Backup Storage Solutions Backup-to-disk Guide
with Computer Associates BrightStor ARCserve Backup, Engineer
ing White Paper, Apr. 16, 2003, pp. 1-28.
EMC CLARiiON Backup Storage Solutions Backup-to-Disk: An
Overview, Engineering White Paper, Mar. 3, 2003, pp. 1-10.
EMC CLARiiON Backup Storage Solutions Backup-to-Disk Guide
with CommVault Galaxy, Engineering White Paper, Mar. 3, 2003,
pp. 1-26.
EMC CLARiiON Backup Storage Solutions Backup-to-Disk Guide
with Computer Associates BrightStor Enterprise Backup, Engineer
Infortrend EonStor A16F-R1211/S1211 FC-to-SATA RAID Sub
ing White Paper, Apr. 16, 2003, pp. 1-23.
system product information, 3 pp., available at: http://www.
IBM Technical Disclosure Bulletin vol. 38, No. 7, Jul. 1995 (New
infortrend.com/2iproduct/a16f-r(s)1211.asp.
York), “Foreground/Background Checking of Parity in a Redundant
Array of Independent Disks-5 Storage Subsystem”, pp. 455-458.
Laboratory Automation and Information Management 32, 1996,
(Elsevier Science B.V.), R E Dessey, “Computer Connections”, pp.
Synetic Inc., SyneRAID -800SA, SCSI/Fibre-toSATA RAID Sub
system product information, 3 pp., available at: http://www.synetic.
net/Synetic -Products/SyneRAID-Units/SyneRAID -800SATN
SyneRaid-800SA.html.
53-62.
“SATA Disk System and Expansiom of Unit offer 3.5 TB storage”,
Product News Network, Nov. 1, 2004 2 pp.
* cited by examiner
U.S. Patent
Apr. 10, 2007
Sheet 1 0f 19
11
US 7,203,135 B2
11
57 56
e4
56 30
57
52 58
12 91
6;
(15;
57
3O
64
67
12
67
30
64
e7
12
67
30
64
e7
12
52
67
12 129
57
52
57
12 12 9
57
52
57
12 12 9
52 57
7
64
64
252
55
58 67
59
2°
63
1°
r ______ “200
‘INFORMATION ;
92
:PROCESSING }
LAEE’ABAIQSJ
U.S. Patent
Apr. 10, 2007
a
Sheet 2 0f 19
am?
Us 7,203,135 B2
U.S. Patent
Apr. 10, 2007
Sheet 3 0f 19
US 7,203,135 B2
U.S. Patent
Apr. 10, 2007
Sheet 4 0f 19
US 7,203,135 B2
U.S. Patent
Apr. 10, 2007
US 7,203,135 B2
Sheet 5 0f 19
INFORMATION
INFORMATION
300/” PROCESSING
PROCESSING A300
APPARATUS
APPARATUS
1&0
MASTER HOUSIV
501 ~\_,’-\ CHANNEL
500~m
503-“
f 501 \
CHANNEL /
CONTROL
PORTION
CPU
/
\ CHANNEL
CONTROL
PORTION
Bag/I
CONTROL
PORTION
CACHE
-
CPU
ME/MORL TROLLER MEMORY
I 505 62
DISK
/500
_
504~~/ MEMORY TROLLER EILIE O\RY
502~~_/\ CONTROL
»501
CONTROL
PORTION
CACHE _ BAOLA
M
I
CHANNEL ,-
62 505 I
DISK
DISK
DISK
CONTROL
CONTROL/—~
PORTION \
/ PORTION
506)\
0
T3
T3
EXPANSION HOUSING
51 VS
T3
1@
/504
I
CONTROL
PORTION
’503
PORTION
/502
U.S. Patent
Apr. 10, 2007
Sheet 6 0f 19
US 7,203,135 B2
FIG.6
CPU
=
A
E CHANNEL CONTROL w 501
' PORTION
MEMORY
DATA READ PROCESS
_
DISK CONTROL
'
PORTION
502
’—\/601
II
DATA WRITE PROCESS
w
"\J 602
II
RAID CONFIGURATION TABLE ’"\/603
DATA CONTROLLER
506
U.S. Patent
Apr. 10, 2007
Sheet 7 0f 19
US 7,203,135 B2
F|G.7
502 ~
DISK
DISK
CONTROL
OONTROL
PORTION
PORTION
506 ~
~502
MASTER HOUSING
M 506
P80
P80
3
Z
701
PC
51
P80
701'
P80
1
I
701
51
i
701
701
i
g
3
M 20
701
I
3
PEG
P80
51
U.S. Patent
Apr. 10, 2007
Sheet 8 0f 19
US 7,203,135 B2
FIG.8
DISK
CONTROL
PORTION
DISK
502 ~
CONTROL
PORTION
5O 6
MA
~ 502
50 6
m
M
O
m14|
%
8w
qvRBOn
CSCTO0ENDuV RH0
EH|.i mu
D.-P C
PP. GC
m
w
\)(|\
BI|.
BI|.
5 14|
)2\.l0O2
m
6
2
U.S. Patent
Apr. 10, 2007
Sheet 10 0f 19
US 7,203,135 B2
SECTOR
NUMBER
DRIVE NUMBER
HDD#2
LBA
# LBA
LBA
#1-128
129-2
#257- 4
0
0
0
o0o
1
0
0
00o
0
1
0
'''
1101
56
,,,
38
U.S. Patent
Apr. 10, 2007
Sheet 11 0f 19
US 7,203,135 B2
FIG.12
C
WRITE PROCESS
I
WRITE DATA
N S1201
EXECUTE SEEK PROCESS
N 81202
READ DATA FROM CACHE MEMORY N $1203
I
READ DATA FROM MAGNETIC DISK N S1204
S1205
DO TWO PIECES
OF DATA COINCIDE ?
YES
INFORM OF FAILURE IN WRITING N S1206
C
J
U.S. Patent
Apr. 10, 2007
Sheet 12 0f 19
US 7,203,135 B2
F|G.13
C WHITE PROCESS D
1
WHITE DATA
N S1301
EXECUTE SEEK PROCESS
"\J S1302
IS DATA SEQUENTIAL ?
READ DATA FRoM CACHE MEMORY
READ ONE SEGMENT AT HEAD
AND ONE SEGMENT AT END FRoM
CACHE MEMORY
N 31304
N 31306
I
READ ONE SEGMENT AT HEAD
AND oNE SEGMENT AT END FRDM
READ DATA FRDM MAGNETIC DISK
f» 81307
MAGNETIC DISK
\ S1305
AI
S1308
DO TWO PIECES
OF DATA COINCIDE ’?
YES
IN FORM OF FAILURE IN WRITING N 31309
C
J
U.S. Patent
Apr. 10, 2007
Sheet 13 0f 19
US 7,203,135 B2
FIG.14
(COMPARISON PROCESS )
l
S1401
IS NUMBER OF TIMES
OF WRITING EXCESSIVE ?
WRITE DATA IN CACHE MEMORY
INTO MAGNETIC DISK
M81402
READ DATA FROM CACHE MEMORY N 31403
READ DATA FROM MAGNTIC DISK N 31404
S1405
DO TWO PIECES
OF DATA COINCIDE ?
YES
INFORM OF FAILURE IN WRITING N 81406
c
1
U.S. Patent
Apr. 10, 2007
Sheet 14 0f 19
US 7,203,135 B2
F|G.15
LBA
#1 -1 28
LBA
#129-256
1501
LBA
#257-384
U.S. Patent
Apr. 10, 2007
Sheet 15 0f 19
US 7,203,135 B2
FIG.16
SETI IN EXAMINATION
HEAD NUMBER
~SI601
S1602
HAS PREDETERMINED
TIME PASSED ?
WRITE EXAMINATION DATA
INTO CONTROL BLOCK OF
HEAD IN QUESTION
~S1603
READ DATA WRITTEN
IN CONTROL BLOCK
~S1604
S1605
DO TWO PIECES
OF DATA COINCIDE ?
INFORM OF DRIVE FAILURE ~S1610
CHANGE EXISTENCE OF
UPDATE OF HEAD IN
QUESTION INTO "o"
~S1606
I
ADD 1 To EXAMINATION
HEAD NUMBER
~S1507
"
END
N0
IS EXAMINATION
HEAD NUMBER LARGER THAN
HEAD NUMBER ?
SET I IN EXAMINATION
HEAD NUMBER
_______I
S1608
“41609
U.S. Patent
Apr. 10, 2007
Sheet 16 0f 19
US 7,203,135 B2
READ PROCESS
ACCEPT READ REQUEST
~S1701
|
ACQUIHE EXISTENCE OF
UPDATE OF TARGET LBA
~ S1702
OF TARGET DRIVE
S1703
IS THERE ANY UPDATE ?
WRITE EXAMINATION DATA
INTO CONTROL BLOCK OF
HEAD IN QUESTION
~S1704
HEAD DATA WRITTEN
IN CONTROL BLOCK
~S1705
S1706
'
00 Two PIECES
OF DATA COINCIDE ?
NO
YES
SET "0" IN EXISTENCE OF
UPDATE OF HEAD IN
QUESTION
INFORM 0F DRIVE FAILURE ~S1709
~s1707
7
EXECUTE READ PROCESS
END
M31708
U.S. Patent
Apr. 10, 2007
Sheet 17 0f 19
US 7,203,135 B2
Fl G. 1 8
Fs#1 | s#2 I s#3 I am | PJ
Fl G . 1 9
1001
ESL-"8“
IllIl\I
2
1803
'..--_---__--------------_‘
m
1803
I3,
m...
nun‘,