Cray T3E T3E system Optimization Guide
বিজ্ঞাপন
বিজ্ঞাপন
CRAY T3E C and C++ Optimization
Guide
SG–2178 3.0.1
Copyright © 1997 Cray Research, Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Research, Inc.
Portions of this product may still be in development. The existence of those portions still in development is not a commitment of actual release or support by Cray Research, Inc. Cray Research, Inc. assumes no liability for any damages resulting from attempts to use any functionality or documentation not officially released and supported. If it is released, the final form and the time of official release and start of support is at the discretion of Cray Research, Inc.
Autotasking, CF77, CRAY, Cray Ada, CraySoft, CRAY Y-MP, CRAY-1, CRInform, CRI/TurboKiva, HSX, LibSci, MPP Apprentice,
SSD, SUPERCLUSTER, UNICOS, and X-MP EA are federally registered trademarks and Because no workstation is an island, CCI,
CCMT, CF90, CFT, CFT2, CFT77, ConCurrent Maintenance Tools, COS, Cray Animation Theater, CRAY APP, CRAY C90,
CRAY C90D, Cray C++ Compiling System, CrayDoc, CRAY EL, CRAY J90, CRAY J90se, CrayLink, Cray NQS,
Cray/REELlibrarian, CRAY S-MP, CRAY SSD-T90, CRAY T90, CRAY T3D, CRAY T3E, CrayTutor, CRAY X-MP, CRAY XMS,
CRAY-2, CSIM, CVT, Delivering the power . . ., DGauss, Docview, EMDS, GigaRing, HEXAR, IOS,
ND Series Network Disk Array, Network Queuing Environment, Network Queuing Tools, OLNET, RQS, SEGLDR, SMARTE,
SUPERLINK, System Maintenance and Remote Testing Environment, Trusted UNICOS, UNICOS MAX, and UNICOS/mk are trademarks of Cray Research, Inc.
AMPEX and DST are trademarks of Ampex Corporation. DEC is a trademark of Digital Equipment Corporation. DLT is a trademark of Quantum Corporation. EXABYTE is a trademark of EXABYTE Corporation. IBM and Magstar are trademarks of
International Business Machines Corporation. STK, TimberLine, RedWood, and WolfCreek are trademarks of Storage Technology
Corporation. Silicon Graphics and the Silicon Graphics logo are registered trademarks of Silicon Graphics, Inc. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited. X/Open is a registered trademark, and the X device is a trademark, of X/Open Company Ltd.
The UNICOS operating system is derived from UNIX
®
System V. The UNICOS operating system is also based in part on the
Fourth Berkeley Software Distribution (BSD) under license from The Regents of the University of California.
Version
3.0.1
Description
August, 1997
Original printing.
Record of Revision
SG–2178 3.0.1
i
Contents
Preface
Related publications .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Ordering Cray Research publications .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Conventions .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Reader comments .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Background Information [1]
Message-passing protocols .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Parallel Virtual Machine (PVM) .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
SHMEM .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Hardware .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Memory .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Procedure 1: Moving data from memory .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Processing element .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Network and peripherals .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Disk support
Tape support .
.
Network protocols
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Memory performance information
Timing your code .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Parallel Virtual Machine (PVM) [2]
Setting the size of a message .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Allocating send buffers .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
The advantage of 32-bit data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
SG–2178 3.0.1
Page
xi
xi xi xii xiii
15
15
16
17
18
19
21
2
2
4
4
9
1
2
23
24
25
26
iii
CRAY T3E C and C++ Optimization Guide
Example 1: Transferring 32-bit data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Sending and receiving stride-1 data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 2: pvm_psend and pvm_precv
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Mixing send and receive routines .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Initializing and packing data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Working while you wait
Avoiding barriers .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using broadcast or multicast .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 3: pvm_bcast
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 4: pvm_mcast
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Minimizing synchronization time when receiving data .
.
.
.
.
.
.
.
.
.
.
.
.
Using the reduction functions .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 5: A
PvmSum example .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Gathering and scattering data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 6: Gather operation
Example 7: Scatter operation
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
SHMEM [3]
Using shmem_get and shmem_put for data transfer
Example 8: Example of a shmem_put transfer .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Optimizing existing MPI and PVM programs by using SHMEM .
.
.
.
.
.
.
.
.
.
.
Example 9: MPI version of the ring program
Optimizing by using shmem_get
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 10: shmem_get version of the ring program .
.
.
.
.
.
.
.
.
.
.
Optimizing by using shmem_put
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 11: shmem_put version of the ring program .
.
.
.
.
.
.
.
.
.
.
Passing 32-bit data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 12: 32-bit version of ring program .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
iv
Page
31
32
33
34
35
26
28
29
30
30
37
38
39
40
42
43
51
55
55
58
45
46
46
51
58
59
60
SG–2178 3.0.1
Copying strided data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 13: Passing strided data using shmem_iget
.
.
.
.
.
.
.
.
.
.
.
.
Example 14: Passing strided data using shmem_iput
.
.
.
.
.
.
.
.
.
.
.
.
Gathering and scattering data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 15: shmem_ixput version of a reordered scatter .
.
.
.
.
.
.
.
.
.
Broadcasting data to multiple PEs .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 16: One-to-all broadcasting .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Merging arrays .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 17: Merging arrays .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Reading and updating in one operation .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 18: Remote fetch and increment .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using reduction functions .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 19: Minimum value reduction routine .
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 20: Summation using a reduction routine .
.
.
.
.
.
.
.
.
.
.
.
.
Single-PE Optimization [4]
Unrolling loops .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Software pipelining .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Optimizing a program with software pipelining
Selecting the level of pipelining .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using the concurrent and ivdep directives .
.
.
.
.
.
.
.
.
.
.
.
.
.
Identifying loops for pipelining .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
How pipelining works .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Optimizing for cache .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Rearranging array dimensions for cache reuse .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 21: Unoptimized code .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Optimizing for stream buffers
Splitting loops .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Contents
Page
90
90
93
94
85
86
87
90
83
83
84
84
85
70
70
72
73
75
62
62
63
67
67
76
76
77
79
SG–2178 3.0.1
v
CRAY T3E C and C++ Optimization Guide
Example 22: Original loop
Example 23: Splitting loops
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 24: Stripmining .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 25: Splitting loops across if statements .
.
.
.
.
.
.
.
.
.
.
.
Example 26: Splitting individual statements .
.
.
.
.
.
.
.
.
.
.
.
.
.
Changed behavior from loop splitting
Maximizing inner loop trip count
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 27: Rearranging array dimensions
Minimizing stream count .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 28: Minimizing streams .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 29: Reduced streams version .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Grouping statements that use the same streams .
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 30: Original code .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 31: Grouping statements within the loop .
.
.
.
.
.
.
.
.
.
.
.
Example 32: Loop that will be split into four .
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 33: Loop that will be split into two .
.
.
.
.
.
.
.
.
.
.
.
.
.
Enabling and disabling stream buffers .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Optimizing division operations
Example 34: Original code .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 35: Modified code .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Vectorization .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using the ivdep directive .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Bypassing cache .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Input/Output [5]
Strategies for I/O .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using a single, shared file .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using multiple files and multiple PEs .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
vi
Page
97
98
98
99
99
95
95
95
96
96
99
100
100
100
100
101
101
102
103
104
105
108
108
111
111
112
114
SG–2178 3.0.1
Using a single PE .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Unformatted I/O .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Sequential, unformatted requests .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Formatted I/O .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Reduce formatted I/O .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Make large I/O requests .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
FFIO .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Memory-resident data files .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Distributed I/O .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using the cache layer .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using library buffers .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Random access
Striping .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Hardware Access [6]
Using E registers .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Basics .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Usage rules .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example of a single-word put .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 36: Inline C function to put an int value to a remote processor’s memory .
Using barrier and eureka synchronization units .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Example 37: Using BESUs in a C function .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Using distributed I/O .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Global application team termination .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Glossary [7]
Index
Contents
Page
129
129
131
131
132
133
135
136
137
137
121
121
123
125
126
126
126
116
117
117
120
120
120
139
151
SG–2178 3.0.1
vii
CRAY T3E C and C++ Optimization Guide
Figures
Figure 1.
Data transfer comparison
Figure 2.
Position of E registers .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 3.
Flow of data on a CRAY T3E node
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 4.
Data flow on the EV5 microprocessor .
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 5.
First value reaches the microprocessor .
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 6.
Ninth value reaches the microprocessor .
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 7.
Output stream .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 8.
An external GigaRing network .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 9.
Fan-out method used by broadcasting routines .
.
.
.
.
.
.
.
.
.
.
.
Figure 10.
A
PvmMax reduction .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 11.
The gather/scatter process .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 12.
SHMEM_PUT64 data transfer .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 13.
Identification of neighbors in the ring program.
.
.
.
.
.
.
.
.
.
.
.
Figure 14.
shmem_iget and shmem_iput transfers .
.
.
.
.
.
.
.
.
.
.
.
.
Figure 15.
Reordering elements during a scatter operation .
.
.
.
.
.
.
.
.
.
.
Figure 16.
shmem_broadcast operation
Figure 17.
Results of a shmem_fcollect
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 18.
Results of a shmem_double_min_to_all
Figure 19.
Overlapped iterations .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 20.
Pipelining a loop with multiplications .
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 21.
Before and after array a has been optimized .
.
.
.
.
.
.
.
.
.
.
.
Figure 22.
Multiple PEs using a single file
Figure 23.
Multiple PEs and multiple files
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 24.
I/O to and from a single PE .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Figure 25.
Data paths between disk and an array .
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
72
75
79
88
89
92
112
115
116
118
34
39
41
10
12
14
16
7
8
3
5
48
54
66
69
viii SG–2178 3.0.1
Figure 26.
Data layout for distributed I/O .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Tables
Table 1.
Latencies and bandwidths for data cache access .
.
.
.
.
.
.
.
.
.
.
.
Table 2.
Latencies and bandwidths for access that does not hit cache .
.
.
.
.
.
.
.
Table 3.
Functional unit .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Contents
Page
124
20
20
90
SG–2178 3.0.1
ix
Preface
This publication documents optimization options for the Cray C and C++ compilers running on CRAY T3E systems.
Related publications
The following documents contain additional information that may be helpful:
• Cray C/C++ Reference Manual, publication SR–2179
• Cray C/C++ Ready Reference, publication SQ–2180
• UNICOS/mk System Libraries Reference Manual, publication SR–2680
• Introducing the MPP Apprentice Tool, publication IN–2511
• Introducing the Cray TotalView Debugger, publication IN–2502
• Message Passing Toolkit: PVM Programmer’s Manual, publication SR–2196
• Message Passing Toolkit: MPI Programmer’s Manual, publication SR–2197
• CRAY T3E and CRAY T3D Programming Environment Differences, publication
SR–2199
• UNICOS/mk System Calls Reference Manual, publication SR–2612
Ordering Cray Research publications
The User Publications Catalog, publication CP–0099, describes the availability and content of all Cray Research hardware and software documents that are available to customers. Cray Research customers who subscribe to the Cray
Inform (CRInform) program can access this information on the CRInform system.
To order a document, either call the Distribution Center in Mendota Heights,
Minnesota, at +1–612–683–5907, or send a facsimile of your request to fax number +1–612–452–0141. Cray Research employees may send electronic mail to orderdsk
(UNIX system users).
Customers who subscribe to the CRInform program can order software release packages electronically by using the
Order Cray Software option.
SG–2178 3.0.1
xi
CRAY T3E C and C++ Optimization Guide
Customers outside of the United States and Canada should contact their local service organization for ordering and documentation information.
Conventions
xii
The following conventions are used throughout this document:
Convention command manpage
variable
(x)
user input
Meaning
5
7
4
4P
7D
8
Denotes a command, library routine or function, system call, part of an application program, program output, or anything else that might appear on your screen.
Man page section identifiers appear in parentheses after man page names. The following list describes the identifiers:
2
3
1
1B
User commands
User commands ported from BSD
System calls
Library routines, macros, and opdefs
Devices (special files)
Protocols
File formats
Miscellaneous topics
DWB-related information
Administrator commands
Some internal routines (for example, the ddcnt
() routine) do not have man pages associated with them.
Italic typeface denotes variable entries and words or concepts being defined.
This bold, fixed-space font denotes literal items that the user enters in interactive sessions.
Output is shown in nonbold, fixed-space font.
SG–2178 3.0.1
Preface
[ ]
(glossary, page number)
Brackets enclose optional portions of a command or directive line.
References the glossary for a definition of the preceding term.
The following machine naming conventions are used throughout this document:
Term
Cray PVP systems
Cray MPP systems
All Cray Research systems
Definition
All configurations of Cray parallel vector processing (PVP) systems.
All configurations of the CRAY T3E series.
All configurations of Cray PVP and Cray MPP systems that support this release
Reader comments
If you have comments about the technical accuracy, content, or organization of this document, please tell us. You can contact us in any of the following ways:
• Send us electronic mail at the following address: [email protected]
• Contact your customer service representative and ask that an SPR or PV be filed. If filing an SPR, use
PUBLICATIONS for the group name,
PUBS for the command, and
NO-LICENSE for the release name.
• Call our Software Publications Group in Eagan, Minnesota, through the
Customer Service Call Center, using either of the following numbers:
1–800–950–2729 (toll free from the United States and Canada)
+1–612–683–5600
• Send a facsimile of your comments to the attention of “Software Publications
Group” in Eagan, Minnesota, at fax number +1–612–683–5599.
We value your comments and will respond to them promptly.
SG–2178 3.0.1
xiii
Background Information [1]
SG–2178 3.0.1
Welcome to CRAY T3E optimization. This chapter gives an overview of the optimization guide and background information on some of its major subjects.
If you want to start optimizing your program right away, just select one of the following topics. You can always come back later.
• The Parallel Virtual Machine (PVM) is a portable message-passing protocol for programming the CRAY T3E system and other parallel systems. See
Chapter 2, page 23.
• SHMEM stands for shared memory protocol. It is not as portable as PVM or the Message Passing Interface (MPI) but has potentially better performance.
See Chapter 3, page 45.
• Single-PE optimizations concern getting the best performance out of each processing element (PE). See Chapter 4, page 83.
• Input/output (I/O) optimizations help you move data between external devices (such as disk) and memory. See Chapter 5, page 111.
• Access to the hardware lets you perform low-level operations that result in additional performance in some situations. See Chapter 6, page 129.
This publication contains a glossary with definitions of terms that might be unfamiliar to you. If you are reading this document online, you can link to the glossary as you encounter a term. Here is an example of a link that will point you to the glossary: PE (glossary, page 146). If you are reading a printed version of the document, you will see a page number in place of the hyperlink.
For background information, see the following topics in this chapter:
• An introduction to two message-passing protocols (see Section 1.1, page 2), including the following subtopics:
– PVM, see Section 1.1.1, page 2.
– SHMEM, see Section 1.1.2, page 2.
• A hardware overview (see Section 1.2, page 4), including the following subtopics:
– Memory characteristics, see Section 1.2.1, page 4.
– The processing element, or PE, see Section 1.2.2, page 15.
1
CRAY T3E C and C++ Optimization Guide
– The network and peripherals, see Section 1.2.3, page 15.
– Memory performance information, see Section 1.2.4, page 19.
• Timing your code, see Section 1.3, page 21.
1.1 Message-passing protocols
When you are optimizing a program on a CRAY T3E system, you may be faced with a number of decisions. One of the first will be which, if any, of the message-passing protocols you should use.
If you want to run the program on more than one vendor’s MPP system, portability is a major concern, and you may want to choose PVM. (The Message
Passing Interface (MPI) is also available and is widely portable.) If your only concern is the performance of the program, you may want to include shared memory access routines, known collectively as SHMEM.
1.1.1 Parallel Virtual Machine (PVM)
The PVM programming style offers a widely used, standardized method of programming a CRAY T3E system. PVM does not offer the performance of
SHMEM, but it is more portable. PVM runs on both Cray massively parallel processing (MPP) systems and Cray parallel vector processing (PVP) systems, as well as on other parallel architectures. It is a message-passing system
(glossary, page 144), meaning it exchanges explicit messages with other PEs.
The messages often contain data, such as array elements.
PVM relieves the programmer of most synchronization concerns. By using explicit calls to send and receive routines, PVM handles its own synchronization in most cases.
For more introductory information on PVM, see the pvm_intro
(1) man page.
1.1.2 SHMEM
2
SHMEM is a set of functions that pass data in a variety of ways, provide synchronization, and perform reductions (glossary, page 146). SHMEM functions are implemented on Cray MPP systems and Cray PVP systems but not on any other company’s computers.
What SHMEM lacks in portability, it makes up for in performance. SHMEM is the fastest of the Cray MPP programming styles.
SG–2178 3.0.1
SG–2178 3.0.1
Background Information [1]
The reason for the speed is SHMEM’s close-to-the-hardware approach. This demands more from the programmer in areas such as synchronization (glossary, page 148), which is provided automatically with some of the other programming styles. The following figure shows how the SHMEM functions enhance performance by dispensing with some of the processes followed by PVM.
On the sending PE
PVM
{
rc = pvm_initsend( ...
rc = pvm_pkint( ...
rc = pvm_send( ...
On the receiving PE
rc = pvm_recv( ...
rc = pvm_upkint( ...
SHMEM
{
shmem_put( ...
shmem_barrier_all shmem_barrier_all
a10510
Figure 1. Data transfer comparison
In this example of typical data transfers, PVM requires five steps on the two
PEs involved in the transfer: initialize a send buffer, pack the data, send the data, receive the data, and unpack the data. SHMEM requires only one step: send the data. However, one or more synchronization routines are almost always necessary when using SHMEM. You usually must ensure that the receiving PE does not try to use the data before it arrives.
SHMEM does a direct memory-to-memory copy, which is the fastest way to move data on a CRAY T3E system. Adding SHMEM functions to your code, or replacing the statements of another programming style with SHMEM functions, will almost always enhance the performance of your program. Replacing only the major data transfers with shmem_put or shmem_get can often give you a major speedup with minimal effort. For more information on the functionality available in SHMEM, see the intro_shmem
(3) man page.
3
CRAY T3E C and C++ Optimization Guide
1.2 Hardware
The CRAY T3E hardware performs at a rate of two to three times that of
CRAY T3D systems. The following sections contain an overview of the memory system, a brief description of the microprocessor, a look at the network and the system’s peripherals, and statistics detailing where the increased performance comes from.
1.2.1 Memory
A memory operation from a PE takes one of two forms:
• A read from, or write to, the PE’s own memory (called local memory). Each
PE has between 64 Mbytes (8 64-bit Mwords) and 2 Gbytes (256 64-bit
Mwords) of memory local to the processor.
• A read from, or write to, remote memory (the memory local to some other
PE).
Note:
A word in this document is assumed to be 64-bits in length, unless otherwise stated.
Operations between the memory of two PEs make use of E registers. E registers are special hardware components that let one PE read from and write to the memory of another PE. E registers are largely reserved for internal use, but users can customize access to them (see Section 6.1, page 129).
E registers are positioned between the PE and the network, as illustrated in the following figure. They are memory-mapped registers, which means they reside in noncached memory and have an address associated with them.
4 SG–2178 3.0.1
Background Information [1]
PE
SG–2178 3.0.1
E registers
Outgoing
PUT data
Returning
GET data
Memory and network interface a10012
Figure 2. Position of E registers
Operations within a PE, between local memory and the microprocessor, are always faster than operations to or from remote memory. Data read from local memory is accessed through two levels of cache: a 96-Kbyte secondary cache
(glossary, page 147) and a high-speed, 8-Kbyte data cache (glossary, page 141).
Data written to local memory passes through a 6-entry write buffer (glossary, page 150) and secondary cache.
Cache coherence (glossary, page 140), which was a user concern on the
CRAY T3D system, is performed automatically on the CRAY T3E system.
Cache is high-speed memory that helps move data quickly between local memory and the EV5 microprocessor registers. It is still an important part of
MPP programming. The cache_align directive aligns each specified variable on a cache line boundary. This is useful for frequently referenced variables and for passing arrays in SHMEM (see Section 3.3, page 59). The cache_align directive can be used with all of the programming styles described in this guide.
Data cache is a direct-mapped cache (glossary, page 141), meaning each local memory location is mapped to one data cache location. When an array, for example, is larger than data cache, a location in data cache can have more than
5
CRAY T3E C and C++ Optimization Guide one of the array addresses mapped to it. Each location is a single, 4-word
(32-byte) line (glossary, page 144).
Secondary cache is three-way set associative, and lines are 8 (64-bit) words long, for a total of 64 bytes. In a three-way, set-associative cache (glossary, page 147), each memory location is associated with three lines in secondary cache. Which of the three lines to which the data is added is chosen at random. Any line can be selected.
For an example of how data cache and secondary cache work, see Procedure 1, page 9, which describes data movement between local memory and the microprocessor. For an illustration of the components of a PE, see Figure 3. The abbreviations on the figure have the following meanings. Many of these terms are also used in Chapter 4, page 83.
EV5
E0, E1
FA, FM
WB
MAF
ICACHE
DCACHE
SCACHE
SB
The RISC microprocessor
Integer functional units
Floating-point functional units
Write buffer
Missed address file
Instruction cache (not relevant to this discussion)
Data cache
Secondary cache
Stream buffer
6 SG–2178 3.0.1
DRAM
Background Information [1]
Local memory
EV5
E0
ICACHE
E1
WB
FA FM
DCACHE
MAF
SCACHE
SG–2178 3.0.1
SB
SB SB SB SB SB
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM a10015
Figure 3. Flow of data on a CRAY T3E node
The size of a local memory page (marked as DRAM in the preceding figure) depends on the amount of memory in your machine. A memory size of 128
7
CRAY T3E C and C++ Optimization Guide
8
Mbytes, for example, has a page size of 16 Kbytes. The following figure shows more detail from the microprocessor part of the data flow.
Write data
Write buffer
Write data
Write data
Read data
Execution units
Read data
Data caches
Control
Instructions
Instruction circuitry
To support circuitry
From support circuitry
Read data
Write data
Missed address file
Control
Secondary cache
Control
Instructions
Read data or instructions
Write data
To support circuitry
From support circuitry
To support circuitry a10014
Figure 4. Data flow on the EV5 microprocessor
Each PE has four functional units: two for floating-point operations and two for integer operations. It can handle six concurrent input and output data streams
(glossary, page 148).
For the following loop, a PE will create streams between memory and the functional units for all of the input operands ( b[i]
, c[i]
, and so on) and one stream between the functional units, through the write buffer, through secondary cache, and back to memory for the output operand ( a[i]
):
SG–2178 3.0.1
SG–2178 3.0.1
Background Information [1] for (i=0; i<1000; i++) { a[i] = b[i] + c[i] + d[i] + e[i] + f[i];
}
By default, as soon as the PE detects two consecutive secondary cache-line misses, it begins to preload subsequent, consecutive locations and form a stream. Data streaming is a major optimization on the CRAY T3E system. For more information on creating streams, see Section 4.4.6, page 101.
The following procedure describes the process of moving data between memory and the microprocessor. It refers only to the key hardware elements:
• EV5 registers
• Write buffer
• Data cache
• Secondary cache
• Stream buffer
• Local memory
The example assumes the following loop: for (i=0; i<n; ++i) { a[i] = b[i] * n;
}
Procedure 1: Moving data from memory
1. An EV5 register requests the value of b[0] from data cache.
2. Data cache does not have b[0]
. It requests b[0] from secondary cache.
3. Secondary cache does not have b[0]
. This is the first secondary cache miss.
It retrieves a line (8 64-bit words for secondary cache) from local memory.
4. Data cache receives a line (4 64-bit words for data cache) from secondary cache.
5. The register receives b[0] from data cache. The state of the data at this point is as illustrated in the following figure.
9
CRAY T3E C and C++ Optimization Guide
10
EV5 b [0]
Data cache b [0-3] b [0-7]
Secondary cache b [0-N]
Local memory
Figure 5. First value reaches the microprocessor a10511
6. When other registers need b[1] through b[3]
, they find them in data cache.
7. When a register needs b[4]
, data cache does not have it.
SG–2178 3.0.1
SG–2178 3.0.1
Background Information [1]
8. Data cache requests b[4] through b[7] from secondary cache, which has them and passes them on.
9. Data cache passes b[4] through b[7] on to the appropriate registers as it gets requests for them. When the microprocessor finishes with them, it requests b[8] from data cache.
10. Data cache requests a new line of data elements from secondary cache, which does not have them. This is the second secondary cache miss, and it is the signal to the system to begin streaming data.
11. Secondary cache requests another 8-word line from local memory and puts it into another of its three-line buckets. It may end up in any of the three lines, since the selection process is random.
12. A 4-word line is passed from secondary cache to data cache, and a single value is moved to a register. When the value of b[8] gets to the register, the situation is as illustrated in the following figure.
11
CRAY T3E C and C++ Optimization Guide
12
EV5 b [8]
Data cache b [0-3] b [4-7] b [8-11] b [0-7] b [8-15]
Secondary cache
Stream buffer b [16-23] b [0-N]
Local memory a10512
Figure 6. Ninth value reaches the microprocessor
SG–2178 3.0.1
SG–2178 3.0.1
Background Information [1]
13. Because streaming has begun, data is now prefetched. A stream buffer anticipates the microprocessor’s continuing need for consecutive data, and begins retrieving b[16] through b[23] from memory before it is requested. As long as the microprocessor continues to request consecutive elements of b
, the data will be ready with a minimum of delay.
14. The process of streaming data between local memory and the registers in the microprocessor continues until the loop is complete.
These steps describe the input stream only. The values of the a array pass through the write buffer and secondary cache, as illustrated in the following figure, on their way back to local memory. Values of a are written to local memory only when a line in secondary cache is dislodged by a write to the same line, or when values of a are requested by another PE.
13
CRAY T3E C and C++ Optimization Guide
EV5 b [8] a [8]
14
Data cache b [0-3] b [4-7] b [8-12]
Secondary cache a [0-7] b [0-7] b [8-15] b [16-23]
Stream buffer
Local memory b [0-N] a [0. . .]
Figure 7. Output stream a [8] a [0-3] a [4-7]
Write buffer a10513
SG–2178 3.0.1
Background Information [1]
1.2.2 Processing element
The CRAY T3E processing element contains an EV5 RISC microprocessor manufactured by Digital Equipment Corporation (DEC). It has the following characteristics:
• Either a 3.3 nanosecond (glossary, page 145), 300 megahertz clock period (CP) or a 2.2 ns (nanosecond), 450 megahertz CP, compared to the 6.6 ns clock in the CRAY T3D system.
• Thirty-two integer and thirty-two floating-point registers.
• Separate addition and multiplication functional units, including pipelines with a 4-CP (13.3 ns for 300 megahertz chip and 8.9 ns for 450 megahertz chip) execution time. The CRAY T3D system had a single functional unit for multiplication and addition, and the pipeline required 6 CPs (39.7 ns).
• An interface to the network for each PE.
1.2.3 Network and peripherals
The network that operates between the CRAY T3E system and external systems is based on the GigaRing technology (glossary, page 143), which is the Standard
Coherent Interface (SCI) with major Cray Research extensions. Physically, the
GigaRing network is a double ring that passes messages between nodes. Figure
8, page 16, illustrates an external network that includes a CRAY T3E system.
(Early CRAY T3E systems do not support multiple hosts on a single GigaRing channel. Cray PVP systems, such as CRAY T90 systems, connect to CRAY T3E systems through standard network interfaces, such as HIPPI.)
SG–2178 3.0.1
15
CRAY T3E C and C++ Optimization Guide
1.2.3.1 Disk support
16
a10013
Figure 8. An external GigaRing network
The GigaRing channel supports the peripherals and networks described in the following sections.
CRAY T3E systems support the following disk drives. On all drives, users can define logical disks. Disk striping (glossary, page 142) and disk mirroring
(glossary, page 142) are also supported.
• MPN-1 SCSI disks:
– SCSI DD-314 disk drives
– SCSI DD-318 disk drives
• FCN-1 Fiber Channel disks:
– DD-308 disk drives.
SG–2178 3.0.1
Background Information [1]
1.2.3.2 Tape support
– DD-308 disk drives, RAID-3.
– DD-308 disk drives, RAID-5. (Support is not yet available.)
• IPI-2 disks:
– DD-60 disk drives and DA-60 disk arrays
– DD-62 disk drives and DA-62 disk arrays
– DD-301 disk drives and DA-301 disk arrays
– DD-302 disk drives and DA-302 disk arrays
• 100 Mbyte/s HIPPI disks. (Support is not yet available.)
– ND-12 network disk array
– ND-14 network disk array
– ND-30 network disk array
– ND-40 network disk array
The following tape hardware is supported on CRAY T3E systems.
• The following models of SCSI tape drives:
– SCSI STK 4781/4480 (18 track).
– SCSI STK 4791/4490 (36 track).
– SCSI DAT HP C1533-A.
– SCSI STK 4890 (Twin Peaks).
– SCSI STK 9490 (TimberLine).
– SCSI STK SD-3 (RedWood).
– SCSI IBM 3590 (Magstar).
– IBM 3490E.
– EXABYTE 8505.
– DLT 4000.
SG–2178 3.0.1
17
CRAY T3E C and C++ Optimization Guide
– DLT 7000. (Support is not yet available.)
– AMPEX DST310. (Support is not yet available.)
• Block multiplexer tape drives:
– IBM 3480
– IBM 3490
– STK 4480
– STK 4490
– 3420-compatible, 9-track reel tape drives with a 256-Kbyte block limit
• ESN-1 ESCON tapes:
– IBM 3490E.
– STK 9490 (TimberlLine).
– STK SD-3 (RedWood).
– IBM 3590 (Magstar). (Support is not yet available.)
• Autoloaders.
– STK 4400
– STK WolfCreek
– IBM 3494
– STK 9710 (Panther)
– IBM 3495
1.2.3.3 Network protocols
The following networking protocols are supported over the GigaRing channel:
• MPN-1 Ethernet (ETN-10).
• MPN-1 Ethernet (ETN-11).
• MPN-1 FDDI (FDI 10).
• HPN-1 100 Mbyte/s HIPPI network, 2 HIPPIs per node.
18 SG–2178 3.0.1
Background Information [1]
• HPN-2 200 Mbyte/s HIPPI network, 1 HIPPI per node.
• MPN-1 ATM OC3.
• ATM OC12. (Support is not yet available.)
Network protocols, such as TCP/IP, that are supported on other Cray Research systems are also supported on CRAY T3E systems.
1.2.4 Memory performance information
This section presents some of the performance specifications of the CRAY T3E memory system. The times presented are theoretically the optimal times. In most cases, you cannot achieve these times in practice for one reason or another. You may decide at some point in the optimization process that the time required to approach the optimal times is not worth spending. In the tables, ns is nanoseconds, CP is clock period (glossary, page 141), and Mbyte/s is megabytes per second.
• Table 1 shows the time required for a CRAY T3E PE, with the microprocessor running at 300 megahertz, to load data from and store data to data cache (glossary, page 141). It compares those figures with the
CRAY T3D system. For information on data cache, see Section 1.2.1, page 4.
• Table 2 shows performance when loads and stores miss in cache and must go to local memory. The statistics reflect a microprocessor running at 300 megahertz. In practice, the peak cacheable load and store bandwidths listed in this table are not likely to be achieved, because in about 50% of the cases, old data must be removed from cache before new data can be brought in.
For more realistic peak numbers, reduce the bandwidths by approximately one-third.
SG–2178 3.0.1
19
CRAY T3E C and C++ Optimization Guide
Data cache load
Table 1. Latencies and bandwidths for data cache access
CRAY T3D latency
20 ns (3 CP per load)
N/A Secondary cache load
Data or secondary cache store
N/A
CRAY T3D bandwidth
1200 Mbyte/s (1 word per CP)
N/A
N/A
CRAY T3E latency
6.67 ns (2 CP per load)
26.67 ns (8 CP per load)
N/A
CRAY T3E bandwidth
4800 Mbyte/s (2 words per CP)
4800 Mbyte/s (2 words per CP)
2400 Mbyte/s (1 word per CP)
Table 2. Latencies and bandwidths for access that does not hit cache
Cacheable load
(stream/ read ahead buffer hit)
Cacheable store
(stream/ read ahead buffer hit)
Infinite vector
A=B+C through cacheable loads and stores (stride
1, stream hit)
Infinite vector
Y-X*s+Y (SAXPY) through cacheable loads and stores
(stride 1, stream hit)
Cacheable load
(local memory page hit)
CRAY T3D latency
86 ns (13 CP per load)
147 ns
CRAY T3D bandwidth
320 Mbyte/s (.26
words per CP)
200 Mbyte/s (.16
words per CP)
283 ns
CRAY T3E latency
80 ns (24 CP per load)
CRAY T3E bandwidth
960 Mbyte/s (.40
words per CP)
1,200 Mbyte/s (.50
words per CP)
720 Mbyte/s (.30
words per CP)
900 Mbyte/s (.37
words per CP)
630 Mbyte/s (.30
words per CP)
20 SG–2178 3.0.1
Background Information [1]
Cacheable store
(local memory page hit)
Cacheable load
(local memory page miss)
Cacheable store
(local memory page miss)
CRAY T3D latency
CRAY T3D bandwidth
533 Mbytes/s (.50
words per CP)
CRAY T3E latency
CRAY T3E bandwidth
355 Mbyte/s (.15
words per CP)
247 ns 123 Mbytes/s (.10
words per CP)
417 ns
209 Mbyte/s (.17
words per CP)
430 Mbyte/s (.20
words per CP)
280 Mbyte/s (.10
words per CP)
1.3 Timing your code
!
You can use a variety of methods to time your code. The following are the most popular:
• The Cray MPP Apprentice tool is good for timing a complete program or a subroutine within a program. It will also give you information on how well you have parallelized your program and where you can make further improvement. Because it estimates its own overhead and subtracts that figure from its timings, MPP Apprentice can be counted on to be accurate to within at least 5% of the numbers it generates.
• The
_rtc intrinsic function returns values from the real-time clock. It is good for timing blocks of code that are part of a subprogram.
• The performance analysis tool (PAT) runs only on the CRAY T3E system. It gives you information on load balancing across multiple PEs, generates and lets you view trace files, displays hardware performance counter information, estimates the amount of time spent in routines, and times individual calls to routines. See the pat
(1) man page for more information.
Caution:
Do not use both the Cray MPP Apprentice tool and
_rtc on the same code at the same time. MPP Apprentice introduces a significant amount of overhead that will be included in the
_rtc numbers but not in the numbers that MPP Apprentice itself reports. Distinguishing between the time used by your code and the overhead is difficult.
If your CRAY T3E system has PEs running at different clock rates (for instance, some at 300 megahertz and others at 450 megahertz), you will have to know
SG–2178 3.0.1
21
CRAY T3E C and C++ Optimization Guide what each PE’s clock rate is in order to time the program correctly. For information on how your mixed-speed PEs are configured, see your system administrator. The grmview
(1) command shows you at what speed each
physical PE runs, but, when you execute your program, physical PEs numbers are mapped to logical PE numbers, which are different.
22 SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2]
SG–2178 3.0.1
The Parallel Virtual Machine (PVM) message-passing library passes messages between PEs to distribute data and to perform other functions necessary for running programs. (The network version of PVM, which enables message passing between computer systems, is not described in this publication.) For background information on PVM, see Section 1.1.1, page 2.
The differences between PVM on a CRAY T3D system and PVM on a
CRAY T3E system are few. The major difference is that the channels feature is not implemented on the CRAY T3E system. But optimizations that worked on the CRAY T3D system should still work on the CRAY T3E system.
This chapter describes the following methods of speeding up your PVM program:
• Saving extra transfers by setting the size of a message properly (see Section
2.1, page 24).
• Allocating the most efficient send buffers, depending on the nature of your message (see Section 2.2, page 25).
• Realizing the performance advantage of 32-bit data (see Section 2.3, page 26).
• Using functions that are optimized for sending and receiving stride-1 data
(see Section 2.4, page 28).
• Making quick improvements by mixing optimized send and receive functions (see Section 2.5, page 30).
• Avoiding performance pitfalls when initializing and packing data (see
Section 2.6, page 30).
• Accomplishing work while you wait for messages (see Section 2.7, page 31).
• Minimizing wait time by avoiding barriers (see Section 2.8, page 32).
• Using broadcast rather than multicast when sending data to multiple PEs
(see Section 2.9, page 33).
• Minimizing synchronization time and maximizing work time when receiving data (see Section 2.10, page 37).
• Using the reduction functions to execute an operation on multiple PEs (see
Section 2.11, page 38).
23
CRAY T3E C and C++ Optimization Guide
• Distributing data from one PE to multiple PEs and gathering data from multiple PEs to a single PE (see Section 2.12, page 40).
2.1 Setting the size of a message
Setting the size of a message properly can save you extra transfers and, consequently, message-passing overhead. You can control the size of a message by setting the
PVM_DATA_MAX environment variable. The default size for the first message sent is 4,096 bytes, or 512 64-bit words, which should be large enough for most messages. If the data in a message is larger than the value of
PVM_DATA_MAX
, however, the data will be divided up into parts, and the parts will be sent in separate messages until all of it has been delivered.
To find the current value of
PVM_DATA_MAX within your program, use the pvm_getopt
(3) function, as follows. The variable max will hold the maximum message size value, in bytes.
max = pvm_getopt(PVM_DATA_MAX);
You cannot, however, change the value within the program by using the pvm_setopt
(3) function. You must reset the value outside of your program, as follows. Specify the new value for
PVM_DATA_MAX in bytes.
% setenv PVM_DATA_MAX 8192
% ./a.out
This example changes the value of the maximum message size to 8,192 bytes (or
1,024 64-bit words) for the entire program. The second line executes the program.
Increasing the size of
PVM_DATA_MAX is not always the best solution. If you have one or two large transfers in your program, but a number of smaller transfers, you may not want to increase the size of all messages. Adjusting the size of
PVM_DATA_MAX may not help your overall performance. It takes away from the memory available to the application, and a large message is not always transferred quickly, especially when it is broadcast to multiple PEs.
Breaking the large messages up into smaller messages may be faster in some cases. Whether this proves to be faster in your program depends upon the application. You may have to time the program to find out. For information on timing your code, see Section 1.3, page 21.
PVM does not handle large amounts of data in the same way as small amounts.
For large transfers (greater than the value of
PVM_DATA_MAX
), the message
24 SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2] contains the first chunk of data and the address of the data block on the sending PE. After the receiving PE unpacks, it uses remote loads to get the remainder of the data.
Often, remote stores used for short messages can occur at the same time as computation on the receiving PE. But with large messages, remote loads require the receiving PE to wait until the loads complete. If the same data is being sent to several PEs, those PEs may all try to do remote loads at the same time, creating a slowdown as they share the limited memory bandwidth.
2.2 Allocating send buffers
The pvm_initsend
(3) function lets you choose what PVM will do with the data it sends. Each of the following three choices can be used to advantage in certain circumstances:
•
PvmDataRaw
•
PvmDataInPlace
•
PvmDataDefault
Assuming your application is running only on the CRAY T3E system, the fastest of the three choices is usually
PvmDataInPlace
, as specified in the following example: bufid = pvm_initsend(PvmDataInPlace);
The
PvmDataInPlace specification has the following advantages and disadvantages:
• It does not copy the data into a send buffer, which is the primary reason for its speed, unless the data streams feature is turned on. If data streams are turned on,
PvmDataInPlace is the same as
PvmDataRaw
. For more information on data streams, see Section 4.4.6, page 101.
• It requires you to wait until the transfer is complete before accessing the data, which can slow the program down at times.
• You must either provide your own synchronization or send a short message from the receiving PE to let the sending PE know the transfer is complete.
• It is optimized for contiguous (stride-1) data. You lose any performance benefit if your data is not contiguous.
SG–2178 3.0.1
25
CRAY T3E C and C++ Optimization Guide
Although it is not always the fastest, the
PvmDataRaw specification is often considered the most useful of the three for the following reasons:
• It does not convert the data into another format, thereby saving on encoding costs.
• It ensures that the data is copied into send buffers, meaning the original data can be reused (for example, changed) immediately.
If you are sending integer data and the data does not need more than 32 bits of accuracy, you could see a performance benefit using
PvmDataDefault
.
Because this form of packing copies only the low-order 32 bits of integer data, you can get twice as much data into the same block packed using
PvmDataRaw
.
This can offer some performance benefit with 32-bit data.
Use of the
PvmDataRaw method is recommended for most transfers, but, as is often the case, which method is best depends on your application.
2.3 The advantage of 32-bit data
Passing 32-bit data can be faster than sending each data item in its own 64-bit word. If, however, your data is not aligned on a 64-bit word and you are using
PvmDataInPlace for packing, your code will probably slow down.
In the following example, data is sent in the float format (32 bits on
CRAY T3E systems):
9.
10.
11.
12.
13.
14.
15.
5.
6.
7.
8.
1.
2.
3.
4.
Example 1: Transferring 32-bit data
/* 32-bit data transfer */
#include <stdio.h>
#include <pvm3.h>
#define N 1000; main()
{ int me; int sender=0, receiver=1, mtag=2; int istat, pkstat, sstat, rstat, upkstat, mytid, i; float d_send[1000], d_recv[1000];
/* Get PE information */ mytid = pvm_mytid(); me = pvm_get_PE(mytid);
26 SG–2178 3.0.1
SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2]
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
16.
17.
18.
19,
20.
21.
22.
23.
34.
35.
36.
37.
38.
39.
40.
}
/* Initialize data */ for (i=0; i <N; i++) { d_send[i] = i * 1.0;
}
/* Send data from PE 0 */ if (me == sender) { istat = pvm_initsend(PvmDataRaw); pkstat = pvm_pkfloat(d_send, N, 1);
} sstat = pvm_send(receiver, mtag);
/* End of if */
/* Receive data */ else { if (me == receiver) { rstat = pvm_recv(sender, mtag); upkstat = pvm_upkfloat(d_recv, N, 1);
}
}
/* Print results */ printf("Receiver = %d\n", me); for (i=0; i !=8; i++) printf("D_recv[%d] = %f\n", i, d_recv[i]);
/* End of else */
The program in this example produces the following output:
Receiver=1 d_recv[0] = 0.
d_recv[1] = 1.
d_recv[2] = 2.
d_recv[3] = 3.
d_recv[4] = 4.
d_recv[5] = 5.
d_recv[6] = 6.
d_recv[7] = 7.
To move on to the next optimization topic, go to Section 2.4, page 28. For a brief description of the above program, continue with this section.
Line 3 references the PVM header file. See your system administrator for its actual location on your system. You can specify the location of the header file with the
-I option on the cc
(1) or
CC
(1) command line.
27
CRAY T3E C and C++ Optimization Guide
3.
#include <pvm3.h>
Line 8 defines the sending PE as PE 0 and the receiving PE as PE 1.
8.
int sender=0, receiver=1, mtag=2;
Line 10 declares the sending and receiving arrays as type float
, meaning they contain 32-bit data on CRAY T3E systems.
10.
float d_send[1000], d_recv[1000];
The program uses the fastest of the data encoding arguments to pvm_initsend
, which is
PvmDataRaw
. If the program were transferring 32-bit integer data,
PvmDataDefault would be faster. Because the sending and receiving arrays both begin with the first element and use a stride-1 increment, the 32-bit values will be packed and unpacked two data items per 64-bit word.
Lines 23 through 25 initialize the send buffer, pack the 32–bit data, and send the data:
23.
24.
25.
istat = pvm_initsend(PvmDataRaw); pkstat = pvm_pkfloat(d_send, N, 1); sstat = pvm_send(receiver, mtag);
Lines 31 and 32 receive and unpack the 32-bit data:
31.
32.
rstat = pvm_recv(sender, mtag); upkstat = pvm_upkfloat(d_recv, N, 1);
2.4 Sending and receiving stride-1 data
The pvm_psend
(3) and pvm_precv
(3) functions transfer either a single data item or stride-1 data between two PEs. You do not have to initialize a send buffer or pack and unpack the data when you use pvm_psend and pvm_precv
. For short messages, they run faster than the traditional send and receive functions, pvm_send and pvm_recv
.
The trade-off for the increase in speed is reduced flexibility. Using pvm_psend
, you are limited to a single block of contiguous data, and it can be sent to just one other PE
You are also limited to receiving a single block of contiguous data with pvm_precv
. But, after pvm_precv completes, it is done with the message.
Using pvm_recv
, one or more unpack calls may follow the receive call, and information about the message must be kept around in case the user calls
28 SG–2178 3.0.1
SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2] pvm_bufinfo
(3). In both the send and the receive, the pvm_psend and pvm_precv functions offer much simpler and faster code.
The speedups from using pvm_psend and pvm_precv are most noticeable for small messages, meaning less than the value of the
PVM_DATA_MAX environment variable (see Section 2.1, page 24). For large messages (greater than
PVM_DATA_MAX
), the performance benefits over pvm_send and pvm_recv are not significant.
The following example shows a program that passes data by using the pvm_psend and pvm_precv functions. The source PE ( src in the program) passes data to the destination PE ( dest
), which in turn passes it back to the src
PE.
Example 2: pvm_psend and pvm_precv
#include <stdio.h>
#include <pvm3.h>
#include <intrinsics.h> main()
{ int src=0, dest=1, len=10, back_and_forth=1000, tag=99; int me, i, retsend, retrec, atid, atag, alen; long array[10], *parray; me = _my_pe(); parray = &array[0];
/* Initialize data */ if (me == src) for (i=0; i < len; i++) array[i] = i * 1.0;
/* Send and receive data back_and_forth times */ for ( i=0; i < back_and_forth; i++) { if (me == src) {
/* Send data to dest PE */ retsend = pvm_psend (dest, tag, parray, len, PVM_LONG);
/* Receive data from dest PE */ retrec = pvm_precv (dest, tag, parray,
29
CRAY T3E C and C++ Optimization Guide len, PVM_LONG, &atid, &atag, &alen);
} else if (me == dest) {
/* Receive data from src PE */ retrec = pvm_precv (src, tag, parray, len, PVM_LONG, &atid, &atag, &alen);
}
/* Send data to src PE */ retsend = pvm_psend (src, tag, parray, len, PVM_LONG);
} /* End of else */
} /* End of for loop */
2.5 Mixing send and receive routines
The pvm_psend
(3) and pvm_precv
(3) functions operate together with the pvm_send and pvm_recv send and receive functions. You can gradually change your code to use them as appropriate, with some benefit accruing with each change. For example, if you find a place that is sending a one-word message using pvm_send
(3), you can change it to use pvm_psend without worrying about finding and changing the matching pvm_recv call. You can also continue to use pvm_bcast
(3) to broadcast a one-word message to all other PEs, something that can be very efficient (see Section 2.9, page 33), but change the receives to use the pvm_precv routine.
2.6 Initializing and packing data
You can use PVM in several different ways to send the same message to multiple targets. Some performance benefit is available by using special techniques.
If possible, avoid the following code construct, which initializes and packs the data buffer for each send.
for (i=0; i < numpes; i++) { bufid = pvm_initsend(PvmDataRaw); retpk = pvm_pklong (parray, n, 1); retsend = pvm_send (i, mtag);
}
30 SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2]
In the simplest case, you can remove the initialization and packing steps from the loop, as follows: bufid = pvm_initsend(PvmDataRaw); retpk = pvm_pklong (parray, n, 1); for (i=0; i < numpes; i++) { retsend = pvm_send (i, mtag);
}
This is more efficient because you pack the data only once. This means you need only one extra data block (for
PvmDataRaw or
PvmDataDefault packing) and one memory copy. Other PVM functions, such as pvm_bcast
(3) and pvm_mcast
(3) (see Section 2.9, page 33), provide alternatives to what remains of the for loop.
Some programs may include this inefficient code construct but hide it, as in the following example: for (i=0; i < numpes; i++) { retmy = myown_send(i, parray, n, 1);
}
In this example, myown_send is a message-passing envelope that contains PVM code that might look something like the following: int myown_send(int pe, long *pdata, int size, int incr)
{ int pe, size, incr, rc, bufid, retpk;
} bufid = pvm_initsend (PvmDataRaw); retpk = pvm_pklong (pdata, size, incr); rc = pvm_send (pe, 99); return rc;
In this case, you are gaining portability but sacrificing performance. You may want to consider using PVM directly or writing a routine that runs more efficiently.
2.7 Working while you wait
Sometimes you will be able to get work done while waiting for a message to arrive. In such cases, the pvm_nrecv
(3) function is a good substitute for pvm_recv
(3). The pvm_nrecv function does a nonblocking receive (glossary
SG–2178 3.0.1
31
CRAY T3E C and C++ Optimization Guide page 145), meaning it does not wait until the message arrives but rather returns immediately if there is no message. By checking with pvm_nrecv periodically, your program can monitor the arrival of a message and execute other statements while it waits.
The following example outlines one way in which you can make use of pvm_nrecv
: arrived = pvm_nrecv(-1, 4); if (arrived == 0) {
/* Do something else */
} else {
/* Process data in message */
}
2.8 Avoiding barriers
Barrier synchronization is appealing because it provides a clear definition of the status of each PE. Although the hardware barrier mechanism on CRAY T3E systems is fast, the waiting time may be long. A barrier requires that all PEs involved arrive at the barrier before any can proceed, so the true speed of a barrier is the speed of the slowest PE. If your application is not well balanced, waiting can slow it down dramatically.
PVM provides a simple form of synchronization. When a PE uses a blocking
receive (glossary, page 140) to receive a message from another PE, you know that the receiving PE will not go beyond that blocking receive until the sending
PE has completed its send. (The
PVMFRECV routine, for instance, uses a blocking receive.) Yet synchronization is accomplished without involving the other PEs and is combined with the transfer of data. Further synchronization, such as using barriers, is usually not needed.
The follow-on to avoiding barriers is to avoid synchronization of any sort, if possible. Synchronization creates idle PEs, especially when some PEs have more work than others. Although synchronous communication may be easier to understand, asynchronous communication provides better performance.
Unlike some message-passing systems, PVM does not have an asynchronous
receive (glossary, page 139), in which a receive is issued in parallel with the send and the application later checks the status of this receive. Instead, it provides a nonblocking receive, the pvm_nrecv function (see Section 2.7, page 31). On the
CRAY T3E system, pvm_nrecv provides comparable performance to the pvm_recv function. If possible, write your code in such a way that it can use a
32 SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2] nonblocking receive, so that if the message has not arrived, the code can do other work. See the example in the preceding subsection.
2.9 Using broadcast or multicast
The pvm_bcast
(3) and pvm_mcast
(3) functions offer two methods of sending messages to multiple PEs in a single call. The broadcast (glossary, page 140) function, pvm_bcast
, sends to all PEs in a group, whether that group consists of all PEs involved in the job or a predefined subset of all PEs. The multicast
(glossary, page 145) function, pvm_mcast
, sends to all PEs with PE numbers that appear in an array that you define.
Although pvm_mcast provides more flexibility concerning which PEs to send to, pvm_bcast is usually faster. If the group name you give to pvm_bcast is the global name, specified by a nil point or a pointer to a null string, PVM uses an optimized method to transfer the data. Instead of the broadcasting PE sending directly to all other PEs, it sends to half the PEs. When these PEs receive the message, they each forward it to half the remaining PEs, and so on.
(For an illustration, see Figure 9.) This provides better and more scalable performance in the following situations:
• If the number of PEs is approximately 32 or larger. There is usually extra time involved in forwarding such messages, meaning the forwarding method may not be as efficient with a smaller number of PEs.
• If the data packets are small (less than or equal to
PVM_DATA_MAX
). If they are larger, the forwarding method is abandoned, and all the receiving PEs try to do remote loads from the sending PE at more or less the same time.
SG–2178 3.0.1
33
CRAY T3E C and C++ Optimization Guide
34
0 1
= sender
2 3
= receiver
4 5 6
Figure 9. Fan-out method used by broadcasting routines
7 a10004
If you use a group name representing a subset of the PEs, there is no special optimization. PVM simply goes through the list of PEs in the group and sends to each PE.
The pvm_mcast function does not offer special optimizations. PVM goes through the specified array of PE numbers and sends to each PE.
The following two examples use pvm_bcast and pvm_mcast
, respectively, to transfer an array of 10 64–bit elements to all other PEs attached to the job:
Example 3: pvm_bcast
#include <pvm3.h>
#include <stdio.h> main()
{ int len = 10, mtag = 99; int mytid, me, npes, bufid, i; int retpk, retbc, retrecv, retupk; double arr[10];
/* Use PVM method of obtaining task id, */
/* PE number, and number of PEs */ mytid = pvm_mytid(); me = pvm_get_PE(mytid); npes = pvm_gsize(NULL);
SG–2178 3.0.1
SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2]
/* PE 0 initializes, packs, and sends */
/* the array of 10 elements */ if (me == 0) {
/* Initialize array */ for (i=0; i<len; i++) arr[i] = (i+1) /2.0;
/* Send array to all PEs */ bufid = pvm_initsend(PvmDataRaw); retpk = pvm_pkdouble(arr, len, 1); retbc = pvm_bcast(NULL, mtag);
}
/* All other PEs receive it */ else { retrecv = pvm_recv(0, mtag); retupk = pvm_upkdouble(arr, len, 1);
/* A representative PE prints the array */ if (me == 2) { printf("The array values are: "); for (i=0; i<len; i++) printf("%f ", arr[i]);
} /* End of inner if */
} /* End of else */
} /* End of main */
Example 4: pvm_mcast
#include <pvm3.h>
#include <stdio.h> main()
{ int len = 10, mtag = 99; int mytid, me, npes, bufid, i; int retpk, retmc, retrecv, retupk; int pe_arr[10]; double arr[10];
35
CRAY T3E C and C++ Optimization Guide
/* Use PVM method of obtaining task id, */
/* PE number, and number of PEs */ mytid = pvm_mytid(); me = pvm_get_PE(mytid); npes = pvm_gsize(NULL);
/* Set up an array of PE numbers */ for(i=0; i<len; i++) pe_arr[i] = i+1;
/* PE 0 initializes, packs, and sends */
/* the array of 10 elements */ if (me == 0) {
/* Initialize array */ for (i=0; i<len; i++) arr[i] = (i+1) /2.0;
}
/* Send array to PEs */ bufid = pvm_initsend(PvmDataRaw); retpk = pvm_pkdouble(arr, len, 1); retmc = pvm_mcast(pe_arr, npes-1, mtag);
/* All other PEs receive it */ else { retrecv = pvm_recv(0, mtag); retupk = pvm_upkdouble(arr, len, 1);
/* A representative PE prints the array */ if (me == 2) { printf("The array values are: "); for (i=0; i!=len; i++) printf("%f ", arr[i]);
} /* End of inner if */
} /* End of else */
} /* End of main */
36 SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2]
The output from both programs is as follows:
The array values are: 0.5
1.
1.5
2.
2.5
3.
3.5
4.
4.5
5.
Because the efficient message-passing system used by pvm_bcast becomes more of a factor as the number of PEs increases, the advantage in using pvm_bcast is most apparent when more PEs are involved in the job. But even when using as few as eight PEs, pvm_bcast still has better performance than pvm_mcast
.
2.10 Minimizing synchronization time when receiving data
When a single PE receives data from multiple PEs, use –1 as the task identifier argument to the pvm_recv
(3) function to save time receiving the data. The –1 value allows a PE to receive data from any PE. The following example forces data to be received in the numeric order of the PEs. It assumes that PE 0 is receiving the data. The first argument to the pvm_recv function specifies the
PE number from which the data is received.
offset = 1; for(i=0; i<npes; i++) { istat = pvm_recv(i, msgtag); istat = pvm_upklong(&x[msgtag-1], length, 1); offset = offset + length;
}
In the preceding example, regardless of which PE gets its data there first, PE 0 will wait until the data from PE 1 arrives and is received before it can receive data from any other PE. The following example receives whatever data arrives first: for (i=0; i<npes; i++) { istat = pvm_recv(-1, msgtag); istat = pvm_upklong(&x[msgtag-1] * length, 1);
}
Unless the data in this example can be put into the x array in random order, you must check the message tag to find out which PE sent a given message.
The example assumes the sending PE has sent its PE number in the message tag. Remember, the data is likely to arrive in a different order for different executions of the program.
A loop such as the following offers a second way to order the arriving data in the receiving array:
SG–2178 3.0.1
37
CRAY T3E C and C++ Optimization Guide for(i=0; i<npes; i++) { istat = pvm_recv(-1, msgtag); istat = pvm_upkint(source, 1, 1); istat = pvm_upklong(&x[source * length], length, 1);
}
This example assumes that each PE has sent its identifier in source
, which is the first part of the message.
The following example assumes the sending PEs did not include their identifiers in the message. Instead, a call to the pvm_bufinfo function retrieves the value of the task identifier (the fourth argument), converts it to a PE number using pvm_get_PE
(3), and places it in the variable nextpe
.
nextpe is then used in the pvm_upklong function to provide the element number in the array x
.
for(i=0; i<npes; i++) { istat = pvm_recv(-1, msgtag); istat = pvm_bufinfo(bufid, &bytes, &msgtag, &source); nextpe = pvm_get_PE(source); istat = pvm_upklong(&x[(nextpe-1) * length], length, 1);
}
The three preceding methods are approximately equivalent in terms of performance.
2.11 Using the reduction functions
A single interface, pvm_reduce
, performs common reduction functions across multiple PEs, returning the results to a single PE. For instance, if each PE contains an array of integers, pvm_reduce can find the largest value in any of the arrays at each location and return those answers to an array on a PE that you specify. By executing the following call, you will end up with an array of the largest values on PE 0, as illustrated in Figure 10: istat = pvm_reduce(PvmMax, ARR, 10, PVM_INT, mtag, "squad1", 0);
38 SG–2178 3.0.1
SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2]
5
6
3
4
0
1
2
7
8
9
PE0 arr
36
17
5
4
31
11
91
18
20
19
PE1 arr
1
0
36
11
6
9
14
3
20
19
5
6
3
4
0
1
2
7
8
9
arr
0
17
3
4
13
7
91
18
12
7
5
6
3
4
0
1
2
7
8
9
PE2 PE3 arr
5
4
14
15
8
5
31
11
18
1
a11385
5
6
3
4
0
1
2
7
8
9
Figure 10. A
PvmMax reduction
Reduction functions are faster than other PVM methods of finding the same answers. The following example adds the values at each array position for each instance of the array and returns the sum to that array position on PE 0:
Example 5: A
PvmSum example
#include <pvm3.h>
#include <stdio.h> main()
{ int mytid, me, npes, istat, len=10; int arr[10], results[10]; int i, mtag; mytid = pvm_mytid(); mtag = 99; me = pvm_get_PE(mytid); npes = pvm_gsize(NULL);
39
CRAY T3E C and C++ Optimization Guide
/* Everyone sums */
/* Initialize the array */ for(i=0; i<len; i++) arr[i] = me * i;
/* Get the summers organized */ pvm_barrier(NULL,npes);
/* Find the sums */ istat = pvm_reduce(PvmSum, arr, len, PVM_INT, mtag, NULL, 0);
/* Write the answers from PE 0 */ if(me == 0) { printf("The array sums are: "); for(i=0; i<len; i++) printf(" %d ", arr[i]);
} /* End of if */
} /* End of program */
Running the program on eight PEs, the output is as follows:
The array sums are: 0 28 56 84 112 140 168 196 224 252
2.12 Gathering and scattering data
An efficient way to process a large array using PVM is to divide its elements among multiple PEs, process those elements, and reassemble the array on a single PE. The pvm_gather
(3) and pvm_scatter
(3) functions are well suited to do just that.
The pvm_scatter function distributes sections of an array among a group of
PEs, and pvm_gather combines arrays from multiple PEs into a single array.
Because they are designed for this purpose, they are faster than other PVM functions at gathering and scattering. The process is illustrated in Figure 11.
40 SG–2178 3.0.1
SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2]
Process
PE1
9
10
11
7
8
5
6
2
3
4
0
1
PE0
Scatter
}
Gather
}
}
}
PE2
PE3
}
}
Figure 11. The gather/scatter process
PE0
a11386
The following example collects each PE’s small_c array into the big_c array on the root PE:
6
7
4
5
2
3
0
1
8
9
10
11
41
CRAY T3E C and C++ Optimization Guide
42
Example 6: Gather operation
#include <pvm3.h>
#include <stdio.h> main() { int root_pe = 0, ixdim = 64, iydim = 128, msgtag = 9; int istat, mype, i, mytid, j, npes; double a[64][128], b[128];
/* Assume 4 PEs, one of which is root_pe */ double small_c[64], big_c[64*4]; mytid = pvm_mytid(); mype = pvm_get_PE(mytid); npes = pvm_gsize(0);
/* Initialize arrays */ for(i=0; i<ixdim; i++) for(j=0; j<iydim; j++) a[i,j] = mype; for(i=0; i<iydim; i+=2) b[i] = -1.2; for(i=1; i<iydim; i+=2) b[i] = 1; for(i=0; i<ixdim; i++) small_c[i] = 0.0;
/* Perform matrix-vector multiplication on each PE */ for(i=0; i<ixdim; i++) for(j=0; j<iydim; j++) small_c[i] = small_c[i] + a[i][j] * b[j];
/* Gather each small_c into root_pe’s big_c */ istat = pvm_gather(big_c, small_c, ixdim, PVM_DOUBLE, msgtag, 0, root_pe); if(mype == root_pe) for(i=0; i<ixdim*4; i++) printf("%f", big_c[i]);
}
SG–2178 3.0.1
SG–2178 3.0.1
Parallel Virtual Machine (PVM) [2]
The following example scatters the big_x array into the smaller small_x arrays.
Example 7: Scatter operation
#include <pvm3.h>
#include <stdio.h> main() {
/* Assume 4 PEs, any of which is root_pe */ int root_pe = 0, iy = 128, ix = iy/4, mtag = 11; int istat, mype, mytid, i; double small_x[32], big_x[128]; mytid = pvm_mytid(); mype = pvm_get_PE(mytid);
/* Initialize data on root_pe */ if(my_pe == root_pe) for(i=0; i<iy; i++) big_x[i] = 1.0 * i; istat = pvm_scatter(small_x, big_x, ix, PVM_DOUBLE, mtag, 0, root_pe);
}
/* Print from any PE */ if(mype == 1) { printf("For PE %d: \n", mype); for(i=0; i<ix; i++) printf("Small_x %d is %f\n", i, small_x[i]);
}
43
SHMEM [3]
SG–2178 3.0.1
You can either use shared memory (SHMEM) routines alone or mix them into a program that primarily uses PVM (glossary, page 146) or MPI (glossary, page
144), thereby offering opportunities for optimizations beyond what the message-passing protocols can provide. Be aware, however, that SHMEM is not a standard protocol and will not be available on machines developed by companies other than Silicon Graphics and Cray Research. SHMEM is supported on Cray PVP systems, Cray MPP systems, and on Silicon Graphics systems.
For background information on SHMEM, see Section 1.1.2, page 2. For an introduction to the SHMEM routines, see the shmem_intro
(3) man page.
This chapter describes the following optimization techniques:
• Improving data transfer rates in any CRAY T3E program by using SHMEM get and put routines (see Section 3.1, page 46). This section provides an introduction to data transfer, which is the most important capability that
SHMEM offers.
• Improving the performance of a PVM or MPI program by adding SHMEM data manipulation routines (see Section 3.2, page 51).
• Avoiding performance pitfalls when passing 32-bit data rather than 64-bit data (see Section 3.3, page 59).
• Copying strided (glossary, page 148) data while maintaining maximum performance. The strided data routines enable you, for example, to divide the elements of an array among a set of processing elements (PEs) or pull elements from arrays on multiple PEs into a single array on one PE (see
Section 3.4, page 62).
• Gathering and scattering data and reordering it in the process (see Section
3.5, page 67).
• Broadcasting (glossary, page 140) data from one PE to all PEs (see Section 3.6, page 70).
• Merging arrays from each PE into a single array on all PEs (see Section 3.7, page 72).
• Executing an atomic memory operation (glossary, page 139) to read and update a remote value in a single process (see Section 3.8, page 75).
45
CRAY T3E C and C++ Optimization Guide
• Using reduction (glossary, page 146) functions to execute a variety of operations across multiple PEs (see Section 3.9, page 76).
3.1 Using shmem_get and shmem_put for data transfer
In general, avoiding communications between PEs (including data transfer) improves performance. The fewer the number of communications, the faster your program can execute. Data transfer is, however, often necessary. Finding the fastest method of passing data is an important optimization, and the
SHMEM routines are usually the fastest method available.
The shmem_put and shmem_get functions avoid the extra overhead (glossary, page 145) sometimes associated with message passing routines by moving data directly between the user-specified memory locations on local and remote PEs.
The shmem_put and shmem_get functions perform at virtually the same rates.
For large transfers, shmem_put offers a different kind of performance advantage by letting the calling PE perform other work while the data is in the network. Because shmem_put is asynchronous, it may allow statements that follow it to execute while the data is in the process of being copied to the memory of the receiving PE. The shmem_get function forces the calling PE to wait until the data is in local memory (glossary, page 144), meaning that no overlapping of data transfer and early work can be done.
Passing data in large chunks is always faster than passing it in small chunks because it saves overhead. Whenever possible, put all of your data (such as an array) into a single shmem_put or shmem_get call rather than calling the function more than once. The same applies to other SHMEM functions.
In the following example, eight 64-bit words are transferred from PE 1 to PE 0 by using shmem_put
. PE numbering always begins with 0.
Example 8: Example of a shmem_put transfer
1.
#include <stdio.h>
2.
#include <mpp/shmem.h>
3.
#include <intrinsics.h>
4.
6.
int me, npes, i;
7.
int source[8], dest[8];
8.
main()
9.
{
10.
11.
/* Get PE information */ me = _my_pe();
46 SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
12.
13.
14.
15.
16.
17.
18.
19.
npes = _num_pes();
/* Initialize and send on PE 1 */ if(me == 1) { for(i=0; i<8; i++) source[i] = i+1; shmem_put64(dest, source, 8*sizeof(dest[0])/8, 0);
}
20.
21.
22.
23.
24.
25.
/* Make sure the transfer is complete */ shmem_barrier_all();
26.
27.
28.
29.
/* Print from the receiving PE */ if(me == 0) {
_shmem_udcflush(); printf(" DEST ON PE 0:"); for(i=0; i<8; i++) printf(" %d%c", dest[i],
30.
31.
}
(i<7) ? ’,’ : ’\n’);
32.
} /* End of main */
See the following figure for an illustration of the transfer.
47
CRAY T3E C and C++ Optimization Guide
PE0
PE1
Dest
7
8
5
6
3
4
1
2
6
7
4
5
2
3
0
1
Source
1
2
0
1
3
4
2
3
5
6
4
5
7
8
6
7 a11384
Figure 12.
SHMEM_PUT64 data transfer
48 SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
The output from the example is as follows:
DEST ON PE 0: 1, 2, 3, 4, 5, 6, 7, 8
Defining the number of PEs in a program and the number in an active set
(glossary, page 139) as powers of 2 (that is, 2, 4, 8, 16, 32, and so on) helped performance on CRAY T3D systems. Also, declaring arrays as powers of 2 was necessary if you were using Cray Research Adaptive Fortran (CRAFT) on
CRAY T3D systems. Both have changed as follows on CRAY T3E systems:
• Declaring arrays such as source and dest as multiples of 8 helps SHMEM speed things up somewhat, since 8 is the vector length of a key component of the PE remote data transfer hardware. Declaring the number of elements as a power of 2 does not affect performance unless that number is also a multiple of 8.
• Defining the number of PEs, whether you are referring to all PEs in a program or to the number involved in an active set, as a power of 2 does not usually enhance performance in a significant way on the CRAY T3E system. Some SHMEM routines, notably
SHMEM_BROADCAST
, still do benefit somewhat from defining the number of PEs as a power of 2.
For information on optimizing an existing MPI program using shmem_get and shmem_put
, see Example 8, page 46. For a complete description of the
MPP-specific statements in the preceding example, continue on with this section.
In the shmem_put example, line 2 imports the SHMEM include file, which defines parameters needed by many of the routines. The location of the file may be different on your system. Check with your system administrator if you do not know the correct path. If you are using the module command and have loaded craytools
, you do not need this line in your program. Subsequent examples will use the
#include method.
2.
#include <mpp/shmem.h>
Line 3 includes the intrinsics include file, intrinsics.h
, which is required if you are using the C++ compiler and are accessing intrinsic functions in your program. This program accesses the
_my_pe and
_num_pes intrinsics in lines
11 and 12. If you are using the C compiler, you do not need to include intrinsics.h
, but it does no harm to do so.
3.
#include <intrinsics.h>
49
CRAY T3E C and C++ Optimization Guide
50
Note:
Most of the examples in this publication use the intrinsic functions
_my_pe and
_num_pes because they are at least as fast as their message-passing system equivalents, such as the shmem_my_pe and shmem_n_pes functions. Both are available on Cray PVP systems as well as
Cray MPP systems.
In lines 11 and 12, the
_my_pe and
_num_pes functions are called on each PE.
Their values are assigned to me and npes
, respectively. Any subsequent reference to the variable me returns the number of the calling PE.
10.
11.
12.
/* Get PE information */ me = _my_pe(); npes = _num_pes();
The value of me is tested in line 15. Because the test returns a value of true on
PE 1, PE 1 initializes the source array in line 17.
15.
16.
17.
if(me == 1) { for(i=0; i<8; i++) source[i] = i+1;
In line 18, PE 1 executes the shmem_put64 function call that sends the data. It sends eight array elements from its source array to the dest array on PE 0.
18.
shmem_put64(dest, source, 8*sizeof(dest[0])/8, 0);
Line 22 is a barrier (glossary, page 139), which provides a synchronization point
(glossary, page 148). No PE proceeds beyond this point in the program until all
PEs have arrived. The effect in this case is to wait until the transfer has finished. Without the barrier, PE 0 might print the dest array before receiving the data. Calling shmem_barrier_all is as fast as calling the barrier function directly.
22.
shmem_barrier_all();
Line 25 selects PE 0, which is passively receiving the data. PVM requires their receiving tasks (glossary, page 148) to call routines that receive and unpack the data, but shmem_put64 places the data directly into PE 0’s local memory. PE 0 is not involved in the transfer operation. After being released from the barrier in line 22, PE 0 prints dest
, and the program exits. The shmem_udc_flush function in line 26 is a no-op on CRAY T3E systems, but keeping it in your code makes the program more portable.
24.
25.
26.
/* Print from the receiving PE */ if(me == 0) {
_shmem_udc_flush();
SG–2178 3.0.1
SHMEM [3]
27.
28.
29.
30.
printf(" DEST ON PE 0:"); for(i=0; i<8; i++) printf("%d", dest[i];
(i<7) ? ’,’ : ’\n’);
31.
}
32.
} /* End of main */
3.2 Optimizing existing MPI and PVM programs by using SHMEM
The following sections show two methods of optimizing an MPI program by using SHMEM routine calls. SHMEM data-transfer routines have lower latencies
(glossary, page 144) and higher bandwidth (glossary, page 139) than comparable message-passing routines. The MPI version of the program is as follows.
Note:
The following programs implement a global summation around a ring.
They are intended to compare equivalent SHMEM and MPI versions of a program, not to provide an optimal implementation of a global summation.
For a faster version of a global summation, see the reduction routines on page 79.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
5.
6.
7.
8.
1.
2.
3.
4.
Example 9: MPI version of the ring program
#include <stdio.h>
#include <mpi.h>
#include <intrinsics.h>
MPI_Status istat; int me, npes, total, send, recv, i; int next, prev, expect, bufid, istat; main(int *argc, char **argv)
{
MPI_Init(argc,&argv);
/* Get PE values */ me = _my_pe(); npes = _num_pes()
/* Define the ring */ next = me + 1; if(next >= npes) next = next - npes; prev = me -1; if(prev < 0) prev = prev + npes;
SG–2178 3.0.1
51
CRAY T3E C and C++ Optimization Guide
52
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
22.
23.
24.
25.
26.
27.
28.
29.
40.
41.
42.
}
/* Initialize data */ send = me; total = me; for(i=2; i<=npes; i++) {
/* Send data to next PE */ ierr=MPI_Send(&send,1,MPI_INT,next,99,MPI_COMM_WORLD);
/* Receive data from previous PE */ ierr=MPI_Recv(&recv,1,MPI_INT,prev,99,MPI_COMM_WORLD,
&istat);
/* Do work */ total = total + recv; send = recv;
} /* End of for loop */ printf("PE= %d Result= %d Expect= %d", me, total,
(int)((npes-1)*npes*.5));
This program simply passes messages around a ring of PEs. All of the PEs execute all of the statements. Each passes its PE number around and adds the number it receives to the variable total
. When each PE has seen the PE number of every other PE, each prints out its own PE number and the total it has calculated.
The output from the program, reflecting the random order in which the PEs finish, is as follows:
PE = 1 Result = 28 Expect = 28.
PE = 5 Result = 28 Expect = 28
PE = 3 Result = 28 Expect = 28
PE = 7 Result = 28 Expect = 28
PE = 2 Result = 28 Expect = 28
PE = 4 Result = 28 Expect = 28
PE = 6 Result = 28 Expect = 28
PE = 0 Result = 28 Expect = 28
For the shmem_get version of the same program, see Section 3.2.1, page 55; otherwise, see the remainder of this section for a more detailed description of the MPI version.
SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
Line 2 references the MPI header file. If you use the module
(1) package to define your environment and have loaded mpt as one of the modules, this line is not necessary. Line 3 references the C intrinsics file.
2.
3.
#include <mpi.h>
#include <intrinsics.h>
Line 10 initializes MPI.
10.
MPI_Init(argc,&argv);
Lines 12 and 13 use functions declared in intrinsics.h
to determine each
PE’s number and how many other PEs are involved in the program. Line 12 returns the PE number for each PE and puts the value into the variable me
.
Line 13 determines the number of PEs in the program and puts the value into the npes variable. The intrinsic functions
_my_pe and
_num_pes are faster than other methods of gaining the same information, and they are implemented on Cray PVP systems. They are not, however, available on systems other than
Cray Research systems:
11.
12.
13.
/* Get PE values */ me = _my_pe(); npes = _num_pes();
15.
16.
17.
18.
19.
20.
21.
The next few statements define a PE’s neighbors. The variables next and prev
, defined in lines 16 and 19, specify which of its neighbors most of the PEs will be passing to and which they will be receiving from. Lines 17, 18 , 20, and 21 define neighbors for the PEs on the end, namely PE and PE 7 in an 8-PE configuration. Lines 17 and 18 cause PE 7 to pass to PE 0, and lines 20 and 21 cause PE to receive from PE 7.
/* Define the ring */ next = me + 1; if(next >= npes) next = next - npes; prev = me -1; if(prev < 0) prev = prev + npes;
The values for next and prev in each PE are as illustrated in the following figure.
53
CRAY T3E C and C++ Optimization Guide
54
PE 0 PE 1 PE 2 PE 3 PE 4 PE 5 PE 6 PE 7
next=1 next=2 next=3 next=4 next=5 next=6 next=7 next=0 prev=7 prev=0 prev=1 prev=2 prev=3 prev=4 prev=5 prev=6
a10827
Figure 13. Identification of neighbors in the ring program.
Line 23 initializes the variable that each PE will pass on to next
, and line 24 initializes the variable that will hold the running total in each PE. Both variables at first contain the number of the respective PE.
23.
24.
25.
/* Initialize data */ send = me; total = me;
Next comes the for loop, within which the MPI statements pass and receive the data. It executes once for every PE except me
, since the value of me is already in the running total. In line 30, the
MPI_Send function sends the data
( send
) to the next PE. All of the MPI functions are described in more detail on their man pages.
29.
30.
/* Send data to next PE */ ierr=MPI_Send(&send,1,MPI_INT,next,99,MPI_COMM_WORLD);
Line 33 receives the data from the PE that is defined in prev and puts it into the variable recv
.
32.
33.
34.
/* Receive data from previous PE */ ierr=MPI_Recv(&recv,1,MPI_INT,prev,99,MPI_COMM_WORLD,
&istat);
At the end of the for loop, each PE updates its running total and moves the number it received into the send variable, preparing to pass it on to next in the next iteration of the loop.
35.
36.
37.
/* Do work */ total = total + recv; send = recv;
SG–2178 3.0.1
SHMEM [3]
3.2.1 Optimizing by using shmem_get
To optimize the MPI version of the ring program shown in Example 9, page 51, you can replace the MPI message-passing statements with SHMEM statements.
You also need explicit synchronization points, in the form of barriers, to replace the implicit synchronization provided by the MPI send and receive routines.
The optimization described in this section and shown in the following example uses the shmem_get function; in Section 3.2.2, page 58, MPI is replaced by shmem_put
.
Instead of passing a single value, the SHMEM versions of the program pass an array of 10 elements. Each of the array elements contains the same value, a PE number.
Example 10: shmem_get version of the ring program
1.
2.
3.
4.
#include <stdio.h>
#include <mpp/shmem.h>
#define N 10 void work(int n,double *d_total,double *d_send,double *d_recv);
5.
6.
double d_send[N]; main()
7.
8.
9.
10.
{ double d_recv[N], d_total[N]; double d_me; int i, me, npes, next, prev;
11.
/*
12.
Get PE info
13.
*/
14.
15.
16.
/* me = _my_pe(); npes = _npes();
17.
18.
*/
Define the ring
19.
20.
next = me + 1; if (next >= npes) next = next - npes;
21.
22.
prev = me - 1; if (prev < 0) prev = prev + npes;
23.
/*
24.
Initialize data
25.
*/
26.
27.
28.
d_me = me; for (i=0; i < N; i++){ d_send[i] = d_me;
SG–2178 3.0.1
55
CRAY T3E C and C++ Optimization Guide
56
29.
30.
31.
32.
} d_total[i] = d_me; for (i=1; i < npes; i++) {
33.
/* Synchronize - Make sure data is ready on other PEs */
34.
shmem_barrier_all();
35.
36.
/* Get data from previous PE */
37.
38.
shmem_double_get(d_recv, d_send, N, prev);
39.
/* Synchronize - Make sure data is ready on other PEs */
40.
shmem_barrier_all();
41.
/*
42.
Perform work
43.
*/
44.
45.
46.
} work(N, d_total, d_send, d_recv);
47.
48.
printf(" PE = %d result = %f expect = %f\n", me, d_total[0], (npes-1)*npes*.5);
49.
}
50.
void work(int n,double *d_total,double *d_send, double *d_recv)
51.
{
52.
53.
54.
55.
56.
57.
} int i; for (i=0; i < n; i++){
} d_total[i] += d_recv[i]; d_send[i] = d_recv[i];
Lines 2 and 3 reference the stdio and SHMEM
#include files. Line 3 declares the size of the remote and local arrays to be 10.
1.
2.
3.
#include <stdio.h>
#include <mpp/shmem.h>
#define N 10
Lines 14 and 15 use the same intrinsic functions and get the same information
(the number of the calling PE and the number of PEs involved in the job) as the
MPI calls in lines 11 and 12 of the MPI version of the program (see Example 9, page 51).
14.
15.
me = _my_pe(); npes = _npes();
SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
The SHMEM version of the program defines the ring, initializes the data, updates the running total, and writes the output exactly as in the MPI version of the program. Only the method of passing data differs.
Lines 34 and 40, which set barriers, are necessary when using the shmem_get function. Synchronization is implicit in the MPI version because of the MPI mode of operation: each send is matched by a receive. You must provide your own synchronization when using SHMEM. The shmem_barrier_all function takes advantage of the fast hardware barrier mechanism, making these calls relatively inexpensive. This can replace the implicit synchronization in MPI by this faster synchronization method when you are converting between MPI and
SHMEM.
33.
/* Synchronize - Make sure data is ready on other PEs */
34.
shmem_barrier_all();
35.
36.
/* Get data from previous PE */
37.
38.
shmem_double_get(d_recv, d_send, N, prev);
39.
/* Synchronize - Make sure data is ready on other PEs */
40.
shmem_barrier_all();
Other performance improvements that you will see when converting from MPI to SHMEM data passing are as follows. They apply whether you are using shmem_put or shmem_get
.
• SHMEM does not require separate calls to routines to initialize, to send the data, and to receive the data.
• SHMEM does not require the remote PE to be involved while doing transfers. That means the remote PE is free to do other work, although it does not do so in this example.
If you have written programs for the CRAY T3D system or for Cray PVP systems, you are probably accustomed to flushing data cache (glossary, page 143) on the PE receiving the data in order to preserve cache coherence (glossary, page
140). That is no longer necessary on the CRAY T3E system. For portability purposes, however, you can leave cache flushing routine calls in your program.
They are essentially ignored on CRAY T3E systems, so they do not affect performance, but they are required by CRAY T90 systems and may be required by future systems.
57
CRAY T3E C and C++ Optimization Guide
3.2.2 Optimizing by using shmem_put
The shmem_put function can deliver the answers as fast as shmem_get
. The following example shows the shmem_put version of the ring program.
Example 11: shmem_put version of the ring program
5.
6.
7.
8.
1.
2.
3.
4.
/* program ring_nsum_3
Summing around a ring shmem_put version
*/
#include <stdio.h>
#include <mpp/shmem.h>
#define N 10
9.
void work(int n,double *d_total,double *d_send,double *d_recv);
10.
double d_recv[N];
11.
main()
12.
{
13.
14.
15.
16.
/* double d_send[N], d_total[N]; double d_me; int i, me, npes, next, prev;
17.
18.
*/
19.
20.
21.
/*
Get PE info me = _my_pe(); npes = _npes();
22.
23.
*/
Define the ring
24.
25.
next = me + 1; if (next >= npes) next = next - npes;
26.
27.
prev = me - 1; if (prev < 0) prev = prev + npes;
28.
/*
29.
Initialize data
30.
*/
31.
32.
33.
34.
35.
36.
37.
d_me = me; for (i=0; i < N; i++){
} d_send[i] = d_me; d_total[i] = d_me; for (i=1; i < npes; i++) {
58 SG–2178 3.0.1
SHMEM [3]
38.
/* Send data to next PE */
39.
shmem_double_put(d_recv, d_send, N, next);
40.
41.
/* Synchronize -
42.
43.
44.
45
Wait for data to arrive at other PE (implicit SHMEM_QUIET) and make sure everyone has data */ shmem_barrier_all();
46.
/* Perform work */
47.
work(N, d_total, d_send, d_recv);
48.
49.
/* Synchronize - Make sure everyone is ready to continue */
50.
51.
} shmem_barrier_all();
52.
53.
54.
55.
} printf(" PE = %d result = %f expect = %f\n", me, d_total[0], (npes-1)*npes*.5);
56.
void work(int n,double *d_total,double *d_send,double *d_recv)
57.
{
58.
59.
int i; for (i=0; i<N; i++) {
60.
61.
62.
63.
}
} d_total[i] += d_recv[i]; d_send[i] = d_recv[i];
The first half of the program is the same as the shmem_get version (see
Example 10, page 55), except that the remote variable is now d_recv
, which is the target of the data transfer.
In line 39, each PE passes the data to the next PE.
38.
/* Send data to next PE */
39.
shmem_double_put(d_recv, d_send, N, next);
3.3 Passing 32-bit data
Two variants of shmem_get and shmem_put
, shmem_get32 and shmem_put32
, are designed and optimized specifically for passing 32-bit data.
When used properly, they can pass 32-bit data faster than shmem_get64 and shmem_put64
.
SG–2178 3.0.1
59
CRAY T3E C and C++ Optimization Guide
When you use the 32-bit routines, try to align either both or neither of the destination array and the source array on a 64-bit boundary. Performance slips significantly when the two are not so aligned, as in the following call: shmem_put32(dest[0], source[5], nlong, pe);
Instead, cache-align the two arrays and begin the transfer either on two even-numbered or two odd-numbered array elements. The C and C++ directive cache_align serves the purpose of aligning cache.
#pragma _CRI cache_align dest, source shmem_put32(dest, source, nlong, pe);
The following 32-bit version of the ring program uses the shmem_put32 function:
Example 12: 32-bit version of ring program
1.
/*
2.
program ring_nsum_3_4
3.
4.
Summing around a ring shmem_put version
5.
*/
6.
#include <stdio.h>
7.
#include <mpp/shmem.h>
8.
#define N 100000 /* Size of transfer */
9.
void work(int n,float *d_total,float *d_send,float *d_recv);
10.
float d_recv[N];
11.
main()
12.
{
13.
14.
15.
16.
/* float d_send[N], d_total[N]; float d_me; int i, me, npes, next, prev;
17.
18.
*/
Get PE info
19.
20.
me = _my_pe(); npes = _n_pes();
21.
/*
22.
23.
*/
Define the ring
24.
25.
26.
next = me + 1; if (next >= npes) next = next - npes; prev = me - 1;
60 SG–2178 3.0.1
SHMEM [3]
27.
28.
/* if (prev < 0) prev = prev + npes;
29.
30.
*/
Initialize data
31.
32.
33.
34.
d_me = me; for (i=0; i < N; i++){ d_send[i] = d_me; d_total[i] = d_me;
35.
36.
} for (i=1; i < npes; i++) {
37.
38.
/* Send data to next PE */
39.
40.
shmem_put32(d_recv, d_send, N, next);
41.
/* Synchronize -
42.
Wait for data to arrive at other PE (implicit SHMEM_QUIET)
43.
44.
and make sure everyone has data */ shmem_barrier_all();
45.
46.
/* Make sure cache is updated */
47.
48.
shmem_udcflush();
49.
/* Perform work */
50.
51.
work(N, d_total, d_send, d_recv);
52.
/* Synchronize - Make sure everyone is ready to continue */
53.
shmem_barrier_all();
54.
55.
}
56.
57.
printf(" PE = %d result = %f expect = %f\n", me, d_total[0], (npes-1)*npes*.5);
58.
}
59.
void work(int n,float *d_total,float *d_send,float *d_recv)
60.
{
61.
62.
63.
int i; for (i=0; i < N; i++) { d_total[i] += d_recv[i]; d_send[i] = d_recv[i]; 64.
65.
66.
}
}
The output from the program is as follows:
SG–2178 3.0.1
61
CRAY T3E C and C++ Optimization Guide
PE = 0 Result = 28. Expect = 28.
PE = 7 Result = 28. Expect = 28.
PE = 3 Result = 28. Expect = 28.
PE = 1 Result = 28. Expect = 28.
PE = 6 Result = 28. Expect = 28.
PE = 2 Result = 28. Expect = 28.
PE = 4 Result = 28. Expect = 28.
PE = 5 Result = 28. Expect = 28.
3.4 Copying strided data
The strided-data (glossary, page 148) copy operation, using the shmem_iput and shmem_iget functions, takes data from an array on one PE and delivers it to an array on another PE. You control the stride for both the source and target arrays through arguments in the function calls. The SHMEM versions of the strided copy operation offer both speed and the ability to reorder the elements.
(For information on reordering data, see Section 3.5, page 67.)
The following examples take elements from an array on PE 0 and move them to an array on PE 1. Both examples, one using shmem_double_iget and the other shmem_double_iput
, use strides other than 1. These examples are assumed to be 2-PE programs.
Example 13: Passing strided data using shmem_iget
5.
6.
7.
8.
1.
2.
3.
4.
/*
*/ shmem_iget version
The sending array is accessed with stride 2
The receiving array is accessed with stride 3
#include <stdio.h>
#include <mpp/shmem.h>
9.
#define N 100
10.
#define SENDER 0
11.
#define RECEIVER 1
12.
13.
double d_send[2*N];
14.
15.
main()
16.
{
17.
18.
int me, i; double d_recv[3*N];
62 SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
19.
20.
/* Get PE info */
21.
22.
me = shmem_my_pe();
23.
24.
if (me == SENDER) {
25.
/* Initialize data */
26.
for (i=0; i < 2*N; i++) {
27.
28.
29.
30.
} d_send[i] = i + me + 1.0; for (i=0; i < 3*N; i++) { d_recv[i] = 0.0;
31.
32.
}
33.
/* Synchronize - Make sure data is ready */
34.
shmem_barrier_all();
35.
/* Note: Sender does nothing but synchronize,
36.
Receiver does all the work */
37.
38.
} else if (me == RECEIVER) {
39.
40.
/* Synchronize - Make sure data is ready on other PE */
41.
44.
45.
shmem_barrier_all();
42.
43.
/* Get data */ shmem_double_iget(d_recv,d_send,3,2,N,SENDER);
46.
/* Print results */
47.
printf("Receiver=%d d_recv=%f %f %f %f %f %f %f\n",
48.
49.
me, d_recv[0],d_recv[1],d_recv[2],d_recv[3], d_recv[4],d_recv[5],d_recv[6]);
50.
51.
}
}
1.
2.
3.
4.
5.
6.
7.
Example 14: Passing strided data using shmem_iput
/* shmem_iput version
The sending array is accessed with stride 2
The receiving array is accessed with stride 3
*/
#include <stdio.h>
#include <mpp/shmem.h>
63
CRAY T3E C and C++ Optimization Guide
64
8.
9.
#define N 100
#define SENDER 0
10.
#define RECEIVER 1
11.
12.
double d_recv[3*N];
13.
14.
main()
15.
{
16.
17.
int me, i; double d_send[2*N];
18.
19.
/* Get PE info */
20.
21.
me = shmem_my_pe();
22.
23.
if (me == SENDER) {
24.
/* Initialize data */
25.
26.
27.
for (i=0; i < 2*N; i++) d_send[i] = i + me + 1.0;
28.
29.
30.
33.
34.
for (i=0; i < 3*N; i++) d_recv[i] = 0.0;
31.
32.
/* Send data */ shmem_iput(d_recv, d_send, 3, 2, N, RECEIVER);
35.
/* Synchronize - Make sure data has arrived */
36.
shmem_barrier_all();
37.
38.
} else if (me == RECEIVER) {
39.
40.
/* Synchronize - Make sure data has arrived */
41.
42.
shmem_barrier_all();
43.
/* Make sure cache is up to date */
44.
shmem_udcflush();
45.
46.
/* Print results */
47.
48.
49.
50.
} printf("Receiver=%d d_recv=%f %f %f %f %f %f %f\n", me, d_recv[0],d_recv[1],d_recv[2],d_recv[3], d_recv[4],d_recv[5],d_recv[6]);
SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
51.
}
Whether shmem_iput or shmem_iget is faster depends on the stride, but shmem_iput is usually the best choice. For one thing, shmem_iput returns before all the data is delivered to the remote PE, but shmem_iget does not return until the data is delivered to the local PE.
The shmem_iget and shmem_iput functions are faster than the shmem_ixget and shmem_ixput functions, but shmem_ixget and shmem_ixput let you reorder the array. To provide the same functionality, use non-unit strides with the PVM packing and unpacking routine. The relevant lines from a PVM strided copy are as follows:
/* Send data */ bufid = pvm_initsend(PvmDataRaw); istat = pvm_pkdouble(d_send, N, 2); istat = pvm_send(receiver, mtag);
/* Receive data */ bufid = pvm_recv(sender, mtag); istat = pvm_upkdouble(d_recv, N, 3);
For a description of how to efficiently collect data from all PEs and distribute it to all PEs, see Section 3.5, page 67. Or continue on with the remainder of this section for a brief description of the two SHMEM strided data programs.
The data is transferred from within an if statement, beginning on line 38 in the shmem_iget version and line 33 in the shmem_iput version.
The structures of the two if statements are identical in that the sender (PE 0) executes the if clause and the receiver (PE 1) executes the else clause, but the placement of the data transfer functions differs. The shmem_double_iget function, executing on PE 1, retrieves the data from PE 0. The shmem_double_iput function, executing on PE 0, copies the data to PE 1.
shmem_iget version:
44.
shmem_double_iget(d_recv,d_send,3,2,N,SENDER); shmem_iput version:
33.
shmem_double_iput(d_recv, d_send, 3, 2, N, RECEIVER);
65
CRAY T3E C and C++ Optimization Guide
66
As the third and fourth arguments of the function calls specify, both functions take every second array element from the source array ( d_send
) and place them in every third element of the target array ( d_recv
). The arguments to the two functions are the same. See the following figure for an illustration of the transfers:
PE 0
d_send
4
5
6
2
3
0
1
3
4
5
1
2
6
7
PE 1
d_recv
0
0
5
0
3
1
0
4
5
6
2
3
0
1 a10828
Figure 14.
shmem_iget and shmem_iput transfers
As mentioned earlier (see page 57), the following shmem_udcflush call is ignored on CRAY T3E systems. It is left in the shmem_iput program for compatibility reasons.
43.
/* Make sure cache is up to date */
44.
shmem_udcflush();
The output, which is the same for both programs, identifies the PE that received the data and the values of the first seven elements of the d_recv array:
Receiver=1 d_recv=1.
0.
0.
3.
0.
0.
5.
SG–2178 3.0.1
SHMEM [3]
3.5 Gathering and scattering data
The functions shmem_ixget and shmem_ixput copy data (such as arrays) from one PE to another. An index array, specified as an argument, gives the
SHMEM functions the additional capability of reordering the array elements that are being passed. SHMEM is faster than PVM because, in order to provide equivalent functionality, PVM must reorder the elements during the pack (for gather) or unpack (for scatter).
The following example shows the SHMEM version of a reordered scatter operation using shmem_ixput
. It assumes a 2-PE job.
Example 15: shmem_ixput version of a reordered scatter
1.
2.
/* shmem_ixput version
3.
4.
5.
6.
7.
8.
9.
*/
Use the index array for the receiving array (scatter)
#include <stdio.h>
#include <mpp/shmem.h>
#define NN 100
#define N 10
#define SENDER 0
10.
#define RECEIVER 1
11.
12.
double d_recv[NN];
13.
long index[N] = { 99, 19, 28, 91, 82, 37, 73, 46, 64, 55 };
14.
15.
main() {
16.
17.
18.
double d_send[N]; int me, i;
19.
20.
/* Get PE info */
21.
22.
me = _my_pe();
23.
/* Initialize data */
24.
for(i=0; i < N; i++) {
25.
26.
27.
28.
} d_send[i] = i + me + 1.0; for(i=0; i < NN; i++) { d_recv[i] = 0.0;
29.
30.
}
SG–2178 3.0.1
67
CRAY T3E C and C++ Optimization Guide
68
31.
/* Synchronize - Make sure data arrays are initialized */
32.
shmem_barrier_all();
33.
34.
if (me == SENDER) {
35.
/* Send data */
36.
shmem_ixput(d_recv, d_send, index, N, RECEIVER);
37.
38.
/* Synchronize - Make sure data has arrived */
39.
40.
41.
42.
} shmem_barrier_all(); else if (me == RECEIVER) {
43.
/* Synchronize - Make sure data has arrived */
44.
shmem_barrier_all();
45.
46.
/* Make sure cache is up to date */
47.
48.
shmem_udcflush();
49.
/* Print results */
50.
printf("Receiver=%d d_recv= %f %f %f %f %f %f\n",
51.
52.
me, d_recv[99], d_recv[19], d_recv[28],d_recv[91],d_recv[82],d_recv[37]);
53.
54.
}
}
The following lines reorder the data in a PVM version of the same program: for(i=0; i<N; i++) iupk = pvm_upkdouble(d_recv[1+index[I]], 1, 1);
In the following lines of the SHMEM program, the index array (defined in line
13) is referenced in the call to the shmem_ixput function, which reorders the array itself.
13.
long index[N] = { 99, 19, 28, 91, 82, 37, 73, 46, 64, 55 };
36.
shmem_ixput(d_recv, d_send, index, N, RECEIVER);
The output from the SHMEM version on a 2-PE configuration is as follows:
Receiver=1 d_recv=1.
2.
3.
4.
5.
6.
SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
For an illustration of the three arrays involved in the shmem_ixput program, see the following figure.
PE 0
d_send
2
3
0
1
4
5
8
9
6
7
1.
2.
7.
8.
9.
10.
5.
6.
3.
4.
PE 1
d_recv
8.
46
●
●
●
10.
55
●
●
●
9.
64
●
●
●
7.
73
●
●
●
5.
82
●
●
●
4.
91
●
●
●
0.
1.
98
99
0.
0.
0.
●
●
●
2.
19
●
●
●
0
1
2
3.
28
●
●
●
6.
37
●
●
● index
8
9
6
7
4
5
2
3
0
1
99
19
73
46
64
55
28
91
82
37
a10836
Figure 15. Reordering elements during a scatter operation
69
CRAY T3E C and C++ Optimization Guide
A program using the shmem_ixget routine would be almost identical to the preceding shmem_ixput program, except that the
RECEIVER would call the following data transfer statement: shmem_ixget(d_recv, d_send, index, N, SENDER)
3.6 Broadcasting data to multiple PEs
The shmem_broadcast
(3) function copies an array on a single PE to a target array on each of the other PEs. It uses the fan-out (glossary, page 142) method of copying the data, which makes it the most efficient function for copying from one PE to all other PEs. With 32 or more PEs and relatively little data, the fan-out process will enhance program performance. For an illustration of how fan-out operates on an 8-PE configuration, see Figure 9, page 34.
shmem_broadcast gives you significant performance advantages when compared to the pvm_bcast
(3) function.
The following program copies an 8-element array named source on PE 0 to an array named dest on the three other PEs in a 4-PE active set (glossary, page
139). Be sure to use the pSync array for synchronization when you use shmem_broadcast
.
Example 16: One-to-all broadcasting
#include <stdio.h>
#include <mpp/shmem.h> int dest[8], source[8]; long pSync[_SHMEM_BCAST_SYNC_SIZE]; main()
{ int i, j; for (i=0; i < _SHMEM_BCAST_SYNC_SIZE; i++) { pSync[i] = _SHMEM_SYNC_VALUE;
} shmem_barrier_all(); /* Wait for PEs to initialize pSync */
/* Initialize the source array * if (_my_pe() == 0) { printf("The original array values are: "); for (i=0; i < 8; i++) {
70 SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
} source[i] = (i+1) * (i+1); printf(" %d", source[i]);
} printf(" \n");
/* Broadcast an 8-element array from PE 0 */ shmem_broadcast(dest, source, 8, 0, 0, 0, 4, pSync);
/* Print the dest array on all PEs */ for (i=0; i < 4; i++) { if (my_pe() == i) { printf("PE %d has ", i); for (j=0; j < 8; j++) printf("%d", dest[j]);
} /*End of if */
} /* End of for */
} /* End of program */
The output from this program is as follows. Notice that the broadcast did not include the dest array on PE 0.
The original array values are: 1 4 9 16 25 36 49 64
PE 0 has 8*0
PE 1 has 1 4 9 16 25 36 49 64
PE 2 has 1 4 9 16 25 36 49 64
PE 3 has 1 4 9 16 25 36 49 64
The following figure illustrates the contents of the arrays after the shmem_broadcast program has executed.
71
CRAY T3E C and C++ Optimization Guide
0
.
.
.
source
1
.
.
.
7
64
PE 0 dest
0
.
.
.
0
.
.
.
0
7
PE 1
PE 2
PE 3
0
.
.
.
dest
1
.
.
.
7
64
0
.
.
.
7
dest
1
.
.
.
64
0
.
.
.
dest
1
.
.
.
7
64
a10838
Figure 16.
shmem_broadcast operation
3.7 Merging arrays
The shmem_fcollect
(3) function quickly combines blocks of data, such as arrays, from multiple PEs into a single array on all PEs. You can consider it a many-to-many broadcast. The following example merges four copies of the array myvals into a single array, allvals
, which is present on all of the PEs.
As in the previous example, be sure to include the pSync array.
72 SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
Example 17: Merging arrays
#include <stdio.h>
#include <shmem.h> int myvals[4], allvals[16]; long pSync[_SHMEM_BCAST_SYNC_SIZE]; main()
{ int i, j, n; for (i=0; i < _SHMEM_BCAST_SYNC_SIZE; i++) { pSync[i] = _SHMEM_SYNC_VALUE;
}
/* Assume 4 PEs */ npes = _num_pes(); me = _my_pe();
/* The values to be passed will be based on PE numbers */ n = npes * me; for(i=0; i < npes; i++) { n = n + 1; myvals[i] = n;
}
/* Wait until all PEs are initialized */ shmem_barrier_all(); shmem_fcollect(allvals, myvals, 4, 0, 0, npes, pSync);
/* Print allvals on each PE */ printf("PE %d has ", me); for (j=0; j < npes*4; j++) printf("%d", allvals[j];
} /* End of program */
The output from the program is as follows. Notice that, unlike shmem_broadcast
, shmem_fcollect sends its data to all of the PEs, including itself.
73
CRAY T3E C and C++ Optimization Guide
PE 3 has 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
PE 0 has 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
PE 1 has 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
PE 2 has 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
The following figure illustrates the contents of both the myvals and the allvals arrays on each PE after the example is run. A call to pvm_gather followed by a call to pvm_bcast would produce the same result as one call to shmem_fcollect
, but the call to shmem_fcollect is faster.
74 SG–2178 3.0.1
SHMEM [3]
PE 0
PE 1 PE 2
PE 3
2
3
0
1
myvals allvals
1
2
3
4
1
2
0
1
3
4
2
5
3
4
6
7
5
6
8
9
15
16
7
8
10
11
13
14
9
10
12
11
12
13
14
15
2
3
0
1
myvals allvals
5
6
7
8
1
2
0
1
3
2
4
5
3
4
6
7
5
6
8
7
9
8
10
11
9
10
12
11
13
14
12
13
15
14
16
15
0
1
2
3
myvals allvals
9
10
11
12
1
2
0
1
3
2
4
5
3
4
6
5
7
6
8
7
9
8
10
9
11
10
12
11
13
12
14
13
15
14
16
15
2
3
0
1
myvals allvals
13
14
15
16
1
2
0
1
3
4
2
5
3
4
6
7
5
6
8
9
15
16
7
8
10
11
13
14
9
10
12
11
12
13
14
15 a10839
Figure 17. Results of a shmem_fcollect
3.8 Reading and updating in one operation
The shmem_swap
(3), shmem_short_finc
(3), and shmem_short_fadd
(3) functions give you the ability to read and change a remote memory address in a single, atomic operation (glossary, page 139). An atomic operation cannot be interrupted by another operation. No other PE can access the same data location while an atomic operation is accessing it. In the following example, the shmem_short_finc function reads and toggles a value. One possible use of
SG–2178 3.0.1
75
CRAY T3E C and C++ Optimization Guide the shmem_short_finc function is to implement your own locks, as shown in the following example.
Example 18: Remote fetch and increment
5.
6.
7.
8.
1.
2.
3.
4.
9.
10.
11.
12.
13.
14.
15.
}
/* These LOCK/UNLOCK routines implement a take-a-number lock using an array of 2 elements initialized to 0. */ lock(lck) { short myno, now_serving, lck(2); short shmem_short_finc;
/* Take the next number. */ myno = shmem_short_finc(lck(1), 0);
/* Wait until my number comes up. */ shmem_get32(now_serving, lck(2), 1, 0); while(now_serving != myno) shmem_get32(now_serving, lck(2), 1, 0);
Using the shmem_barrier_all function is faster than writing your own synchronization function on CRAY T3E systems, but there are instances in which you may prefer an atomic read and update function. Synchronizing between two PEs is just one example.
3.9 Using reduction functions
The reduction (glossary, page 146) functions combine array elements from each active PE to yield an array of results, which are distributed to all PEs. For instance, one function adds the values at each array location for an array spread across multiple PEs and distributes an array of those sums to each PE. The result is that each PE has an array of answers when the function completes. In its simplest form, a reduction function is a collective (glossary, page 141) operation that involves all PEs.
Once again, the SHMEM reduction functions are usually faster than those of
PVM and MPI because of the difference in overhead, and they are faster than other means of executing operations across PEs (such as shmem_put
, shmem_get
, and standard compiler or library operations).
The reduction routines perform the following operations on arrays:
• Return the smallest value for each array location (see Example 19, page 77)
76 SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
• Return the largest value for each array location
• Return the product of each array location
• Return the sum of each array location (see Example 20, page 79)
• Return the logical OR of each array location
• Return the logical exclusive OR (XOR) of each array location
• Return the logical product (AND) of each array location
The following example finds the smallest value at each position in four arrays, sends those values to arrays on all PEs, and has PE 0 print the values from its array.
Example 19: Minimum value reduction routine
#include <mpp/shmem.h>
#include <stdio.h>
#include <intrinsics.h>
#define NR 4 main() { int me, i; long pSync[SHMEM_REDUCE_SYNC_SIZE]; double foo[NR], foomin[NR], pWrk[SHMEM_REDUCE_MIN_WRKDATA_SIZE]; for(i=0; i < SHMEM_REDUCE_SYNC_SIZE; i++) pSync[i] = SHMEM_REDUCE_SYNC_SIZE*SHMEM_SYNC_VALUE; me = _my_pe(); srand48(); for(i=0; i < 4; i++) { foo[i] = drand48() * 1000.0;
}
/* Print the preliminary numbers on each PE */ printf("At first, PE %d has: ", me); printf("%f %f %f %f\n", foo[0], foo[1], foo[2], foo[3]); shmem_barrier_all();
77
CRAY T3E C and C++ Optimization Guide shmem_double_min_to_all(foomin, foo, NR, 0, 0, 4, pWrk, pSync);
}
/* All the values should be the same */ for(i=0; i<NR; i++) { printf("Result on PE %d : ", me); printf("%f %f %f %f\n", foomin[0], foomin[1], foomin[2], foomin[3]);
}
The output from the program is as follows:
At first, PE 0 has 158.04 422.40 267.21 272.92
At first, PE 3 has 790.22 318.94 692.47 859.08
At first, PE 1 has 474.13 54.58 847.66 801.64
At first, PE 2 has 474.13 686.76 112.02 330.36
Result on PE 0 : 158.04 54.58 112.02 272.92
Result on PE 1 : 158.04 54.58 112.02 272.92
Result on PE 2 : 158.04 54.58 112.02 272.92
Result on PE 3 : 158.04 54.58 112.02 272.92
The following figure illustrates the contents of the two arrays on each PE at the end of the program.
78 SG–2178 3.0.1
SG–2178 3.0.1
SHMEM [3]
2
3
0
1
foo
158.04
422.40
267.21
272.92
PE 0 foomin
158.04
54.58
112.02
272.92
2
3
0
1
foo
474.13
54.58
847.66
801.64
PE 1 foomin
158.04
54.58
112.02
272.92
PE 2
PE 3
0
1
2
3
foo
474.13
686.76
112.02
330.36
foomin
158.04
54.58
112.02
272.92
0
1
2
3
foo
790.22
318.92
692.47
859.08
foomin
158.04
54.58
112.02
272.92
a10840
Figure 18. Results of a shmem_double_min_to_all
In the next example, the summation performed by the ring program is implemented by using the shmem_int_sum_to_all
(3) reduction function.
The reduction is faster than using either shmem_get or shmem_put
, which would pass data 15 times on a 16-PE configuration. It is a valid optimization technique that replaces a slower algorithm with a more efficient algorithm.
Example 20: Summation using a reduction routine
#include <mpp/shmem.h>
#include <stdio>
#include <intrinsics.h>
#define N 100000 main() {
79
CRAY T3E C and C++ Optimization Guide
80
int me, npes, next, prev, istat, i; int total[N], send[N]; double pWrk(max(N/2+1, _SHMEM_REDUCE_MIN_WRKDATA_SIZE)); long pSync[_SHMEM_REDUCE_SYNC_SIZE]; for(i=0; i<_SHMEM_REDUCE_SYNC_SIZE; i++) pSync[i] = _SHMEM_REDUCE_SYNC_SIZE*_SHMEM_SYNC_VALUE;
/* Get PE information */ me = _my_pe(); npes = _num_pes();
/* Initialize data */ for(i=0; i<N; i++) send[i] = me;
/* Synchronize, make sure data is on all PEs */ shmem_barrier_all();
/* Perform the work */ shmem_int_sum_to_all(total, send, N, 0, 0, npes, pWrk, pSync);
} printf("PE = %d Result = %d Expect = %f", me, total[0], (npes - 1) * npes * .5);
The output from running the program on 16 PEs is as follows. As with most programs involving output from multiple PEs, the order in which the PEs finish is random.
PE = 5 Result = 120 Expect = 120.
PE = 12 Result = 120 Expect = 120.
PE = 13 Result = 120 Expect = 120.
PE = 2 Result = 120 Expect = 120.
PE = 11 Result = 120 Expect = 120.
PE = 10 Result = 120 Expect = 120.
PE = 14 Result = 120 Expect = 120.
PE = 8 Result = 120 Expect = 120.
PE = 7 Result = 120 Expect = 120.
PE = 15 Result = 120 Expect = 120.
PE = 6 Result = 120 Expect = 120.
PE = 9 Result = 120 Expect = 120.
SG–2178 3.0.1
SHMEM [3]
PE = 0 Result = 120 Expect = 120.
PE = 3 Result = 120 Expect = 120.
PE = 4 Result = 120 Expect = 120.
PE = 1 Result = 120 Expect = 120.
Comparing this to the PVM, shmem_get
, and shmem_put versions of the ring program, shmem_int_sum_to_all delivers the best performance.
SG–2178 3.0.1
81
Single-PE Optimization [4]
Some of the most significant improvements you can make to your program are not linked to parallelism. They fall into the category of single-PE optimizations.
This chapter describes what you can do to get each processing element (PE) running at as close to peak performance as possible.
This chapter makes frequent reference to the hardware, especially the path between local memory and the functional units. For background information on
CRAY T3E hardware, see Section 1.2.1, page 4.
To identify the parts on your program that take the most time and to get feedback on performance improvements, use a performance analyzer such as pat
(1) or the MPP Apprentice tool. For more information, see Section 1.3, page
21.
This chapter addresses the following optimization topics:
• Unrolling loops (see Section 4.1, page 83).
• Using pipelining for loop optimization (see Section 4.2, page 84).
• Making the best use of cache (see Section 4.3, page 90).
• Optimizing stream buffers, which are key to many of the single-PE optimizations (see Section 4.4, page 93).
• Optimizing division operations (see Section 4.5, page 102).
• Vectorizing some math operations within a loop (see Section 4.6, page 105).
• Bypassing cache (see Section 4.7, page 108).
4.1 Unrolling loops
SG–2178 3.0.1
Loop unrolling is a technique that is beneficial on many computer systems, not just the CRAY T3E system. It can provide the following performance benefits:
• Increasing the basic block size, thus increasing the potential for instruction-level parallelism and covering the latency of memory references.
• Reducing loop overhead, thus potentially increasing the instruction issue rate.
• Eliminating redundant memory references.
83
CRAY T3E C and C++ Optimization Guide
• Increasing merging in the missed address file (MAF) of the EV5 processor.
For an illustration of where the MAF fits in, see Figure 3, page 7, and Figure
4, page 8.
Although the compiler does unroll loops for you, unrolling is not done by default on CRAY T3E systems. You can enable the unrolling of all loops generated by the compiler by including the
-h unroll option on the cc
(1) or
CC
(1) command line. You can direct the compiler to unroll select loops by setting the command line option to
-h nounroll and placing the unroll directive immediately in front of a loop, as follows:
#pragma _CRI unroll for(i=0; i<n; i++) {
You can direct the compiler not to unroll a specific loop by placing the nounroll directive in front of it.
The unroll directive can be applied to any loop of a loop nest, not just the innermost loop. For a loop that is not the innermost loop, the compiler performs a technique called unroll and jam. A loop must meet special criteria, however, to ensure that correct behavior is maintained. In particular, the loop must have no data dependencies (glossary, page 141) across its iterations. Also, the compiler performs unroll and jam only on nests in which each loop (except the innermost) contains only one loop. If these criteria are not met, the compiler does not take the risk of performing the optimization.
4.2 Software pipelining
Software pipelining is an advanced scheduling technique performed by the compiler that overlaps the execution of successive loop iterations in an attempt to optimize utilization of the processor’s scheduled resources (such as floating-point functional units, integer functional units, and cache bandwidth).
Software pipelining applies to innermost for loops, do loops, and while loops, provided the loops contain no conditional code or subroutine calls.
Software pipelining is often effective when used in conjunction with the
-h vector3 option to vectorize intrinsic functions. For information on vectorization, see Section 4.6, page 105.
4.2.1 Optimizing a program with software pipelining
Before you run your program you can select the level of automatic pipelining with a command-line option. Currently, pipelining can only be turned on or off,
84 SG–2178 3.0.1
Single-PE Optimization [4] but more options are planned for future releases. Along with the command-line option, you can provide the compiler with additional information on selected loops with a directive.
4.2.1.1 Selecting the level of pipelining
The
-h pipeline
n option specifies the level of pipelining in effect for the program. The levels are as follows: pipeline0 pipeline1 pipeline2 pipeline3
Disables pipelining (the default).
Enables standard pipelining. Numeric results obtained at this level do not differ from results obtained at the pipeline0 level.
Currently equivalent to pipeline1
.
Currently equivalent to pipeline1
.
Enabling pipelining increases compile times, but execution times are often shorter. Timing your program will tell you if the payoff in faster execution is worth the slower compile time.
4.2.1.2 Using the concurrent and ivdep directives
The concurrent directive lets you provide information about array dependencies. Sometimes the compiler cannot understand ambiguous array references, forcing it to assume dependencies exists (for safety reasons) when there are none. Using the concurrent directive, you can tell the compiler that no dependencies exist. The pipeliner uses that information to more aggressively schedule memory references into preceding iterations.
Using the concurrent directive with the optional safe_distance=
n
argument allows the compiler to assume that no dependencies exist between the current iteration of the loop and n preceding or subsequent iterations.
The directive immediately precedes the loop that benefits from it, as in the following example:
#pragma _CRI concurrent safe_distance=3 for(i=p+1; i<=n; i++) x[i] = a[i] + x[i-p]
The concurrent directive in this example informs the software pipeliner that the relationship p
>3 is true. This allows the pipeliner, for example, to safely
SG–2178 3.0.1
85
CRAY T3E C and C++ Optimization Guide load any of the array references x[i-p]
, x[i-p+1]
, x[i-p+2]
, and x[i-p+3] during the i
-th loop iteration.
The ivdep directive can also be used on the CRAY T3E system to communicate the absence of data dependence to the pipeliner, but the information it provides is more limited than that of the concurrent directive, and thus it does not allow as much optimization. The ivdep directive only indicates the absence of vector dependencies, which are data dependencies between a memory reference and those memory references that lexically precede it in the loop. For more information on the ivdep directive, see Section 4.6.1, page 108.
4.2.2 Identifying loops for pipelining
Theoretically, the software pipeliner is guaranteed to optimize utilization for one of the functional units only when there are no recurrences (glossary, page
146) in the loop. (Updates of induction variables do not count as recurrences.)
This means that parallel loops and vector loops should provide the best candidates for pipelining.
In practice, traditional instruction scheduling will already optimize the use of functional units whenever the loop body contains enough parallel instructions.
But if either of the following cases applies to your loop, it is likely that pipelining will significantly increase the performance of a parallel or vector loop:
• The loop body is not too large. (An approximate definition for large is a loop that translates to more than 64 assembly instructions.) On large loop bodies with many parallel instructions, the pipeliner will exhaust the available registers sooner than the default scheduler.
• The loop body is not memory bound. Since most memory events are difficult to predict at compile time, the pipeliner cannot manage the memory bandwidth resource accurately.
Sometimes unrolling loops, either manually or by letting the compiler do it, results in loops with large bodies and too many parallel instructions. Pipelining such unrolled loops yields minimal, if any, performance improvement.
Because it relies on overlapping loop iterations to increase performance, anything that decreases the amount of overlap makes it harder for the pipeliner to increase performance. One or more recurrences within the loop can fall into that category. However, you can still get significant performance increases if one or more of the following conditions are satisfied:
• The recurrence can be ignored because it applies to iterations that are too distant to be overlapped. In the following example, if p has a value larger
86 SG–2178 3.0.1
Single-PE Optimization [4] than about 3, the loop will be translated into a high-performance software pipeline, provided the compiler knows about the lower bound of p
. That information can be provided by using the concurrent directive.
for(i=p+1; i<=n; i++) x[i] = a[i] + x[i-p];
If p is a constant known at compile time, the compiler will eliminate the load instructions to carry the value across iterations in registers. In that case the concurrent directive is no longer required or useful.
• The loop body contains enough instructions that are not involved in a recurrence cycle. In that case, the loop initiation interval is probably constrained more by functional unit availability than by the recurrence itself.
A typical example might look like the following: for(i=2, i<=n; i++) { x[i] = a[i] +x[i-1]; /* Recurrence */ y[i] = b[i+1] * r +b[i-1] * s /*Unrelated work */ z[i] = z[i] * y[i] * c[i] /*Unrelated work */
}
Combining parallel and vector loops with recurrent loops before pipelining is worthwhile provided the resulting loop body does not grow too large and does not become memory bound.
4.2.3 How pipelining works
The way pipelining attempts to optimize utilization of the scheduled processor resources is best understood through an example: for(i=0; i<n; i++) y[i] = x[i];
When this loop is compiled, it is translated into assembly instructions, which are then block-scheduled. The resulting code appears as pseudo code in the next example, in which each line corresponds to one clock period (CP). The t identifier represents a register.
i = 0; for { t = x[i]; y[i] = t; i = i+1;
SG–2178 3.0.1
87
CRAY T3E C and C++ Optimization Guide
88
if (i==n) exit;
}
Without software pipelining, the processor issues an average of less than one instruction per CP. The execution of successive loop iterations is sequential, as opposed to overlapped. A given iteration does not begin until the previous iteration completes. This loop takes 5 CPs per iteration (assuming hits in data cache, which has a latency of 2 CPs).
However, by overlapping the execution of successive iterations and by creating a new loop body, pipelining produces an average throughput of one iteration every 2 CPs. The initiation interval (the time required to start each iteration) of
2 CPs, along with the fact that every iteration now takes 6 CPs to complete, proves that an overlap of 3 (6
42) has been achieved.
After the transformation, the new loop has parts of multiple iterations executing at the same time (see the following figure), has multiple exits, uses twice as much register space, and reorders the update to the loop induction variable ( i
= i+1;
) relative to its use in the store to y
. But the throughput has increased by a factor of 2.5, and the two integer functional units of the EV5 processor are kept busy within the loop.
i=0; t1=x[i]; i=i+1; for { y[i-1]-t1; if(i==n) exit; t2=x[i]; i=i+1; y[i-1]=t2; if(i==n) exit; t1=x[i]; i=i+1;
}
Figure 19. Overlapped iterations t2=x[i];
Figure 20, page 89, illustrates the overlap of iterations for the following loop: for(i=0; i<n; i++) {
... = a[i] * b[i];
... = c[i] * d[i];
}
SG–2178 3.0.1
SG–2178 3.0.1
Single-PE Optimization [4]
Time
Initiation interval
Iterations
Iteration i
Iteration i + 1 a[i] + b[i] a[i + 1]
*
b[i + 1]
Iteration i + 2
Multiply pipeline is 100% busy c[i]
*
d[i] a[i + 2]
*
b[i + 2] c[i + 1]
*
d[i + 1] c[i + 2]
*
d[i + 2] a11336
Figure 20. Pipelining a loop with multiplications
The multiplication functional unit has nine stages, each requiring 1 CP to complete its task. But stages 0 through 3 of each pipeline are for instruction decode and issue, and bypasses between the pipelines give you latencies less than the total pipeline length once the instruction is issued. A snapshot of the functional unit when it is completely busy (see the following table) shows
89
CRAY T3E C and C++ Optimization Guide multiplication operations from three separate iterations of the loop, each at a different stage.
Table 3. Functional unit c[i+1] * d[i+1] a[i+2] * b[i+2] c[i] * d[i] a[i+1] * b[i+1]
4.3 Optimizing for cache
Like CRAY T3D systems, the CRAY T3E system has an 8-Kbyte primary data cache that is direct mapped (glossary, page 141). But it also has a 96-Kbyte secondary cache that is three-way set associative (glossary, page 147). Although this makes optimizing for cache use less critical on the CRAY T3E system, programming for cache is still an important source of potential performance improvement.
The following section describes how to rearrange array dimensions. For background information on how data cache and secondary cache work, see
Procedure 1, page 9, and the associated figures.
4.3.1 Rearranging array dimensions for cache reuse
You can decrease the execution time for your code by increasing the number of times a piece of data is used while it is resident in cache. One way to increase cache reuse is by making reused array dimensions the fastest moving, or rightmost, dimensions of an array.
The array dimensions in the following example can be rearranged to increase reuse.
Example 21: Unoptimized code
double a[3][3][n], b[3][3][n], c[3][3][n] for(i=0; i<3; i++) { for(k=0; k<3; k++) { for(j=0; j<n; j++) {
90 SG–2178 3.0.1
SG–2178 3.0.1
Single-PE Optimization [4]
}
}
} c[k][i][j] = a[1][i][j] * b[k][1][j]
+ a[2][i][j] * b[k][2][j]
+ a[3][i][j] * b[k][3][j];
One way to detect potential cache reuse in a loop nest is by looking for array references that do not contain all of the loop nest’s loop control variables.
Dimensions that have references without a loop control variable, or that have loop control variables that differ from reference to reference, are generally candidates for reuse.
In the preceding example, the first and second dimensions have reuse potential because they each have references in which the subscript is not a loop control variable. These dimensions should be made the fastest running, rightmost dimensions. The dimension declarations can be changed as follows: double a[n][3][3], b[n][3][3], c[n][3][3];
Rearranging a dimension should be accompanied by changing the order of the nested for loops. This optimization maximizes the number of loop invariant
(glossary, page 144) references in the inner loop. Maximizing the proportion of stride-1 reference streams is also important and often easier to do. In the loop from the preceding example, the number of invariant references is the same for both the k and i loops. Both the k and i loops have more invariant references than the j loop. The i loop gives more stride-1 references, thus the loop nest should be changed in the following way: for(j=0; j<n; j++) { for(k=0; k<3; k++) { for(i=0; i<3; i++) { c[j][k][i] = a[j][1][i] * b[j][k][1]
+ a[j][2][i] * b[j][k][2]
+ a[j][3][i] * b[j][k][3];
}
}
}
For an illustration of how the internal ordering of array a changes as a result of the optimization, see the following figure. The array is now processed in the same order it is stored. The illustration assumes that n has a value of 6.
91
CRAY T3E C and C++ Optimization Guide
92
41
42
43
44
37
38
39
40
33
34
35
36
30
31
32
49
50
51
52
45
46
47
48
53
55
26
27
28
29
22
23
24
25
18
19
20
21
14
15
16
17
10
11
12
13
8
9
6
7
4
5
2
3
0
1 a [2][2][5] a [1][0][3] a [2][0][3] a [0][1][3] a [1][1][3] a [2][1][3] a [0][2][3] a [1][2][3] a [2][2][3] a [0][0][4] a [1][0][4] a [2][0][4] a [0][1][4] a [1][1][4] a [2][1][4] a [0][2][4] a [1][2][4] a [2][2][4] a [0][0][5] a [1][0][5] a [2][0][5] a [0][1][5] a [1][1][5] a [2][1][5] a [0][2][5] a [1][2][5] a [2][2][5] a [0][2][1] a [1][2][1] a [2][2][1] a [0][0][2] a [1][0][2] a [2][0][2] a [0][1][2] a [1][1][2] a [2][1][2] a [0][2][2] a [1][2][2] a [2][2][2] a [0][0][3] a [0][0][0] a [1][0][0] a [2][0][0] a [0][1][0] a [1][1][0] a [2][1][0] a [0][2][0] a [1][2][0] a [2][2][0] a [0][0][1] a [1][0][1] a [2][0][1] a [0][1][1] a [1][1][1] a [2][1][1] a11152
Figure 21. Before and after array a has been optimized
48
49
50
51
44
45
46
47
52
53
55
40
41
42
43
37
38
39
33
34
35
36
29
30
31
32
25
26
27
28
21
22
23
24
17
18
19
20
13
14
15
16
10
11
12
8
9
6
7
4
5
2
3
0
1 a [5][2][2] a [3][1][1] a [4][1][1] a [5][1][1] a [0][2][1] a [1][2][1] a [2][2][1] a [3][2][1] a [4][2][1] a [5][2][1] a [0][0][2] a [1][0][2] a [2][0][2] a [3][0][2] a [4][0][2] a [5][0][2] a [0][1][2] a [1][1][2] a [2][1][2] a [3][1][2] a [4][1][2] a [5][1][2] a [0][2][2] a [1][2][2] a [2][2][2] a [3][2][2] a [4][2][2] a [5][2][2] a [1][2][0] a [2][2][0] a [3][2][0] a [4][2][0] a [5][2][0] a [0][0][1] a [1][0][1] a [2][0][1] a [3][0][1] a [4][0][1] a [5][0][1] a [0][1][1] a [1][1][1] a [2][1][1] a [0][0][0] a [1][0][0] a [2][0][0] a [3][0][0] a [4][0][0] a [5][0][0] a [0][1][0] a [1][1][0] a [2][1][0] a [3][1][0] a [4][1][0] a [5][1][0] a [0][2][0]
Of course, loop interchange can only be done if it does not violate data dependencies. The compiler will usually perform interchange under default
SG–2178 3.0.1
Single-PE Optimization [4] optimization if it detects that doing so is both profitable in terms of performance and does not cause incorrect behavior.
Cache reuse in loops such as the one in the preceding example can be further increased by tiling (glossary, page 149). Tiling involves further stripmining
(glossary, page 148) the inner loops and interchanging loops. This is especially profitable when the fastest-moving dimensions are large and less likely to stay entirely in cache. Tiling is currently not performed by the compiler. You must do it manually. For an example of stripmining, see Example 24, page 95.
4.4 Optimizing for stream buffers
Each CRAY T3E PE has six stream (glossary, page 148) buffers located between secondary cache and local memory (see Figure 3, page 7). When allocated, the stream buffers prefetch data from local memory before it is actually requested, increasing memory bandwidth (glossary, page 139) and decreasing latency
(glossary, page 144). A new stream buffer is allocated when the hardware detects two secondary cache (glossary, page 147) line misses that are consecutive in memory. (For an example, see Procedure 1, page 9.)
If an inner loop contains references that allocate more than six different streams, stream buffer thrashing (glossary, page 149) occurs, and the stream buffers are ineffective. The following programming techniques will let your program make the best use of stream buffers.
• Letting the Cray Research compiler enhance the performance of your program by splitting loops (see the following section).
• Avoiding undesirable side effects from loop splitting (see Section 4.4.2, page
97).
• Getting the most out of a stream by maximizing the inner loop trip count
(see Section 4.4.3, page 98).
• Rearranging the dimensions of an array in order to cut down on the number of streams (see Section 4.4.4, page 99).
• Reducing overhead by grouping statements that use the same stream (see
Section 4.4.5, page 100).
• Enabling or disabling stream buffers to get the most benefit from the hardware (see Section 4.4.6, page 101).
SG–2178 3.0.1
93
CRAY T3E C and C++ Optimization Guide
4.4.1 Splitting loops
94
Splitting an inner loop that allocates more than six stream buffers into a sequence of smaller inner loops that each allocate six or fewer stream buffers is a profitable optimization in many cases. It eliminates stream buffer thrashing and can reduce the execution time of the loop by up to 40%.
Because splitting loops by hand is tedious and will not always be a performance improvement on other systems, the Cray Research C and C++ compilers provide command-line options (
-h split and
-h nosplit
) and a source directives ( split
) to help you split loops. The compiler does not split loops by default, because it can degrade performance in some cases, and the compiler cannot currently detect all such cases. For more information on problems with loop splitting and how to detect them, see Section 4.4.2, page 97.
Place the split compiler directive immediately before the for or while statement of the loop to be split. Good candidates for loop splitting are loops with all of the following characteristics:
• Trip counts are higher than 24.
• Performance is bound by memory bandwidth or latency.
• Most references in the loop cause sequences of consecutive cache lines to be read from memory.
• They have few, if any, if statements. Loops with if statements cannot be split as profitably as those without if statements.
When you place the split directive in front of a loop, you are telling the compiler only that the performance of the loop will profit from splitting. You are not telling it that the loop is safe for splitting. The compiler decides on its own if the loop can be safely split. The compiler splits the loop only if it can be done without changing the results of the computation. Therefore, it will not cause incorrect behavior in codes that conform to the International Standards
Organization (ISO) and the American National Standards Institute standards for
C and C++. Usually, a loop is safe to split under the same conditions that a loop is vectorizable for Cray PVP systems. The compiler splits only inner loops.
The split directive also causes the original loop to be stripmined (glossary, page 148). Stripmining increases the potential for cache hits between the resulting smaller loops. On the negative side, stripmining can reduce the average stream length for a loop nest, thus reducing the effectiveness of the streams. A strip length of 256 represents a good balance between cache reuse and stream effectiveness.
SG–2178 3.0.1
SG–2178 3.0.1
Single-PE Optimization [4]
The following examples show optimizations based on splitting loops:
Example 22: Original loop
#pragma _CRI split for(i=0; i<1000; i++) { a[i] = b[i] * c[i]; t = d[i] * a[i]; e[i] = f[i] + t * g[i]; h[i] = h[i] + e[i];
}
First, the compiler generates the following loops. Notice the expansion of the scalar temporary t into the compiler temporary array ta in the following example.
Example 23: Splitting loops
for(i=0; i<1000; i++) { a[i] = b[i] * c[i]; ta[i] = d[i] * a[i];
} for(i=0; <1000; i++) { e[i] = f[i] + ta[i] * g[i]; h[i] = h[i] + e[i];
}
Finally, in the following example, the compiler stripmines the loops by 256 to increase the potential for cache hits and reduce the size of arrays created for scalar expansion. Stripmining itself does not provide a performance benefit, but in combination with other optimizations (especially loop splitting and vectorization), it can speed up your program.
Example 24: Stripmining
for(i1=0; i1<1000; i1+=256) { if(i1 + 256 < 1000) i2 = i1 + 256; else i2 = 1000; for(i=i1; i<i2; i++) { a[i] = b[i] * c[i]; ta[i-i1+1] = d[i] * a[i];
}
95
CRAY T3E C and C++ Optimization Guide
96
} for(i=i1; i<i2; i++) { e[i] = f[i] + ta[i-i1+1] * g[i]; h[i] = h[i] + e[i];
}
In the following example, the compiler splits a loop that includes an if statement. The result is two loops, each with an if statement.
Example 25: Splitting loops across if statements
#pragma _CRI split for(i=0; i<1000; i++) { if(a[i] < 0.0) b[i] = c[i] * d[i]; else e[i] = f[i] * g[i];
}
The preceding loop is split as follows: for(i=0; i<1000; i++) { l[i] = a[i] < 0.0; if(l[i] != 0) b[i] = c[i] * d[i];
} for(i=0; i<1000; i++) { if(l[i] == 0) e[i] = f[i] * g[i];
}
The compiler does not split up if statements that are nested within other if statements. Nested if statements remain intact at the end of the splitting process.
The compiler also splits individual statements that would allocate more than six stream buffers, as in the following example.
Example 26: Splitting individual statements
#pragma _CRI split for(i=0; i<1000; i++) a[i] = b[i] * c[i] + d[i] * e[i] + f[i] * g[i];
The preceding loop would be split as follows and then stripmined:
SG–2178 3.0.1
Single-PE Optimization [4] for(i=0; i<1000; i++) t[i] = b[i] * c[i] + d[i] * e[i]; for(i=0; i<1000; i++) a[i] = t[i] + f[i] * g[i];
Statements such as those in the preceding example are split only on add, subtract, and multiply operations.
The
-h split command-line option directs the compiler to try to split all loops in a file. The
-h nosplit option, which is the default, splits only loops preceded by the split directive.
Note:
There is potential for increasing the execution time of certain loops by splitting them. Loop splitting also increases compile time, especially when loop unrolling is also enabled. The split compiler directive and the
-h nosplit command-line specification let you select only those loops that will benefit from splitting. Timing a loop both ways can help you determine whether splitting it will enhance performance.
For more information on loop splitting options and directives, see the Cray
C/C++ Reference Manual, publication SR–2179.
4.4.2 Changed behavior from loop splitting
Enabling loop splitting on CRAY T3E systems may cause previously working, but nonstandard, codes to change behavior.
Loop splitting assumes that all arrays are referenced with subscripts that are within the array’s declared bounds. If a code indexes an array with a subscript that is outside of the declared bounds (a nonstandard practice), loop splitting may change the behavior of the code. The change in behavior may result in different numerical results, or it might cause an exception, such as an operand range error or a floating-point exception.
To find the source of the problem, first try to isolate the loop that is causing the change in behavior. Try adding the
-h overindex option to the compiler command line. It disables some assumptions made by the compiler that
overindexing (glossary, page 145) is not done. Depending on the nature of the overindexing, this option may cause the program to behave correctly and still give most of the benefit of loop splitting.
SG–2178 3.0.1
97
CRAY T3E C and C++ Optimization Guide
4.4.3 Maximizing inner loop trip count
Once a stream is allocated, it is usually most advantageous to maximize the number of references that go through the stream in order to recover the startup time and increase performance. Reducing the number of streams also reduces the number of stream startups.
One technique for increasing the references to a stream is to rearrange array dimensions in order to maximize inner loop trip counts. The technique is also used on Cray PVP systems to increase performance through increased vector length. Because split loops are stripmined by 256, 256 is the largest inner loop count possible for loops that are split by the compiler.
Currently, this technique is not performed automatically by the compiler. You must do it manually.
The loop in the following example processes the arrays in streams of 32 elements:
Example 27: Rearranging array dimensions
double x[1000][32], y[1000][32], z[1000][32]; double s[1000][32], t[1000][32], w[1000][32], u[1000][32]; for(j=0; j<n1000; j++) { for(i=0; i<n32; i++) { x[j][i] = y[j][i] + z[j][i]; w[j][i] = x[j][i] * t[j][i] - s[j][i] * u[j][i];
}
}
By switching dimensions, as in the following code, the loop is processed in streams of 256 elements (the stripmine length), which reduces the aggregate stream startup count by 88%: double x[32][1000], y[32][1000], z[32][1000]; double s[32][1000], t[32][1000], w[32][1000], u[32][1000]; for(j=0; j<32; j++) { for(i=0; i<1000; i++) { x[j][i] = y[j][i] + z[j][i]; w[j][i] = x[j][i] * t[j][i] - s[j][i] * u[j][i];
}
}
98 SG–2178 3.0.1
Single-PE Optimization [4]
4.4.4 Minimizing stream count
Minimizing the number of different streams in a loop reduces the amount of loop splitting required.
One technique is to rearrange the dimensions of an array to make short dimensions (usually between 2 and 20 elements) the fastest running (or rightmost) dimension. This is the opposite of the technique described in Section
4.4.3, page 98, and it is only profitable if all the array’s references are unwound along the short dimension and grouped within the loop so as to allocate only one stream.
Currently, this technique is not performed automatically by the compiler. You must do it manually.
Although the loop in the following example makes the best use of vectors on
Cray PVP systems, it will allocate 12 streams on CRAY T3E systems:
Example 28: Minimizing streams
double x[6][1000], y[6][1000]; for(i=0; i<1000; i++) { x[0][i] = y[0][i]; x[1][i] = y[1][i]; x[2][i] = y[2][i]; x[3][i] = y[3][i]; x[4][i] = y[4][i]; x[5][i] = y[5][i];
}
The loop in the following example allocates only two streams on CRAY T3E systems:
Example 29: Reduced streams version
double x[1000][6], y[1000][6]; for(i=0; i<1000; i++) { x[i][0] = y[i][0]; x[i][1] = y[i][1]; x[i][2] = y[i][2]; x[i][3] = y[i][3]; x[i][4] = y[i][4]; x[i][5] = y[i][5];
}
SG–2178 3.0.1
99
CRAY T3E C and C++ Optimization Guide
4.4.5 Grouping statements that use the same streams
Although loop splitting is beneficial to program performance, it does introduce some overhead. The fewer times a loop is split, the less overhead you will have. As the compiler splits loops, it creates new loops by processing statements in the order in which they occur in the original loop. It is beneficial to group statements that use the same streams.
Currently, this technique is not performed automatically by the compiler. You must do it manually.
The compiler would split the loop in the following example into four loops:
Example 30: Original code
double x[1000][2], y[1000][2], z[1000][2]; double s[1000][2], t[1000][2], w[1000][2], u[1000][2]; for(i=0; i<1000; i++) { x[i][0] = y[i][0] + z[i][0]; w[i][0] = x[i][0] * t[i][0] - s[i][0] * u[i][0]; x[i][1] = y[i][1] + z[i][1]; w[i][1] = x[i][1] * t[i][1] - s[i][1] * u[i][1];
}
If the statements using the same streams are grouped together, the loop only needs to be split into two loops, as shown in the following example:
Example 31: Grouping statements within the loop
for(i=0; i<1000; i++) { x[i][0] = y[i][0] + z[i][0]; x[i][1] = y[i][1] + z[i][1]; w[i][0] = x[i][0] * t[i][0] - s[i][0] * u[i][0]; w[i][1] = x[i][1] * t[i][1] - s[i][1] * u[i][1];
}
The situation in the preceding example might be found in codes that contain loops that have been unrolled manually. These loops should either have their statements grouped, or the loops should be rerolled. The following loop, unrolled manually, will be split into four different loops:
Example 32: Loop that will be split into four
double x[1000], y[1000], z[1000]; double s[1000], t[1000], w[1000], u[1000];
100 SG–2178 3.0.1
Single-PE Optimization [4] for(i=0; i<999; i+=2) { x[i] = y[i] + z[i]; w[i] = x[i] * t[i] - s[i] * u[i]; x[i+1] = y[i+1] + z[i+1]; w[i+1] = x[i+1] * t[i+1] - s[i+1] * u[i+1];
}
As in Example 30, page 100, and Example 31, page 100, the first and third lines of the loop access the same streams, as do the second and fourth lines. By grouping the first and third lines and second and fourth lines, the loop in the following example, which was unrolled manually, will be split into two loops.
(Rerolling this loop would probably be best, but the intent of the example is to demonstrate grouping.)
Example 33: Loop that will be split into two
for(i=0; i<999; i+=2) { x[i] = y[i] + z[i]; x[i+1] = y[i+1] + z[i+1]; w[i] = x[i] * t[i] - s[i] * u[i]; w[i+1] = x[i+1] * t[i+1] - s[i+1] * u[i+1];
}
4.4.6 Enabling and disabling stream buffers
There are two levels for the stream buffers, and 1. Level 1 enables stream buffers and is usually the default.
At level 1, stream buffers are allocated when two secondary cache misses on consecutive local memory locations are detected. The allocation occurs only if the second miss occurs within eight memory locations of the first miss. Once a stream is activated, it reads memory in paging mode. This causes memory reads to be done in blocks of four cache lines, optimizing memory bandwidth.
For an example of starting a stream, see Procedure 1, page 9.
At level 0, no stream buffers are allocated. Level 0 may be needed for certain rare loops with references that activate stream buffers but never access them, causing unused memory traffic.
Currently, the stream buffer level is not adjusted for an individual loop by the compiler. You must change the level manually before each loop if you want a setting other than the default of 1. The stream buffer level can be set at run
SG–2178 3.0.1
101
CRAY T3E C and C++ Optimization Guide time by using the set_d_stream
(3) library function. The stream buffer level is changed to 0 in the following example:
#include <mpp/rastream.h> set_d_stream(0);
The get_d_stream
(3) function saves the current stream buffer level for the purpose of restoring it later.
For certain versions of the CRAY T3E system, stream buffers are disabled by default for the following classes of programs:
• A program that calls subroutines from the SHMEM library.
• A program that uses the cache_bypass directive (see Section 4.7, page 108).
The document CRAY T3E Programming with Coherent Memory Streams outlines conditions under which you can safely enable streams for programs in these categories. The document is available online at the following URL: http://www.cray.com/PUBLIC/product-info/sw/PE/T3E_streams.html
Once you ensure that your program is safe, you can enable streams by using the set_d_stream
(3) library function or by setting the
SCACHE_D_STREAMS environment variable to 1.
4.5 Optimizing division operations
The division operation is relatively expensive in terms of performance.
Depending on how many bits need to be generated (that is, 1.0 divided by 2.0
is quicker than 1.0 divided by 3.0), the operation varies between 22 and 60 CPs for a 64-bit divide and between 15 and 31 CPs for a 32-bit divide. Division operations are not pipelined, so a second divide cannot be issued while the first is in progress.
The best strategy for division is to avoid it whenever possible. Changing a division operation into a reciprocal multiplication operation, which the C and
C++ compilers do by default, can improve performance dramatically. In the following example, a
, b
, c
, and d are all cache-resident arrays:
#pragma_CRI unroll for(i=0; i<256; i++) a[i] = (b[i] + 2.0 * c[i] + d[i]) / x;
102 SG–2178 3.0.1
SG–2178 3.0.1
Single-PE Optimization [4]
Because the divide is loop invariant (glossary, page 144), the divide can be changed to a multiply by the reciprocal, as shown in the following example.
xinv = 1.0/x; for(i=0; i<256; i++) a[i] = (b[i] + 2.0 * c[i] + d[i]) * xinv;
By default, the C and C++ compilers change a divide into a reciprocal multiply for you. You do not have to change the code at all. Unless you disable it by specifying the
-h nofastfpdivide option on the cc or
CC command line, the compiler will use the faster reciprocal multiply at every opportunity.
Other operations can proceed when a divide operation is in progress. If moving a divide operation outside of a loop is not possible, you can sometimes preschedule it from within your source code. The following inner loop has a divide operation in line 7 that causes a wait of about 60 CPs, and the result is immediately used.
11.
12.
13.
14.
15.
16.
17.
18.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
19.
20.
21.
22.
23.
24.
Example 34: Original code
for(jn=0; jn<lpr; jn++) { j = iar2[jn + lpair]; ic = ico[iaci_iac[j]]; xw1 = tmp1 - x[1][j]; xw2 = tmp2 - x[2][j]; rwtmp = tmp3 - x[3][j]; r2inv = 1.0/((xw1 * 100) + (xw2 * 100) + (rwtmp * 100));
/* Line 7 is the problem.
The result of the divide is used in the next calculation. */ df2 = cgi * cg[j] * r2inv; eelt = eelt + df2; r6 = r2inv * 1000; f1 = cn12[1][ic] * (r6 * r6); f2 = cn12[2][ic] * r6; enbt = enbt + (f2 - f1); df = (df2 + 6.0 * ((f2 - f1) - f1)) * r2inv; fw1 = xw1 * df; fw2 = xw2 * df; fw3 = rwtmp * df; f[1][j] += fw1; f[2][j] += fw2; f[3][j] += fw3; tmp4 -= fw1;
103
CRAY T3E C and C++ Optimization Guide
104
25.
26.
27.
} tmp5 -= fw2; tmp6 -= fw3;
Using a technique similar to bottom loading (glossary, page 140), the division required for the next iteration of the loop is computed in advance. The divide operation itself is in line 14 of the following example. The result of the divide is not needed until the next pass of the loop, so the floating-point operations following the divide can overlap with the 60 CPs, assuming a 64-bit divide.
This kind of division is unconventional, but it increases the performance of the code. The compiler does not make the following changes automatically.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
9.
10.
11.
12.
13.
14.
15.
5.
6.
7.
8.
1.
2.
3.
4.
Example 35: Modified code
/* First divide computed */ j = iar2[1 + lpair]; xw1 = tmp1 - x[1][j]; xw2 = tmp2 - x[2][j]; rwtmp = tmp3 - x[3][j]; r2inv = 1.0/((xw1 * 100) + (xw2 * 100) + (rwtmp * 100)); for(jn=0; jn<lpr; jn++) {
/* Compute the divide for next pass */ j_next = iar2[(jn + 1) + lpair]; xw1_next = tmp1 - x[1][j_next]; xw2_next = tmp2 - x[2][j_next]; rwtmp_next = tmp3 - x[3][j_next]; rnxt=1.0/((xw1_next*100)+(xw2_next*100)+(rwtmp_next*100)); ic = ico[iaci + iac[j]]; df2 = cgi * cg[j] * r2inv; eelt = eelt + df2; r6 = r2inv * 1000; f1 = cn12[1][ic] * (r6 * r6); f2 = cn12[2][ic] * r6; enbt = enbt + (f2 - f1); df = (df2 + 6.0 * ((f2 - f1) - f1)) * r2inv; fw1 = xw1 * df; fw2 = xw2 * df; fw3 = rwtmp * df; f[1][j] += fw1; f[2][j] += fw2; f[3][j] += fw3;
SG–2178 3.0.1
Single-PE Optimization [4]
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
} tmp4 -= fw1; tmp5 -= fw2; tmp6 -= fw3;
/* Juggle the values for the next pass */ j = j_next; xw1 = xw1_next; xw2 = xw2_next; rwtmp = rwtmp_next;
/*The result of the divide is not needed until here. All the work above this can proceed concurrently with the divide. */ r2inv = rnxt;
The early divide for the last iteration in the preceding example is potentially unsafe because it may go out of bounds. You may have to provide a special case for the last iteration.
4.6 Vectorization
SG–2178 3.0.1
The CRAY T3E compiler offers a method to vectorize select math operations inside loops. This is not the same kind of vectorization available on a Cray PVP systems On a CRAY T3E system, the compiler restructures loops containing scalar operations and generates calls to specially coded vector versions of the underlying math routines. The vector versions are between two and four times faster than the scalar versions. The compiler uses the following process:
1. Stripmine the loop. (For more information on stripmining, see Example 24, page 95.)
2. Split vectorizable operations into separate loops, if necessary. (For more information on loop splitting, see Example 24, page 95.)
3. Replace loops containing vectorizable operations with calls to vectorized intrinsics.
Vectorizing reduces execution time in the following ways:
• By reducing aggregate call overhead, including the subroutine linkage and the latency to bring scalar values into registers needed by the intrinsic routine.
105
CRAY T3E C and C++ Optimization Guide
106
• By improving functional unit utilization. It provides better instruction scheduling by processing a vector of operands rather than a single operand.
• By producing loops that can be pipelined by the software. (For more information on pipelining, see Section 4.2, page 84.)
The programming environment also offers the libmfastv library of faster, but less accurate, vector versions of the libc math functions. These routines deliver results that are generally 1 to 2 bits less accurate than the results given by the libc math functions. Less accurate scalar versions of the library functions are also used to provide identical results between vector and non-vector invocations within the same program. The libmfastv routines reduce execution time spent in math intrinsics by between 50% and 70%.
Because the vector functions may not provide a performance improvement in all cases (due to necessary loop splitting), vectorization is not on by default. It is enabled through the compiler command-line option
-h vector3
. The default is vector2
, which currently does no intrinsic vectorization. The vector1 and vector0 options have their own meanings on Cray PVP systems, but on the
CRAY T3E system, they are the same as vector2
: they turn vectorization off.
If you have selected
-h vector3
, you can further control vectorization using the following:
• The vectorization directive novector
. It lets you turn vectorization off for selected loops in your program.
• In the case of ambiguous data dependences within the loop, you can express a loop’s vectorization potential with the ivdep pragma directive.
ivdep tells the compiler to ignore vector dependencies in the following loop. For more information, see Section 4.6.1, page 108.
• Access to the libmfastv routines can be controlled with the
-l compiler option. For example, the following command line links in the faster, less accurate math routines rather than the slower, default routines in libc
:
% cc -hvector3 -lmfastv test.c
Transformation from scalar to vector is implemented by splitting loops. This may cause extra memory traffic due to the expansion of scalars into arrays and reduce the opportunity for other scalar optimizations. This could negatively impact the profitability of the vectorization.
The less accurate version offered through libmfastv varies from default libc results generally within 1 to 2 bits, although some results could differ by larger amounts. Exceptions may also differ from the libc versions, where some calls
SG–2178 3.0.1
SG–2178 3.0.1
Single-PE Optimization [4] to libmfastv may generate only a NaN for a particular operand rather than an exception, causing exceptions further down the line. A NaN is a value that is not a number but rather a symbolic entity encoded in floating-point format.
Vectorization is only performed on loops that the compiler judges to be vectorizable. This determination is based on perceived data dependencies and the regularity of the loop control. These loops will likely be a significant subset of those seen as vectorizable by the Cray PVP compiler. Vectorization of conditionally executed operators is deferred. Vectorization of loops that contain potentially early exits from the loop is also deferred.
Vectorization will be performed on the following intrinsics and operators. The first set supports both 32-bit and 64-bit floating-point data: sqrt
(3C)
1/sqrt
(replaced by a call to sqrtinv
(3C) log
(3C) exp
(3C) sin
(3C) cos
(3C) coss
(3C) (replaced by a combined call to sin and cos
)
The following support 64-bit floating-point only: pow
(3C) rand
(3c)
The vector intrinsic routines are designed to read an arbitrary number of operands from memory and write their results to memory. They can also handle operands and results that do not have a stride of one.
The compiler stripmines and splits (if necessary) any loop for which intrinsic vectorization is indicated by the programmer. The stripmine factor is currently
256. The loop is stripmined to limit the size of scalar expansion arrays and to decrease the likelihood of cache conflict.
The following example illustrates the kind of loop to which the vectorization optimization can be applied: for(i=0; i<n; i++) a[i] = b[i] * sqrt(c[i] + d[i]);
107
CRAY T3E C and C++ Optimization Guide
4.6.1 Using the ivdep directive
Use the ivdep directive to tell the compiler that the following loop can be vectorized, despite the presence of what might appear to be vector dependencies.
Placing the ivdep directive in front of a loop also enables the compiler to perform the following optimizations:
• Loop splitting.
• Using the vectorizing intrinsic functions listed in Section 4.6, page 105.
• Using the cache_bypass directive (see Section 4.7, page 108).
• Improving pipelining (see Section 4.2, page 84).
Vector dependencies prevent the compiler from analyzing a loop because of the unknown value of variables at the time the program is executed. In the following example, the first statement in the loop may read a value that was written by the second statement in an earlier iteration: for(i=0; i<=n; i++) { x = a[i-k]; a[i] = b[i] * c[i] - w;
}
In this example, there may be a dependence, depending on the value of k
. If k is positive, there is a dependence, because the first statement is trying to access a value written by a later statement. If k is negative, there is no dependence.
The compiler cannot know the value of k ahead of time. It will not vectorize the loop unless you place an ivdep directive immediately before it.
4.7 Bypassing cache
The cache_bypass directive offers some applications a semi-automatic way to speed up memory references. The execution time for some loops can be reduced by up to 45%.
Local memory references prefaced by the cache_bypass directive are routed through E registers rather than through cache. Because E registers offer a finer granularity of access to local memory (see Section 6.1, page 129), they give you higher bandwidth for sparse index array accesses, such as gather/scatter and large stride accesses that do not take advantage of the spatial locality of cache references.
108 SG–2178 3.0.1
SG–2178 3.0.1
Single-PE Optimization [4]
The cache_bypass directive can also be used to initialize large arrays if the contents are not immediately needed in cache, avoiding unnecessary reads into cache and improving the memory bandwidth.
The directive precedes a for
, while
, do while
, or if
...
goto loop and affects all of the named arrays whose base data types are 64 bits. (Support for
32-bit and complex data types is not yet implemented.) In the following loop, arrays a and b will be accessed through E registers rather than through cache:
#pragma _CRI cache_bypass a, b for(i=0; i<n; i++) x[i] = a[i] * b[i];
Note:
The cache_bypass directive makes no guarantees about the state of cache before or after the specified loop. The directive is strictly a performance hint to the compiler. In particular:
• It does not guarantee that the specified variables are not in cache before the loop.
• It does not guarantee that the specified variables are not in cache after the loop.
• It does not invalidate cache.
• It does not affect program results in any way.
Even if you include a cache_bypass directive before a loop, the compiler ignores it if it determines it cannot generate code efficiently. The loop must meet the following requirements before the compiler uses E registers:
• The loop must be an inner loop, if it is nested in another loop.
• The loop must be vectorizable. Use the ivdep directive in cases where ambiguous data dependencies are preventing the loop from vectorizing.
(For more information on the ivdep directive, see Section 4.6.1, page 108.
For more information on vectorization, see Section 4.6, page 105.)
You will probably have to enable loop unrolling to realize the full benefit from this feature. For information on the unrolling command-line option and directive, see Section 4.1, page 83.
The benefit is greater the more random the index stream, however, benefit has been seen from index streams with secondary cache hit rates as high as 50%.
Bypassing cache does generate more code for candidate loops, potentially increasing the compile time slightly. It also increases the latency of memory
109
CRAY T3E C and C++ Optimization Guide
!
references in return for greater bandwidth. Applying the cache_bypass directive may increase the execution time for loops that would otherwise benefit from cache references.
Caution:
For some CRAY T3E installations, this feature causes the stream buffer hardware feature to be disabled by default for the entire application.
Streaming is disabled if the compiler cannot guarantee correctness in the interaction of the stream buffers and the E register operations generated by this feature. Disabling stream buffers can cause considerable performance degradation for other sections of the program. The stream buffer feature can be reenabled using the set_d_stream
(3) library routine. Consult your system administrator to determine if your CRAY T3E installation falls into this category.
For background information on streaming, see Figure 4, page 8, and the example that follows. See the document CRAY T3E Programming with
Coherent Memory Streams for details on how and when streams can be safely reenabled in the presence of E register operations. For the online address of the document, plus information on enabling and disabling stream buffers, see
Section 4.4.6, page 101.
110 SG–2178 3.0.1
Input/Output [5]
Optimizing I/O on the CRAY T3E system is not, for the most part, very different from optimizing I/O on Cray PVP systems. If you are already acquainted with Cray PVP I/O, much of this chapter should be familiar to you.
As on other Cray Research systems, there are a few optimizations that apply regardless of the kind of I/O your program is performing. For instance, using binary (or unformatted) data rather than ASCII data is a good idea that should be used whenever possible. To reduce redundancy, it is not listed as an optimization in every section of this chapter.
The following optimization topics are covered:
• Choosing a strategy for doing I/O in a parallel programming environment
(see Section 5.1, page 111).
• Using unformatted I/O whenever possible (see Section 5.2, page 117).
• Coping with formatted I/O when necessary (see Section 5.3, page 120).
• Making use of the performance-enhancing FFIO layers in your program (see
Section 5.4, page 121 ).
• Optimizing random access I/O (see Section 5.5, page 126).
• Striping a file over disk partitions (see Section 5.6, page 126).
5.1 Strategies for I/O
One of the first questions to answer when optimizing on a parallel system is what your I/O strategy will be. Should you do all of your I/O from a single
PE? Should each PE perform its own I/O? Should you work with a single data file or multiple files?
If you want to write data from multiple PEs to one file, you must choose one of the following methods:
• Have all of the PEs open one or more shared files. A shared file can be on a single disk or striped (glossary, page 142) over many disks. This method requires you to synchronize carefully when you are writing to the file (see
Section 5.1.1, page 112).
SG–2178 3.0.1
111
CRAY T3E C and C++ Optimization Guide
• Have each PE involved in I/O, perhaps all of them, open a separate file.
This method often requires you to divide data up into multiple files before reading and to merge files after writing. You can do the dividing and merging outside the scope of the program. Each file can be read from and written to different disks (see Section 5.1.2, page 114).
• Have one PE do all of the I/O. The PE performing the read shares the data with the rest of the PEs and collects it again before writing the output. This method can be very fast when you make use of disk striping (see Section
5.1.3, page 116).
The following sections describe the performance benefits and detriments of these three methods.
5.1.1 Using a single, shared file
PEs can read the same file at the same time. They can also, if you are careful, write to the same file at the same time. For an illustration of this process, see the following figure.
PE 0
PE 1 PE 2 PE 3
Synchronization mechanism for write
112
Data file
Read
Write a11300
Figure 22. Multiple PEs using a single file
Having many PEs reading from the same file can cause a slowdown due to I/O contention. Reading or writing to a single file is most effective under the following circumstances:
SG–2178 3.0.1
SG–2178 3.0.1
Input/Output [5]
• Your program is doing distributed I/O, which automatically spreads an input file across the memory of multiple PEs. This is a supported and optimized method of doing shared file I/O, and it is the recommended approach. For more information on this method, see Section 5.4.2, page 123.
• Your program is doing random access I/O.
• The number of PEs involved is not greater than the number of I/O channels on your CRAY T3E system. This need not be a restriction unless your program is I/O bound. If your program is not I/O bound, you can overlap computation with I/O. Choosing an asynchronous I/O mechanism, such as either the bufa or cachea
FFIO layer, lets you continue to execute statements in your program while I/O is taking place. For information on asynchronous I/O, see Section 5.4.3, page 125, and Section 5.4.4, page 126.
• Your file is striped over multiple disk drives. For information on disk striping, see Section 5.6, page 126.
If each PE reads its own part of the input file, use an offset into the file based on each PE’s number. If all PEs need the same data, reading from one PE and broadcasting the data to the others might be faster than having each PE read from the same file, especially if a large number of PEs are involved in the job.
The following example demonstrates how multiple PEs can share the job of reading and writing a single random-access file. In the example, the function findnext returns a record number based on the PE that invokes it and the number of PEs participating in the I/O. This example will run fast whether the shared file is on a single disk or striped over many disks.
#include <stdio.h>
#include <stdlib.h>
#include <ffio.h>
#include <fcntl.h> main()
{ int me, i, next, workers, size=100000; int num_elements, fd, j, ret, fret; int array[size]; me = _my_pe(); workers = _num_pes; size = 100000; num_elements = size/workers;
113
CRAY T3E C and C++ Optimization Guide fd = ffopen("datafile", O_RDWR); for(i=0; i<num_elements; i++) { next=findnext(next, me, num_elements); fret = ffseek(fd, next, 0); if (fret < 0) exit(); ret = ffread(fd, array[i], 8); if (ret < 0) abort();
/* Perform calculations */
}
} ret = ffwrite(fd, array[i], 8);
When executing this program, pipe the input and output data files as follows: mpprun -n 10 a.out <infile >outfile
5.1.2 Using multiple files and multiple PEs
Reading from and writing to multiple files may be the easiest and fastest I/O strategy, given a number of PEs less than or equal to the number of data streams on your CRAY T3E system. The I/O library routines, the I/O hardware paths between local memory and disk, and the disk devices themselves can all operate in parallel.
Use conditional ffopen
(3C) functions, as in the following example, to read from and write to four files in a 4-PE program. You may be able to optimize this example further by performing I/O to different file systems located on different GigaRings. See your system administrator for information on how file systems are partitioned across GigaRings.
#include <ffio.h
#include <fcntl.h>
#include <cs_os_def.h> main()
{ int fd, errno; char inbuf[8*4096], outbuf[8*4096]; if(_my_pe() == 0)
114 SG–2178 3.0.1
SG–2178 3.0.1
Input/Output [5] fd=ffopen("file0",O_RDWR|O_RAW); if(_my_pe() == 1) fd=ffopen("file1",O_RDWR|O_RAW); if(_my_pe() == 2) fd=ffopen("file2",O_RDWR|O_RAW); if(_my_pe() == 3) fd=ffopen("file3",O_RDWR|O_RAW); ffread(fd, inbuf, 8*4096, errno, FULL, 0);
/* Work on data */
} ffwrite(fd, outbuf, 8*4096);
In this example, each PE reads from and writes to its own file using the FFIO routines ffread
(3C) and ffwrite
(3C). For more information on FFIO, see
Section 5.4, page 121. The external file for each PE is identified by its own copy of fd
. The number 8*4096 is assumed to be eight disk sectors.
PE 0
PE 1 PE 2 PE 3
File0 File1 File2 File3
Read
Write a11301
Figure 23. Multiple PEs and multiple files
115
CRAY T3E C and C++ Optimization Guide
5.1.3 Using a single PE
When a single PE performs the I/O, you will usually have to share the data with other PEs. For an illustration, see the following figure.
PE 1
PE 0
PE 2
PE 3
116
Data file
Read
Write
An array put routine
An array get routine a11302
Figure 24. I/O to and from a single PE
You can share the data using one of the following methods:
• You can use one of the Message Passing Toolkit (MPT) products to pass the data on. PVM, MPI, and SHMEM all have broadcast routines that pass data to other PEs. If every PE needs all the data, the shmem_broadcast routine is the fastest of the three, but it is not portable to other vendors’ systems.
For examples of shmem_broadcast
, see Section 3.6, page 70. For information on pvm_bcast
(3), see Section 2.9, page 33.
• If each PE only needs part of the data, use get and put functions. Again, the
SHMEM put and get routines are the fastest. For information on using shmem_put64 and shmem_get64
, see Section 3.1, page 46. If portability is a concern, PVM and MPI have put and get functions, but they are slower than SHMEM.
SG–2178 3.0.1
Input/Output [5]
5.2 Unformatted I/O
Because formatted I/O requires data conversion, it will add overhead to your program. Avoid formatted I/O whenever practical.
For example, if you are moving data between machines, you can send the binary (unformatted) version instead of ASCII. You can best convert the data as follows:
• Use the assign
(1) command before running your program to convert to other formats automatically. The following command converts to CRAY T3E format:
% assign -N t3e f:myfile1
• Convert the scanf
(3) and printf
(3) function calls in C and C++, and the cout
, cin
,
<<
, and
>>
C++ functions to unformatted functions. Use the ffread
(3) and ffwrite
(3) functions, along with the assign command before executing your program, as follows:
% cc -X4 myprog.c
% assign -s bin f:outfile
% ./a.out
By default, the system always accesses the next record automatically during a read or a write. That means sequential I/O will be fast. When you are also manipulating unformatted data, you have the potential for very fast data transfers. The following section describes how to get the most out of a good combination.
5.2.1 Sequential, unformatted requests
This section describes how best to optimize I/O when you are reading or writing sequential, unformatted data.
A permanent file exists on an external device, such as a disk. Instead of reading from the disk every time, you can save time by moving your file into memory and reading from there. The file in memory is temporary, since you will probably write it out at some point in the program.
Reading from or writing to a disk involves accessing system calls, which move the data between disk and system buffers. Once data has been read into the system buffer, the I/O library moves it from there to its own set of buffers or into cache, both of which are in the memory of another PE on a CRAY T3E system.
SG–2178 3.0.1
117
CRAY T3E C and C++ Optimization Guide
If the whole file does not fit in memory, you can read in parts of it at a time, possibly getting work done on the current data while waiting for the next chunk to be read. The following figure illustrates the data flow for an array named
A in PE 4.
PE 3 PE 4
PE 5
118
A
Library buffers
Library layer
System layer
System buffer
Library buffers
System buffer
Write
Read a11333
Figure 25. Data paths between disk and an array
To move the data between disk and the system buffers, use the following optimizations:
• Choose system call I/O. System call I/O is specified on an assign command either explicitly, by selecting the system or syscall
FFIO layer, or implicitly; if it is not specified, it is added automatically. (For more information on FFIO, see Section 5.4, page 121.) The following example selects syscall
:
SG–2178 3.0.1
SG–2178 3.0.1
Input/Output [5]
% assign -F syscall
• Make I/O requests that begin and end on disk sector boundaries. Most disk sectors are the same as a block size, 512 words (or 4,096 bytes). Check with your system administrator to make sure of the size of a disk sector on your
CRAY T3E system.
The following command preallocates an area that is 80 512-word blocks in size:
% assign -n 80
• If you are using the read
(2) and write
(2) system calls directly, switch to ffread
(3C) and ffwrite
(3C). Doing so will reduce the number of calls to the system, and performance should be at least as good as using the system calls.
Optimizations such as double buffering (or even triple and quadruple buffering) and disk striping are performed by the operating system. User striping can still gain you performance improvements, but it is more labor intensive than other optimizations described in this chapter. (For an example of user striping, see Section 5.6, page 126.)
By adjusting the arguments to the cachea and bufa layers of FFIO, you can have double buffering done automatically. The following assign command creates two buffers, each 50 blocks in size:
& assign -F bufa:50:2
To optimize the process of moving data between the system buffers and an array in your program, use the following techniques:
• Access the FFIO libraries using the ffread
(3) and ffwrite
(3) functions.
• Take advantage of asynchronous I/O if you can accomplish other work while the I/O is taking place. If sequential, unformatted I/O requests take most of your program’s time, you can probably improve performance by combining computation with the inherent asynchronous capability of I/O.
First, select an asynchronous FFIO layer by running assign commands such as the following before executing your program:
% assign -F cachea:80:7 f:indata
% assign -F bufa:100:12 f:indata
The preceding examples take advantage of library caching and buffering, respectively. The bufa and cachea layers have read-ahead (glossary, page
119
CRAY T3E C and C++ Optimization Guide
146) and write-behind (glossary, page 150) capabilities that can improve performance significantly.
• Use the setvbuf
(3) function to specify the size of the library buffer. The following sets the buffer size to 48 512-word blocks, which is the default: setvbuf (FILE stream-file, *NULL, _IOFBF, 48*512)
• If the file is small enough to fit entirely into the memory of a PE, or if a certain part of the file is heavily accessed, use the memory-resident layer in
FFIO. The memory-resident layer involves less overhead than, for example, the cachea layer. For more information, see Section 5.4.1, page 121.
5.3 Formatted I/O
If using formatted I/O cannot be avoided, you can use some of the optimization techniques described in the following sections to speed it up.
5.3.1 Reduce formatted I/O
You can create incremental speedups to your program by reducing the amount of the formatted I/O as follows:
• If you are not going to use all of the output, print only a sample.
• Do not format intermediate results.
5.3.2 Make large I/O requests
Rather than reading an array an element at a time, read multiple data items in a single statement. This will reduce overhead and speed up your program.
For example, the following codes (the first reads in C and the second writes in
C++) make many small I/O requests by reading an array within a loop, element-by-element: float a[isize]; for(i=0, i<isize; i++) scanf("%f",a[i]); float a[isize]; for(i=0; i<isize;) { myfile << a[i] << ’ ’;
120 SG–2178 3.0.1
Input/Output [5]
}
++i; if(i%5 == 0) myfile << ’/n’;
The following example reads five elements in a single I/O request. It makes
80% fewer I/O calls and helps the program to execute faster: float a[i]; for(i=0; i<isize; i+=5) scanf("%f %f %f %f %f", a[i], a[i+1], a[i+2], a[i+3], a[i+4]);
5.4 FFIO
The bufa and cachea layers of flexible file I/O (FFIO) do asynchronous buffering and caching internally. Those two, along with global
, which distributes a data file across multiple PEs, and mr
, which stores part or all of a data file in the memory of a single PE, are the high performance FFIO layers.
The following sections describe how to improve the performance of your program by using them.
5.4.1 Memory-resident data files
You can keep part or all of a data file in the local memory of the PE performing the I/O by using the memory-resident layer of FFIO. This technique can speed up data access dramatically for a small, frequently accessed file or for a large file with most of the I/O activity occurring at the beginning of the file.
Memory is a more precious commodity on a CRAY T3E PE than on a Cray PVP system. Creating a memory-resident file that is too large could take up memory space necessary to your application. To determine how much space you have for data, you first must know how much memory is available on each PE. A
CRAY T3E system comes with between 64 Mbytes and 2 Gbytes of local memory per PE. If you do not know how much your system has, you can find out by entering the grmview -l command. The following example shows the first part of the output, with several columns that are irrelevant to this subject removed:
SG–2178 3.0.1
121
CRAY T3E C and C++ Optimization Guide
+ APP
+ APP
+ APP
+ APP
+ APP
+ CMD
+ CMD
+ CMD
+ CMD
+ CMD
PE Map: 20 PEs configured
Ap. Size Number Aps.
Type PE min max running limit
+ APP 0 2 10 0 1
+ APP
+ APP
+ APP
+ APP
1
2
3
4
2
2
2
2
10
10
10
10
0
1
1
1
1
1
1
1
5
6
7
8
9
10
11
12
13
14
+ CMD
+ CMD
15
16
+ CMD 17
+ OS 18
+ OS 19
2
2
2
2
2
1
1
1
1
1
1
1
1
0
0
10
10
10
10
10
1
1
1
1
1
1
1
1
0
0
1
1
1
0
0
1
1
1
1
1
4 unlim
5 unlim
2 unlim
2 unlim
2 unlim
3 unlim
2 unlim
1 unlim
0
0
0
0 x y z Clock UsrMem FreMem
0 0 0 300 119 119
1 0 0 300
0 1 0 300
119
119
119
116
1 1 0 300
0 2 0 300
1 2 0 300
0 3 0 300
119
119
119
119
116
116
116
117
1 3 0 300
1 0 1 300
0 1 1 300
1 1 1 300
0 2 1 300
1 2 1 300
0 3 1 300
1 3 1 300
0 0 2 300
1 0 2 300
0 1 2 300
1 1 2 300
0 0 1 300
119
119
119
114
111
113
115
113
115
106
107
95
75
55
50
45
67
0
116
119
119
52
78
57
57
52
The application PEs (
APP under the
Type column) are the ones to look at. The
UsrMem column shows 119 Mbytes available to a user program, meaning each
PE probably has 128 Mbytes of local memory. If the combined size of your data and your executable file do not approach 119 Mbytes, you may be able to move the entire data file into the memory of a single PE. To enable the memory-resident layer of FFIO, enter an assign command such as the following before executing your program. This example allocates 10 512-word blocks (about .4 Mbytes) of memory for the data coming from the file myfile
.
% assign -F mr:10 f:myfile
The data is automatically read into the memory-resident area when the file myfile is opened and written back out when myfile is closed. If the data area proves to be too small, the data file is split automatically between local memory and disk.
122 SG–2178 3.0.1
Input/Output [5]
5.4.2 Distributed I/O
The distributed FFIO layer distributes input data across the PEs involved in a job. One PE makes an I/O request that reads the data and automatically divides it among the local memories of the PEs involved in the job.
When executing your program, a read retrieves the data from the right PE, using the shmem_get
(3) function.You do not need to know where the data is stored or how it is retrieved. The performance is not as fast as using memory-resident ( mr
) data, but it is approximately equivalent to using bufa
. A sample assign command looks as follows:
% assign -F global:5:1 f:filea
This example allocates 50 blocks for each page, with each block capable of containing 512 64-bit words of data. For a 10-PE program, it allocates 1 page for each PE, meaning there are 50 blocks on each of the 10 PEs, for a total of 500 blocks. The distribution for an array of 500 blocks would place the first 50 on the first PE to access a file page, the second 50 on the next PE to access a file page, and so on. The following figure represents the layout of the words of data on each PE. It does not reflect the random order in which the PEs access the file.
SG–2178 3.0.1
123
CRAY T3E C and C++ Optimization Guide
PE 0
Program area
PE 1
PE 2
Data area
PE 3
PE 4
Words
0-25,599
PE 5
25,600-
51,199
PE 6
51,200-
76,799
PE 7
76,800-
102,399
PE 8
102,400-
127,999
PE 9
124
128,000-
153,599
153,600-
179,199
179,200-
204,799
204,800-
230,399
230,400-
249,999 a10996
Figure 26. Data layout for distributed I/O
If you use distributed I/O during an operation in which all of the PEs were involved, each PE would hold data for every other PE. Although this might seem like a confusing arrangement, it is a good use of memory for most applications.
The advantages of using distributed I/O are as follows:
• You get more buffer space without severely impacting the memory of any single PE.
SG–2178 3.0.1
Input/Output [5]
• The data becomes essentially a globally accessible file. You do not have to know on which PE any particular data element is stored.
The following are disadvantages:
• You are using memory that might be needed by a PE.
• PEs might need a large amount of data residing in the memory of other PEs, creating many remote transfers.
5.4.3 Using the cache layer
Using the cache layer is especially productive on random access files when many requests are made for data that has already been read into cache, though it is also effective on sequential data.
Be sure to choose the asynchronous FFIO cache layer ( cachea
) for your I/O.
The following command, executed before you run your program, will set up a cache in the local memory of all PEs involved in the job:
% assign -F cachea:100:40:2 f:indata
This example sets up a cache of 40 pages, each page of which is 100 blocks of
512 64-bit words (51,200 words). If the I/O libraries detect sequential access, they perform either asynchronous read-ahead or asynchronous write-behind.
The third parameter to cachea
, which is
2 in the example, tells the libraries how many pages you want to be read ahead. Setting the third parameter is important to the performance for a program using sequential I/O, since the default is no read ahead.
Using cache is similar to using library buffers (see the following section); both have an asynchronous capability and both are stored in the memory of PEs.
There are differences between the two, however.
Cache contains an indexing system for the data in an active cache. You can choose any indexed data in cache and quickly move it into a register.
A buffer does not have an indexing scheme; it knows only the file position at the top of the buffer. A buffer is designed for sequential access. If you reposition within a buffer, the current buffer is flushed and a new set of data is read from disk.
SG–2178 3.0.1
125
CRAY T3E C and C++ Optimization Guide
5.4.4 Using library buffers
When you are dealing with sequential access data, create large buffers. Set the buffer size to the size of the entire data file if possible, or to a fraction of the data size that keeps transfers to a minimum. Use the assign
(1) statement as follows:
% assign -F bufa:40:1 f:file1
This allocates one buffer of 40 512-word blocks, or about 164 Kbytes. The buffers are allocated during program execution. Each PE opening a file receives a buffer.
5.5 Random access
Random access (also called direct access) does not read or write files in sequential order. Because there is no pattern, the I/O libraries cannot optimize random I/O in the same way as sequential I/O. But there are some things you can do to speed up random access:
• Use binary files that bypass the system cache by entering the assign -l none command before executing your program. Because random access usually involves small reads and writes, going through system cache creates extra overhead. Also, name the array in a cache_align directive to align the array in secondary cache and further speed up processing. If you do have large reads or writes, system cache may benefit your program.
• Do not use formatted or blocked file formats. Converting or unblocking data takes extra time.
• If your data file fits in local memory, consider using the memory-resident layer of FFIO. For more information, see Section 5.4.1, page 121.
• If it is practical, rearrange your data so that you can process it sequentially.
5.6 Striping
126
Striping a file over disk partitions adds a level of parallel processing to the slowest part of I/O: the physical reading data from and writing data to disk.
You can specify automatic striping by entering an assign
(1) statement such as the following before executing your program:
% assign -p 0-3 -n 8400 -q 21 -s u f:mydata
SG–2178 3.0.1
Input/Output [5]
This command stripes over four partitions (0, 1, 2, and 3), putting 21 sectors on each partition.
SG–2178 3.0.1
127
Hardware Access [6]
To provide a convenient and efficient way to run distributed programs, the
UNICOS/mk operating system manages special groups of processes called
application teams (glossary, page 139). An application team is a group of one or more processes, running on one or more PEs, that have capabilities that are extensions to standard UNIX. Application teams provide support for
multithreaded programs (glossary, page 145)onCRAY T3E systems.
The special capabilities of an application team include the following hardware-level operations:
• Communication using E registers, which are a special set of hardware registers that provide the ability to directly read and write to the memory of another process within an application team. See Section 6.1.
• Hardware barrier and eureka synchronization units, which permit very fast barrier and eureka (glossary, page 142) capabilities within an application team. See Section 6.2, page 135.
• Distributed I/O. One process may make an I/O request to or from memory associated with another process in the application team. See Section 6.3, page 137.
• Global stop, exit, and abort processing. One process can terminate the execution of all processes in an application team. See Section 6.4, page 137.
On CRAY T3E systems, application teams provide a user environment based on
UNIX process semantics. On CRAY T3D systems, PEs within an application team had some characteristics of a process (separate address spaces) and some characteristics of threads (every PE in an application team has the same process identification number (PID)). On CRAY T3E systems, threads packages, like
Pthreads and DCE threads, are implemented within a process on a single PE.
The CRAY T3E system does not lend itself to running multiple threads for the same process across multiple PEs.
6.1 Using E registers
E registers provide the following special capabilities, in addition to the buffer space that holds data being transferred to or from a remote processor:
• Memory reads or writes to any process within the application team.
SG–2178 3.0.1
129
CRAY T3E C and C++ Optimization Guide
130
!
• Atomic memory operations, including masked atomic swap, compare-and-swap, fetch-and-increment, and fetch-and-add. These operations can be executed on any data address in any process within the application team.
• Population count and leading zero count operations.
The easiest and most portable method of accessing E registers on CRAY T3E systems is by using SHMEM routines from libsma
. SHMEM routines offer a consistent interface on both Cray MPP systems and Cray PVP systems. (For more information on SHMEM, see Chapter 3, page 45.)
If you want to customize your access to E registers, you may want to perform low-level, direct programming.
Caution:
The use of E registers documented in this section represents the conventions followed by CRAY T3E system software at initial release. The compiler, library, and header file interfaces described in this section are not formally supported and are subject to change at major software release levels.
If you are manipulating low-level CRAY T3E hardware features, watch new software release notices for any changes you need to make to affected user code.
For more information on the interface to E registers, see one of the following hardware publications:
• System Architecture Kit (CRAY T3E System), publication HMK-255-0.
• Commands Quick Reference - CRAY T3E Series Systems, publication
HMQ-151-A
Note:
You must sign a nondisclosure agreement with Cray Research to order either of these manuals.
You can find a convenient source of E register opcodes and bit field definitions in the mpp/mpphw_t3e.h
header file, available in the CrayLibs package. It has both physical and virtual addresses for the E register commands. The physical addresses correspond to the values listed in the Commands booklet, but you must use the virtual addresses in user and library code. The operating system maps the virtual addresses to the physical address when an E register command is issued.
The header file has examples of inline functions that use E registers. Inline functions are also available in the CrayLibs package for CRAY T3E systems.
SG–2178 3.0.1
Hardware Access [6]
6.1.1 Basics
6.1.2 Usage rules
SG–2178 3.0.1
The following sections include an overview of material in the hardware reference documents and add information about related support for E register manipulation in the compilers, loaders, and operating system.
The stq
, ldq
, stl
, and ldl load/store hardware instructions trigger E register operations when accompanied by special addresses. The opcodes and operands for the E register commands are packaged in the following locations:
• The address of the load/store instruction.
• The data argument in the load/store instruction.
• The more operands block (MOBE) of E registers. A bit field in the data argument points to up to 4 E registers.
• The source or destination (SADE) E register, which contains data to be transferred. This E register is specified in a bit field within the address for the load/store instruction.
The operating system, the compilers, and the system libraries also use E registers. You must follow certain conventions when using E registers to avoid breaking the system software. The following conventions are analogous to
Alpha register use conventions that must be observed by assembly language programmers:
• Use context 0. The E register commands allow two contexts. Context 1 is reserved for the operating system. User-level code must use context 0. The context number is packed into the E register command address.
• Select E registers from the correct pool. The mpp/mpphw_t3e.h
header file documents the four pools of E registers. The starting offsets within the E registers of these pools are identified by the following constants:
–
_MPC_E_REG_STRIDE1
–
_MPC_E_REG_STRIDE_LW
–
_MPC_E_REG_SAV_MOBE
–
_MPC_E_REG_SCR_MOBE
–
_MPC_E_REG_SADE
131
CRAY T3E C and C++ Optimization Guide
• Use a compiler directive to indicate E register usage. The following directives allocate E registers:
–
#pragma _CRI uses_eregs
–
#pragma _CRI sade
n and
#pragma _CRI mobe
n
Because future C compiler releases might utilize E registers to perform single-processor optimizations, it would be advantageous for user code to explicitly specify the number of scratch source-and-destination E registers
(SADE) and more operands block E registers (MOBE) needed. Also, the loader sets a special bit in the a.out
(5) header if any of the modules linked into a program use E registers.
• Use
MB and
WMB instructions where needed. The memory barrier (
MB
) and write memory barrier (
WMB
) instructions are used when issuing E register commands. The
MB instruction enforces the ordering of loads and stores, and the
WMB instruction enforces the ordering of stores. Ordering is needed to ensure, for example, that previous stores to set up MOBE or SADE E registers have completed before issuing an E register command through a subsequent stq store instruction. The C and C++ compilers provide the
_memory_barrier() and
_write_memory_barrier() intrinsic functions, which generate
MB and
WMB instructions.
• Use volatile qualifiers to force loads and stores to memory. The C and C++ compilers ensure that a dereference of a volatile pointer always accesses memory. To operate the E registers, set up pseudo addresses in volatile pointers, and then load or store to the pseudo address to trigger the E register operation.
6.1.3 Example of a single-word put
This section contains an example of a C inlined function that stores an int value to an address on a remote processor. A remote update like this is called a put operation. The
_I_shmem_int_p function shown here is present in the
CRAY T3E version of the mpp/shmem.h
header file.
Of course, E register operations can be coded in assembly code as well. The mpp/mpphw_t3e.h
header file can be included in assembler code to import the same helpful E register opcode and bit-field definitions.
132 SG–2178 3.0.1
SG–2178 3.0.1
Hardware Access [6]
Example 36: Inline C function to put an int value to a remote processor’s memory
1.
#include <mpp/mpphw_t3e.h>
2.
3.
static void
4.
_I_shmem_int_p(int *addr, int value, int pe)
5.
{
6.
7.
8.
9.
volatile long * const Ecmd =
(volatile long *)(_PUT(_MPC_E_REG_SADE)); volatile int * const sade =
((volatile int*)_MPC_E_REG_BASE) + _MPC_E_REG_SADE;
10. #pragma _CRI sade 1
11.
12.
*sade = value; /* value to be transferred */
_write_memory_barrier(); /* wait for SADE store */
13.
14.
*Ecmd = /* issue the PUT command */
(_MPC_E_REG_STRIDE1 << _MPC_BS_EDATA_MOBE) |
15.
16.
17.
18.
(
(
(pe) << _MPC_BS_DFLTCENTPE) |
(long)(addr) );
19.
20. }
_memory_barrier( /* PUT must precede load */
/* of EREG_PENDING */
21. #pragma _CRI inline _I_shmem_int_p
For a line-by-line explanation of this example, continue on with this section. To move on to information on how to use the barrier and eureka synchronization units, see Section 6.2, page 135.
Line 1 includes the header file that defines bit fields and opcodes.
1.
#include <mpp/mpphw_t3e.h>
Lines 3 and 4 declare the function to be static so that it can be included in a header file and be declared an inline function. If a function is inlined, there is no need for external linkage. If you do not want to create an inlined function, omit the static qualifier.
3.
static void
4.
_I_shmem_int_p(int *addr, int value, int pe)
The
Ecmd pointer in line 6 is a pseudo-pointer that contains the E register put command code and a pointer to the E register (
_MPC_E_REG_SADE
) that will contain the data to be transferred to the remote processor. The volatile
133
CRAY T3E C and C++ Optimization Guide
134
keyword in the declaration of
Ecmd ensures that the pseudo-store to
*Ecmd that follows will be translated by the compiler into an actual store instruction.
6.
7.
volatile long * const Ecmd =
(volatile long *)(_PUT(_MPC_E_REG_SADE));
The sade pointer in line 8 is assigned the memory-mapped address of the E register that will hold the value being transferred by the put command. Again, it is declared volatile so that the store to
*sade will be translated by the compiler into a store instruction.
8.
9.
volatile int * const sade =
((volatile int*)_MPC_E_REG_BASE) + _MPC_E_REG_SADE;
The
#pragma directive in line 10 tells the C compiler that the user is accessing the first source-and-destination E register (SADE). The compiler will not itself generate code that uses this SADE behind the scenes to do its own optimizations. It will use a different SADE, if needed.
10. #pragma _CRI sade 1
Line 11 stores the value to be transferred into the SADE.
11.
*sade = value; /* value to be transferred */
The
_write_memory_barrier intrinsic function is translated by the compiler into a
WMB instruction, ensuring that the store to the SADE in line 11 precedes the E register command, which is issued in line 13.
12.
_write_memory_barrier(); /* wait for SADE store */
The store to
*Ecmd in lines 13 through 16 is translated by the compiler into a stq instruction, which stores a value that contains encoding for the following:
• Specification of a MOBE (
_MPC_E_REG_STRIDE1
), which is pre-initialized in library program startup processing.
• Specification of a PE number. This is a 0-based index that selects the process within the application team that will receive the data.
• The address on the remote PE to which data will be delivered.
13.
14.
15.
16.
*Ecmd = /* issue the PUT command */
(_MPC_E_REG_STRIDE1 << _MPC_BS_EDATA_MOBE) |
( (pe) << _MPC_BS_DFLTCENTPE) |
( (long)(addr) );
SG–2178 3.0.1
Hardware Access [6]
The
_memory_barrier intrinsic function in line 18 is translated by the compiler into an
MB instruction. An
MB instruction ensures that all prior loads and stores are processed before all loads and stores that follow. In this case, the put command triggered by the prior pseudo-store to
*Ecmd must be processed before any succeeding loads of the
_MPP_EREG_PENDING memory-mapped register (MMR). This MMR is loaded by the barrier routine when waiting for prior remote put commands to complete.
18.
19.
_memory_barrier( /* PUT must precede load */
/* of EREG_PENDING */
The
#pragma _CRI inline directive in line 21 inlines the function just defined. It is placed wherever it is called in the C source file. If you do not want to inline the function, omit this directive.
21. #pragma _CRI inline _I_shmem_int_p
6.2 Using barrier and eureka synchronization units
An application team can access up to 27 barrier/eureka synchronization units
(BESUs). Each BESU can be used to implement a barrier or eureka event flag. A barrier flag changes state after all processes in an application team store to it. A eureka event flag changes state when one or more processes in an application team stores to it.
A barrier flag is most often used in same program, multiple data (SPMD) programs, in which all processes must synchronize around some type of communication. A eureka event flag is most often used to indicate the success of a searching algorithm by one process when all processes have been conducting similar searches in parallel.
The most portable way to use barrier and eureka event flags is to use library routines that interface to the hardware BESUs. Barrier routines to choose from include the following:
• barrier
(3)
• pvm_barrier
(3)
• shmem_barrier_all
(3)
The eureka event routines include the following:
• set_event
(3), callable from Fortran, C, and C++
• wait_event
(3), callable from Fortran, C, and C++
SG–2178 3.0.1
135
CRAY T3E C and C++ Optimization Guide
136
• test_event
(3), callable from Fortran, C, and C++
• clear_event
(3), callable from Fortran, C, and C++
For more information on accessing BESUs, see Barrier and Eureka Synchronization
(CRAY T3E Systems), publication HMM-141-0. (A nondisclosure agreement must be signed with Cray Research before you can obtain this document.) For a convenient source of BESU state codes, see header file mpp/mpphw_t3e.h
.
As is the case with E registers, BESUs are used internally by the libraries. To avoid introducing conflicts with system software, observe the following conventions:
• Use the following compiler directive to allocate BESUs:
#pragma _CRI besu
n
This directive indicates that n BESUs should be allocated for use in the compilation unit. The sum of the BESU counts specified with directives in a program is recorded at link time and placed in the a.out
(5) header. The operating system allocates the specified number to the application team at program startup. As a special case, the operating system does not allocate a
BESU if the BESU count in the a.out
header is 1 and the program uses one
PE.
• Use the besu_alloc() function to obtain the correct memory-mapped address. The synopsis for besu_alloc is as follows:
#include <mpp/besu.h> besu_t *besu_alloc(void);
The besu_alloc function should be called during program execution exactly as many times as there are BESUs allocated by compiler directives.
Furthermore, all PEs should call besu_alloc as a concurrent, collective operation so that all PEs obtain the same memory-mapped address for each
BESU.
The following is an example of a barrier routine that illustrates correct program style for the use of BESUs:
Example 37: Using BESUs in a C function
#include <mpp/mpphw_t3e.h>
#include <mpp/besu.h>
#pragma _CRI besu 1
SG–2178 3.0.1
Hardware Access [6] void mybarrier(void)
{ static int firstcall = 1; static besu_t *besuptr; if (firstcall) { firstcall = 0; besuptr = besu_alloc();
}
/* Arm the barrier, indicating this PE’s arrival */
*besuptr = _MPC_OP_BAR;
/* Wait for all other PEs to arrive at the barrier */ while (*besuptr != _MPC_S_BAR) ;
}
6.3 Using distributed I/O
The listio
(2) system call is extended on CRAY T3E systems to provide the ability to issue read and write I/O requests for memory locations on a different
PE in the application team.
To make a remote listio request, set the
LF_REMOTE bit in the li_flags field and the PE number in the li_remote_vpe field of the listreq structure. The iosw structure, which flags I/O completion, must reside on the
PE issuing the I/O request, not on the remote PE.
For more information on distributed I/O, see Section 5.4.2, page 123.
6.4 Global application team termination
Pure process semantics apply to the following forms of termination processing:
• exit()
•
_exit()
Only the process calling one of the above terminates. The other processes in the application team continue running.
When a process in an application team calls the abort function from C or C++, or when a program receives an exception signal (operand range error and floating-point errors are two examples), the entire application team is terminated. The following library extensions to C and C++ terminate the whole application team:
SG–2178 3.0.1
137
CRAY T3E C and C++ Optimization Guide
• globalexit()
. This C-callable function causes all processes in the application team to go through normal exit processing. This function takes a single int argument to indicate exit status: void globalexit(int status);
• The
C_APTEAM category can be passed to the killm
(2) system call to cause the signal to be sent to all processes in an application team.
138 SG–2178 3.0.1
Glossary [7]
SG–2178 3.0.1
active set
In SHMEM, a set of PEs defined to participate in a collective operation, such as sending a value from one PE to multiple PEs.
See also group for the PVM equivalent, page 143, and
communicator for the MPI equivalent, page 141.
application team
A group of one or more processes, running on one or more PEs, that have capabilities that are extensions to standard UNIX.
Members of a team can work on the same or different parts of a program in parallel.
asynchronous receive
A receive operation that proceeds in parallel with other operations on the receiving PE. The send process must check later to find out if the receive has completed.
atomic operation
An operation that cannot be interrupted.
atomic swap
An atomic read-and-update operation on a remote or local data object. The value read is guaranteed to be the value of the data object at the time of the update.
bandwidth
The amount of data that can be moved from one place to another in a given period of time; it is usually expressed in megabytes per second.
barrier
A location in a program at which all PEs (or tasks) must stop until the final PE arrives. A barrier synchronizes the PEs and prevents situations such as having one PE reading a memory location that does not have the correct data yet.
139
CRAY T3E C and C++ Optimization Guide
140 blocking receive bottom loading
A message receiving operation that waits until a message has arrived. Only after the message is received will the next instruction be executed.
A single-PE optimization technique used on some scalar loops in which operands are prefetched during each loop iteration for use in the next iteration. The operand is available as soon as the first loop instruction executes. A prefetch is performed even during the final loop iteration, before the loop’s final jump test has been performed.
broadcast operation
Sending data from a single PE to all other PEs.
buffer
A block of memory used to store data temporarily before transferring it somewhere else.
cache-aligned data cache coherence
Data stored at the beginning of a cache line. The C and C++ cache_align directive aligns data in cache.
All processors see the same value for any memory location, regardless of which cache the actual data is in, or which processor most recently changed the data. On the CRAY T3E system, only local memory references can be cached (all remote memory references use external E registers). Hardware on each
CRAY T3E processor maintains cache coherence with the local memory, including when data is modified by a remote processor.
cache hit
A memory reference to a data object already in primary or secondary cache. Such references are closer and faster than references to data objects in local memory.
SG–2178 3.0.1
SG–2178 3.0.1
Glossary [7]
clock period
The time it takes an instruction to complete (sometimes called the cycle time) on the EV5 microprocessor. This is the smallest measurable unit of time used by the computer hardware. The length of a clock period varies from machine to machine. On the CRAY T3E system, the clock period is 3.3 nanoseconds (or
3.3 billionths of a second), and on a CRAY T3E-900 system, the clock period is 2.2 nanoseconds.
collective routine
A SHMEM routine that must be called by all PEs simultaneously. Such a routine requires the cooperation of all participating PEs. See also individual routine, page 143.
commit
In MPI, saving the formal description of a newly defined data type so that it can be used again.
communicator
In MPI, a group of processes that can send messages to each other. See also group, page 143, for the PVM equivalent and
active set, page 139, for the SHMEM equivalent.
data cache
In each PE, a high-speed, random-access memory that temporarily stores frequently or recently accessed data. For an illustration showing where data cache fits in, see Figure 4, page
8. See also secondary cache, page 147.
dependency
When data from one section of code relies on a value from an earlier section of code.
direct-mapped cache
Method of relating areas of local memory to areas of data cache.
Each line in memory is mapped to a bucket, which is a specific area in data cache. If a line in local memory is read, the data is moved into its bucket in data cache. Each bucket holds just one line. See also set-associative cache, page 147.
141
CRAY T3E C and C++ Optimization Guide
disk mirroring
A logical disk device composed of two or more physical disk slices. Mirroring is used to provide data redundancy when data integrity is important. It is implemented by using two to eight slices, usually on as many different physical disks, each of the same size. A write operation to a mirrored device causes separate write operations to be performed on each of the components. A read operation can be performed on any of the component devices.
disk striping
Splitting a disk file across two or more disk drives to enhance
I/O performance. The performance gain is a function of the number of drives and channels used.
envelope
In MPI, message information used to identify and selectively receive messages. The four parts of the envelope are as follows:
• The rank of the receiver
• The rank of the sender
• A message tag
• A communicator
eureka
Hardware search mechanism. A eureka is like a barrier. When all of the PEs are searching for something, the one that finds it posts an eureka that is visible to all of the other PEs. The posting of the eureka stops the search.
fan-out
An optimized method of passing messages when a single PE is sending a message to multiple PEs. Rather than the sending PE sending the message to every PE, it sends the message to a subset of the PEs. Then the PEs that received the message in turn send it to a subset of the remaining PEs, and so on. See
Figure 9, page 34, for an illustration of the process.
142 SG–2178 3.0.1
SG–2178 3.0.1
Glossary [7]
flushing cache
Clearing cache of its contents and storing the data of a write-back cache, such as data cache. On CRAY T3E systems, cache is automatically flushed.
gather operation
Collecting arrays from multiple remote PEs into a single array on the local PE. Also, collecting scattered elements of one array on one PE to consecutive elements on another PE. See also
scatter operation, page 147.
GigaRing
The networking protocol that connects the CRAY T3E system to other resources (such as I/O devices and other computer systems). For an example of a GigaRing, see Figure 8, page 16.
group
In PVM, a defined set of tasks (or PEs) that participate in the same synchronization and communication processes. A group can either be all the tasks defined for a job or a user-defined subset of all the tasks. See also active set for the SHMEM equivalent, page 139, and communicator for the MPI equivalent, page 141.
GSEG
A global segment is used by the operating system to map a remote virtual address to a remote physical address.
individual routine
A SHMEM routine that must be called by a single PE, regardless of how many PEs are involved in the same operation. See also collective routine, page 141.
instance number
In PVM, if you specify the global group, the instance number is the same as the PE number. If you specify a group other than the global group, you can find the instance number from the pvm_getinst
(3) function or from the return value of the pvm_joingroup
(3) function.
143
CRAY T3E C and C++ Optimization Guide
144 latency of memory
The startup time. The period of time between when a PE requests data and when it can use the data.
line
A division of cache memory. Every location in local memory is mapped to an area of both data cache and secondary cache.
This area in data cache contains one line, which is 4 64-bit words long. Each area in secondary cache contains three lines, each of which are 8 words long.
local memory
The memory available to a microprocessor on its own PE.
Although any PE can access the memory of any other PE, the most efficient method is always a PE accessing its own memory.
In some publications, memory is also called DRAM (for direct random access memory).
loop invariant
A value that does not change between iterations of a loop. In the following loop,
2.0
, val
, c[j]
, and j are loop invariant: for (i=0; i<n; i+) a[i] = b[i] * 2.0 + val * c[j];
message-passing system
A software system that transfers explicit messages between PEs.
The messages often contain data, such as array elements.
MFLOPS (or megaflops)
A timing measurement indicating how many millions of floating-point operations execute each second. A loop, for instance, that runs at 24 MFLOPS executes 24 million floating-point operations each second.
MPI, or Message Passing Interface
A message passing library that conforms to the Message
Passing Interface Standard. It offers portability to other Cray
SG–2178 3.0.1
SG–2178 3.0.1
Glossary [7]
Research computer systems and to computer systems developed by other vendors.
multicast operation
In PVM, the multicast operation sends a message to each PE that has its task identifier number in an array.
multithreaded program
A program that is executed by multiple threads in parallel.
NaN
A value that is not a number but rather a symbolic entity encoded in floating-point format.
nanosecond
A measurement of time equal to one billionth of a second. A clock period on a CRAY T3E system running at 300 megahertz is equal to 3.3 nanoseconds.
nonblocking receive
A message receiving operation that will receive a message if one is present but will return immediately if one is not present.
overhead
In the context of this publication, time spent doing something other than the actual work of a program. For instance, the time required to move data, as opposed to the time spent processing data, is viewed as overhead.
overindexing
When a program attempts to access an array element that is outside the declared bounds of that array.
pad array
An unused array that aligns arrays containing data in cache in an optimal way.
145
CRAY T3E C and C++ Optimization Guide
146
PEs, or processing elements
The microprocessors that execute code on the CRAY T3E system.
The CRAY T3E system can have up to 2,048 PEs configured.
process protocol
A standardized set of rules for transmitting data that allows communication between various entities.
PVM, or Parallel Virtual Machine
A message-passing library used when programming Cray MPP systems, Cray PVP systems, or certain other vendors’ products.
The principle virtues of PVM are portability and flexibility.
rank
In MPI, an independent, parallel code that runs on a PE. It is equivalent to a PVM task. See also task, page 148.
In MPI, the number identifying a process. This is equivalent to a PE number.
read ahead
Reading data from local memory before it is needed. First, a block of memory is read from local memory and sent to the microprocessor. Then, the following block in memory is read, anticipating that it will be needed next. See also, write behind, page 150.
recurrence
A computation in a loop in which the values produced in the current iteration depend on values produced in previous iterations. A typical example is: for (i=1; i<n; i++) x[i] = x[i-1] + a[i];
reduction
A loop or operation that reduces an array to a scalar value by doing a cumulative operation on many of the array elements.
SG–2178 3.0.1
SG–2178 3.0.1
Glossary [7]
For example, a summation is a reduction that adds all the elements of an array together to yield one number.
scatter operation secondary cache
Distributing data from a single array on a single PE to multiple arrays on multiple PEs. Also, distributing consecutive data from one PE to nonconsecutive elements on another PE. See also gather operation, page 143.
A 96-Kbyte data cache, located between local memory and data cache, that optimizes memory access. See also set-associative
cache, page 147, and data cache, page 141.
set-associative cache
A method of associating locations in local memory with locations in secondary cache. Each location in local memory is associated with a 3-line area of secondary cache. When a line is moved from local memory to secondary cache, it is moved into one of the three lines. CRAY T3E systems use a three-way associative cache, meaning each area in secondary cache can hold three lines at a time. See also direct-mapped cache, page 141.
SHMEM
A library of optimized subroutines that take advantage of shared memory. The most frequently used routines move data between the memory of a remote PE and the local PE, but a number of other facilities are available. The routines can either be used by themselves or in conjunction with another programming style, such as PVM. The principle virtue of
SHMEM is its performance.
SM pool
In PVM, the shared memory (SM) pool lets a receiving PE continue to compute while receiving data.
software pipelining
A loop scheduling technique in which the execution of successive loop iterations are overlapped. Overlapping loop iterations exposes more instruction-level parallelism to the
147
CRAY T3E C and C++ Optimization Guide
148
processor, usually resulting in 100% utilization of one of its scheduled resources (such as functional units, cache bandwidth or memory bandwidth).
stream
On CRAY T3E systems, a stream is a series of data items between memory and the functional units of a PE. Similar to a pipeline on Cray PVP systems, a stream feeds data to the functional units in optimal fashion. For more information on streams, see Section 1.2.1, page 4. For information on how to make use of streams in your program, see Chapter 4, page 83.
stride
A term derived from the concept of walking through the data, from one location to the next. For instance, if every other element of an array were to be transferred, the stride through the array would be two.
stripmining symmetric synchronization
A data object is said to be symmetric if it has an address mapping across PEs that allows remote memory access. On
CRAY T3E systems, a data object is symmetric if it has the same address on all PEs.
Timing the actions of PEs to avoid problems. For instance, the barrier function might be used to prevent one PE from accessing a data location before another PE has updated that location.
task
A single-PE optimization technique in which an array is broken down into chunks of convenient size (on the CRAY T3E system, the size of the cache) to allow optimal use of computational hardware.
In PVM, an independent, parallel process that executes on a PE.
There is a one-to-one relationship between a task and a PE. See also process, page 146, for the MPI equivalent.
SG–2178 3.0.1
SG–2178 3.0.1
Glossary [7]
thrashing thread tiling
A phenomenon that occurs when you have a fixed quantity of a reusable resource, the resource is allocated on a least-recently-used basis, and the cycle length of the reuse is larger than the quantity of the resource. A common example of thrashing involves pages in a paging operating system. Assume that there is only room for two pages in main memory, allocated on a least-recently-used basis, but a loop is referencing three arrays, each on a different page. The following code fragment suggests the situation: for (i=0; i<n; i++) {
... = a[i];
... = b[i];
... = c[i];
}
After the references to a[0] and b[0]
, the page for a and the page for b are in memory. When c[0] is referenced, the operating system removes a
’s page, because it has room for only two pages, and a
’s page was least recently used. Next a[1] is referenced, but a
’s page is now out of memory, so the operating system removes b
’s page. Likewise, the reference to b[1] ends up removing c
’s page. Because more pages are referenced than there are room for in memory, because they are referenced cyclically, and because they are allocated in a least-recently-used basis, reuse never occurs, and the paging mechanism gives no benefit.
The activity entity of execution. A sequence of instructions together with machine context (CPU registers) and a stack.
An optimization technique that combines stripmining and loop interchange. It operates on inner loops to increase cache reuse.
Tiling is not done automatically by the compiler. See also,
stripmining page 148.
149
CRAY T3E C and C++ Optimization Guide
ulp
Unit of least precision. It is used to discuss the accuracy of floating-point data. It represents the minimum step between representable values near a desired number: the true result to infinite precision, assuming the argument is exact. For instance, the ulp of 1.977436582E+22 is 1.0E+13, since the least significant
(leftmost) digit of the mantissa is in the 10
13 place. Within 0.5
ulp is the best approximation representable.
unrolling virtual address
A single-PE optimization technique that lets the compiler exploit parallelism at the functional unit level and take maximum advantage of data in cache.
A normal user address that starts at 0 or some other consistent value in every program. The hardware translates a virtual address to a physical address, which is the true location in a machine’s memory.
word
A data item that is 64 bits, or 8 bytes, long on Cray Research systems.
write behind
An I/O operation in which a block of data is written to disk, and the next block of data is buffered, anticipating that it will be written sequentially to disk next. See also read ahead, page 146.
write buffer
A 6-entry buffer through which a write operation passes on its way to, first, secondary cache, and eventually to local memory.
write through
When a data item is being written from a microprocessor to local memory, it makes a brief stopover in data cache. No memory is allocated in data cache for a write-through operation.
150 SG–2178 3.0.1
32–bit data
SHMEM, 59
A
active set definition, 139 add operation and splitting loops, 97 advantages of unrolling, 83 aggressiveness levels, 101
Apprentice timing tool, 21 array how elements are stored, 91 operations on, SHMEM, 76 rearranging dimensions, 90, 98, 99 arrays merging with SHMEM, 72 asynchronous receive definition, 139 asynchronous send, SHMEM, 46
ATM networks, 19 atomic operation definition, 139 atomic swap definition, 139 atomic swap, SHMEM, 75 autoloaders, 18
B
background
Parallel Virtual Machine (PVM), 2 programming styles, 2
SHMEM, 2 topics, 1 background information, 1
SG–2178 3.0.1
Index
bandwidth data and secondary cache, 19 definition, 139 barrier definition, 139 example, SHMEM, 50 in SHMEM program, 55 barriers avoiding in PVM, 32 benefits of unrolling, 83 block multiplexer tape drives, 18 blocking receive definition, 140 blocking receive, PVM, 32 bottom loading definition, 140 broadcast operation definition, 140 broadcast vs. multicast, PVM, 33 broadcast, SHMEM, 70 buffer definition, 140 buffering, double, 119 buffers setting size for sequential I/O, 126 buffers, send allocating in PVM, 25 buffers, stream, optimizing, 93
C
cache coherence, 5 data and secondary, 5 how it works, 9 load and store timings, 19 miss, 9, 11
151
CRAY T3E C and C++ Optimization Guide optimization, 90 reuse, 90 cache coherence definition, 140 cache hit definition, 140 cache layer, FFIO with random access I/O, 125 cache, system bypassing, 126 cache-aligned data definition, 140 cache_align directive, 5 cache_align directive, SHMEM, 60 channels feature not available, 23 clock period definition, 141 clock, real time, 21 collective routine definition, 141 commit definition, 141 communicator definition, 141 compile time increases with loop splitting, 97 concurrent directive, 85 conditional OPEN statements, 114 converting data, 117 converting data, PVM, 26
D
dangers of loop splitting, 97 data
32–bit in SHMEM, 59
32–bit optimization, PVM, 26
32–bit packing, PVM, 26 broadcast with SHMEM, 70 conversion, 117 dependencies, 92 gather/scatter, PVM, 40
152
gather/scatter, SHMEM, 67 initializing and packing, PVM, 30 movement, 9 reuse, 26 streams, 8 stride-1, PVM, 28 strided, SHMEM, 62 transfer, SHMEM, 46 data cache, 5 definition, 141 data conversion, PVM, 26 data flow, 7 data transfer comparing PVM and SHMEM, 3 differences, to PVM on CRAY T3D systems, 23 dimensions, rearranging, 98, 99 direct access I/O, 126 direct-mapped cache definition, 141 directive pipelining, 85 disk mirroring definition, 142 disk sector boundaries for I/O requests, 119 disk striping automatic, 126 definition, 142 disk support, 16 distributed I/O, 123 division operations, 102 division outside of loop, 103 division, how to avoid it, 102 division, IEEE, 103 double buffering, 119
DRAM, same as local memory, 7
E
E registers, 4 envelope
SG–2178 3.0.1
definition, 142
ESCON tapes, 18
EtherNet, 18 eureka definition, 142
F
fan-out definition, 142 fan-out distribution, PVM, 34
FDDI network, 18
FFIO description, 121
Fiber Channel disks, 16
Flexible File I/O description, 121 flow of data, 7 flushing cache definition, 143 formatted I/O optimizations, 120 reducing, 120 functional unit and pipelining, 89 functional units, 8
G
gather data, SHMEM, 67 gather operation definition, 143 gathering data, PVM, 40 get_d_stream routine, 102
GigaRing definition, 143
GigaRing network, 15 global I/O, 123 glossary description, 1 grmview example, 121
SG–2178 3.0.1
Index group definition, 143 grouping statements, 100
GSEG definition, 143
H
hardware illustration, 7, 8 hardware overview, 4
HIPPI disks, 17
HIPPI network, 19
I
I/O, 111
I/O from a single PE, 116
I/O requests using large, 120
I/O strategies, 111
IEEE division, 103 if statement, splitting loop with, 96 individual routine definition, 143 initializing data, PVM, 30 inner loop trip count, maximizing, 98 instance number, 143 intrinsic routines vectorized, 107 intrinsics.h include file, 49 invariant references, maximizing, 91
IPI-2 disks, 17 ivdep directive and pipelining, 86
L
large transfers
153
CRAY T3E C and C++ Optimization Guide how handled by PVM, 24 latencies data and secondary cache, 19 latency, memory definition, 144 line definition, 144 local memory checking your system for, 121 definition, 144 logical PE number, 21 loop overlapping examples, 87 loop invariant definition, 144 loop iterations overlapping, 84 loop splitting can change program, 97 examples, 95 loops identifying for pipelining, 86 loops, splitting, 94 loops, unrolling, 83
M
memory checking your system for, 121 performance information, 19 memory overview, 4 memory-resident FFIO layer with random access I/O, 126 memory-resident I/O description, 121 merging arrays, SHMEM, 72
Message Passing Interface definition, 144 message size finding and changing with PVM, 24 message size, PVM, 24
154
message-passing system definition, 144
PVM, 2
Mflops, or megaflops definition, 144 microprocessor, description, 15 mirroring, disk, 16 mixing send and receive routines, PVM, 30
MPI definition, 144
MPI, switching to SHMEM, 51 multicast operation definition, 145 multicast vs. broadcast, PVM, 33 multiple file, multiple PE I/O, 114 multiply operation and splitting loops, 97
_my_pe intrinsic function, 50
N
NaN definition, 145 nanosecond definition, 145 network illustration, 16 network overview, 15 network protocols, 18 nonblocking receive definition, 145 nonblocking receive, PVM, 31 nonstandard, SHMEM, 45 nosplit directive, 94
_num_pes intrinsic function, 50
O
organization of manual, 1 output stream, 13 overhead
SG–2178 3.0.1
by grouping statements, 100 definition, 145
SHMEM, 46 overindex command-line option, 97 overindexing dangerous with loop splitting, 97 definition, 145 overlapping loop iterations, 84
P
packing data, PVM, 30 pad array definition, 145
Parallel Virtual Machine (PVM), 23
PE, 15 logical number, 21 physical number, 21 performance
SHMEM, 2 peripherals overview, 15
PEs definition, 146 physical PE number, 21 pipelined, division is not, 102 pipelining, 84 command-line options, 85 definition, 147 directive, 85 how it works, 87 portability
PVM, 2
SHMEM, 2 portability, SHMEM, 45 powers of 2, 49 prescheduling division, 103 process definition, 146 processing element (PE), 15 processing elements definition, 146 programming styles
SG–2178 3.0.1
Index background, 2 protocol definition, 146
PVM, 23 background, 2
PVM, or Parallel Virtual Machine definition, 146 pvm_bcast function, 33 pvm_bufinfo routine, 38 pvm_bufinfo(3), 28
PVM_DATA_MAX environment variable, 24 pvm_gather function, 40 pvm_getopt, use of, 24 pvm_initsend routine, 25 pvm_mcast function, 33, 34 pvm_nrecv routine, 31 pvm_precv routine, 28 pvm_psend routine, 28 pvm_reduce, 38 pvm_scatter function, 40
PvmDataDefault, 25
PvmDataInPlace, 25
PvmDataRaw, 25, 28
R
random access I/O, 126 rank definition, 146 read ahead definition, 146 real-time clock, 21 rearranging array dimensions, 90 receiving stride-1 data, PVM, 28 reciprocal multiplication, 102 reduction definition, 146 reduction routines, SHMEM, 76 reductions, PVM, 38 remote memory, 4 reorder an array, SHMEM, 67
155
CRAY T3E C and C++ Optimization Guide reusing data, 26
_rtc timing tool, 21
S
scatter data, SHMEM, 67 scatter operation definition, 147 scattering data, PVM, 40
SCSI disks, 16
SCSI tape drives, 17 search, eureka, 142 secondary cache, 5 definition, 147 send buffers allocating in PVM, 25 sending stride-1 data, PVM, 28 sequential I/O, 117 set-associative cache definition, 147 set_d_stream function, 102 shared memory (SHMEM), 45
SHMEM, 45 background, 2 definition, 147 shmem.h include file, 49 shmem_broadcast function, 70 shmem_double_min_to_all function, 77 shmem_fcollect function, 72 shmem_get function, 46 shmem_get32 function, 59 shmem_iget function, 62 shmem_int_sum_to_all function, 79 shmem_iput function, 62 shmem_ixget function, 67 shmem_ixput function, 67 shmem_my_pe function, 50 shmem_n_pes function, 50 shmem_put function, 46 shmem_put32 function, 59 shmem_short_fadd function, 75 shmem_short_finc function, 75
156
shmem_swap function, 75 single file, multiple PE I/O, 112 single-PE optimizations, 83 size of memory, 4 size of message finding and changing with PVM, 24 size of message, PVM, 24
SM pool definition, 147 software pipelining, 84 definition, 147 split command-line option, 94 split directive, 94 splitting loop with if statement, 96 splitting loops, 94 statistics memory performance, 19 strategies for I/O, 111 stream definition, 148 reducing startup costs, 98 stream buffers setting aggressiveness level, 101 stream buffers, optimizing, 93 stream references, maximizing, 98 streams example of, 8 reducing number, 99 stride definition, 148 stride-1 streams, maximizing, 91 stride-1 data, PVM, 28 strided data, SHMEM, 62 striping, disk, 16 automatic, 126 stripmining definition, 148 stripmining example, 95 subtract operation and splitting loops, 97 swap, atomic, SHMEM, 75
SG–2178 3.0.1
symmetric definition, 148 synchronization avoiding in PVM, 32 definition, 148 minimizing with receive, PVM, 37
PSYNC array, SHMEM, 70
PVM, 2, 32
SHMEM, 3 synchronization, SHMEM, 55 system cache bypassing, 126 system call I/O, 118
T
tapes supported, 17 task definition, 148 task identifier, PVM, 37
TCP/IP network, 19 thrashing definition, 149 tiling definition, 149 time to compile increases with loop splitting, 97 timing your code, 21 timings memory operations, 19 transfer
32–bit data, PVM, 26 of data, SHMEM, 46 transfer of data comparing PVM and SHMEM, 3 transfer, large how handled by PVM, 24
U
ulp definition, 150 unformatted I/O, 117 unroll directive, 84 unrolling by the compiler, 84 command-line option, 84 definition, 150 outer loops, 84 unrolling loops, 83
V
vectorization, 105
W
work while waiting,
PVM, 31 write behind definition, 150
Index
SG–2178 3.0.1
157
ডাউনলোড
পাবলিক লিঙ্ক আপডেট করা হয়েছে
আপনার চ্যাটের পাবলিক লিঙ্ক আপডেট করা হয়েছে। You may view and manage your chat links after registration here.
নথির পাবলিক লিঙ্ক
বিজ্ঞাপন
