Bull AIX 4.3 - General Programming Guide

Bull
AIX 4.3 General Programming Concepts
Writing and Debugging Programs
AIX
ORDER REFERENCE
86 A2 34JX 02
Bull
AIX 4.3 General Programming Concepts
Writing and Debugging Programs
AIX
Software
November 1999
BULL ELECTRONICS ANGERS
CEDOC
34 Rue du Nid de Pie – BP 428
49004 ANGERS CEDEX 01
FRANCE
ORDER REFERENCE
86 A2 34JX 02
The following copyright notice protects this book under the Copyright laws of the United States of America
and other countries which prohibit such actions as, but not limited to, copying, distributing, modifying, and
making derivative works.
Copyright
Bull S.A. 1992, 1999
Printed in France
Suggestions and criticisms concerning the form, content, and presentation of
this book are invited. A form is provided at the end of this book for this purpose.
To order additional copies of this book or other Bull Technical Publications, you
are invited to use the Ordering Form also provided at the end of this book.
Trademarks and Acknowledgements
We acknowledge the right of proprietors of trademarks mentioned in this book.
AIXR is a registered trademark of International Business Machines Corporation, and is being used under
licence.
UNIX is a registered trademark in the United States of America and other countries licensed exclusively through
the Open Group.
Year 2000
The product documented in this manual is Year 2000 Ready.
The information in this document is subject to change without notice. Groupe Bull will not be liable for errors
contained herein, or for incidental or consequential damages in connection with the use of this material.
Contents
Trademarks and Acknowledgements . . iii
Legal Notices .
.
.
.
.
.
.
.
.
.
.
.
.
. iii
About This Book . . . . . . . . . . xvii
Who Should Use This Book . . .
How to Use This Book . . . . .
Highlighting . . . . . . .
ISO 9000 . . . . . . . . . .
AIX 32-Bit and 64-Bit Support for the
Specification . . . . . . . .
Related Publications . . . . . .
Ordering Publications . . . . .
Chapter 1. Tools and Utilities
. . .
. . .
. . .
. . .
UNIX98
. . .
. . .
. . .
.
.
.
.
.
.
.
.
.
. xvii
. xviii
. xviii
.
.
xvii
xvii
xvii
xvii
. . . . . 1
Entering a Program into the System . . . . .
Checking a Program . . . . . . . . . . .
Compiling and Linking a Program . . . . . .
Correcting Errors in a Program . . . . . .
Building and Maintaining a Program . . . .
Subroutines . . . . . . . . . . . . . .
Shell Commands . . . . . . . . . . . .
AIX Support for the X/Open UNIX95 Specification
AIX 32-Bit and 64-Bit Support for the UNIX98
Specification . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
1
1
1
2
2
3
3
3
. 3
Chapter 2. The Curses Library . . . . . 5
Terminology . . . . . . . . . . . .
Naming Conventions . . . . . . . . .
Structure of a Curses Program . . . . . .
Return Values . . . . . . . . . . .
Initializing Curses . . . . . . . . . .
Windows in the Curses Environment . . . .
The Default Window Structure . . . . .
The Current Window Structure . . . . .
Subwindows . . . . . . . . . . .
Pads . . . . . . . . . . . . . .
Manipulating Window Data with Curses . .
Creating Windows . . . . . . . . .
Removing Windows, Pads, and Subwindows
Changing the Screen or Window Images. .
Manipulating Window Content . . . . .
Support for Filters . . . . . . . . .
Controlling the Cursor with Curses . . . .
Manipulating Characters with Curses. . . .
Character Size . . . . . . . . . .
Adding Characters to the Screen Image . .
Enabling Text Scrolling . . . . . . .
Deleting Characters. . . . . . . . .
Getting Characters . . . . . . . . .
Understanding Terminals with curses. . . .
Manipulating Multiple Terminals . . . .
Setting Terminal Input and Output Modes .
Using the terminfo and termcap Files. . .
Low-Level Screen Subroutines . . . . .
Manipulating TTYs . . . . . . . . .
© Copyright IBM Corp. 1997, 1999
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 5
. 6
. 7
. 7
. 7
. 8
. 8
. 9
. 9
. 10
. 10
. 10
. 11
. 11
. 12
. 13
. 13
. 14
. 14
. 14
. 19
. 19
. 20
. 25
. 25
. 25
. 28
. 29
. 30
Synchronous and Networked Asynchronous
Terminals . . . . . . . . . . . .
Working with Color . . . . . . . . .
Manipulating Video Attributes . . . . . .
Video Attributes, Bit Masks, and the Default
Colors . . . . . . . . . . . . .
Setting Video Attributes . . . . . . .
Setting Curses Options . . . . . . .
Manipulating Soft Labels . . . . . . . .
Obsolete Curses Subroutines. . . . . . .
AIX Version 3.2 Curses Compatibility. . . .
List of Additional Curses Subroutines . . .
Manipulating Windows . . . . . . .
Manipulating Characters . . . . . . .
Manipulating Terminals . . . . . . .
Manipulating Color. . . . . . . . .
Miscellaneous Utilities . . . . . . . .
.
.
.
. 30
. 30
. 31
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
31
32
33
33
34
34
35
35
35
35
36
36
Chapter 3. Debugging Programs. . . . 37
adb Debug Program Overview . . . . . . . .
Getting Started with the adb Debug Program . . .
Starting adb with a Program File . . . . . .
Starting adb with a Nonexistent or Incorrect File
Starting adb with the Default File . . . . . .
Starting adb with a Core Image File . . . . .
Starting adb with a Data File . . . . . . .
Starting adb with the Write Option . . . . .
Using a Prompt . . . . . . . . . . . .
Using Shell Commands from within the adb
Program . . . . . . . . . . . . . .
Exiting the adb Debug Program . . . . . .
Controlling Program Execution . . . . . . . .
Preparing Programs for Debugging with the adb
Program . . . . . . . . . . . . . .
Running a Program. . . . . . . . . . .
Continuing Program Execution . . . . . . .
Using adb Expressions. . . . . . . . . . .
Using Integers in Expressions . . . . . . .
Using Symbols in Expressions . . . . . . .
Using Operators in Expressions. . . . . . .
Customizing the adb Debug Program. . . . . .
Combining Commands on a Single Line . . . .
Creating adb Scripts . . . . . . . . . .
Setting Output Width . . . . . . . . . .
Setting the Maximum Offset . . . . . . . .
Setting Default Input Format . . . . . . .
Changing the Disassembly Mode . . . . . .
Computing Numbers and Displaying Text . . . .
Displaying and Manipulating the Source File with
the adb Program. . . . . . . . . . . . .
Displaying Instructions and Data . . . . . .
Forming Addresses . . . . . . . . . . .
Displaying an Address . . . . . . . . .
Displaying the C Stack Backtrace . . . . . .
Choosing Data Formats . . . . . . . . .
37
37
37
37
38
38
38
39
39
39
39
40
40
40
42
44
44
44
44
45
45
46
48
48
48
49
49
50
50
50
51
51
52
v
Changing the Memory Map . . . . . . . .
Patching Binary Files . . . . . . . . . .
Locating Values in a File . . . . . . . . .
Writing to a File . . . . . . . . . . . .
Making Changes to Memory . . . . . . .
Using adb Variables . . . . . . . . . .
Finding the Current Address . . . . . . .
Displaying External Variables . . . . . . .
Displaying the Address Maps . . . . . . .
adb Debug Program Reference Information . . . .
adb Debug Program Addresses . . . . . . .
adb Debug Program Expressions . . . . . .
adb Debug Program Operators . . . . . . .
adb Debug Program Subcommands . . . . .
adb Debug Program Variables . . . . . . .
Example adb Program: adbsamp . . . . . . .
Example adb Program: adbsamp2 . . . . . . .
Example adb Program: adbsamp3 . . . . . . .
Example of Directory and i-node Dumps in adb
Debugging . . . . . . . . . . . . . .
Example of Data Formatting in adb Debugging . .
Example of Tracing Multiple Functions in adb
Debugging . . . . . . . . . . . . . .
Starting the adb Program . . . . . . . . .
Setting Breakpoints . . . . . . . . . . .
Displaying a Set of Instructions. . . . . . .
Starting the adsamp3 Program . . . . . . .
Removing a Breakpoint . . . . . . . . .
Continuing the Program . . . . . . . . .
Tracing the Path of Execution . . . . . . .
Displaying a Variable Value . . . . . . . .
Skipping Breakpoints . . . . . . . . . .
dbx Symbolic Debug Program Overview . . . .
Using the dbx Debug Program . . . . . . . .
Starting the dbx Debug Program . . . . . .
Running Shell Commands from dbx . . . . .
Command Line Editing in dbx . . . . . . .
Using Program Control . . . . . . . . .
Running a Program. . . . . . . . . . .
Separating dbx Output from Program Output . .
Tracing Execution . . . . . . . . . . .
Displaying and Manipulating the Source File with
the dbx debug Program . . . . . . . . . .
Changing the Source Directory Path . . . . .
Displaying the Current File . . . . . . . .
Changing the Current File or Procedure . . . .
Debugging Programs Involving Multiple Threads
Debugging Programs Involving Multiple
Processes . . . . . . . . . . . . . .
Examining Program Data . . . . . . . . . .
Handling Signals . . . . . . . . . . .
Calling Procedures . . . . . . . . . . .
Displaying a Stack Trace . . . . . . . . .
Displaying and Modifying Variables . . . . .
Displaying Thread-Related Information . . . .
Scoping of Names . . . . . . . . . . .
Using Operators and Modifiers in Expression . .
Checking of Expression Types . . . . . . .
Folding Variables to Lowercase and Uppercase
Changing Print Output with Special Debug
Program Variables . . . . . . . . . . .
vi
General Programming Concepts
52
53
53
54
54
54
55
56
56
57
57
57
58
58
62
62
62
63
64
65
68
68
68
69
69
69
69
69
70
70
70
71
71
71
71
72
72
73
74
74
74
75
75
75
78
79
79
81
81
82
82
82
83
83
84
84
Debugging at the Machine Level with dbx . . .
Using Machine Registers . . . . . . . .
Examining Memory Addresses . . . . . .
Running a Program at the Machine Level . .
Debugging fdpr Reordered Executables . . .
Displaying Assembly Instructions . . . . .
Customizing the dbx Debugging Environment. .
Defining a New dbx Prompt. . . . . . .
Creating dbx Subcommand Aliases . . . .
Using the .dbxinit File . . . . . . . . .
List of dbx Subcommands . . . . . . . .
Setting and Deleting Breakpoints . . . . .
Running Your Program . . . . . . . .
Tracing Program Execution . . . . . . .
Ending Program Execution . . . . . . .
Displaying the Source File . . . . . . .
Printing and Modifying Variables, Expressions,
and Types . . . . . . . . . . . . .
Thread Debugging . . . . . . . . . .
Multiprocess Debugging . . . . . . . .
Procedure Calling . . . . . . . . . .
Signal Handling . . . . . . . . . . .
Machine-Level Debugging . . . . . . .
Debugging Environment Control . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
85
85
86
86
87
87
87
87
88
89
89
89
90
91
91
91
.
.
.
.
.
.
.
91
91
91
91
91
92
92
Chapter 4. Error Notification . . . . . 93
Security . . . . .
Examples . . . . .
Related Information
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 95
. 95
. 96
Chapter 5. File Systems and
Directories . . . . . . . . . . . . . 97
File Types . . . . . . . . . . . . . .
Working with Files . . . . . . . . . .
Directories . . . . . . . . . . . . .
Directory Structures . . . . . . . . .
Working with Directories (Programming) . .
Subroutines That Control Directories . . .
Working with JFS i-nodes . . . . . . . .
Disk i-node Structure . . . . . . . . .
In-core i-node Structure . . . . . . . .
File Space Allocation . . . . . . . . . .
Full and Partial Logical Blocks . . . . .
Allocation in Fragmented File Systems . . .
Allocation in Compressed File Systems . . .
Allocation in File Systems Enabled for Large
Files . . . . . . . . . . . . . .
Disk Address Format . . . . . . . . .
Indirect Blocks . . . . . . . . . . .
Quotas . . . . . . . . . . . . .
Writing Programs That Access Large Files . . .
Implications for Existing Programs . . . .
Open Protection . . . . . . . . . .
Porting Applications to the Large File
Environment . . . . . . . . . . .
Using _LARGE_FILES . . . . . . . .
Using the 64-Bit File System Subroutines . .
Common Pitfalls using the Large File
Environment . . . . . . . . . . .
Linking for Programmers . . . . . . . .
. 97
. 99
. 100
. 100
. 102
. 102
. 103
. 103
. 104
. 104
. 105
. 105
. 106
.
.
.
.
.
.
.
106
106
108
110
111
111
111
. 112
. 113
. 114
. 115
. 119
Hard Links . . . . . . . . .
Symbolic Links . . . . . . . .
Directory Links . . . . . . . .
Using File Descriptors . . . . . .
System File and File Descriptor Tables
Managing File Descriptors . . . .
Preset File Descriptor Values . . .
File Descriptor Resource Limit. . .
Related Information . . . . . .
File Creation and Removal . . . . .
Creating a File . . . . . . . .
Opening a File . . . . . . . .
Closing a File . . . . . . . .
Working with File I/O . . . . . .
Manipulating the Current Offset . .
Reading a File . . . . . . . .
Writing a File . . . . . . . .
Writing Programs to Use Direct I/O .
Direct I/O vs. Normal Cached I/O .
Benefits of Direct I/O . . . . .
Working with Pipes . . . . . .
Synchronous I/O . . . . . . .
File Status . . . . . . . . . .
File Accessibility . . . . . . . .
File System Layout . . . . . . .
Boot Block . . . . . . . . .
Superblock . . . . . . . . .
Allocation Bitmaps . . . . . .
Fragments . . . . . . . . .
Disk I-Nodes . . . . . . . .
Allocation Groups . . . . . . .
Using File System Subroutines. . .
Creating New File System Types . . .
File System Helpers . . . . . .
Mount Helpers . . . . . . . .
Major Control Block Header Files . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
119
119
120
121
121
122
125
126
127
127
127
128
128
128
128
129
130
131
131
132
133
135
136
136
138
138
138
138
138
138
139
139
140
140
143
144
Chapter 6. Floating-Point Exceptions
145
State Diagram . . . . . . . . . . . .
Floating-Point Exception Subroutines . . . .
Floating-Point Trap Handler Operation . . . .
Exceptions: Disabled and Enabled Comparison
Imprecise Trapping Modes . . . . . . .
Hardware-Specific Subroutines . . . . .
Example of a Floating-Point Trap Handler . .
. 145
. 146
. 147
147
. 148
. 148
. 149
Chapter 7. Input and Output Handling
155
Low-Level I/O Interfaces .
Stream I/O Interfaces . .
Terminal I/O Interfaces . .
Asynchronous I/O Interfaces
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
156
157
157
158
Chapter 8. Large Program Support
161
Understanding the Large Address-Space Model
Enabling the Large Address-Space Model . . .
Executing Programs with Large Data Areas . .
Special Considerations . . . . . . . .
161
. 162
. 162
. 163
Chapter 9. Parallel Programming . . . 165
Understanding Threads .
.
.
.
.
.
.
.
.
. 165
Threads and Processes . . . . . . . . .
Threads Implementation. . . . . . . . .
Thread Programming Concepts . . . . . .
Writing Reentrant and Thread-Safe Code . . .
Developing Multi-Threaded Programs . . . .
Developing Multi-Threaded Program Debuggers
Multi-Threaded Call Back Functions . . . . .
Benefits of Threads . . . . . . . . . . .
Parallel Programming Concepts . . . . . .
Performance Consideration . . . . . . . .
Limitations . . . . . . . . . . . . .
165
166
169
174
179
182
187
189
189
192
193
Chapter 10. Programming on
Multiprocessor Systems. . . . . . . 195
Identifying Processors . . . . . . . . .
ODM Processor Names . . . . . . . .
Logical Processor Numbers . . . . . . .
ODM Processor States . . . . . . . .
Controlling Processor Use . . . . . . . .
The cpu_state Command . . . . . . .
Example Processor Configurations . . . .
Binding Processes and Kernel Threads . . .
Creating Locking Services . . . . . . . .
Multiprocessor-Safe Locking Services . . .
Locking Services Example . . . . . . .
Kernel Programming . . . . . . . . . .
32-bit and 64-bit Addressability . . . . .
Differences between 32-bit and 64-bit execution
environments . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
195
195
196
196
196
196
196
199
200
200
201
201
202
. 203
Chapter 11. Threads Programming
Guidelines. . . . . . . . . . . . . 207
Thread Implementation Model . . . . . . .
Thread-safe and Threaded Libraries in AIX . . .
Threads Versions On AIX . . . . . . . . .
Threads Basic Operation Overview . . . . . .
Creating Threads . . . . . . . . . . . .
Thread Attributes Object . . . . . . . .
Thread Creation . . . . . . . . . . .
Handling Thread IDs . . . . . . . . . .
A First Multi-Threaded Program . . . . . .
Terminating Threads . . . . . . . . . . .
Exiting a Thread . . . . . . . . . . .
Canceling a Thread . . . . . . . . . .
Using Cleanup Handlers . . . . . . . .
List of Threads Basic Operation Subroutines . . .
Synchronization Overview . . . . . . . . .
Using Mutexes . . . . . . . . . . . . .
Mutex Attributes Object . . . . . . . . .
Creating and Destroying Mutexes . . . . .
Locking and Unlocking Mutexes . . . . . .
Protecting Data with Mutexes . . . . . . .
Using Condition Variables . . . . . . . . .
Condition Attributes Object . . . . . . .
Creating and Destroying Condition Variables
Using Condition Variables . . . . . . . .
Synchronizing Threads with Condition Variables
Joining Threads . . . . . . . . . . . .
Waiting for a Thread . . . . . . . . . .
Returning Information from a Thread . . . .
Contents
207
207
208
208
208
209
210
211
212
212
212
214
218
219
220
220
220
221
222
222
224
224
225
226
228
230
230
233
vii
List of Synchronization Subroutines . . . . .
Scheduling Overview. . . . . . . . . .
Threads Scheduling . . . . . . . . . .
Basic Scheduling Facilities . . . . . . .
Scheduling Policy and Priority . . . . .
Contention Scope . . . . . . . . . .
Synchronization Scheduling . . . . . . .
Priority Inversion . . . . . . . . . .
Mutex Protocols . . . . . . . . . .
Choosing a Mutex Protocol . . . . . . .
List of Scheduling Subroutines . . . . . .
Threads Advanced Features . . . . . . .
One-Time Initializations . . . . . . . . .
One-Time Initialization Object . . . . . .
One-Time Initialization Routine . . . . .
Thread-Specific Data . . . . . . . . . .
Creating and Destroying Keys . . . . . .
Using Thread-Specific Data . . . . . . .
Advanced Attributes . . . . . . . . . .
Stack Attributes . . . . . . . . . .
Process Sharing. . . . . . . . . . .
Making Complex Synchronization Objects . . .
Long Locks . . . . . . . . . . . .
Semaphores . . . . . . . . . . . .
Write-Priority Read/Write Locks . . . . .
List of Threads Advanced-Feature Subroutines .
Threads-Processes Interactions Overview . . .
Signal Management . . . . . . . . . .
Signal Handlers and Signal Masks . . . .
Signal Generation . . . . . . . . . .
Handling Signals . . . . . . . . . .
Signal Delivery . . . . . . . . . . .
Process Duplication and Termination . . . .
Forking . . . . . . . . . . . . .
Fork Handlers . . . . . . . . . . .
Process Termination . . . . . . . . .
Scheduling . . . . . . . . . . . . .
Process-Level Scheduling . . . . . . .
Timer and Sleep Subroutines . . . . . .
List of Threads-Processes Interactions Subroutines
Threads Library Options . . . . . . . .
List of Options . . . . . . . . . . .
Checking the Availability of an Option . . .
Threads Library Quick Reference . . . . . .
Supported Interfaces . . . . . . . . .
Threads Data Types . . . . . . . . .
Limits and Default Values . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
234
235
235
235
236
237
239
239
240
241
243
243
243
243
244
244
245
247
248
248
250
250
250
251
252
254
254
255
255
255
256
257
258
258
258
259
259
259
259
260
260
260
261
262
262
265
265
Chapter 12. lex and yacc Program
Information . . . . . . . . . . . . 267
Creating an Input Language with the lex and yacc
Commands . . . . . . . . . . . . . .
Writing a Lexical Analyzer Program with the lex
Command . . . . . . . . . . . . .
Extended Regular Expressions in the lex
Command . . . . . . . . . . . . .
lex Actions . . . . . . . . . . . . .
Passing Code to the Generated lex Program . .
Defining lex Substitution Strings . . . . . .
lex Start Conditions . . . . . . . . . .
Compiling the Lexical Analyzer . . . . . .
viii
General Programming Concepts
267
267
268
272
276
276
277
277
lex Library . . . . . . . . . . . .
Using the lex Program with the yacc Program .
Creating a Parser with the yacc Program . .
yacc Grammar File . . . . . . . . .
Using the yacc Grammar File . . . . . .
yacc Declarations . . . . . . . . . .
yacc Rules . . . . . . . . . . . .
yacc Actions . . . . . . . . . . . .
yacc Error Handling . . . . . . . . .
Lexical Analysis for the yacc Command . .
yacc-Generated Parser Operation . . . . .
Using Ambiguous Rules in the yacc Program
Turning on Debug Mode for a yacc-Generated
Parser . . . . . . . . . . . . . .
Example Program for the lex and yacc Programs
Compiling the Example Program . . . . .
.
.
.
.
.
.
.
.
.
.
.
278
278
279
280
281
283
285
287
288
290
290
292
. 293
293
. 294
Chapter 13. Logical Volume
Programming . . . . . . . . . . . 299
Logical Volume Program Example .
List of Logical Volume Subroutines .
.
.
.
.
.
.
.
.
. 299
. 305
Chapter 14. make Command Overview 307
Creating a Description File . . . . . . . . .
Format of a make Description File Entry . . .
Using Commands in a make Description File
Calling the make Program from a Description
File . . . . . . . . . . . . . . . .
Preventing the make Program from Writing
Commands . . . . . . . . . . . . .
Preventing the make Program from Stopping on
Errors . . . . . . . . . . . . . . .
Example of a Description File . . . . . . .
Making the Description File Simpler. . . . .
Internal Rules for the make Program . . . . .
Example of Default Rules File . . . . . . .
Single-Suffix Rules . . . . . . . . . .
Using the Make Command with Archive
Libraries . . . . . . . . . . . . . .
Changing Macros in the Rules File . . . . .
Defining Default Conditions in a Description
File . . . . . . . . . . . . . . . .
Including Other Files in a Description File. . .
Defining and Using Macros in a Description File
Using Macros in a Description File . . . . .
Internal Macros. . . . . . . . . . . .
Changing Macro Definitions in a Command . .
How the make Command Creates a Target File . .
Using the make Command with Source Code
Control System (SCCS) Files . . . . . . . .
Description Files Stored in the Source Code
Control System (SCCS) . . . . . . . . . .
Using the make Command with Non-Source Code
Control System (SCCS) Files . . . . . . . .
How the make Command Uses the Environment
Variables . . . . . . . . . . . . . . .
Example of a Description File . . . . . . . .
307
308
308
309
309
310
310
310
311
312
312
313
313
313
314
314
314
315
318
318
319
320
320
321
321
Chapter 15. m4 Macro Processor
Overview . . . . . . . . . . . . . 325
Using the m4 Macro Processor . . .
Creating a User-Defined Macro . . .
Using the Quote Characters . . .
Arguments . . . . . . . . .
Using a Built-In m4 Macro . . . . .
Removing a Macro Definition . . .
Checking for a Defined Macro. . .
Using Integer Arithmetic . . . .
Manipulating Files . . . . . .
Redirecting Output . . . . . .
Using System Programs in a Program
Using Unique File Names . . . .
Using Conditional Expressions . .
Manipulating Strings . . . . . .
Printing . . . . . . . . . .
List of Additional m4 Macros . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
325
325
326
327
328
328
329
329
330
330
331
331
331
332
333
333
Chapter 16. National Language
Support . . . . . . . . . . . . . . 335
NLS Capabilities . . . . . . . . . . . .
Locale-Specific and Culture-Specific
Conventions . . . . . . . . . . . . .
User Messages in Native Languages . . . . .
Code Set Support . . . . . . . . . . .
Input Method Support . . . . . . . . .
Overview of Chapter Contents . . . . . . .
Locale Overview for Programming . . . . . .
Working with Code Sets . . . . . . . . .
Data Representation . . . . . . . . . .
Character Properties . . . . . . . . . .
Localization . . . . . . . . . . . . .
Multibyte Subroutines . . . . . . . . .
Wide Character Subroutines . . . . . . .
Bidirectionality and Character Shaping . . . .
Code Set Independence . . . . . . . . .
File Name Matching . . . . . . . . . .
Radix Character Handling . . . . . . . .
Programming Model . . . . . . . . . .
National Language Support Subroutines Overview
Introducing Locale Subroutines . . . . . .
Introducing Time Formatting Subroutines . . .
Introducing Monetary Formatting Subroutines
Introducing Multibyte and Wide Character
Subroutines . . . . . . . . . . . . .
Introducing Internationalized Regular
Expression Subroutines . . . . . . . . .
Locale Subroutines . . . . . . . . . . .
Setting the Locale . . . . . . . . . . .
Accessing Locale Information . . . . . . .
Examples . . . . . . . . . . . . . .
Time Formatting Subroutines . . . . . . . .
Examples . . . . . . . . . . . . . .
Monetary Formatting Subroutines . . . . . .
Euro Currency Support via the @euro Modifier
Examples . . . . . . . . . . . . . .
Multibyte and Wide Character Subroutines . . .
Multibyte Code and Wide Character Code
Conversion Subroutines . . . . . . . . .
Wide Character Classification Subroutines . . .
Wide Character Display Column Width
Subroutines . . . . . . . . . . . . .
335
335
335
336
336
336
337
337
338
339
340
343
343
344
344
345
345
346
348
348
349
349
349
350
350
350
350
351
355
355
355
356
356
358
358
362
364
Multibyte and Wide Character String Collation
Subroutines . . . . . . . . . . . .
Multibyte and Wide Character String
Comparison Subroutines . . . . . . .
Wide Character String Conversion Subroutines
Wide Character String Copy Subroutines . .
Wide Character String Search Subroutines . .
Wide Character Input/Output Subroutines .
Working with the Wide Character Constant .
Internationalized Regular Expression Subroutines
Examples . . . . . . . . . . . . .
Layout (Bidirectional Text and Character Shaping)
Overview. . . . . . . . . . . . . .
Data Streams . . . . . . . . . . .
Cursor Movement . . . . . . . . . .
Character Shaping . . . . . . . . . .
Introducing Layout Library Subroutines . .
Use of the libcur Package . . . . . . . .
Code Set Overview . . . . . . . . . .
ASCII Characters . . . . . . . . . .
Other ASCII Characters . . . . . . . .
Code Set Strategy . . . . . . . . . .
Code Set Structure. . . . . . . . . .
ISO Code Sets . . . . . . . . . . .
IBM PC Code Sets . . . . . . . . . .
UCS-2 and UTF-8 . . . . . . . . . .
Converters Overview for Programming. . . .
Converters Introduction . . . . . . . .
Standard Converters . . . . . . . . .
Understanding libiconv . . . . . . . .
Using Converters . . . . . . . . . .
List of Converters . . . . . . . . . .
Writing Converters Using the iconv Interface . .
Code Sets and Converters . . . . . . .
iconv Framework - Overview of Structures .
Writing a Code Set Converter . . . . . .
Examples . . . . . . . . . . . . .
Input Method Overview. . . . . . . . .
Input Method Introduction . . . . . . .
Input Method Names. . . . . . . . .
Input Method Areas . . . . . . . . .
Programming Input Methods . . . . . . .
Initialization. . . . . . . . . . . .
Input Method Management . . . . . .
IM Keymap Management . . . . . . .
Key Event Processing. . . . . . . . .
Callbacks . . . . . . . . . . . . .
Input Method Structures . . . . . . .
Working with Keyboard Mapping . . . . .
IM Keymaps . . . . . . . . . . .
Inbound and Outbound Mapping . . . .
Using Callbacks . . . . . . . . . . .
Initializing Callbacks . . . . . . . . .
Bidirectional Input Method . . . . . . . .
Cyrillic Input Method (CIM) . . . . . . .
Keymap: . . . . . . . . . . . . .
Keysyms: . . . . . . . . . . . . .
Reserved Keysyms: . . . . . . . . .
Modifiers . . . . . . . . . . . . .
Greek Input Method (GIM) . . . . . . . .
Keymap: . . . . . . . . . . . . .
Contents
. 366
. 368
368
. 370
. 371
. 373
. 378
379
. 379
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
381
382
383
384
385
385
387
388
389
390
391
392
407
420
423
424
424
425
428
429
451
451
452
454
458
465
465
467
468
468
469
470
470
471
471
471
471
472
473
473
476
476
477
478
478
478
478
479
479
ix
Keysyms: . . . . . . . . . . . . . .
Reserved keysyms: . . . . . . . . . .
Japanese Input Method (JIM) . . . . . . . .
Japanese Character Processing . . . . . . .
Kana-To-Kanji Conversion (KKC) Technology
Input Modes . . . . . . . . . . . .
Keyboard Mapping . . . . . . . . . .
Character Size . . . . . . . . . . . .
Romaji-To-Kana Conversion (RKC) . . . . .
Kanji Pre-edit . . . . . . . . . . . .
Keymaps: . . . . . . . . . . . . .
Keysyms: . . . . . . . . . . . . . .
Reserved Keysyms: . . . . . . . . . .
Korean Input Method (KIM) . . . . . . . .
Latvian Input Method (LVIM) . . . . . . . .
Keymap: . . . . . . . . . . . . . .
Lithuanian Input Method (LTIM) . . . . . . .
Keymap: . . . . . . . . . . . . . .
Thai Input Method (THIM) . . . . . . . . .
Keymap: . . . . . . . . . . . . . .
Vietnamese Input Method (VNIM) . . . . . .
Keymap: . . . . . . . . . . . . . .
Simplified Chinese Input Method (ZIM) . . . .
Simplified Chinese Character Processing . . .
Simplified Chinese Input Method (ZIM-UCS). . .
Chinese (CJK) Character Processing . . . . .
Single-Byte Input Method . . . . . . . . .
Traditional Chinese Input Method (TIM) . . . .
TIM Features . . . . . . . . . . . .
Traditional Chinese Character Processing . . .
Universal Input Method . . . . . . . . . .
Keymap: . . . . . . . . . . . . . .
List of Reserved Keysyms . . . . . . . . .
Reserved Keysyms for Traditional Chinese . .
Reserved Keysyms for Simplified Chinese (ZIM
and ZIM-UCS) . . . . . . . . . . . .
Message Facility Overview for Programming . . .
Creating a Message Source File . . . . . .
Creating a Message Catalog . . . . . . .
Displaying Messages outside of an Application
Program . . . . . . . . . . . . . .
Displaying Messages with an Application
Program . . . . . . . . . . . . . .
Example of Retrieving a Message from a
Catalog . . . . . . . . . . . . . .
Culture-Specific Data Processing . . . . . . .
Culture-Specific Tables . . . . . . . . .
Culture-Specific Algorithms . . . . . . .
Example: Load a Culture-Specific Module for
Arabic Text for an Application. . . . . . .
NLS Sample Program . . . . . . . . . .
Message Source File for foo . . . . . . .
Creation of Message Header File for foo . . .
Single Path Code Set Independent Version. . .
Dual-Path Version Optimized for Single-Byte
Code Sets . . . . . . . . . . . . .
National Language Support (NLS) Quick Reference
National Language Support Do’s and Don’ts
National Language Support Checklist . . . .
Message Suggestions . . . . . . . . . .
List of National Language Support Subroutines
x
General Programming Concepts
479
479
480
481
481
481
483
483
483
484
486
486
486
487
489
489
489
489
489
490
490
490
490
491
492
492
493
495
495
496
497
497
497
498
498
498
498
503
505
505
507
508
508
508
509
510
510
511
511
513
516
516
517
523
526
List
List
List
List
List
List
List
List
List
of
of
of
of
of
of
of
of
of
Locale Subroutines . . . . . . . . .
Time and Monetary Formatting Subroutines
Multibyte Character Subroutines . . . .
Wide Character Subroutines . . . . . .
Layout Library Subroutines . . . . . .
Message Facility Subroutines . . . . . .
Converter Subroutines . . . . . . . .
Input Method Subroutines . . . . . .
Regular Expression Subroutines . . . . .
527
527
527
527
529
529
530
530
531
Chapter 17. Object Data Manager
(ODM) . . . . . . . . . . . . . . 533
ODM Object Classes and Objects . . . . . .
Creating an Object Class. . . . . . . .
Adding Objects to an Object Class . . . .
Locking Object Classes . . . . . . . .
Storing Object Classes and Objects . . . .
ODM Descriptors . . . . . . . . . . .
ODM Terminal Descriptors . . . . . . .
ODM Link Descriptor . . . . . . . .
ODM Method Descriptor . . . . . . .
ODM Object Searches . . . . . . . . .
Descriptor Names in ODM Predicates . . .
Comparison Operators in ODM Predicates .
LIKE Comparison Operator . . . . . .
Constants in ODM Predicates . . . . . .
AND Logical Operator for Predicates . . .
List of ODM Commands and Subroutines . . .
Commands . . . . . . . . . . . .
Subroutines . . . . . . . . . . . .
ODM Example Code and Output. . . . . .
ODM Example Input Code for Creating Object
Classes . . . . . . . . . . . . .
ODM Example Output for Object Class
Definitions . . . . . . . . . . . .
ODM Example Code for Adding Objects to
Object Classes . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
533
535
535
535
536
537
538
538
540
541
542
542
542
543
544
544
544
545
545
. 546
. 547
. 548
Chapter 18. sed Program Information
551
Manipulating Strings with sed. . .
Starting the Editor. . . . . .
How sed Works . . . . . .
Using Regular Expressions . . .
Using the sed Command Summary
Using Text in Commands . . .
Using String Replacement . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
551
551
551
552
552
556
556
Chapter 19. Shared Libraries, Shared
Memory, and The malloc Subsystem . 559
Shared Objects and Runtime Linking . . . . .
Operation of the Runtime Linker . . . . . .
Creating a Shared Object with Runtime Linking
Enabled . . . . . . . . . . . . . .
Shared Libraries and Lazy Loading . . . . . .
Lazy Loading Execution Tracing . . . . . .
Creating a Shared Library . . . . . . . . .
Prerequisite Tasks . . . . . . . . . . .
Procedure . . . . . . . . . . . . .
Program Address Space Overview . . . . . .
System Memory Architecture Introduction. . .
559
560
561
562
563
563
563
564
565
566
The Physical Address Space of 32-bit Systems
The Physical Address Space of 64-bit Systems
Segment Register Addressing . . . . . . .
Paging Space . . . . . . . . . . . .
Memory Management Policy . . . . . . .
Memory Allocation . . . . . . . . . .
Understanding Memory Mapping . . . . . .
mmap Comparison with shmat . . . . . .
mmap Compatibility Considerations. . . . .
Using the Semaphore Subroutines . . . . .
Mapping Files with the shmat Subroutine . . .
Mapping Shared Memory Segments with the
shmat Subroutine . . . . . . . . . . .
Creating a Mapped Data File with the shmat
Subroutine . . . . . . . . . . . . . .
Prerequisite Condition . . . . . . . . .
Procedure . . . . . . . . . . . . .
Creating a Copy-On-Write Mapped Data File with
the shmat Subroutine . . . . . . . . . . .
Prerequisite Condition . . . . . . . . .
Procedure . . . . . . . . . . . . .
Creating a Shared Memory Segment with the
shmat Subroutine . . . . . . . . . . . .
Prerequisite Tasks or Conditions . . . . . .
Procedure . . . . . . . . . . . . .
System Memory Allocation Using the malloc
Subsystem . . . . . . . . . . . . . .
Working with the Heap . . . . . . . . .
Working with the Heap . . . . . . . . .
Understanding System Allocation Policy . . .
Understanding the Version 3.2/Version 4.1
Allocation Policy . . . . . . . . . . .
Understanding the Version 3.1 Allocation Policy
User Defined Malloc Replacement . . . . . .
Enablement . . . . . . . . . . . . .
32/64bit Considerations . . . . . . . . .
Thread Considerations . . . . . . . . .
Limitations . . . . . . . . . . . . .
Error Reporting . . . . . . . . . . .
Paging Space Programming Requirements . . . .
List of Memory Manipulation Services . . . . .
List of Memory Mapping Services . . . . . .
566
567
567
568
568
569
569
570
572
572
573
574
574
574
574
575
575
575
576
576
576
576
577
577
578
578
579
581
583
583
583
584
584
584
585
586
Chapter 20. Packaging Software for
Installation . . . . . . . . . . . . 587
Installation Procedure Requirements. . . .
Package Control Information Requirements .
Package Partitioning Requirements . . . .
Software Product Packaging Parts . . . .
Sample File System Guide for Package
Partitioning . . . . . . . . . . .
Format of a Software Package . . . . . .
Package and Fileset Naming Conventions . .
Fileset Extension Naming Conventions . .
Special Naming Considerations for Device
Driver Packaging . . . . . . . . .
Special Naming Considerations for Message
Catalog Packaging. . . . . . . . .
File Names . . . . . . . . . . .
Fileset Revision Level Identification . . . .
Fileset Level Overview . . . . . . .
.
.
.
.
.
.
.
.
587
587
588
588
.
.
.
.
.
.
.
.
588
589
589
589
.
. 590
.
.
.
.
.
.
.
.
590
590
591
591
Fileset Level Rules and Conventions for AIX
Version 4.1-Formatted Filesets . . . . . .
Compatibility Information For Version
3.2-Formatted Fileset Updates . . . . . .
Contents of a Software Package . . . . . .
Example Contents of a Software Package . .
The lpp_name Package Information File . . .
Requisite Information Section . . . . . .
Size Information Section . . . . . . . .
Supersede Information Section . . . . .
Fix Information Section . . . . . . . .
The liblpp.a Installation Control Library File . .
Data Files Contained in the liblpp.a File . .
Optional Executable Files Contained in the
liblpp.a File . . . . . . . . . . . .
Optional Executable File Contained in the
Fileset.al File . . . . . . . . . . .
Further Description of Installation Control Files
The Fileset.cfgfiles File . . . . . . . .
The Fileset.fixdata File . . . . . . . .
The Fileset.inventory File . . . . . . .
Installation Control Files Specifically for
Repackaged Products . . . . . . . . . .
The Fileset.installed_list File . . . . . .
The Fileset.namelist File . . . . . . . .
The Fileset.rm_inv File . . . . . . . .
Installation Files for Supplemental Disk
Subsystems . . . . . . . . . . . . .
Format of Distribution Media . . . . . . .
Tape . . . . . . . . . . . . . .
CD-ROM . . . . . . . . . . . . .
Diskette . . . . . . . . . . . . .
The Table of Contents File . . . . . . . .
Date and Time Stamp Format . . . . . .
Location Format for Tape and Diskette . . .
The installp Processing of Product Packages . .
Processing for the Apply Operation . . . .
Processing for the Reject and Cleanup
Operations . . . . . . . . . . . .
Processing for the Remove Operation . . .
The Installation Status File . . . . . . .
Installation Commands Used During Installation
and Update Processing . . . . . . . . .
. 591
.
.
.
.
.
.
.
.
.
.
592
592
593
594
596
600
601
603
604
604
. 605
. 607
607
. 607
. 609
. 609
.
.
.
.
611
611
612
612
.
.
.
.
.
.
.
.
.
.
613
614
614
614
615
616
616
616
617
618
. 621
. 622
. 623
. 624
Chapter 21. Documentation Library
Service . . . . . . . . . . . . . . 625
Language Support . . . . . . . . . . . .
Writing your HTML Documents . . . . . . .
Calling the Documentation Library Service From
Your Documentation . . . . . . . . . . .
Section A: Calling the Documentation Library
Service From Your Documentation (for versions
AIX Version 4.3, AIX Version 4.3.1 and AIX
Version 4.3.2) . . . . . . . . . . . .
Section B: Calling the Documentation Library
Service From Your Documentation (for AIX
Version 4.3.3 and later) . . . . . . . . .
Section C: Calling the Documentation Library
Service From Your Documentation (for all
versions of AIX) . . . . . . . . . . .
Creating Indexes of your Documentation . . . .
Contents
626
626
627
628
639
647
647
xi
Requirements . . . . . . . . . .
Building the Indexes . . . . . . . .
Removing Indexes of your Documentation .
Packaging your Application’s Documentation
Include a Search Index . . . . . . .
Register your Documentation . . . . .
Create an install package . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
647
648
657
657
657
659
659
Chapter 22. Software Vital Product
Data (SWVPD) . . . . . . . . . . . 661
Object Classes .
Files . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 661
. 662
Chapter 23. Source Code Control
System (SCCS) . . . . . . . . . . 663
Introduction to SCCS . . . . . . . . .
Delta Table in SCCS files . . . . . . .
Control and Tracking Flags in SCCS Files . .
Body of an SCCS file . . . . . . . . .
SCCS Flag and Parameter Conventions . . .
Creating, Editing, and Updating an SCCS File
Creating an SCCS File . . . . . . .
Editing an SCCS file . . . . . . . .
Updating an SCCS File . . . . . . .
Controlling and Tracking SCCS File Changes .
Controlling Access to SCCS files . . . .
Tracking Changes to an SCCS File . . .
Detecting and Repairing Damaged SCCS Files
Procedure . . . . . . . . . . .
List of Additional SCCS Commands . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
663
663
664
664
664
665
665
665
666
666
666
667
667
668
668
Chapter 24. Subroutines, Example
Programs, and Libraries. . . . . . . 669
128-Bit Long Double Floating-Point Data Type . .
Compiling Programs that Use the 128-bit Long
Double Data Type . . . . . . . . . . .
Compliance with IEEE 754 Standard. . . . .
Implementing the 128-Bit Long Double Format
Values of Numeric Macros . . . . . . . .
List of Character Manipulation Subroutines . . .
Character Testing . . . . . . . . . . .
Character Translation . . . . . . . . . .
Miscellaneous Character Manipulation . . . .
List of Executable Program Creation Subroutines
List of Files and Directories Subroutines . . . .
Controlling Files . . . . . . . . . . .
Working with Directories . . . . . . . .
Manipulating File Systems . . . . . . . .
List of FORTRAN BLAS Level 1: Vector-Vector
Subroutines . . . . . . . . . . . . . .
List of FORTRAN BLAS Level 2: Matrix-Vector
Subroutines . . . . . . . . . . . . . .
List of FORTRAN BLAS Level 3: Matrix-Matrix
Subroutines . . . . . . . . . . . . . .
List of Numerical Manipulation Subroutines . . .
List of Long Long Integer Numerical Manipulation
Subroutines . . . . . . . . . . . . . .
List of 128-Bit Long Double Numerical
Manipulation Subroutines . . . . . . . . .
List of Processes Subroutines . . . . . . . .
xii
General Programming Concepts
670
670
670
671
671
672
672
672
673
673
674
674
674
675
675
676
676
677
678
678
679
Process Initiation . . . . . . . . . .
Process Suspension . . . . . . . . .
Process Termination . . . . . . . . .
Process and Thread Identification . . . .
Process Accounting . . . . . . . . .
Process Resource Allocation . . . . . .
Process Prioritization . . . . . . . . .
Process and Thread Synchronization. . . .
Process Signals and Masks . . . . . . .
Process Messages . . . . . . . . . .
List of Multi-threaded Programming Subroutines
List of Programmer’s Workbench Library
Subroutines . . . . . . . . . . . . .
File . . . . . . . . . . . . . . .
List of Security and Auditing Subroutines . . .
Access Control Subroutines. . . . . . .
Auditing Subroutines. . . . . . . . .
Identification and Authentication Subroutines
Process Subroutines . . . . . . . . .
List of String Manipulation Subroutines . . .
Programming Example for Manipulating
Characters . . . . . . . . . . . . .
Searching and Sorting Example Program . . .
List of Operating System Libraries . . . . .
librs2.a Library . . . . . . . . . . . .
General-Use sqrt and itrunc Subroutines . .
POWER2-Specific sqrt and itrunc Subroutines
.
.
.
.
.
.
.
.
.
.
679
679
679
680
680
680
680
680
681
681
681
.
.
.
.
.
681
682
683
683
683
683
. 684
. 684
.
.
.
.
.
685
687
689
690
690
690
Chapter 25. System Management
Interface Tool (SMIT) . . . . . . . . 693
SMIT Screen Types . . . . . . . . . . .
Menu Screens . . . . . . . . . . . .
Selector Screens . . . . . . . . . . .
Dialog Screens . . . . . . . . . . . .
SMIT Object Classes . . . . . . . . . . .
The SMIT Database . . . . . . . . . .
SMIT Aliases and Fast Paths . . . . . . . .
SMIT Information Command Descriptors . . . .
The cmd_to_discover Descriptor . . . . . .
The cmd_to_*_postfix Descriptors . . . . .
SMIT Command Generation and Execution . . .
Generating Dialog Defined Tasks . . . . . .
Executing Dialog Defined Tasks . . . . . .
Adding Tasks to the SMIT Database . . . . . .
Procedure . . . . . . . . . . . . .
Debugging SMIT Database Extensions . . . . .
Prerequisite Tasks or Conditions . . . . . .
Procedure . . . . . . . . . . . . .
Creating SMIT Help Information for a New Task
Man Pages Method . . . . . . . . . .
Message Catalog Method . . . . . . . .
Softcopy Libraries Method . . . . . . . .
sm_menu_opt (SMIT Menu) Object Class . . . .
The sm_menu_opt Object Class Used for Aliases
sm_name_hdr (SMIT Selector Header) Object Class
sm_cmd_opt (SMIT Dialog/Selector Command
Option) Object Class . . . . . . . . . . .
sm_cmd_hdr (SMIT Dialog Header) Object Class
SMIT Example Program . . . . . . . . . .
693
694
695
696
696
702
702
706
707
708
709
709
710
711
711
712
712
712
712
713
713
714
714
715
716
719
723
726
Chapter 26. System Resource
Controller . . . . . . . . . . . . . 739
Subsystem Interaction with the SRC . . . . . .
The SRC and the init Command . . . . . . .
Compiling Programs to Interact With the srcmstr
Daemon . . . . . . . . . . . . . . .
SRC Operations . . . . . . . . . . . .
SRC Capabilities . . . . . . . . . . . .
SRC Objects . . . . . . . . . . . . . .
Subsystem Object Class . . . . . . . . .
Subserver Type Object Class . . . . . . .
Notify Object Class . . . . . . . . . .
SRC Communication Types. . . . . . . . .
Signals Communication . . . . . . . . .
Sockets Communication . . . . . . . . .
IPC Message Queue Communication . . . .
Programming Subsystem Communication with the
SRC . . . . . . . . . . . . . . . .
Programming Subsystems to Receive SRC
Requests . . . . . . . . . . . . . .
Programming Subsystems to Process SRC
Request Packets . . . . . . . . . . .
Processing SRC Status Requests . . . . . .
Programming Subsystems to Send Reply
Packets . . . . . . . . . . . . . .
Programming Subsystems to Return SRC Error
Packets . . . . . . . . . . . . . .
Responding to Trace Requests . . . . . . .
Responding to Refresh Requests . . . . . .
Defining Your Subsystem to the SRC . . . . .
List of Additional SRC Subroutines . . . . . .
739
740
740
740
740
741
741
744
745
746
747
748
749
750
750
752
753
754
755
756
756
757
757
Chapter 27. Trace Facility . . . . . . 759
Trace Facility Commands . . . .
Trace Facility Calls and Subroutines .
Trace Facility Files . . . . . . .
Trace Event Data . . . . . . .
Trace Hook Identifiers . . . .
Hook Types . . . . . . . .
Trace Facility Generic Trace Channels
Trace Hook IDs: 001 through 10A .
001 : HKWD TRACE TRCON . .
002 : HKWD TRACE TRCOFF. .
003 : HKWD TRACE HEADER .
004 : HKWD TRACE NULL . .
005 : HKWD TRACE LWRAP . .
006 : HKWD TRACE TWRAP . .
007 : HKWD TRACE UNDEFINED
100 : HKWD KERN FLIH . . .
101 : HKWD KERN SVC . . .
102 : HKWD KERN SLIH . . .
103 : HKWD KERN SLIHRET . .
104 : HKWD KERN SYSCRET . .
105 : HKWD KERN LVM . . .
106 : HKWD KERN DISPATCH .
107 : HKWD LFS LOOKUP. . .
108 : HKWD SYSC LFS . . . .
10A : HKWD KERN PFS . . .
Trace Hook IDs: 10B through 14E. .
10B : HKWD KERN LVMSIMP .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
759
760
761
761
761
761
761
762
762
762
762
762
763
763
763
763
764
764
764
764
764
766
766
766
767
767
768
10C : HKWD KERN IDLE . . . .
10F : HKWD KERN EOF . . . .
110 : HKWD KERN STDERR . . .
112 : HKWD KERN LOCK . . . .
113 : HKWD KERN UNLOCK . . .
114 : HKWD KERN LOCKALLOC .
115 : HKWD KERN SETRECURSIVE
116 : HKWD KERN XMALLOC . .
117 : HKWD KERN XMFREE . . .
118 : HKWD KERN FORKCOPY . .
119 : HKWD KERN SENDSIGNAL .
11A : HKWD KERN RCVSIGNAL .
11B : HKWD KERN LOCKL . . .
11C : HKWD KERN P SLIH . . .
11D : HKWD KERN SIG SLIH. . .
11E : HKWD KERN ISSIG . . . .
11F : HKWD KERN SORQ . . . .
120 : HKWD SYSC ACCESS . . .
121 : HKWD SYSC ACCT . . . .
122 : HKWD SYSC ALARM . . .
12E : HKWD SYSC CLOSE . . . .
134 : HKWD SYSC EXECVE . . .
135 : HKWD SYSC EXIT. . . . .
139 : HKWD SYSC FORK . . . .
145 : HKWD SYSC GETPGRP . . .
146 : HKWD SYSC GETPID . . .
147 : HKWD SYSC GETPPID . . .
14C : HKWD SYSC IOCTL . . . .
14E : HKWD SYSC KILL . . . .
Trace Hook IDs: 152 through 19C. . .
152 : HKWD SYSC LOCKF . . . .
154 : HKWD SYSC LSEEK . . . .
15F : HKWD SYSC PIPE. . . . .
160 : HKWD SYSC PLOCK . . . .
169 : HKWD SYSC SBREAK . . .
16E : HKWD SYSC SETPGRP . . .
16F : HKWD SYSC SETPRIO . . .
180 : HKWD SYSC SIGACTION . .
181 : HKWD SYSC SIGCLEANUP .
18E : HKWD SYSC TIMES . . . .
18F : HKWD SYSC ULIMIT . . .
195 : HKWD SYSC USRINFO . . .
19B : HKWD SYSC WAIT . . . .
Trace Hook IDs: 1A4 through 1BF . .
1A4 : HKWD SYSC GETRLIMIT . .
1A5 : HKWD SYSC SETRLIMIT . .
1A6 : HKWD SYSC GETRUSAGE .
1A7 : HKWD SYSC GETPRIORITY .
1A8 : HKWD SYSC SETPRIORITY .
1A9 : HKWD SYSC ABSINTERVAL .
1AA : HKWD SYSC GETINTERVAL.
1AB : HKWD SYSC GETTIMER . .
1AC : HKWD SYSC INCINTERVAL .
1AD : HKWD SYSC RESTIMER . .
1AE : HKWD SYSC RESABS . . .
1AF : HKWD SYSC RESINC . . .
1B0 : HKWD VMM ASSIGN . . .
1B1 : HKWD VMM DELETE . . .
1B2 : HKWD VMM PGEXCT . . .
1B3 : HKWD VMM PROTEXCT . .
1B4 : HKWD VMM LOCKEXCT . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Contents
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
769
769
769
770
770
770
771
771
771
771
771
772
772
772
772
772
772
773
773
773
773
773
774
774
774
774
774
774
775
775
775
775
776
776
776
776
777
777
777
777
778
778
778
778
779
779
779
779
780
780
780
780
780
780
781
781
781
782
782
782
782
xiii
1B5 : HKWD VMM RECLAIM . . . .
1B6 : HKWD VMM GETPARENT . . .
1B7 : HKWD VMN COPYPARENT . . .
1B8 : HKWD VMN VMAP . . . . . .
1B9 : HKWD VMN ZFOD . . . . . .
1BA : HKWD VMN SIO . . . . . . .
1BB : HKWD VMM SEGCREATE. . . .
1BC : HKWD VMM SEGDELETE. . . .
1BD : HKWD VMM DALLOC. . . . .
1BE : HKWD VMM PFEND . . . . .
1BF : HKWD VMM EXCEPT . . . . .
Trace Hook IDs: 1C8 through 1CE . . . .
1C8 : HKWD DD PPDD . . . . . . .
1C9 : HKWD DD CDDD . . . . . .
1CA : HKWD DD TAPEDD . . . . .
1CD : HKWD DD ENTDD . . . . . .
1CE : HKWD DD TOKDD . . . . . .
Trace Hook IDs: 1CF through 211. . . . .
1CF : HKWD DD C327DD . . . . . .
1D1 : HKWD RAS ERRLG . . . . . .
1D2 : HKWD RAS DUMP . . . . . .
1F0 : HKWD SYSC SETTIMER . . . .
200 : HKWD KERN RESUME . . . . .
20E: HKWD KERN LOCKL . . . . .
20F: HKWD KERN UNLOCKL . . . .
211 : HKWD NFS VOPSRW . . . . .
Trace Hook IDs: 212 through 220 . . . . .
212 : HKWD NFS VOPS . . . . . . .
213 : HKWD NFS RFSRW . . . . . .
214 : HKWD NFS RFS . . . . . . .
215 : HKWD NFS DISPATCH . . . . .
216 : HKWD NFS CALL. . . . . . .
218 : HKWD RPC LOCKD . . . . . .
220 : HKWD DD FDDD . . . . . . .
Trace Hook IDs: 221 through 223 . . . . .
221 : HKWD DD SCDISKDD . . . . .
222 : HKWD DD BADISKDD . . . . .
223 : HKWD DD SCSIDD . . . . . .
Trace Hook IDs: 224 through 226 . . . . .
224 : HKWD DD MPQPDD . . . . .
225 : HKWD DD X25DD . . . . . .
226 : HKWD DD GIO . . . . . . .
Trace Hook IDs: 230 through 233 . . . . .
230: HKWD PTHREAD MUTEX LOCK. .
231: HKWD PTHREAD MUTEX UNLOCK
232: HKWD PTHREAD SPIN LOCK . .
233: HKWD PTHREAD SPIN UNLOCK .
Trace Hook IDs: 240 through 252 . . . . .
240 : HKWD SYSX DLC START . . . .
241 : HKWD SYSX DLC HALT . . . .
242 : HKWD SYSX DLC TIMER . . . .
243 : HKWD SYSX DLC XMIT . . . .
244 : HKWD SYSX DLC RECV . . . .
245 : HKWD SYSX DLC PERF. . . . .
246 : HKWD SYSX DLC MONITOR . . .
251 : HKWD NETERR . . . . . . .
252 : HKWD SYSC TCPIP . . . . . .
Trace Hook IDs: 253 through 25A . . . .
253 : HKWD SOCKET . . . . . . .
254 : HKWD MBUF . . . . . . . .
255 : HKWD IFEN. . . . . . . . .
xiv
General Programming Concepts
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
783
783
783
783
784
784
784
785
785
785
785
786
786
787
788
789
790
790
790
791
792
794
794
794
794
794
795
795
796
797
798
799
799
800
801
801
803
804
806
806
809
813
813
813
814
814
814
814
814
815
816
816
817
817
818
819
821
822
822
823
824
256 : HKWD IFTR . . . . . . . .
257 : HKWD IFET . . . . . . . .
258 : HKWD IFXT . . . . . . . .
259 : HKWD IFSL . . . . . . . .
25A : HKWD TCPDBG . . . . . .
Trace Hook IDs: 271 through 280 . . . .
271: HKWD SNA API . . . . . .
280: HKWD HIA . . . . . . . .
Trace Hook IDs: 301 through 315 . . . .
301: HKWD KERN ASSERTWAIT . .
302: HKWD KERN CLEARWAIT . . .
303: HKWD KERN THREADBLOCK .
304: HKWD KERN EMPSLEEP . . .
305: HKWD KERN EWAKEUPONE . .
306: HKWD SYSC CRTHREAD . . .
307: HKWD KERN KTHREADSTART .
308 : HKWD SYSC TERMTHREAD . .
309 : HKWD KERN KSUSPEND . . .
310 : HKWD SYSC THREADSETSTATE
311 : HKWD SYSC THREADTERM ACK
312 : HKWD SYSC THREADSETSCHED
313 : HKWD KERN TIDSIG . . . .
314 : HKWD KERN WAITLOCK . . .
315 : HKWD KERN WAKEUPLOCK .
Trace Hook IDs: 3C5 through 3E2 . . .
3c5 : HKWD SYSC IPCACCESS . . .
3c6 : HKWD SYSC IPCGET . . . .
3c7 : HKWD SYSC MSGCONV . . .
3c8 : HKWD SYSC MSGCTL . . . .
3c9 : HKWD SYSC MSGGET . . . .
3ca : HKWD SYSC MSGRCV . . . .
3cb : HKWD SYSC MSGSELECT . . .
3cc : HKWD SYSC MSGSND . . . .
3cd : HKWD SYSC MSGXRCV . . .
3ce : HKWD SYSC SEMCONV . . .
3cf : HKWD SYSC SEMCTL . . . .
3d0 : HKWD SYSC SEMGET . . . .
3d1 : HKWD SYSC SEMOP . . . .
3d2 : HKWD SYSC SEM. . . . . .
3d3 : HKWD SYSC SHMAT . . . .
3d4 : HKWD SYSC SHMCONV . . .
3d5 : HKWD SYSC SHMCTL . . . .
3d6 : HKWD SYSC SHMDT . . . .
3d7 : HKWD SYSC SHMGET . . . .
3d8 : HKWD SYSC MADVISE . . . .
3d9 : HKWD SYSC MINCORE . . .
3da : HKWD SYSC MMAP . . . . .
3db : HKWD SYSC MPROTECT . . .
3dc : HKWD SYSC MSYNC . . . .
3dd : HKWD SYSC MUNMAP . . .
3de : HKWD SYSC MVALID . . . .
3df : HKWD SYSC MSEM_INIT . . .
3e0 : HKWD SYSC MSEM_LOCK . .
3e1 : HKWD SYSC MSEM_REMOVE .
3e2 : HKWD SYSC MSEM_UNLOCK .
Trace Hook IDs: 401 . . . . . . . .
401 : HKWD TTY TTY . . . . . .
Trace Hook IDs: 402 . . . . . . . .
402 : HKWD TTY PTY . . . . . .
Trace Hook IDs: 403 . . . . . . . .
403 : HKWD TTY RS . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
825
826
827
828
829
830
830
832
839
839
840
840
840
840
841
841
841
841
841
842
842
842
843
843
843
843
843
844
844
844
844
845
845
845
845
845
846
846
846
846
847
847
847
847
847
848
848
848
848
848
849
849
849
849
850
850
850
855
855
860
860
Trace Hook IDs: 404 . . . . . . . .
404 : HKWD TTY LION . . . . . .
Trace Hook IDs: 405 . . . . . . . .
405 : HKWD TTY HFT . . . . . .
Trace Hook IDs: 406 . . . . . . . .
406 : HKWD TTY RTS . . . . . .
Trace Hook IDs: 407 . . . . . . . .
407 : HKWD TTY XON . . . . . .
Trace Hook IDs: 408 . . . . . . . .
408 : HKWD TTY DTR . . . . . .
Trace Hook IDs: 409 . . . . . . . .
409 : HKWD TTY DTRO . . . . .
Trace Hook IDs: 411 through 418 . . . .
411: HKWD STTY STRTTY . . . . .
412: HKWD STTY LDTERM . . . .
413: HKWD STTY SPTR . . . . . .
414: HKWD STTY NLS . . . . . .
415: HKWD STTY PTY . . . . . .
416: HKWD STTY RS. . . . . . .
417: HKWD STTY LION. . . . . .
418: HKWD STTY CXMA . . . . .
Trace Hook IDs: 460 through 46E . . . .
460: HKWD KERN ASSERTWAIT . .
461: HKWD KERN CLEARWAIT . . .
462: HKWD KERN THREADBLOCK .
463: HKWD KERN EMPSLEEP . . .
464: HKWD KERN EWAKEUPONE . .
465: HKWD SYSC CRTHREAD . . .
466: HKWD KERN KTHREADSTART .
467: HKWD SYSC TERMTHREAD . .
468: HKWD KERN KSUSPEND . . .
469: HKWD SYSC THREADSETSTATE .
46A: HKWD SYSC THREADTERM ACK
46B: HKWD SYSC THREADSETSCHED
46C: HKWD KERN TIDSIG . . . .
46D: HKWD KERN WAITLOCK . . .
46E: HKWD KERN WAKEUPLOCK . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
865
865
870
870
875
875
880
880
885
885
890
890
894
894
895
896
898
900
901
905
909
913
913
913
914
914
914
914
915
915
915
915
916
916
916
916
916
Chapter 28. tty Subsystem. . . . . . 919
TTY Subsystem Objectives
tty Subsystem Modules .
tty Drivers . . . .
Line Disciplines . .
Converter Modules .
TTY Subsystem Structure
Common Services . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
919
919
919
920
920
920
922
Hardware Control Services . . . . .
Flow Control Services . . . . . .
Terminal Information and Control . .
Window and Terminal Size Services . .
Process Group Management Services .
Buffer Size Operations . . . . . .
Synchronization . . . . . . . . .
tty Revision History . . . . . . . .
tty Structure . . . . . . . . . .
Unsupported ioctl Operations . . . .
Supported ioctl Operations . . . . .
Line Discipline Module (ldterm) . . . .
Terminal Parameters . . . . . . .
Process Group Session Management (Job
Control) . . . . . . . . . . .
Terminal Access Control . . . . . .
Reading Data and Input Processing . .
Writing Data and Output Processing .
Modem Management . . . . . . .
Closing a Terminal Device File . . .
Converter Modules . . . . . . . .
NLS Module . . . . . . . . .
SJIS Modules . . . . . . . . .
TTY Drivers . . . . . . . . . . .
Asynchronous Line Drivers. . . . .
Pseudo-Terminal Driver . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
923
923
923
923
924
924
924
924
924
925
926
926
926
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
927
927
927
930
930
930
930
930
931
931
931
932
Chapter 29. High-Resolution Time
Measurements Using PowerPC Time
Base or POWER Real-Time Clock. . . 933
Chapter 30. Loader Domains . . . . . 935
Using Loader Domains . . . . .
Creating/Deleting Loader Domains .
.
.
.
.
.
.
.
.
. 935
. 937
Chapter 31. Power
Management-Aware Application
Program . . . . . . . . . . . . . 939
Index . . . . . . . . . . . . . . . 941
Readers’ Comments — We’d Like to
Hear from You . . . . . . . . . . . 953
Contents
xv
xvi
General Programming Concepts
About This Book
This book introduces you to the programming tools and interfaces available for
writing and debugging application programs using the AIX operating system.
Who Should Use This Book
This book is intended for programmers who write and debug application programs
on the AIX operating system. Users of this book should be familiar with the C
programming language and AIX usage (entering commands, creating and deleting
files, editing files, and moving around in the file system).
How to Use This Book
Highlighting
The following highlighting conventions are used in this book:
Bold
Italics
Monospace
Identifies commands, subroutines, keywords, files, structures, directories, and
other items whose names are predefined by the system. Also identifies
graphical objects such as buttons, labels, and icons that the user selects.
Identifies parameters whose actual names or values are to be supplied by the
user.
Identifies examples of specific data values, examples of text similar to what
you might see displayed, examples of portions of program code similar to
what you might write as a programmer, messages from the system, or
information you should actually type.
ISO 9000
ISO 9000 registered quality systems were used in the development and
manufacturing of this product.
AIX 32-Bit and 64-Bit Support for the UNIX98 Specification
Beginning with AIX Version 4.3, the operating system is designed to support the
X/Open UNIX98 Specification for portability of UNIX-based operating systems.
Many new interfaces, and some current ones, have been added or enhanced to
meet this specification. Making AIX Version 4.3 even more open and portable for
applications.
At the same time, compatibility with previous AIX releases is preserved. This is
accomplished by the creation of a new environment variable, which can be used to
set the system environment on a per-system, per-user, or per-process basis.
To determine the proper way to develop a UNIX98-portable application, you may
need to refer to the X/Open UNIX98 Specification, which can be obtained on a
CD-ROM by ordering the printed copy of AIX Version 4.3 Commands Reference,
order number SBOF-1877, or by ordering Go Solo: How to Implement and Go Solo
with the Single Unix Specification, order number SR28-5705, a book which includes
the X/Open UNIX98 Specification on a CD-ROM.
© Copyright IBM Corp. 1997, 1999
xvii
Related Publications
The following books contain information about or related to writing programs:
v AIX and Related Products Documentation Overview, order number SC23-2456.
v AIX Version 4.3 Kernel Extensions and Device Support Programming Concepts, order
number SC23-4125.
v AIX Version 4.3 Communications Programming Concepts, order number SC23-4124.
v AIX Version 4.3 AIXwindows Programming Guide, available on the AIX Version 4.3
Base Documentation CD only.
v AIX Versions 3.2 and 4 Performance Tuning Guide, order number SC23-2365.
v AIX Version 4.3 System Management Guide: Operating System and Devices, order
number SC23-4126.
v AIX Version 4.3 System Management Guide: Communications and Networks, order
number SC23-4127.
v AIX Version 4.3 Commands Reference, order number SBOF-1877.
v AIX Version 4.3 Technical Reference, order number SBOF-1878.
v AIX Version 4.3 Files Reference, order number SC23-4168.
v AIX Version 4.3 Assembler Language Reference, available on the AIX Version 4.3
Extended Documentation CD only.
v AIX Version 4.3 INed Editor User’s Guide, available on the AIX Version 4.3 Base
Documentation CD only.
Bach, Maurice. The Design of the UNIX Operating System. Englewood, NJ:
Prentice-Hall Press,1986.
Leffler, Samuel J., McKusic, Marshall Kirk, Karels, Michael J., Quarterman, John S.
The Design and Implementation of the 4.3 BSD UNIX Operating System. USA:
Addison-Wesley Publishing Company, 1989.
Kernighan, Brian W., Pike, Rob. The UNIX Programming Environment. Englewood
Cliffs, NJ: Prentice-Hall Press, 1984.
Levine, John R., Tony Mason, and Doug Brown. lex & yacc. Sebastopol, CA:
O’Reilly & Associates, 1992.
Dougherty, Dale. sed & awk. Sebastopol, CA: O’Reilly & Associates, 1990.
Oram, Andrew, and Steve Talbott. Managing Projects with make. Sebastopol, CA:
O’Reilly & Associates, 1991.
X/Open Company, Ltd. and Prentice Hall, Inc. Go Solo: How to Implement and Go
Solo with the Single Unix Specification. Order Number SR28-5705.
Ordering Publications
You can order publications from your sales representative or from your point of
sale.
To order additional copies of this book, use order number SC23-4128.
Use AIX and Related Products Documentation Overview for information on related
publications and how to obtain them.
xviii
General Programming Concepts
Chapter 1. Tools and Utilities
This chapter provides an overview of the operating system tools and utilities that
you can use to develop C language programs.
The AIX Operating System provides many tools to help you develop C language
programs. To access these tools, you enter a command on the command line. The
tools provide help in the following programming areas:
v “Entering a Program into the System”
v “Checking a Program”
v “Compiling and Linking a Program”
v “Correcting Errors in a Program” on page 2
v “Building and Maintaining a Program” on page 2
“Subroutines” on page 3 and “Shell Commands” on page 3 are provided for use in
a C language program.
Entering a Program into the System
The system has a line editor called ed for use in entering a program into a file. The
system also has the full-screen editors INed (or e) and vi. These editors display
one full screen of data at a time and allow interactive editing of a file.
Checking a Program
The following commands allow you to check the format of a program for
consistency and accuracy:
lint
cb
cflow
cxref
Checks for syntax and data type errors in a C language source program. The lint
command checks these areas of a program more carefully than the C language
compiler does, and displays many messages that point out possible problems.
Reformats a C language source program into a consistent format that uses
indentation levels to show the structure of the program.
Generates a diagram of the logic flow of a C language source program.
Generates a list of all external references for each module of a C language source
program, including where the reference is resolved (if it is resolved in the
program).
Compiling and Linking a Program
To make source code into a program that the system can run, you need to process
the source file with a compiler program and a linkage editor.
A compiler is a program that reads program text from a file and changes the
programming language in that file to a form that the system understands. The
linkage editor connects program modules together and determines how to put the
finished program into memory. To create this final form of the program, the system
does the following:
v If a file contains compiler source code, the compiler translates it into object code.
v If a file contains assembler language, the assembler translates it into object code.
© Copyright IBM Corp. 1997, 1999
1
v The linkage editor links the object files created in the previous step with any
other object files specified in the compiler command.
Other programming languages available for use on the operating system include
the FORTRAN, Pascal, and Assembler languages. Refer to documentation on these
programming languages for information on compiling and linking programs
written in them.
You can write parts of a program in different languages and have one main routine
call and start the separate routines to execute, or use the cc program to both
assemble and link the program.
Correcting Errors in a Program
The following debugging tools are available for use:
v “dbx Symbolic Debug Program Overview” on page 70 symbolic debugger can be
used to debug programs written in C language, Pascal, FORTRAN, and
Assembler language.
v “adb Debug Program Overview” on page 37 debugger provides subcommands
to examine, debug, and repair executable binary files and to examine non-ASCII
data files.
v Kernel Debug Program can help to determine errors in code running in the
kernel. The primary application of this debugger is debugging device drivers.
v The Trace Facility helps isolate system problems by monitoring selected system
events.
When syntax errors or parameter naming inconsistencies are discovered in a
program file, a text editor or string-searching and string-editing programs can be
used to locate and change strings in the file. String-searching and string-editing
programs include the grep, sed, and awk commands. To make many changes in
one or more program files, you can include the commands in a shell program and
then run the shell program to locate and change the code in the files.
Building and Maintaining a Program
Two facilities are provided to help you control program changes and build a
program from many source modules. These commands can be particularly useful
in software development environments in which many source modules are
produced.
The make command builds a program from source modules. Since the make
command compiles only those modules changed since the last build, its use can
reduce compilation time when many source modules must be processed.
The Source Code Control System (SCCS) allows you to maintain separate versions
of a program without storing separate, complete copies of each version. (See
“Chapter 23. Source Code Control System (SCCS)” on page 663 for more
information.) The use of SCCS can reduce storage requirements and help in
tracking the development of a project that requires keeping many versions of large
programs.
2
General Programming Concepts
Subroutines
Subroutines from system libraries handle many complex or repetitive programming
situations so that you can concentrate on unique programming situations. See
“Chapter 24. Subroutines, Example Programs, and Libraries” on page 669 for
information on using subroutines and for lists of many of the subroutines available
on the system.
Shell Commands
You can include the functions of many of the shell commands in a C language
program. Any shell command used in a program must be available on all systems
that use the program.
You can then use the fork and exec subroutines in a program to run the command
as a process in a part of the system that is separate from the program. The system
subroutine also runs a shell command in a program, and the popen subroutine
uses shell filters.
AIX Support for the X/Open UNIX95 Specification
Beginning with AIX Version 4.2, the operating system is designed to support the
X/Open UNIX95 Specification for portability of UNIX-based operating systems.
Many new interfaces, and some current ones, have been added or enhanced to
meet this specification. Beginning with Version 4.2, AIX is more open and portable
for applications.
The default AIX environment is one that maintains compatibility with previous
AIX releases. This is accomplished by the creation of a new environment variable,
which can be used to set the system environment on a per-system, per-user, or
per-process basis. To obtain an environment designed to conform to the UNIX95
specification, the environment variable XPG_SUS_ENV must be assigned the value
ON.
When XPG_SUS_ENV is set, every program in that environment will operate in
the UNIX95-specified operating system environment. It is possible that some
applications compiled for the AIX environment (perhaps for an earlier version of
AIX) will not operate correctly when XPG_SUS_ENV is set. If XPG_SUS_ENV is
set to any other value, or is unset, the default AIX behavior will be used.
AIX 32-Bit and 64-Bit Support for the UNIX98 Specification
Beginning with AIX Version 4.3, the operating system is designed to support the
X/Open UNIX98 Specification for portability of UNIX-based operating systems.
Many new interfaces, and some current ones, have been added or enhanced to
meet this specification, making AIX Version 4.3 even more open and portable for
applications.
At the same time, compatibility with previous AIX releases is preserved. This is
accomplished by the creation of a new environment variable, which can be used to
set the system environment on a per-system, per-user, or per-process basis.
Chapter 1. Tools and Utilities
3
4
General Programming Concepts
Chapter 2. The Curses Library
The Curses library provides a set of functions that enable you to manipulate a
terminal’s display regardless of the terminal type. Throughout this documentation,
the Curses library is referred to as curses.
The basis of curses programming is the window data structure. Using this
structure, you can manipulate data on a terminal’s display. You can instruct curses
to treat the entire terminal display as one large window or you can create multiple
windows on the display. The windows can be different sizes and can overlap one
another. The following figure shows a typical curses application with a single large
window and one subwindow
TTY
Move cursor to desired item and press Enter.
List All Defined TTYs
Add a TTY
Moving a TTY to Another Port
Change / Show characteristics of a
TTY
Remove a TTY
Configure a Defined TTY
Generate Error Report
Trace a TTY
TTY Type
Move cursor to desired item and press Enter.
tty rs232 Asynchronous Terminal
tty rs422 Asynchronous Terminal
F1=Help
F8=Image
/=Find
F1=Help
F9
F2=Refresh
F10=Exit
n=Find Next
F3=Cancel
Enter=Do
Each window on a terminal’s display has its own window data structure. This
structure keeps state information about the window such as its size and where it is
located on the display. Curses uses the window data structure to obtain relevant
information it needs to carry out your instructions.
Terminology
When programming with curses, you should be familiar with the following terms:
Term
current character
current line
© Copyright IBM Corp. 1997, 1999
Definition
The character that the logical cursor is currently on.
The line that the logical cursor is currently on.
5
Term
curscr
display
logical cursor
pad
physical cursor
screen
stdscr
window
Definition
A virtual default window provided by curses. The curscr
(current screen) is an internal representation of what currently
appears on the terminal’s external display. Do not modify the
curscr.
A physical display connected to a workstation.
The cursor location within each window. The window data
structure keeps track of the location of its logical cursor.
A type of window that is larger than the dimensions of the
terminal’s display.
The cursor that appears on a display. The workstation uses this
cursor to write to the display. There is only one physical cursor
per display.
The window that fills the entire display. The screen is
synonymous with the stdscr.
A virtual default window (standard screen) provided by curses
that represents the entire display.
A pointer to a C data structure and the graphic representation of
that data structure on the display. A window can be thought of
as a two-dimensional array representing how all or part of the
display looks at any point in time.
Naming Conventions
A single curses subroutine can have two or more versions. Curses subroutines with
multiple versions follow distinct naming conventions that identify the separate
versions. These conventions add a prefix to a standard curses subroutine and
identify what arguments the subroutine requires or what actions take place when
the subroutine is called. The different versions of curses subroutine names use
three prefixes:
Prefix
w
p
mv
Description
Identifies a subroutine that requires a window argument.
Identifies a subroutine that requires a pad argument.
Identifies a subroutine that first performs a move to the program-supplied
coordinates.
Some curses subroutines with multiple versions do not include one of the
preceding prefixes. These subroutines use the curses default window stdscr
(standard screen). The majority of subroutines that use the stdscr are macros
created in the /usr/include/curses.h file using #define statements. The preprocessor
replaces these statements at compilation time. As a result, these macros do not
appear in the compiled assembler code, a trace, a debugger, or the curses source
code.
If a curses subroutine has only a single version, it does not necessarily use stdscr.
For example, the printw subroutine prints a string to the stdscr. The wprintw
subroutine prints a string to a specific window by supplying the window argument.
The mvprintw subroutine moves the specified coordinates to the stdscr and then
performs the same function as the printw subroutine. Likewise, the mvwprintw
subroutine moves the specified coordinates to the specified window and then
performs the same function as the wprintw subroutine.
6
General Programming Concepts
Structure of a Curses Program
In general, a curses program has the following progression:
v Start curses.
v Check for color support (optional).
v Start color (optional).
v Create one or more windows.
v Manipulate windows.
v Destroy one or more windows.
v Stop curses.
Your program does not have to follow this progression exactly.
Return Values
With a few exceptions, all curses subroutines return either the integer value ERR or
the integer value OK. Subroutines that do not follow this convention are noted
appropriately. Subroutines that return pointers always return a null pointer on an
error.
Initializing Curses
initscr
newterm
setupterm
endwin
isendwin
Initializes the curses subroutine library and its data structures
Sets up a new terminal
Sets up the TERMINAL structure for use by curses
Terminates the curses subroutine libraries and their data structures
Returns TRUE if the endwin subroutine has been called without any
subsequent calls to the wrefresh subroutine
You must include the curses.h file at the beginning of any program that calls
curses subroutines. To do this, use the following statement:
#include <curses.h>
Before you can call subroutines that manipulate windows or screens, you must call
the initscr or newterm subroutine. These subroutines first save the terminal’s
settings. These subroutines then call the setupterm subroutine to establish a curses
terminal.
If you need to temporarily suspend curses, use a shell escape or system call for
example. To resume after a temporary escape, you should call the wrefresh or
doupdate subroutine. Before exiting a curses program, you must call the endwin
subroutine. The endwin subroutine restores tty modes, moves the cursor to the
lower left corner of the screen, and resets the terminal into the proper nonvisual
mode.
Most interactive, screen-oriented programs require character-at-a-time input
without echoing the result to the screen. To establish your program with
character-at-a-time input, call the cbreak and noecho subroutines after calling the
initscr subroutine. When accepting this type of input, programs should also call
the following subroutines:
Chapter 2. The Curses Library
7
v nonl subroutine.
v intrflush subroutine with the Window parameter set to the stdscr and the Flag
parameter set to FALSE. The Window parameter is required but ignored. You can
use stdscr as the value of the Window parameter, because stdscr is already
created for you.
v keypad subroutine with the Window parameter set to the stdscr and the Flag
parameter set to TRUE.
The isendwin subroutine is helpful if, for optimization reasons, you don’t want to
call the wrefresh subroutine needlessly. You can determine if the endwin
subroutine was called without any subsequent calls to the wrefresh subroutine by
using the isendwin subroutine.
Windows in the Curses Environment
A curses program manipulates windows that appear on a terminal’s display. A
window can be as large as the entire display or as small as a single character in
length and height.
Note: Pads are the exception. A pad is a window that is not restricted by the
size of the screen. (See “Pads” on page 10 for more information.)
The following figure shows the different types of windows that exist in the curses
environment:
screen (stdscr)
subwindow
subpad
pad
Within a curses program, windows are variables declared as type WINDOW. The
WINDOW data type is defined in the /usr/include/curses.h file as a C data
structure. You create a window by allocating a portion of a machine’s memory for
a window structure. This structure describes the characteristics of the window.
When a program changes the window data internally in memory, it must use the
wrefresh subroutine (or equivalent subroutine) to update the external, physical
screen to reflect the internal change in the appropriate window structure.
The Default Window Structure
Curses provides a virtual default window called stdscr. The stdscr represents, in
memory, the entire terminal display. The stdscr window structure is created
automatically when the curses library is initialized and it describes the display.
When the library is initialized, the length and width variables are set to the length
and width of the physical display.
8
General Programming Concepts
Programs that use the stdscr first manipulate the stdscr and then call the refresh
subroutine to refresh the external display so that it matches the stdscr window.
In addition to the stdscr, you can define your own windows. These windows are
known as user-defined windows to distinguish them from the stdscr. Like the
stdscr, user-defined windows exist in machine memory as structures. Except for the
amount of memory available to a program, there is no limit to the number of
windows you can create. A curses program can manipulate the default window,
user-defined windows, or both.
The Current Window Structure
Curses supports another virtual window called curscr (current screen). The curscr
window is an internal representation of what currently appears on the terminal’s
external display.
When a program requires the external representation to match the internal
representation, it must call a subroutine, such as the wrefresh subroutine, to
update the physical display (or the refresh subroutine if the program is working
with the stdscr).
refresh, or wrefresh
Updates the terminal and curscr to reflect changes made to
a window.
The curscr is reserved for internal use by curses. You should not manipulate the
curscr.
Subwindows
Curses also allows you to construct subwindows. Subwindows are rectangular
portions within other windows. A subwindow is also of type WINDOW. The
window that contains a subwindow is known as the subwindow’s parent and the
subwindow is known as the containing window’s child. The following figure
demonstrates the parent child relationship.
parent (window)
child
(subwindow)
Changes to either the parent window or the child window within the area
overlapped by the subwindow are made to both windows. After modifying a
subwindow, you should call the touchline or touchwin subroutine on the parent
window before refreshing it.
touchline
Forces a range of lines to be refreshed at the next call to the wrefresh
subroutine.
Chapter 2. The Curses Library
9
touchwin
Forces every character in a window’s character array to be refreshed at the
next call of the wrefresh subroutine. The touchwin subroutine does not save
optimization information. This subroutine is useful with overlapping
windows.
A refresh called on the parent refreshes the children as well.
A subwindow can also be a parent window. The process of layering windows
inside of windows is called nesting.
Before you can delete a parent window, you must first delete all of its children
using the delwin subroutine.
delwin
Removes a window data structure.
Curses returns an error if you try to delete a window before removing all of its
children.
Pads
A pad is a type of window that is not restricted by the terminal’s display size or
associated with a particular part of the display. Because a pad is usually larger
than the physical display, only a portion of a pad is visible to the user at a given
time.
Use pads if you have a large amount of related data that you want to keep all
together in one window but you do not need to display all of the data at once.
Windows within pads are known as subpads. Subpads are positioned within a pad
at coordinates relative to the parent pad. This placement differs from subwindows
which are positioned using screen coordinates.
prefresh or pnoutrefresh
Updates the terminal and curscr to reflect changes
made to a pad.
Unlike other windows, scrolling or echoing of input does not automatically refresh
a pad. Like subwindows, when changing the image of a subpad, you must call
either the touchline or touchwin subroutine on the parent pad before refreshing
the parent.
You can use all the curses subroutines with pads except for the newwin, subwin,
wrefresh, and wnoutrefresh subroutines. These subroutines are replaced with the
newpad, subpad, prefresh, and pnoutrefresh subroutines.
Manipulating Window Data with Curses
When curses is initialized, the stdscr is provided automatically. You can manipulate
the stdscr using the curses subroutine library or you can create your own,
user-defined windows. This section discusses the following topics as they relate to
manipulating window data:
Creating Windows
You can create your own window using the newwin subroutine.
newwin
10
General Programming Concepts
Creates a new window data structure.
Each time you call the newwin subroutine, curses allocates a new window
structure in memory. This structure contains all the information associated with the
new window. Curses does not put a limit on the number of windows you can
create. The number of nested subwindows is limited to the amount of memory
available up to the value of SHRT_MAX as defined in the /usr/include/limits.h file.
You can change windows without regard to the order in which they were created.
Updates to the terminal’s display occur through calls to the wrefresh subroutine.
Subwindows
subwin
Creates a subwindow of an existing window.
You must supply coordinates for the subwindow relative to the terminal’s display.
The subwindow must fit within the bounds of the parent window; otherwise, a
null value is returned.
Pads
newpad
subpad
Creates a new pad data structure.
Creates and returns a pointer to a subpad within a pad.
The new subpad is positioned relative to its parent.
Removing Windows, Pads, and Subwindows
To remove a window, pad, or subwindow, use the delwin subroutine. Before you
can delete a window or pad, you must have already deleted its children;
otherwise, the delwin subroutine returns an error.
Changing the Screen or Window Images
When curses subroutines change the appearance of a window, the internal
representation of the window is updated while the display remains unchanged
until the next call to the wrefresh subroutine. The wrefresh subroutine uses the
information in the window structure to update the display.
Refreshing Windows
Any time you write output to a window or pad structure, you must refresh the
terminal’s display to match the internal representation. A refresh does the
following:
v Compares the contents of the curscr to the contents of the user-defined or stdscr
v Updates the curscr structure to match the user-defined or stdscr
v Redraws the portion of the physical display that changed
refresh, or wrefresh
wnoutrefresh or doupdate
Updates the terminal and curscr to reflect changes
made to a window.
Updates the designated windows and outputs them
all at once to the terminal. These subroutines are
useful for faster response when there are multiple
updates.
The refresh and wrefresh subroutines first call the wnoutrefresh subroutine to
copy the window being refreshed to the current screen. They then call the
doupdate subroutine to update the display.
Chapter 2. The Curses Library
11
If you need to refresh multiple windows at the same time, use one of the two
available methods. You can use a series of calls to the wrefresh subroutine that
result in alternating calls to the wnoutrefresh and doupdate subroutines. You can
also call the wnoutrefresh subroutine once for each window and then call the
doupdate subroutine once. With the second method, only one burst of output is
sent to the display.
Subroutines Used for Refreshing Pads
The prefresh and pnoutrefresh subroutines are similar to the wrefresh and
wnoutrefresh subroutines.
prefresh or pnoutrefresh
Updates the terminal and curscr to reflect changes
made to a user-defined pad.
The prefresh subroutine updates both the current screen and the physical display,
while the pnoutrefresh subroutine updates curscr to reflect changes made to a
user-defined pad. Because pads instead of windows are involved, these
subroutines require additional parameters to indicate which part of the pad and
screen are involved.
Refreshing Areas that Have Not Changed
During a refresh, only those areas that have changed are redrawn on the display. It
is possible to refresh areas of the display that have not changed using the
touchwin and touchline subroutines.
touchline
touchwin
Forces a range of lines to be refreshed at the next call to the wrefresh
subroutine.
Forces every character in a window’s character array to be refreshed at the
next call of the wrefresh subroutine. The touchwin subroutine does not save
optimization information. This subroutine is useful with overlapping
windows.
Combining the touchwin and wrefresh subroutines is helpful when dealing with
subwindows or overlapping windows. To bring a window forward from behind
another window, call the touchwin subroutine followed by the wrefresh
subroutine.
Garbled Displays
If text is sent to the terminal’s display with a noncurses subroutine, such as the
echo or printf subroutine, the external window can become garbled. In this case,
the display changes, but the current screen is not updated to reflect these changes.
Problems can arise when a refresh is called on the garbled screen because after a
screen is garbled, there is no difference between the window being refreshed and
the current screen structure. As a result, spaces on the display caused by garbled
text are not changed.
A similar problem can also occur when a window is moved. The characters sent to
the display with the noncurses subroutines do not move with the window
internally.
If the screen becomeS garbled, call the wrefresh subroutine on the curscr to update
the display to reflect the current physical display.
Manipulating Window Content
After a window or subwindow is created, programs often must manipulate them
in some way.
12
General Programming Concepts
Draws a box in or around a window.
Provides more precise control over the overlay and
overwrite subroutine.
Indicates to curses that a screen line is garbaged and
should be thrown away before having anything written
over the top of it.
Moves a window or subwindow to a new location.
Copies one window on top of another.
Removes a line from the default screen.
box
copywin
garbagedlines
mvwin
overlay or overwrite
ripoffline
The mvwin subroutine moves a window or subwindow. The box subroutine draws
a box around the edge of a window or subwindow.
To use the overlay and overwrite subroutines, the two windows must overlap.
Also, be aware that the overwrite subroutine is destructive whereas the overlay
subroutine is not. When text is copied from one window to another using the
overwrite subroutine, blank portions from the copied window overwrite any
portions of the window copied to. The overlay subroutine is nondestructive
because it does not copy blank portions from the copied window.
Similar to the overlay and overwrite subroutines, the copywin subroutine allows
you to copy a portion of one window to another. Unlike overlay and overwrite
subroutines, the windows do not have to overlap for you to use the copywin
subroutine.
You can use the ripoffline subroutine to remove a line from the stdscr. If you pass
this subroutine a positive line argument, the specified number of lines is removed
from the top of the stdscr. Otherwise, if you pass the subroutine a negative line
argument, the lines are removed from the bottom of the stdscr.
Finally, you can use the garbagedlines subroutine to discard a specified range of
lines before writing anything new.
Support for Filters
The filter subroutine is provided for curses applications that are filters.
filter
Sets the size of the terminal screen to 1 line.
This subroutine causes curses to operate as if the stdscr was only a single line on
the screen. When running with the filter subroutine, curses does not use any
terminal capabilities that require knowledge of the line that curses is on.
Controlling the Cursor with Curses
In the Curses library, there are two types of cursors:
logical cursor
physical cursor
The cursor location within each window. A window’s data structure
keeps track of the location of its logical cursor. Each window has a
logical cursor.
The display cursor. The workstation uses this cursor to write to the
display. There is only one physical cursor per display.
You can only add to or erase characters at the current cursor location in a window.
The following subroutines are provided for controlling the cursor:
move
Moves the logical cursor associated with the stdscr.
Chapter 2. The Curses Library
13
wmove
getbegyx
getmaxyx
getsyx
getyx
leaveok
mvcur
setsyx
Moves the logical cursor associated with a user-defined window.
Places the beginning coordinates of the window in integer variables y and x.
Places the size of the window in integer variables y and x.
Returns the current coordinates of the virtual screen cursor.
Returns the position of the logical cursor associated with a specified window.
Controls physical cursor placement after a call to the wrefresh subroutine.
Moves the physical cursor.
Sets the virtual screen cursor to the specified coordinate.
After a call to the refresh or wrefresh subroutine, curses places the physical cursor
at the last updated character position in the window. To leave the physical cursor
where it is and not move it after a refresh, call the leaveok subroutine with the
Window parameter set to the desired window and the Flag parameter set to TRUE.
Manipulating Characters with Curses
You can add characters to a curses window using a keyboard or a curses
application. This section provides an overview of the ways you can add, remove,
or change characters that appear in a curses window.
Character Size
Historically, a position on the screen has corresponded to a single stored byte. This
correspondence is no longer true for several reasons:
v Some characters may occupy several columns when displayed on the screen.
v Some characters may be non-spacing characters, defined only in association with
a spacing character.
v The number of bytes to hold a character from the extended character sets
depends on the LC_CTYPE locale category.
Some character sets define multi-column characters that occupy more than one
column position when displayed on the screen.
Writing a character whose width is greater than the width of the destination
window is an error.
Adding Characters to the Screen Image
The Curses library provides a number of subroutines that write text changes to a
window and mark the area to be updated at the next call to the wrefresh
subroutine.
waddch Subroutines
The waddch subroutines overwrite the character at the current logical cursor
location with a specified character. After overwriting, the logical cursor is moved
one space to the right. If the waddch subroutines are called at the right margin,
these subroutines also add an automatic newline character. Additionally, if you call
one of these subroutines at the bottom of a scrolling region and scrollok is enabled,
the region is scrolled up one line. For example, if you added a new line at the
bottom line of a window, the window would scroll up one line.
If the character to add is a tab, newline, or backspace character, curses moves the
cursor appropriately in the window to reflect the addition. Tabs are set at every
eighth column. If the character is a newline, curses first uses the wclrtoeol
subroutine to erase the current line from the logical cursor position to the end of
the line before moving the cursor. The waddch subroutine family is made up of
14
General Programming Concepts
the following:
waddch subroutine
addch macro
mvaddch macro
mvwaddch macro
Adds a character to the user-defined window.
Adds a character to the stdscr.
Moves a character to the specified location before adding it to
the stdscr.
Moves a character to the specified location before adding it to
the user-defined window.
By using the winch and waddch subroutine families together, you can copy text
and video attributes from one place to another. Using the winch subroutine family,
you can retrieve a character and its video attributes. You can then use one of the
waddch subroutines to add the character and its attributes to another location.
You can also use the waddch subroutines to add control characters to a window.
Control characters are drawn in the |X notation.
Note: Calling the winch subroutine on a position in the window containing a
control character does not return the character. Instead, it returns one
character of the control character representation.
Outputting Single, Noncontrol Characters: When outputting single, noncontrol
characters, there is significant performance gain to using the wechochar
subroutines. These subroutines are functionally equivalent to a call to the
corresponding waddchr subroutine followed by the corresponding wrefresh
subroutine. The wechochar subroutines include the wechochar subroutine, the
echochar macro, and the pechochar subroutine.
Some character sets may contain non-spacing characters. (Non-spacing characters
are those, other than the ’ \ 0 ’ character, for which wcwidth ( ) returns a width of
zero.) The application may write non-spacing characters to a window. Every
non-spacing character in a window is associated with a spacing character and
modifies the spacing character. Non-spacing characters in a window cannot be
addressed separately. A non-spacing character is implicitly addressed whenever a
Curses operation affects the spacing character with which the non-spacing
character is associated.
Non-spacing characters do not support attributes. For interfaces that use wide
characters and attributes, the attributes are ignored if the wide character is a
non-spacing character. Multi-column characters have a single set of attributes for
all columns. The association of non-spacing characters with spacing characters can
be controlled by the application using the wide character interfaces. The wide
character string functions provide codeset-dependent association.
Two typical effects of a non-spacing character associated with a spacing character
called c, are as follows:
v The non-spacing character may modify the appearance of c. (For instance, there
may be non-spacing characters that add diacritical marks to characters.
However, there may also be spacing characters with built-in diacritical marks.)
v The non-spacing characters may bridge c to the character following c. (Examples
of this usage are the formation of ligatures and the conversion of characters into
compound display forms, words, or ideograms.)
Implementations may limit the number of non-spacing characters that can be
associated with a spacing character, provided any limit is at least 5.
Chapter 2. The Curses Library
15
Complex Characters
A complex character is a set of associated characters, which may include a spacing
character and may also include any non-spacing characters associated with it. A
spacing complex character is a complex character that includes one spacing
character and any non-spacing characters associated with it. An example of a code
set that has complex characters is ISO/IEC 10646-1:1993.
A complex character can be written to the screen. If the complex character does not
include a spacing character, any non-spacing characters are associated with the
spacing complex character that exists at the specified screen position. When the
application reads information back from the screen, it obtains spacing complex
characters.
The cchar_t data type represents a complex character and its rendition. When a
cchar_t represents a non-spacing complex character (that is, when there is no
spacing character within the complex character), then its rendition is not used;
when it is written to the screen, it uses the rendition specified by the spacing
character already displayed.
An object of type cchar_t can be initialized using setchar( ) and its contents can be
extracted using getchar( ). The behavior of functions that take a cchar_t value that
was not initialized in this way or obtained from a curses function that has a
cchar_t output argument.
Special Characters
Some functions process special characters as specified below.
In functions that do not move the cursor based on the information placed in the
window, these special characters would only be used within a string in order to
affect the placement of subsequent characters; the cursor movement specified
below does not persist in the visible cursor beyond the end of the operation. In
functions that do not move the cursor, these special characters can be used to affect
the placement of subsequent characters and to achieve movement of the physical
cursor.
Backspace
Carriage return
newline
Unless the cursor was already in column 0, Backspace moves the
cursor one cloumn toward the start of the current line and any
characters after the Backspace are added or inserted starting there.
Unless the cursor was already in column 0, Carriage return moves
the cursor to the start of the current line. Any characters after the
Carriage return are added or inserted starting there.
In an add operation, curses adds the background character into
successive columns until reaching the end of the line. Scrolling
occurs, and any characters after the newline character are added,
starting at the beginning of the new line.
In an insert operation, newline erases the remainder of the current
line with the background character, effectively a wclrtoeol ( ), and
moves the cursor to the start of a new line. When scrolling is
enabled, advancing the cursor to a new line may cause scrolling.
Any characters after the newline character are inserted at the
beginning of the new line.
The filter ( ) function may inhibit this processing.
16
General Programming Concepts
Tab
Tab characters in text move subsequent characters to the next
horizontal tab stop. By default, tab stops are in columns 0, 8, 16,
and so on.
In an insert or add operation, curses inserts or adds, respectively,
the background character into successive columns until reaching the
next tab stop. If there are no more tab stops in the current line,
wrapping and scrolling occur.
Control Characters: The curses functions that perform special-character
processing conceptually convert control characters to the ( ’ | ’ ) character followed
by a second character (which is an upper-case letter if it is alphabetic) and write
this string to the window in place of the control character. The funtions that
retrieve text from the window will not retrieve the original control character.
Line Graphics: You can use the following variables to add line-drawing
characters to the screen with the waddch subroutine. When defined for the
terminal, the variable will have the A_ALTCHARSET bit turned on. Otherwise,
the default character listed in the following table will be stored in the variable.
Variable Name
Default Character
Glyph Description
ACS_ULCORNER
+
upper left corner
ACS_LLCORNER
+
lower left corner
ACS_URCORNER
+
upper right corner
ACS_LRCORNER
+
lower right corner
ACS_RTEE
+
right tee
ACS_LTEE
+
left tee
ACS_BTEE
+
bottom tee
ACS_TTEE
+
top tee
ACS_HLINE
—
horizontal line
ACS_VLINE
|
vertical line
ACS_PLUS
+
plus
ACS_S1
-
scan line 1
ACS_S9
_
scan line 9
ACS_DIAMOND
+
diamond
ACS_CKBOARD
:
checker board (stipple)
ACS_DEGREE
,
degree symbol
ACS_PLMINUS
#
plus/minus
ACS_BULLET
o
bullet
ACS_LARROW
<
arrow pointing left
ACS_RARROW
>
arrow pointing right
ACS_DARROW
v
arrow pointing down
ACS_UARROW
|
arrow pointing up
ACS_BOARD
#
board of squares
ACS_LANTERN
#
lantern symbol
ACS_BLOCK
#
solid square block
Chapter 2. The Curses Library
17
waddstr Subroutines
The waddstr subroutines add a null-terminated character string to a window,
starting with the current character. If you are adding a single character, use the
waddch subroutine. Otherwise, use the waddstr subroutine. The following are part
of the waddstr subroutine family:
waddstr subroutine
addstr macro
mvaddstr macro
wmvaddstr macro
Adds a character string to a user-defined window.
Adds a character string to the stdscr.
Moves the logical cursor to a specified location before adding a
character string to the stdscr.
Moves the logical cursor to a specified location before adding a
character string to a user-defined window.
winsch Subroutines
The winsch subroutines insert a specified character before the current character in
a window. All characters to the right of the inserted character are moved one space
to the right. As a result, the rightmost character on the line may be lost. The
positions of the logical and physical cursors do not change after the move. The
winsch subroutines include the following:
winsch subroutine
insch macro
mvinsch macro
mvwinsch macro
Inserts a character in a user-defined window.
Inserts a character in the stdscr.
Moves the logical cursor to a specified location in the stdscr
before inserting a character.
Moves the logical cursor to a specified location in a
user-defined window before inserting a character.
winsertln Subroutines
The winsertln subroutines insert a blank line above the current line in a window.
The insertln subroutine inserts a line in the stdscr. The bottom line of the window
is lost. The winsertln subroutine performs the same action in a user-defined
window.
wprintw Subroutines
The wprintw subroutines replace a series of characters (starting with the current
character) with formatted output. The format is the same as for the printf
command. The following subroutine and macros belong to the printw family:
wprintw subroutine
printw macro
mvprintw macro
mvwprintw macro
Replaces a series of characters in a user-defined window.
Replaces a series of characters in the stdscr.
Moves the logical cursor to a specified location in the stdscr
before replacing any characters.
Moves the logical cursor to a specified location in a
user-defined window before replacing any characters.
The wprintw subroutines make calls to the waddch subroutine to replace
characters.
unctrl Macro
The unctrl macro returns a printable representation of the specified control
character, displayed in the |X notation. The unctrl macro returns print characters
as is.
18
General Programming Concepts
Enabling Text Scrolling
Allows curses to use the hardware insert/delete line
feature.
Enables a window to scroll when the cursor is moved
off the right edge of the last line of a window.
Sets a software scrolling region within a window.
idlok
scrollok
setscrreg or wsetscrreg
Scrolling occurs when a program or user moves a cursor off a window’s bottom
edge. For scrolling to occur, you must first use the scrollok subroutine to enable
scrolling for a window. A window is scrolled if scrolling is enabled and if any of
the following occurs:
v The cursor is moved off the edge of a window.
v A new-line character is encountered on the last line.
v A character is inserted in the last position of the last line.
When a window is scrolled, curses will update both the window and the display.
However, to get the physical scrolling effect on the terminal, you must call the
idlok subroutine with the Flag parameter set to TRUE.
If scrolling is disabled, the cursor is left on the bottom line at the location where
the character was entered.
When scrolling is enabled for a window, you can use the setscrreg subroutines to
create a software scrolling region inside the window. You pass the setscrreg
subroutines values for the top line and bottom line of the region. If setscrreg is
enabled for the region and scrolling is enabled for the window, any attempt to
move off the specified bottom line causes all the lines in the region to scroll up one
line. You can use the setscrreg macro to define a scrolling region in the stdscr.
Otherwise, you use the wsetscrreg subroutine to define scrolling regions in
user-defined windows.
Note: Unlike the idlok subroutine, the setscrreg subroutines have nothing to
do with the use of the physical scrolling region capability that the terminal
may or may not have.
Deleting Characters
You can delete text by replacing it with blank spaces or by removing characters
from a character array and sliding the rest of the characters on the line one space
to the left.
werase Subroutines
erase or werase
Copies blank spaces to every position in a window.
The erase macro copies blank space to every position in the stdscr. The werase
subroutine puts a blank space at every position in a user-defined window. To
delete a single character in a window, use the wdelch subroutine.
wclear Subroutines
clear, or wclear
clearok
Clears the screen and sets a clear flag for the next refresh.
Determines whether curses clears a window on the next call to
the refresh or wrefresh subroutine.
Chapter 2. The Curses Library
19
The wclear subroutines are similar to the werase subroutines. However, in addition
to putting a blank space at every position of a window, the wclear subroutines also
call the wclearok subroutine. As a result, the screen is cleared on the next call to
the wrefresh subroutine.
The wclear subroutine family contains the wclear subroutine, the clear macro, and
the clearok subroutine. The clear macro puts a blank at every position in the
stdscr.
wclrtoeol Subroutines
clrtoeol or wclrtoeol
Erases the current line to the right of the logical cursor.
The clrtoeol macro operates in the stdscr, while the wclrtoeol subroutine performs
the same action within a user-defined window.
wclrtobot Subroutines
clrtobot or wclrtobot
Erases the lines below and to the right of the logical
cursor.
The clrtobot macro operates in the stdscr, while the wclrtobot performs the same
action in a user-defined window.
wdelch Subroutines
wdelch subroutine
delch macro
mvdelch macro
mvwdelch macro
Deletes the current character in a user-defined window.
Deletes the current character from the stdscr.
Moves the logical cursor before deleting a character from the
stdscr.
Moves the logical cursor before deleting a character from a
user-defined window.
The wdelch subroutines delete the current character and move all the characters to
the right of the current character on the current line one position to the left. The
last character in the line is filled with a blank. The delch subroutine family consists
of the following subroutine and macros:
wdeleteln Subroutines
deleteln or wdeleteln
Deletes the current line.
The wdeleteln subroutines delete the current line and move all lines below the
current line up one line. This clears the window’s bottom line.
Getting Characters
Your program can retrieve characters from the keyboard or from the display. The
wgetch subroutines retrieve characters from the keyboard. The winch subroutines
retrieve characters from the display.
wgetch Subroutines
The wgetch subroutines read characters from the keyboard attached to the terminal
associated with the window. Before getting a character, these subroutines call the
wrefresh subroutines if anything in the window has changed: for example, if the
cursor has moved or text has changed. If the wgetch subroutine encounters a
Ctrl-D key sequence during processing, it returns.
20
General Programming Concepts
The following belong to the wgetch subroutine family:
wgetch subroutine
getch macro
mvgetch macro
mvwgetch macro
Gets a character from a user-defined window.
Gets a character from the stdscr.
Moves the cursor before getting a character from the stdscr.
Moves the cursor before getting a character from a user-defined
window.
To place a character previously obtained by a call to the wgetch subroutine back in
the input queue, use the ungetch subroutine. The character is retrieved by the next
call to the wgetch subroutine.
The Importance of Terminal Modes: The output of the wgetch subroutines is, in
part, determined by the mode of the terminal. The following list describes the
action of the wgetch subroutines in each type of terminal mode:
Mode
NODELAY
DELAY
HALF-DELAY
Action of wgetch Subroutines
Returns a value of ERR if there is no input waiting.
Stops reading until the system passes text through the program. If CBREAK
mode is also set, the program stops after one character. If CBREAK mode is
not set (NOCBREAK mode), the wgetch subroutine stops reading after the
first new-line character. If ECHO is set, the character is also echoed to the
window.
Stops reading until a character is typed or a specified timeout is reached. If
ECHO mode is set, the character is also echoed to the window.
Note: When you use the wgetch subroutines do not set both the NOCBREAK
mode and the ECHO mode at the same time. Setting both modes can cause
undesirable results depending on the state of the tty driver when each
character is typed.
Function Keys: Function keys are defined in the curses.h file. Function keys can
be returned by the wgetch subroutine if the keypad is enabled. A terminal may not
support all of the function keys. To see if a terminal supports a particular key,
check its terminfo database definition. The following table lists the function keys
defined in the curses.h file:
Name
Key Name
KEY_BREAK
Break key (unreliable).
KEY_DOWN
Down arrow key.
KEY_UP
Up arrow key.
KEY_LEFT
Left arrow key.
KEY_RIGHT
Right arrow key.
KEY_HOME
Home key (upward + left arrow).
KEY_BACKSPACE
Backspace (unreliable).
KEY F0
Function keys. Space for 64 keys is reserved.
KEYF(n)
Formula for fn.
KEY_DL
Delete line.
KEY_IL
Insert line.
KEY_DC
Delete character.
KEY_IC
Insert character or enter insert mode.
KEY_EIC
Exit insert character mode.
Chapter 2. The Curses Library
21
22
KEY_CLEAR
Clear screen.
KEY_EOS
Clear to end of screen.
KEY_EOL
Clear to end of line.
KEY_SF
Scroll 1 line forward.
KEY_SR
Scroll 1 line backwards (reverse).
KEY_NPAGE
Next page.
KEY_PPAGE
Previous page.
KEY_STAB
Set tab.
KEY_CTAB
Clear tab.
KEY_CATAB
Clear all tabs.
KEY_ENTER
Enter or send.
KEY_SRESET
Soft (partial) reset.
KEY_RESET
Reset or hard reset.
KEY_PRINT
Print or copy.
KEY_IL
Home down or bottom (lower left) keypad.
KEY_A1
Upper left of keypad.
KEY_A3
Upper right of keypad.
KEY_B2
Center of keypad.
KEY_C1
Lower left of keypad.
KEY_C3
Lower right of keypad.
KEY_BTAB
Back tab key.
KEY_BEG
Beginning key.
KEY_CANCEL
Cancel key.
KEY-CLOSE
Close key.
KEY_COMMAND
Command key.
KEY_COPY
Copy key.
KEY_CREATE
Create key.
KEY_END
End key.
KEY_EXIT
Exit key.
KEY_FIND
Find key.
KEY_HELP
Help key.
KEY_MARK
Mark key.
KEY_MESSAGE
Message key.
KEY_MOVE
Move key.
KEY_NEXT
Next object key.
KEY_OPEN
Open key.
KEY_OPTIONS
Options key.
KEY_PREVIOUS
Previous object key.
KEY_REDO
Redo key.
KEY_REFERENCE
Reference key.
KEY_REFRESH
Refresh key.
KEY_REPLACE
Replace key.
General Programming Concepts
KEY_RESTART
Restart key.
KEY_RESUME
Resume key.
KEY_SAVE
Save key.
KEY_SBEG
Shifted beginning key.
KEY_SCANCEL
Shifted cancel key.
KEY_SCOMMAND
Shifted command key.
KEY_SCOPY
Shifted copy key.
KEY_SCREATE
Shifted create key.
KEY_SDC
Shifted delete-character key.
KEY_SDL
Shifted delete-line key.
KEY_SELECT
Select key.
KEY_SEND
Shifted end key.
KEY_SEOL
Shifted clear-line key.
KEY_SEXIT
Shifted exit key.
KEY_SFIND
Shifted find key.
KEY_SHELP
Shifted help key.
KEY_SHOME
Shifted home key.
KEY_SIC
Shifted input key.
KEY_SLEFT
Shifted left arrow key.
KEY_SMESSAGE
Shifted message key.
KEY_SMOVE
Shifted move key.
KEY_SNEXT
Shifted next key.
KEY_SOPTIONS
Shifted options key.
KEY_SPREVIOUS
Shifted previous key.
KEY_SPRINT
Shifted print key.
KEY_SREDO
Shifted redo key.
KEY_SREPLACE
Shifted replace key.
KEY_SRIGHT
Shifted right arrow key.
KEY_SRSUME
Shifted resume key.
KEY_SSAVE
Shifted save key.
KEY_SSUSPEND
Shifted suspend key.
KEY_SUNDO
Shifted undo key.
KEY_SUSPEND
Suspend key.
KEY_UNDO
Undo key.
Getting Function Keys: If your program enables the keyboard with the keypad
subroutine, and the user presses a function key, the token for that function key is
returned instead of raw characters. The possible function keys are defined in the
/usr/include/curses.h file. Each define statement begins with a KEY_ prefix and the
keys are defined as integers beginning with the value 03510.
If a character is received that could be the beginning of a function key (such as an
Escape character), curses sets a timer (a structure of type timeval that is defined in
/usr/include/sys/time.h). If the remainder of the sequence is not received before the
timer expires, the character is passed through. Otherwise, the function key’s value
Chapter 2. The Curses Library
23
is returned. For this reason, after a user presses the escape key there is a delay
before the escape is returned to the program. You should avoid using the escape
key where possible when you call a single-character subroutine such as the wgetch
subroutine. This timer can be overridden or extended by the use of the
environment variable ESCDELAY.
The ESCDELAY environment variable sets the length of time to wait before timing
out and treating the ESC keystroke as the Escape character rather than combining
it with other characters in the buffer to create a key sequence. The ESCDELAY
value is measured in fifths of a millisecond. If the ESCDELAY variable is 0, the
system immediately composes the Escape response without waiting for more
information from the buffer. You may choose any value from 0 to 99,999. The
default setting for the ESCDELAY variable is 500 (1/10th of a second).
To prevent the wgetch subroutine from setting a timer, call the notimeout
subroutine. If notimeout is set to TRUE, curses does not distinguish between
function keys and characters when retrieving data.
keyname Subroutine
The keyname subroutine returns a pointer to a character string containing a
symbolic name for the Key argument. The Key argument can be any key returned
from the wgetch, getch, mvgetch, or mvwgetch subroutines.
winch Subroutines
The winch subroutines retrieve the character at the current position. If any
attributes are set for the position, the attribute values are ORed into the value
returned. You can use the winch subroutines to extract only the character or its
attributes. To do this, use the predefined constants A_CHARTEXT and
A_ATTRIBUTES with the logical & (ampersand) operator. These constants are
defined in the curses.h file. The following are the inch subroutines:
winch subroutine
inch macro
mvinch macro
mvwinch macro
Gets the current character from a user-defined window.
Gets the current character from the stdscr.
Moves the logical cursor before calling the inch subroutine on the
stdscr.
Moves the logical cursor before calling the winch subroutine in
the user-defined window.
wscanw Subroutines
The wscanw subroutines read character data, interpret it according to a conversion
specification, and store the converted results into memory. The wscanw
subroutines use the wgetstr subroutines to read the character data. The following
are the wscanw subroutines:
wscanw subroutine
scanw macro
mvscanw macro
mvwscanw macro
Scans a user-defined window.
Scans the stdscr.
Moves the logical cursor before scanning the stdscr.
Moves the logical cursor in the user-defined window before
scanning.
The vwscanw subroutine scans a window using a variable argument list. For
information about manipulating variable argument lists, see the varargs macros.
24
General Programming Concepts
Understanding Terminals with curses
The capabilities of your program are limited, in part, by the capabilities of the
terminal on which it runs. This section provides information about initializing
terminals and identifying their capabilities.
Manipulating Multiple Terminals
With curses, you can use one or more terminals for input and output. The terminal
subroutines enable you to establish new terminals, to switch input and output
processing, and to retrieve terminal capabilities.
You can start curses on a single default screen using the initscr subroutine. This
should be sufficient for most applications. However, if your application sends
output to more than one terminal, you should use the newterm subroutine. Call
this subroutine for each terminal. You should also use the newterm subroutine if
your application wants an indication of error conditions so that it can continue to
run in a line-oriented mode if the terminal cannot support a screen-oriented
program.
When it completes, a program must call the endwin subroutine for each terminal it
used. If you call the newterm subroutine more than once for the same terminal,
the first terminal referred to must be the last one for which you call the endwin
subroutine.
The set_term subroutine switches input and output processing between different
terminals.
Determining Terminal Capabilities
curses supplies the following subroutines to help you determine the capabilities of
a terminal:
longname
has_ic
has_il
Returns the verbose name of the terminal.
Determines whether a terminal has the insert-character capability.
Determines whether a terminal has the insert-line capability.
The longname subroutine returns a pointer to a static area containing a verbose
description of the current terminal. This area is defined only after a call to the
initscr or newterm subroutine. If you intend to use the longname subroutine with
multiple terminals, you should know that each call to the newterm subroutine
overwrites this area. Calls to the set_term subroutine do not restore the value.
Instead, save this area between calls to the newterm subroutine.
The has_ic subroutine returns TRUE if the terminal has insert and delete character
capabilities.
The has_il subroutine returns TRUE if the terminal has insert and delete line
capabilities or can simulate the capabilities using scrolling regions. Use the has_il
subroutine to check whether it is appropriate to turn on physical scrolling using
the scrollok or idlok subroutines.
Setting Terminal Input and Output Modes
The subroutines that control input and output determine how your application
retrieves and displays data to users.
Chapter 2. The Curses Library
25
Input Modes
Special input characters include the flow-control characters, the interrupt character,
the erase character, and the kill character. Four mutually-exclusive curses modes let
the application control the effect of the input characters.
Cooked Mode
This achieves normal line-at-a-time processing with all special characters
handled outside the application. This achieves the same effect as
canonical-mode input processing. The state of the ISIG and IXON flags are
not changed upon entering this mode by calling nocbreak( ), and are set
upon entering this mode by calling noraw( ). The implementation supports
erase and kill characters from any supported locale, no matter what the
width of the character is.
cbreak Mode
Characters typed by the user are immediately available to the application
and curses does not perform special processing on either the erase
character or the kill character. An application can select cbreak mode to do
its own line editing but to let the abort character be used to abort the task.
This mode achieves the same effect as non-canonical-mode, Case B input
processing (with MIN set to 1 and ICRNL cleared). The state of the ISIG
and IXON flags are not changed upon entering this mode.
Half-Delay Mode
The effect is the same as cbreak, except that input functions wait until a
character is available or an interval defined by the application elapses,
whichever comes first. This mode achieves the same effect as
non-canonical-mode, Case C input processing (with TIME set to the value
specified by the application). The state of the ISIG and IXON flags are not
changed upon entering this mode.
Raw Mode
Raw mode gives the application maximum control over terminal input.
The application sees each character as it is typed. This achieves the same
effect as non-canonical mode, Case D input processing. The ISIG and IXON
flags are cleared upon entering this mode.
The terminal interface settings are recorded when the process calls initscr( ) or
newterm( ) to initialize curses and restores these settings when endwin( ) is called.
The initial input mode for curses operations is unspecified unless the
implementation supports Enhanced curses compliance, in which the initial input
mode is cbreak mode.
The behavior of the BREAK key depends on other bits in the display driver that
are not set by curses.
Delay Mode
Two mutually-exclusive delay modes specify how quickly certain curses functions
return to the application when there is no terminal input waiting when the
function is called:
No Delay
Delay
The function fails.
The application waits until the implementation passes text through to the
application. If cbreak or Raw Mode is set, this is after one character.
Otherwise, this is after the first <newline> character, end-of-line character, or
end-of-file character.
The effect of No Delay Mode on function key processing is unspecified.
26
General Programming Concepts
Echo Processing
echo mode determines whether curses echoes typed characters to the screen. The
effect of echo mode is analogous to the effect of the echo flag in the local mode
field of the termios structure associated with the terminal device connected to the
window. However, curses always clears the echo flag when invoked, to inhibit the
operating system from performing echoing. The method of echoing characters is
not identical to the operating system’s method of echoing characters, because
curses performs additional processing of terminal input.
If in echo mode, curses performs its own echoing. Any visible input character is
stored in the current or specified window by the input function that the
application called, at that window’s cursor position, as though addch( ) were
called, with all consequent effects such as cursor movement and wrapping.
If not in echo mode, any echoing of input must be performed by the application.
Applications often perform their own echoing in a controlled area of the screen, or
do not echo at all, so they disable echo mode.
It may not be possible to turn off echo processing for synchronous and network
asynchronous terminals because echo processing is done directly by the terminals.
Applications running on such terminals should be aware that any characters typed
will appear on the screen at wherever the cursor is positioned.
cbreak or nocbreak
delay_output
echo or noecho
halfdelay
nl or nonl
raw or noraw
Puts the terminal into or takes it out of CBREAK mode.
Sets the output delay in milliseconds.
Controls echoing of typed characters to the screen.
Returns ERR if no input was typed after blocking for a
specified amount of time.
Determines whether curses translates a new line into a
carriage return and line feed on output, and translates a
return into a new line on input.
Places the terminal into or out of RAW mode.
The echo subroutine puts the terminal into echo mode. In echo mode, curses writes
characters typed by the user to the terminal at the physical cursor position. The
noecho subroutine takes the terminal out of echo mode.
The raw subroutine puts the terminal into raw mode. In raw mode, characters
typed by the user are immediately available to the program. Additionally, the
interrupt, quit, suspend, and flow-control characters are passed uninterpreted
instead of generating a signal as they do in cbreak mode. The noraw subroutine
takes the terminal out of raw mode.
The cbreak subroutine performs a subset of the functions performed by the raw
subroutine. In cbreak mode, characters typed by the user are immediately available
to the program and erase or kill character processing is not done. Unlike RQW
mode, interrupt and flow characters are acted upon. Otherwise, the tty driver
buffers the characters typed until a new line or carriage return is typed.
Note: cbreak mode disables translation by the tty driver.
The nocbreak subroutine takes the terminal out of cbreak mode.
The delay_output subroutine sets the output delay to the specified number of
milliseconds. You should not use this subroutine extensively because it uses
padding characters instead of a processor pause.
Chapter 2. The Curses Library
27
The nl and nonl subroutines, respectively, control whether curses translates new
lines into carriage returns and line feeds on output, and whether curses translates
carriage returns into new lines on input. Initially, these translations do occur. By
disabling these translations, the curses subroutine library has more control over the
line-feed capability, resulting in faster cursor motion.
Using the terminfo and termcap Files
When curses is initialized, it checks the TERM environment variable to identify the
terminal type. Then, curses looks for a definition explaining the capabilities of the
terminal. Usually this information is kept in a local directory specified by the
TERMINFO environment variable or in the /usr/share/lib/terminfo directory. All
curses programs first check to see if the TERMINFO environment variable is
defined. If this variable is not defined, the /usr/share/lib/terminfo directory is
checked.
For example, if the TERM variable is set to vt100 and the TERMINFO variable is
set to the /usr/mark/myterms file, curses checks for the /usr/mark/myterms/v/vt100
file. If this file does not exist, curses checks the /usr/share/lib/terminfo/v/vt100 file.
Additionally, the LINES and COLUMNS environment variables can be set to
override the terminal description.
Writing Programs That Use the terminfo Subroutines
Use the terminfo subroutines when your program needs to deal directly with the
terminfo database. For example, use these subroutines to program function keys. In
all other cases, curses subroutines are more suitable and their use is recommended.
Initializing Terminals: Your program should begin by calling the setupterm
subroutine. Normally, this subroutine is called indirectly by a call to the initscr or
newterm subroutine. The setupterm subroutine reads the terminal-dependent
variables defined in the terminfo database. The terminfo database includes
boolean, numeric, and string variables. All of these terminfo variables use the
values defined for the specified terminal. After reading the database, the setupterm
subroutine initializes the cur_term variable with the terminal definition. When
working with multiple terminals, you can use the set_curterm subroutine to set the
cur_term variable to a specific terminal.
Another subroutine, restartterm, is similar to the setupterm subroutine. However,
it is called after memory is restored to a previous state. For example, you would
call the restartterm subroutine after a call to the scr_restore subroutine. The
restartterm subroutine assumes that the input and output options are the same as
when memory was saved, but that the terminal type and baud rate may differ.
The del_curterm subroutine frees the space containing the capability information
for a specified terminal.
Header Files: Include the curses.h and term.h files in your program in the
following order:
#include <curses.h>
#include <term.h>
These files contain the definitions for the strings, numbers, and flags in the
terminfo database.
28
General Programming Concepts
Handling Terminal Capabilities: Pass all parametized strings through the tparm
subroutine to instantiate them. You should print all terminfo strings and the
output of the tparm subroutine with the tputs or putp subroutine.
putp
tparm
tputs
Provides a shortcut to the tputs subroutine.
Instantiates a string with parameters.
Applies padding information to the given string and outputs it.
Use the following subroutines to obtain and pass terminal capabilities:
tigetflag
tigetnum
tigetstr
Returns the value of a specified boolean capability. If the capability is not
boolean, a -1 is returned.
Returns the value of a specified numeric capability. If the capability is not
numeric, a -2 is returned.
Returns the value of a specified string capability. If the capability specified is
not a string, the tigetstr subroutine returns the value of (char *) -1.
Exiting the Program: When your program exits you should restore the tty modes
to their original state. To do this, call the reset_shell_mode subroutine. If your
program uses cursor addressing, it should output the enter_ca_mode string at
startup and the exit_ca_mode string when it exits.
Programs that use shell escapes should call the reset_shell_mode subroutine and
output the exit_ca_mode string before calling the shell. After returning from the
shell, the program should output the enter_ca_mode string and call the
reset_prog_mode subroutine. This process differs from standard curses operations
which call the endwin subroutine on exit.
Low-Level Screen Subroutines
Use the following subroutines for low-level screen manipulations:
scr_restore
scr_dump
scr_init
ripoffline
Restores the virtual screen to the contents of a previously dumped file.
Dumps the contents of the virtual screen to the specified file.
Initializes the curses data structures from a specified file.
Strips a single line from the stdscr.
termcap Subroutines
If your program uses the termcap file for terminal information, the termcap
subroutines are included as a conversion aid. The parameters are the same for the
termcap subroutines. curses emulates the subroutines using the terminfo database.
The following termcap subroutines are supplied:
tgetent
tgetflag
tgetnum
tgetstr
tgoto
Emulates the setupterm subroutine.
Returns the boolean entry for a termcap identifier.
Returns the numeric entry for a termcap identifier.
Returns the string entry for a termcap identifier.
Duplicates the tparm subroutine. The output from the tgoto subroutine should
be passed to the tputs subroutine.
Converting termcap Descriptions to terminfo Descriptions
The captoinfo command converts termcap descriptions to terminfo descriptions.
The following example illustrates how the captoinfo command works:
captoinfo /usr/lib/libtermcap/termcap.src
This command converts the /usr/lib/libtermcap/termcap.src file to terminfo source.
The captoinfo command writes the output to standard output and preserves
comments and other information in the file.
Chapter 2. The Curses Library
29
Note: To maintain compatibility with the AIX 3.2 version of curses, you can
install a version of the captoinfo command that converts termcap data to the
AIX 3.2-style terminfo data. See “AIX Version 3.2 Curses Compatibility” on
page 34 for details.
Manipulating TTYs
The following functions save and restore the state of terminal modes:
savetty
resetty
Saves the state of the tty modes.
Restores the state of the tty modes to what they were the last time the savetty
subroutine was called.
Synchronous and Networked Asynchronous Terminals
Synchronous, networked synchronous (NWA) or non-standard directly-connected
asynchronous terminals are often used in a mainframe environment and
communicate to the host in block mode. That is, the user types characters at the
terminal then presses a special key to initiate transmission of the characters to the
host.
Although it may be possible to send arbitrary sized blocks to the host, it is not
possible or desirable to cause a character to be transmitted with only a single
keystroke. Doing so could cause severe problems to an application wishing to
make use of single-character input.
Output
The curses interface can be used in the normal way for all operations pertaining to
output to the terminal, with the possible exception that on some terminals the
refresh( ) routine may have to redraw the entire screen contents in order to
perform any update.
If it is additionally necessary to clear the screen before each such operation, the
result could be undesirable.
Input
Because of the nature of operation of synchronous (block-mode) and NWA
terminals, it might not be possible to support all or any of the curses input
functions. In particular, the following points should be noted:
v Single-character input might not possible. It may be necessary to press a special
key to cause all characters typed at the terminal to be transmitted to the host.
v It is sometimes not possible to disable echo. Character echo may be performed
directly by the terminal. On terminals that behave in this way, any curses
application that performs input should be aware that any characters typed will
appear on the screen at wherever the cursor is positioned. This does not
necessarily correspond to the position of the cursor in the window.
Working with Color
If a terminal supports color, you can use the color manipulation subroutines to
include color in your curses program. Before manipulating colors, you should test
whether a terminal supports color. To do this, you can use either the has_colors
subroutine or the can_change_color subroutine. The can_change_color subroutine
also checks to see if a program can change the terminal’s color definitions. Neither
of these subroutines requires an argument.
30
General Programming Concepts
Checks to see if the terminal supports colors and changing of the
color definition.
Checks that the terminal supports colors.
Initializes the eight basic colors and two global variables,
COLORS and COLOR_PAIRS.
can_change_color
has_colors
start_color
Once you have determined that the terminal supports color, you must call the
start_color subroutine before calling other color subroutines. It is a good practice to
call this subroutine right after the initscr subroutine and after a successful color
test. The COLORS global variable defines the maximum number of colors the
terminal supports. The COLOR_PAIRS global variable defines the maximum
number of color pairs the terminal supports.
Manipulating Video Attributes
Your program can manipulate a number of video attributes. The following sections
provide information on video attributes and the subroutines that affect them.
Video Attributes, Bit Masks, and the Default Colors
Curses enables you to control the following attributes:
A_STANDOUT
A_UNDERLINE
A_REVERSE
A_BLINK
A_DIM
A_BOLD
A_ALTCHARSET
A_NORMAL
COLOR_PAIR (Number)
Terminal’s best highlighting mode.
Underline.
Reverse video.
Blinking.
Half-bright.
Extra bright or bold.
Alternate character set.
Normal attributes.
Displays the color pair represented by Number. You must have
already initialized the color pair using the init_pair
subroutine.
These attributes are defined in the curses.h file. You can pass attributes to the
wattron, wattroff , and wattrset subroutines or you can OR them with the
characters passed to the waddch subroutine. The C logical OR operator is a |
(pipe symbol). The following bit masks are also provided:
A_NORMAL
A_CHARTEXT
A_ATTRIBUTES
A_COLOR
Turns all video attributes off.
Extracts a character.
Extracts attributes.
Extracts color-pair field information.
Two macros are provided for working with color pairs: COLOR_PAIR( Number)
and PAIR_NUMBER( Attribute). The COLOR_PAIR( Number) macro and the
A_COLOR mask are used by the PAIR_NUMBER( Attribute) macro to extract the
color-pair number found in the attributes specified by the Attribute parameter.
If your program uses color, the curses.h file defines a number of macros that
identify default colors. These colors are the following:
Color
COLOR_BLACK
COLOR_BLUE
COLOR_GREEN
COLOR_CYAN
Integer Value
0
1
2
3
Chapter 2. The Curses Library
31
Color
COLOR_RED
COLOR_MAGENTA
COLOR_YELLOW
COLOR_WHITE
Integer Value
4
5
6
7
Curses assumes that the default background color for all terminals is 0
(COLOR_BLACK ).
Setting Video Attributes
The current window attributes are applied to all characters written into the
window with the addch subroutines. These attributes remain as a property of the
characters. The characters retain these attributes during terminal operations.
attroff or wattroff
Turns off attributes.
attron or wattron
Turns on attributes.
attrset or wattrset
Sets the current attributes of a window.
standout, wstandout, standend, or wstandend
Puts a window into and out of the terminal’s
best highlight mode.
vidputs or vidattr
Outputs a string that puts the terminal in a
video-attribute mode.
The attrset subroutine sets the current attributes of the default screen. The wattrset
subroutine sets the current attributes of the user-defined window.
Use the attron and attroff subroutines to turn on and off the specified attributes in
the stdscr without affecting any others. The wattron and wattroff subroutines
perform the same actions in user-defined windows.
The standout subroutine is the same as a call to the attron subroutine with the
A_STANDOUT attribute. It puts the stdscr into the terminal’s best highlight mode.
The wstandout subroutine is the same as a call to the wattron( Window,
A_STANDOUT) subroutine. It puts the user-defined window into the terminal’s
best highlight mode. The standend subroutine is the same as a call to the attrset(0)
subroutine. It turns off all attributes for stdscr. The wstandend subroutine is the
same as a call to the wattrset( Window, 0) subroutine. It turns off all attributes for
the specified window.
The vidputs subroutine outputs a string that puts the terminal in the specified
attribute mode. Characters are output through the putc subroutine. The vidattr
subroutine is the same as the vidputs subroutine except that characters are output
through the putchar subroutine.
Working with Color Pairs
The COLOR_PAIR (Number) macro is defined in the curses.h file so you can
manipulate color attributes as you would any other attributes. You must initialize a
color pair with the init_pair subroutine before you use it. The init_pair subroutine
has three parameters Pair, Foreground, and Background. The Pair parameter must be
between 1 and COLOR_PAIRS -1. The Foreground and Background parameters must
be between 0 and COLORS -1. For example, to initialize color pair 1 to a
foreground of black with a background of cyan, you would use the following:
init_pair(1, COLOR_BLACK, COLOR_CYAN);
You could then set the attributes for the window as:
32
General Programming Concepts
wattrset(win, COLOR_PAIR(1));
If you then write the string Let’s add Color to the terminal, the string appears as
black characters on a cyan background.
Extracting Attributes
You can use the results from the call to the winch subroutine to extract attribute
information, including the color-pair number. The following example uses the
value returned by a call to the winch subroutine with the C logical AND operator
(&) and the A_ATTRIBUTES bit mask to extract the attributes assigned to the
current position in the window. The results from this operation are used with the
PAIR_NUMBER macro to extract the color-pair number, and the number 1 is
printed on the screen.
win = newwin(10, 10, 0, 0);
init_pair(1, COLOR_RED, COLOR_YELLOW);
wattrset(win, COLOR_PAIR(1));
waddstr(win, "apple");
number = PAIR_NUMBER((mvwinch(win, 0, 0) & A_ATTRIBUTES));
wprintw(win, "%d\n", number);
wrefresh(win);
Lights and Whistles
The curses library provides alarm subroutines to signal the user.
beep
flash
Sounds an audible alarm on the terminal
Displays a visible alarm on the terminal
Setting Curses Options
All curses options are initially turned off, so it is not necessary to turn them off
before calling the endwin subroutine. The following subroutines allow you to set
various options with curses:
curs_set
idlok
intrflush
keypad
typeahead
Sets the cursor visibility to invisible, normal, or very visible.
Specifies whether curses can use the hardware insert and delete line
features of terminals so equipped.
Specifies whether an interrupt key (interrupt, quit, or suspend) flushes all
output in the tty driver. This option’s default is inherited from the tty
driver.
Specifies whether curses retrieves the information from the terminal’s
keypad. If enabled, the user can press a function key (such as an arrow
key) and the wgetch subroutine returns a single value representing that
function key. If disabled, curses will not treat the function keys specially
and your program must interpret the escape sequences. For a list of these
function keys, see the wgetch subroutine.
Instructs curses to check for type ahead in an alternative file descriptor.
See the wgetch subroutines and “Setting Terminal Input and Output Modes” on
page 25 for descriptions of additional curses options.
Manipulating Soft Labels
Curses provides subroutines for manipulating soft function-key labels. These labels
appear at the bottom of the screen and give applications, such as editors, a more
user-friendly look. To use soft labels, you must call the slk_init subroutine before
calling the initscr or newterm subroutines.
slk_clear
Clears soft labels from the screen.
Chapter 2. The Curses Library
33
slk_init
slk_label
slk_noutrefresh
slk_refresh
slk_restore
slk_set
slk_touch
Initializes soft function key labels.
Returns the current label.
Refreshs soft labels. This subroutine is functionally equivalent to the
wnoutrefresh subroutine.
Refreshs soft labels. This subroutine is functionally equivalent to the
refresh subroutine.
Restores the soft labels to the screen after a call to the slk_clear
subroutine.
Sets a soft label.
Updates soft labels on the next call to the slk_noutrefresh
subroutine.
To manage soft labels, curses reduces the size of the stdscr by one line. It reserves
this line for use by the soft-label functions. This reservation means that the
environment variable LINES is also reduced. Many terminals support built-in soft
labels. If built-in soft labels are supported, curses uses them. Otherwise, curses
simulates the soft-labels with software.
Because many terminals that support soft labels have 8 labels, curses follows the
same standard. A label string is restricted to 8 characters. Curses arranges labels in
one of two patterns: 3-2-3 (3 left, 2 center, 3 right) or 4-4 (4 left, 4 right).
To specify a string for a particular label, call the slk_set subroutine. This
subroutine also instructs curses as to left-justify, right-justify, or center the string on
the label. If you wish to obtain a label name before it was justified by the slk_set
subroutine, use the slk_label subroutine. The slk_clear and slk_restore subroutines
clear and restore soft labels respectively. Normally, to update soft labels, your
program should call the slk_noutrefresh subroutine for each label and then use a
single call to the slk_refresh subroutine to perform the actual output. To output all
the soft labels on the next call to the slk_noutrefresh subroutine, use the slk_touch
subroutine.
Obsolete Curses Subroutines
Several curses subroutines are obsolete beginning in AIX Version 4. These obsolete
subroutines are emulated as indicated in the following list:
Obsolete
crmode
fixterm
getcap
nocrmode
resetterm
saveterm
setterm
Replaced by
cbreak
reset_prog_mode
tgetstr
nocbreak
reset_shell_mode
def_prog_mode
setupterm
The touchoverlap, flushok, and _showstring subroutines are obsolete and there
are no direct replacements. The gettmode subroutine is available as a no-op.
AIX Version 3.2 Curses Compatibility
v In this version of AIX, curses is compatible with AT&T System V Release 3.2
curses.
v The older AIX curses have been kept in a form useful for supporting existing
binaries only. This change was made to provide support for color and to increase
application portability to AIX systems.
34
General Programming Concepts
v Applications already running under AIX will still operate using the old curses.
v Applications compiled, rebound, or relinked beginning with AIX Version 4 may
need source code changes for compatibility with the newer version of curses.
The newer curses library does not have or use AIX extended curses functions.
v Applications requiring multibyte support may still compile and link with
extended curses. However, because the extended curses library may be removed
in the future, use of the extended curses library is discouraged except for
applications that require multibyte support.
List of Additional Curses Subroutines
For information on the X/Open UNIX95 Specification curses subroutines available
on AIX Version 4.2 (and later), see the X/Open CAE Specification.
Manipulating Windows
scr_dump
scr_init
scr_restore
Writes the current contents of the virtual screen to the specified file.
Uses the contents of a specified file to initialize the curses data structures.
Sets the virtual screen to the contents of the specified file.
Manipulating Characters
echochar, wechochar, or pechochar
flushinp
insertln or winsertln
keyname
meta
nodelay
scroll
unctrl
vwprintw
vwscanw
Functionally equivalent to a call to the
addch (or waddch) subroutine followed
by a call to the refresh (or wrefresh)
subroutine.
Flushes any type-ahead characters typed
by the user but not yet read by the
program.
Inserts a blank line in a window.
Returns a pointer to a character string
containing a symbolic name for the Key
parameter.
Determines whether 8-bit character
return for the wgetch subroutine is
allowed.
Causes a call to the wgetch subroutine
to be a nonblocking call. If no input is
ready, the wgetch subroutine returns
ERR.
Scrolls a window up one line.
Returns the printable representation of a
character. Control characters are
punctuated with a | (caret).
Performs the same operation as the
wprintw subroutine but takes a variable
list of arguments.
Performs the same operation as the
wscanw subroutine but takes a variable
list of arguments.
Manipulating Terminals
def_prog_mode
def_shell_mode
del_curterm
Identifies the current terminal mode as the in-curses mode.
Saves the current terminal mode as the not-in-curses mode.
Frees the space pointed to by the oterm variable.
Chapter 2. The Curses Library
35
Prevents the wgetch subroutine from setting a timer when
interpreting an input escape sequence.
Equivalent to a call to the waddch subroutine followed by a call
to the prefresh subroutine.
Restores the terminal into the in-curses program mode.
Restores the terminal to shell mode (out-of-curses mode). The
endwin subroutine does this automatically.
Sets up a TERMINAL structure for use by curses. This
subroutine is similar to the setupterm subroutine. Call the
restartterm subroutine after restoring memory to a previous
state. For example, call this subroutine after a call to the
scr_restore subroutine.
notimeout
pechochar
reset_prog_mode
reset_shell_mod e
restartterm
Manipulating Color
color_content
init_color
init_pair
pair_content
Returns the composition of a color.
Changes a color to the desired composition.
Initializes a color pair to the specified foreground and background
colors.
Returns the foreground and background colors for a specified
color-pair number.
Miscellaneous Utilities
baudrate
erasechar
killchar
36
General Programming Concepts
Queries the current terminal and returns its output speed.
Returns the erase character chosen by the user.
Returns the line-kill character chosen by the user.
Chapter 3. Debugging Programs
There are several debug programs available for debugging your programs: the adb,
dbx, dex, softdb, and kernel debug programs. The adb program enables you to
debug executable binary files and examine non-ASCII data files. The dbx program
enables source-level debugging of C, C++, Pascal, and FORTRAN language
programs, as well as assembler-language debugging of executable programs at the
machine level. The (dex) provides an X interface for the dbx debug program,
providing windows for viewing the source, context, and variables of the
application program. The softdb debug program works much like the dex debug
program, but softdb is used with AIX Software Development Environment
Workbench. The kernel debug program is used to help determine errors in code
running in the kernel.
adb Debug Program Overview
The adb command provides a general purpose debug program. You can use this
command to examine object and core files and provide a controlled environment
for running a program.
While the adb command is running, it takes standard input and writes to standard
output. The command does not recognize the Quit or Interrupt keys. If these keys
are used, the adb command waits for a new command.
Getting Started with the adb Debug Program
The following sections explain how to start the adb debugging program from a
variety of files, use the adb prompt, use shell commands from within the adb
program, and stop the adb program.
Starting adb with a Program File
You can debug any executable C or assembly language program file by entering a
command line of the form:
adb FileName
where FileName is the name of the executable program file to be debugged. The
adb program opens the file and prepares its text (instructions) and data for
subsequent debugging. For example, the command:
adb sample
prepares the program named sample for examination and operation.
Once started, the adb debug program places the cursor on a new line and waits for
you to type commands.
Starting adb with a Nonexistent or Incorrect File
If you start the adb debug program with the name of a nonexistent or incorrectly
formatted file, the adb program first displays an error message and then waits for
commands. For example, if you start the adb program with the command:
© Copyright IBM Corp. 1997, 1999
37
adb sample
and the sample file does not exist, the adb program displays the message:
sample: no such file or directory.
Starting adb with the Default File
You can start the adb debug program without a file name. In this case, the adb
program searches for the default a.out file in your current working directory and
prepares it for debugging. Thus, the command:
adb
is the same as entering:
adb a.out
The adb program starts with the a.out file and waits for a command. If the a.out
file does not exist, the adb program starts without a file and does not display an
error message.
Starting adb with a Core Image File
You can use the adb debug program to examine the core image files of programs
that caused irrecoverable system errors. Core image files maintain a record of the
contents of the CPU registers, stack, and memory areas of your program at the
time of the error. Therefore, core image files provide a way to determine the cause
of an error.
To examine a core image file with its corresponding program, you must give the
name of both the core and the program file. The command line has the form:
adb ProgramFile CoreFile
where ProgramFile is the file name of the program that caused the error, and
CoreFile is the file name of the core image file generated by the system. The adb
program then uses information from both files to provide responses to your
commands.
If you do not give the filename of the core image file, the adb program searches
for the default core file, named core, in your current working directory. If such a
file is found, the adb program determines whether the core file belongs to the
ProgramFile. If so, the adb program uses it. Otherwise, the adb program discards
the core file by giving an appropriate error message.
Note: The adb command cannot be used to examine 64-bit objects and AIX
Version 4.3 core format. adb still works with pre-AIX Version 4.3 core format.
On AIX Version 4.3, user can make kernel to generate pre-AIX Version 4.3
style core dumps using smitty.
Starting adb with a Data File
The adb program provides a way to look at the contents of the file in a variety of
formats and structures. You can use the adb program to examine data files by
giving the name of the data file in place of the program or core file. For example,
to examine a data file named outdata, enter:
38
General Programming Concepts
adb outdata
The adb program opens a file called outdata and lets you examine its contents.
This method of examining files is useful if the file contains non-ASCII data. The
adb command may display a warning when you give the name of a non-ASCII
data file in place of a program file. This usually happens when the content of the
data file is similar to a program file. Like core files, data files cannot be executed.
Starting adb with the Write Option
If you open a program or data file with the -w flag of the adb command, you can
make changes and corrections to the file. For example, the command:
adb -w sample
opens the program file sample for writing. You can then use adb commands to
examine and modify this file. The -w flag causes the adb program to create a given
file if it does not already exist. The option also lets you write directly to memory
after running the given program.
Using a Prompt
After you have started the adb program you can redefine your prompt with the $P
subcommand.
To change the [adb:scat]>> prompt to Enter a debug command—->, enter:
$P"Enter a debug command—->"
The quotes are not necessary when redefining the new prompt from the adb
command line.
Using Shell Commands from within the adb Program
You can run shell commands without leaving the adb program by using the adb
escape command (!) (exclamation point). The escape command has the form:
! Command
In this format Command is the shell command you want to run. You must provide
any required arguments with the command. The adb program passes this
command to the system shell that calls it. When the command is finished, the shell
returns control to the adb program. For example, to display the date, enter the
following command:
! date
The system displays the date and restores control to the adb program.
Exiting the adb Debug Program
You can stop the adb program and return to the system shell by using the $q or
$Q subcommands. You can also stop the adb program by typing the Ctrl-D key
sequence. You cannot stop the adb program by pressing the Interrupt or Quit keys.
These keys cause adb to wait for a new command.
Chapter 3. Debugging Programs
39
Controlling Program Execution
The following sections explain the commands and subcommands necessary to
prepare programs for debugging; execute programs; set, display, and delete
breakpoints; continue programs; single-step through a program; stop programs;
and kill programs.
Preparing Programs for Debugging with the adb Program
Compile the program using the cc command to a file such as adbsamp2 by
entering the following:
cc adbsamp2.c -o adbsamp2
To start the debug session, enter:
adb adbsamp2
The C language does not generate statement labels for programs. Therefore, you
cannot refer to individual C language statements when using the debug program.
To use execution commands effectively, you must be familiar with the instructions
that the C compiler generates and how those instructions relate to individual C
language statements. One useful technique is to create an assembler language
listing of your C program before using the adb program. Then, refer to the listing
as you use the debug program. To create an assembler language listing, use the -S
or -qList flag of the cc command.
For example, to create an assembler language listing of the example program,
adbsamp2.c, use the following command:
cc -S adbsamp2.c -o adbsamp2
This command creates the adbsamp2.s file, that contains the assembler language
listing for the program, and compiles the program to the executable file,
adbsamp2.
Running a Program
You can execute a program by using the :r or :R subcommand. The commands
have the form:
[ Address ][,Count ] :r [Arguments ]
OR
[ Address ][,Count ] :R [Arguments ]
In this format, the Address parameter gives the address at which to start running
the program; the Count parameter is the number of breakpoints to skip before one
is taken; and the Arguments parameter provides the command-line arguments, such
as file names and options, to pass to the program.
If you do not supply an Address value, the adb program uses the start of the
program. To run the program from the beginning enter:
:r
40
General Programming Concepts
If you supply a Count value, the adb program ignores all breakpoints until the
given number has been encountered. For example, to skip the first five named
breakpoints, use the command:
,5:r
If you provide arguments, separate them by at least one space each. The arguments
are passed to the program in the same way the system shell passes command-line
arguments to a program. You can use the shell redirection symbols.
The :R subcommand passes the command arguments through the shell before
starting program operation. You can use shell pattern-matching characters in the
arguments to refer to multiple files or other input values. The shell expands
arguments containing pattern-matching characters before passing them to the
program. This feature is useful if the program expects multiple file names. For
example, the following command passes the argument [a-z]* to the shell where it
is expanded to a list of the corresponding file names before being passed to the
program:
:R [a-z]*.s
The :r and :R subcommands remove the contents of all registers and destroy the
current stack before starting the program. This operation halts any previous copy
of the program that may be running.
Setting Breakpoints
To set a breakpoint in a program, use the :b subcommand. Breakpoints stop
operation when the program reaches the specified address. Control then returns to
the adb debug program. The command has the form:
[Address] [,Count ] :b [Command]
In this format, the Address parameter must be a valid instruction address; the Count
parameter is a count of the number of times you want the breakpoint to be
skipped before it causes the program to stop; and the Command parameter is the
adb command you want to execute each time that the instruction is executed
(regardless of whether the breakpoint stops the program). If the specified
command sets . (period) to a value of 0, the breakpoint causes a stop.
Set breakpoints to stop program execution at a specific place in the program, such
as the beginning of a function, so that you can look at the contents of registers and
memory. For example, when debugging the example program (see “Example adb
Program: adbsamp2” on page 62), the following command sets a breakpoint at the
start of the function named f:
.f :b
The breakpoint is taken just as control enters the function and before the function’s
stack frame is created.
A breakpoint with a count is used within a function that is called several times
during the operation of a program, or within the instructions that correspond to a
for or while statement. Such a breakpoint allows the program to continue to run
until the given function or instructions have been executed the specified number of
times. For example, the following command sets a breakpoint for the second time
that the f function is called in the adbsamp2 program:
.f,2 :b
Chapter 3. Debugging Programs
41
The breakpoint does not stop the function until the second time the function is
run.
Displaying Breakpoints
Use the $b subcommand to display the location and count of each currently
defined breakpoint. This command displays a list of the breakpoints by address
and any count or commands specified for the breakpoints. For example, the
following example sets two breakpoints in the file and then uses the $b
subcommand to display those breakpoints:
.f+4:b
.f+8:b$v
$b
breakpoints
count brkpt
1
.f+8
1
.f+4
command
$v
When the program runs, it stops at the first breakpoint that it finds, such as .f+4.
If you use the :c subcommand to continue execution, the program stops again at
the next breakpoint and starts the $v subcommand. The command and response
sequence looks like the following example:
:r
adbsamp2:running
breakpoint
.f+4:
:c
adbsamp2:running
variables
b = 268435456
d = 236
e = 268435512
m = 264
breakpoint
.f+8
st
r3,32(r1)
l
r15,32(r1)
Deleting Breakpoints
To use the :d subcommand to delete a breakpoint from a program, enter:
Address :d
In this format, the Address parameter gives the address of the breakpoint to delete.
For example, when debugging the example program, entering the following
command deletes the breakpoint at the start of the f function:
.f:d
Continuing Program Execution
To use the :c subcommand to continue the execution of a program after it has been
stopped by a breakpoint enter:
[Address ] [,Count ] :c [Signal ]
In this format, the Address parameter gives the address of the instruction at which
to continue operation; the Count parameter gives the number of breakpoints to
ignore; and the Signal parameter is the number of the signal to send to the
program.
42
General Programming Concepts
If you do not supply an Address parameter, the program starts at the next
instruction after the breakpoint. If you supply a Count parameter, the adb debug
program ignores the first Count breakpoints.
If the program is stopped using the Interrupt or Quit key, this signal is
automatically passed to the program upon restarting. To prevent this signal from
being passed, enter the command in the form:
[Address] [,Count] :c 0
The command argument 0 prevents a signal from being sent to the subprocess.
Single-Stepping a Program
Use the :s subcommand to run a program in single steps or one instruction at a
time. This command issues an instruction and returns control to the adb debug
program. The command has the form:
[Aaddress ] [,Count ] :s [Signal]
In this format, the Address parameter gives the address of the instruction you want
to execute, and the Count parameter is the number of times you want to repeat the
command. If there is no current subprocess, the ObjectFile parameter is run as a
subprocess. In this case, no signal can be sent and the remainder of the line is
treated as arguments to the subprocess. If you do not supply a value for the
Address parameter, the adb program uses the current address. If you supply the
Count parameter, the adb program continues to issue each successive instruction
until the Count parameter instructions have been run. Breakpoints are ignored
while single-stepping. For example, the following command issues the first five
instructions in the main function:
.main,5:s
Stopping a Program with the Interrupt and Quit Keys
Use either the Interrupt or Quit key to stop running a program at any time.
Pressing either of these keys stops the current program and returns control to the
adb program. These keys are useful with programs that have infinite loops or
other program errors.
When you press the Interrupt or Quit key to stop a program, the adb program
automatically saves the signal. If you start the program again using the :c
command, the adb program automatically passes the signal to the program. This
feature is useful when testing a program that uses these signals as part of its
processing. To continue running the program without sending signals, use the
command:
:c 0
The command argument 0 (zero) prevents a signal from being sent to the program.
Stopping a Program
To stop a program you are debugging, use the :k subcommand. This command
stops the process created for the program and returns control to the adb debug
program. The command clears the current contents of the system unit registers and
stack and begins the program again. The following example shows the use of the
:k subcommand to clear the current process from the adb program:
Chapter 3. Debugging Programs
43
:k
560:
killed
Using adb Expressions
The following sections describe the use of adb expressions.
Using Integers in Expressions
When creating an expression, you can use integers in three forms: decimal, octal,
and hexadecimal. Decimal integers must begin with a non-zero decimal digit. Octal
numbers must begin with a 0 (zero) and have octal digits only (0-7). Hexadecimal
numbers must begin with the prefix 0x and can contain decimal digits and the
letters a through f (in both uppercase and lowercase). The following are examples
of valid numbers:
Decimal
34
4090
Octal
042
07772
Hexadecimal
0x22
0xffa
Using Symbols in Expressions
Symbols are the names of global variables and functions defined within the
program being debugged. Symbols are equal to the address of the given variable
or function. They are stored in the program symbol table and are available if the
symbol table has not been stripped from the program file.
In expressions, you can spell the symbol exactly as it is in the source program or
as it has been stored in the symbol table. Symbols in the symbol table are no more
than 8 characters long.
When you use the ? subcommand, the adb program uses the symbols found in the
symbol table of the program file to create symbolic addresses. Thus, the ?
subcommand sometimes gives a function name when displaying data. This does
not happen if the ? subcommand is used for text (instructions) and the / command
is used for data.
Local variables can only be addressed if the C language source program is
compiled with the -g flag.
If the C language source program is not compiled using the -g flag the local
variable cannot be addressed. The following command displays the value of the
local variable b in a function sample:
.sample.b / x - value of local variable.
.sample.b = x - Address of local variable.
Using Operators in Expressions
You can combine integers, symbols, variables, and register names with the
following operators:
Unary Operators
x (tilde)
- (dash)
* (asterisk)
44
General Programming Concepts
Bitwise complementation
Integer negation
Returns contents of location
Binary Operators:
+ (plus)
- (minus)
* (asterisk)
% (percent)
& (ampersand)
] (right bracket)
| (caret)
# (number sign)
Addition
Subtraction
Multiplication
Integer division
Bitwise conjunction
Bitwise disjunction
Modulo
Round up to the next multiple
The adb debug program uses 32-bit arithmetic. Values that exceed 2,147,483,647
(decimal) are displayed as negative values. The following example shows the
results of assigning two different values to the variable n, and then displaying the
value in both decimal and hexadecimal:
2147483647>n<
n=D
2147483647<
n=X
7fffffff
2147483648>n<
n=D
-2147483648<
n=X
80000000
Unary operators have higher precedence than binary operators. All binary
operators have the same precedence and are evaluated in order from left to right.
Thus, the adb program evaluates the following binary expressions as shown:
2*3+4=d
10
4+2*3=d
18
You can change the precedence of the operations in an expression by using
parentheses. The following example shows how the previous expression is changed
by using parentheses:
4+(2*3)=d
10
The unary operator, * (asterisk), treats the given address as a pointer into the data
segment. An expression using this operator is equal to the value pointed to by that
pointer. For example, the expression:
*0x1234
is equal to the value at the data address 0x1234, whereas the example:
0x1234
is equal to 0x1234.
Customizing the adb Debug Program
Combining Commands on a Single Line
You can give more than one command on a line by separating the commands with
a ; (semicolon). The commands are performed one at a time, starting at the left.
Changes to the current address and format carry over to the next command. If an
Chapter 3. Debugging Programs
45
error occurs, the remaining commands are ignored. For example, the following
sequence displays both the adb variables and then the active subroutines at one
point in the program “Example adb Program: adbsamp2” on page 62:
$v;$c
variables
b = 10000000
d = ec
e = 10000038
m = 108
t = 2f8.
f(0,0) .main+26.
main(0,0,0) start+fa
Creating adb Scripts
You can direct the adb debug program to read commands from a text file instead
of from the keyboard by redirecting the standard input file when you start the adb
program. To redirect standard input, use the input redirection symbol, < (less
than), and supply a file name. For example, use the following command to read
commands from the file script:
adb sample <script
The file must contain valid adb subcommands. Use the adb program script files
when the same set of commands can be used for several different object files.
Scripts can display the contents of core files after a program error. The following
example shows a file containing commands that display information about a
program error. When that file is used as input to the adb program using the
following command to debug the file “Example adb Program: adbsamp2” on
page 62, the specified output is produced.
120$w
4095$s.
f:b:
r
=1n"=======
$v
=1n"=======
$m
=1n"=======
$C
=1n"=======
$e
=1n"=======
$r
0$s
=1n"=======
b,10/8xna
adb Variables ======="
Address Map ======="
C Stack Backtrace ======="
C External Variables ======="
Registers ======="
Data Segment ======="<
$ adb adbsamp2 <script
adbsamp2: running
breakpoint .f: b .f+24
======= adb Variables =======
variables
0 = TBD
1 = TBD
2 = TBD
9 = TBD
b = 10000000
d = ec
e = 10000038
m = 108
t = 2f8
======= Address Map =======
46
General Programming Concepts
[0]? map .adbsamp2.
b1 = 10000000 e1 = 100002f8 f1 = 0
b2 = 200002f8 e2 = 200003e4 f2 = 2f8
[0]/ map .-.
b1 = 0
e1 = 0
f1 = 0
b2 = 0
e2 = 0
f2 = 0
======= C Stack Backtrace =======.
f(0,0) .main+26.
main(0,0,0) start+fa
======= C External Variables =======Full word.
errno: 0.
environ: 3fffe6bc.
NLinit: 10000238.
main: 100001ea.
exit: 1000028c.
fcnt: 0
.loop .count: 1.
f: 100001b4.
NLgetfile: 10000280.
write: 100002e0.
NLinit. .X: 10000238 .
NLgetfile. .X: 10000280 .
cleanup: 100002bc.
exit: 100002c8 .
exit . .X: 1000028c . .
cleanup . .X: 100002bc
mq
cs
ics
pc
r15
r14
r13
r12
r11
r10
r9
r8
r7
r6
r5
r4
r3
r2
r1
r0
f:
======= Registers =======
20003a24 .errno+3634
100000 gt
1000004
100001b4 .f
10000210 .main+26
20000388 .main
200003ec .loop .count
3fffe3d0
3fffe44c
0
20004bcc
200041d8 .errno+3de8
0
200030bc .errno+2ccc
1
200003ec .loop .count
f4240
1
3fffe678
20000380 .f.
b .f+24
======= Data Segment =======
10000000: 103 5313 3800 0 0 2f8 0 ec
10000010: 0 10 1000 38 0 0 0 1f0
10000020: 0 0 0 0 1000 0 2000 2f8
10000030: 0 0 0 0 4 6000 0 6000
10000040: 6e10 61d0 9430 a67 6730 6820 c82e 8
10000050: 8df0 94 cd0e 60 6520 a424 a432 c84e
10000060: 8 8df0 77 cd0e 64 6270 8df0 86
10000070: cd0e 60 6520 a424 a432 6470 8df0 6a
10000080: cd0e 64 c82e 19 8df0 78 cd0e 60
10000090: 6520 a424 a432 c84e 19 8df0 5b cd0e
100000a0: 64 cd2e 5c 7022 d408 64 911 c82e
100000b0: 2e 8df0 63 cd0e 60 6520 a424 a432
100000c0: c84e 2e 8df0 46 cd0e 64 15 6280
100000d0: 8df0 60 cd0e 68 c82e 3f 8df0 4e
100000e0: cd0e 60 6520 a424 a432 c84e 3f 8df0
100000f0: 31 cd0e 64 c820 14 8df0 2b cd0e
10000100:
Chapter 3. Debugging Programs
47
Setting Output Width
Use the $w subcommand to set the maximum width (in characters) of each line of
output created by the adb program. The command has the form:
Width$w
In this format, the Width parameter is an integer that specifies the width in
characters of the display. You can give any width convenient for your display
device. When the adb program is first invoked, the default width is 80 characters.
This command can be used when redirecting output to a line printer or special
output device. For example, the following command sets the display width to 120
characters, a common maximum width for line printers:
120$w
Setting the Maximum Offset
The adb debug program normally displays memory and file addresses as the sum
of a symbol and an offset. This format helps to associate the instructions and data
on the display with a particular function or variable. When the adb program starts
up, it sets the maximum offset to 255, so that symbolic addresses are assigned only
to instructions or data that occur less than 256 bytes from the start of the function
or variable. Instructions or data beyond that point are given numeric addresses.
In many programs, the size of a function or variable is actually larger than 255
bytes. For this reason the adb program lets you change the maximum offset to
accommodate larger programs. You can change the maximum offset by using the
$s subcommand.
The subcommand has the form:
Offset$s
In this format, the Offset parameter is an integer that specifies the new offset. For
example, the following command increases the maximum possible offset to 4095:
4095$s
All instructions and data that are less than 4096 bytes away are given symbolic
addresses. You can disable all symbolic addressing by setting the maximum offset
to zero. All addresses are given numeric values instead.
Setting Default Input Format
To alter the default format for numbers used in commands, use the $d or $o (octal)
subcommands. The default format tells the adb debug program how to interpret
numbers that do not begin with 0 (octal) or 0x (hexadecimal), and how to display
numbers when no specific format is given. Use these commands to work with a
combination of decimal, octal, and hexadecimal numbers.
The $o subcommand sets the radix to 8 and thus sets the default format for
numbers used in commands to octal. After you enter that subcommand, the adb
program displays all numbers in octal format except those specified in some other
format.
48
General Programming Concepts
The format for the $d subcommand is the Radix$d command, where the Radix
parameter is the new value of the radix. If the Radix parameter is not specified, the
$d subcommand sets the radix to a default value of 16. When you first start the
adb program, the default format is hexadecimal. If you change the default format,
you can restore it as necessary by entering the $d subcommand by itself:
$d
To set the default format to decimal, use the following command:
0xa$d
Changing the Disassembly Mode
Use the $i and $n subcommands to force the adb debug program to disassemble
instructions using the specified instruction set and mnemonics. The $i
subcommand specifies the instruction set to be used for disassembly. The $n
subcommand specifies the mnemonics to be used in disassembly.
If no value is entered, these commands display the current settings.
The $i subcommand accepts the following values:
Specifies the instruction set
POWER architectures.
Specifies the instruction set
POWER architecture.
Specifies the instruction set
the POWER architecture.
Specifies the instruction set
Specifies the instruction set
Microprocessor.
com
pwr
pwrx
ppc
601
603
604
ANY
for the common intersection mode of the PowerPC and
and mnemonics for the POWER implementation of the
and mnemonics for the POWER2 implementation of
and mnemonics for the PowerPC architecture.
and mnemonics for the PowerPC 601 RISC
Specifies the instruction set and mnemonics for the PowerPC 603 RISC
Microprocessor.
Specifies the instruction set and mnemonics for the PowerPC 604 RISC
Microprocessor.
Specifies any valid instruction. For instruction sets that overlap, the mnemonics will
default to PowerPC mnemonics.
The $n subcommand accepts the following values:
pwr
ppc
Specifies the mnemonics for the POWER implementation of the POWER architecture.
Specifies the mnemonics for the PowerPC architecture.
Computing Numbers and Displaying Text
You can perform arithmetic calculations while in the adb debug program by using
the = (equal sign) subcommand. This command directs the adb program to display
the value of an expression in a specified format. The command converts numbers
in one base to another, double-checks the arithmetic performed by a program, and
displays complex addresses in simpler form. For example, the following command
displays the hexadecimal number 0x2a as the decimal number 42:
0x2a=d
42
Similarly, the following command displays 0x2a as the ASCII character * (asterisk):
0x2a=c
*
Chapter 3. Debugging Programs
49
Expressions in a command can have any combination of symbols and operators.
For example, the following command computes a value using the contents of the
r0 and r1 registers and the adb variable b.
<r0-12*<r1+<b+5=X
8fa86f95
You can also compute the value of external symbols to check the hexadecimal
value of an external symbol address, by entering:
main+5=X
2000038d
The = (equal sign) subcommand can also display literal strings. Use this feature in
the adb program scripts to display comments about the script as it performs its
commands. For example, the following subcommand creates three lines of spaces
and then prints the message C Stack Backtrace:
=3n"C Stack Backtrace"
Displaying and Manipulating the Source File with the adb Program
The following sections describe how you can use the adb program to display and
manipulate the source file.
Displaying Instructions and Data
The adb program provides several subcommands for displaying the instructions
and data of a given program and the data of a given data file. The subcommands
and their formats are:
Display address
Display instruction
Display value of variable
Address [, Count ] = Format
Address [, Count ] ? Format
Address [, Count ] / Format
In this format, the symbols and variables have the following meaning:
Address
Count
Format
=
?
/
Gives the location of the instruction or data item.
Gives the number of items to be displayed.
Defines how to display the items.
Displays the address of an item.
Displays the instructions in a text segment.
Displays the value of variables.
Forming Addresses
In the adb program addresses are 32-bit values that indicate a specific memory
address. They can, however, be represented in the following forms:
Absolute address
Symbol name
Entry points
The 32-bit value is represented by an 8-digit hexadecimal number,
or its equivalent in one of the other number-base systems.
The location of a symbol defined in the program can be
represented by the name of that symbol in the program.
The entry point to a routine is represented by the name of the
routine preceded by a . (period). For example, to refer to the
address of the start of the main routine, use the following notation:
.main
50
General Programming Concepts
Displacements
Other points in the program can be referred to by using
displacements from entry points in the program. For example, the
following notation references the instruction that is 4 bytes past
the entry point for the symbol main:
.main+4
Displaying an Address
Use the = (equal sign) subcommand to display an address in a given format. This
command displays instruction and data addresses in a simpler form and can
display the results of arithmetic expressions. For example, entering:
main=an
displays the address of the symbol main:
10000370:
The following example shows a command that displays (in decimal) the sum of
the internal variable b and the hexadecimal value 0x2000, together with its output:
<b+0x2000=D
268443648
If a count is given, the same value is repeated that number of times. The following
example shows a command that displays the value of main twice and the output
that it produces:
main,2=x
370 370
If no address is given, the current address is used. After running the above
command once (setting the current address to main), the following command
repeats that function:
,2=x
370 370
If you do not specify a format, the adb debug program uses the last format that
was used with this command. For example, in the following sequence of
commands, both main and one are displayed in hexadecimal:
main=x
370
one=
33c
Displaying the C Stack Backtrace
To trace the path of all active functions, use the $c subcommand. This
subcommand lists the names of all functions that have been called and have not
yet returned control. It also lists the address from which each function was called
and the arguments passed to each function. For example, the following command
sequence sets a breakpoint at the function address .f+2 in the program. The
breakpoint calls the $c subcommand. The program is started, runs to the
breakpoint, and then displays a backtrace of the called C language functions:
.f+2:b$c
:r
adbsamp2:running
.f(0,0) .main+26
.main(0,0,0) start+fa
breakpoint
f+2:
tgte
r2,r2
Chapter 3. Debugging Programs
51
By default, the $c subcommand displays all calls. To display fewer calls, supply a
count of the number of calls to display. For example, the following command
displays only one of the active functions at the preceding breakpoint:
,1$c
.f(0,0) .main+26
Choosing Data Formats
A format is a letter or character that defines how data is to be displayed. The
following are the most commonly used formats:
Letter
a
b
c
d
D
i
n
o
O
r
s
t
u
x
X
Format
The current symbolic address
One byte in octal (displays data associated with instructions, or the high or low
byte of a register)
One byte as a character (char variables)
Halfword in decimal (short variables)
Fullword in decimal (long variables)
Machine instructions in mnemonic format
A new line
Halfword in octal (short variables)
Fullword in octal (long variables)
A blank space
A null-terminated character string (null-terminated arrays of char variables)
A horizontal tab
Halfword as an unsigned integer (short variables)
Halfword in hexadecimal (short variables)
Fullword in hexadecimal (long variables)
For example, the following commands produce the indicated output when using
the example program “Example adb Program: adbsamp” on page 62:
Command
main=o
main=O
main=d
main=D
main=x
main=X
main=u
Response
1560
4000001560
880
536871792
370
20000370
880
A format can be used by itself or combined with other formats to present a
combination of data in different forms. You can combine the a, n, r, and t formats
with other formats to make the display more readable.
Changing the Memory Map
You can change the values of a memory map by using the ?m and /m
subcommands. These commands assign specified values to the corresponding map
entries. The commands have the form:
[,count] ?m b1 e1 f1
[,count] /m b1 e1 f2
The following example shows the results of these commands on the memory map
displayed with the $m subcommand in the previous example:
52
General Programming Concepts
,0?m
10000100
10000470
0
/m
100
100
100
$m
[0] : ?map : 'adbsamp3'
b1 = 0x10000100, e1 = 10000470, f1 = 0
b2 = 0x20000600, e2 = 0x2002c8a4, f2 = 0x600
[1] : ?map : 'shr.o' in library '/usr/ccs/lib/libc.a'
b1 = 0xd00d6200, e1 = 0xd01397bf, f1 = 0xd00defbc
b2 = 0x20000600, e2 = 0x2002beb8, f2 = 0x4a36c
[-] : /map : '-'
b1 = 100,
e1 = 100, f1 = 100
b2 = 0,
e2 = 0,
f2 = 0
To change the data segment values, add an * (asterisk) after the / or ?.
,0?*m 20000270
20000374
270
/*m
200
200
200
$m
[0] : ?map : 'adbsamp3'
b1 = 0x10000100, e1 = 10000470, f1 = 0
b2 = 0x20000270, e2 = 0x20000374, f2 = 0x270
[1] : ?map : 'shr.o' in library '/usr/ccs/lib/libc.a'
b1 = 0xd00d6200, e1 = 0xd01397bf, f1 = 0xd00defbc
b2 = 0x20000600, e2 = 0x2002beb8, f2 = 0x4a36c
[-] : /map : '-'
b1 = 100,
e1 = 100, f1 = 100
b2 = 0,
e2 = 0,
f2 = 0
Patching Binary Files
You can make corrections or changes to any file, including executable binary files,
by starting the adb program with the -w flag and by using the w and W
subcommands.
Locating Values in a File
Locate specific values in a file by using the l and L subcommands. The
subcommands have the form:
?l Value
OR
/l Value
The search starts at the current address and looks for the expression indicated by
Value. The l subcommand searches for 2-byte values. The L subcommand searches
for 4-byte values.
The ?l subcommand starts the search at the current address and continues until the
first match or the end of the file. If the value is found, the current address is set to
that value’s address. For example, the following command searches for the first
occurrence of the file: f symbol in the file “Example adb Program: adbsamp2” on
page 62.
?l .f.
write+a2
The value is found at .write+a2 and the current address is set to that address.
Chapter 3. Debugging Programs
53
Writing to a File
Write to a file by using the w and W subcommands. The subcommands have the
form:
[ Address ] ?w Value
In this format, the Address parameter is the address of the value you want to
change, and the Value parameter is the new value. The w subcommand writes
2-byte values. The W subcommand writes 4-byte values. For example, the
following commands change the word ″This″ to ″The″:
?l .Th.
?W .The.
The W subcommand changes all four characters.
Making Changes to Memory
Make changes to memory whenever a program has run. If you have used an :r
subcommand with a breakpoint to start program operation, subsequent w
subcommands cause the adb program to write to the program in memory rather
than to the file. This command is used to make changes to a program’s data as it
runs, such as temporarily changing the value of program flags or variables.
Using adb Variables
The adb debug program automatically creates a set of its own variables when it
starts. These variables are set to the addresses and sizes of various parts of the
program file as defined in the following table:
Variable
0
1
2
9
b
d
e
m
s
t
Content
Last value printed
Last displacement part of an instruction source
Previous value of the 1 variable
Count on the last $< or $<< command
Base address of the data segment
Size of the data segment
Entry address of the program
″Magic″ number
Size of the stack segment
Size of the text segment
The adb debug program reads the program file to find the values for these
variables. If the file does not seem to be a program file, then the adb program
leaves the values undefined.
To display the values that the adb debug program assigns to these variables, use
the $v subcommand. This subcommand lists the variable names followed by their
values in the current format. The subcommand displays any variable whose value
is not 0 (zero). If a variable also has a non-zero segment value, the variable’s value
is displayed as an address. Otherwise, it is displayed as a number. The following
example shows the use of this command to display the variable values for the
sample program “Example adb Program: adbsamp” on page 62:
$v
Variables
54
General Programming Concepts
0 = undefined
1 = undefined
2 = undefined
9 = undefined
b = 10000000
d = 130
e = 10000038
m = 108
t = 298
Specify the current value of an adb variable in an expression by preceding the
variable name with < (less than sign). The following example displays the current
value of the b base variable:
<b=X
10000000
Create your own variables or change the value of an existing variable by assigning
a value to a variable name with > (greater than sign). The assignment has the
form:
Expression > VariableName
where the Expression parameter is the value to be assigned to the variable and the
VariableName parameter is the variable to receive the value. The VariableName
parameter must be a single letter. For example, the assignment:
0x2000>b
assigns the hexadecimal value 0x2000 to the b variable. Display the contents of b
again to show that the assignment occurred:
<b=X
2000
Finding the Current Address
The adb program has two special variables that keep track of the last address used
in a command and the last address typed with a command. The . (period) variable,
also called the current address, contains the last address used in a command. The ″
(double quotation mark) variable contains the last address typed with a command.
The . and ″ variables usually contain the same address except when implied
commands, such as the newline and | (caret) characters, are used. These characters
automatically increase and decrease the . variable but leave the ) (right parenthesis)
variable unchanged.
Both the . and the ″ variables can be used in any expression. The < (less than sign)
is not required. For example, the following commands display these variables at
the start of debugging with the program “Example adb Program: adbsamp” on
page 62
.=
0.
=
0
Chapter 3. Debugging Programs
55
Displaying External Variables
Use the $e subcommand to display the values of all external variables in the adb
program. External variables are the variables in your program that have global
scope or have been defined outside of any function, and include variables defined
in library routines used by your program, as well as all external variables of
shared libraries.
The $e subcommand is useful to get a list of the names for all available variables
or a summary of their values. The command displays one name on each line with
the variable’s value (if any) on the same line. If the Count parameter is specified,
only the external variables associated with that file are printed.
The following example illustrates the setting of a breakpoint in the sample
program that calls the $e subcommand, and the output that results when the
program runs (be sure to delete any previous breakpoints that you may have set):
.f+2:b,0$e
:r
adbsamp2: running
_errno: 0
_environ: 3fffe6bc
__NLinit: 10000238
_main: 100001ea
_exit: 1000028c
_fcnt: 0
_loop_count: 1
_f: 100001b4
_NLgetfile: 10000280
_write: 100002e0
__NLinit__X: 10000238
_NLgetfile__X: 10000280
__cleanup: 100002bc
__exit: 100002c8
_exit__X: 1000028c
__cleanup__X: 100002bc
breakpoint .f+2: st r2,1c(r1)
Displaying the Address Maps
The adb program prepares a set of maps for the text and data segments in your
program and uses these maps to access items that you request for display. Use the
$m subcommand to display the contents of the address maps. The subcommand
displays the maps for all segments in the program and uses information taken
from either the program and core files or directly from memory.
The $m subcommand displays information similar to the following:
$m
[0] : ?map : 'adbsamp3'
b1 = 0x10000200, e1 = 0x10001839, f1 = 0x10000200
b2 = 0x2002c604, e2 = 0x2002c8a4, f2 = 0x600
[1] : ?map : 'shr.o' in library 'lib/libc.a'
b1 = 0xd00d6200, e1 = 0xd013976f, f1 = 0xd00defbc
b2 = 0x20000600, e2 = 0x2002bcb8, f2 = 0x4a36c
[-] : /map :
'-'
b1 = 0x0000000, e1 = 0x00000000, f1 = 0x00000000
b2 = 0x0000000, e2 = 0x00000000, f2 = 0x00000000
The display defines address-mapping parameters for the text (b1, e1, and f1) and
data (b2, e2, and f2) segments for the two files being used by the adb debug
56
General Programming Concepts
program. This example shows values for the sample program only. The second set
of map values are for the core file being used. Since none was in use, the example
shows the file name as - (dash).
The value displayed inside the square brackets can be used as the Count parameter
in the ?e and ?m subcommands.
adb Debug Program Reference Information
The adb debug program uses addresses, expressions, operators, subcommands,
and variables to organize and manipulate data.
adb Debug Program Addresses
The address in a file associated with a written address is determined by a mapping
associated with that file. Each mapping is represented by two triples (B1, E1, F1)
and (B2, E2, F2). The FileAddress parameter that corresponds to a written Address
parameter is calculated as follows:
B1<=Address<E1=>FileAddress=Address+F1-B1
OR
B2<=Address<E2=>FileAddress=Address+F2-B2
If the requested Address parameter is neither between B1 and E1 nor between B2
and E2, the Address parameter is not valid. In some cases, such as programs with
separated I and D space, the two segments for a file may overlap. If a ? (question
mark) or / (slash) subcommand is followed by an * (asterisk), only the second
triple is used.
The initial setting of both mappings is suitable for normal a.out and core files. If
either file is not of the kind expected, the B1 parameter for that file is set to a
value of 0, the E1 parameter is set to the maximum file size, and the F1 parameter
is set to a value of 0. In this way, the whole file can be examined with no address
translation.
adb Debug Program Expressions
The following expressions are supported by the adb debug program:
. (period)
+ (plus)
| (caret)
″ (double quotes)
Integer
′Cccc’
Specifies the last address used by a subcommand. The last
address is also known as the current address.
Increases the value of . (period) by the current increment.
Decreases the value of . (period) by the current increment.
Specifies the last address typed by a command.
Specifies an octal number if this parameter begins with 0o, a
hexadecimal number if preceded by 0x or #, or a decimal
number if preceded by 0t. Otherwise, this expression specifies a
number interpreted in the current radix. Initially, the radix is 16.
Specifies the ASCII value of up to 4 characters. A \ (backslash)
can be used to escape an ’ (apostrophe).
Chapter 3. Debugging Programs
57
< Name
Symbol
.Symbol
Routine.Name
(Expression)
Reads the current value of the Name parameter. The Name
parameter is either a variable name or a register name. The adb
command maintains a number of variables named by single
letters or digits. If the Name parameter is a register name, the
value of the register is obtained from the system header in the
CoreFile parameter. Use the $r subcommand to see the valid
register names.
Specifies a sequence of uppercase or lowercase letters,
underscores, or digits, though the sequence cannot start with a
digit. The value of the Symbol parameter is taken from the
symbol table in the ObjectFile parameter. An initial _ (underscore)
is prefixed to the Symbol parameter, if needed.
Specifies the entry point of the function named by the Symbol
parameter.
Specifies the address of the Name parameter in the specified C
language routine. Both the Routine and Name parameters are
Symbol parameters. If the Name parameter is omitted, the value is
the address of the most recently activated C stack frame
corresponding to the Routine parameter.
Specifies the value of the expression.
adb Debug Program Operators
Integers, symbols, variables, and register names can be combined with the
following operators:
Unary Operators
*Expression
@Expression
-Expression
xExpression
#Expression
Binary Operators
Expression1+Expression2
Expression1-Expression2
Expression1*Expression2
Expression1%Expression2
Expression1&Expression2
Expression1|Expression2
Expression1#Expression2
Returns contents of the location addressed by the Expression
parameter in the CoreFile parameter.
Returns contents of the location addressed by the Expression
parameter in the ObjectFile parameter.
Performs integer negation.
Performs bit-wise complement.
Performs logical negation.
Performs integer addition.
Performs integer subtraction.
Performs integer multiplication.
Performs integer division.
Performs bit-wise conjunction.
Performs bit-wise disjunction.
Rounds up the Expression1 parameter to the next
multiple of the Expression2 parameter.
Binary operators are left-associative and are less binding than unary operators.
adb Debug Program Subcommands
You can display the contents of a text or data segment with the ? (question mark)
or the / (slash) subcommand. The = (equal sign) subcommand displays a given
address in the specified format. The ? and / subcommands can be followed by an *
(asterisk).
?Format
58
General Programming Concepts
Displays the contents of the ObjectFile parameter starting at the Address
parameter. The value of . (period) increases by the sum of the increment for
each format letter.
/Format
=Format
Displays the contents of the CoreFile parameter starting at the Address parameter.
The value of . (period) increases by the sum of the increment for each format
letter.
Displays the value of the Address parameter. The i and s format letters are not
meaningful for this command.
The Format parameter consists of one or more characters that specify print style.
Each format character may be preceded by a decimal integer that is a repeat count
for the format character. While stepping through a format, the . (period) increments
by the amount given for each format letter. If no format is given, the last format is
used.
The available format letters are as follows:
a
b
c
C
Prints the value of . (period) in symbolic form. Symbols are checked to
ensure that they have an appropriate type.
Prints the addressed byte in the current radix, unsigned.
Prints the addressed character.
Prints the addressed character using the following escape conventions:
v Prints control characters as x (tilde) followed by the corresponding
printing character.
d
D
f
F
i Number
n
o
O
p
q
Q
r
s Number
S Number
t
u
U
x
X
Y
/
?
=
″...″
|
+
newline
v Prints nonprintable characters as x (tilde) <Number>, where Number
specifies the hexadecimal value of the character. The x character prints as
x x (tilde tilde).
Prints in decimal.
Prints long decimal.
Prints the 32-bit value as a floating-point number.
Prints double floating point.
Prints as instructions. Number is the number of bytes occupied by the
instruction.
Prints a new line.
Prints 2 bytes in octal.
Prints 4 bytes in octal.
Prints the addressed value in symbolic form using the same rules for
symbol lookup as the a format letter.
Prints 2 bytes in the current radix, unsigned.
Prints 4 unsigned bytes in the current radix.
Prints a space.
Prints the addressed character until a zero character is reached.
Prints a string using the x (tilde) escape convention. The Number variable
specifies the length of the string including its zero terminator.
Tabs to the next appropriate tab stop when preceded by an integer. For
example, the 8t format command moves to the next 8-space tab stop.
Prints as an unsigned decimal number.
Prints a long unsigned decimal.
Prints 2 bytes in hexadecimal.
Prints 4 bytes in hexadecimal.
Prints 4 bytes in date format.
Local or global data symbol.
Local or global text symbol.
Local or global absolute symbol.
Prints the enclosed string.
Decreases the . (period) by the current increment. Nothing prints.
Increases the . (period) by a value of 1. Nothing prints.
Decreases the . (period) decrements by a value of 1. Nothing prints.
Repeats the previous command incremented with a Count of 1.
Chapter 3. Debugging Programs
59
[?/]lValue Mask
[?/]wValue...
[,Count][?/]m
B1 E1 F1[?/]
>Name
!
$Modifier
60
General Programming Concepts
Words starting at the . (period) are masked with the Mask value and
compared with the Value parameter until a match is found. If L is used, the
match is for 4 bytes at a time instead of 2 bytes. If no match is found, then .
(period) is unchanged; otherwise, . (period) is set to the matched location. If
the Mask parameter is omitted, a value of -1 is used.
Writes the 2-byte Value parameter into the addressed location. If the
command is W, write 4 bytes. If the command is V, write 1 byte.
Alignment restrictions may apply when using the w or W command.
Records new values for the B1, E1, and F1 parameters. If less than three
expressions are given, the remaining map parameters are left unchanged. If
the ? (question mark) or / (slash) is followed by an * (asterisk), the second
segment (B2, E2, F2) of the mapping is changed. If the list is terminated by
? or /, the file (ObjectFile or CoreFile, respectively) is used for subsequent
requests. (For example, the /m? command causes / to refer to the ObjectFile)
file. If the Count parameter is specified, the adb command changes the
maps associated with that file or library only. The $m command shows the
count that corresponds to a particular file. If the Count parameter is not
specified, a default value of 0 is used.
Assigns a . (period) to the variable or register specified by the Name
parameter.
Calls a shell to read the line following ! (exclamation mark).
Miscellaneous commands. The available values for Modifier are:
<File
Reads commands from the specified file and returns to standard
input. If a count is given as 0, the command will be ignored. The
value of the count is placed in the adb 9 variable before the first
command in the File parameter is executed.
<<File
Reads commands from the specified file and returns to standard
input. The <<File command can be used in a file without causing
the file to be closed. If a count is given as 0, the command is
ignored. The value of the count is placed in the adb 9 variable
before the first command in File is executed. The adb 9 variable is
saved during the execution of the <<File command and restored
when <<File completes. There is a limit to the number of <<File
commands that can be open at once.
>File
Sends output to the specified file. If the File parameter is omitted,
output returns to standard output. The File parameter is created if
it does not exist.
b
Prints all breakpoints and their associated counts and commands.
c
Stacks back trace. If the Address parameter is given, it is taken as
the address of the current frame (instead of using the frame
pointer register). If the format letter C is used, the names and
values of all automatic and static variables are printed for each
active function. If the Count parameter is given, only the number
of frames specified by the Count parameter are printed.
d
Sets the current radix to the Address value or a value of 16 if no
address is specified.
e
Prints the names and values of external variables. If a count is
specified, only the external variables associated with that file are
printed.
$Modifier
(continued)
f
Prints the floating-point registers in hexadecimal.
i instruction set
Selects the instruction set to be used for disassembly.
I
Changes the default directory as specified by the -I flag to the
Name parameter value.
m
Prints the address map.
n mnem_set
Selects the mnemonics to be used for disassembly.
o
Sets the current radix to a value of 8.
q
Exits the adb command.
r
Prints the general registers and the instruction addressed by iar
and sets the . (period) to iar. The Number$r parameter prints the
register specified by the Number variable. The Number,Count$r
parameter prints registers Number+Count-1,...,Number.
s
Sets the limit for symbol matches to the Address value. The default
is a value of 255.
v
Prints all non-zero variables in octal.
w
Sets the output page width for the Address parameter. The default
is 80.
P Name Uses the Name value as a prompt string.
?
:Modifier
Prints the process ID, the signal that caused stoppage or
termination, and the registers of $r.
Manages a subprocess. Available modifiers are:
bCommand
Sets the breakpoint at the Address parameter. The breakpoint runs
the Count parameter -1 times before causing a stop. Each time the
breakpoint is encountered, the specified command runs. If this
command sets . (period) to a value of 0, the breakpoint causes a
stop.
cSignal Continues the subprocess with the specified signal. If the Address
parameter is given, the subprocess is continued at this address. If
no signal is specified, the signal that caused the subprocess to stop
is sent. Breakpoint skipping is the same as for the r modifier.
d
Deletes the breakpoint at the Address parameter.
k
Stops the current subprocess, if one is running.
r
Runs the ObjectFile parameter as a subprocess. If the Address
parameter is given explicitly, the program is entered at this point.
Otherwise, the program is entered at its standard entry point. The
Count parameter specifies how many breakpoints are to be ignored
before stopping. Arguments to the subprocess can be supplied on
the same line as the command. An argument starting with < or >
establishes standard input or output for the command. On entry to
the subprocess, all signals are turned on.
sSignal Continues the subprocess in single steps up to the number
specified in the Count parameter. If there is no current subprocess,
the ObjectFile parameter is run as a subprocess. In this case no
signal can be sent. The remainder of the line is treated as
arguments to the subprocess.
Chapter 3. Debugging Programs
61
adb Debug Program Variables
The adb command provides a number of variables. When the adb program is
started, the following variables are set from the system header in the specified core
file. If the CoreFile parameter does not appear to be a core file, these values are set
from the ObjectFile parameter:
0
1
2
9
b
d
e
m
s
t
Last value printed
Last displacement part of an instruction source
Previous value of the 1 variable
Count on the last $< or $<< subcommand
Base address of the data segment
Size of the data segment
Entry address of the program
″Magic″ number
Size of the stack segment
Size of the text segment
Example adb Program: adbsamp
The following sample program is used in this example:
/* Program Listing for adbsamp.c */
char str1[ ] = "This is a character string";
int one = 1;
int number = 456;
long lnum = 1234;
float fpt = 1.25;
char str2[ ] = "This is the second character string";
main()
{
one = 2;
printf("First String = %s\n",str1);
printf("one = %d\n",one);
printf("Number = %d\n",lnum);
printf("Floating point Number = %g\n",fpt);
printf("Second String = %s\n",str2);
}
Compile the program using the cc command to the adbsamp file as follows:
cc -g adbsamp.c -o adbsamp
To start the debug session, enter:
adb adbsamp
Example adb Program: adbsamp2
The following sample program is used in this example:
/*program listing for adbsamp2.c*/
int
fcnt,loop_count;
f(a,b)
int a,b;
{
a = a+b;
fcnt++;
return(a);
}
main()
{
loop_count = 0;
62
General Programming Concepts
while(loop_count <= 100)
{
loop_count = f(loop_count,1);
printf("%s%d\n","Loop count is: ", loop_count);
printf("%s%d\n","fcnt count is: ",fcnt);
}
}
Compile the program using the cc command to the adbsamp2 file with the
following command:
cc -g adbsamp2.c -o adbsamp2
To start the debug session, enter:
adb adbsamp2
Example adb Program: adbsamp3
The following sample program adbsamp3.c contains an infinite recursion of
subfunction calls. If you run this program to completion, it causes a memory fault
error and quits.
int
fcnt,gcnt,hcnt;
h(x,y)
int x,y;
{
int hi;
register int hr;
hi = x+1;
hr = x-y+1;
hcnt++;
hj:
f(hr,hi);
}
g(p,q)
int p,q;
{
int gi;
register int gr;
gi = q-p;
gr = q-p+1;
gcnt++;
gj:
h(gr,gi);
}
f(a,b)
int a,b;
{
int fi;
register int fr;
fi = a+2*b;
fr = a+b;
fcnt++;
fj:
g(fr,fi);
}
main()
{
f(1,1);
}
Compile the program using the cc command to create the adbsamp3 file with the
following command:
cc -g adbsamp3.c -o adbsamp3
Chapter 3. Debugging Programs
63
To start the debug session, enter:
adb adbsamp3
Example of Directory and i-node Dumps in adb Debugging
This example shows how to create adb scripts to display the contents of a
directory and the i-node map of a file system. In the example, the directory is
named dir and contains a variety of files. The file system is associated with the
/dev/hd3 device file (/tmp), which has the necessary permissions to be read by
the user.
To display a directory, create an appropriate script. A directory normally contains
one or more entries. Each entry consists of an unsigned i-node number (i-number)
and a 14-character file name. You can display this information by including a
command in your script file. The adb debug program expects the object file to be
an xcoff format file. This is not the case with a directory. The adb program
indicates that the directory, because it is not an xcoff format file, has a text length
of 0. Use the m command to indicate to the adb program that this directory has a
text length of greater than 0. Therefore, display entries in your adb session by
entering:
,0?m 360 0
For example, the following command displays the first 20 entries separating the
i-node number and file name with a tab:
0,20?ut14cn
You can change the second number, 20, to specify the number of entries in the
directory. If you place the following command at the beginning of the script, the
adb program displays the strings as headings for each column of numbers:
="inumber"8t"Name"
Once you have created the script file, redirect it as input when you start the adb
program with the name of your directory. For example, the following command
starts the adb program on the geo directory using command input from the ddump
script file:
adb geo - <ddump
The minus sign (-) prevents the adb program from opening a core file. The adb
program reads the commands from the script file.
To display the i-node table of a file system, create a new script and then start the
adb program with the file name of the device associated with the file system. The
i-node table of a file system has a complex structure. Each entry contains:
v A word value for status flags
v A byte value for number links
v 2-byte values for the user and group IDs
v A byte and word value for the size
v 8-word values for the location on disk of the file’s blocks
v 2-word values for the creation and modification dates
The following is an example directory dump output:
64
General Programming Concepts
0:
inumber Name
26
.
2
..
27
.estate
28
adbsamp
29
adbsamp.c
30
calc.lex
31
calc.yacc
32
cbtest
68
.profile
66
.profile.bak
46
adbsamp2.c
52
adbsamp2
35
adbsamp.s
34
adbsamp2.s
48
forktst1.c
49
forktst2.c
50
forktst3.c
51
lpp&us1.name
33
adbsamp3.c
241 sample
198 adbsamp3
55
msgqtst.c
56
newsig.c
The i-node table starts at the address 02000. You can display the first entry by
putting the following command in your script file:
02000,-1?on3bnbrdn8un2Y2na
The command specifies several new-line characters for the output display to make
it easier to read.
To use the script file with the i-node table of the /dev/hd3 file, enter the following
command:
adb /dev/hd3 - <script
Each entry in the display has the form:
02000: 073145
0163 0164 0141
0162 10356
28770 8236 25956 27766 25455 8236 25956 25206
1976 Feb 5 08:34:56 1975 Dec 28 10:55:15
Example of Data Formatting in adb Debugging
To display the current address after each machine instruction, enter:
main , 5 ? ia
This produces output such as the following when used with the example program
:
.main:
.main:
.main+4:
.main+8:
.main+c:
.main+10:
.main+14:
mflr 0
st r0, 0x8(r1)
stu rs,
(r1)
li l r4, 0x1
oril r3, r4, 0x0
To make it clearer that the current address does not belong to the instruction that
appears on the same line, add the new-line format character (n) to the command:
Chapter 3. Debugging Programs
65
.main , 5 ? ian
In addition, you can put a number before a formatting character to indicate the
number of times to repeat that format.
To print a listing of instructions and include addresses after every fourth
instruction, use the following command:
.main,3?4ian
This instruction produces the following output when used with the example
program adbsamp:
.main:
mflr 0
st r0, 0x8(r1)
stu r1, -56(r1)
lil r4, 0x1
.main+10:
oril r3, r4, 0x0
bl .f
l r0, Ox40(r1)
ai r1, r1, 0x38
.main+20:
mtlr r0
br
Invalid opcode
Invalid opcode
.main+30:
Be careful where you put the number.
The following command, though similar to the previous command, does not
produce the same output:
main,3?i4an
.main:
.main:
.main+4:
.main+8:
.main+c:
mflr 0
.main+4:
st r0, 0x8(r1)
.main+8:
stu r1,
(r1)
.main+c:
.main+4:
.main+4:
.main+8:
.main+8:
.main+c:
.main+c:
You can combine format requests to provide elaborate displays. For example,
entering the following command displays instruction mnemonics followed by their
hexadecimal equivalent:
.main,-1?i|xn
In this example, the display starts at the address main. The negative count (-1)
causes an indefinite call of the command, so that the display continues until an
error condition (such as end-of-file) occurs. In the format, i displays the mnemonic
instruction at that location, the | (caret) moves the current address back to the
beginning of the instruction, and x re-displays the instruction as a hexadecimal
number. Finally, n sends a newline character to the terminal. The output is similar
to the following, only longer:
.main:
.main:
66
General Programming Concepts
mflr 0
7c0802a6
st r0, 0x8(r1)
9001008
st r1, -56(r1)
9421ffc8
lil r4, 0x1
38800001
oril r3, r4, 0x0
60830000
bl - .f
4bffff71
l r0, 0x40(r1)
80010040
ai r1, r1, 0x38
30210038
mtlr r0
7c0803a6
The following example shows how to combine formats in the ? or / subcommand
to display different types of values when stored together in the same program. It
uses the adbsamp program. For the commands to have variables with which to
work, you must first set a breakpoint to stop the program, and then run the
program until it finds the breakpoint. Use the :b command to set a breakpoint:
.main+4:b
Use the $b command to show that the breakpoint is set:
$b
breakpoints
count bkpt
1
.main+4
command
Run the program until it finds the breakpoint by entering:
:r
adbsamp: running
breakpoint .main+4:
st r0, 0x8(r1)
You can now display conditions of the program when it stopped. To display the
value of each individual variable, give its name and corresponding format in a /
(slash) command. For example, the following command displays the contents of
str1 as a string:
str1/s
str1:
str1:
This is a character string
The following command displays the contents of number as a decimal integer:
number/D
number:
number:
456
You can choose to view a variable in a variety of formats. For example, you can
display the long variable lnum as a 4-byte decimal, octal, and hexadecimal number
by entering the commands:
lnum/D
lnum:
lnum:
lnum/O
lnum:
lnum:
lnum/X
lnum:
lnum:
1234
2322
4d2
You can also examine variables in other formats. For example, the following
command displays some variables as eight hexadecimal values on a line and
continues for five lines:
Chapter 3. Debugging Programs
67
str1,5/8x
str1:
str1: 5468 6973 2069 7320 6120 6368 6172 6163
7465 7220 7374 7269 6e67 0 0 0
0
number: 0
1c8 0
0
0
4d2
0
0
3fa0
0 0
0
5468 6973 2069 7320
7468 6520 7365 636f 6e64 2063 6861 7261
Since the data contains a combination of numeric and string values, display each
value as both a number and a character to see where the actual strings are located.
You can do this with one command:
str1,5/4x4|8Cn
str1:
str1:
5468 6973 2069 7320 This is
6120 6368 6172 6163 a charac
7465 7220 7374 7269 ter stri
6e67 0
0
0
ngx@x@x@x@x@x@
0
1c8 0
0
x@x@xAx<c8>x@x@x@x@
In this case, the command displays four values in hexadecimal, then displays the
same values as eight ASCII characters. The | (caret) is used four times just before
displaying the characters to set the current address back to the starting address for
that line.
To make the display easier to read, you can insert a tab between the values and
characters and give an address for each line:
str1,5/4x4|8t8Cna
str1:
str1:
5468 6973 2069 7320
str1+8: 6120 6368 6172 6163
str1+10: 7465 7220 7374 7269
str1+18: 6e67
0
0
1
number:
number:
0
1c8 0
0
fpt:
This is
a charac
ter stri
ngx@x@x@x@x@xA
x@x@xAx<c8>x@x@x@x@
Example of Tracing Multiple Functions in adb Debugging
Note: The example program used in this section, “Example adb Program:
adbsamp3” on page 63, contains an infinite recursion of subfunction calls. If
you run this program to completion, it causes a memory fault error and quits.
The following example shows how to execute a program under adb control and
carry out the basic debugging operations described in the following sections.
The source program for this example is stored in a file named adbsamp3.c.
Compile this program to an executable file named adbsamp3 using the cc
command:
cc adbsamp3.c -o adbsamp3
Starting the adb Program
To start the session and open the program file, use the following command (no
core file is used):
adb adbsamp3
Setting Breakpoints
First, set breakpoints at the beginning of each function using the :b subcommand:
68
General Programming Concepts
.f:b
.g:b
.h:b
Displaying a Set of Instructions
Next, display the first five instructions in the f function:
.f,5?ia
.f:
.f:
.f+4:
.f+8:
.f+c:
.f+10:
.f+14:
mflr r0
st r0, 0x8(r1)
stu r1, -64(r1)
st r3, 0x58(r1)
st r4, 0x5c(r1)
Display five instructions in function g without their addresses:
.g,5?i
.g:
mflr r0
st r0, 0x8(r1)
stu r1, -64(r1)
st r3, 0x58(r1)
st r4, 0x5c(r1)
Starting the adsamp3 Program
Start the program by entering the following command:
:r
adbsamp3: running
breakpoint .f:
mflr r0
The adb program runs the sample program until it reaches the first breakpoint
where it stops.
Removing a Breakpoint
Since running the program to this point causes no errors, you can remove the first
breakpoint:
.f:d
Continuing the Program
Use the :c subcommand to continue the program:
:c
adbsamp3: running
breakpoint .g:
mflr r0
The adb program restarts the adbsamp3 program at the next instruction. The
program operation continues until the next breakpoint, where it stops.
Tracing the Path of Execution
Trace the path of execution by entering:
$c
.g(0,0) .f+2a
.f(1,1) .main+e
.main(0,0,0) start+fa
The $c subcommand displays a report that shows the three active functions: main,
f and g.
Chapter 3. Debugging Programs
69
Displaying a Variable Value
Display the contents of the fcnt integer variable by entering the command:
fcnt/D
fcnt:
fcnt:
1
Skipping Breakpoints
Next, continue running the program and skip the first 10 breakpoints by entering:
,10:c
adbsamp3: running
breakpoint .g:
mflr r0
The adb program starts the adbsamp3 program and displays the running message
again. It does not stop the program until exactly 10 breakpoints have been
encountered. To ensure that these breakpoints have been skipped, display the
backtrace again:
$c
.g(0,0) .f+2a
.f(2,11) .h+28
.h(10,f) .g+2a
.g(11,20) .f+2a
.f(2,f) .h+28
.h(e,d) .g+2a
.g(f,1c) .f+2a
.f(2,d) .h+28
.h(c,b) .g+2a
.g(d,18) .f+2a
.f(2,b) .h+28
.h(a,9) .g+2a
.g(b,14) .f+2a
.f(2,9) .h+28
.h(8,7) .g+2a
.g(9,10) .f+2a
.f(2,7) .h+28
.h(6,5) .g+2a
.g(7,c) .f+2ae
.f(2,5) .h+28
.h(4,3) .g+2a
.g(5,8) .f+2a
.f(2,3) .h+28
.h(2,1) .g+2a
.g(2,3) .f+2a
.f(1,1) .main+e
.main(0,0,0) start+fa
dbx Symbolic Debug Program Overview
The dbx symbolic debug program allows you to debug a program at two levels:
the source-level and the assembler language-level. Source level debugging allows
you to debug your C, C++, Pascal, or FORTRAN language program. Assembler
language level debugging allows you to debug executable programs at the
machine level. The commands used for machine level debugging are similar to
those used for source-level debugging.
Using the dbx debug program, you can step through the program you want to
debug one line at a time or set breakpoints in the object program that will stop the
debug program. You can also search through and display portions of the source
files for a program.
70
General Programming Concepts
The following sections contain information on how to perform a variety of tasks
with the dbx debug program:
v “Using the dbx Debug Program”
v “Displaying and Manipulating the Source File with the dbx debug Program” on
page 74
v “Examining Program Data” on page 79
v “Debugging at the Machine Level with dbx” on page 85
v “Customizing the dbx Debugging Environment” on page 87
Using the dbx Debug Program
The following sections contain information on how to use the dbx debug program.
Starting the dbx Debug Program
The dbx program can be started with a variety of flags. The three most common
ways to start a debug session with the dbx program are:
v Running the dbx command on a specified object file
v Using the -r flag to run the dbx command on a program that ends abnormally
v Using the -a flag to run the dbx command on a process that is already in
progress
When the dbx command is started, it checks for a .dbxinit file (see “Using the
.dbxinit File” on page 89) in the user’s current directory and in the user’s $HOME
directory. If a .dbxinit file exists, its subcommands run at the beginning of the
debug session. If a .dbxinit file exists in both the home and current directories,
then both are read in that order. Because the current directory .dbxinit file is read
last, its subcommands can supercede those in the home directory.
If no object file is specified, then the dbx program asks for the name of the object
file to be examined. The default is a.out. If the core file exists in the current
directory or a CoreFile parameter is specified, then the dbx program reports the
location where the program faulted. Variables, registers, and memory held in the
core image may be examined until execution of the object file begins. At that point
the dbx debug program prompts for commands.
Running Shell Commands from dbx
You can run shell commands without exiting from the debug program using the sh
subcommand.
If sh is entered without any commands specified, the shell is entered for use until
it is exited, at which time control returns to the dbx program.
Command Line Editing in dbx
The dbx command provides command line editing features similar to those
provided by Korn Shell. vi mode provides vi-like editing features, while emacs
mode gives you controls similar to emacs.
Chapter 3. Debugging Programs
71
You can turn these features on by using dbx subcommand set -o or set edit. So, to
turn on vi-style command line editing, you would type the subcommand set edit
vi or set -o vi.
You can also use the EDITOR environment variable to set the editing mode.
The dbx command saves commands entered to history file .dbxhistory. If the
DBXHISTFILE environment variable is not set, then the history file used is
$HOME/.dbxhistory.
By default, the dbx command saves the text of the last 128 commands entered. The
DBXHISTSIZE environment variable can be used to increase this limit.
Using Program Control
The dbx debug program allows you to set breakpoints (stopping places) in the
program. After entering the dbx program you can specify which lines or addresses
are to be breakpoints and then run the program you want to debug with the dbx
program. The program halts and reports when it reaches a breakpoint. You can
then use dbx commands to examine the state of your program.
An alternative to setting breakpoints is to run your program one line or instruction
at a time, a procedure known as single-stepping.
Setting and Deleting Breakpoints
Use the stop subcommand to set breakpoints in the dbx program. The stop
subcommand halts the application program when certain conditions are fulfilled:
v The Variable is changed when the Variable parameter is specified.
v The Condition is true when the if Condition flag is used.
v The Procedure is called when the in Procedure flag is used.
v The SourceLine line number is reached when the at SourceLine flag is used.
Note: The SourceLine variable can be specified as an integer or as a file
name string followed by a : (colon) and an integer.
After any of these commands, the dbx program responds with a message reporting
the event ID associated with your breakpoint along with an interpretation of your
command.
Running a Program
The run subcommand starts your program. It tells the dbx program to begin
running the object file, reading any arguments just as if they were typed on the
shell command line. The rerun subcommand has the same form as run; the
difference is that if no arguments are passed, the argument list from the previous
execution is used. After your program begins, it continues until one of the
following events occurs:
v The program reaches a breakpoint.
v A signal occurs that is not ignored, such as INTERRUPT or QUIT.
v A multiprocess event occurs while multiprocess debugging is enabled.
v The program performs a load, unload, or loadbind subroutine.
72
General Programming Concepts
Note: The dbx program ignores this condition if the $ignoreload debug
variable is set. This is the default. For more information see the set
subcommand.
v The program completes.
In each case, the dbx debug program receives control and displays a message
explaining why the program stopped.
There are several ways to continue the program once it stops:
cont
detach
return
skip
step
next
Continues the program from where it stopped.
Continues the program from where it stopped, exiting the debug program. This is
useful after you have patched the program and want to continue without the
debug program.
Continues execution until a return to Procedure is encountered, or until the current
procedure returns if Procedure is not specified.
Continues execution until the end of the program or until Number + 1 breakpoints
execute.
Runs one or a specified Number of source lines.
Runs up to the next source line, or runs a specified Number of source lines.
A common method of debugging is to step through your program one line at a
time. The step and next subcommands serve that purpose. The distinction between
these two commands is apparent only when the next source line to be run involves
a call to a subprogram. In this case, the step subcommand stops in the
subprogram; the next subcommand runs until the subprogram has finished and
then stops at the next instruction after the call.
The $stepignore debug variable can be used to modify the behavior of the step
subcommand. See the dbx command in AIX Version 4.3 Commands Reference, Volume
2 for more information.
There is no event number associated with these stops because there is no
permanent event associated with stopping a program.
If your program has multiple threads, they all run normally during the cont, next,
nexti, and step subcommands. These commands act on the running thread (the
thread which stopped execution by hitting a breakpoint), so even if another thread
executes the code which is being stepped, the cont, next, nexti, or step operation
continues until the running thread has also executed that code.
If you want these subcommands to execute the running thread only, you can set
the dbx debug program variable $hold_next; this causes the dbx debug program to
hold all other user threads during cont, next, nexti, and step subcommands.
Note: If you use this feature, remember that a held thread will not be able to
release any locks which it has acquired; another thread which requires one of
these locks could deadlock your program.
Separating dbx Output from Program Output
Use the screen subcommand for debugging programs that are screen-oriented,
such as text editors or graphics programs. This subcommand opens an Xwindow
for dbx command interaction. The program continues to operate in the window in
which it originated. If screen is not used, dbx program output is intermixed with
the screen-oriented program output.
Chapter 3. Debugging Programs
73
Tracing Execution
The trace subcommand tells the dbx program to print information about the state
of the program being debugged while that program is running. The trace
subcommand can slow a program considerably, depending on how much work the
dbx program has to do. There are five forms of program tracing:
v You can single-step the program, printing out each source line that is executed.
The $stepignore debug variable can be used to modify the behavior of the trace
subcommand. See the set subcommand for more information.
v You can restrict the printing of source lines to when the specified procedure is
active. You can also specify an optional condition to control when trace
information is produced.
v You can display a message each time a procedure is called or returned.
v You can print the specified source line when the program reaches that line.
v You can print the value of an expression when the program reaches the specified
source line.
Deleting trace events is the same as deleting stop events. When the trace
subcommand is executed, the event ID associated is displayed along with the
internal representation of the event.
Displaying and Manipulating the Source File with the dbx debug
Program
You can use the dbx debug program to search through and display portions of the
source files for a program.
You do not need a current source listing for the search. The dbx debug program
keeps track of the current file, current procedure, and current line. If a core file
exists, the current line and current file are set initially to the line and file
containing the source statement where the process ended.
Note: This is only true if the process stopped in a location compiled for
debugging.
Changing the Source Directory Path
By default, the dbx debug program searches for the source file of the program
being debugged in the following directories:
v Directory where the source file was located when it was compiled. This
directory is searched only if the compiler placed the source path in the object.
v Current directory.
v Directory where the program is currently located.
You can change the list of directories to be searched by using the -I option on the
dbx invocation line or issuing the use subcommand within the dbx program. For
example, if you moved the source file to a new location since compilation time,
you might want to use one of these commands to specify the old location, the new
location, and some temporary location.
74
General Programming Concepts
Displaying the Current File
The list subcommand allows you to list source lines.
The $ (dollar sign) and @ (at sign) symbols represent SourceLineExpression and are
useful with the list, stop, and trace subcommands. The $ symbol represents the
next line to be run. The @ symbol represents the next line to be listed.
The move subcommand changes the next line number to be listed.
Changing the Current File or Procedure
Use the func and file subcommands to change the current file, current procedure,
and current line within the dbx program without having to run any part of your
program.
Search through the current file for text that matches regular expressions. If a match
is found, the current line is set to the line containing the matching text. The syntax
of the search subcommand is:
/ RegularExpression [/]
? RegularExpression [?]
Searches forward in the current source file for the given
expression.
Searches backward in the current source file for the
given expression.
If you repeat the search without arguments, the dbx command searches again for
the previous regular expression. The search wraps around the end or beginning of
the file.
You can also invoke an external text editor for your source file using the edit
subcommand. You can override the default editor (vi) by setting the EDITOR
environment variable to your desired editor before starting the dbx program.
The dbx program resumes control of the process when the editing session is
completed.
Debugging Programs Involving Multiple Threads
Programs involving multiple user threads call the subroutine pthread_create.
When a process calls this subroutine, the operating system creates a new thread of
execution within the process. When debugging a multi-threaded program, it is
necessary to work with individual threads instead of with processes. The dbx
program only works with user threads: in the dbx documentation, the word thread
is usually used alone to mean user thread. The dbx program assigns a unique
thread number to each thread in the process being debugged, and also supports
the concept of a running and current thread:
Running thread
Current thread
The user thread that was responsible for stopping the program by
hitting a breakpoint. Subcommands that single-step the program
work with the running thread.
The user thread that you are examining. Subcommands that display
information work in the context of the current thread.
By default, the running thread and current thread are the same. You can select a
different current thread by using the thread subcommand. When the thread
Chapter 3. Debugging Programs
75
subcommand displays threads, the current thread line is preceded by a >. If the
running thread is not the same as the current thread, its line is preceded by a *.
Identifying Thread-Related Objects
Threads use mutexes and condition variables to synchronize access to resources.
Threads, mutexes, and condition variables are created with attribute objects that
define how they behave. The dbx program automatically creates several variables
that identify these various thread-related objects. For each object class, dbx
maintains a numbered list and creates an associated variable for each object in the
list. These variable names begin with a $ (dollar sign), followed by a letter
indicating the object class (a, c, m, or t), followed by a number indicating the
object’s position in the class list. The letters and their associated object classes are
as follows:
v a for attributes
v c for condition variables
v m for mutexes
v t for threads.
For example, $t2 corresponds to the second thread in the dbx thread list. In this
case, 2 is the object’s thread number, which is unrelated to the kernel thread
identifier (tid). You can list the objects in each class using the following dbx
subcommands: attribute, condition, mutex, and thread. For example, you can
simply use the thread subcommand to list all threads.
The dbx program automatically defines and maintains the variable
$running_thread, which identifies the thread that was running when a breakpoint
was hit.
Breakpoints and Threads
If your program has multiple user threads, simply setting a breakpoint on a source
line will not guarantee that a particular thread will hit the breakpoint, because
several threads can execute the same code. If any thread hits the breakpoint, all the
threads of the process will stop.
If you want to specify which thread is to hit the breakpoint, you can use the stop
or stopi subcommands to set a conditional breakpoint. The following aliases set the
necessary conditions automatically:
v bfth (Function, ThreadNumber)
v blth (LineNumber, ThreadNumber)
These aliases stop the thread at the specified function or source line number,
respectively. ThreadNumber is the number part of the symbolic thread name as
reported by the thread subcommand (for example, 2 is the ThreadNumber for the
thread name $t2).
For example, the following subcommand stops thread $t1 at function func1:
(dbx) bfth (func1, 1)
and the following subcommand stops thread $t2 at source line 103:
(dbx) blth (103, 2)
If no particular thread was specified with the breakpoint, any thread that executes
the code where the breakpoint is set could become the running thread.
76
General Programming Concepts
Thread-Related subcommands
The dbx debug program has the following subcommands that enable you to work
with individual attribute objects, condition variables, mutexes, and threads:
attribute
condition
mutex
thread
Displays information about all attribute objects, or attribute objects specified
by attribute number.
Displays information about all condition variables, condition variables that
have waiting threads, condition variables that have no waiting threads, or
condition variables specified by condition number.
Displays information about all mutexes, locked or unlocked mutexes, or
mutexes specified by mutex number.
Displays information about threads, selects the current thread, and holds and
releases threads.
A number of subcommands that do not deal with threads directly are also affected
when used to debug a multi-threaded program:
print
stop, stopi
If passed a symbolic object name reported by the thread, mutex,
condition, or attribute subcommands, displays status
information about the object. For example, to display the third
mutex and the first thread:
(dbx) print $m3, $t1
If a single thread hits a breakpoint, all other threads are stopped
as well, and the process timer is halted. This means that the
breakpoint does not affect the global behavior of the process.
These normal breakpoints are global, meaning that they can stop
any thread.
If you want to specify which thread will hit the breakpoint, you
must use a condition as shown in the following example, which
ensures that only thread $t5 can hit the breakpoint set on
function f1:
(dbx) stopi at &f1 if ($running_thread == 5)
step, next, nexti
stepi
trace, tracei
This syntax also works with the stop subcommand. Another way
to specify these conditions is to use the bfth and blth aliases, as
explained in the section “Breakpoints and Threads” on page 76.
All threads resume execution during the step, next, and nexti
subcommands. If you want to step the running thread only, set
the $hold_next dbx debug program variable; this holds all
threads except the running thread during these subcommands.
The stepi subcommand executes the specified number of
machine instructions in the running thread only. Other threads in
the process being debugged will not run during the stepi
subcommand.
A specific user thread can be traced by specifying a condition
with the trace and tracei subcommands as shown in the
following example, which traces changes made to var1 by thread
$t1:
(dbx) trace var1 if ($running_thread == 1)
If a multi-threaded program does not protect its variables with mutexes, the dbx
debug program behavior may be affected by the resulting race conditions. For
example, suppose that your program contains the following lines:
59 var = 5;
Chapter 3. Debugging Programs
77
60 printf(″var=%d\n″, var);
If you want to verify that the variable is being initialized correctly, you could type:
stop at 60 if var==5
The dbx debug program puts a breakpoint at line 60, but if access to the variable is
not controlled by a mutex, another thread could update the variable before the
breakpoint is hit. This means that the dbx debug program would not see the value
of five and would continue execution.
Debugging Programs Involving Multiple Processes
Programs involving multiple processes call the fork and exec subroutines. When a
program forks, the operating system creates another process that has the same
image as the original. The original process is called the parent process, the created
process is called the child process.
When a process performs an exec subroutine, a new program takes over the
original process. Under normal circumstances, the debug program debugs only the
parent process. However, the dbx program can follow the execution and debug the
new processes when you issue the multproc subcommand. The multproc
subcommand enables multiprocess debugging.
When multiprocess debugging is enabled and a fork occurs, the parent and child
processes are halted. A separate virtual terminal Xwindow is opened for a new
version of the dbx program to control running of the child process:
(dbx) multproc on
(dbx) multproc
multi-process debugging is enabled
(dbx) run
78
General Programming Concepts
When the fork occurs, execution is stopped in the parent, and the dbx program
displays the state of the program:
application forked, child pid = 422, process stopped, awaiting input
stopped due to fork with multiprocessing enabled in fork at 0x1000025a (fork+0xe)
(dbx)
Another virtual terminal Xwindow is then opened to debug the child process:
debugging child, pid=422, process stopped, awaiting input
stopped due to fork with multiprocessing enabled in fork at 0x10000250
10000250 (fork+0x4) )80010010
1
r0,0x10(r1)
(dbx)
At this point, two distinct debugging sessions are running. The debugging session
for the child process retains all the breakpoints from the parent process, but only
the parent process can be rerun.
When a program performs an exec subroutine in multiprocess debugging mode,
the program overwrites itself, and the original symbol information becomes
obsolete. All breakpoints are deleted when the exec subroutine runs; the new
program is stopped and identified for the debugging to be meaningful. The dbx
program attaches itself to the new program image, makes a subroutine to
determine the name of the new program, reports the name, and then prompts for
input. The prompt is similar to the following:
(dbx) multproc
Multi-process debugging is enabled
(dbx) run
Attaching to program from exec . . .
Determining program name . . .
Successfully attached to /home/user/execprog . . .
Reading symbolic information . . .
(dbx)
If a multi-threaded program forks, the new child process will have only one
thread. The process should call the exec subroutine. Otherwise, the original symbol
information is retained, and thread-related subcommands (such as thread) display
the objects of the parent process, which are obsolete. If an exec subroutine is called,
the original symbol information is reinitialized, and the thread-related
subcommands display the objects in the new child process.
It is possible to follow the child process of a fork without a new Xwindow being
opened by using the child flag of the multproc subcommand. When a forked
process is created, dbx follows the child process. The parent flag of the multproc
subcommand causes dbx to stop when a program forks, but then follows the
parent. Both the child and parent flags follow an execed process. These flags are
very useful for debugging programs when Xwindows is not running.
Examining Program Data
The following topics contain more information about examining, testing, and
modifying program data.
Handling Signals
The dbx debug program can either trap or ignore signals before they are sent to
your program. Each time your program is to receive a signal, the dbx program is
notified. If the signal is to be ignored, it is passed to your program; otherwise, the
Chapter 3. Debugging Programs
79
dbx program stops the program and notifies you that a signal has been trapped.
The dbx program cannot ignore the SIGTRAP signal if it comes from a process
outside of the debug process. In a multi-threaded program, a signal can be sent to
a particular thread via the pthread_kill subroutine. By default, the dbx program
stops and notifies you that a signal has been trapped. If you request a signal be
passed on to your program using the ignore subcommand, the dbx program
ignores the signal and passes it on to the thread. Use the catch and ignore
subcommands to change the default handling.
In the following example, a program uses SIGGRANT and SIGREQUEST to
handle allocation of resources. In order for the dbx program to continue each time
one of these signals is received, enter:
(dbx) ignore GRANT
(dbx) ignore SIGREQUEST
(dbx) ignore
CONT CLD ALARM KILL GRANT REQUEST
The dbx debug program can block signals to your program if you set the
$sigblock variable. By default, signals received through the dbx program are sent
to the source program or the object file specified by the dbx ObjectFile parameter. If
the $sigblock variable is set using the set subcommand, signals received by the
dbx program are not passed to the source program. If you want a signal to be sent
to the program, use the cont subcommand and supply the signal as an operand.
You can use this feature to interrupt execution of a program running under the
dbx debug program. Program status can be examined before continuing execution
as usual. If the $sigblock variable is not set, interrupting execution causes a
SIGINT signal to be sent to the program. This causes execution, when continued,
to branch to a signal handler if one exists.
The following example program illustrates how execution using the dbx debug
program changes when the $sigblock variable is set:
#include <signal.h>
#include <stdio.h>
void inthand( ) {
printf("\nSIGINT received\n");
exit(0);
}
main( )
{
signal(SIGINT, inthand);
while (1) {
printf(".");
fflush(stdout);
sleep(1);
}
}
The following sample session with the dbx program uses the preceding program as
the source file. In the first run of the program, the $sigblock variable is not set.
During rerun, the $sigblock variable is set. Comments are placed between angle
brackets to the right:
dbx version 3.1.
Type 'help' for help.
reading symbolic information ...
(dbx) run
80
General Programming Concepts
.........|C
<User pressed Ctrl-C here!>
interrupt in sleep at 0xd00180bc
0xd00180bc (sleep+0x40) 80410014
1
r2,0x14(r1)
(dbx) cont
SIGINT received
execution completed
(dbx) set $sigblock
(dbx) rerun
[ looper ]
..............|C
<User pressed Ctrl-C here!>
interrupt in sleep at 0xd00180bc
0xd00180bc (sleep+0x40) 80410014
1
r2,0x14(r1)
(dbx) cont
....|C <Program did not receive signal, execution continued>
interrupt in sleep at 0xd00180bc
0xd00180bc (sleep+0x40) 80410014
1
r2,0x14(r1)
(dbx) cont 2
<End program with a signal 2>
SIGINT received
execution completed
(dbx)
Calling Procedures
You can call your program procedures from the dbx program to test different
arguments. You can also call diagnostic routines that format data to aid in
debugging. Use the call subcommand or the print subcommand to call a
procedure.
Displaying a Stack Trace
To list the procedure calls preceding a program halt, use the where command.
In the following example, the executable object file, hello, consists of two source
files and three procedures, including the standard procedure main. The program
stopped at a breakpoint in procedure sub2.
(dbx) run
[1] stopped in sub2 at line 4 in file "hellosub.c"
(dbx) where
sub2(s = "hello", n = 52), line 4 in "hellosub.c"
sub(s = "hello", a = -1, k = delete), line 31 in "hello.c"
main(), line 19 in "hello.c"
The stack trace shows the calls in reverse order. Starting at the bottom, the
following events occurred:
1. Shell called main.
2. main called sub procedure at line 19 with values s = ″hello″, a = -1, and k =
delete.
3. sub called sub2 procedure at line 31 with values s = ″hello″ and n = 52.
4. The program stopped in sub2 procedure at line 4.
Note: Set the debug program variable $noargs to turn off the display of
arguments passed to procedures.
You can also display portions of the stack with the up and down subcommands.
Chapter 3. Debugging Programs
81
Displaying and Modifying Variables
To display an expression, use the print subcommand. To print the names and
values of variables, use the dump subcommand. If the given procedure is a period,
then all active variables are printed. To modify the value of a variable, use the
assign subcommand.
In the following example, a C program has an automatic integer variable x with
value 7, and s and n parameters in the sub2 procedure:
(dbx) print x, n
7 52
(dbx) assign x = 3*x
(dbx) print x
21
(dbx) dump
sub2(s = "hello", n = 52)
x = 21
Displaying Thread-Related Information
To display information on user threads, mutexes, conditions, and attribute objects,
use the thread, mutex, condition, and attribute subcommands. You can also use
the print subcommand on these objects. In the following example, the running
thread is thread 1. The user sets the current thread to be thread 2, lists the threads,
prints information on thread 1, and finally prints information on several
thread-related objects.
(dbx) thread current 2
(dbx) thread
thread state-k wchan state-u
*$t1
run
running
>$t2
run
running
k-tid mode held scope function
12755 u no pro main
12501 k no sys thread_1
(dbx) print $t1
(thread_id = 0x1, state = run, state_u = 0x0, tid = 0x31d3, mode = 0x1, held = 0x0,
priority = 0x3c, policy = other, scount = 0x1, cursig = 0x5, attributes = 0x200050f8)
(dbx) print $a1,$c1,$m2
(attr_id = 0x1, type = 0x1, state = 0x1, stacksize = 0x0, detachedstate = 0x0,
process_shared = 0x0, contentionscope = 0x0, priority = 0x0, sched = 0x0, inherit = 0x0,
protocol = 0x0, prio_ceiling = 0x0) (cv_id = 0x1, lock = 0x0,
semaphore_queue = 0x200032a0, attributes = 0x20003628) (mutex_id = 0x2, islock = 0x0,
owner = (nil), flags = 0x1, attributes = 0x200035c8)
Scoping of Names
Names resolve first using the static scope of the current function. The dynamic
scope is used if the name is not defined in the first scope. If static and dynamic
searches do not yield a result, an arbitrary symbol is chosen and the message using
QualifiedName is printed. You can override the name resolution procedure by
qualifying an identifier with a block name (such as Module.Variable). Source files
are treated as modules named by the file name without the suffix. For example, the
x variable, which is declared in the sub procedure inside the hello.c file, has the
fully qualified name hello.sub.x. The program itself has a period for a name.
The which and whereis subcommands can be helpful in determining which
symbol is found when multiple symbols with the same name exist.
82
General Programming Concepts
Using Operators and Modifiers in Expression
The dbx program can display a wide range of expressions. Specify expressions
with a common subset of C and Pascal syntax, with some FORTRAN extensions.
* (asterisk) or | (caret)
[ ] (brackets) or ( ) (parentheses)
. (period)
& (ampersand)
.. (two periods)
Denotes indirection or pointer
dereferencing.
Denotes subscript array expressions.
Use this field reference operator with
pointers and structures. This makes
the C operator -> (arrow)
unnecessary, although it is allowed.
Gets the address of a variable.
Separates the upper and lower
bounds when specifying a subsection
of an array. For example: n[1..4].
The following types of operations are valid in expressions:
Algebraic
Bitwise
Logical
Comparison
Other
=, -, *,/(floating division), div (integral division), mod, exp (exponentiation)
-, I, bitand, xor, x, <<, >>
or, and, not, II, &&
<, >, <=, >=, <> or !=, = or ==
sizeof
Logical and comparison expressions are allowed as conditions in stop and trace
subcommands.
Checking of Expression Types
The dbx debug program checks expression types. You can override the expression
type by using a renaming or casting operator. There are three forms of type
renaming:
Typename (Expression)
Expression \ Typename
(Typename) Expression
Note: When you cast to or from a structure, union, or class, the casting is
left-justified. However, when casting from a class to a base class, C++ syntax
rules are followed.
For example, to rename the x variable where x is an integer with a value of 97,
enter:
(dbx) print char (x), x \ char, (char) x, x,
'a' 'a' 'a' 97
The following examples show how you can use the (Typename) Expression form of
type renaming:
print (float) i
print ((struct qq *) void_pointer)->first_element
The following restrictions apply to C-style typecasting for the dbx debug program:
v The FORTRAN types (integer*1, integer*2, integer*4, logical*1, logical*2,
logical*4, and so on) are not supported as cast operators.
Chapter 3. Debugging Programs
83
v If an active variable has the same name as one of the base types or user-defined
types, the type cannot be used as a cast operator for C-style typecasting.
The whatis subcommand prints the declaration of an identifier, which you can
then qualify with block names.
Use the $$TagName construct to print the declaration of an enumeration, structure,
or union tag (or the equivalent in Pascal).
The type of the assign subcommand expression must match the variable type you
assigned. If the types do not match, an error message is displayed. Change the
expression type using a type renaming. Disable type checking by setting a special
dbx debug program $unsafeassign variable.
Folding Variables to Lowercase and Uppercase
By default, the dbx program folds symbols based on the current language. If the
current language is C, C++, or undefined, the symbols are not folded. If the current
language is FORTRAN or Pascal, the symbols are folded to lowercase. The current
language is undefined if the program is in a section of code that has not been
compiled with the debug flag. You can override default handling with the case
subcommand.
Using the case subcommand without arguments displays the current case mode.
The FORTRAN and Pascal compilers convert all program symbols to lowercase;
the C compiler does not.
Changing Print Output with Special Debug Program Variables
Use the set subcommand to set the following special dbx debug program variables
to get different results from the print subcommand:
$hexints
$hexchars
$hexstrings
$octints
$expandunions
Prints
Prints
Prints
Prints
Prints
integer expressions in hexadecimal.
character expressions in hexadecimal.
the address of the character string, not the string itself.
integer expressions in octal.
fields within a union.
Set and unset the debug program variables to get the desired results. For example:
(dbx) whatis x; whatis i; whatis s
int x;
char i;
char *s;
(dbx) print x, i, s
375 'c' "hello"
(dbx) set $hexstrings; set $hexints; set $hexchars
(dbx) print x, i, s
0x177 0x63 0x3fffe460
(dbx) unset $hexchars; set $octints
(dbx) print x, i
0567 'c'
84
General Programming Concepts
Debugging at the Machine Level with dbx
You can use the dbx debug program to examine programs at the assembly
language level. You can display and modify memory addresses, display assembler
instructions, single-step instructions, set breakpoints and trace events at memory
addresses, and display the registers.
In the commands and examples that follow, an address is an expression that
evaluates to a memory address. The most common forms of addresses are integers
and expressions that take the address of an identifier with the & (ampersand)
operator. You can also specify an address as an expression enclosed in parentheses
in machine-level commands. Addresses can be composed of other addresses and
the operators + (plus), - (minus), and indirection (unary *).
The following sections contain more information on debugging at the machine
level with the dbx program.
Using Machine Registers
Use the registers subcommand to see the values of the machine registers. Registers
are divided into three groups: general-purpose, floating-point, and system-control.
General-purpose registers:
General-purpose registers are denoted by $rNumber, where Number represents the
number of the register.
Note: The register value may be set to a hexadecimal value of 0xdeadbeef.
This is an initialization value assigned to all general-purpose registers at
process initialization.
Floating-point registers:
Floating-point registers are denoted by $frNumber, where Number represents the
number of the register. Floating-point registers are not displayed by default. Unset
the $noflregs debug program variable to enable the floating-point register display
(unset $noflregs).
System-control registers:
Supported system-control registers are denoted by:
v The Instruction Address register, $iar or $pc
v The Condition Status register, $cr
v The Multiplier Quotient register, $mq
v The Machine State register, $msr
v The Link register, $link
v The Count register, $ctr
v The Fixed Point Exception register, $xer
v The Transaction ID register, $tid
v The Floating-Point Status register, $fpscr
Chapter 3. Debugging Programs
85
Examining Memory Addresses
Use the following command format to print the contents of memory starting at the
first address and continuing up to the second address, or until the number of items
specified by the Count variable are displayed. The Mode specifies how memory is
to print.
Address, Address / [Mode][> File]
Address / [Count][Mode] [> File]
If the Mode variable is omitted, the previous mode specified is reused. The initial
mode is X. The following modes are supported:
b
c
D
d
f
g
h
i
lld
llo
llu
llx
O
o
q
s
X
x
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
Prints
a byte in octal.
a byte as a character.
a long word in decimal.
a short word in decimal.
a single-precision floating-point number.
a double-precision floating-point number.
a byte in hexadecimal.
the machine instruction.
an 8-byte signed decimal number.
an 8-byte unsigned octal number.
an 8-byte unsigned decimal number.
an 8-byte unsigned hexadecimal number.
a long word in octal.
a short word in octal.
an extended-precision floating-point number.
a string of characters terminated by a null byte.
a long word in hexadecimal.
a short word in hexadecimal.
In the following example, expressions in parentheses can be used as an address:
(dbx) print &x
0x3fffe460
(dbx) &x/X
3fffe460: 31323300
(dbx) &x,&x+12/x
3fffe460: 3132 3300 7879 7a5a 5958 5756 003d 0032
(dbx) ($pc)/2i
100002cc (sub) 7c0802a6
mflr
r0
100002d0 (sub + 0x4)
bfc1fff8
stm
r30,-8(r1)
Running a Program at the Machine Level
The commands for debugging your program at the machine-level are similar to
those at the symbolic level. The stopi subcommand stops the machine when the
address is reached, the condition is true, or the variable is changed. The tracei
subcommands are similar to the symbolic trace commands. The stepi subcommand
executes either one or the specified Number of machine instructions.
If you performed another stepi subcommand at this point, you would stop at
address 0x10000618, identified as the entry point of procedure printf. If you do
not intend to stop at this address, you could use the return subcommand to
continue execution at the next instruction in sub at address 0x100002e0. At this
point, the nexti subcommand will automatically continue execution to 0x10000428.
86
General Programming Concepts
If your program has multiple threads, the symbolic thread name of the running
thread is displayed when the program stops. For example:
stopped in sub at 0x100002d4 ($t4)
10000424 (sub+0x4) 480001f5 bl 0x10000618 (printf)
Debugging fdpr Reordered Executables
You can debug programs that have been reordered with fdpr (feedback directed
program restructuring, part of Performance Toolbox for AIX) at the instruction
level. If optimization options -R0 or -R2 are used, additional information is
provided enabling dbx to map most reordered instruction addresses to the
corresponding addresses in the original executable as follows:
0xRRRRRRRR = fdpr[0xYYYYYYYY]
In this example, 0xRRRRRRRR is the reordered address and 0xYYYYYYYY is the
original address. In addition, dbx uses the traceback entries in the original
instruction area to find associated procedure names for the stopped in message,
the func subcommand, and the traceback.
(dbx) stepi
stopped in proc_d at 0x1000061c = fdpr[0x10000278]
0x1000061c (???) 9421ffc0
stwu r1,-64(r1)
(dbx)
In the preceding example, dbx indicates the program is stopped in the proc_d
subroutine at address 0x1000061c in the reordered text section originally located at
address 0x10000278. For more information about fdpr, see the fdpr command.
Displaying Assembly Instructions
The listi subcommand for the dbx command displays a specified set of instructions
from the source file. In the default mode, the dbx program lists the instructions for
the architecture on which it is running. You can override the default mode with the
$instructionset and $mnemonics variables of the set subcommand for the dbx
command.
For more information on displaying instructions or disassembling instructions, see
the listi subcommand for the dbx command. For more information on overriding
the default mode, see the $instructionset and $mnemonics variables of the set
subcommand for the dbx command.
Customizing the dbx Debugging Environment
You can customize the debugging environment by creating subcommand aliases
and by specifying options in the .dbxinit file. You can read dbx subcommands
from a file using the -c flag. The following sections contain more information about
customization options.
Defining a New dbx Prompt
The dbx prompt is normally the name used to start the dbx program. If you
specified /usr/ucb/dbx a.out on the command line, then the prompt is
/usr/ucb/dbx.
Chapter 3. Debugging Programs
87
You can change the prompt with the prompt subcommand, or by specifying a
different prompt in the prompt line of the .dbxinit file. Changing the prompt in
the .dbxinit file causes your prompt to be used instead of the default each time
you initialize the dbx program.
For example, to initialize the dbx program with the debug prompt debug—>, enter
the following line in your .dbxinit file:
prompt "debug—>"
Creating dbx Subcommand Aliases
You can build your own commands from the dbx primitive subcommand set. The
following commands allow you to build a user alias from the arguments specified.
All commands in the replacement string for the alias must be dbx primitive
subcommands. You can then use your aliases in place of the dbx primitives.
The alias subcommand with no arguments displays the current aliases in effect;
with one argument the command displays the replacement string associated with
that alias.
alias [AliasName[ CommandName] ]
alias AliasName ″CommandString″
alias AliasName (Parameter1, Parameter2, . . . ) ″CommandString″
The first two forms of the alias subcommand are used to substitute the
replacement string for the alias each time it is used. The third form of aliasing is a
limited macro facility. Each parameter specified in the alias subcommand is
substituted in the replacement string.
The following aliases and associated subcommand names are defaults:
attr
bfth
blth
c
cv
d
e
h
j
l
m
mu
n
p
q
r
s
st
t
th
x
attribute
stop (in given thread at specified function)
stop (in given thread at specified source line)
cont
condition
delete
edit
help
status
list
map
mutex
next
print
quit
run
step
stop
where
thread
registers
You can remove an alias with the unalias command.
88
General Programming Concepts
Using the .dbxinit File
Each time you begin a debugging session, the dbx program searches for special
initialization files named .dbxinit, which contain lists of dbx subcommands to
execute. These subcommands are executed before the dbx program begins to read
subcommands from standard input. When the dbx command is started, it checks
for a .dbxinit file in the user’s current directory and in the user’s $HOME
directory. If a .dbxinit file exists, its subcommands run at the beginning of the
debug session. If a .dbxinit file exists in both the home and current directories,
then both are read in that order. Because the current directory .dbxinit file is read
last, its subcommands can supersede those in the home directory.
Normally, the .dbxinit file contains alias subcommands, but it can contain any
valid dbx subcommands. For example:
$ cat .dbxinit
alias si "stop in"
prompt "dbg—>"
$ dbx a.out
dbx version 3.1
Type 'help' for help.
reading symbolic information
dbg—> alias
si stop in
t
where . . .
dbg—>
. . .
Reading dbx Subcommands from a File
The -c invocation option and .dbxinit file provide mechanisms for executing dbx
subcommands before reading from standard input. When the -c option is specified,
the dbx program does not search for a .dbxinit file. Use the source subcommand
to read dbx subcommands from a file once the debugging session has begun.
After executing the list of commands in the cmdfile file, the dbx program displays
a prompt and waits for input.
You can also use the -c option to specify a list of subcommands to be executed
when initially starting the dbx program.
List of dbx Subcommands
The commands and subcommands for the dbx debug program are located in the
AIX Version 4.3 Commands Reference.
The dbx debug program provides subcommands for performing the following task
categories:
Setting and Deleting Breakpoints
clear
cleari
delete
status
stop
Removes all stops at a given source line.
Removes all breakpoints at an address.
Removes the traces and stops corresponding to the specified numbers.
Displays the currently active trace and stop subcommands.
Stops execution of the application program.
Chapter 3. Debugging Programs
89
Running Your Program
cont
detach
down
goto
gotoi
next
nexti
rerun
return
run
skip
step
stepi
up
90
General Programming Concepts
Continues running the program from the current breakpoint until the program
finishes or another breakpoint is encountered.
Exits the debug program, but continues running the application.
Moves a function down the stack.
Causes the specified source line to be the next line run.
Changes program counter addresses.
Runs the application program up to the next source line.
Runs the application program up to the next source instruction.
Begins running an application.
Continues running the application program until a return to the specified
procedure is reached.
Begins running an application.
Continues execution from the current stopping point.
Runs one source line.
Runs one source instruction.
Move a function up the stack.
Tracing Program Execution
Prints tracing information.
Turns on tracing.
Displays a list of all active procedures and functions.
trace
tracei
where
Ending Program Execution
quit
Quits the dbx debug program.
Displaying the Source File
Invokes an editor on the specified file.
Changes the current source file to the specified file.
Changes the current function to the specified function or procedure.
Displays lines of the current source file.
Lists instructions from the application.
Changes the next line to be displayed.
Searches forward in the current source file for a pattern.
Searches backward in the current source file for a pattern.
Sets the list of directories to be searched when looking for a file.
edit
file
func
list
listi
move
/ (Search)
? (Search)
use
Printing and Modifying Variables, Expressions, and Types
Assigns a value to a variable.
Changes the way in which dbx interprets symbols.
Displays the names and values of variables in the specified procedure.
Prints the value of an expression or runs a procedure and prints the return
code.
Assigns a value to a nonprogram variable.
Deletes a nonprogram variable.
Displays the declaration of application program components.
Displays the full qualifications of all the symbols whose names match the
specified identifier.
Displays the full qualification of the specified identifier.
assign
case
dump
print
set
unset
whatis
whereis
which
Thread Debugging
attribute
condition
mutex
thread
Displays
Displays
Displays
Displays
information about all or selected attributes objects.
information about all or selected condition variables.
information about all or selected mutexes.
and controls threads.
Multiprocess Debugging
Enables or disables multiprocess debugging.
multproc
Procedure Calling
call
print
Runs the object code associated with the named procedure or function.
Prints the value of an expression or runs a procedure and prints the return code.
Signal Handling
catch
ignore
Starts trapping a signal before that signal is sent to the application program.
Stops trapping a signal before that signal is sent to the application program.
Chapter 3. Debugging Programs
91
Machine-Level Debugging
display memory
gotoi
map
nexti
registers
stepi
stopi
tracei
Displays the contents of memory.
Changes program counter addresses.
Displays address maps and loader information for the application
program.
Runs the application program up to the next machine instruction.
Displays the values of all general-purpose registers, system-control
registers, floating-point registers, and the current instruction register.
Runs one source instruction.
Sets a stop at a specified location.
Turns on tracing.
Debugging Environment Control
alias
help
prompt
screen
sh
source
unalias
92
General Programming Concepts
Displays and assigns aliases for dbx subcommands.
Displays help information for dbx subcommands or topics.
Changes the dbx prompt to the specified string.
Opens an Xwindow for dbx command output.
Passes a command to the shell for execution.
Reads dbx commands from a file.
Removes an alias.
Chapter 4. Error Notification
The Error Notification object class specifies the conditions and actions to be taken
when errors are recorded in the system error log. The user specifies these
conditions and actions in an Error Notification object.
Each time an error is logged, the error notification daemon determines if the error
log entry matches the selection criteria of any of the Error Notification objects. If
matches exist, the daemon runs the programmed action, also called a notify
method, for each matched object.
The Error Notification object class is located in the /etc/objrepos/errnotify file.
Error Notification objects are added to the object class by using Object Data
Manager (ODM) commands. Error Notification objects contain the following
descriptors:
en_alertflg
Identifies whether the error is alertable. This descriptor is
provided for use by alert agents associated with network
management applications. The valid alert descriptor values are:
TRUE
alertable
FALSE not alertable
en_class
en_crcid
en_label
© Copyright IBM Corp. 1997, 1999
Identifies the class of the error log entries to match. The valid
en_class descriptor values are:
H
Hardware Error class
S
Software Error class
O
Messages from the errlogger command
U
Undetermined
Specifies the error identifier associated with a particular error.
Specifies the label associated with a particular error identifier as
defined in the output of the errpt -t command.
93
en_method
Specifies a user-programmable action, such as a shell script or
command string, to be run when an error matching the selection
criteria of this Error Notification object is logged. The error
notification daemon uses the sh -c command to execute the
notify method.
The following key words are automatically expanded by the
error notification daemon as arguments to the notify method.
en_name
en_persistenceflg
$1
Sequence number from the error log entry
$2
Error ID from the error log entry
$3
Class from the error log entry
$4
Type from the error log entry
$5
Alert flags value from the error log entry
$6
Resource name from the error log entry
$7
Resource type from the error log entry
$8
Resource class from the error log entry
$9
Error label from the error log entry
Uniquely identifies the object. The creator uses this unique name
when removing the object.
Designates whether the Error Notification object should be
automatically removed when the system is restarted. For
example, to avoid erroneous signaling, Error Notification objects
containing methods which send a signal to another process
should not persist across system restarts. This is because the
receiving process and its process ID do not persist across system
restarts.
The creator of the Error Notification object is responsible for
removing the Error Notification object at the appropriate time. In
the event that the process terminates and fails to remove the
Error Notification object, the en_persistenceflg descriptor
ensures that obsolete Error Notification objects are removed
when the system is restarted.
The valid en_persistenceflg descriptor values are:
en_pid
en_rclass
en_resource
en_rtype
en_symptom
94
General Programming Concepts
0
non-persistent (removed at boot time)
1
persistent (persists through boot)
Specifies a process ID (PID) for use in identifying the Error
Notification object. Objects that have a PID specified should have
the en_persistenceflg descriptor set to 0.
Identifies the class of the failing resource. For the hardware error
class, the resource class is the device class. The resource error
class is not applicable for the software error class.
Identifies the name of the failing resource. For the hardware
error class, a resource name is the device name.
Identifies the type of the failing resource. For the hardware error
class, a resource type is the device type a resource is known by
in the devices object class.
Enables notification of an error accompanied by a symptom
string when set to TRUE.
en_type
Identifies the severity of error log entries to match. The valid
en_type descriptor values are:
INFO
Informational
PEND
Impending loss of availability
PERM
Permanent
PERF
Unacceptable performance degradation
TEMP
Temporary
UNKN Unknown
TRUE
Matches alertable errors.
FALSE Matches non-alertable errors.
0
Removes the Error Notification object at system restart.
non-zero
Retains the Error Notification object at system restart.
Security
Only processes running with the root user authority can add objects to the Error
Notification object class.
Examples
1. To create a notify method that mails a formatted error entry to root each time a
disk error of type PERM is logged, create a file called /tmp/en_sample.add
containing the following Error Notification object:
errnotify:
en_name = "sample"
en_persistenceflg = 0
en_class = "H"
en_type = "PERM"
en_rclass = "disk"
en_method = "errpt -a -l $1 | mail -s 'Disk Error' root"
To add the object to the Error Notification object class, enter:
odmadd /tmp/en_sample.add
The odmadd command adds the Error Notification object contained in
/tmp/en_sample.add to the errnotify file.
2. To verify that the Error Notification object was added to the object class, enter:
odmget -q"en_name='sample'" errnotify
The odmget command locates the Error Notification object within the
errnotify file that has an en_name value of ″sample″ and displays the object.
The following output is returned:
errnotify:
en_pid = 0
en_name = "sample"
en_persistenceflg = 0
en_label = ""
en_crcid = 0
en_class = "H"
en_type = "PERM"
en_alertflg = ""
Chapter 4. Error Notification
95
en_resource = ""
en_rtype = ""
en_rclass = "disk"
en_method = "errpt -a -l $1 | mail -s 'Disk Error' root"
3. To delete the sample Error Notification object from the Error Notification object
class, enter:
odmdelete -q"en_name='sample'" -o errnotify
The odmdelete command locates the Error Notification object within the
errnotify file that has an en_name value of ″sample″ and removes it from the
Error Notification object class.
Related Information
Error Logging Overview in AIX Version 4.3 Problem Solving Guide and Reference.
Error Logging Special Files in AIX Version 4.3 Files Reference.
The errdemon daemon in AIX Version 4.3 Commands Reference.
The errclear command, errdead command, errinstall command, errlogger
command, errmsg command, errpt command, errstop command, errupdate
command, odmadd command, odmdelete command, odmget command in AIX
Version 4.3 Commands Reference.
The errlog subroutine in AIX Version 4.3 Technical Reference: Base Operating System
and Extensions Volume 1.
The errsave kernel service in AIX Version 4.3 Technical Reference: Kernel and
Subsystems Volume 1.
96
General Programming Concepts
Chapter 5. File Systems and Directories
A file is a one-dimensional array of bytes that can contain ASCII or binary
information. In this operating system, files can contain data, shell scripts, and
programs. File names are also used to represent abstract objects such as sockets or
device drivers.
Internally, files are represented by index nodes (i-nodes). Within this file system, an
i-node is a 128-byte structure that contains all access, timestamp, ownership, and
data location information for each file. Pointers within the i-node structure
designate the real disk address of the data blocks associated with the file. An
i-node is identified by an offset number (i-number) and has no file name
information. The connection of i-numbers and file names is called a link.
File names exist only in directories. Directories are a unique type of file that give
hierarchical structure to the file system. Directories contain directory entries. Each
directory entry contains a file name and an i-number.
The journaled file system (JFS) is native to this operating system. The file system
links the file and directory data to the structure used by storage and retrieval
mechanisms. This chapter contains the following sections that further describe the
journaled file system programming model:
v File Types
v Directories
v Working with JFS i-nodes
v File Space Allocation
v Writing Programs Which Access Large Files
v Linking for Programmers
v Using File Descriptors
v File Creation and Removal
v Working with File I/O
v File Status
v File Accessibility
v File System Layout
v Creating New File System Types
File Types
A file is a one-dimensional array of bytes with at least one hard link (file name).
Files can contain ASCII or binary information. Files contain data, shell scripts, or
programs. File names are also used to represent abstract objects such as sockets,
pipes, and device drivers.
The kernel does not distinguish record boundaries in regular files. Programs can
establish their own boundary markers if desired. For example, many programs use
line-feed characters to mark the end of lines. “Working with Files” on page 99
contains a list of the subroutines used to control files.
© Copyright IBM Corp. 1997, 1999
97
Files are represented in the journaled file system (JFS) by disk index nodes
(i-node). Information about the file (such as ownership, access modes, access time,
data addresses, and modification time) is stored in the i-node. For more
information about the internal structure of files, see “Working with JFS i-nodes” on
page 103.
The journaled file system supports the following file types:
File Types Supported By Journaled File System
Type of File
Macro Name Used in
mode.h
Description
Regular
S_ISREG
A sequence of bytes with one
or more names. Regular files
can contain ASCII or binary
data. These files can be
randomly accessed (read
from or written to) from any
byte in the file.
Directory
S_ISDIR
Contains directory entries
(file name and i-number
pairs). Directory formats are
determined by the file
system. Processes read
directories as they do
ordinary files, but the kernel
reserves the right to write to
a directory. Special sets of
subroutines control directory
entries.
Block Special
S_ISBLK
Associates a structured
device driver with a file
name.
Character Special
S_ISCHR
Associates an unstructured
device driver with a file
name.
Pipes
S_ISFIFO
Designates an interprocess
communication channel
(IPC). The mkfifo subroutine
creates named pipes. The
pipe subroutine creates
unnamed pipes.
Symbolic Links
S_ISLNK
A file that contains either an
absolute or relative path
name to another file name.
Sockets
S_ISSOCK
An IPC mechanism that
allows applications to
exchange data. The socket
subroutine creates sockets,
and the bind subroutine
allows sockets to be named.
The maximum size of a regular file in a file system enabled for large files
(available beginning in Version 4.2) is slightly less than 64 gigabytes (68589453312).
All nonregular files in a file system enabled for large files and all files in other JFS
file system types have a maximum file size of 2 gigabytes minus 1 (2147483647).
98
General Programming Concepts
The maximum length of a file name is 255 characters, and the maximum length of
a path name is 1023 bytes. For more information, see “File Space Allocation” on
page 104.
Working with Files
The operating system offers many subroutines that manipulate files. Brief
descriptions of the most common file-control subroutines are provided in two
categories:
v “Creating Files”
v “Manipulating Files (Programming)”
Creating Files
creat
open
mkfifo
mkdir
mknod
socket
pipe
link
Creates
Creates
Creates
Creates
Creates
Creates
Creates
Creates
a new, empty, regular file
a new, empty file if the O_CREAT flag is set
a named pipe
a directory
a file that defines a device
a socket
an IPC
an additional name (directory entry) for an existing file
Manipulating Files (Programming)
open
read
write
lseek or llseek
close
rmdir
chown
chmod
stat
access
rename
truncate
ioctl
fclear
fsync
fcntl, dup, or dup2
lockf or flock
Returns a file descriptor used by other subroutines to
reference the opened file. The open operation takes a regular
file name and a permission mode that indicates whether the
file is to be read from, written to, or both.
Removes data from an open file if the appropriate
permissions (O_RDONLY or O_RDWR) were set by the open
subroutine.
Puts data into an open file if the appropriate permissions
(O_WRONLY or O_RDWR) were set by the open subroutine.
Move the I/O pointer position in an open file.
Closes open file descriptors (including sockets).
Removes directories from the file system.
Changes ownership of a file.
Changes the access modes of a file.
Reports the status of a file including the owner and access
modes.
Determines the accessibility of a file.
Changes the name of a file.
Changes the length of a file.
Controls functions associated with open file descriptors,
including special files, sockets, and generic device support
like the termio general terminal interface.
Creates space in file.
Writes changes in a file to permanent storage.
Control open file descriptors.
Control open file descriptors.
For more information on types and characteristics of file systems, see ″File Systems
Overview″ in AIX Version 4.3 System Management Guide: Operating System and
Devices.
Chapter 5. File Systems and Directories
99
Directories
Directories provide a hierarchical structure to the file system and link file and
subdirectory names to i-nodes. There is no limit on the depth of nested directories.
Disk space is allocated for directories in 4096-byte blocks, but the operating system
allocates directory space in 512-byte records.
Processes can read directories as regular files. However, the kernel reserves the
right to write directories. For this reason, directories are created and maintained by
a set of subroutines unique to them.
Directory Structures
Directories contain a sequence of directory entries. Each directory entry contains
three fixed-length fields (the index number associated with the file’s i-node, the
length of the file name, and the number of bytes for the entry) and one variable
length field for the file name. The file name field is null-terminated and padded to
4 bytes. File names can be up to 255 bytes long.
Directory entries are variable-length to allow file names the greatest flexibility.
However, all directory space is allocated at all times. See the JFS Directory Record
Space Allocation figure for an illustration.
100
General Programming Concepts
i-node number (4 bytes)
Size of the entry (2 bytes)
Length of entry name (2 bytes)
File name (rounded to 4 bytes)
f
o
o
f
o
o
f
o
o
f
o
o
f
o
o
0
An Empty, Well-Formed
Directory
mkdir (“foo”);
0
0
0
0
0
0
0
0
a
r
0
0
0
0
0
0
a
r
0
f
r
e
d
0
0
0
0
j
a
c
k
s
o
n
0
0
0
0
0
0
23
12
1
467
500
2
23
12
1
467
12
2
189
488
3
23
12
1
467
12
2
189
12
3
b
2098
16
4
0
b
creat (“foo/bar”, 0666);
creat (“foo/fred”, 0666);
creat (“foo/jackson”, 0666);
4887
460
7
23
12
1
467
24
2
189
12
3
b
a
r
0
2098
16
4
f
r
e
d
0
0
0
0
4887
460
7
j
a
c
k
s
o
n
0
23
12
1
467
40
2
189
12
3
b
2098
16
4
460
7
4887
0
remove (“foo/bar”);
0
0
0
0
a
r
0
f
r
e
d
0
0
0
0
j
a
c
k
s
o
n
0
remove (“foo/fred”);
JFS Directory Record Space Allocation
No directory entry is allowed to span 512-byte sections of a directory. When a
directory requires more than 512 bytes, another 512-byte record is appended to the
original record. If all of the 512-byte records in the allocated data block are filled,
an additional data block (4096 bytes) is allotted.
When a file is removed, the space the file occupied in the directory structure is
added to the preceding directory entry. The information about the directory
remains until a new entry fits into the space vacated.
Every well-formed directory contains the entries . (dot) and .. (dot, dot). The . (dot)
directory entry points to the i-node for the directory itself. The .. (dot, dot)
Chapter 5. File Systems and Directories
101
directory entry points to the i-node for the parent directory. The mkfs program
initializes a file system so that the . (dot) and .. (dot, dot) entries in the new root
directory point to the root i-node of the file system.
Access modes for directories have the following meanings:
read
write
execute
Allows a process to read directory entries
Allows a process to create new directory entries or remove old ones by using
the creat, mknod, link, and unlink subroutines
Allows a process to use the directory as a current working directory or to search
below the directory in the file tree
Working with Directories (Programming)
The mkdir and rmdir subroutines create and remove directories, respectively.
The opendir, readdir, telldir, seekdir, rewinddir and closedir subroutines
manipulate directories. The opendir subroutine returns a structure pointer that is
used by the readdir subroutine to obtain the next directory entry, by rewinddir to
reset the read position to the beginning, and by closedir to close the directory. The
seekdir subroutine returns to a position previously obtained with the telldir
subroutine. In earlier versions, programs treated directories as regular files and
used the open, read, lseek, and close subroutines to access them. This is no longer
recommended.
Changing Current Directory of a Process
When the system is booted, the first process uses the root directory of the root file
system as its current directory. New processes created with the fork subroutine
inherit the current directory used by the parent process. The chdir subroutine
changes the current directory of a process.
The chdir subroutine parses the path name to ensure that the target file is a
directory and that the process owner has permissions to the directory. After the
chdir subroutine is run, the process uses the new current directory to search all
path names that do not begin with a / (slash).
Changing the Root Directory of a Process
Processes can change their understanding of the root directory through the chroot
subroutine. Child processes of the calling process consider the directory indicated
by the chroot subroutine as the logical root directory of the file system.
Processes use the global file system root directory for all path names starting with
a / (slash).All path name searches beginning with a / (slash) begin at this new
root directory.
Subroutines That Control Directories
Due to the unique nature of directory files, directories are controlled by a special
set of subroutines. The following subroutines are designed to control directories:
chdir
Changes the current working directory
chroot
Changes the effective root directory
opendir, readdir, telldir, seekdir, rewinddir, or closedir
Perform various actions on directories
102
General Programming Concepts
getcwd or getwd
mkdir
rename
rmdir
Gets path to current directory
Creates a directory
Renames a directory
Removes a directory
Working with JFS i-nodes
Files in the journaled file system (JFS) are represented internally as index nodes
(i-nodes). Journaled file system i-nodes exist in a static form on disk and contain
access information for the file as well as pointers to the real disk addresses of the
file’s data blocks. The number of disk i-nodes available to a file system is
dependent on the size of the file system, the allocation group size (8 MB by
default), and the number of bytes per i-node ratio (4096 by default). These
parameters are given to the mkfs command at file system creation. When enough
files have been created to use all the available i-nodes, no more files can be
created, even if the file system has free space. The number of available i-nodes can
be determined by using the df -v command. Disk i-nodes are defined in the
/usr/include/jfs/ino.h file.
When a file is opened, an in-core i-node is created by the operating system. The
in-core i-node contains a copy of all the fields defined in the disk i-node, plus
additional fields for tracking the in-core i-node. In-core i-nodes are defined in the
/usr/include/jfs/inode.h file.
Disk i-node Structure
Each disk i-node in the journaled file system (JFS) is a 128-byte structure.
The offset of a particular i-node within the i-node list of the file system produces
the unique number (i-number) by which the operating system identifies the i-node.
A bit map, known as the i-node map, tracks the availability of free disk i-nodes for
the file system.
Disk i-nodes include the following information:
Field
i_mode
i_size
i_uid
i_gid
i_nblocks
i_mtime
i_atime
i_ctime
i_nlink
i_rdaddr[8]
i_rindirect
Contents
Type of file and access permission mode bits
Size of file in bytes
Access permissions for the user ID
Access permissions for the group ID
Number of blocks allocated to the file
Last time file was modified
Last time file was accessed
Last time i-node was modified
Number of hard links to the file
Real disk addresses of the data
Real disk address of the indirect block, if any
It is impossible to change the data of a file without changing the i-node, but it is
possible to change the i-node without changing the contents of the file. For
example, when permission is changed, the information within the i-node (i_ctime)
is modified, but the data in the file remains the same.
The i_rdaddr field within the disk i-node contains 8 disk addresses. These
addresses point to the first 8 data blocks assigned to the file. The i_rindirect field
Chapter 5. File Systems and Directories
103
address points to an indirect block. Indirect blocks are either single indirect or
double indirect. Thus, there are three possible geometries of block allocation for a
file: direct, indirect, or double indirect. Use of the indirect block and other file
space allocation geometries are discussed in the article ″File Space Allocation″.
Disk i-nodes do not contain file or path name information. Directory entries are
used to link file names to i-nodes. Any i-node can be linked to many file names by
creating additional directory entries with the link or symlink subroutine. To
discover the i-node number assigned to a file, use the ls -i command.
The i-nodes that represent files that define devices contain slightly different
information from i-nodes for regular files. Files associated with devices are called
special files. There are no data block addresses in special device files, but the major
and minor device numbers are included in the i_rdev field.
In normal situations, a disk i-node is released when the link count (i_nlink) to the
i-node equals 0. Links represent the file names associated with the i-node. When
the link count to the disk i-node is 0, all the data blocks associated with the i-node
are released to the bit map of free data blocks for the file system. The i-node is
then placed on the free i-node map.
In-core i-node Structure
When a file is opened, the information in the disk i-node is copied into an in-core
i-node for easier access. The in-core i-node structure contains additional fields
which manage access to the disk i-node’s valuable data. The fields of the in-core
i-node are defined in the inode.h file. Some of the additional information tracked
by the in-core i-node is:
v Status of the in-core i-node, including flags that indicate:
– An i-node lock
– A process waiting for the i-node to unlock
– Changes to the file’s i-node information
– Changes to the file’s data
v Logical device number of the file system that contains the file
v i-number used to identify the i-node
v Reference count. When the reference count field equals 0, the in-core i-node is
released.
When an in-core i-node is released (for instance with the close subroutine), the
in-core i-node reference count is reduced by 1. If this reduction results in the
reference count to the in-core i-node becoming 0, the i-node is released from the
in-core i-node table, and the contents of the in-core i-node are written to the disk
copy of the i-node (if the two versions differ).
File Space Allocation
File space allocation is the method by which data is apportioned physical storage
space in the operating system. The kernel allocates disk space to a file or directory
in the form of logical blocks. A Logical block refers to the division of a file or
directory’s contents into 4096-byte units. Logical blocks are not tangible entities;
however, the data in a logical block consumes physical storage space on the disk.
104
General Programming Concepts
Each file or directory consists of 0 or more logical blocks. Fragments, instead of
logical blocks, are the basic units for allocated disk space in the journaled file
system (JFS).
Full and Partial Logical Blocks
A file or directory may contain full or partial logical blocks. A full logical block
contains 4096 bytes of data. Partial logical blocks occur when the last logical block
of a file or directory contains less than 4096 bytes of data.
For example, a file of 8192 bytes is two logical blocks. The first 4096 bytes reside in
the first logical block and the following 4096 bytes reside in the second logical
block. Likewise, a file of 4608 bytes consists of two logical blocks. However, the
last logical block is a partial logical block containing the last 512 bytes of the file’s
data. Only the last logical block of a file can be a partial logical block.
Allocation in Fragmented File Systems
The default fragment size is 4096 bytes. You can specify smaller fragment sizes
with the mkfs command during a file system’s creation. Allowable fragment sizes
are: 512, 1024, 2048, and 4096 bytes. You can use only one fragment size in a file
system. See “File System Layout” on page 138 for more information on the file
system structure.
To maintain efficiency in file system operations, the JFS allocates 4096 bytes of
fragment space to files and directories that are 32KB or larger. A fragment that
covers 4096 bytes of disk space is allocated to a full logical block. When data is
added to a file or directory, the kernel allocates disk fragments to store the logical
blocks. Thus, if the file system’s fragment size is 512 bytes, a full logical block is
the allocation of 8 fragments.
The kernel allocates disk space so that only the last bytes of data receive a partial
block allocation. As the partial block grows beyond the limits of its current
allocation, additional fragments are allocated. If the partial block increases to 4096
bytes, the data stored in its fragments are reallocated into 4096-byte allocations. A
partial logical block that contains less than 4096 bytes of data is allocated the
number of fragments that best matches its storage requirements.
Fragment reallocation also occurs if data is added to logical blocks that represent
file holes. A file hole is an ″empty″ logical block located prior to the last logical block
that stores data. (File holes do not occur within directories.) These empty logical
blocks are not allocated fragments. However, as data is added to file holes,
allocation occurs. Each logical block that was not previously allocated disk space is
allocated 4096 byte of fragment space.
Additional fragment allocation is not required if existing data in the middle of a
file or directory is overwritten. The logical block containing the existing data has
already been allocated fragments.
JFS tries to maintain contiguous allocation of a file or directory’s logical blocks on
the disk. Maintaining contiguous allocation lessens seek time because the data for a
file or directory can be accessed sequentially and found on the same area of the
disk. However, disk fragments for one logical block are not always contiguous to
the disk fragments for another logical block. The disk space required for
Chapter 5. File Systems and Directories
105
contiguous allocation may not be available if it has already been written to by
another file or directory. An allocation for a single logical block, however, always
contains contiguous fragments.
The file system uses a bitmap called the fragment allocation map to record the status
of every fragment in the file system. When the file system needs to allocate a new
fragment, it refers to the fragment allocation map to identify which fragments are
available. A fragment can only be allocated to a single file or directory at a time.
Allocation in Compressed File Systems
In a file system that supports data compression, directories are allocated disk
space. Data compression also applies to regular files and symbolic links whose size
is larger than that of their i-nodes.
The allocation of disk space for compressed file systems is the same as that of
fragments in fragmented file systems. A logical block is allocated 4096 bytes when
it is modified. This allocation guarantees that there will be a place to store the
logical block if the data does not compress. The system requires that a write or
store operation report an out-of-disk-space condition into a memory-mapped file at
a logical block’s initial modification. After modification is complete, the logical
block is compressed before it is written to a disk. The compressed logical block is
then allocated only the number of fragments required for its storage.
In a fragmented file system, only the last logical block of a file (not larger than
32KB) can be allocated less than 4096 bytes. The logical block becomes a partial
logical block. In a compressed file system, every logical block can be allocated less
than a full block.
A logical block is no longer considered modified after it is written to a disk. Each
time a logical block is modified, a full disk block is allocated again, according to
the system requirements. Reallocation of the initial full block occurs when the
logical block of compressed data is successfully written to a disk.
Allocation in File Systems Enabled for Large Files
Beginning in Version 4.2, in a file system enabled for large files, the JFS allocates
two sizes of fragments for regular files. A ″large″ fragment (32 X 4096) is allocated
for logical blocks after the 4 MB boundary, and a 4096-byte fragment is allocated
for logical blocks before the 4 MB boundary. All nonregular files allocate 4096-byte
fragments. This geometry allows a maximum file size of slightly less than 64
gigabytes (68589453312).
A ″large″ fragment is made up of 32 contiguous 4096-byte fragments. Because of
this requirement, it is recommended that file systems enabled for large files have
predominantly large files in them. Storing many small files (files less than 4 MB)
can cause free-space fragmentation problems. This can cause large allocations to
fail with ENOSPC because the file system does not contain 32 contiguous disk
addresses.
Disk Address Format
JFS fragment support requires fragment-level addressability. As a result, disk
addresses have a special format for mapping where the fragments of a logical
block reside on the disk. Fragmented and compressed file systems use the same
106
General Programming Concepts
method for representing disk addresses. Disk addresses are contained in the
i_rdaddr field of the i-nodes or in the indirect blocks. All fragments referenced in a
single address must be contiguous on the disk.
The disk address format consists of two fields, the nfrags and addr fields. These
fields describe the area of disk covered by the address.
addr
nfrags
Indicates which fragment on the disk is the starting fragment
Indicates the total number of contiguous fragments not used by the address
For example, if the fragment size for the file system is 512 bytes and the logical
block is divided into eight fragments, the nfrags value is 3, indicating that five
fragments are included in the address.
The following examples illustrate possible values for the addr and nfrags fields for
different disk addresses. These values assume a fragment size of 512 bytes,
indicating that the logical block is divided into eight fragments.
Address for a single fragment:
addr:
143
nfrags: 7
This address indicates that the starting location of the data is fragment 143 on the
disk. The nfrags value indicates that the total number of fragments included in the
address is one. The nfrags value changes in a file system that has a fragment size
other than 512 bytes. To correctly read the nfrags value, the system, or any user
examining the address, must know the fragment size of the file system.
Address for five fragments:
addr:
1117
nfrags: 3
In this case, the address starts at fragment number 1117 on the disk and continues
for five fragments (including the starting fragment). There are three fragments
remaining, as illustrated by the nfrags value.
The disk addresses are 32 bits in size. The bits are numbered from 0 to 31. The 0
bit is always reserved. Bits 1 through 3 contain the nfrags field. Bits 4 through 31
contain the addr field.
The Disk Address Format figure illustrates the mapping of a file’s data to the
fragments contained on the disk.
Chapter 5. File Systems and Directories
107
I-Node with Two Addresses
117
125
133
118
126
134
0
119
127
nfrags
addr
5
125
4
1
Block’s
Address
31
135
120
128
136
121
129
137
122
130
138
123
131
139
0
1
nfrags
addr
0
117
4
31
A File with One Full
Logical Block and a
Partial Logical Block
0
124
132
...
1
Notes:
Fragment Size
= 512 Bytes
addr Value = Starting
Fragment
Disk with Numbered
Fragments
nfrags Value = Number of
Fragments Less than a Full
Block
Disk Address Format
Indirect Blocks
The JFS uses the indirect blocks to address the disk space allocated to larger files.
Indirect blocks allow the greatest flexibility for file sizes and the fastest retrieval
time. The indirect block is assigned using the i_rindirect field of the disk i-node.
This field allows for three geometries or methods for addressing the disk space:
v Direct
v Single indirect
v Double indirect
Each of these methods uses the same disk address format as compressed and
fragmented file systems. Because files larger than 32KB are allocated fragments of
4096 bytes, the nfrags field for addresses using the single indirect or double
indirect method has a value of 0.
Direct Method
When the direct method of disk addressing is used, each of the eight addresses
listed in the i_rdaddr field of the disk i-node points directly to a single allocation
of disk fragments. The maximum size of a file using direct geometry is 32,768
bytes (32KB), or 8 x 4096 bytes. When the file requires more than 32KB, an indirect
108
General Programming Concepts
block is used to address the file’s disk space. The direct disk address method is
illustrated in the Geometry One figure:
Fragment
Allocations
I-Node
Address 0
Address 1
Address 2
i_rdaddr
Address 3
Address 4
Address 5
Address 6
Address 7
i_rindirect
Unused
Geometry One: Direct (8x4KB) = 32KB)
Single Indirect Method
The i_rindirect field contains an address that points to either a single indirect
block or a double indirect block. When the single indirect disk addressing method
is used, the i_rindirect field contains the address of an indirect block containing
1024 addresses. These addresses point to the disk fragments for each allocation.
Using the single indirect block geometry, the file can be up to 4,194,304 bytes
(4MB), or 1024 x 4096 bytes. The indirect disk address method is illustrate in the
Geometry Two figure:
Fragment
Allocations
I-Node
Unused
Unused
Unused
i_rdaddr
Unused
Single Indirect
Block
Unused
Unused
1024
Addresses
Unused
Unused
i_rindirect
Address
Geometry Two: Indirect (1024x4KB) = 4MB)
Double Indirect Method
The double indirect addressing method uses the i_rindirect field to point to a
double indirect block. The double indirect block contains 512 addresses that point
to indirect blocks, which contain pointers to the fragment allocations. The largest
file size that can be used with the double indirect geometry in a file system not
enabled for large files is 2,147,483,648 bytes (2GB), or 512(1024 x 4096) bytes. The
double indirect disk address method is illustrated in the Geometry Three figure:
Chapter 5. File Systems and Directories
109
Fragment
Allocations
I-Node
Unused
Unused
Unused
Unused
i_rdaddr
Unused
Unused
Unused
Double Indirect
Block
Single Indirect
Blocks
512
Addresses
Unused
Address
i_rindirect
Geometry Three: Double Indirect (512(1024x4KB) = 4GB)
Note: The maximum file size that the read and write system calls would
allow is 2GB minus 1 (231-1 ). When memory map interface is used, 2GB can
be addresed.
Beginning in Version 4.2, file systems enabled for large files allow a maximum file
size of slightly less than 64 gigabytes (68589453312). The first single indirect block
contains 4096 byte fragments, and all subsequent single indirect blocks contain (32
X 4096) byte fragments. The following produces the maximum file size for file
systems enabling large files:
(1 * (1024 * 4096)) + (511 * (1024 * 131072))
The fragment allocation assigned to a directory is divided into records of 512 bytes
each and grows in accordance with the allocation of these records.
Quotas
Disk quotas restrict the amount of file system space any single user or group can
monopolize.
quotactl
110
General Programming Concepts
Subroutine that sets limits on both the number of files and the number of disk
blocks allocated to each user or group on a file system. Quotas enforce two
kinds of limits:
hard
Maximum limit allowed. When a process hits its hard limit, requests
for more space fail.
soft
Practical limit. If a process hits the soft limit, a warning is printed to
the user’s terminal. The warning is often displayed at login. If the
user fails to correct the problem after several login sessions, the soft
limit can become a hard limit.
System warnings are designed to encourage users to heed the soft limit. However,
the quota system allows processes access to the higher hard limit when more
resources are temporarily required.
Writing Programs That Access Large Files
Beginning in Version 4.2, the operating system allows files that are larger than 2
gigabytes (2GB). This article is intended to assist programmers in understanding
the implications of ″large″ files on their applications and to assist them in
modifying their applications. A new set of programming interfaces is defined, so
that application programs can be modified to be aware of large files.
The file system programming interfaces generally revolve around the off_t data
type. In Version 4.1, the off_t data type was defined as a signed 32-bit integer. As a
result, the maximum file size that these interfaces would allow was 2 gigabytes
minus 1.
Implications for Existing Programs
The 32-bit application environment that all applications used in prior releases
remains unchanged. Existing application programs will execute exactly as they did
before. However, existing application programs will not be able to deal with large
files.
For example, the st_size field in the stat structure, which is used to return file
sizes, is a signed, 32-bit long. Therefore, that stat structure cannot be used to return
file sizes that are larger than LONG_MAX. If an application attempts to stat a file
that is larger than LONG_MAX, the stat subroutine will fail, and errno will be set
to EOVERFLOW, indicating that the file size overflows the size field of the
structure being used by the program.
This behavior is significant because existing programs that might not appear to
have any impacts as a result of large files will experience failures in the presence of
large files even though they may not even be interested in the file size.
The errno EOVERFLOW can also be returned by lseek and by fcntl if the values
that need to be returned are larger than the data type or structure that the program
is using. For lseek, if the resulting offset is larger than LONG_MAX, lseek will fail
and errno will be set to EOVERFLOW. For fcntl, if the caller uses F_GETLK and
the blocking lock’s starting offset or length is larger than LONG_MAX, the fcntl
call will fail, and errno will be set to EOVERFLOW.
Open Protection
Many of the existing application programs were written under the assumption that
a file size could never be larger than could be represented in a signed, 32-bit long.
These programs could have unexpected behavior, including data corruption, if
allowed to operate on large files. Beginning in Version 4.2, the operating system
implements an open-protection scheme to protect applications from this class of
failure.
When an application that has not been enabled for large-file access attempts to
open a file that is larger than LONG_MAX, the open subroutine will fail and errno
will be set to EOVERFLOW. Application programs that have not been enabled will
be unable to access a large file, and the possibility of inadvertent data corruption is
Chapter 5. File Systems and Directories
111
avoided. Applications that need to be able to open large files must be ported to the
large-file environment described in “Porting Applications to the Large File
Environment”.
In addition to open protection, a number of other subroutines offer protection by
providing an execution environment, which is identical to the environment under
which these programs were developed. If an application uses the write family of
subroutines and the write request crosses the 2 gigabyte boundary, the write
subroutines will transfer data only up to 2 gigabytes minus 1. If the application
attempts to write at or beyond the 2Gb-1 boundary, the write subroutines will fail
and set errno to EFBIG. The behavior of mmap, ftruncate, and fclear are similar.
The read family of subroutines also participates in the open protection scheme. If
an application attempts to read a file across the 2 gigabyte threshold, only the data
up to 2 gigabytes minus 1 will be read. Reads at or beyond the 2Gb-1 boundary
will fail, and errno will be set to EOVERFLOW.
Open protection is implemented by a flag associated with an open file description.
The current state of the flag can be queried with the fcntl subroutine using the
F_GETFL command. The flag can be modified with the fcntl subroutine using the
F_SETFL command.
Since open file descriptions are inherited across the exec family of subroutines,
application programs that pass file descriptors that are enabled for large-file access
to other programs should consider whether the receiving program can safely access
the large file.
Porting Applications to the Large File Environment
Beginning in Version 4.2, the operating system provides two different ways for
applications to be enabled for large-file access. Application programmers must
decide which approach best suits their needs. The first approach is to define
_LARGE_FILES, which carefully redefines all of the relevant data types, structures,
and subroutine names to their large-file enabled counterparts. The second
approach is to recode the application to call the large-file enabled subroutines
explicitly.
Defining _LARGE_FILES has the advantage of maximizing application portability
to other platforms since the application is still written to the normal POSIX and
XPG interfaces. It has the disadvantage of creating some ambiguity in the code
since the size of the various data items is not obvious from looking at the code.
Recoding the application has the obvious disadvantages of requiring more effort
and reducing application portability. It can be used when the redefinition effect of
_LARGE_FILES would have a considerable negative impact on the program or
when it is desirable to convert only a very small portion of the program.
It is very important to understand that in either case, the application program
MUST be carefully audited to ensure correct behavior in the new environment.
Some of the common programming pitfalls are discussed in “Common Pitfalls
using the Large File Environment” on page 115.
112
General Programming Concepts
Using _LARGE_FILES
In the default compilation environment, the off_t data type is defined as a signed,
32-bit long. Beginning in Version 4.2, if the application defines _LARGE_FILES
before the inclusion of any header files, then the large-file programming
environment is enabled. and off_t is defined to be a signed, 64-bit long long. In
addition, all of the subroutines that deal with file sizes or file offsets are redefined
to be their large-file enabled counterparts. Similarly, all of the data structures with
embedded file sizes or offsets are redefined.
Assuming that the application is coded without any dependencies on off_t being a
32-bit quantity, the resulting binary should work properly in the new environment.
In practice, application programs rarely require a porting effort this small.
The following table shows the redefinitions that occur in the _LARGE_FILES
environment beginning in Version 4.2.
Item
Redefined To Be
Header File
off_t
long long
<sys/types.h>
fpos_t
long long
<sys/types.h>
struct stat
struct stat64
<sys/stat.h>
stat()
stat64()
<sys/stat.h>
fstat()
fstat64()
<sys/stat.h>
lstat()
lstat64()
<sys/stat.h>
mmap()
mmap64()
<sys/mman.h>
lockf()
lockf64()
<sys/lockf.h>
struct flock
struct flock64
<sys/flock.h>
open()
open64()
<fcntl.h>
creat()
creat64()
<fcntl.h>
F_GETLK
F_GETLK64
<fcntl.h>
F_SETLK
F_SETLK64
<fcntl.h>
F_SETLKW
F_SETLKW64
<fcntl.h>
ftw()
ftw64()
<ftw.h>
nftw()
nftw64()
<ftw.h>
fseeko()
fseeko64()
<stdio.h>
ftello()
ftello64()
<stdio.h>
fgetpos()
fgetpos64()
<stdio.h>
fsetpos()
fsetpos64()
<stdio.h>
fopen()
fopen64()
<stdio.h>
freopen()
freopen64()
<stdio.h>
lseek()
lseek64()
<unistd.h>
ftruncate()
ftruncate64()
<unistd.h>
truncate()
truncate64()
<unistd.h>
fclear()
fclear64()
<unistd.h>
struct aiocb
struct aiocb64
<sys/aio.h>
aio_read()
aio_read64()
<sys/aio.h>
Chapter 5. File Systems and Directories
113
aio_write()
aio_write64()
<sys/aio.h>
aio_cancel()
aio_cancel64()
<sys/aio.h>
aio_suspend
aio_suspend64()
<sys/aio.h>
aio_listio()
aio_listio64()
<sys/aio.h>
aio_return()
aio_return64()
<sys/aio.h>
aio_error
aio_error64()
<sys/aio.h>
Using the 64-Bit File System Subroutines
Using the _LARGE_FILES environment may be impractical for some applications
due to the far-reaching implications of changing the size of off_t to 64 bits. If the
number of changes is small, it may be more practical to convert a relatively small
part of the application to be large-file enabled. The 64-bit file system data types,
structures, and subroutines are listed below:
<sys/types.h>
typedef long long off64_t;
typedef long long fpos64_t;
<fcntl.h>
extern int
open64(const char *, int, ...);
extern int
creat64(const char *, mode_t);
#define F_GETLK64
#define F_SETLK64
#define F_SETLKW64
<ftw.h>
extern int ftw64(const char *, int (*)(const char *,const struct stat64 *, int), int);
extern int nftw64(const char *, int (*)(const char *, const struct stat64 *,
int,struct FTW *),int, int);
<stdio.h>
extern int
fgetpos64(FILE *, fpos64_t *);
extern FILE
*fopen64(const char *, const char *);
extern FILE
*freopen64(const char *, const char *, FILE *);
extern int
fseeko64(FILE *, off64_t, int);
extern int
fsetpos64(FILE *, fpos64_t *);
extern off64_t ftello64(FILE *);
<unistd.h>
extern off64_t lseek64(int, off64_t, int);
extern int
ftruncate64(int, off64_t);
extern int
truncate64(const char *, off64_t);
extern off64_t fclear64(int, off64_t);
<sys/flock.h>
struct flock64;
<sys/lockf.h>
extern int lockf64 (int, int, off64_t);
<sys/mman.h>
extern void
*mmap64(void *, size_t, int, int, int, off64_t);
<sys/stat.h>
struct stat64;
extern int
stat64(const char *, struct stat64 *);
extern int
fstat64(int, struct stat64 *);
extern int
lstat64(const char *, struct stat64 *);
<sys/aio.h>
struct aiocb64
int
aio_read64(int, struct aiocb64 *):
int
aio_write64(int, struct aiocb64 *);
int
aio_listio64(int, struct aiocb64 *[],
int, struct
sigevent *);
int
aio_cancel64(int, struct aiocb64 *);
int
aio_suspend64(int, struct aiocb64 *[]);
114
General Programming Concepts
Common Pitfalls using the Large File Environment
Porting of application programs to the large-file environment can expose a number
of different problems in the application. These problems are frequently the result of
poor coding practices, which are harmless in a 32-bit off_t environment, but which
can manifest themselves when compiled in a 64-bit off_t environment. The
information below illustrates some of the more common problems and solutions.
Note: In the examples below, off_t is assumed to be a 64-bit file offset.
Improper Use of Data Types
The most obvious source of problems with application programs is a failure to use
the proper data types. If an application attempts to store file sizes or file offsets in
an integer variable, the resulting value will be truncated and lose significance. The
proper technique for avoiding this problem is to use the off_t data type to store file
sizes and offsets.
Incorrect:
int file_size;
struct stat s;
file_size = s.st_size;
Better:
off_t file_size;
struct stat s;
file_size = s.st_size;
Parameter Mismatches
Care must be taken when passing 64-bit integers to functions as arguments or
when returning 64-bit integers from functions. Both the caller and the called
function must agree on the types of the arguments and the return value in order to
get correct results.
Passing a 32-bit integer to a function that expects a 64-bit integer causes the called
function to misinterpret the caller’s arguments, leading to unexpected behavior.
This type of problem is especially severe if the program passes scalar values to a
function that expects to receive a 64-bit integer.
Many of the problems can be avoided by careful use of function prototypes as
illustrated below. In the code fragments below, fexample() is a function that takes a
64-bit file offset as a parameter. In the first example, the compiler generates the
normal 32-bit integer function linkage, which would be incorrect since the
receiving function expects 64-bit integer linkage. In the second example, the LL
specifier is added, forcing the compiler to use the proper linkage. In the last
example, the function prototype causes the compiler to promote the scalar value to
a 64-bit integer. This is the preferred approach since the source code remains
portable between 32- and 64-bit environments.
Incorrect:
fexample(0);
Better:
fexample(0LL);
Chapter 5. File Systems and Directories
115
Best:
void fexample(off_t);
fexample(0);
Arithmetic Overflows
Even when an application uses the correct data types, it is still vulnerable to
failures due to arithmetic overflows. This problem usually occurs when the
application performs an arithmetic overflow before it is promoted to the 64-bit data
type. In the following example, blkno is a 32-bit block number. Multiplying the
block number by the block size occurs before the promotion, and overflow will
occur if the block number is sufficiently large. This problem is especially
destructive because the code is using the proper data types and the code works
properly for small values, but fails for large values. The problem can be fixed by
typecasting the values before the arithmetic operation.
Incorrect:
int blkno;
off_t offset;
offset = blkno * BLKSIZE;
Better:
int blkno;
off_t offset;
offset = (off_t) blkno * BLKSIZE;
This problem can also appear when passing values based on fixed constants to
functions that expect 64-bit parameters. In the example below, LONG_MAX+1
results in a negative number, which is sign-extended when it is passed to the
function.
Incorrect:
void fexample(off_t);
fexample(LONG_MAX+1);
116
General Programming Concepts
Better:
void fexample(off_t);
fexample((off_t)LONG_MAX+1);
Fseek/Ftell
The data type used by fseek and ftell subroutines is long and cannot be redefined
to the appropriate 64-bit data type in the _LARGE_FILES environment. Application
programs that access large files and that use fseek and ftell need to be converted.
This can be done in a number of ways. The fseeko and ftello subroutines are
functionally equivalent to fseek and ftell except that the offset is given as an off_t
instead of a long. Make sure to convert all variables that can be used to store
offsets to the appropriate type.
Incorrect:
long cur_offset, new_offset;
cur_offset = ftell(fp);
fseek(fp, new_offset, SEEK_SET);
Better:
off_t cur_offset, new_offset;
cur_offset = ftello(fp);
fseeko(fp, new_offset, SEEK_SET);
Failure to Include Proper Header Files
In order for application programs to see the function and data type redefinitions,
they must include the proper header files. This has the additional benefit of
exposing the function prototypes for various subroutines, which enables stronger
type-checking in the compiler.
Many application programs that call the open and creat subroutines do not include
<fcntl.h>, which contains the defines for the various open modes. These programs
typically hard code the open modes. This will cause runtime failures when the
program is compiled in the _LARGE_FILES environment because the program
does call the proper open subroutine, and the resulting file descriptor is not
enabled for large-file access. Programs must make sure to include the proper
header files, especially in the _LARGE_FILES environment, to get visibility to the
redefinitions of the environment.
Incorrect:
fd = open("afile",2);
Better:
#include <fcntl.h>
fd = open("afile",O_RDWR);
String Conversions
Converting file sizes and offsets to and from strings can cause problems when
porting applications to the large-file environment. The printf format string for a
64-bit integer is different than for a 32 bit integer. Programs that do these
conversions must be careful to use the proper format specifier. This is especially
difficult when the application needs to be portable between 32- and 64-bit
environments since there is no portable format specifier between the two
Chapter 5. File Systems and Directories
117
environments. One way to deal with this problem is to write offset converters that
use the proper format for the size of off_t.
off_t
atooff(const char *s)
{
off_t o;
if (sizeof(off_t) == 4)
sscanf(s,"%d",&o);
else if (sizeof(off_t) == 8)
sscanf(s,"%lld",&o);
else
error();
return o;
}
main(int argc, char **argv)
{
off_t offset;
offset = atooff(argv[1]);
fexample(offset);
}
Imbedded File Offsets
Application programs that imbed file offsets or sizes in data structures may be
affected by the change to the size of the off_t in the large-file environment. This
problem can be especially severe if the data structure is shared between various
applications or if the data structure is written into a file. In cases like this, the
programmer must decide if it should continue to contain a 32-bit offset or if it
should be converted to contain a 64-bit offset. If the application program needs to
have a 32-bit file offset even if off_t is 64 bits, the program may use the new data
type soff_t, a short off_t. This data type remains 32 bits even in the large-file
environment. If the data structure is converted to a 64-bit offset, then all of the
programs that deal with that structure must be converted to understand the new
data structure format.
File Size Limits
Application programs that are converted to be aware of large files may fail in their
attempts to create large files due to the file-size resource limit. The file-size
resource limit is a signed, 32-bit value which limits maximum file offset to which a
process can write to a regular file. Programs that need to write large files must
have their file size limit set to RLIM_INFINITY.
struct rlimit r;
r.rlim_cur = r.rlim_max = RLIM_INFINITY;
setrlimit(RLIMIT_FSIZE,&r);
This limit may also be set from the Korn shell by issuing the command:
ulimit -f unlimited
To set this value permanently for a specific user, use the chuser command:
Example: chuser fsize_hard = -1 root
JFS File Size Limits
The maximum size of a file is ultimately a characteristic of the file system itself,
not just the file size limit or the environment. For the JFS, the maximum file size is
determined by the parameters used at the time the file system was made. For JFS
file systems that are enabled for large files, the maximum file size is slightly less
than 64 gigabytes (0xff8400000). For all other JFS file systems, the maximum file
118
General Programming Concepts
size is 2Gb-1 (0x7fffffff). Attempts to write a file past the maximum file size in any
file system format will fail, and errno will be set to EFBIG.
Linking for Programmers
A link is a connection between a file name and an i-node (hard link) or between
file names (symbolic link). Linking allows access to an i-node from multiple file
names. (See “Working with JFS i-nodes” on page 103.) Directory entries pair file
names with i-nodes. File names are easy for users to identify, and i-nodes contain
the real disk addresses of the file’s data. A reference count of all links into an
i-node is maintained in the i_nlink field of the i-node. Subroutines that create and
destroy links use file names, not file descriptors, (see “Using File Descriptors” on
page 121). Therefore, it is not necessary to open files when creating a link.
Processes can access and change the contents of the i-node by any of the linked file
names. Two kinds of links exist in this operating system, hard links and symbolic
links.
Hard Links
link
unlink
Subroutine that creates hard links.The presence of a hard link guarantees the
existence of a file because a hard link increments the link count in the i_nlink
field of the i-node.
Subroutine that releases links. When all hard links to an i-node are released, the
file is no longer accessible.
The user ID that created the original file owns the file and retains access mode
authority over the file. Otherwise, all hard links are treated equally by the
operating system. Hard links must link file names and i-nodes within the same file
system since the i-node number is relative to a single file system.
Hard links always refer to a specific file because the directory entry created by the
hard link pairs the new file name to an i-node.
Example: If the /u/tom/bob file is linked to the /u/jack/foo file, the link count in
the i_nlink field of the foo file is 2. Both hard links are equal. If /u/jack/foo is
removed, the file continues to exist by the name /u/tom/bob and can be accessed
by users with access to the tom directory. However, the owner of the file is jack
even though /u/jack/foo was removed. The space occupied by the file is charged
to jack’s quota account. Change file ownership using the chown subroutine.
Symbolic Links
symlink
Subroutine that creates symbolic links
Symbolic links are implemented as a file that contains a path name. When a
process encounters a symbolic link, the path contained in the symbolic link is
prepended to the path the process was searching. If the path name in the symbolic
link is an absolute path name, the process searches from the root directory for the
named file. If the path name in the symbolic link does not begin with a / (slash),
the process interprets the rest of the path relative to the position of the symbolic
link. The unlink subroutine also removes symbolic links.
Symbolic links can traverse file systems because they are treated as regular files by
the operating system rather than as part of the file system structure. The presence
of a symbolic link does not guarantee the existence of the target file because a
Chapter 5. File Systems and Directories
119
symbolic link has no effect on the i_nlink field of the i-node.
Subroutine that reads the contents of a symbolic link. Many subroutines
(including the open and stat subroutines) follow symbolic paths.
Subroutine created to report on the status of the file containing the symbolic
link and does not follow the link. See the symlink subroutine for a list of
subroutines that traverse symbolic links.
readlink
lstat
Symbolic links are also called soft links because they link to a file by path name. If
the target file is renamed or removed, the symbolic link cannot resolve.
Example: The symbolic link to /u/joe/foo is a file that contains the literal data
/u/joe/foo. When the owner of the foo file removes this file, subroutine calls
made to the symbolic link cannot succeed. If the file owner then creates a new file
named foo in the same directory, the symbolic link leads to the new file. Therefore,
the link is considered soft because it is linked to interchangeable i-nodes.
In the ls -l command listing, an l in the first position indicates a linked file. In the
final column of that listing, the links between files are represented as Path2 ->
Path1 (or Newname -> Oldname).
Subroutine that removes a directory entry. The Path parameter in the subroutine
identifies the file to be disconnected. At the completion of the unlink call, the
link count of the i-node is reduced by the value of 1.
Subroutine that also removes a file name by calling either the unlink or rmdir
subroutine.
unlink
remove
Directory Links
mkdir
Subroutine that creates directory entries for new directories, which creates hard
links to the i-node representing the directory
Symbolic links are recommended for creating additional links to a directory.
Symbolic links do not interfere with the . and .. directory entries and will maintain
the empty, well-formed directory status. See the Understanding Directory Links
figure, for a graphic example of the empty, well-formed directory /u/joe/foo.
120
General Programming Concepts
/u
68
j
e
0
0
0
i = 68
0
0
i_nlink
3
o
o
0
0
0
0
0
0
o
/u/joe
mkdir (”foo”, 0666)
0
68
235
f
/u/joe/foo
235
68
i = 235
i_nlink
2
Understanding Directory Links
rmdir or remove
Remove links to directories
Using File Descriptors
A file descriptor is an unsigned integer used by a process to identify an open file.
Two thousand file descriptors are available to each process. The open, pipe, creat,
and fcntl subroutines all generate file descriptors. File descriptors are generally
unique to each process, but they can be shared by child processes created with a
fork subroutine or copied by the fcntl, dup, and dup2 subroutines.
File descriptors are indexes to the file descriptor table in the u_block area
maintained by the kernel for each process. The most common ways for processes
to obtain file descriptors are through open or creat operations or through
inheritance from a parent process. When a fork operation occurs, the descriptor
table is copied for the child process, which allows the child process equal access to
the files used by the parent process.
System File and File Descriptor Tables
The system file and file descriptor data structures track each process’ access to a
file and ensure data integrity.
Chapter 5. File Systems and Directories
121
Structure
file descriptor table
Activity and Contents
Translates an index number (file descriptor) in the table
to an open file. File descriptor tables are created for each
process and are located in the u_block area set aside for
that process. Each of the entries in a file descriptor table
has two fields: the flags area and the file pointer. The
structure of the file descriptor table is:
struct ufd
{
struct file *fp;
int flags;
}u_ufd[OPEN_MAX]
system open file table
The close-on-exec (FD_CLOEXEC bit) flag can be set in
the file descriptor table using the fcntl subroutine. The
dup subroutine copies one file descriptor entry into
another position in the same table. The fork subroutine
creates an identical copy of the entire file descriptor table
for a child process.
Contains entries for each open file. Two of the most
important pieces of information tracked in a file table
entry are the current offset referenced by all read or write
operations to the file and the open mode (O_RDONLY,
O_WRONLY, or O_RDWR) of the file.
The open file data structure contains the current I/O
offset for the file. The system treats each read/write
operation as an implied seek to the current offset. Thus if
x bytes are read or written, the pointer advances x bytes.
The lseek subroutine can be used to reassign the current
offset to a specified location in files that are randomly
accessible. Stream-type files (such as pipes and sockets)
do not use the offset because the data in the file is not
randomly accessible.
The File Descriptor and System File Table Relationship
figure illustrates the anatomy of and interaction between
the process’ file descriptor table and the system file table.
File descriptors are recycled in ascending order.
Per–Process
u_block Area
User’s File
Descriptor Table
Flags
0
1
2
.
.
.
File
Pointer
System File Table
Current I/O Offset
Open Mode
Ref Count
Current I/O Offset
Open Mode
Ref Count
1999
The File Descriptor and System File Table Relationship
Managing File Descriptors
Because files can be shared by many users, it is necessary to allow related
processes to share a common offset pointer and have a separate current offset
122
General Programming Concepts
pointer for independent processes that access the same file. The open file table
entry maintains a reference count to track the number of file descriptors assigned
to the file.
Multiple references to a single file can be caused by:
v A separate process opening the file
v Child processes retaining the file descriptors assigned to the parent process
v The fcntl or dup subroutine creating copies of the file descriptors
Sharing Open Files
The File Sharing figure demonstrates what happens in the file descriptor and
system file tables when two processes open the same file. Each open operation
creates a system table entry. Individual table entries ensure each process a separate
current I/O offsets. Independent offsets protect the integrity of the data.
Per-Process
u_block Area
System File Table
User’s File
Descriptor Table
File
Flags
Pointer
File 954
Offset = 67
Read only
Ref Count = 1
0
1
2
.
.
.
File 63
Offset = 4390
Write
Ref Count = 1
1999
vnode
User’s File
Descriptor Table
File
Flags
Pointer
File 63
Offset = 76450
Read/Write
Ref Count = 1
0
1
2
.
.
.
1999
File Sharing
When a file descriptor is duplicated, two processes then share the same offset and
interleaving can occur. Interleaving means that bytes are not read or written
sequentially.
Duplicating File Descriptors
There are three ways file descriptors can be duplicated between processes: the dup
or dup2 subroutine, the fork subroutine, and the fcntl (file descriptor control)
subroutine.
Chapter 5. File Systems and Directories
123
The dup and dup2 Subroutines:
dup Creates a copy of a file descriptor
The duplicate is created at an empty space in the user file descriptor table that
contains the original descriptor. A dup process increments the reference count in
the file table entry by 1 and returns the index number of the file-descriptor where
the copy was placed. See the Duplicate File Descriptors Created by the dup
Subroutine figure.
Per-Process
u_block area
System File Table
Descriptor Table
Flags
File
Pointer
0
1
2
.
.
.
File 63
Offset = 76450
Read/Write
Ref Count = 2
1999
Duplicate File Descriptors Created by the dup Subroutine
dup2
Scans for the requested descriptor assignment and closes the requested file
descriptor if it is open
The dup2 subroutine allows the process to designate which descriptor entry the
copy will occupy, if a specific descriptor-table entry is required.
The fork Subroutine:
fork
124
Creates a child process that inherits the file descriptors assigned to the parent
process. See the Duplicate File Descriptors as a Result of a Fork Operation figure.
The child process then execs a new process. Inherited descriptors that had the
close-on-exec flag set by the fcntl subroutine close.
General Programming Concepts
The fcntl (File Descriptor Control) Subroutine:
fcntl
Manipulates file structure and controls open file descriptors.
Per Process
u_block area
System File Table
Parent
Descriptor Table
File
Flags
Pointer
File 954
0
1
2
.
.
.
Offset = 67
Read/Write
Ref Count = 2
1999
File 63
Child
Descriptor Table
File
Flags
Pointer
Offset = 7645
Read/Write
Ref Count = 2
0
1
2
.
.
.
1999
Duplicate File Descriptors as a result of a Fork Operation
The fcntl subroutine can be used to make the following changes to a descriptor:
v Duplicate a file descriptor (identical to the dup subroutine).
v Get or set the close-on-exec flag.
v Set nonblocking mode for the descriptor.
v Append future writes to the end of the file (O_APPEND).
v Enable the generation of a signal to the process when it is possible to do I/O.
v Set or get the process ID or the group process ID for SIGIO handling.
v Close all file descriptors.
Preset File Descriptor Values
When the shell runs a program, it opens three files with file descriptors 0, 1, and 2.
The default assignments for these descriptors are:
0
1
2
Represents standard input.
Represents standard output.
Represents standard error.
These default file descriptors are connected to the terminal, so that if a program
reads file descriptor 0 and writes file descriptors 1 and 2, the program collects
input from the terminal and sends output to the terminal. As the program uses
other files, file descriptors are assigned in ascending order.
Chapter 5. File Systems and Directories
125
If I/O is redirected using the < (less than) or > (greater than) symbols, the shell’s
default file descriptor assignments are changed. For instance:
prog < FileX > FileY
changes the default assignments for file descriptors 0 and 1 from the terminal to
the appropriate files. In this example, file descriptor 0 now refers to FileX and file
descriptor 1 refers to FileY. File descriptor 2 has not been changed. The program
does not need to know where its input comes from nor where it is sent, as long as
file descriptor 0 represents the input file and 1 and 2 represent output files.
The following sample program illustrates the redirection of standard output:
#include <fcntl.h>
#include <stdio.h>
void redirect_stdout(char *);
main()
{
printf("Hello world\n");
/*this printf goes to
* standard output*/
fflush(stdout);
redirect_stdout("foo");
/*redirect standard output*/
printf("Hello to you too, foo\n");
/*printf goes to file foo */
fflush(stdout);
}
void
redirect_stdout(char *filename)
{
int fd;
if ((fd = open(filename,O_CREAT|O_WRONLY,0666)) < 0)
/*open a new file */
{
perror(filename);
exit(1);
}
close(1);
/*close old */
*standard output*/
if (dup(fd) !=1)
/*dup new fd to
*standard input*/
{
fprintf(stderr,"Unexpected dup failure\n");
exit(1);
}
close(fd);
/*close original, new fd,*/
* no longer needed*/
}
The value for file descriptor 2 can also be reassigned, but this is rarely done.
Within the file descriptor table, file descriptor numbers are assigned the lowest
descriptor number available at the time of a request for a descriptor. However, any
value can be assigned within the file descriptor table by using the dup subroutine.
File Descriptor Resource Limit
The number of file descriptors that can be allocated to a process is governed by a
resource limit. The default value is set in the /etc/security/limits file and is
typically 2000 (for compatibility with earlier releases). The limit can be changed by
the ulimit command or the setrlimit subroutine. The maximum size is defined by
the constant OPEN_MAX.
126
General Programming Concepts
Related Information
“Chapter 5. File Systems and Directories” on page 97
“Working with JFS i-nodes” on page 103
“File Creation and Removal”
Special Files Overview
fcntl, dup, or dup2 subroutine, lseek subroutine, open, openx, or create subroutine
File Creation and Removal
The internal procedures performed by the operating system when creating,
opening, or closing files are described in the following sections.
Creating a File
Different subroutines create specific types of files. They are:
Subroutine
creat
open
mknod
mkfifo
pipe
socket
mkdir
symlink
Type of File Created
Regular
Regular (when the O_CREAT flag is set)
Regular, first-in-first-out (FIFO), or special
Named pipe (FIFO)
Unnamed pipe
Sockets
Directories
Symbolic link
Creating a Regular File (creat, open, or mknod Subroutines)
You use the creat subroutine to create a file according to the values set in the
Pathname and Mode parameters. If the file named in the Pathname parameter exists
and the process has write permission to the file, the creat subroutine truncates the
file. Truncation releases all data blocks and sets the file size to 0. You can also
create new, regular files using the open subroutine with the O_CREAT flag.
Files created with the creat, mkfifo, or mknod subroutine take the access
permissions set in the Mode parameter. Regular files created with the open
subroutine take their access modes from the O_CREAT flag Mode parameter. The
umask subroutine sets a file-mode creation mask (set of access modes) for new
files created by processes and returns the previous value of the mask.
The permission bits on a newly created file are a result of the reverse of the umask
bits ANDed with the file-creation mode bits set by the creating process. When a
new file is created by a process, the operating system performs the following
actions:
v Determines the permissions of the creating process.
v Retrieves the appropriate umask value.
v Reverses the umask value.
v Uses the AND operation to combine the permissions of the creating process with
the reverse of the umask value.
Chapter 5. File Systems and Directories
127
Creating a Special File (mknod or mkfifo Subroutine)
You can use the mknod and mkfifo subroutines to create new special files. The
mknod subroutine handles named pipes (FIFO), ordinary, and device files. It
creates an i-node for a file identical to that created by the creat subroutine. When
you use the mknod subroutine, the file-type field is set to indicate the type of file
being created. If the file is a block or character-type device file, the names of the
major and minor devices are written into the i-node.
The mkfifo subroutine is an interface for the mknod subroutine and is used to
create named pipes.
Opening a File
The open subroutine is the first step required for a process to access an existing
file. The open subroutine returns a file descriptor. Reading, writing, seeking,
duplicating, setting I/O parameters, determining file status and closing the file all
use the file descriptor returned by the open call. The open subroutine creates
entries for a file in the file descriptor table when assigning file descriptors.
The open subroutine:
v Checks for appropriate permissions that allow the process access to the file.
v Assigns a entry in the file descriptor table for the open file. The open subroutine
sets the initial read/write byte offset to 0, the beginning of the file.
The ioctl or ioctlx subroutines perform control operations on opened special device
files.
Closing a File
When a process no longer needs access to the open file, the close subroutine
removes the entry for the file from the table. If more than one file descriptor
references the file table entry for the file, the reference count for the file is
decreased by 1, and the close completes. If a file has only 1 reference to it, the file
table entry is freed. Attempts by the process to use the disconnected file descriptor
result in errors until another open subroutine reassigns a value for that file
descriptor value. When a process exits, the kernel examines its active user file
descriptors and internally closes each one. This ensures that all files close before
the process ends.
Working with File I/O
All input and output (I/O) operations use the current file offset information stored
in the system file structure (see “System File and File Descriptor Tables” on
page 121). The current I/O offset designates a byte offset that is constantly tracked
for every open file. It is called the current I/O offset because it signals a read or
write process where to begin operations in the file. The open subroutine resets it to
0. The pointer can be set or changed using the lseek subroutine.
Manipulating the Current Offset
Read and write operations can access a file sequentially. This is because the current
I/O offset of the file tracks the byte offset of each previous operation. The offset is
stored in the system file table.
128
General Programming Concepts
You can adjust the offset on files that can be randomly accessed, such as regular
and special-type files, using the lseek subroutine.
Allows a process to position the offset at a designated byte. The lseek subroutine
positions the pointer at the byte designated by the Offset variable. The Offset value
can be calculated from three places in the file (designated by the value of the
Whence variable):
lseek
absolute offset
Beginning byte of the file
relative offset
Position of the former pointer
end_relative offset
End of the file
The return value for the lseek subroutine is the current value of the pointer’s
position in the file. For example:
cur_off= lseek(fd, 0, SEEK_CUR);
The lseek subroutine is implemented in the file table. All following read and write
operations use the new position of the offset as their starting location.
Note: The offset cannot be changed on pipes or socket-type files.
fclear
Subroutine that creates an empty space in a file. It sets to zero the number of
bytes designated in the NumberOfBytes variable beginning at the current offset.
The fclear subroutine cannot be used if the O_DEFER flag was set at the time the
file was opened.
Reading a File
The read
Subroutine that copies a specified number of bytes from an open file to a
specified buffer. The copy begins at the point indicated by the current offset.
The number of bytes and buffer are specified by the NBytes and Buffer
parameters.
The read subroutine:
1. Assures that the FileDescriptor parameter is valid and that the process has read
permissions. The subroutine then gets the file table entry specified by the
FileDescriptor parameter.
2. Sets a flag in the file to indicate a read operation is in progress. This locks other
processes out of the file during the operation.
3. Converts the offset byte value and the value of the NBytes variables into a
block address.
4. Transfers the contents of the identified block into a storage buffer.
5. Copies the contents of the storage buffer into the area designated by the Buffer
variable.
6. Updates the current offset according to the number of bytes actually read.
Resetting the offset assures that the data is read in sequence by the next read
process.
7. Deducts the number of bytes read from the total specified in the NByte variable.
8. Loops until the number of bytes to be read is satisfied.
9. Returns the total number of bytes read.
Chapter 5. File Systems and Directories
129
The cycle completes when the file to be read is empty, the number of bytes
requested is met, or a reading error is encountered during the process.
Errors can occur while the file is being read from disk or in copying the data to the
system file space.
It is advantageous for read requests to start at the beginning of data block
boundaries and to be multiples of the data block size. An extra iteration in the read
loop can be avoided. If a process reads blocks sequentially, the operating system
assumes all subsequent reads will be sequential too.
During the read operation, the i-node is locked. No other processes are allowed to
modify the contents of the file while a read is in progress. However the file is
unlocked immediately on completion of the read operation. If another process
changes the file between two read operations, the resulting data is different, but
the integrity of the data structure is maintained.
The following example illustrates how to use the read subroutine to count the
number of null bytes in the foo file:
#include <fcntl.h>
#include <sys/param.h>
main()
{
int fd;
int nbytes;
int nbytes;
int nnulls;
int i;
char buf[PAGESIZE];
/*A convenient buffer size*/
nnulls=0;
if ((fd = open("foo",O_RDONLY)) < 0)
exit();
while ((nbytes = read(fd,buf,sizeof(buf))) > 0)
for (i = 0; i < nbytes; i++)
if (buf[i] == '\0';
nnulls++;
printf("%d nulls found\n", nnulls);
}
Writing a File
write
Subroutine that adds the amount of data specified in the NBytes variable from the
space designated by the Buffer variable to the file described by the FileDescriptor
variable. It functions similar to the read subroutine. The byte offset for the write
operation is found in the system file table’s current offset.
Sometimes when you write to a file the file does not contain a block corresponding
to the byte offset resulting from the write process. When this happens, the write
subroutine allocates a new block. This new block is added to the i-node
information that defines the file. If adding the new block produces an indirect
block position (i_rindirect), the subroutine allocates more than one block when a
file moves from direct to indirect geometry.
During the write operation, the i-node is locked. No other processes are allowed to
modify the contents of the file while a write is in progress. However the file is
unlocked immediately on completion of the write operation. If another process
changes the file between two write operations, the resulting data is different, but
the integrity of the data structure is maintained.
130
General Programming Concepts
The write subroutine loops in a way similar to the read subroutine, logically
writing one block to disk for each iteration. At each iteration, the process either
writes an entire block or only a portion of one. If only a portion of a data block is
required to accomplish an operation, the write subroutine reads the block from
disk to avoid overwriting existing information. If an entire block is required, it
does not read the block because the entire block is overwritten. The write
operation proceeds block by block until the number of bytes designated in the
NBytes parameter is written.
Delayed Write
You can designate a delayed write process with the O_DEFER flag. Then, the data
is transferred to disk as a temporary file. The delayed write feature caches the data
in case another process reads or writes the data sooner. Delayed write saves extra
disk operations. Many programs, such as mail and editors create temporary files in
the directory /tmp and quickly remove them.
When a file is opened with the deferred update (O_DEFER) flag, the data is not
written to permanent storage until a process issues an fsync subroutine call or a
process issues a synchronous write to the file (opened with O_SYNC flag). The
fsync subroutine saves all changes in an open file to disk. See the open subroutine
for a description of the O_DEFER and O_SYNC flags.
Truncating Files
The truncate or ftruncate subroutines change the length of regular files. The
truncating process must have write permission to the file. The Length variable
value indicates the size of the file after the truncation operation is complete. All
measures are relative to the first byte of the file, not the current offset. If the new
length (designated in the Length variable) is less than the previous length, the data
between the two is removed. If the new length is greater than the existing length,
zeros are added to extend the file size. When truncation is complete, full blocks are
returned to the file system, and the file size is updated.
Writing Programs to Use Direct I/O
Beginning in AIX 4.3, an application will be able to use Direct I/O on JFS files.
This article is intended to assist programmers in understanding the intricacies
involved with writing programs to take advantage of this feature.
Direct I/O vs. Normal Cached I/O
Normally, the JFS caches file pages in kernel memory. When the application does a
file read request, if the file page is not in memory, the JFS reads the data from the
disk into the file cache, then copies the data from the file cache to the user’s buffer.
For application writes, the data is merely copied from the user’s buffer into the
cache. The actual writes to disk are done later.
This type of caching policy can be extremely effective when the cache hit rate is
high. It also enables read-ahead and write-behind policies. Lately, it makes file
writes to the asynchronous, allowing the application to continue processing instead
of waiting for I/O requests to complete.
Direct I/O is an alternative caching policy which causes the file dta to be
transferred from the disk to/from the user’s buffer. Direct I/O for files is
functionally equivalent to raw I/O for devices.
Chapter 5. File Systems and Directories
131
Benefits of Direct I/O
The primary benefit of direct I/O is to reduce CPU utilization for file reads and
writes by eliminating the copy from the cache to the user buffer. This can also be a
benefit for file data which has a very poor cache hit rate. If the cache hit rate is
low, then most read requests have to go to the disk. Direct I/O can also benefit
applications which must use synchronous writes since these writes have to go to
disk. In both of these cases, CPU usage is reduced since the data copy is
eliminated.
A second benefit if direct I/O is that it allows applications to avoid diluting the
effectiveness of caching of other files. Any time a file is read or written, that file
competes for space in the cache. This may cause other file data to be pushed out of
the cache. If the newly cached data has very poor reuse characterisitics, the
effectiveness of the cache can be reduced. Direct I/O gives applications the ability
to identify files where the normal caching policies are ineffective, thus freeing up
more cache space for files where the policies are effective.
Performance Costs of Direct I/O
Although Direct I/O can reduce cpu usage, it typically results in longer wall clock
times, especially for relatively small requests. This penalty is caused by the
fundamental differences between normal cached I/O and Direct I/O.
Direct I/O Reads
Every Direct I/O read causes a synchronous read from disk; unlike the normal
cached I/O policy where read may be satisfied from the cache. This can result in
very poor performance if the data was likely to be in memory under the normal
caching policy.
Direct I/O also bypasses the normal JFS read-ahead algorithms. These algorithms
can be extremely effective for sequential access to files by issuing larger and larger
read requests and by overlapping reads of future blocks with application
processing.
Applications can compensate for the loss of JFS read-ahead by issuing larger reads
requests. At a minimum, Direct I/O readers should issue read requests of at least
128k to match the JFS read-ahead characteristics.
Applications can also simulate JFS read-ahead by issuing asynchronous Direct I/O
read-ahead either by use of multiple threads or by using aio_read.
Direct I/O Writes
Every direct I/O write causes a synchronous write to disk; unlike the normal
cached I/O policy where the data is merely copied and then written to disk later.
This fundamental difference can cause a significant performance penalty for
applications which are converted to use Direct I/O.
Conflicting File Access Modes
In order to avoid consistency issues between programs that use Direct I/O and
programs that use normal cached I/O, Direct I/O is an exclusive use mode. If
there are multiple opens of a file and some of them are direct and others are not,
the file will stay in its normal cached access mode. Only when the file is open
exclusively by Direct I/O programs will the file be placed in Direct I/O mode.
Similarly, if the file is mapped into virtual memory via the shmat or mmap system
calls, then file will stay in normal cached mode.
132
General Programming Concepts
The JFS will attempt to move the file into Direct I/O mode any time the last
conflicting. non-direct access is eliminated (either by close, munmap, or shmdt).
Changing the file from normal mode to Direct I/O mode can be rather expensive
since it requires writing all modified pages to disk and removing all the file’s
pages from memory.
Enabling Applications to use Direct I/O
Applications enable Direct I/O access to a file by passing the O_DIRECT flag to
the fcntl.h. This flag is defined in open. Applications must be compiled with
_ALL_SOURCE enabled to see the definition of O_DIRECT.
Offset/Length/Address Alignment Requirements of the Target
Buffer
In order for Direct I/O to work efficiently, the request should be suitably
conditioned. Applications can query the offset, length, and address alignment
requirements by using the finfo and ffinfo subroutines. When the FI_DIOCAP
command is used, finfo and ffinfo return information in the diocapbuf structure as
described in sys/finfo.h. This structure contains the following fields:
dio_offset
dio_max
dio_min
dio_align
Contains the recommended
this file system
Contains the recommended
files in this system
Contains the recommended
files in this file system
Contains the recommended
in this file system
offset alignment for direct I/O writes to files in
maximum write length for Direct I/O writes to
minimum write length for Direct I/O writes to
buffer alignment for Direct I/O writes to files
Failure to meet these requirements may cause file reads and writes to use the
normal cached model. Different file systems may have different requirements.
FS Format
fixed, 4k blk
fragmented
compressed
big file
dio_offset
4k
4k
n/a
128k
dio_max
2m
2m
n/a
2m
dio_min
4k
4k
n/a
128k
dio_align
4k
4k
n/a
4k
Direct I/O Limitations
Direct I/O is not supported for files in a compressed file filesystem. Attempts to
open these files with O_DIRECT will be ignored and the files will be accessed with
the normal cached I/O methods.
Direct I/O and Data I/O Integrity Completion
Although Direct I/O writes are done synchronously, they do not provide
synchronized I/O data integrity completion, as defined by POSIX. Applications
which need this feature should use O_DSYNC in addition O_DIRECT. O_DSYNC
guarantees that all of the data and enough of the meta-data (eg. indirect blocks)
have written to the stable store to be able to retrieve the data after a system crash.
O_DIRECT only writes the data; it does not write the meta-data.
Working with Pipes
Pipes are unnamed objects created to allow two processes to communicate. One
process reads and the other process writes to the pipe file. This unique type of file
is also called a first-in-first-out (FIFO) file. The data blocks of the FIFO are
manipulated in a circular queue, maintaining read and write pointers internally to
Chapter 5. File Systems and Directories
133
preserve the FIFO order of data. The PIPE_BUF system variable, defined in the
limits.h file, designates the maximum number of bytes guaranteed to be atomic
when written to a pipe.
The shell uses unnamed pipes to implement command pipelining. Most unnamed
pipes are created by the shell. The | (vertical) symbol represents a pipe between
processes. For example:
ls | pr
the output of the ls command is printed to the screen.
Pipes are treated as regular files as far is possible. Normally, the current offset
information is stored in the system file table. However, because pipes are shared
by processes, the read/write pointers must be specific to the file, not to the
process. File table entries are created by the open subroutine and are unique to the
open process, not to the file. Processes with access to pipes share the access
through common system file table entries.
Using Pipe Subroutines
The pipe subroutine creates an interprocess channel and returns two file
descriptors. File descriptor 0 is opened for reading. File descriptor 1 is opened for
writing. The read operation accesses the data on a FIFO basis. These two file
descriptors are used with read, write, and close subroutines.
In the following example, a child process is created and sends its process ID back
through a pipe:
#include <sys/types.h>
main()
{
int p[2];
char buf[80];
pid_t pid;
if (pipe(p))
{
perror("pipe failed");
exit(1)'
}
if ((pid=fork()) == 0)
{
/* in child process */
close(p[0]);
/*close unused read */
*side of the pipe */
sprintf(buf,"%d",getpid());
/*construct data */
/*to send */
write(p[1],buf,strlen(buf)+1);
/*write it out, including
/*null byte */
exit(0);
}
/*in parent process*/
close(p[1]);
/*close unused write side
read(p[0],buf,sizeof(buf));
/*read the pipe*/
printf("Child process said: %s/n", buf);
/*display the result */
exit(0);
}
134
General Programming Concepts
If a process reads an empty pipe, the process waits until data arrives. If a process
writes to a pipe that is too full (PIPE_BUF), the process waits until space is
available. If the write side of the pipe is closed, a subsequent read operation to the
pipe returns an end-of-file.
Two other subroutines that control pipes are the popen and pclose subroutines.
popen
Creates the pipe (using the pipe subroutine) then forks to create a copy of the
caller. The child process decides whether it is supposed to read or write, closes the
other side of the pipe, then calls the shell (using the execl subroutine) to run the
desired process.
The parent closes the end of the pipe it did not use. These closes are necessary to
make end-of-file tests work properly. For example, if a child process intended to
read the pipe does not close the write end of the pipe, it will never see the end of
file condition on the pipe, because there is one write process potentially active.
The conventional way to associate the pipe descriptor with the standard input of a
process is:
close(p[1]);
close(0);
dup(p[0]);
close(p[0]);
The close subroutine disconnects file descriptor 0, the standard input. The dup
subroutine returns a duplicate of an already open file descriptor. File descriptors
are assigned in increasing order and the first available one is returned. The effect
of the dup subroutine is to copy the file descriptor for the pipe (read side) to file
descriptor 0, thus standard input becomes the read side of the pipe. Finally, the
previous read side is closed. The process is similar for a child process to write from
a parent.
pclose
Closes a pipe between the calling program and a shell command to be executed.
Use the pclose subroutine to close any stream opened with the popen subroutine.
The pclose subroutine waits for the associated process to end, then closes and
returns the exist status of the command. This subroutine is preferable to the close
subroutine because pclose waits for child processes to finish before closing the
pipe. Equally important, when a process creates several children, only a bounded
number of unfinished child processes can exist, even if some of them have
completed their tasks. Performing the wait allows child processes to complete their
tasks.
Synchronous I/O
By default, writes to files in JFS file systems are asynchronous. However, JFS file
systems support three types of synchronous I/O. One type is specified by the
O_DSYNC open flag. When a file is opened using the O_DSYNC open mode, the
write () system call will not return until the file data and all file system meta-data
required to retrieve the file data are both written to their permanent storage
locations.
Another type of synchronous I/O is specified by the O_SYNC open flag. In
addition to items specified by O_DSYNC, O_SYNC specifies that the write ()
system call will not return until all file attributes relative to the I/O are written to
their permanent storage locations, even if the attributes are not required to retrieve
the file data.
Chapter 5. File Systems and Directories
135
Before the O_DSYNC open mode existed, AIX applied O_DSYNC semantics to
O_SYNC. For binary compatibility reasons, this behavior can never change. If true
O_SYNC behavior is required, then both O_DSYNC and O_SYNC open flags must
be specified. Exporting the XPG_SUS_ENV=ON environment variable also enables
true O_SYNC behavior.
The last type of synchronous I/O is specified by the O_RSYNC open flag, and it
simply applies the behaviors associated with O_SYNC or _DSYNC to reads. For
files in JFS file systems, only the combination of O_RSYNC | O_SYNC has
meaning. It means that the read system call will not return until the file’s access
time is written to its permanent storage location.
File Status
File status information resides in the i-node. The stat subroutines are used to
return information on a file. The stat subroutines report file type, file owner, access
mode, file size, number of links, i-node number, and file access times. These
subroutines write information into a data structure designated by the Buffer
variable. The process must have search permission for the directories in the path to
the designated file.
stat
lstat
Subroutine that returns the information about files named by the Path parameter. If
the size of the file cannot be represented in the structure designated by the Buffer
variable, stat will fail with the errno set to EOVERFLOW.
Subroutine that provides information about a symbolic link, and the stat
subroutine returns information about the file referenced by the link. The fstat
subroutine returns information from an open file using the file descriptor.
The statfs, fstafs, and ustat subroutines return status information about a file
system. The statfs subroutine returns information about the file system that
contains the file specified by the Path parameter.
fstatfs
ustat
utimes and utime
Returns the information about the file system that contains the file
associated with the given file descriptor. The structure of the
returned information is described in the /usr/include/sys/statfs.h
file for the statfs and fstatfs subroutines and in the ustat.h file for
the ustat subroutine.
Returns information about a mounted file system designated by
the Device variable. This device identifier is for any given file and
can be determined by examining the st_dev field of the stat
structure defined in the /usr/include/sys/stat.h file. The ustat
subroutine is superseded by the statfs and fstatfs subroutines.
Also affect file status information. They change the file access and
modification time in the i-node.
File Accessibility
Every file is created with an access mode. Each access mode grants read, write, or
execute permission to users, the user’s group, and all other users.
The access bits on a newly created file are a result of the reverse of the umask bits
ANDed with the file-creation mode bits set by the creating process. When a new
file is created by a process, the operating system performs the following actions:
v Determines the permissions of the creating process
v Retrieves the appropriate umask value
v Reverses the umask value
136
General Programming Concepts
v Uses the AND operation to combine the permissions of the creating process with
the reverse of the umask value
For example, if an existing file has the 027 permissions bits set, the user is not
allowed any permissions. Write permission is granted to the group. Read, write,
and execute access is set for all others. The umask value of the 027 permissions
modes would be 750 (the opposite of the original permissions). When 750 is
ANDed with 666 (the file creation mode bits set by the system call that created the
file), the actual permissions for the file are 640. Another representation of this
example is:
027 = _ _ _ _ W _ R W X
750 = R W X R _ X _ _ _
666 =
ANDED
750 =
640 =
R W _ R W _ R W _
TO
R W X R _ X _ _ _
R W _ R _ _ _ _ _
umask
chmod and fchmod
access
Existing file access mode
Reverse (umask) of original
permissions
File creation access mode
The umask value
Resulting file access mode
Subroutine that sets and gets the value of the file creation mask.
Subroutines that change file access permissions.
Subroutine that investigates and reports on the accessibility mode
of the file named in the Pathname parameter. This subroutine uses
the real user ID and the real group ID instead of the effective user
and group ID. Using the real user and group IDs allows programs
with the set-user-ID and set-group-ID access modes to limit access
only to users with proper authorization.
Consider the following example:
$ ls -l
total 0
-r-s—x—x
1 root system 8290 Jun 09 17:07 special
-rw———1 root system 1833 Jun 09 17:07 secrets
$ cat secrets
cat: cannot open secrets
In this example, the user does not have access to the file secrets. However, when
the program special is run and the access mode for the program is set-uID root,
the program can access the file. The program must use the access subroutine to
prevent subversion of system security.
The access subroutine must be used by any set-uID or set-gID program to forestall
this type of intrusion.
chown
Subroutine resets the ownership field of the i-node for the file and clears the
previous owner. The new information is written to the i-node and the process
finishes.
The chmod subroutine works in similar fashion, but the permission mode flags are
changed instead of the file ownership.
Changing file ownership and access modes are actions that affect the i-node, not
the data in the file. The owner of the process must have root user authority or own
the file to make these changes.
Chapter 5. File Systems and Directories
137
File System Layout
A file system is a set of files, directories, and other structures. File systems
maintain information and identify where a file or directory’s data is located on the
disk. In addition to files and directories, file systems contain a boot block, a
superblock, bitmaps, and one or more allocation groups. An allocation group
contains disk i-nodes and fragments. Each file system occupies one logical volume.
Boot Block
The boot block occupies the first 4096 bytes of the file system starting at byte offset
0 on the disk. The boot block is available to start the operating system.
Superblock
The superblock is 4096 bytes in size and starts at byte offset 4096 on the disk. The
super- block maintains information about the entire file system and includes the
following fields:
v Size of the file system
v Number of data blocks in the file system
v A flag indicating the state of the file system
v Allocation group sizes
Allocation Bitmaps
The file system contains two allocation bitmaps:
v The fragment allocation map records the allocation state of each fragment.
v The disk i-node allocation map records the status of each i-node.
Fragments
Many file systems have disk blocks or data blocks. These blocks divide the disk
into units of equal size to store the data in a file or directory’s logical blocks. The
disk block may be further divided into fixed-size allocation units called fragments.
Some systems do not allow fragment allocations to span the boundaries of the disk
block. In other words, a logical block cannot be allocated fragments from different
disk blocks.
The journaled file system (JFS), however, provides a view of the file system as a
contiguous series of fragments. JFS fragments are the basic allocation unit and the
disk is addressed at the fragment level. Thus, fragment allocations can span the
boundaries of what might otherwise be a disk block. The default JFS fragment size
is 4096 bytes, although you can specify smaller sizes. In addition to containing data
for files and directories, fragments also contain disk addresses and data for indirect
blocks. “File Space Allocation” on page 104 explains how the operating system
allocates fragments.
Disk I-Nodes
A logical block contains a file or directory’s data in units of 4096 bytes. Each
logical block is allocated fragments for the storage of its data. Each file and
directory has an i-node that contains access information such as file type, access
138
General Programming Concepts
permissions, owner’s ID, and number of links to that file. These i-nodes also
contain ″addresses″ for finding the location on the disk where the data for a logical
block is stored.
Each i-node has an array of numbered sections. Each section contains an address
for one of the file or directory’s logical blocks. These addresses indicate the starting
fragment and the total number of fragments included in a single allocation. For
example, a file with a size of 4096 bytes has a single address on the i-node’s array.
Its 4096 bytes of data are contained in a single logical block. A larger file with a
size of 6144 bytes has two addresses. One address contains the first 4096 bytes and
a second address contains the remaining 2048 bytes (a partial logical block). If a
file has a large number of logical blocks, the i-node does not contain the disk
addresses. Instead, the i-node points to an indirect block which contains the
additional addresses.
Allocation Groups
The set of fragments making up the file system are divided into one or more
fixed-sized units of contiguous fragments. Each unit is an allocation group. The
first of these groups begins the file system and contains a reserved area occupying
the first 32 x 4096 bytes of the group. The first 4096 bytes of this area hold the boot
block and the second 4096 bytes hold the file system superblock.
Each allocation group contains a static number of contiguous disk i-nodes which
occupy some of the group’s fragments. These fragments are set aside for the
i-nodes at file system creation and extension time. For the first allocation group,
the disk i-nodes occupy the fragments immediately following the reserved block
area. For subsequent groups, the disk i-nodes are found at the start of each group.
Disk i-nodes are 128 bytes in size and are identified by a unique disk i-node
number or i-number. The i-number maps a disk i-node to its location on the disk
or to an i-node within its allocation group.
A file system’s allocation groups are described by three sizes:
v The fragment allocation group size and the disk i-node allocation group size are
specified as the number of fragments and disk i-nodes that exist in each
allocation group.
v The default allocation group size is 8 MB.
v Beginning in Version 4.2, it can be as large as 64 MB.
These three values are stored in the file system superblock, and they are set at file
system creation.
Allocation groups allow the JFS resource allocation policies to use effective
methods for achieving good file system I/O performance. These allocation policies
try to cluster disk blocks and disk i-nodes for related data to achieve good locality
for the disk. Files are often read and written sequentially and files within a
directory are often accessed together. Also, these allocation policies try to distribute
unrelated data throughout the file system in an attempt to minimize free space
fragmentation.
Using File System Subroutines
The most used file system subroutines are:
fscntl
Controls file system control operations
Chapter 5. File Systems and Directories
139
getfsent, getfsspec, getfsfile, getfstype, setfsent, or endfsent
Obtain information about a file system
lseek
Moves the read-write pointer
mntctl
Returns mount status information
vmount or mount
Make a file system ready for use
statfs, fstsfs, or ustat
Report file system statistics
sync
Updates file systems to disk
Other subroutines are designed for use on virtual file systems (VFS):
getvfsent, getvfsbytype, getvfsbyname, getvfsbyflag, sevfsent, or endvfsent
Retrieve a VFS entry
umount or uvmount
Remove VFS from the file tree
Creating New File System Types
If it is necessary to create a new type of file system, file system helpers and mount
helpers must be created. The following sections provide information about the
implementation specifics and execution syntax of file system and mount helpers.
File System Helpers
To enable support of multiple file system types, most file system commands do not
contain the code that communicates with individual file systems. Instead, the
commands collect parameters, file system names, and other information not
specific to one file system type and then pass all this information to a back-end
program (the helper).
The back end understands specific information about the relevant file system type
and does the detail work of communicating with the file system. Back-end
programs used by file system commands are known as file system and mount
helpers.
To determine the appropriate file system helper for each file system type, the
front-end command examines the /etc/vfs file. This file contains a line for each file
system type. Each line includes a field defining the appropriate helper program.
There are two helper fields in the /etc/vfs file: one for the mount helper and one
for the file system helper. The file system helper is a single executable that handles
all requests for file system specifics, except for file system mounting, which is
controlled by the mount helper.
For example:
fsck -fp /users
invokes the fsck command on a file system named /users. /users is a journaled
file system (JFS). JFS is the default file system type.
In this example, the fsck command does several things including:
v Notes the fast and preen (-fp) options request
v Looks into the /etc/filesystems file to see which file system type corresponds to
the name /users
140
General Programming Concepts
v Determines the device for the /users file system as specified in the
/etc/filesystems file
v Passes all this information to the appropriate file system helper
The actual invocation of the helper by the fsck command includes an indication of
which operation to perform, which file system or device to perform the action on,
and any options requested (fast and preen in the preceding example).
File System Helper Operations
The following table lists the possible operations requested of a helper in the
/usr/include/fshelp.h file:
Helper Operations
#define FSHOP_NULL
#define FSHOP_CHECK
#define FSHOP_CHGSIZ
#define FSHOP_FINDATA
#define FSHOP_FREE
#define FSHOP_MAKE
#define FSHOP_REBUILD
#define FSHOP_STATFS
#define FSHOP_STAT
#define FSHOP_USAGE
#define FSHOP_NAMEI
#define FSHOP_DEBUG
Value
0
1
2
3
4
5
6
7
8
9
10
11
However, the JFS file system supports only the following operations:
Operation Value Corresponding Command
#define FSHOP_CHECK
1
fsck
#define FSHOP_CHGSIZ
2
chfs
#define FSHOP_MAKE
5
mkfs
#define FSHOP_STATFS
7
df
#define FSHOP_NAMEI
10 ff
File System Helper Execution Syntax
The execution syntax of the file system helper is:
OpName OpKey FilsysFileDescriptor PipeFileDescriptor Modeflags
DebugLevel OpFlags
Field
OpName
OpKey
FilsysFileDescriptor
Definition
Specifies the arg0 parameter when the program invokes the
helper. The value of the OpName field appears in a list of
processes (see the ps command).
Corresponds to the available helper operations. Thus, if the
OpKey value is 1, the fsck (file system check) operation is
being requested.
Indicates the file descriptor on which the file system has
been opened by the program.
Chapter 5. File Systems and Directories
141
Field
PipeFileDescriptor
Modeflags
Definition
Indicates the file descriptor of the pipe (see the pipe
subroutine) that is open between the original program and
the helper program. This channel allows the helper to
communicate with the front-end program.
Example: The helper sends an indication of its success or
failure through the pipe, which can affect further front-end
processing. Also, if the debug level is high enough, the
helper can have additional information to send to the
original program.
Provides an indication of how the helper is being invoked
and can affect the behavior of the helper, especially in
regard to error reporting. Mode flags are defined in the
/usr/include/fshelp.h file:
Flags
Indicator
#define FSHMOD_INTERACT_FLAG
″i″
#define FSHMOD_FORCE_FLAG
″f″
#define FSHMOD_NONBLOCK_FLAG
″n″
#define FSHMOD_PERROR_FLAG
″p″
#define FSHMOD_ERRDUMP_FLAG
″e″
#define FSHMOD_STANDALONE_FLAG
″s″
#define FSHMOD_IGNDEVTYPE_FLAG
″I″
DebugLevel
OpFlags
Example: The FSHMOD_INTERACT flag indicates whether
the command is being run interactively (determined by
testing the isatty subroutine on the standard input). Not
every operation uses all (or any) of these modes.
Determines the amount of debugging information required:
the higher the debugging level, the more detailed the
information returned.
Includes the actual device (or devices) on which the
operation is to be performed and any other options specified
by the front end.
Sample Helper Invocations
The actual invocation of the file system helper for fsck -fp /user is:
execl("/etc/helpers/v3fshelpers","fshop_check","1","3","5","ifp",
"0","devices=/dev/lv02,fast,preen,mounted")
In this example:
v The executable execd is /etc/helper/v3fshelpers.
v The name appearing in a listing of processes (ps command) is fshop_check.
v The operation requested is FSHOP_CHECK as represented by the value ″1″.
v The file system is open on file descriptor ″3″.
142
General Programming Concepts
v The pipe through which the helper can send information to the front-end
program is open on file descriptor ″5″.
v The ModeFlags string is ″-ifp″, indicating interactive mode plus force and
perror modes.
v The DebugLevel is 0, so no extra debugging output is returned to the fsck
command.
v Finally, the OpFlags string tells the back-end programs which device the
operation is to be performed on (/dev/lv02), the options requested (fast and
preen), and notes that the device is mounted. For the fsck command, this means
no changes will be made since fsck does not work on mounted file systems.
Another example of how file system helpers are invoked uses the mkfs command.
To create a JFS file system on the existing logical volume named /dev/lv02, enter:
mkfs /junk
which creates a mount point. If you want to create a file system and only know the
device you wish to mount it to, enter:
mkfs /dev/lv02
Either way, the following file system helper is invoked:
execl ("/etc/helpers/v3fshelpers",
"fshop_make", "5", "3", "5", "-ip", "0", "name=/junk,label=/junk,dev=/dev/lv02")
The operation requested is now FSHOP_MAKE. The mode is interactive and
perror. The OpFlags string includes both the mount point and the device.
Mount Helpers
The mount command is a front-end program that uses a helper to communicate
with specific file systems. Helper programs for the mount and umount (or
unmount) commands are called mount helpers.
Like other file system-specific commands, the mount command collects the
parameters and options given at the command line and interprets that information
within the context of the file system configuration information found in the
/etc/filesystems file. Using the information in the /etc/filesystems file, the
command invokes the appropriate mount helper for the type of file system
involved. For example, if the user enters:
mount /test
the mount command checks the /etc/filesystems file for the stanza that describes
the /test file system. From the /etc/filesystems file, the mount command
determines that the /test file system is a remote NFS mount from the node named
host1. The mount command also notes any options associated with the mount.
An example /etc/filesystems file stanza is:
/test:
dev
= /export
vfs
= nfs
nodename
= host1
options = ro,fg,hard,intr
The file system type (nfs in our example) determines which mount helper to
invoke. The command compares the file system type to the first fields in the
/etc/vfs file. The field that matches will have the mount helper as its third field.
Chapter 5. File Systems and Directories
143
Mount Helper Execution Syntax
The following is a sample of the execution syntax of the mount helper:
/etc/helpers/nfsmnthelp M 0 host1 /export /test ro,fg,hard,intr
Both the mount and unmount commands have six parameters. The first four
parameters are the same for both commands:
operation
debuglevel
nodename
object
Indicates operation requested of the helper. Values are either M (mount
operation), Q (query operation), or U (unmount operation). The query
operation is obsolete.
Gives the numeric parameter for the -D flag. Neither the mount nor the
unmount commands support the -D flag, so the value is 0.
Names the node for a remote mount or a null string for local mounts. The
mount or unmount commands do not invoke a mount helper if the
nodename parameter is null.
Names the local or remote device, directory, or file that is being mounted or
unmounted. Not all file systems support all combinations. For example,
most remote file systems do not support device mounts, while most local
file systems do not support anything else.
The remaining parameters for the mount command are:
mount point
options
Names the local directory or file where the object is to be mounted.
Lists any file system-specific options, separated by commas. Information
for this parameter comes from the options field of the relevant stanza in
the /etc/filesystems file or from the -o Options flag on the command line
(mount -o Options). The mount command also recognizes the -r
(read-only) flag and converts it to the string ro in this field.
The remaining parameters for the unmount command are:
vfsNumber
flag
Gives the unique number that identifies the mount being undone. The
unique number is returned by the vmount call and can be retrieved by
calling the mntctl or stat subroutine. For the mount helper, this parameter is
used as the first parameter to the uvmount subroutine call that actually does
the unmount.
Gives the value of the second parameter to the uvmount subroutine. The
value is 1 if the unmount operation is forced using the -f flag (umount -f).
Otherwise, the value is 0. Not all file systems support forced unmounts.
Major Control Block Header Files
The Base Operating System constants and control block structure definitions are
contained in header files in the /usr/include and /usr/include/sys directories. For a
list of the major constants and control blocks and their corresponding header files,
see List of Major Control Block Header Files.
144
General Programming Concepts
Chapter 6. Floating-Point Exceptions
This chapter provides information about floating-point exceptions and how your
programs can detect and handle them.
The Institute of Electrical and Electronics Engineers (IEEE) defines a standard for
floating-point exceptions called the IEEE Standard for Binary Floating-Point
Arithmetic (IEEE 754). This standard defines five types of floating-point exception
that must be signaled when detected:
v Invalid operation
v Division by zero
v Overflow
v Underflow
v Inexact calculation
When one of these exceptions occurs in a user process, it is signaled either by
setting a flag or taking a trap. By default, the system sets a status flag in the
Floating-Point Status and Control registers (FPSCR), indicating the exception has
occurred. Once the status flags are set by an exception, they are cleared only when
the process clears them explicitly or when the process ends. The operating system
provides subroutines to query, set, or clear these flags.
The system can also cause the floating-point exception signal (SIGFPE) to be raised
if a floating-point exception occurs. Because this is not the default behavior, the
operating system provides subroutines to change the state of the process so the
signal is enabled. When a floating-point exception raises the SIGFPE signal, the
process terminates and produces a core file if no signal-handler subroutine is
present in the process. Otherwise, the process calls the signal-handler subroutine.
State Diagram
The Subroutine Impact figure illustrates the default state when floating-point
exceptions occur and how this behavior can be changed by the appropriate
operating system subroutine calls.
© Copyright IBM Corp. 1997, 1999
145
Floating-Point
Exception Occurs
Exception Enabled
Exception Disabled
Default Result Delivered
Sticky Bit Set in FPSCR
fp_enable
1
fp_enable_all
Pipeline State
Serial State
Result May Be Undefined
Sticky Bit Set in FPSCR
FEX Bit Set in FPSCR
fp_trap 1
Signal Handler
No Signal Handler
sigaction 1
Process Terminates
1
Signal Handler Entered
Call to Modify from Default
= Default
Subroutine Impact on Default State When
Floating-Point Exceptions Occur
Floating-Point Exception Subroutines
Floating-point exception subroutines can be used to:
v Change the execution state of the process
v Enable the signaling of exceptions
v Disable exceptions or clear flags
v Determine which exceptions caused the signal
v Test the exception sticky flags
The following subroutines are provided to accomplish these tasks:
fp_any_xcp or fp_divbyzero
fp_enable or fp_enable_all
fp_inexact, fp_invalid_op, fp_iop_convert,
fp_iop_infdinf, fp_iop_infmzr,
fp_iop_infsinf, fp_iop_invcmp, fp_iop_snan,
fp_iop_sqrt, fp_iop_vxsoft, fp_iop_zrdzr, or
fp_overflow
fp_sh_info
fp_sh_set_stat
fp_trap
fp_underflow
146
General Programming Concepts
Test the exception sticky flags
Enable the signaling of exceptions
Test the exception sticky flags
Determines which exceptions caused the
signal
Disables exceptions or clear flags
Changes the execution state of the process
Tests the exception sticky flags
sigaction
Installs signal handler
Floating-Point Trap Handler Operation
To generate a trap, a program must change the execution state of the process using
the fp_trap subroutine and enable the exception to be trapped using the fp_enable
or fp_enable_all subroutine.
Changing the execution state of the program may slow performance because
floating-point trapping causes the process to execute in serial mode.
When a floating-point trap occurs, the SIGFPE signal is raised. By default, the
SIGFPE signal causes the process to terminate and produce a core file. To change
this behavior, the program must establish a signal handler for this signal. See the
sigaction, sigvec, or signal subroutines for more information on signal handlers.
Exceptions: Disabled and Enabled Comparison
Refer to the following lists for an illustration of the differences between the
disabled and enabled processing states and the subroutines that are used.
Exceptions-Disabled Model
The following subroutines test exception flags in the disabled processing state:
v fp_any_xcp
v
v
v
v
v
v
v
v
v
v
v
v
v
v
v
fp_clr_flag
fp_divbyzero
fp_inexact
fp_invalid_op
fp_iop_convert
fp_iop_infdinf
fp_iop_infmzr
fp_iop_infsi
fp_iop_invcmp
fp_iop_snan
fp_iop_sqrt
fp_iop_vxsoft
fp_iop_zrdzr
fp_overflow
fp_underflow
Exceptions-Enabled Model
The following subroutines function in the enabled processing state:
fp_enable or fp_enable_all
fp_sh_info
fp_sh_set_stat
fp_trap
sigaction
Enable the signaling of exceptions
Determines which exceptions caused the signal
Disables exceptions or clear flags
Changes the execution state of the process
Installs signal handler
Chapter 6. Floating-Point Exceptions
147
Imprecise Trapping Modes
Some systems have imprecise trapping modes. This means the hardware can detect a
floating-point exception and deliver an interrupt, but processing may continue
while the signal is delivered. As a result, the instruction address register (IAR) is at
a different instruction when the interrupt is delivered.
Imprecise trapping modes cause less performance degradation than precise trapping
mode. However, some recovery operations are not possible, because the operation
that caused the exception cannot be determined or because subsequent instruction
may have modified the argument that caused the exception.
To use imprecise exceptions, a signal handler must be able to determine if a trap
was precise or imprecise.
Precise Traps
In a precise trap, the instruction address register (IAR) points to the instruction
that caused the trap. A program can modify the arguments to the instruction and
restart it, or fix the result of the operation and continue with the next instruction.
To continue, the IAR must be incremented to point to the next instruction.
Imprecise Traps
In an imprecise trap, the IAR points to an instruction beyond the one that caused
the exception. The instruction to which the IAR points has not been started. To
continue execution, the signal handler does not increment the IAR.
To eliminate ambiguity, the trap_mode field is provided in the fp_sh_info structure.
This field specifies the trapping mode in effect in the user process when the signal
handler was entered. This information can also be determined by examining the
Machine Status register (MSR) in the mstsave structure.
The fp_sh_info subroutine allows a floating-point signal handler to determine if
the floating-point exception was precise or imprecise.
Note: Even when precise trapping mode is enabled some floating-point
exceptions may be imprecise (such as operations implemented in software).
Similarly, in imprecise trapping mode some exceptions may be precise.
When using imprecise exceptions, some parts of your code may require that all
floating-point exceptions are reported before proceeding. The fp_flush_imprecise
subroutine is provided to accomplish this. It is also recommended that the atexit
subroutine be used to register the fp_flush_imprecise subroutine to run at
program exit. Running at exit ensures that the program does not exit with
unreported imprecise exceptions.
Hardware-Specific Subroutines
Some systems have hardware instructions to compute the square root of a
floating-point number and to convert a floating-point number to an integer. Models
not having these hardware instructions use software subroutines to do this. Either
method can cause a trap if the invalid operation exception is enabled. The software
subroutines report, through the fp_sh_info subroutine, that an imprecise exception
occurred, because the IAR does not point to a single instruction that can be
restarted to retry the operation.
148
General Programming Concepts
Example of a Floating-Point Trap Handler
/*
* This code demonstates a working floating-point exception
* trap handler. The handler simply identifies which
* floating-point exceptions caused the trap and return.
* The handler can return via either the default signal return
* mechanism (sigreturn() in AIX version 3) or longjmp();
* a global handshaking variable determines which. In
* practice a working trap handler would probably use one
* of these methods, not both.
*/
#include
#include
#include
#include
#include
#include
<signal.h>
<setjmp.h>
<fpxcp.h>
<fptrap.h>
<stdlib.h>
<stdio.h>
#define EXIT_BAD -1
#define EXIT_GOOD 0
/*
* Handshaking variable with the signal handler. If zero,
* then the signal hander returns via the default signal
* return mechanism; if non-zero, then the signal handler
* returns via longjmp.
*/
static int fpsigexit;
#define SIGRETURN_EXIT 0
#define LONGJUMP_EXIT 1
static jmp_buf jump_buffer;
#define JMP_DEFINED 0
#define JMP_FPE
2
/* jump buffer */
/* setjmp rc on initial call */
/* setjmp rc on return from */
/* signal handler */
/*
* The fp_list structure allows text descriptions
* of each possible trap type to be tied to the mask
* that identifies it.
*/
typedef struct
{
fpflag_t mask;
char
*text;
} fp_list_t;
/* IEEE required trap types */
fp_list_t
trap_list[] =
{
{ FP_INVALID,
"FP_INVALID"},
{ FP_OVERFLOW,
"FP_OVERFLOW"},
{ FP_UNDERFLOW,
"FP_UNDERFLOW"},
{ FP_DIV_BY_ZERO, "FP_DIV_BY_ZERO"},
{ FP_INEXACT,
"FP_INEXACT"}
};
/* INEXACT detail list — this is an system extension */
fp_list_t
detail_list[] =
{
{ FP_INV_SNAN,
{ FP_INV_ISI,
{ FP_INV_IDI,
{ FP_INV_ZDZ,
{ FP_INV_IMZ,
{ FP_INV_CMP,
"FP_INV_SNAN" } ,
"FP_INV_ISI" } ,
"FP_INV_IDI" } ,
"FP_INV_ZDZ" } ,
"FP_INV_IMZ" } ,
"FP_INV_CMP" } ,
Chapter 6. Floating-Point Exceptions
149
{ FP_INV_SQRT, "FP_INV_SQRT" } ,
{ FP_INV_CVI,
"FP_INV_CVI" } ,
{ FP_INV_VXSOFT, "FP_INV_VXSOFT" }
};
/*
* the TEST_IT macro is used in main() to raise
* an exception.
*/
#define TEST_IT(WHAT, RAISE_ARG)
{
puts(strcat("testing: ", WHAT));
fp_clr_flag(FP_ALL_XCP);
fp_raise_xcp(RAISE_ARG);
}
\
\
\
\
\
/*
* NAME: my_div
*
* FUNCTION: Perform floating-point division.
*
*/
double
my_div(double x, double y)
{
return x / y;
}
/*
* NAME: sigfpe_handler
*
* FUNCTION: A trap handler that is entered when
*
a floating-point exception occurs. The
*
function determines what exceptions caused
*
the trap, prints this to stdout, and returns
*
to the process which caused the trap.
*
* NOTES:
This trap handler can return either via the
*
default return mechanism or via longjmp().
*
The global variable fpsigexit determines which.
*
*
When entered, all floating-point traps are
*
disabled.
*
*
This sample uses printf(). This should be used
*
with caution since printf() of a floating*
point number can cause a trap, and then
*
another printf() of a floating-point number
*
in the signal handler will corrupt the static
*
buffer used for the conversion.
*
* OUTPUTS: The type of exception that caused
*
the trap.
*/
static void
sigfpe_handler(int sig,
int code,
struct sigcontext *SCP)
{
struct mstsave * state = &SCP->sc_jmpbuf.jmp_context;
fp_sh_info_t flt_context;
/* structure for fp_sh_info()
/* call */
int i;
/* loop counter */
extern int fpsigexit;
/* global handshaking variable */
extern jmp_buf jump_buffer
/* */
/*
* Determine which floating-point exceptions caused
* the trap. fp_sh_info() is used to build the floating signal
150
General Programming Concepts
* handler info structure, then the member
* flt_context.trap can be examined. First the trap type is
* compared for the IEEE required traps, and if the trap type
* is an invalid operation, the detail bits are examined.
*/
fp_sh_info(SCP, &flt_context, FP_SH_INFO_SIZE);
static void
sigfpe_handler(int sig,
int code,
struct sigcontext *SCP)
{
struct mstsave * state = &SCP->sc_jmpbuf.jmp_context;
fp_sh_info_t flt_context;
/* structure for fp_sh_info()
/* call */
int i;
/* loop counter */
extern int fpsigexit;
/* global handshaking variable */
extern jmp_buf jump_buffer; /* */
/*
* Determine which floating-point exceptions caused
* the trap. fp_sh_info() is used to build the floating signal
* handler info structure, then the member
* flt_context.trap can be examined. First the trap type is
* compared for the IEEE required traps, and if the trap type
* is an invalid operation, the detail bits are examined.
*/
fp_sh_info(SCP, &flt_context, FP_SH_INFO_SIZE);
for (i = 0; i < (sizeof(trap_list) / sizeof(fp_list_t)); i++)
{
if (flt_context.trap & trap_list[i].mask)
(void) printf("Trap caused by %s error\n", trap_list[i].text);
}
if (flt_context.trap & FP_INVALID)
{
for (i = 0; i < (sizeof(detail_list) / sizeof(fp_list_t)); i++)
{
if (flt_context.trap & detail_list[i].mask)
(void) printf("Type of invalid op is %s\n", detail_list[i].text);
}
}
/* report which trap mode was in effect */
switch (flt_context.trap_mode)
{
case FP_TRAP_OFF:
puts("Trapping Mode is OFF");
break;
case FP_TRAP_SYNC:
puts("Trapping Mode is SYNC");
break;
case FP_TRAP_IMP:
puts("Trapping Mode is IMP");
break;
case FP_TRAP_IMP_REC:
puts("Trapping Mode is IMP_REC");
break;
default:
puts("ERROR: Invalid trap mode");
}
if (fpsigexit == LONGJUMP_EXIT)
{
/*
* Return via longjmp. In this instance there is no need to
* clear any exceptions or disable traps to prevent
* recurrence of the exception, because on return the
* process will have the signal handler's floating-point
Chapter 6. Floating-Point Exceptions
151
* state.
*/
longjmp(jump_buffer, JMP_FPE);
}
else
{
/*
* Return via default signal handler return mechanism.
* In this case you must take some action to prevent
* recurrence of the trap, either by clearing the
* exception bit in the fpscr or by disabling the trap.
* In this case, clear the exception bit.
* The fp_sh_set_stat routine is used to clear
* the exception bit.
*/
fp_sh_set_stat(SCP, (flt_context.fpscr & ((fpstat_t) xflt_context.trap)));
/*
* Increment the iar of the process that caused the trap,
* to prevent re-execution of the instruction.
* The FP_IAR_STAT bit in flt_context.flags indicates if
* state->iar points to an instruction that has logically
* started. If this bit is true, state->iar points to
* an operation that has started and will cause another
* exception if it runs again. In this case you want to
* continue execution and ignore the results of that
* operation, so the iar is advanced to point to the
* next instruction. If the bit is false, the iar already
* points to the next instruction that must run.
*/
if ( flt_context.flags & FP_IAR_STAT )
{
puts("Increment IAR");
state->iar += 4;
}
}
return;
}
/*
* NAME: main
*
* FUNCTION: Demonstrate the sigfpe_handler trap handler.
*
*/
int
main(void)
{
struct sigaction response;
struct sigaction old_response;
extern int fpsigexit;
extern jmp_buf jump_buffer;
int jump_rc;
int trap_mode;
double arg1, arg2, r;
/*
* Set up for floating-point trapping. Do the following:
* 1. Clear any existing floating-point exception flags.
* 2. Set up a SIGFPE signal handler.
* 3. Place the process in synchronous execution mode.
* 4. Enable all floating-point traps.
*/
fp_clr_flag(FP_ALL_XCP);
(void) sigaction(SIGFPE, NULL, &old_response);
(void) sigemptyset(&response.sa_mask);
152
General Programming Concepts
response.sa_flags = FALSE;
response.sa_handler = (void (*)(int)) sigfpe_handler;
(void) sigaction(SIGFPE, &response, NULL);
fp_enable_all();
/*
* Demonstate trap handler return via default signal handler
* return. The TEST_IT macro will raise the floating-point
* exception type given in its second argument. Testing
* is done in this case with precise trapping, because
* it is supported on all platforms to date.
*/
trap_mode = fp_trap(FP_TRAP_SYNC);
if ((trap_mode == FP_TRAP_ERROR) ||
(trap_mode == FP_TRAP_UNIMPL))
{
printf("ERROR: rc from fp_trap is %d\n",
trap_mode);
exit(-1);
}
(void) printf("Default signal handler return: \n");
fpsigexit = SIGRETURN_EXIT;
TEST_IT("div by zero", FP_DIV_BY_ZERO);
TEST_IT("overflow",
FP_OVERFLOW);
TEST_IT("underflow", FP_UNDERFLOW);
TEST_IT("inexact",
FP_INEXACT);
TEST_IT("signaling nan",
TEST_IT("INF - INF",
TEST_IT("INF / INF",
TEST_IT("ZERO / ZERO",
TEST_IT("INF * ZERO",
TEST_IT("invalid compare",
TEST_IT("invalid sqrt",
TEST_IT("invalid coversion",
TEST_IT("software request",
FP_INV_SNAN);
FP_INV_ISI);
FP_INV_IDI);
FP_INV_ZDZ);
FP_INV_IMZ);
FP_INV_CMP);
FP_INV_SQRT);
FP_INV_CVI);
FP_INV_VXSOFT);
/*
* Next, use fp_trap() to determine what the
* the fastest trapmode available is on
* this platform.
*/
trap_mode = fp_trap(FP_TRAP_FASTMODE);
switch (trap_mode)
{
case FP_TRAP_SYNC:
puts("Fast mode for this platform is PRECISE");
break;
case FP_TRAP_OFF:
puts("This platform dosn't support trapping");
break;
case FP_TRAP_IMP:
puts("Fast mode for this platform is IMPRECISE");
break;
case FP_TRAP_IMP_REC:
puts("Fast mode for this platform is IMPRECISE RECOVERABLE");
break;
default:
printf("Unexpected return code from fp_trap(FP_TRAP_FASTMODE): %d\n",
trap_mode);
exit(-2);
}
/*
* if this platform supports imprecise trapping, demonstate this.
*/
Chapter 6. Floating-Point Exceptions
153
trap_mode = fp_trap(FP_TRAP_IMP);
if (trap_mode != FP_TRAP_UNIMPL)
{
puts("Demonsrate imprecise FP execeptions");
arg1 = 1.2;
arg2 = 0.0;
r = my_div(arg1, arg2);
fp_flush_imprecise();
}
/* demonstate trap handler return via longjmp().
*/
(void) printf("longjmp return: \n");
fpsigexit = LONGJUMP_EXIT;
jump_rc = setjmp(jump_buffer);
switch (jump_rc)
{
case JMP_DEFINED:
(void) printf("setjmp has been set up; testing ...\n");
TEST_IT("div by zero", FP_DIV_BY_ZERO);
break;
case JMP_FPE:
(void) printf("back from signal handler\n");
/*
* Note that at this point the process has the floating* point status inherited from the trap handler. If the
* trap hander did not enable trapping (as the example
* did not) then this process at this point has no traps
* enabled. We create a floating-point exception to
* demonstrate that a trap does not occur, then re-enable
* traps.
*/
(void) printf("Creating overflow; should not trap\n");
TEST_IT("Overflow", FP_OVERFLOW);
fp_enable_all();
break;
default:
(void) printf("unexpected rc from setjmp: %d\n", jump_rc);
exit(EXIT_BAD);
}
exit(EXIT_GOOD);
}
154
General Programming Concepts
Chapter 7. Input and Output Handling
This chapter provides an introduction to programming considerations for input
and output handling and the input and output handling (I/O) subroutines.
The input and output (I/O) library subroutines can send data to or from either
devices or files. The system treats devices as if they were I/O files. For example,
you must also open and close a device just as you do a file.
Some of the subroutines use standard input and standard output as their input and
output channels. For most of the subroutines, however, you can specify a different
file for the source or destination of the data transfer. For some subroutines, you
can use a file pointer to a structure that contains the name of the file; for others,
you can use a file descriptor (that is, the positive integer assigned to the file when
it is opened).
The I/O subroutines stored in the C Library (libc.a) provide stream I/O. To access
these stream I/O subroutines, you must include the stdio.h file using the following
statement:
#include <stdio.h>
Some of the I/O library subroutines are macros defined in a header file and some
are object modules of functions. In many cases, the library contains a macro and a
function that do the same type of operation. Consider the following when deciding
whether to use the macro or the function:
v You cannot set a breakpoint for a macro using the dbx program.
v Macros are usually faster than their equivalent functions because the
preprocessor replaces the macros with actual lines of code in the program.
v Macros result in larger object code after being compiled.
v Functions can have side effects to avoid.
The files, commands, and subroutines used in I/O handling provide the following
interfaces:
“Low-Level I/O
Interfaces” on
page 156
“Stream I/O
Interfaces” on
page 157
“Terminal I/O
Interfaces” on
page 157
“Asynchronous I/O
Interfaces” on
page 158
“Creating an Input
Language with the
lex and yacc
Commands” on
page 267
© Copyright IBM Corp. 1997, 1999
Basic open and close functions for files and devices.
Read and write I/O for pipes and FIFOs.
Formatted output and buffering.
Concurrent I/O and processing.
The lex and yacc commands generate a lexical analyzer and a parser
program for interpreting I/O.
155
Low-Level I/O Interfaces
Low-level I/O interfaces are direct entry points into a kernel, providing functions
such as opening files, reading to and writing from files, and closing files.
The line command provides the interface that allows one line from standard input
to be read and the following subroutines provide other low-level I/O functions:
open, openx, or creat
read, readx, readv, or readvx
write, writex, writev, or writevx
close
Prepare a file, or other path object, for
reading and writing by means of an
assigned file descriptor
Read from an open file descriptor
Write to an open file descriptor
Relinquish a file descriptor
The open and creat subroutines set up entries in three system tables. A file
descriptor indexes the first table, which functions as a per process data area that
can be accessed by read and write subroutines. Each entry in this table has a
pointer to a corresponding entry in the second table.
The second table is a per-system data base, or file table, that allows an open file to
be shared among several processes. The entries in this table indicate if the file was
open for reading, writing, or as a pipe, and when the file was closed. There is also
an offset to indicate where the next read or write will take place and a final pointer
to indicates entry to the third table, which contains a copy of the file’s i-node.
The file table contains entries for every instance of an open or create subroutine on
the file, but the i-node table contains only one entry for each file.
Note: While processing an open or creat subroutine for a special file, the
system always calls the device’s open subroutine to allow any special
processing (such as rewinding a tape or turning on a data-terminal-ready
modem lead). However, the system uses the close subroutine only when the
last process closes the file (that is, when the i-node table entry is deallocated).
This means that a device cannot maintain or depend on a count of its users
unless an exclusive-use device (that prevents a device from being reopened
before its closed) is implemented.
When a read or write operation takes place, the user’s arguments and the file table
entry are used to set up the following variables:
v User address of the I/O target area
v Byte-count for the transfer
v Current location in the file
If the file referred to is a character-type special file, the appropriate read or write
subroutine is called to transfer data and update the count and current location.
Otherwise, the current location is used to calculate a logical block number in the
file.
If the file is an ordinary file, the logical block number must be mapped to a
physical block number. A block-type special file need not be mapped. The resulting
physical block number is used to read or write the appropriate device.
Block device drivers can provide the ability to transfer information directly
between the user’s core image and the device in blocks as large as the caller
requests without using buffers. The method involves setting up a character-type
156
General Programming Concepts
special file corresponding to the raw device and providing read and write
subroutines to create a private, non-shared buffer header with the appropriate
information. If desired, separate open and close subroutines can be provided, and a
special-function subroutine can be called for magnetic tape.
Stream I/O Interfaces
Stream I/O interfaces provide data as a stream of bytes that is not interpreted by
the system, which offers more efficient implementation for networking protocols
than character I/O processing. There are no record boundaries when reading and
writing using stream I/O. For example, a process reading 100 bytes from a pipe
cannot tell if the process that wrote the data into the pipe did a single write of 100
bytes, or two writes of 50 bytes, or even if the 100 bytes came from two different
processes.
Stream I/Os can be pipes or FIFOs, first in, first out files. FIFOs are similar to
pipes because they allow the data to flow only one way (left to right). However, a
FIFO can be given a name and can be accessed by unrelated processes, unlike a
pipe. FIFOs are sometimes referred to as named pipes. Because it has a name, a
FIFO can be opened using the standard I/O fopen subroutine. To open a pipe, you
must call the pipe subroutine, which returns a file descriptor, and the standard
I/O fdopen subroutine to associate an open file descriptor with a standard I/O
stream.
Stream I/O interfaces are accessed through the following subroutines and macros:
Closes a stream
Check the status of a stream
Write all currently buffered characters from a
stream
fopen, freopen, or fdopen
Open a stream
fread or fwrite
Perform binary input
fseek, rewind, ftell, fgetpos, or fsetpos
Reposition the file pointer of a stream
getc, fgetc, getchar, or getw
Get a character or word from an input
stream
gets or fgets
Get a string from a stream
getwc, fgetwc, or getwchar
Get a wide character from an input stream
getws or fgetws
Get a string from a stream
printf, fprintf, sprintf, wsprintf, vprintf, vfprintf, vsprintf, or vwsprintf
Print formatted output
putc, putchar, fputc, or putw
Write a character or a word to a stream
puts or fputs
Write a string to a stream
putwc, putwchar, or fputwc
Write a character or a word to a stream
putws or fputws
Write a wide character string to a stream
scanf, fscanf, sscanf, or wsscanf
Convert formatted input
setbuf, setvbuf, setbuffer, or setlinebuf
Assign buffering to a stream
ungetc or ungetwc
Push a character back into the input stream
fclose
feof, ferror, clearerr, or fileno
fflush
Terminal I/O Interfaces
Terminal I/O interfaces operate between a process and the kernel, providing
functions such as buffering and formatted output.
Every terminal and pseudo-terminal has a tty structure that contains the current
terminal group ID. This field identifies the process group to receive the signals
associated with the terminal.
Chapter 7. Input and Output Handling
157
Terminal I/O interfaces are accessed through the iostat command, which monitors
I/O system device loading, and the uprintfd daemon, which allows kernel
messages to be written to the terminal screen.
A daemon opens a terminal device in order to log error messages to the /dev/tty or
/dev/console file. If background writes are not allowed, disassociate the daemon
process from the controlling terminal.
Terminal characteristics can be enabled or disabled through the following
subroutines:
cfgetospeed, cfsetospeed, cfgetispeed, or cfsetispeed
Get and set input and output baud rates
ioctl
Performs control functions associated with
open file descriptors, such as controlling the
ability of background processes to produce
output on the control terminal
termdef
Queries terminal characteristics
tcdrain
Waits for output to complete
tcflow
Performs flow control functions
tcflush
Discards data from the specified queue
tcgetaattr
Gets terminal state
tcgetpgrp
Gets foreground process group ID
tcsendbreak
Sends a break on an asynchronous serial data
line
tcsetattr
Sets terminal state
ttylock, ttywait, ttyunlock, or ttylocked
Control tty locking functions
ttyname or isatty
Get the name of a terminal
ttyslot
Finds the slot in the utmp file for the current
user
Asynchronous I/O Interfaces
Asynchronous I/O subroutines allow a process to start an I/O operation and have
the subroutine return immediately after the operation is started or queued.
Another subroutine is required to wait for the operation to complete (or return
immediately if the operation is already finished). This means that a process can
overlap its execution with its I/O or overlap I/O between different devices.
Although asynchronous I/O does not significantly improve performance for a
process that is reading from a disk file and writing to another disk file,
asynchronous I/O provides significant performance improvements for other types
of I/O driven programs, such as programs that dump a disk to a magnetic tape or
display an image on an image display.
Although not required, a process performing asynchronous I/O can tell the kernel
to notify it when a specified descriptor is ready for I/O (also called signal-driven
I/O). The kernel notifies the user process with the SIGIO signal.
To use asynchronous I/O, a process must perform three steps:
1. Establish a handler for the SIGIO signal. This step is only necessary if
notification by the signal is requested.
2. Set the process ID or the process group ID to receive the SIGIO signals. This
step is only necessary if notification by the signal is requested.
158
General Programming Concepts
3. Enable asynchronous I/O. The system administrator usually determines
whether asynchronous I/O is loaded (enabled). Enabling occurs at system
startup.
The following asynchronous I/O subroutines are provided:
aio_cancel
aio_error
aio_read
aio_return
aio_suspend
aio_write
lio_listio
poll or select
Cancels one or more outstanding asynchronous I/O requests
Retrieves the error status of an asynchronous I/O request
Reads asynchronously from a file descriptor
Retrieves the return status of an asynchronous I/O request
Suspends the calling process until one or more asynchronous I/O
requests is completed
Writes asynchronously to a file descriptor
Initiates a list of asynchronous I/O requests with a single call
Check I/O status of multiple file descriptors and message queues
For use with the poll subroutine, the following header files are supplied:
poll.h
aio.h
Defines the structures and flags used by the poll subroutine
Defines the structure and flags used by the aio_read, aio_write, and aio_suspend
subroutines
Chapter 7. Input and Output Handling
159
160
General Programming Concepts
Chapter 8. Large Program Support
This chapter provides information about using the large address-space model to
accommodate programs requiring data areas that are larger than conventional
segmentation can handle.
Note: The discussion in this chapter only applies to 32-bit processes. For
information about the default 32-bit address space model and the 64-bit address
space model, see “Program Address Space Overview” on page 565 and “System
Memory Allocation Using the malloc Subsystem” on page 576.
The system hardware divides the currently active 32-bit virtual address space into
16 independent segments, each addressed by a separate segment register. The
operating system refers to segment 2 (virtual address 0x20000000) as the process
private segment. This segment contains most of the per-process information,
including user data, user stack, kernel stack, and user block.
Because the system places user data and the user stack within a single segment,
the system limits the maximum amount of stack and data to slightly less than
256MB. This size is adequate for most applications. The kernel stack and u-block
are relatively small and of fixed size. However, certain applications require large
initialized or uninitialized data areas in the data section of a program. Other large
data areas can be created dynamically with the malloc, brk or sbrk subroutine.
Some programs need larger data areas than allowed by the default address-space
model. Programs that need the larger data areas can use the large address-space
model to request the necessary amount of data space.
Understanding the Large Address-Space Model
The large address-space model enables large data applications while allowing
programs that use a smaller space to follow the smaller model. To allow a program
to use the large address-space model, you must set the o_maxdata field in the
XCOFF header of the program to indicate the amount of data needed.
In the large address-space model, the data in the program is laid out beginning in
segment 3 when the value is non-zero. (The data is laid out beginning in segment
3, even if the value is smaller than a segment size.) The program consumes as
many segments as needed to hold the amount of data indicated by the o_maxdata
field, up to a maximum of 8 segments. The program can therefore have up to 2
gigabytes of data.
Other aspects of the program address space remain unchanged. The user stack,
kernel stack, and u-block continue to reside in segment 2. Also, the data resulting
from loading a private copy of a shared library is placed in segment 2. Only
program data is placed in segment 3 or higher.
As a result of this organizational scheme, the user stack is still limited by the size
of segment 2. (However, the user stack can be relocated into a shared memory
segment.) In addition, fewer segments are available for mapped files.
While the size of initialized data in a program can be large, there is still a
restriction on the size and placement of text. In the executable file associated with
© Copyright IBM Corp. 1997, 1999
161
a program, the offset of the end of the text section plus the size of the loader
section must be less than 256MB. This is required so that this read-only portion of
the executable will fit into segment 1 (the TEXT segment). Because of these
restrictions, a program cannot have a very large text section.
Enabling the Large Address-Space Model
The large address space model is used if any nonzero value is given for the
maxdata keyword. Use the -bmaxdata option only if the program needs very large
data areas.
Use the -bmaxdata flag with the ld command to enable the large address-space
model.
For example, to link a program that will have the maximum 8 segments reserved
to it, the following command line could be used:
cc sample.o -bmaxdata:0x80000000
The number 0x80000000 is the number of bytes, in hexadecimal format, equal to
eight 256MB segments. Although larger numbers can be used, they are ignored
because a maximum of 8 segments can be reserved. The value following the
-bmaxdata flag can also be specified in decimal or octal format.
Using the following shell commands, you can patch large programs to use large
data without relinking them:
/usr/bin/echo '\0200\0\0\0'|dd of=executable_file_name bs=4
count=1 seek=19 conv=notrunc
Note: Use the full name of the echo command (/usr/bin/echo) to avoid
invoking any of the shell echo subcommands by mistake.
The echo string generates the binary value 0x80000000. This dd command seeks to
the proper offset in the executable file and modifies the o_maxdata field. Do not use
the dd command on nonexecutable object files, loadable modules, or shared
libraries.
Executing Programs with Large Data Areas
When a program attempts to execute a program with large data areas, the system
recognizes the requirement for large data and attempts to modify the soft limit on
data size to accommodate that requirement. However, if it does not have
permission to modify the soft limit, the program ends.
In addition, it is also possible that the data size specified in the o_maxdata field
may be too small to accommodate the amount of space required for initialized or
uninitialized data. In this case, the process ends, and an error is reported.
The attempt is also unsuccessful if the new soft limit is above the hard limit for the
process. For example, the login process usually sets the hard limit to infinity.
However, if the calling process has modified its hard limit using either the ulimit
command in the Bourne shell or the limit command in the C shell, the newly
modified soft limit may be above the hard limit for the process. In this case, the
process will be killed during exec processing. In this situation, the only message
you receive is killed, which informs you that the process was killed.
162
General Programming Concepts
For more information on the ulimit command in the Bourne shell, see ″Bourne
Shell Special Commands″ in AIX Version 4.3 System User’s Guide: Operating System
and Devices. For more information about the limit command in the C shell, see
″Command Substitution in the C Shell″ and ″Filename Substitution in the C Shell″
in AIX Version 4.3 System User’s Guide: Operating System and Devices.
After placing the program’s initialized and uninitialized data in segments 3 and
beyond, the system computes the break value. The break value defines the end of
the process’s static data and the beginning of its dynamically allocatable data.
Using the malloc, brk or sbrk subroutine, the process is free to move the break
value toward the end of the segment identified by the maxdata field in the a.out
header file.
For example, if the value specified in the maxdata field in the a.out header file is
0x80000000, then the maximum break value is up to the end of segment 10 or
0xafffffff. The brk subroutine extends the break across segment boundaries, but
not beyond the point specified in the maxdata field.
The majority of subroutines are unaffected by large data programs. The semantics
of the fork subroutine remain unchanged. Large data programs can run other large
or small programs, as well as load and unload other modules.
The setrlimit subroutine allows the soft data limit to be set to any value that does
not exceed the hard limit. However, because of the inherent limitation of the
address space model used by the process, it may not be able to increase its size to
the value that is set.
Special Considerations
Programs with large data spaces require a large amount of paging space. For
example, if a program with a 2-gigabyte address space tries to access every page in
its address space, the system must have 2 gigabytes of paging space. The operating
system page-space monitor terminates processes when paging space runs low.
Programs with large data spaces are terminated first because they typically
consume a large amount of paging space.
Debugging programs with large data is similar to debugging other programs. The
dbx command can debug these large programs actively or from a core dump. A
full core dump should not be performed because programs with large data areas
produce large core dumps, which consume large amounts of file-system space.
Some application programs may be written in such a way that they rely on
characteristics of the address space model. Programs in which the large address
space is enabled use a different address space model than programs without the
large address space enabled. This could cause problems for applications which
make assumptions about the address space model they are running in. In general,
avoid application programs that make assumptions about the address space model.
Chapter 8. Large Program Support
163
164
General Programming Concepts
Chapter 9. Parallel Programming
Parallel programming should be used to get benefits of new multiprocessor
systems, while maintaining a full binary compatibility with existing monoprocessor
systems. The parallel programming facilities are based on a new concept of the
operating system: threads. The following information introduces threads and the
associated programming facilities. It also discusses general topics concerning
parallel programming.
Understanding Threads
A thread is an independent flow of control that operates within the same address
space as other independent flows of controls within a process. In previous versions
of AIX, and in most of UNIX systems, thread and process characteristics are
grouped into a single entity called a process. In other operating systems, threads
are sometimes called ″lightweight processes,″ or the meaning of the word ″thread″
is sometimes slightly different.
In the following pages, we will learn the differences between a thread and a
process, and see what ″thread″ really means in AIX.
Read the following to learn more about threads in AIX:
Threads and Processes
In traditional single-threaded process systems, a process has a set of properties. In
multi-threaded systems, these properties are divided between processes (“Process
Properties”) and threads (“Thread Properties”).
Process Properties
A process in a multi-threaded system is the changeable entity. It must be
considered as an execution frame. It has all traditional process attributes, such as:
v Process ID, process group ID, user ID, and group ID
v Environment
v Working directory.
A process also provides a common address space and common system resources:
v File descriptors
v Signal actions
v Shared libraries
v Inter-process communication tools (such as message queues, pipes, semaphores,
or shared memory).
Thread Properties
A thread is the schedulable entity. It has only those properties that are required to
ensure its independent flow of control. These include the following properties:
v Stack
v Scheduling properties (such as policy or priority)
v Set of pending and blocked signals
v Some thread-specific data.
© Copyright IBM Corp. 1997, 1999
165
An example of thread-specific data is the error indicator, errno. In multi-threaded
systems, errno is no longer a global variable, but usually a subroutine returning a
thread-specific errno value. Some other systems may provide other
implementations of errno.
Threads within a process must not be considered as a group of processes. All
threads share the same address space. This means that two pointers having the
same value in two threads refer to the same data. Also, if any thread changes one
of the shared system resources, all threads within the process are affected. For
example, if a thread closes a file, the file is closed for all threads.
The Initial Thread
When a process is created, one thread is automatically created. This thread is called
the initial thread. It ensures the compatibility between the old processes with a
unique implicit thread and the new multi-threaded processes. The initial thread
has some special properties, not visible to the programmer, that ensure binary
compatibility between the old single-threaded programs and the multi-threaded
operating system. It is also the initial thread that executes the main routine in
multi-threaded programs.
Threads Implementation
A thread is the schedulable entity, which means that the system scheduler handles
threads. These threads, known by the system scheduler, are strongly
implementation-dependent. To facilitate the writing of portable programs, libraries
provide another kind of thread.
Kernel Threads and User Threads
A kernel thread is a kernel entity, like processes and interrupt handlers; it is the
entity handled by the system scheduler. A kernel thread runs within a process, but
can be referenced by any other thread in the system. The programmer has no
direct control over these threads, unless writing kernel extensions or device
drivers. See AIX Version 4.3 Kernel Extensions and Device Support Programming
Concepts for more information about kernel programming.
A user thread is an entity used by programmers to handle multiple flows of controls
within a program. The API for handling user threads is provided by a library, the
threads library. A user thread only exists within a process; a user thread in process
A cannot reference a user thread in process B. The library uses a proprietary
interface to handle kernel threads for executing user threads. The user threads API,
unlike the kernel threads interface, is part of a portable programming model. Thus,
a multi-threaded program developed on an AIX system can easily be ported to
other systems.
On other systems, user threads are simply called threads, and lightweight process
refers to kernel threads.
Thread Models and Virtual Processors
User threads are mapped to kernel threads by the threads library. The way this
mapping is done is called the thread model. There are three possible thread models,
corresponding to three different ways to map user threads to kernel threads.
166
General Programming Concepts
v M:1 model
v 1:1 model
v M:N model.
The mapping of user threads to kernel threads is done using virtual processors. A
virtual processor (VP) is a library entity that is usually implicit. For a user thread,
the virtual processor behaves as a CPU for a kernel thread. In the library, the
virtual processor is a kernel thread or a structure bound to a kernel thread.
In the M:1 model all user threads are mapped to one kernel thread; all user threads
run on one VP. The mapping is handled by a library scheduler. All user threads
programming facilities are completely handled by the library. This model can be
used on any system, especially on traditional single-threaded systems. The M:1
Threads Model figure illustrates this model.
User Threads
Library scheduler
Threads Library
VP
Kernel Threads
M:1 Threads Model
In the 1:1 model, each user thread is mapped to one kernel thread; each user
thread runs on one VP. Most of the user threads programming facilities are directly
handled by the kernel threads. The 1:1 Threads Model figure illustrates this model.
User Threads
VP
VP
VP
Threads Library
Kernel Threads
1:1 Threads Model
In the M:N model, all user threads are mapped to a pool of kernel threads; all user
threads run on a pool of virtual processors. A user thread may be bound to a
Chapter 9. Parallel Programming
167
specific VP, as in the 1:1 model. All unbound user threads share the remaining VPs.
This is the most efficient and most complex thread model; the user threads
programming facilities are shared between the threads library and the kernel
threads. The M:N Threads Model figure illustrates this model.
User Threads
Library scheduler
VP
VP
VP
Threads Library
Kernel Threads
M:N Threads Model
Contention Scope and Concurrency Level
The contention scope of a user thread defines how it is mapped to a kernel thread.
There are two possible contention scopes:
v System contention scope, sometimes called global contention scope
A system contention scope user thread is a user thread that is directly mapped
to one kernel thread. All user threads in a 1:1 thread model have system
contention scope.
v Process contention scope, sometimes called local contention scope.
A process contention scope user thread is a user thread that shares a kernel
thread with other (process contention scope) user threads in the process. All user
threads in a M:1 thread model have process contention scope.
In an M:N thread model, user threads can have either system or process contention
scope. In the previous figure, for example, the user thread on the left side has
system contention scope; the other ones all have process contention scope.
Therefore, an M:N thread model is often referred as a mixed-scope model.
The concurrency level is a property of M:N threads libraries. It defines the number
of VPs used to run the process contention scope user threads. This number cannot
exceed the number of process contention scope user threads, and is usually
dynamically set by the threads library. The system also sets a limit to the number
of available kernel threads.
libpthreads.a POSIX Threads Library
AIX provides a threads library, called libpthreads.a, based on the POSIX 1003.1c
industry standard for a portable user threads API. Any program written for use
with a POSIX thread library can easily be ported for use with another POSIX
threads library; only the performance and very few subroutines of the threads
library are implementation-dependent. For this reason, multi-threaded programs
written for this version of AIX will work on any future version of AIX.
168
General Programming Concepts
To enhance the portability of the threads library, the POSIX standard made the
implementation of several programming facilities optional. See Threads Library
Options for more information about checking the POSIX options.
libpthreads_compat.a POSIX Draft 7 Threads Library
AIX provides binary compatibility for existing multi-threads applications that were
coded to Draft 7 of the POSIX thread standard. These applications will run without
re-linking.
The libpthreads_compat.a library is actually provided for program development.
AIX Version 4.3 provides program support for both Draft 7 of the POSIX Thread
Standard and Xopen Version 5 Standard, which includes the final POSIX 1003.1c
Pthread Standard.
See “Developing Multi-Threaded Programs” on page 179 for more information.
Thread Programming Concepts
The following information provides an overview of the threads library and
introduces major programming concepts for multi-threaded programming. Unless
otherwise specified, the threads library always operates within a single process.
Basic Operations
Basic thread operations include thread creation (see “Thread Creation”) and
termination (see “Thread Termination”).
Thread Creation:
Thread creation differs from process creation in that no parent-child relation exists
between threads. All threads, except the initial thread automatically created when a
process is created, are on the same hierarchical level. A thread does not maintain a
list of created threads, nor does it know the thread that created it.
When creating a thread, an entry-point routine and an argument must be specified.
Every thread has an entry-point routine with one argument. The same entry-point
routine may be used by several threads. See “Creating Threads” on page 208 for
more information about thread creation.
Thread Termination:
Threads can terminate themselves by either returning from their entry-point
routine or calling a library subroutine. Threads can also terminate other threads,
using a mechanism called cancellation. Any thread can request the cancellation of
another thread. Each thread controls whether it may be canceled or not. Cleanup
handlers may also be registered to perform operations when a cancellation request
is acted upon. See “Terminating Threads” on page 212 for more information about
thread termination.
Synchronization
Threads need to synchronize their activities to effectively interact. This includes:
v Implicit communication through the modification of shared data
v Explicit communication by informing each other of events that have occurred.
The threads library provides three synchronization mechanisms: mutexes, condition
variables, and joins. These are primitive but powerful mechanisms, which can be
used to build more complex mechanisms.
Chapter 9. Parallel Programming
169
Mutexes and Race Conditions:
Mutual exclusion locks (mutexes) can prevent data inconsistencies due to race
conditions. A race condition often occurs when two or more threads need to
perform operations on the same memory area, but the results of computations
depends on the order in which these operations are performed.
Consider, for example, a single counter, X, that is incremented by two threads, A
and B. If X is originally 1, then by the time threads A and B increment the counter,
X should be 3. Both threads are independent entities and have no synchronization
between them. Although the C statement X++ looks simple enough to be atomic,
the generated assembly code may not be, as shown in the following
pseudo-assembler code:
move
inc
move
X, REG
REG
REG, X
If both threads are executed concurrently on two CPUs, or if the scheduling makes
the threads alternatively execute on each instruction, the following steps may
occur:
1. Thread A executes the first instruction and puts X, which is 1, into the thread A
register. Then thread B executes and puts X, which is 1, into the thread B
register. The following figure illustrates the resulting registers and the contents
of memory X.
Thread A Register
1
Thread B Register
1
Memory X
1
2. Next, thread A executes the second instruction and increments the content of its
register to 2. Then thread B increments its register to 2. Nothing is moved to
memory X, so memory X stays the same. The following figure illustrates the
resulting registers and the contents of memory X.
Thread A Register
2
Thread B Register
2
Memory X
1
3. Last, thread A moves the content of its register, which is now 2, into memory X.
Then thread B moves the content of its register, which is also 2, into memory X,
overwriting thread A’s value. The following figure illustrates the resulting
registers and the contents of memory X.
170
General Programming Concepts
Thread A Register
2
Thread B Register
2
Memory X
2
Note that in most cases thread A and thread B will execute the three instructions
one after the other, and the result would be 3, as expected. Race conditions are
usually difficult to discover, because they occur intermittently.
To avoid this race condition, each thread should lock the data before accessing the
counter and updating memory X. For example, if thread A takes a lock and
updates the counter, it leaves memory X with a value of 2. Once thread A releases
the lock, thread B takes the lock and updates the counter, taking 2 as its initial
value for X and incrementing it to 3, the expected result.
See “Using Mutexes” on page 220 for more information about mutexes.
Waiting for Threads:
Condition variables allow threads to block until some event or condition has
occurred. Boolean predicates indicate whether the program has satisfied a
condition variable. The complexity of a condition variable predicate is defined by
the programmer. A condition can be signaled by any thread to either one or all
waiting threads. See “Using Condition Variables” on page 224 to get more
information.
When a thread is terminated, its storage may not be reclaimed, depending on an
attribute of the thread. Such threads can be joined by other threads and return
information to them. A thread that wants to join another thread is blocked until the
target thread terminates. This joint mechanism is a specific case of
condition-variable usage, the condition is the thread termination. See “Joining
Threads” on page 230 for more information about joins.
Scheduling
The threads library allows the programmer to control the execution scheduling of
the threads. The control is performed in different ways:
v By setting scheduling attributes when creating a thread
v By dynamically changing the scheduling attributes of a created thread
v By defining the effect of a mutex on the thread’s scheduling when creating a
mutex
v By dynamically changing the scheduling of a thread during synchronization
operations.
The two last types of controls are known as synchronization scheduling.
Scheduling Parameters:
A thread has three scheduling parameters:
Scope
Policy
The contention scope of a thread is defined by the thread model used in the
threads library.
The scheduling policy of a thread defines how the scheduler treats the thread
once it gains control of the CPU.
Chapter 9. Parallel Programming
171
Priority
The scheduling priority of a thread defines the relative importance of the work
being done by each thread.
The scheduling parameters can be set before the thread’s creation or during the
thread’s execution. In general, controlling the scheduling parameters of threads is
important only for threads that are compute-intensive. Thus the threads library
provides default values that are sufficient for most cases. See “Threads Scheduling”
on page 235 for more information about controlling the scheduling parameters of
threads.
Synchronization Scheduling:
Synchronization scheduling is a complex topic. Some implementations of the
threads library do not provide this facility.
Synchronization scheduling defines how the execution scheduling, especially the
priority, of a thread is modified by holding a mutex. This allows custom-defined
behavior and avoids priority inversions. It is useful when using complex locking
schemes. See “Synchronization Scheduling” on page 239 for more information.
Other Facilities
The threads library provides other useful facilities to help programmers implement
powerful functions. It also manages the interactions between threads and
processes.
Advanced Facilities: The threads library provides an API for handling
synchronization and scheduling of threads. It also provides facilities for the
following purposes:
v One-time initializations allow dynamic package initializations.
v Thread-specific data allows each thread to maintain its own private data.
v Advanced thread attributes allow control of the size and the address of the
thread’s stack.
Threads-Processes Interactions: Threads and processes interact when handling
specific actions:
v Signal management are shared between the process and its threads.
v Process duplication and destruction imply thread creation and termination.
Threads Library API
This section provides some general comments about the threads library API. The
following information is not required for writing multi-threaded programs, but
may help the programmer understand the threads library API.
Object-Oriented Interface: The threads library API provides an object-oriented
interface. The programmer manipulates opaque objects using pointers or other
universal identifiers. This ensures the portability of multi-threaded programs
between systems that implement the threads library. It also allows implementation
changes between two releases of AIX that necessitate only programs to be
re-compiled. Although some definitions of data types may be found in the threads
library header file (pthread.h), programs should not rely on these
implementation-dependent definitions to directly handle the contents of structures.
The regular threads library subroutines must always be used to manipulate the
objects.
172
General Programming Concepts
The threads library essentially uses three kinds of objects (opaque data types):
threads, mutexes, and condition variables. These objects have attributes which
specify the object properties. When creating an object, the attributes must be
specified. In the threads library, these creation attributes are themselves objects,
called attributes objects.
Therefore, there are three pairs of objects manipulated by the threads library:
v Threads and thread attributes objects
v Mutexes and mutex attributes objects
v Condition variables and condition attributes objects.
Creating an object requires the creation of an attributes object. An attributes object
is created with attributes having default values. Attributes can then be individually
modified using subroutines. This ensures that a multi-threaded program will not
be affected by the introduction of new attributes or changes in the implementation
of an attribute. An attributes object can thus be used to create one or several
objects, and then destroyed without affecting objects created with the attributes
object.
Using an attributes object also allows the use of object classes. One attributes object
may be defined for each object class. Creating an instance of an object class would
be done by creating the object using the class attributes object.
Naming Convention: The identifiers used by the threads library follow a strict
naming convention. All identifiers of the threads library begin with pthread_. User
programs should not use this prefix for private identifiers. This prefix is followed
by a component name. The following components are defined in the threads
library:
pthread_
pthread_attr
pthread_cond
pthread_condattr
pthread_key
pthread_mutex
pthread_mutexattr
Threads themselves and miscellaneous subroutines
Thread attributes objects
Condition variables
Condition attributes objects
Thread-specific data keys
Mutexes
Mutex attributes objects.
Data types identifiers end with _t. Subroutines and macros end with an _
(underscore), followed by a name identifying the action performed by the
subroutine or the macro. For example, pthread_attr_init is a threads library
identifier (pthread_) concerning thread attributes objects (attr) and is an
initialization subroutine (_init).
Explicit macro identifiers are in uppercase letters. Some subroutines may, however,
be implemented as macros, although their names are in lowercase letters.
Related Files: The following AIX files provide the AIX implementation of
pthreads:
/usr/include/pthread.h
/usr/include/sched.h
/usr/include/unistd.h
C/C++ header with most
pthread definitions.
C/C++ header with some
scheduling definitions.
C/C++ header with
pthread_atfork() definition.
Chapter 9. Parallel Programming
173
/usr/include/sys/limits.h
/usr/include/sys/pthdebug.h
/usr/include/sys/sched.h
/usr/include/sys/signal.h
/usr/include/sys/types.h
/usr/lib/libpthreads.a
/usr/lib/libpthreads_compat.a
/usr/lib/profiled/libpthreads.a
/usr/lib/profiled/libpthreads_compat.a
C/C++ header with some
pthread definitions.
C/C++ header with most
pthread debug definitions.
C/C++ header with some
scheduling definitions.
C/C++ header with
pthread_kill() and
pthread_sigmask() definitions.
C/C++ header with some
pthread definitions.
32-bit/64-bit library providing
UNIX98 and POSIX 1003.1c
pthreads.
32-bit only library providing
POSIX 1003.1c Draft 7 pthreads.
Profiled 32-bit/64-bit library
providing UNIX98 and POSIX
1003.1c pthreads.
Profiled 32-bit only library
providing POSIX 1003.1c Draft 7
pthreads.
Writing Reentrant and Thread-Safe Code
In single-threaded processes there is only one flow of control. The code executed
by these processes thus need not to be reentrant or thread-safe. In multi-threaded
programs, the same functions and the same resources may be accessed
concurrently by several flows of control. To protect resource integrity, code written
for multi-threaded programs must be reentrant and thread-safe.
This section provides information for writing reentrant and thread-safe programs.
It does not cover the topic of writing thread-efficient programs. Thread-efficient
programs are efficiently parallelized programs. This can only be done during the
design of the program. Existing single-threaded programs can be made
thread-efficient, but this requires that they be completely redesigned and rewritten.
Understanding Reentrance and Thread-Safety
Reentrance and thread-safety are both related to the way functions handle
resources. Reentrance and thread-safety are separate concepts: a function can be
either reentrant, thread-safe, both, or neither.
Reentrance: A reentrant function does not hold static data over successive calls,
nor does it return a pointer to static data. All data is provided by the caller of the
function. A reentrant function must not call non-reentrant functions.
A non-reentrant function can often, but not always, be identified by its external
interface and its usage. For example, the strtok subroutine is not reentrant, because
it holds the string to be broken into tokens. The ctime subroutine is also not
reentrant; it returns a pointer to static data that is overwritten by each call.
Thread-Safety: A thread-safe function protects shared resources from concurrent
access by locks. Thread-safety concerns only the implementation of a function and
does not affect its external interface.
174
General Programming Concepts
In C, local variables are dynamically allocated on the stack. Therefore, any function
that does not use static data or other shared resources is trivially thread-safe. For
example, the following function is thread-safe:
/* thread-safe function */
int diff(int x, int y)
{
int delta;
delta = y - x;
if (delta < 0)
delta = -delta;
return delta;
}
The use of global data is thread-unsafe. It should be maintained per thread or
encapsulated, so that its access can be serialized. A thread may read an error code
corresponding to an error caused by another thread. In AIX, each thread has its
own errno value.
Making a Function Reentrant
In most cases, non-reentrant functions must be replaced by functions with a
modified interface to be reentrant. Non-reentrant functions cannot be used by
multiple threads. Furthermore, it may be impossible to make a non-reentrant
function thread-safe.
Returning Data: Many non-reentrant functions return a pointer to static data.
This can be avoided in two ways:
v Returning dynamically allocated data. In this case, it will be the caller’s
responsibility to free the storage. The benefit is that the interface does not need
to be modified. However, backward compatibility is not ensured; existing
single-threaded programs using the modified functions without changes would
not free the storage, leading to memory leaks.
v Using caller-provided storage. This method is recommended, although the
interface needs to be modified.
For example, a strtoupper function, converting a string to uppercase, could be
implemented as in the following code fragment:
/* non-reentrant function */
char *strtoupper(char *string)
{
static char buffer[MAX_STRING_SIZE];
int index;
for (index = 0; string[index]; index++)
buffer[index] = toupper(string[index]);
buffer[index] = 0
return buffer;
}
This function is not reentrant (nor thread-safe). Using the first method to make the
function reentrant, the function would be similar to the following code fragment:
/* reentrant function (a poor solution) */
char *strtoupper(char *string)
{
char *buffer;
int index;
/* error-checking should be performed! */
buffer = malloc(MAX_STRING_SIZE);
for (index = 0; string[index]; index++)
Chapter 9. Parallel Programming
175
buffer[index] = toupper(string[index]);
buffer[index] = 0
return buffer;
}
A better solution consists of modifying the interface. The caller must provide the
storage for both input and output strings, as in the following code fragment:
/* reentrant function (a better solution) */
char *strtoupper_r(char *in_str, char *out_str)
{
int index;
for (index = 0; in_str[index]; index++)
out_str[index] = toupper(in_str[index]);
out_str[index] = 0
return out_str;
}
The non-reentrant standard C library subroutines were made reentrant using the
second method. This is discussed in “Reentrant and Thread-Safe Libraries” on
page 178.
Keeping Data over Successive Calls: No data should be kept over successive
calls, because different threads may successively call the function. If a function
needs to maintain some data over successive calls, such as a working buffer or a
pointer, this data should be provided by the caller.
Consider the following example. A function returns the successive lowercase
characters of a string. The string is provided only on the first call, as with the
strtok subroutine. The function returns 0 when it reaches the end of the string. The
function could be implemented as in the following code fragment:
/* non-reentrant function */
char lowercase_c(char *string)
{
static char *buffer;
static int index;
char c = 0;
/* stores the string on first call */
if (string != NULL) {
buffer = string;
index = 0;
}
/* searches a lowercase character */
for (; c = buffer[index]; index++) {
if (islower(c)) {
index++;
break;
}
}
return c;
}
This function is not reentrant. To make it reentrant, the static data, the index
variable, needs to be maintained by the caller. The reentrant version of the function
could be implemented as in the following code fragment:
/* reentrant function */
char reentrant_lowercase_c(char *string, int *p_index)
{
char c = 0;
/* no initialization - the caller should have done it */
/* searches a lowercase character */
for (; c = string[*p_index]; (*p_index)++) {
if (islower(c)) {
176
General Programming Concepts
(*p_index)++;
break;
}
}
return c;
}
The interface of the function changed and so did its usage. The caller must provide
the string on each call and must initialize the index to 0 before the first call, as in
the following code fragment:
char *my_string;
char my_char;
int my_index;
...
my_index = 0;
while (my_char = reentrant_lowercase_c(my_string, &my_index)) {
...
}
Making a Function Thread-Safe
In multi-threaded programs, all functions called by multiple threads must be
thread-safe. However, there is a workaround for using thread unsafe subroutines in
multi-threaded programs. Note also that non-reentrant functions usually are
thread-unsafe, but making them reentrant often makes them thread-safe, too.
Locking Shared Resources: Functions that use static data or any other shared
resources, such as files or terminals, must serialize the access to these resources by
locks in order to be thread-safe. For example, the following function is
thread-unsafe:
/* thread-unsafe function */
int increment_counter()
{
static int counter = 0;
counter++;
return counter;
}
To be thread-safe, the static variable counter needs to be protected by a static lock,
as in the following (pseudo-code) example:
/* pseudo-code thread-safe function */
int increment_counter();
{
static int counter = 0;
static lock_type counter_lock = LOCK_INITIALIZER;
lock(counter_lock);
counter++;
unlock(counter_lock);
return counter;
}
In a multi-threaded application program using the threads library, mutexes should
be used for serializing shared resources. Independent libraries may need to work
outside the context of threads and, thus, use other kinds of locks.
A Workaround for Thread-Unsafe Functions: It is possible to use thread-unsafe
functions called by multiple threads using a workaround. This may be useful,
especially when using a thread-unsafe library in a multi-threaded program, for
testing or while waiting for a thread-safe version of the library to be available. The
workaround leads to some overhead, because it consists of serializing the entire
function or even a group of functions.
Chapter 9. Parallel Programming
177
v Use a global lock for the library, and lock it each time you use the library
(calling a library routine or using a library global variable), as in the following
pseudo-code fragments:
/* this is pseudo-code! */
lock(library_lock);
library_call();
unlock(library_lock);
lock(library_lock);
x = library_var;
unlock(library_lock);
This solution can create performance bottlenecks because only one thread can
access any part of the library at any given time. The solution is acceptable only
if the library is seldom accessed, or as an initial, quickly implemented
workaround.
v Use a lock for each library component (routine or global variable) or group of
components, as in the following pseudo-code fragments:
/* this is pseudo-code! */
lock(library_moduleA_lock);
library_moduleA_call();
unlock(library_moduleA_lock);
lock(library_moduleB_lock);
x = library_moduleB_var;
unlock(library_moduleB_lock);
This solution is somewhat more complicated to implement than the first one, but
it can improve performance.
Because this workaround should only be used in application programs and not in
libraries, mutexes can be used for locking the library.
Reentrant and Thread-Safe Libraries
Reentrant and thread-safe libraries are useful in a wide range of parallel (and
asynchronous) programming environments, not just within threads. Thus it is a
good programming practice to always use and write reentrant and thread-safe
functions.
Using Libraries: Several libraries shipped with the AIX Base Operating System
are thread-safe. In the current version of AIX, the following libraries are
thread-safe:
v Standard C library (libc.a)
v Berkeley compatibility library (libbsd.a).
Some of the standard C subroutines are non-reentrant, such as the ctime and strtok
subroutines. The reentrant version of the subroutines have the name of the original
subroutine with a suffix _r (underscore r).
When writing multi-threaded programs, the reentrant versions of subroutines
should be used instead of the original version. For example, the following code
fragment:
token[0] = strtok(string, separators);
i = 0;
do {
i++;
token[i] = strtok(NULL, separators);
} while (token[i] != NULL);
should be replaced in a multi-threaded program by the following code fragment:
178
General Programming Concepts
char *pointer;
...
token[0] = strtok_r(string, separators, &pointer);
i = 0;
do {
i++;
token[i] = strtok_r(NULL, separators, &pointer);
} while (token[i] != NULL);
Thread-unsafe libraries may be used by only one thread in a program. The
uniqueness of the thread using the library must be ensured by the programmer;
otherwise, the program will have unexpected behavior, or may even crash.
Converting Libraries: This information highlights the main steps in converting an
existing library to a reentrant and thread-safe library. It applies only to C language
libraries.
v Identifying exported global variables. Those variables are usually defined in a
header file with the export keyword.
Exported global variables should be encapsulated. The variable should be made
private (defined with the static keyword in the library source code). Access (read
and write) subroutines should be created.
v Identifying static variables and other shared resources. Static variables are
usually defined with the static keyword.
Locks should be associated with any shared resource. The granularity of the
locking, thus choosing the number of locks, impacts the performance of the
library. To initialize the locks, the one-time initialization facility may be used. See
“One-Time Initializations” on page 243.
v Identifying non-reentrant functions and making them reentrant. See “Making a
Function Reentrant” on page 175.
v Identifying thread-unsafe functions and making them thread-safe. See “Making a
Function Thread-Safe” on page 177.
Developing Multi-Threaded Programs
Developing multi-threaded programs is not much more complicated than
developing programs with multiple processes. See “Chapter 11. Threads
Programming Guidelines” on page 207 for detailed information about using the
threads library. Developing programs also implies compiling and debugging the
code.
Compiling a Multi-Threaded Program
This section explains how to generate a multi-threaded program. It describes:
v The required “Header File”
v How to invoke the compiler to generate multi-threaded programs. See
“Compiler Invocation” on page 180.
Header File: All subroutine prototypes, macros, and other definitions for using
the threads library are in one header file, pthread.h, located in the /usr/include
directory.
The pthread.h header file must be the first included file of each source file using
the threads library, because it defines some important macros that affect other
header files. Having the pthread.h header file as the first included file ensures the
usage of thread-safe subroutines. The following global symbols are defined in the
pthread.h file:
Chapter 9. Parallel Programming
179
_POSIX_REENTRANT_FUNCTIONS
_POSIX_THREADS
Specifies that all functions should be reentrant.
Several header files use this symbol to define
supplementary reentrant subroutines, such as the
localtime_r subroutine.
Denotes the POSIX threads API. This symbol is
used to check if the POSIX threads API is available.
Macros or subroutines may be defined in different
ways, depending on whether the POSIX or some
other threads API is used.
The pthread.h file also redefines the errno global variable as a function returning a
thread-specific errno value. The errno identifier is, therefore, no longer an l-value
in a multi-threaded program.
Compiler Invocation: When compiling a multi-threaded program, you should
invoke the C compiler using one of the following commands:
xlc_r
cc_r
Invokes the compiler with default language level of ansi.
Invokes the compiler with default language level of extended.
These commands ensure that the adequate options and libraries are used to be
compliant with the X/Open Version 5 Standard. The POSIX Threads Specification
1003.1c is a subset of the X/Open Specification.
The following libraries are automatically linked with your program when using
these commands:
libpthreads.a
libc.a
Threads library.
Standard C library
For example, the following command compiles the foo.c multi-threaded C source
file and produces the foo executable file:
cc_r -o foo foo.c
Compiler Invocation for Draft 7 of POSIX 1003.1c:
AIX provides source code compatibility for Draft 7 applications. It is recommended
that developers port their threaded application to the latest standard, which is
covered by the compiler directions provided above.
When compiling a multi-threaded program for Draft 7 support of threads, you
should invoke the C compiler using one of the following commands:
xlc_r7
cc_r7
Invokes the compiler with default language level of ansi.
Invokes the compiler with default language level of extended.
The following libraries are automatically linked with your program when using
these commands:
libpthreads_compat.a
libpthreads.a
libc.a
Draft 7 Compatibility Threads library.
Threads library.
Standard C library.
Source code compatibility has been achieved through the use of the compiler
directive _AIX_PTHREADS_D7. It is also necessary to link the libraries in the
following order: libpthreads_compat.a, libpthreads.a, and libc.a. Most users do
180
General Programming Concepts
not need to know this information, since the commands listed above provide the
necessary options. These options are provided for those that don’t have the latest
AIX compiler.
Porting Draft 7 applications to the X/Open Version 5 Standard: There are very
few differences between Draft 7 and the final standard.
There are some minor errno differences. The most prevalent is the use of ESRCH
to denote the specified pthread could not be found. Draft 7 frequently returned
EINVAL for this failure.
Pthreads are joinable by default. This is a significant change since it can result in a
memory leak if ignored. See “Creating Threads” on page 208 for more information
about thread creation.
Pthreads have process scheduling scope by default. See “Threads Scheduling” on
page 235 for more information about scheduling.
The subroutine pthread_yield has been replaced by sched_yield.
The various scheduling policies associated with the mutex locks are slightly
different.
Memory Requirements of a Multi-Threaded Program
AIX supports up to 32768 threads in a single process. Each individual pthread
requires some amount of process address space so the actual maximum number of
pthreads a process can have depends on the memory model and the use of process
address space for other purposes. The amount of memory a pthread needs includes
the stack size and the guard region size plus some amount for internal use. The
user can control the size of the stack with pthread_attr_setstacksize() and the size
of the guard region with pthread_attr_setguardsize(). The following table points
out the maximum number of pthreads which could be created in a 32-bit process
using a simple program which does nothing other than create pthreads in a loop
using the NULL pthread attribute. In a real program the actual numbers will
depend on other memory usage in the program. For a 64-bit process the ulimit
controls how many threads can be created therefore the big data model is not
necessary and in fact can decrease the maximum number of threads.
32-bit Process:
Data Model
-bmaxdata:
Maximum Pthreads
Small Data
n/a
1084
Big Data
0x10000000
2169
Big Data
0x20000000
4340
Big Data
0x30000000
6510
Big Data
0x40000000
8681
Big Data
0x50000000
10852
Big Data
0x60000000
13022
Big Data
0x70000000
15193
Big Data
0x80000000
17364
Debugging a Multi-Threaded Program
This section provides an introduction to debugging multi-threaded programs.
Chapter 9. Parallel Programming
181
Using dbx: Application programmers can use the dbx program to perform
debugging. Several new subcommands are available for displaying thread-related
objects: attribute, condition, mutex, and thread.
Using the Kernel Debug Program: Kernel programmers can use the kernel debug
program to perform debugging on kernel extensions and device drivers. The kernel
debug program provides no access to user threads but handles kernel threads.
Several new commands have been added to support multiple kernel threads and
processors: cpu, ppd, thread, and uthread. These commands respectively change
the current processor, display per-processor data structures, display thread table
entries, and display the uthread structure of a thread.
Core File Requirements of a Multi-Threaded Program
By default processes do not generate a full core file. Before AIX 4.3.0 this meant
only the stack for the thread causing the core dump was written to the core file.
Before AIX 4.3.2 this meant the part of the process address space made up of
shared memory region was not written to the core file. If an application needs to
debug data in shared memory regions, particular thread stacks it will be necessary
to generate a full core dump. To generate full core file information the following
command must be run as root:
chdev -l sys0 -a fullcore=true
Each individual pthread adds to the size of the generated core file. The amount of
core file space a pthread needs includes the stack size which the user can control
with pthread_attr_setstacksize(). For pthreads created with the NULL pthread
attribute each pthread in a 32-bit process adds 128KB to the size of the core file
and each pthread in a 64-bit process ads 256KB to the size of the core file.
Developing Multi-Threaded Program Debuggers
The pthread debug library (libpthdebug.a) provides a set of functions which will
allow debugger developers to provide debug capabilities for applications using the
pthread library.
This library is used to debug both 32-bit and 64-bit pthreaded applications. This
library is used to debug targeted debug processes only, it cannot be used
introspectively (i.e. linked to a application that uses pthreads) to examine pthread
information of its own application. This library can be used by a multi-threaded
debugger to debug a multi-threaded application. Multi-threaded debuggers are
supported via libpthreads.a. This library is thread safe. The pthread debug library
contains a 32-bit shared object. This library does not support 64-bit debuggers since
the functions ptrace() and ptracex() are not supported in 64-bit mode.
The pthread debug library provides debuggers access to pthread library
information. This includes information on pthreads, pthread attributes, mutexes,
mutex attributes, condition variables, condition variable attributes, read/write
locks, read/write lock attributes, and information about the state of the pthread
library. This library also provides help with controlling the execution of pthreads.
Note: All data (addresses, registers) returned by this library will be in 64-bit
format both for 64-bit and 32-bit application. It is the debuggers responsibility
to convert these values into 32-bit format for 32-bit applications. When
debugging a 32-bit application the top half of addresses and registers will be
ignored.
182
General Programming Concepts
Initialization
The debugger must initialize a pthread debug library session for each debug
process. This cannot be done until the pthread library has been initialized in the
debug process. The pthdb_session_pthreaded() function has been provided to tell
the debugger when the pthread library has been initialized in the debug process.
Each time, the pthdb_session_pthreaded() function is called it checks to see if the
pthread library has been initialized. If initialized, it returns PTHDB_SUCCESS.
Otherwise it returns PTHDB_NOT_PTHREADED. In both cases, it returns a function name
which can be used to set a breakpoint for immediate notification that the pthread
library has been initialized. Therefore, the pthdb_session_pthreaded() function
provides two methods to determine when the pthread library has been initialized:
v The first method requires the debugger to call the function each time the debug
process stops, to see if the debuggee is pthreaded.
v The second method requires the debugger to call the function once and if the
debuggee is not pthreaded to set a breakpoint to notify the debugger when the
debug process is pthreaded.
Once the debug process is pthreaded, the debugger must call the
pthdb_session_init() function, to initialize a session for the debug process. The
pthread debug library supports one session for a single debug process. The
debugger must assign a unique user identifier and pass it to pthdb_session_init()
which in turn will assign a unique session identifier which must be passed as the
first parameter to all other pthread debug library functions, except
pthdb_session_pthreaded(), in return. Whenever the pthread debug library
invokes a call back function, it will pass the unique debugger assigned user
identifier back to the debugger. The pthdb_session_init() function checks the list of
call back functions provided by the debugger, and initializes the session’s data
structures. Also, this function sets the session flags, see the pthdb_session_setflags
function.
Call Back Functions
The pthread debug library uses call back functions to obtain addresses, to obtain
data, to write data, to give storage management to the debugger, and to aid in
debugging the pthread debug library. See “Multi-Threaded Call Back Functions”
on page 187 for more information.
Update Function
Each time the debugger is stopped, after the session has been initialized, it is
necessary to call the pthdb_session_update() function. This function sets or reset
the lists of pthreads, pthread attributes, mutexes, mutex attributes, condition
variables, condition variable attributes, read/write locks, read/write lock attributes,
pthread specific keys and active keys. It uses call back functions to manage
memory for the lists.
Hold and Unhold Functions
Debuggers need to support hold and unhold of threads for two reasons:
v In order to allow a user to single step a single thread, it must be possible to hold
one or more of the other threads.
v For users to continue a subset of available threads, it must be possible to hold
threads not in the set.
The pthdb_pthread_hold() function sets the hold state of a pthread to hold.
The pthdb_pthread_unhold() function sets the hold state of a pthread to unhold.
Note: The pthdb_pthread_hold() and pthdb_pthread_unhold() functions
must always be used whether a pthread has a kernel thread or not.
Chapter 9. Parallel Programming
183
The pthdb_pthread_holdstate() function returns the hold state of the pthread.
The pthdb_session_committed() function reports the function name of the function
that is called after all of the hold and unhold changes are committed. A break
point can be placed at this function to notify the debugger when the hold and
unhold changes have been committed.
The pthdb_session_stop_tid() function informs the pthread debug library, which
informs the pthread library the tid of the thread that stopped the debugger.
The pthdb_session_commit_tid() function returns the list of kernel threads, one
kernel thread at a time, that must be continued to commit the hold and unhold
changes. This function must be called repeatedly, until PTHDB_INVALID_TID is
reported. If the list of kernel threads is empty, it is not necessary to continue any
threads for the commit operation.
The debugger can determine when all of the hold and unhold changes have been
committed in two ways:
v Before the commit operation (continuing all of the tids returned by the
pthdb_session_commit_tid() function) is started, the debugger can call the
pthdb_session_committed() function to get the function name and set a
breakpoint. (This method can be done once for the life of the process.)
v Before the commit operation is started, the debugger calls the
pthdb_session_stop_tid() function with the tid of the thread that stopped the
debugger. When the commit operation is complete, the pthread library will
ensure that the same stop tid is stopped as before the commit operation.
In order to hold or unhold pthreads it is necessary to follow the following
procedure, before continuing a group of pthreads or single stepping a single
pthread:
1. Use the pthdb_pthread_hold() and pthdb_pthread_unhold() functions to set
up which pthreads will be held and which will be unheld.
2. Set-up the method that will determine when all of the hold and unhold
changes have been committed.
3. Use the pthdb_session_commit_tid() function to determine the list of tids that
must be continued to commit the hold and unhold changes.
4. Continue the tids in the previous step and the thread which stopped the
debugger.
The pthdb_session_continue_tid() function allows the debugger to obtain the list
of kernel threads that must be continued before it proceeds with single stepping a
single pthread or continuing a group of pthreads. This function must be called
repeatedly, until PTHDB_INVALID_TID is reported. If the list of kernel threads is not
empty, the debugger will need to continue these kernel threads along with the
others it is explicitly interested in. The debugger is responsible for parking the stop
thread and continuing the stop thread. The stop thread, is the thread that caused
the debugger to be entered.
Context Functions
The pthdb_pthread_context() function is used to get the context information and
the pthdb_pthread_setcontext() function is used to set the context. The
pthdb_pthread_context() function obtains the context information of a pthread
from either the kernel or the pthread data structure in the debug process’s address
space. If the pthread is not associated with a kernel thread, then the context
information saved by pthread library is obtained. If a pthread is associated with a
kernel thread, the information is obtained from the debugger using call backs, it is
184
General Programming Concepts
the debuggers responsibility to determine if the kernel thread is in kernel mode or
user mode and provide the correct information for that mode.
When a pthread with kernel thread is in kernel mode code it is impossible to get
the full user mode context because the kernel does not save it off in one place. The
getthrds() function can be used to get part of this information. It always saves the
user mode stack and the debugger can discover this by checking
thrdsinfo64.ti_scount. If this is non-zero the user mode stack is available in
thrdsinfo64.ti_ustk. From user mode stack it is possible to determine the iar and
the call back frames but not the other register values. The thrdsinfo64 structure is
defined in procinfo.h file.
List Functions
The pthread debug library maintains lists for pthreads, pthread attributes,
mutexes, mutex attributes, condition variables, condition variables attributes,
read/write locks, read/write lock attributes, pthread specific keys and active keys,
each represented by a type specific handle. The pthdb_object() functions return the
next handle in the appropriate list. If the list is empty or the end of the list is
reached, PTHDB_INVALID_object is reported, where object is one of the following:
PTHREAD, ATTR, MUTEX, MUTEXATTR, COND, CONDATTR, RWLOCK, RWLOCKATTR or KEY.
Field Functions
Detailed information about an object can be obtained by using the appropriate
object member function, pthdb_object_field(), where object is one of the following:
pthread, attr, mutex, mutexattr, cond, condattr, rwlock, rwlockattr or key
and where field is the name of a field of the detailed information for the object.
Customizing the Session
The pthdb_session_setflags() function allows the debugger to change the flags
which customize the session. These flags are used to control the number of
registers that are read or wrote during context operations, and to control the
printing of debug information.
The pthdb_session_flags() function gets the current flags for the session.
Session Termination
At the end of the debug session, the session data structures need to be deallocated
and the session data needs to be deleted. This is accomplished by calling the
pthdb_session_destroy() function, which uses a call back functions to deallocate
the memory. All of the memory allocated by the pthdb_session_init(), and
pthdb_session_update() functions will be deallocated.
Example
Pseudo-code showing how the debugger should make use of the hold/unhold
code:
/* includes */
#include <sys/pthdebug.h>
main()
{
tid_t stop_tid; /* thread which stopped the process */
pthdb_user_t user = <unique debugger value>;
pthdb_session_t session; /* <unique library value> */
pthdb_callbacks_t callbacks = <callback functions>;
char *pthreaded_symbol=NULL;
char *committed_symbol;
int pthreaded = 0;
int pthdb_init = 0;
char *committed_symbol;
/* fork/exec or attach to debuggee */
Chapter 9. Parallel Programming
185
/* debuggee uses ptrace()/ptracex() with PT_TRACE_ME */
while (/* waiting on an event */)
{
/* debugger waits on debuggee */
if (pthreaded_symbol==NULL) {
rc = pthdb_session_pthreaded(user, &callbacks, pthreaded_symbol);
if (rc == PTHDB_NOT_PTHREADED)
{
/* set breakpoint at pthreaded_symbol */
}
else
pthreaded=1;
}
if (pthreaded == 1 && pthdb_init == 0) {
rc = pthdb_session_init(user, &session, PEM_32BIT, flags, &callbacks);
if (rc)
/* handle error and exit */
pthdb_init=1;
}
rc = pthdb_session_update(session)
if ( rc != PTHDB_SUCCESS)
/* handle error and exit */
while (/* accepting debugger commands */)
{
switch (/* debugger command */)
{
...
case DB_HOLD:
/* regardless of pthread with or without kernel thread */
rc = pthdb_pthread_hold(session, pthread);
if (rc)
/* handle error and exit */
case DB_UNHOLD:
/* regardless of pthread with or without kernel thread */
rc = pthdb_pthread_unhold(session, pthread);
if (rc)
/* handle error and exit */
case DB_CONTINUE:
/* unless we have never held threads for the life */
/* of the process */
if (pthreaded)
{
/* debugger must handle list of any size */
struct pthread commit_tids;
int commit_count = 0;
/* debugger must handle list of any size */
struct pthread continue_tids;
int continue_count = 0;
rc = pthdb_session_committed(session, committed_symbol);
if (rc != PTHDB_SUCCESS)
/* handle error */
/* set break point at committed_symbol */
/* gather any tids necessary to commit hold/unhold */
/* operations */
do
{
rc = pthdb_session_commit_tid(session,
&commit_tids.th[commit_count++]);
if (rc != PTHDB_SUCCESS)
/* handle error and exit */
} while (commit_tids.th[commit_count - 1] != PTHDB_INVALID_TID);
/* set up thread which stopped the process to be */
/* parked using the stop_park function*/
if (commit_count > 0) {
rc = ptrace(PTT_CONTINUE, stop_tid, stop_park, 0,
186
General Programming Concepts
&commit_tids);
if (rc)
/* handle error and exit */
/* wait on process to stop */
}
/* gather any tids necessary to continue */
/* interesting threads */
do
{
rc = pthdb_session_continue_tid(session,
&continue_tids.th[continue_count++]);
if (rc != PTHDB_SUCCESS)
/* handle error and exit */
} while (continue_tids.th[continue_count - 1] != PTHDB_INVALID_TID);
/* add interesting threads to continue_tids */
/* set up thread which stopped the process to be parked */
/* unless it is an interesting thread */
rc = ptrace(PTT_CONTINUE, stop_tid, stop_park, 0,
&continue_tids);
if (rc)
/* handle error and exit */
}
case DB_EXIT:
rc = pthdb_session_destroy(session);
/* other clean up code */
exit(0);
...
}
}
}
exit(0);
}
Multi-Threaded Call Back Functions
v
v
v
v
v
v
v
v
v
symbol_addrs
read_data
write_data
read_regs
write_regs
alloc
realloc
dealloc
print
Purpose
Provide access to facilities needed by the pthread debug library and supplied by the
debugger.
Library
These functions are provided by the debugger which links in the pthread debug
library.
Syntax
#include <sys/pthdebug.h>
int symbol_addrs(pthdb_user_t
user,
pthdb_symbol_t symbols[],
int
count)
int read_data(pthdb_user_t
user,
void
*buf,
pthdb_addr_t
addr,
Chapter 9. Parallel Programming
187
int
size)
int write_data(pthdb_user_t
user,
void
*buf,
pthdb_addr_t
addr,
int
size)
int read_regs(pthdb_user_t
user,
tid_t
tid,
unsigned long long flags,
struct context64 *context)
int write_regs(pthdb_user_t
user,
tid_t
tid,
unsigned long long flags,
struct context64 *context)
int alloc(pthdb_user_t
user,
size_t
len,
void
**bufp)
int realloc(pthdb_user_t
user,
void
*buf,
size_t
len,
void
**bufp)
int dealloc(pthdb_user_t
user,
void
*buf)
int print(pthdb_user_t
user,
char
*str)
Description
int symbol_addrs()
Resolves the address of symbols in the debuggee. The pthdebug library
will call this function to get the address of known debug symbols. If
symbol has a name of NULL or ″″, then just set the address to 0LL, instead
of doing a lookup or returning an error. If successful, 0 is returned, else
non-zero is returned.
int read_data()
Reads the requested number of bytes of data at requested location from an
active process or from a core file and returns the data through a buffer. If
successful then return 0 else return non-zero.
int write_data()
Writes the requested number of bytes of data at requested location. The
pthdebug library may use this to write data to the active process. If
successful return 0, else non-zero is returned.
int read_regs()
Read registers call back function should be able to read the context
information of a debuggee kernel thread from an active process or from a
core file. The information should be formatted in context64 form for both
32-bit and 64-bit process. If successful return 0, else non-zero is returned.
int write_regs()
Write register function should be able to write requested context
information to specified debuggee’s kernel thread id. If successful return 0,
else non-zero is returned.
int alloc()
Takes len and allocates len bytes of memory and returns the address. If
successful return 0, else non-zero is returned.
int realloc()
Takes len and the buf and re-allocates the memory and returns an address
to the realloc memory. If successful return 0, else non-zero is returned.
188
General Programming Concepts
int dealloc()
Takes a buffer and frees it. If successful return 0, else non-zero is returned.
int print()
Prints the character string on the debugger’s stdout. If successful return 0,
else non-zero is returned. This call back is for debugging the library only,
the messages printed will not be translated and will not be explained in
our user level documentation. If not debugging the pthread debug library
pass a NULL value for this call back.
Note: If write_data() and write_regs() are NULL then the pthread debug
library will not try to write data or write regs. If pthdb_pthread_set_context() is
called when write_data() and write_regs() are NULL, then it will return
PTHDB_NOTSUP.
Parameters
user
symbols
count
buf
addr
size
flags
tid
flags
context
len
bufp
str
User handle.
Array of symbols.
Number of symbols.
Buffer.
Address to be read from or wrote to.
Size of buffer.
Session flags, must accept
PTHDB_FLAG_GPRS, PTHDB_FLAG_SPRS,
PTHDB_FLAG_FPRS and
PTHDB_FLAG_REGS.
Thread id.
Flags which control which registers are read
or wrote.
Context structure.
Length of buffer to be allocated or
re-allocated.
Pointer to buffer.
String to be printed.
Return Values
If successful these function returns 0 else returns a non-zero value.
Benefits of Threads
The following explains the benefits of writing multi-threaded programs. Major
improvements of threads programming are:
v “Parallel Programming Concepts”, techniques are easier to implement.
v Multi-threaded programs provide better performance (see “Performance
Consideration” on page 192).
Threads do have some limitations (see “Limitations” on page 193) and cannot be
used for some special purposes which still require multi-processed programs.
Parallel Programming Concepts
There are two main advantages for using parallel programming instead of serial
programming techniques:
v Parallel programming can improve the performance of a program.
v Some common software models are well suited to parallel programming
techniques.
Chapter 9. Parallel Programming
189
Traditionally, multiple single-threaded processes have been used to achieve
parallelism, but some programs can benefit from a finer level of parallelism.
Multi-threaded processes offer parallelism within a process and share many of the
concepts involved in programming multiple single-threaded processes.
Modularity
Programs are often modeled as a number of distinct parts interacting with each
other to produce a desired result or service. A program can be implemented as a
single, complex entity that performs multiple functions among the different parts
of the program. A more simple solution consists of implementing several entities,
each entity performing a part of the program and sharing resources with other
entities.
By using multiple entities, a program can be separated according to its distinct
activities, each having an associated entity. These entities do not have to know
anything about the other parts of the program except when they exchange
information. In these cases, they must synchronize with each other to ensure data
integrity.
Threads are well-suited entities for modular programming. Threads provide simple
data sharing (all threads within a process share the same address space) and
powerful synchronization facilities (such as mutexes and condition variables).
Software Models
The following common software models can easily be implemented with threads.
v “Master/Slave Model”
v “Divide-and-Conquer Models” on page 191
v “Producer/Consumer Models” on page 191
All these models lead to modular programs. Models may also be combined to
efficiently solve complex tasks.
These models can apply to either traditional multi-process solutions, or to single
process multi-thread solutions, on multi-threaded systems such as AIX. In the
following descriptions, the word entity refers to either a single-threaded process or
to a single thread in a multi-threaded process.
Master/Slave Model
In the master/slave (sometimes called boss/worker) model, a master entity
receives one or more requests, then creates slave entities to execute them. Typically,
the master controls how many slaves there are and what each slave does. A slave
runs independently of other slaves.
An example of this model is a print job spooler controlling a set of printers. The
spooler’s role is to ensure that the print requests received are handled in a timely
fashion. When the spooler receives a request, the master entity chooses a printer
and causes a slave to print the job on the printer. Each slave prints one job at a
time on a printer, handling flow control and other printing details. The spooler
may support job cancellation or other features which require the master to cancel
slave entities or reassign jobs. The Master/Slave Print Model Example figure
illustrates this model.
190
General Programming Concepts
Spooler
M
Requests
M
S
Master
Slaves
S
P
S
P
S
P
P
Printers
Queued Jobs
Master/Slave Print Model Example
Divide-and-Conquer Models
In the divide-and-conquer (sometimes called simultaneous computation or work
crew) model, one or more entities perform the same tasks in parallel. There is no
master entity; all entities run in parallel independently.
An example of a divide-and-conquer model is a parallelized grep command
implementation, which could be done as follows. The grep command first
establishes a pool of files to be scanned. It then creates a number of entities. Each
entity takes a different file from the pool and searches for the pattern, sending the
results to a common output device. When an entity completes its file search, it
obtains another file from the pool or stops if the pool is empty.
Producer/Consumer Models
The producer/consumer (sometimes called pipelining) model is typified by a
production line. An item proceeds from raw components to a final item in a series
of stages. Usually a single worker at each stage modifies the item and passes it on
to the next stage. In software terms, an AIX command pipe, such as the cpio
command, is a good example of a this model.
The Producer/Consumer Model Example figure illustrates a typical
producer/consumer model. In this example, the Reader entity reads raw data from
standard input and passes it to the processor entity, which processes the data and
passes it to the writer entity, which writes it to standard output. Parallel
programming allows the activities to be performed concurrently: the writer entity
may output some processed data while the reader entity gets more raw data.
Chapter 9. Parallel Programming
191
Raw Data
Reader
Processor
Standard Input
Writer
Processed Data
Filter
Standard Output
Producer/Consumer Model Example
Performance Consideration
Multi-threaded programs can improve performance in many ways compared to
traditional parallel programs using multiple processes. Furthermore, higher
performance can be obtained on multiprocessor systems using threads.
Managing Threads
Managing threads, that is creating threads and controlling their execution, requires
fewer system resources than managing processes. Creating a thread, for example,
only requires the allocation of the thread’s private data area, usually 64KB, and
two system calls. Creating a process is far more expensive, because the entire
parent process addressing space is duplicated.
The threads library API is also easier to use than the one for managing processes.
Programmers should think about the six ways of calling the exec subroutine.
Thread creation requires just one syntax: the pthread_create subroutine.
Inter-Thread Communications
Inter-thread communication is far more efficient and easier to use than
inter-process communication. Because all threads within a process share the same
address space, they need not use shared memory. Shared data should just be
protected from concurrent access using mutexes or other synchronization tools.
Synchronization facilities provided by the threads library allow easy
implementation of flexible and powerful synchronization tools. These tools can
easily replace traditional inter-process communication facilities, such as message
queues. Note that pipes can be used as an inter-thread communication path.
Multiprocessor Systems
On a multiprocessor system, multiple threads can concurrently run on multiple
CPUs. Therefore, multi-threaded programs can run much faster than on a
uniprocessor system. They will also be faster than a program using multiple
processes, because threads require fewer resources and generate less overhead. For
example, switching threads in the same process can be faster, especially in the M:N
library model where context switches can often be avoided. Finally, a major
advantage of using threads is that a single multi-threaded program will work on a
uniprocessor system, but can naturally take advantage of a multiprocessor system,
without recompiling.
192
General Programming Concepts
Limitations
Multi-threaded programming is useful for implementing parallelized algorithms
using several independent entities. However, there are some cases where multiple
processes should be used instead of multiple threads.
Many operating system identifiers, resources, states, or limitations are defined at
the process level and, thus, are shared by all threads in a process. For example,
user and group IDs and their associated permissions are handled at process level.
Programs that need to assign different user IDs to their programming entities need
to use multiple processes, instead of a single multi-threaded process. Other
examples include file system attributes such as the current working directory, and
the state and maximum number of open files. Multi-threaded programs may not be
appropriate if these attributes are better handled independently. For example, a
multi-processed program can let each process open a large number of files without
interference from other processes.
Chapter 9. Parallel Programming
193
194
General Programming Concepts
Chapter 10. Programming on Multiprocessor Systems
On a uniprocessor system, threads execute one after another in a time-sliced
manner. This contrasts with a multiprocessor system, where several threads
execute at the same time, one on each available processor. Overall performance is
improved by running different process threads on different processors. However,
an individual program cannot take advantage of multiprocessing, unless it has
multiple threads.
For most users, multiprocessing is invisible, being completely handled by the
operating system and the programs it runs. If desired, users may bind their
processes (force them to run on a certain processor); however, this is not required,
nor recommended for ordinary use. Even for most programmers, taking advantage
of multiprocessing simply amounts to using multiple threads. On the other hand,
kernel programmers have to deal with several issues when porting or creating
code for multiprocessor systems.
Identifying Processors
Symmetric multiprocessor machines have one or more CPU boards, each of which
can accommodate two processors. For example, a four processor machine has two
CPU boards, each having two processors. Commands, subroutines, or messages
that refer to processors need to use an identification scheme. Processors are
identified by physical and logical numbers, and by Object Data Manager (ODM)
processor names and location codes.
ODM Processor Names
ODM is a system used to identify various parts throughout a machine, including
bus adapters, peripherals such as printers or terminals, disks, memory boards, and
processor boards. See “Chapter 17. Object Data Manager (ODM)” on page 533 for
more information about ODM.
ODM assigns numbers to processor boards and processors in order, starting from 0
(zero), and creates names based on these numbers by adding a prefix cpucard or
proc. Thus, the first processor board is called cpucard0, and the second processor
on it is called proc1.
ODM location codes for processors consist of four 2-digit fields, in the form
AA-BB-CC-DD, as explained below:
AA
BB
CC
DD
Always 00. It indicates the main unit.
Indicates the processor board number. It can be 0P, 0Q, 0R, or 0S, indicating
respectively the first, second, third or fourth processor card.
Always 00.
Indicates the processor position on the processor board. It can be 00 or 01.
These codes are illustrated in “Example Processor Configurations” on page 196.
© Copyright IBM Corp. 1997, 1999
195
Logical Processor Numbers
Processors can also be identified using logical numbers, which start with 0 (zero).
Only enabled processors have a logical number.
The logical processor number 0 (zero) identifies the first physical processor in the
enabled state; the logical processor number 1 (one) identifies the second enabled
physical processor, and so on.
Generally, all operating system commands and library subroutines use logical
numbers to identify processors. The cpu_state command is an exception and uses
ODM processor names instead of logical processor numbers.
ODM Processor States
If a processor functions correctly, it can be enabled or disabled using a software
command. A processor is marked faulty if it has a detected hardware problem.
ODM classifies processors using three states. A processor can only be in one of the
following states:
enabled
disabled
faulty
Processor works and can be used by AIX.
Processor works, but cannot be used by AIX.
Processor does not work (a hardware fault was detected).
Controlling Processor Use
On a multiprocessor system, the use of processors can be controlled in two ways:
v A system administrator can control the availability of the processors for the
whole system.
v A user can force a process or kernel threads to run on a specific processor.
The cpu_state Command
A system administrator (or any user with root authority) can use the cpu_state
command to list system processors or to control available processors. This
command can be used to list the following information for each configured
processor in the system:
Name
Cpu
Status
Location
ODM processor name (see “ODM Processor Names” on page 195), shown in
the form procx, where x is the physical processor number
Logical processor number of enabled processor (see “Logical Processor
Numbers”)
Processor state (see “ODM Processor States”) for the next boot
ODM processor location code (“ODM Processor Names” on page 195), shown
in the form AA-BB-CC-DD
Note: The cpu_state command does not display the current processor state,
but instead displays the state to be used for the next system start up (enabled
or disabled). If the processor does not respond, it is either faulty (an ODM
state) or a communication error occurred. In this case, the cpu_state
command displays No Reply.
Example Processor Configurations
The examples that follow show various processor configurations and how they
affect the output of the cpu_state command. These examples illustrate the
196
General Programming Concepts
relationships among physical processor numbers, logical processor numbers, the
current processor state, and the processor state used at the next boot.
Simple Processor Configurations
The simplest case to consider is when all available processors on a system are
functioning and enabled. Consider a simple two processor system with both
processors enabled. The various ODM and number conventions are shown in the
following table.
Processor Naming Conventions
ODM Card
Name
ODM Processor
Name
Logical Number ODM Current
Processor State
cpu_state Status
Field
cpucard0
proc0
0
Enabled
Enabled
cpucard0
proc1
1
Enabled
Enabled
For the above configuration, the cpu_state -l command produces a listing similar
to the following:
Name
proc0
proc1
Cpu
0
1
Status
Enabled
Enabled
Location
00-0P-00-00
00-0P-00-01
The following example shows the system upgraded by adding an additional CPU
card with two processors. By default, processors are enabled, so the new
configuration is shown in the following table.
Processor Naming Conventions
ODM Card
Name
ODM Processor
Name
Logical Number ODM Current
Processor State
cpu_state Status
Field
cpucard0
proc0
0
Enabled
Enabled
cpucard0
proc1
1
Enabled
Enabled
cpucard1
proc2
2
Enabled
Enabled
cpucard1
proc3
3
Enabled
Enabled
For this configuration, the cpu_state -l command produces a listing similar to the
following:
Name
proc0
proc1
proc2
proc3
Cpu
0
1
2
3
Status
Enabled
Enabled
Enabled
Enabled
Location
00-0P-00-00
00-0P-00-01
00-0Q-00-00
00-0Q-00-01
Complex Processor Configurations
In some conditions, a processor is not enabled and does not have a logical
processor number. A processor can fail a boot power-on test and be marked faulty
by ODM. A processor can also be disabled for maintenance or test reasons. Also,
when a processor is enabled or disabled using the cpu_state command, its current
state remains unchanged until the next boot, but its state at the next boot
(displayed in the Status field of the cpu_state command) is changed immediately.
Disabled Processor Configurations: Using the four processor configurations in
the previous section, the physical processor 1 can be disabled with the command:
cpu_state -d proc1
Chapter 10. Programming on Multiprocessor Systems
197
The processor configuration is shown in the following table.
Processor Naming Conventions
ODM Card
Name
ODM Processor
Name
Logical Number ODM Current
Processor State
cpu_state Status
Field
cpucard0
proc0
0
Enabled
Enabled
cpucard0
proc1
1
Enabled
Disabled
cpucard1
proc2
2
Enabled
Enabled
cpucard1
proc3
3
Enabled
Enabled
For this configuration, the cpu_state -l command produces a listing similar to the
following:
Name
proc0
proc1
proc2
proc3
Cpu
0
1
2
3
Status
Enabled
Disabled
Enabled
Enabled
Location
00-0P-00-00
00-0P-00-01
00-0Q-00-00
00-0Q-00-01
When the system is rebooted, the processor configuration is as shown in the
following table.
Processor Naming Conventions
198
ODM Card
Name
ODM Processor
Name
Logical Number ODM Current
Processor State
cpu_state Status
Field
cpucard0
proc0
0
Enabled
Enabled
cpucard0
proc1
1
Disabled
Disabled
cpucard1
proc2
2
Enabled
Enabled
cpucard1
proc3
3
Enabled
Enabled
General Programming Concepts
The output of the cpu_state -l command is similar to the following:
Name
proc0
proc1
proc2
proc3
Cpu
0
1
2
Status
Enabled
Disabled
Enabled
Enabled
Location
00-0P-00-00
00-0P-00-01
00-0Q-00-00
00-0Q-00-01
Faulty Processor Configurations: The following example uses the last processor
configuration discussed in the previous section. The system is rebooted with
processors proc0 and proc3 failing their power-on tests. The processor
configuration is as shown in the following table.
Processor Naming Conventions
ODM Card
Name
ODM Processor
Name
Logical Number ODM Current
Processor State
cpu_state Status
Field
cpucard0
proc0
-
Faulty
No Reply
cpucard0
proc1
-
Disabled
Disabled
cpucard1
proc2
0
Enabled
Enabled
cpucard1
proc3
-
Faulty
No Reply
The output of the cpu_state -l command is similar to the following:
Name
proc0
proc1
proc2
proc3
Cpu
0
-
Status
No Reply
Disabled
Enabled
No Reply
Location
00-0P-00-00
00-0P-00-01
00-0Q-00-00
00-0Q-00-01
Binding Processes and Kernel Threads
Users may also force their processes to run on a given processor; this action is
called binding. A system administrator may bind any process. From the command
line, binding is controlled with the bindprocessor command.
It is important to understand that a process itself is not bound, but rather its kernel
threads are bound. Once kernel threads are bound, they are always scheduled to
run on the chosen processor, unless they are later unbound. When a new kernel
thread is created, it has the same bind properties as its creator.
This applies to the initial thread in the new process created by the fork subroutine;
the new thread inherits the bind properties of the thread that called the fork
subroutine. When the exec subroutine is called, bind properties are left unchanged.
Once a process is bound to a processor, if no other binding or unbinding action is
performed, all child processes will be bound to the same processor.
It is only possible to bind processes to enabled processors using logical processor
numbers. Available logical processor numbers can be listed using the
bindprocessor -q command. For a system with four enabled processors, this
command produces output similar to:
The available processors are: 0 1 2 3
Binding may also be controlled within a program using the bindprocessor
subroutine, which allows the programmer to bind a single kernel thread or all
kernel threads in a process. The programmer can also unbind either a single kernel
thread or all kernel threads in a process.
Chapter 10. Programming on Multiprocessor Systems
199
Creating Locking Services
Some programmers may want to implement their own high-level locking services
instead of using the standard locking services (mutexes) provided in the threads
library. For example, a database product may already use a set of internally
defined services; it can be easier to adapt these locking services to a new system
than to adapt all the internal modules that use these services.
For this reason, AIX provides atomic locking service primitives which can be used
to build higher level locking services. To create services that are
multiprocessor-safe (like the standard mutex services), programmers must use the
atomic locking services described in this section and not atomic operations
services, such as compare_and_swap.
Multiprocessor-Safe Locking Services
Locking services are used to serialize access to resources that may be used
concurrently. For example, locking services can be used for insertions in a linked
list, which require several pointer updates. If the update sequence by one process
is interrupted by a second process that tries to access the same list, an error can
occur. A sequence of operations that should not be interrupted is called a critical
section.
Locking services use a lock word to indicate the lock status: 0 (zero) can be used
for free, and 1 (one) for busy. Therefore, a service to acquire a lock would do the
following:
test the lock word
if the lock is free
set the lock word to busy
return SUCCESS
...
Because this sequence of operations (read, test, set) is itself a critical section, special
handling is required. On a uniprocessor system, disabling interrupts during the
critical section prevents interruption by a context switch. But on a multiprocessor
system, the hardware must provide a so-called test-and-set primitive, usually with
a special machine instruction. In addition, special processor dependent
synchronization instructions called import and export fences are used to temporarily
block other reads or writes. They protect against concurrent access by several
processors and against the read and write reordering performed by modern
processors.
To mask this complexity and provide independence from these machine-dependent
instructions, three subroutines are defined:
_check_lock
_clear_lock
_safe_fetch
200
General Programming Concepts
Conditionally updates a single word variable atomically, issuing an import
fence for multiprocessor systems. The compare_and_swap routine is
similar, but it does not issue an import fence and, therefore, is not usable
to implement a lock.
Atomically writes a single word variable, issuing an export fence for
multiprocessor systems.
Atomically reads a single word variable, issuing an import fence for
multiprocessor systems. The import fence ensures that the read value is
not a stale value resulting from an early pre-fetch. This subroutine is
rarely needed.
Locking Services Example
The multiprocessor-safe locking subroutines can be used to create custom
high-level routines independent of the threads library. The example that follows
shows partial implementations of subroutines similar to the pthread_mutex_lock
and pthread_mutex_unlock subroutines in the threads library:
#include <sys/atomic_op.h>
#define SUCCESS
0
#define FAILURE
-1
#define LOCK_FREE
0
#define LOCK_TAKEN
1
typdef struct {
atomic_p
tid_t
...
} my_mutex_t;
/* for locking primitives */
lock; /* lock word */
owner; /* identifies the lock owner */
/* implementation dependent fields */
...
int my_mutex_lock(my_mutex_t *mutex)
{
tid_t self; /* caller's identifier */
/*
Perform various checks:
is mutex a valid pointer?
has the mutex been initialized?
*/
...
/* test that the caller does not have the mutex */
self = thread_self();
if (mutex->owner == self)
return FAILURE;
/*
Perform a test-and-set primitive in a loop.
In this implementation, yield the processor if failure.
Other solutions include: spin (continuously check);
or yield after a fixed number of checks.
*/
while (!_check_lock(&mutex->lock, LOCK_FREE, LOCK_TAKEN))
yield();
mutex->owner = self;
return SUCCESS;
} /* end of my_mutex_lock */
int my_mutex_unlock(my_mutex_t *mutex)
{
/*
Perform various checks:
is mutex a valid pointer?
has the mutex been initialized?
*/
...
/* test that the caller owns the mutex */
if (mutex->owner != thread_self())
return FAILURE;
_clear_lock(&mutex->word, LOCK_FREE);
return SUCCESS;
} /* end of my_mutex_unlock */
Kernel Programming
Kernel programming is thoroughly explained in AIX Version 4.3 Kernel Extensions
and Device Support Programming Concepts. This section only highlights the major
changes required for multiprocessor systems.
Chapter 10. Programming on Multiprocessor Systems
201
Serialization is often required when accessing certain critical resources. Locking
services can be used to serialize thread access in the process environment, but they
will not protect against an access occurring in the interrupt environment.
Previously, a kernel service disabled interrupts using the i_disable kernel service
to serialize interrupt level access. However, this strategy does not work in a
multiprocessor environment. Therefore, new or ported code should use the
disable_lock and unlock_enable kernel services, which use simple locks in
addition to interrupt control. These kernel services can also be used for
uniprocessor systems, on which they simply use interrupt services without locking.
See Locking Kernel Services in AIX Version 4.3 Kernel Extensions and Device Support
Programming Concepts for detailed information.
Device drivers by default run in a logical uniprocessor environment, in what is
called funnelled mode. Most well-written drivers for AIX uniprocessor systems will
work without modification in this mode, but must be carefully examined and
modified to benefit from multiprocessing. Finally, kernel services for timers now
have return values because they will not always succeed in a multiprocessor
environment. Therefore, new or ported code must check these return values. See
Using Multiprocessor-Safe Timer Services in AIX Version 4.3 Kernel Extensions and
Device Support Programming Concepts for more information.
32-bit and 64-bit Addressability
In AIX, applications are either 32-bit applications or 64-bit applications.
A 32-bit application is an application that executes in an environment where
addresses are 32 bits long and can represent 4 gigabytes of addressability (virtual
address space).
A 64-bit application is an application that executes in an environment where
addresses are 64 bits long and can represent much larger addressability (over a
billion gigabytes).
When creating an application, a decision must be made whether to create a 32-bit
application or a 64-bit application. 32-bit applications can be run on any RS/6000
system. 64-bit applications can only be run on 64-bit RS/6000 systems. Both 32-bit
and 64-bit applications (and libraries) can be compiled and linked on both 32-bit
and 64-bit systems.
Performance
If the same source code is used to create both a 32-bit and a 64-bit application, the
64-bit application will be the same size or larger than the 32-bit application and
will generally run no faster (and often slower) than the 32-bit application, unless it
makes use of the larger 64-bit addressability to improve its performance. Therefore,
the correct choice should generally be to create a 32-bit application unless 64-bit
addressability is required by the application or can be used to dramatically
improve its performance. For this reason, the default mode for development tools
is to create 32-bit objects and applications.
The 64-bit address space can be used to dramatically improve the performance of
applications that manipulate large amounts of data. This data can either be created
within the application or obtained from files. Generally the performance gain
comes from the fact that the 64-bit application can contain the data in its address
space (either created in data structures or mapped into memory), where the data
would not fit into a 32-bit address space. The data would need to be multiple
gigabytes in size or larger to show this benefit.
202
General Programming Concepts
There are two reasons for this performance improvement. First, the system call
overhead of reading and writing files can be avoided by mapping the files into
memory. Second, 64-bit systems can support physical memories that are larger than
the addressability of 32-bit applications, so 64-bit applications are needed to make
full use of the physical memory available.
64-bit objects and archive file types
64-bit libraries and applications can only be created from 64-bit objects. A 64-bit
object is an object type (64-bit XCOFF format) created by compilation or assembly
in 64-bit mode. (This does not mean that the compiler or assembler executes in a
64-bit execution environment, just that the compiler or assembler has been
requested to create 64-bit objects rather than 32-bit objects.) 32-bit XCOFF format
was the only object type in releases of AIX before release 4.3.
There is no way to create an object or application using both 32-bit and 64-bit
object files. A system provided library contains both 32-bit and 64-bit object files.
The linker selects the appropriate objects from the library based on the type of
linking that is requested (32-bit or 64-bit) and creates an object or application of
that type.
There are two archive file types in AIX. The first does not recognize 64-bit object
files and cannot be larger than 2 gigabytes. This was the only archive file type in
releases of AIX before release 4.3. The second archive file type recognizes both
32-bit and 64-bit object files and will work with files larger than 2 gigabytes.
Differences between 32-bit and 64-bit execution environments
In addition to the differences in addressability, there are the following differences
between the 32-bit and 64-bit execution environments (or modes):
v The C ″long″ type (and types derived from it) in 64-bit mode is 64 bits in size.
v All pointer types in 64-bit mode are 64 bits in size.
v 64-bit applications can make use of 64-bit PowerPC instructions.
v The size of a machine register is 64 bits in 64-bit mode.
v The maximum theoretical limits for the size of 64-bit applications, their heaps,
stacks, shared libraries, and loaded objects is millions of gigabytes. The practical
limits are dependent on the file system limits, paging space sizes, and system
resources available.
All C fundamental types other than ″long″ and pointer types will be the same size
in 32-bit and 64-bit compilation modes.
Tools support for 64-bit development
AIX provides support in all the standard tools for building, examining, and
debugging 64-bit applications.
yacc, lex, and lint work with source code destined for both 32-bit and 64-bit
compilation.
The C and Fortran compilers and the assembler allow the creation of both 32-bit
and 64-bit objects. The linker allows the creation of both 32-bit and 64-bit objects
and applications.
make, ar, strip, dump, nm, prof, gprof, lorder, ranlib, size, strings, and sum work
with both 32-bit and 64-bit objects and applications.
Chapter 10. Programming on Multiprocessor Systems
203
dbx and xldb allow the debugging of both 32-bit and 64-bit applications.
Porting source code from 32-bit to 64-bit execution environments
The following issues must be examined and dealt with when porting source code
for 32-bit applications to be compiled in 64-bit mode to create 64-bit applications:
v Remove any assumption that a pointer type can fit in a C integer type (or types
derived from integer).
v Remove any assumption that a C long type can fit in a C integer type (or types
derived from long and integer).
v Remove any assumption about the number of bits in a C long type object when
bit shifting or doing bitwise operations.
v Remove any assumption that a C integer can be passed to an unprototyped
long or pointer parameter.
v Remove any assumption that an unprototyped function can return a pointer or
long.
The -t flag to lint can be used to find issues when porting source code from 32-bit
to 64-bit compilation mode.
64-bit application development
The APIs (Application Programming Interfaces) provided to 32-bit applications are
also generally provided to 64-bit applications. Some libraries that have been
superseded or deprecated for 32-bit applications are not being provided to 64-bit
applications, so their APIs will be missing in 64-bit execution mode.
The names of types, global variables, preprocessor macros, and predefined
constants are the same in 32-bit and 64-bit compilation mode. The sizes (and
layouts in the case of structures) and values for these are often different in 32-bit
and 64-bit compilation mode, to account for the different sizes of the address
spaces and fundamental types involved.
The names of API functions, the types of parameters passed, and the return types
are the same in 32-bit and 64-bit compilation modes. The sizes (and layouts in the
case of structures) of the parameters and return values are often different in 32-bit
and 64-bit compilation modes to account for the different size of the address spaces
involved.
64-bit library development
Libraries should provide the same functionality to 64-bit applications that they
provide to 32-bit applications. This is to minimize the amount of porting that
needs to be done when changing the execution mode of an application from 32-bit
to 64-bit. To ease porting, the names of functions provided, their parameter types
and return types should be the same for 32-bit and 64-bit applications.
The choice of the types of integral parameters and return values should be made
based upon what a parameter or return value is representing. If it represents the
size of an object in the address space, its type should be based upon a C ″long″
type. Otherwise, the type should be made one of the C types ″char″, ″short″, ″int″,
or ″long long″, depending on what the maximum possible value is. (These types
are the same size in 32-bit and 64-bit compilation mode.)
Libraries should contain both the 32-bit and 64-bit objects files for the API they
support. This will minimize the porting effort for makefiles for applications that
are being ported from 32-bit to 64-bit execution mode. System libraries provide
object files for using both 32-bit and 64-bit applications in the same library archive
file.
204
General Programming Concepts
Only 32-bit objects can be loaded by a 32-bit application. Only 64-bit objects can be
loaded by 64-bit applications. If an API is provided by loading objects, a separate
32-bit and 64-bit version of the object must be provided with a different pathname.
64-bit kernel extension development
AIX kernel extensions run in 32-bit mode, even when processing requests made by
64-bit applications. This allows kernel extensions which were built for AIX releases
which only supported 32-bit systems to run on AIX releases which support both
32-bit and 64-bit systems.
For these old kernel extensions and all kernel extensions which have not been
designed to work with 64-bit applications, only 32-bit applications can be
supported. A 64-bit application will fail to link if it attempts to make use of a
system call from a kernel extension that has not been modified to support 64-bit
applications. A kernel extension can indicate that it supports 64-bit applications by
setting the SYS_64BIT flag when it is loaded using the sysconfig routine.
Kernel extension support for 64-bit applications has two aspects.
The first aspect is the use of new kernel services for working with the 64-bit user
address space. The new 64-bit services for examining and manipulating the 64-bit
address space are as_att64, as_det64, as_geth64, as_puth64, as_seth64, and
as_getsrval64. The new services for copying data to or from 64-bit address spaces
are copyin64, copyout64, copyinstr64, fubyte64, fuword64, subyte64, and
suword64. The new service for doing cross-memory attaches to memory in a 64-bit
address space is xmattach64. The new services for creating real memory mappings
are rmmap_create64 and rmmap_remove64. The major difference between all these
services and their 32-bit counterparts is that they use 64 bit user addresses rather
than 32 bit user addresses.
The new service for determining whether a process (and its address space) is 32-bit
or 64-bit is IS64U.
The second aspect of supporting 64-bit applications is taking 64 bit user data
passed to system calls and transforming that data to 32 bit data which can be
passed through the system call handler to the kernel extension. If the types of the
parameters passed to a system call are all 32 bit sized or smaller (in 64-bit
compilation mode), then no extra work is required. However, if 64-bit data (long or
long long C types) or addresses are passed to the system call, then the data must
be split and passed in two parameters or transformed into a 32-bit value.
To assist in using 64-bit user addresses in kernel mode, a set of ″remapping″
services are provided to transform the 64 bit user addresses into 32 bit addresses,
which can be used by most of the old 32-bit user address manipulation services in
the kernel. In this way, kernel code which has not been modified to work with
64-bit user addresses an always use the 32 bit address (either from a 32-bit
application or as a transformed 64-bit address from a 64-bit application). Services
such as copyin, copyout, xmattach, fuword, fubyte, suword, and subyte will
correctly work with the 32-bit address by transforming it back to a 64-bit address.
The remapping services consist of a set of routines which can be called (in 64-bit
execution mode) from a library and a matching set of routines which can be called
(in 32-bit mode) from a kernel extension. The library routines determine which
32-bit addresses should be matched to each 64-bit address and package this
information in a data structure to be passed to the kernel. The library then passes
Chapter 10. Programming on Multiprocessor Systems
205
all the parameters (including the 32-bit addresses which were associated with the
64-bit addresses) through the 64-bit system call to the kernel extension.
The kernel extension passes the remapping information (created by the library
remapping routine) to the remapping kernel service, which saves it for use by
address space services for the duration of that system call.
A limited number of addresses can be remapped because remapping must be done
on a segment (256K bytes) basis. One address remapping can use multiple of these
segments, depending on the location of the address relative to a segment boundary
and depending on the length of the address range being remapped. Multiple
address remappings can utilize a single segment of remapping if they all happen
to fall in the range of a single segment.
To simplify the use of these remapping services, a set of macros have been
declared (in sys/remap.h) which hide many of the underlying details. These
macros should be used to avoid mistakes in the use of the underlying services.
REMAP_DCL
REMAP_SETUP and
REMAP_SETUP_WITH_LEN
REMAP, REMAP_VOID,
REMAP_IDENTITY, and
REMAP_IDENTITY_VOID
206
General Programming Concepts
Used to declare the data structures used by
the other services
Used to fill in the data structures with the
64-bit addresses and lengths
Called by the library to determine the
remappings
Chapter 11. Threads Programming Guidelines
The following information provides guidelines for writing multi-threaded
programs using the threads library (libpthreads.a). The AIX threads library is
based on the emerging POSIX 1003.1c standard. For this reason, the following
information presents the threads library as the AIX implementation of the POSIX
standard.
Thread Implementation Model
There are various thread implementation models. One model is the ″library-thread
model.″ In such a model, the threads of a process are not visible to the operating
system kernel, and the threads are not kernel scheduled entities. The process is the
only kernel scheduled entity. The process is scheduled onto the processor by the
kernel according to the scheduling attributes of the process. The threads are
scheduled onto the single kernel scheduled entity (the process) by the run-time
library according to the scheduling attributes of the threads. This is the model of
threads provided on AIX Version 3.
At the other end of the spectrum is the ″kernel-thread model.″ In this model, all
threads are visible to the operating system kernel. Thus, all threads are kernel
scheduled entities, and all threads can concurrently execute. The threads are
scheduled onto processors by the kernel according to the scheduling attributes of
the threads. This model is the model provided in AIX Version 4.1 and AIX Version
4.2.
AIX Version 4.3 uses a hybrid model that offers the speed of library threads and
the concurrency of kernel threads. In hybrid models, a process has a varying
number of kernel scheduled entities associated with it. It also has a potentially
much larger number of library threads associated with it. Some library threads
may be bound to kernel scheduled entities, while the other library threads are
multiplexed onto the remaining kernel scheduled entities. For this reason, a hybrid
model is referred to as a ″M:N″ model. In this model, the process can have
multiple concurrently executing threads; specifically, it can have as many
concurrently executing threads as it has kernel scheduled entities.
Thread-safe and Threaded Libraries in AIX
In AIX Version 4 Releases 1 and 2, special versions of selected libraries were
provided, that were for use by threaded applications. These libraries were
counterparts of the non-thread-safe libraries, but with the suffix ″_r″ added to the
name. These libraries were:
libc.a/libc_r.a
libm.a/libm_r.a
libs.a/libs_r.a
libsvid.a/libsvid_r.a
libxti.a/libxti_r.a
libbsd.a/libbsd_r.a
libnetsvc.a/libnetsvc_r.a
libs2.a/libs2_r.a
libtli.a/libtli_r.a
In AIX Version 4 Release 3, the need for these ″_r″ versions has been eliminated.
By default, all applications are now considered ″threaded,″ even though most are
of the case ″single threaded.″ These thread-safe libraries are now:
© Copyright IBM Corp. 1997, 1999
207
libbsd.a
libsvid.a
libnetsvc.a
libc.a
libtli.a
libm.a
libxti.a
The ″_r″ versions have been kept as links to these libraries, to enable compatibility
with user applications.
Threads Versions On AIX
In order to bring threaded application support to our users, AIX introduced
threads API models based on preliminary drafts of the now-official IEE POSIX
standard. AIX Version 4 Release 3 conforms fully to the IEEE POSIX standard for
threads APIs, IEEE POSIX 1003.1-1996.
v In AIX Version 3 Release 2 threads were supported at a ″Draft 4″ level.
v In AIX Version 4 Release 1 and 2 threads were supported at a ″Draft 4″ level.
AIX Version 4 Release 3 provides full support for applications compiled on AIX
Version 3 Release 2 and AIX Version 4. It also provides compilation support for
applications written to the ″Draft 7″ level that are not able to modify their source
code to full standard conformance.
Compiling a Threaded Application
In AIX Version 4 Releases 1 and 2, ″_r″ versions of the C compiler invocations were
offered that allowed the proper libraries and command line options to be set for
creating a threaded application.
In AIX Version 4 Release 3 the use of the ″_r″ invocations is no longer required for
creating a threaded application.
v To Compile a Threaded Application on AIX 4.3, use either the normal or ″_r″
version of the compiler.
v To Compile a Threaded Application at ″Draft 7″ level, use the ″_r7″ invocation of
the compiler.
Threads Basic Operation Overview
To write a multi-threaded program, it is necessary to understand how to create and
terminate threads. Synchronization facilities and scheduling control are not
required for a basic usage of threads.
The following information will help you in writing your first multi-threaded
program:
v “Creating Threads”
v “Terminating Threads” on page 212
v “List of Threads Basic Operation Subroutines” on page 219.
Creating Threads
A thread has attributes, which specify the characteristics of the thread. The
attributes default values fit for most common cases. To control thread attributes, a
thread attributes object must be defined before creating the thread.
Read the following to learn more about creating threads:
208
General Programming Concepts
Thread Attributes Object
The thread attributes are stored in an opaque object, the thread attributes object, used
when creating the thread. It contains several attributes, depending on the
implementation of POSIX options. It is accessed through a variable of type
pthread_attr_t. In AIX, the pthread_attr_t data type is a pointer to a structure; on
other systems it may be a structure or another data type.
Thread Attributes Object Creation and Destruction
The thread attributes object is initialized to default values by the pthread_attr_init
subroutine. The attributes are handled by subroutines. The thread attributes object
is destroyed by the pthread_attr_destroy subroutine. This subroutine may free
storage dynamically allocated by the pthread_attr_init subroutine, depending on
the implementation of the threads library.
In the following example, a thread attributes object is created and initialized with
default values, then used and finally destroyed:
pthread_attr_t attributes;
/* the attributes object is created */
...
if (!pthread_attr_init(&attributes)) {
/* the attributes object is initialized */
...
/* using the attributes object */
...
pthread_attr_destroy(&attributes);
/* the attributes object is destroyed */
}
The same attributes object can be used to create several threads. It can also be
modified between two thread creations. When the threads are created, the
attributes object can be destroyed without affecting the threads created with it.
Detachstate Attribute
The following attribute is always defined.
Detachstate
Specifies the detached state of a thread.
The value of the attribute is returned by the pthread_attr_getdetachstate
subroutine; it can be set by the pthread_attr_setdetachstate subroutine. Possible
values for this attributes are the following symbolic constants:
PTHREAD_CREATE_DETACHED Specifies that the thread will be created in the detached
state.
PTHREAD_CREATE_JOINABLE Specifies that the thread will be created in the joinable
state.
The default value is PTHREAD_CREATE_JOINABLE.
If you create a thread in the joinable state, you must pthread_join with the thread.
Otherwise, you may run out of storage space when creating new threads, because
each thread takes up a signficant amount of memory.
Other Attributes
The following attributes are also defined in AIX. They are intended for advanced
programs and may require special execution privilege to take effect. Most
programs will operate correctly with the default settings.
Chapter 11. Threads Programming Guidelines
209
Contention Scope
Inheritsched
Schedparam
Schedpolicy
Specifies
Specifies
Specifies
Specifies
the
the
the
the
contention scope of a thread.
inheritance of scheduling properties of a thread.
scheduling parameters of a thread.
scheduling policy of a thread.
The use of these attributes is explained in Scheduling Attributes.
Stacksize
Stackaddr
Guardsize
Specifies the size of the thread’s stack.
Specifies the address of the thread’s stack.
Specifies the size of the guard area of the thread’s stack.
The use of these attributes is explained in Stack Attributes.
Thread Creation
Creating a thread is accomplished by calling the pthread_create subroutine. This
subroutine creates a new thread and makes it runnable.
Using the Thread Attributes Object
When calling the pthread_create subroutine, you may specify a thread attributes
object. If you specify a NULL pointer, the created thread will have the default
attributes. Thus, the code fragment:
pthread_t thread;
pthread_attr_t attr;
...
pthread_attr_init(&attr);
pthread_create(&thread, &attr, init_routine, NULL);
pthread_attr_destroy(&attr);
210
General Programming Concepts
is equivalent to:
pthread_t thread;
...
pthread_create(&thread, NULL, init_routine, NULL);
Entry Point Routine
When calling the pthread_create subroutine, you must specify an entry-point
routine. This routine, provided by your program, is similar to the main routine for
the process. It is the first user routine executed by the new thread. When the
thread returns from this routine, the thread is automatically terminated.
The entry-point routine has one parameter, a void pointer, specified when calling
the pthread_create subroutine. You may specify a pointer to some data, such as a
string or a structure. The creating thread (the one calling the pthread_create
subroutine) and the created thread must agree upon the actual type of this pointer.
The entry-point routine returns a void pointer. After the thread termination, this
pointer is stored by the threads library unless the thread is detached. See
“Synchronization Overview” on page 220 for more information about using this
pointer.
Returned Information
The pthread_create subroutine returns the thread ID of the new thread. The caller
can use this thread ID to perform various operations on the thread.
Depending on the scheduling parameters of both threads, the new thread may
start running before the call to the pthread_create subroutine returns. It may even
happen that, when the pthread_create subroutine returns, the new thread has
already terminated. The thread ID returned by the pthread_create subroutine
through the thread parameter is then already invalid. It is, therefore, important to
check for the ESRCH error code returned by threads library subroutines using a
thread ID as a parameter.
If the pthread_create subroutine is unsuccessful, no new thread is created, the
thread ID in the thread parameter is invalid, and the appropriate error code is
returned.
Handling Thread IDs
The thread ID of a newly created thread is returned to the creating thread through
the thread parameter. The current thread ID is returned by the pthread_self
subroutine.
A thread ID is an opaque object; its type is pthread_t. In AIX, the pthread_t data
type is an integer. On other systems, it may be a structure, a pointer, or any other
data type.
To enhance the portability of programs using the threads library, the thread ID
should always be handled as an opaque object. For this reason, thread IDs should
be compared using the pthread_equal subroutine. Never use the C equality
operator (==), because the pthread_t data type may be neither an arithmetic type
nor a pointer.
Chapter 11. Threads Programming Guidelines
211
A First Multi-Threaded Program
The first multi-threaded program discussed is short. It displays ″Hello!″ in both
English and French for five seconds. Compile with cc_r or xlc_r. See ″Developing
Multi-Threaded Programs″ for more information on compiling thread programs.
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
/* include file for pthreads - the 1st */
/* include file for printf()
*/
/* include file for sleep()
*/
void *Thread(void *string)
{
while (1)
printf("%s\n", (char *)string);
pthread_exit(NULL);
}
int main()
{
char *e_str = "Hello!";
char *f_str = "Bonjour !";
pthread_t e_th;
pthread_t f_th;
int rc;
rc = pthread_create(&e_th, NULL, Thread, (void *)e_str);
if (rc)
exit(-1);
rc = pthread_create(&f_th, NULL, Thread, (void *)f_str);
if (rc)
exit(-1);
sleep(5);
/* usually the exit subroutine should not be used
see below to get more information */
exit(0);
}
The initial thread (executing the main routine) creates two threads. Both threads
have the same entry-point routine (the Thread routine), but a different parameter.
The parameter is a pointer to the string that will be displayed.
Terminating Threads
A thread automatically terminates when it returns from its entry-point routine. A
thread can also explicitly terminate itself or terminate any other thread in the
process. Because all threads share the same data space, a thread must perform
cleanup operations at termination time; cleanup handlers are provided by the
threads library for this purpose.
Exiting a Thread
A process can exit at any time by any thread by calling the exit subroutine.
Similarly, a thread can exit at any time by calling the pthread_exit subroutine.
Calling the exit subroutine terminates the entire process, including all its threads.
In a multi-threaded program, the exit subroutine should only be used when the
entire process needs to be terminated; for example, in the case of an unrecoverable
error. The pthread_exit subroutine should be preferred, even for exiting the initial
thread.
Calling the pthread_exit subroutine terminates the calling thread. The status
parameter is saved by the library and can be further used when joining the
terminated thread. Calling the pthread_exit subroutine is similar, but not identical,
212
General Programming Concepts
to returning from the thread’s initial routine. The result of returning from the
thread’s initial routine depends on the thread:
v Returning from the initial thread implicitly calls the exit subroutine, thus
terminating all the threads in the process.
v Returning from another thread implicitly calls the pthread_exit subroutine. The
return value has the same role as the status parameter of the pthread_exit
subroutine.
It is recommended always to use the pthread_exit subroutine to exit a thread to
avoid implicitly calling the exit subroutine.
Exiting the initial thread (for example by calling the pthread_exit subroutine from
the main routine) does not terminate the process. It only terminates the initial
thread. If the initial thread is terminated, the process will be terminated when the
last thread in it terminates. In this case, the process return code (usually the return
value of the main routine or the parameter of the exit subroutine) is 0 if the last
thread was detached or 1 otherwise.
The following example is a slightly modified version of our first multi-threaded
program. The program displays exactly 10 messages in each language. This is
accomplished by calling the pthread_exit subroutine in the main routine after
creating the two threads, and creating a loop in the Thread routine.
#include <pthread.h>
#include <stdio.h>
/* include file for pthreads - the 1st */
/* include file for printf()
*/
void *Thread(void *string)
{
int i;
for (i=0; i<10; i++)
printf("%s\n", (char *)string);
pthread_exit(NULL);
}
int main()
{
char *e_str = "Hello!";
char *f_str = "Bonjour !";
pthread_t e_th;
pthread_t f_th;
int rc;
rc = pthread_create(&e_th, NULL, Thread, (void *)e_str);
if (rc)
exit(-1);
rc = pthread_create(&f_th, NULL, Thread, (void *)f_str);
if (rc)
exit(-1);
pthread_exit(NULL);
}
It is important to note that the pthread_exit subroutine frees any thread-specific
data, including the thread’s stack. Any data allocated on the stack becomes invalid,
since the stack is freed and the corresponding memory may be reused by another
thread. Therefore, thread synchronization objects (mutexes and condition variables)
allocated on a thread’s stack must be destroyed before the thread calls the
pthread_exit subroutine.
Unlike the exit subroutine, the pthread_exit subroutine does not clean up system
resources shared among threads. For example, files are not closed by the
pthread_exit subroutine, since they may be used by other threads.
Chapter 11. Threads Programming Guidelines
213
Canceling a Thread
The thread cancellation mechanism allows a thread to terminate the execution of
any other thread in the process in a controlled manner. The target thread (that is,
the one that’s being canceled) can hold cancellation requests pending in a number
of ways and perform application-specific cleanup processing when the notice of
cancellation is acted upon. When canceled, the thread implicitly calls the
pthread_exit((void *)-1) subroutine.
The cancellation of a thread is requested by calling the pthread_cancel subroutine.
When the call returns, the request has been registered, but the thread may still be
running. The call to the pthread_cancel subroutine is unsuccessful only when the
specified thread ID is not valid.
Cancelability State and Type
The cancelability state and type of a thread determines the action taken upon
receipt of a cancellation request. Each thread controls its own cancelability state
and type with the pthread_setcancelstate and pthread_setcanceltype subroutines.
There are two possible cancelability states and two possible cancelability types,
leading to three possible cases, as shown in the following table.
Cancelability State
Cancelability Type
Resulting Case
Disabled
Any (the type is ignored)
Case 1
Enabled
Deferred
Case 2
Enabled
Asynchronous
Case 3
The following discusses the three possible cases.
1. Disabled cancelability. Any cancellation request is set pending, until the
cancelability state is changed or the thread is terminated in another way.
A thread should disable cancelability only when performing operations that
cannot be interrupted. For example, if a thread is performing some complex file
save operations (such as an indexed database) and is canceled during the
operation, the files may be left in an inconsistent state. To avoid this, the thread
should disable cancelability during the file save operations.
2. Deferred cancelability. Any cancellation request is set pending until the thread
reaches the next cancellation point. It is the default cancelability state.
This cancelability state ensures that a thread can be cancelled, but limits the
cancellation to specific moments in the thread’s execution, called cancellation
points. A thread canceled on a cancellation point leaves the system in a safe
state; however, user data may be inconsistent or locks may be held by the
canceled thread. To avoid these situations, you may use cleanup handlers or
disable cancelability within critical regions. See “Using Cleanup Handlers” on
page 218 for more information about cleanup handlers.
3. Asynchronous cancelability. Any cancellation request is acted upon immediately.
A thread that is asynchronously canceled while holding resources may leave
the process, or even the system, in a state from which it is difficult or
impossible to recover. See “Async-Cancel Safety” on page 215 for more
information about async-cancel safety.
214
General Programming Concepts
Async-Cancel Safety
A function is said to be async-cancel safe if it is written so that calling the function
with asynchronous cancelability enabled does not cause any resource to be
corrupted, even if a cancellation request is delivered at any arbitrary instruction.
Any function that gets a resource as a side effect cannot be made async-cancel safe.
For example, if the malloc subroutine is called with asynchronous cancelability
enabled, it might acquire the resource successfully, but as it was returning to the
caller, it could act on a cancellation request. In such a case, the program would
have no way of knowing whether the resource was acquired or not.
For this reason, most library routines cannot be considered async-cancel safe. It is
recommended not to use asynchronous cancelability unless you are sure only to
perform operations that do not hold resources and only to call async-cancel safe
library routines.
The following three subroutines are async-cancel safe; they ensure that cancellation
will be properly handled, even if asynchronous cancelability is enabled:
v pthread_cancel
v pthread_setcancelstate
v pthread_setcanceltype
An alternative to asynchronous cancelability is to use deferred cancelability and to
add explicit cancellation points by calling the pthread_testcancel subroutine (see
“Cancellation Points” for more information).
Cancellation Points
Cancellation points are points inside of certain subroutines where a thread must
act on any pending cancellation request if deferred cancelability is enabled. All
these subroutines may block the calling thread or compute indefinitely.
An explicit cancellation point can also be created by calling the pthread_testcancel
subroutine. This subroutine simply creates a cancellation point. If deferred
cancelability is enabled, and if a cancellation request is pending, the request is
acted upon and the thread is terminated. Otherwise, the subroutine simply returns.
Other cancellation points occur when calling the following subroutines:
v pthread_cond_wait
v pthread_cond_timedwait
v pthread_join
The pthread_mutex_lock and pthread_mutex_trylock subroutines do not provide a
cancellation point. If they did, all functions calling these subroutines (and many
functions do) would provide a cancellation point. Having too many cancellation
points makes programming very difficult, requiring either lots of disabling and
restoring of cancelability or extra effort in trying to arrange for reliable cleanup at
every possible place. See “Using Mutexes” on page 220 for more information about
these subroutines.
Cancellation points occur when a thread is executing the following functions:
aio_suspend
creat
close
fcntl
Chapter 11. Threads Programming Guidelines
215
fsync
getpmsg
mq_receive
msgrcv
msync
open
poll
pthread_cond_timedwait
pthread_join
putpmsg
read
select
sigpause
sigtimedwait
sigwaitinfo
system
usleep
wait3
waitpid
writev
getmsg
lockf
mq_send
msgsnd
nanosleep
pause
pread
pthread_cond_wait
pthread_testcancel
pwrite
readv
sem_wait
sigsuspend
sigwait
sleep
tcdrain
wait
waitid
write
A cancellation point may also occur when a thread is executing the following
functions:
catclose
closedir
dbm_close
dbm_nextkey
dlclose
endpwent
fwscanf
getchar
getdate
getgrgid_r
getlogin
printf
putchar
pututxline
putwchar
remove
endutxent
fflush
fgets
fopen
fputs
getpwnam_r
gets
getutxline
getwchar
scanf
setgrent
strerror
tmpnam
fputwc
freopen
fseeko
216
General Programming Concepts
catgets
closelog
dbm_delete
dbm_open
dlopen
fwprintf
getc
getchar_unlocked
getgrent
getgrnam
getlogin_r
putc
putchar_unlocked
putw
readdir
rename
fclose
fgetc
fgetwc
fprintf
getpwent
getpwuid
getutxent
getw
getwd
seekdir
setpwent
syslog
ttyname
fputws
fscanf
fsetpos
catopen
ctermid
dbm_fetch
dbm_store
endgrent
fwrite
getc_unlocked
getcwd
getgrgid
getgrnam_r
popen
putc_unlocked
puts
putwc
readdir_r
rewind
fcntl
fgetpos
fgetws
fputc
getpwnam
getpwuid_r
getutxid
getwc
rewinddir
semop
setutxent
tmpfile
ttyname_r
fread
fseek
ftell
ftello
iconv_close
lseek
opendir
perror
unlink
vprintf
wscanf
ftw
iconv_open
mkstemp
openlog
ungetc
vfprintf
vwprintf
glob
ioctl
nftw
pclose
ungetwc
vfwprintf
wprintf
The side effects of acting upon a cancellation request while suspended during a
call of a function is the same as the side effects that may be seen in a
single-threaded program when a call to a function is interrupted by a signal and
the given function returns [EINTR]. Any such side effects occur before any
cancellation cleanup handlers are called.
Whenever a thread has cancelability enabled and a cancellation request has been
made with that thread as the target and the thread calls pthread_testcancel, then
the cancellation request is acted upon before pthread_testcancel returns. If a thread
has cancelability enabled and the thread has an asynchronous cancellation request
pending and the thread is suspended at a cancellation point waiting for an event
to occur, then the cancellation request will be acted upon. However, if the thread is
suspended at a cancellation point and the event that it is waiting for occurs before
the cancellation request is acted upon, it is dependent upon the sequence of events
whether the cancellation request is acted upon or whether the request remains
pending and the thread resumes normal execution.
Cancellation Example
The following example is a variant of our first multi-threaded program. Both
″writer″ threads are canceled after 10 seconds, and after they have written their
message at least 5 times.
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
/* include file for pthreads - the 1st */
/* include file for printf()
*/
/* include file for sleep()
*/
void *Thread(void *string)
{
int i;
int o_state;
/* disables cancelability */
pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &o_state);
/* writes five messages */
for (i=0; i<5; i++)
printf("%s\n", (char *)string);
/* restores cancelability */
pthread_setcancelstate(o_state, &o_state);
/* writes further */
while (1)
printf("%s\n", (char *)string);
pthread_exit(NULL);
}
int main()
{
char *e_str = "Hello!";
char *f_str = "Bonjour !";
pthread_t e_th;
pthread_t f_th;
int rc;
/* creates both threads */
rc = pthread_create(&e_th, NULL, Thread, (void *)e_str);
if (rc)
Chapter 11. Threads Programming Guidelines
217
return -1;
rc = pthread_create(&f_th, NULL, Thread, (void *)f_str);
if (rc)
return -1;
/* sleeps a while */
sleep(10);
/* requests cancellation */
pthread_cancel(e_th);
pthread_cancel(f_th);
/* sleeps a bit more */
sleep(10);
pthread_exit(NULL);
}
Using Cleanup Handlers
Cleanup handlers provide an easy way to implement a portable mechanism for
releasing resources and restoring invariants when a thread terminates.
Calling Cleanup Handlers
Cleanup handlers are specific to each thread. A thread can have several cleanup
handlers; cleanup handlers are stored in a thread-specific LIFO stack. They are all
called in the following cases:
v The thread returns from its entry-point routine.
v The thread calls the pthread_exit subroutine.
v The thread acts on a cancellation request.
A cleanup handler is pushed onto the cleanup stack, by the pthread_cleanup_push
subroutine. The pthread_cleanup_pop subroutine pops the topmost cleanup
handler from the stack, and optionally executes it. Use this subroutine when the
cleanup handler is no longer needed.
The cleanup handler is a user-defined routine. It has one parameter, a void pointer,
specified when calling the pthread_cleanup_push subroutine. You may specify a
pointer to some data the cleanup handler needs to perform its operation.
In the following example, a buffer is allocated for performing some operation. With
deferred cancelability enabled, the operation may be stopped at any cancellation
point. A cleanup handler is established to free the buffer in that case.
/* the cleanup handler */
cleaner(void *buffer)
{
free(buffer);
}
/* fragment of another routine */
...
myBuf = malloc(1000);
if (myBuf != NULL) {
pthread_cleanup_push(cleaner, myBuf);
/*
*
perform any operation using the buffer,
*
including calls to other functions
*
and cancellation points
*/
/* pops the handler and frees the buffer in one call */
pthread_cleanup_pop(1);
}
218
General Programming Concepts
Using deferred cancelability ensures that the thread will not act on any cancellation
request between the buffer allocation and the registration of the cleanup handler,
because neither the malloc subroutine nor the pthread_cleanup_push subroutine
provides any cancellation point. When popping the cleanup handler, the handler is
executed, freeing the buffer. More complex programs may not execute the handler
when popping it, because the cleanup handler should be thought of as an
emergency exit for the protected portion of code.
Balancing the Push and Pop Operations
The pthread_cleanup_push and pthread_cleanup_pop subroutines should always
appear in pairs within the same lexical scope, that is, within the same function and
the same statement block. They can be thought of as left and right parentheses
enclosing a protected portion of code.
The reason for this rule is that on some systems these subroutines are implemented
as macros. The pthread_cleanup_push subroutine is implemented as a left brace,
followed by other statements:
#define pthread_cleanup_push(rtm,arg) { \
/* other statements */
The pthread_cleanup_pop subroutine is implemented as a right brace following
other statements:
#define pthread_cleanup_pop(ex) \
/* other statements */ \
}
Not following the balancing rule for the pthread_cleanup_push and
pthread_cleanup_pop subroutines may lead to compiler errors or to unexpected
behavior of your programs when porting to other systems.
In AIX, the pthread_cleanup_push and pthread_cleanup_pop subroutines are
library routines, and can be unbalanced within the same statement block. However,
they must be balanced in the program, since the cleanup handlers are stacked.
List of Threads Basic Operation Subroutines
pthread_attr_destroy
pthread_attr_getdetachstate
pthread_attr_init
pthread_create
pthread_cancel
pthread_cleanup_pop
pthread_cleanup_push
pthread_equal
pthread_exit
pthread_self
pthread_setcancelstate
pthread_setcanceltype
pthread_testcancel
Deletes a thread attributes object.
Returns the value of the detachstate attribute of a
thread attributes object.
Creates a thread attributes object and initializes it
with default values.
Creates a new thread, initializes its attributes, and
makes it runnable.
Requests the cancellation of a thread.
Removes, and optionally executes, the routine at
the top of the calling thread’s cleanup stack.
Pushes a routine onto the calling thread’s cleanup
stack.
Compares two thread IDs.
Terminates the calling thread.
Returns the calling thread’s ID.
Sets the calling thread’s cancelability state.
Sets the calling thread’s cancelability type.
Creates a cancellation point in the calling thread.
Chapter 11. Threads Programming Guidelines
219
Synchronization Overview
One main benefit of using threads is the ease of using synchronization facilities.
Three basic synchronization techniques are implemented in the threads library:
mutexes, condition variables, and joining. More complex synchronization objects
can be built using the primitive objects. This is discussed in 250.
Using Mutexes
A mutex is a mutual exclusion lock. Only one thread can hold the lock. Mutexes
are used to protect data or other resources from concurrent access. A mutex has
attributes, which specify the characteristics of the mutex. In the current version of
AIX, the mutex attributes are not used. The mutex attributes object can therefore be
ignored when creating a mutex.
Mutex Attributes Object
Like threads, mutexes are created with the help of an attributes object. The mutex
attributes object is an abstract object, containing several attributes, depending on
the implementation of POSIX options. It is accessed through a variable of type
pthread_mutexattr_t. In AIX, the pthread_mutexattr_t data type is a pointer; on
other systems, it may be a structure or another data type.
Mutex Attributes Object Creation and Destruction
The mutex attributes object is initialized to default values by the
pthread_mutexattr_init subroutine. The attributes are handled by subroutines. The
thread attributes object is destroyed by the pthread_mutexattr_destroy subroutine.
This subroutine may free storage dynamically allocated by the
pthread_mutexattr_init subroutine, depending on the implementation of the
threads library.
In the following example, a mutex attributes object is created and initialized with
default values, then used and finally destroyed:
pthread_mutexattr_t attributes;
/* the attributes object is created */
...
if (!pthread_mutexattr_init(&attributes)) {
/* the attributes object is initialized */
...
/* using the attributes object */
...
pthread_mutexattr_destroy(&attributes);
/* the attributes object is destroyed */
}
The same attributes object can be used to create several mutexes. It can also be
modified between two mutex creations. When the mutexes are created, the
attributes object can be destroyed without affecting the mutexes created with it.
Mutex Attributes
In AIX, no mutex attribute is defined. They depend on POSIX options that are not
implemented in AIX. However, the following attributes may be defined on other
systems:
220
General Programming Concepts
Protocol
Prioceiling
Process-shared
Specifies the protocol used to prevent priority inversions for a mutex.
This attribute depends on either the priority inheritance or the
priority protection POSIX option.
Specifies the priority ceiling of a mutex. This attribute depends on
the priority protection POSIX option.
Specifies the process sharing of a mutex. This attribute depends on
the process sharing POSIX option.
The default values for these attributes are sufficient for most simple cases. See
“Synchronization Scheduling” on page 239 for more information about the protocol
and prioceiling attributes; see 248 for more information about the process-shared
attribute.
Creating and Destroying Mutexes
A mutex is created by calling the pthread_mutex_init subroutine. You may specify
a mutex attributes object. If you specify a NULL pointer, the mutex will have the
default attributes. Thus, the code fragment:
pthread_mutex_t mutex;
pthread_mutex_attr_t attr;
...
pthread_mutexattr_init(&attr);
pthread_mutex_init(&mutex, &attr);
pthread_mutexattr_destroy(&attr);
is equivalent to:
pthread_mutex_t mutex;
...
pthread_mutex_init(&mutex, NULL);
The ID of the created mutex is returned to the calling thread through the mutex
parameter. The mutex ID is an opaque object; its type is pthread_mutex_t. In AIX,
the pthread_mutex_t data type is a structure; on other systems, it may be a pointer
or another data type.
A mutex must be created once. Calling the pthread_mutex_init subroutine more
than once with the same mutex parameter (for example, in two threads
concurrently executing the same code) should be avoided. The second call will fail,
returning an EBUSY error code. Ensuring the uniqueness of a mutex creation can
be done in three ways:
v Calling the pthread_mutex_init subroutine prior to the creation of other threads
that will use this mutex; in the initial thread, for example.
v Calling the pthread_mutex_init subroutine within a one time initialization
routine; see “One-Time Initializations” on page 243.
v Using a static mutex initialized by the PTHREAD_MUTEX_INITIALIZER static
initialization macro; the mutex will have default attributes.
Once the mutex is no longer needed, it should be destroyed by calling the
pthread_mutex_destroy subroutine. This subroutine may reclaim any storage
allocated by the pthread_mutex_init subroutine. After having destroyed a mutex,
the same pthread_mutex_t variable can be reused for creating another mutex. For
example, the following code fragment is legal, although not very realistic:
pthread_mutex_t mutex;
...
for (i = 0; i < 10; i++) {
/* creates a mutex */
Chapter 11. Threads Programming Guidelines
221
pthread_mutex_init(&mutex, NULL);
/* uses the mutex */
/* destroys the mutex */
pthread_mutex_destroy(&mutex);
}
Like any system resource that can be shared among threads, a mutex allocated on
a thread’s stack must be destroyed before the thread is terminated. The threads
library maintains a linked list of mutexes; thus if the stack where a mutex is
allocated is freed, the list will be corrupted.
Locking and Unlocking Mutexes
A mutex is a simple lock, having two states: locked and unlocked. When it is
created, a mutex is unlocked. The pthread_mutex_lock subroutine locks the
specified mutex:
v If the mutex is unlocked, the subroutine locks it.
v If the mutex is already locked by another thread, the subroutine blocks the
calling thread until the mutex is unlocked.
v If the mutex is already locked by the calling thread, the subroutine returns an
error.
The pthread_mutex_trylock subroutine acts like the pthread_mutex_lock
subroutine without blocking the calling thread:
v If the mutex is unlocked, the subroutine locks it.
v If the mutex is already locked by any thread, the subroutine returns an error.
The thread that locked a mutex is often called the owner of the mutex.
The pthread_mutex_unlock subroutine resets the specified mutex to the unlocked
state if it is owned by the calling mutex:
v If the mutex was already unlocked, the subroutine returns an error.
v If the mutex was owned by the calling thread, the subroutine unlocks the mutex.
v If the mutex was owned by another thread, the subroutine returns an error.
Because locking does not provide a cancellation point (see “Cancellation Points” on
page 215), a thread blocked while waiting for a mutex cannot be canceled
(“Canceling a Thread” on page 214). Therefore, it is recommended to use mutexes
only for short periods of time, like protecting data from concurrent access.
Protecting Data with Mutexes
Mutexes are intended to serve either as a low level primitive from which other
thread synchronization functions can be built or as a data protection lock. Making
Complex Synchronization Objects provides more information about implementing
long locks and writer-priority readers/writers locks with mutexes.
Mutex Usage Example
Mutexes can be used to protect data from concurrent access. For example, a
database application may create several threads to handle several requests
concurrently. The database itself is protected by a mutex, called db_mutex.
/* the initial thread */
pthread_mutex_t mutex;
int i;
...
pthread_mutex_init(&mutex, NULL);
222
General Programming Concepts
/* creates the mutex
*/
for (i = 0; i < num_req; i++)
/* loop to create threads */
pthread_create(th + i, NULL, rtn, &mutex);
...
/* waits end of session */
pthread_mutex_destroy(&mutex);
/* destroys the mutex
*/
...
/* the request handling thread */
...
pthread_mutex_lock(&db_mutex);
...
pthread_mutex_unlock(&db_mutex);
...
/*
/*
/*
/*
waits for a request
locks the database
handles the request
unlocks the database
*/
*/
*/
*/
The initial thread creates the mutex and all the request handling threads. The
mutex is passed to the thread using the parameter of the thread’s entry point
routine. In a real program, the address of the mutex may be a field of a more
complex data structure passed to the created thread.
Avoiding Deadlocks
In AIX, mutexes cannot be re-locked by the same thread. This may not be the case
on other systems. To enhance portability of your programs, assume that the
following code fragment may produce a deadlock:
pthread_mutex_lock(&mutex);
pthread_mutex_lock(&mutex);
This kind of deadlock may occur when locking a mutex and then calling a routine
that will itself lock the same mutex. For example:
pthread_mutex_t mutex;
struct {
int a;
int b;
int c;
} A;
f()
{
pthread_mutex_lock(&mutex);
A.a++;
g();
A.c = 0;
pthread_mutex_unlock(&mutex);
/* call 1 */
pthread_mutex_lock(&mutex);
A.b += A.a;
pthread_mutex_unlock(&mutex);
/* call 2 */
}
g()
{
/* call 3 */
}
On some non-AIX systems, calling the f subroutine would produce a deadlock; call
2 would block the thread, because call 1 already locked the mutex. In AIX, this
code fragment would still not have the expected behavior. Call 2 would be
unsuccessful, but call 3 would succeed. Thus, when returning for the g subroutine,
the mutex would already be unlocked and the A variable would no longer be
protected; when returning from the f routine, the A.c variable may not contain
zero.
To avoid this kind of deadlock or data inconsistency, you should use either one of
the following schemes:
v Fine granularity locking. Each data atom should be protected by a mutex, locked
only by low-level functions. For example, this would result in locking each
Chapter 11. Threads Programming Guidelines
223
record of a database. Benefits: high-level functions do not need to care about
locking data. Drawbacks: it increases the number of mutexes, and great care
should be taken to avoid deadlocks.
v High-level locking. Data should be organized into areas, each area protected by a
mutex; low-level functions do not need to care about locking. For example, this
would result in locking a whole database before accessing it. Benefits: there are
few mutexes, and thus few risks of deadlocks. Drawbacks: performance may be
bad, especially if many threads want access to the same data.
Deadlocks may also occur when locking mutexes in reverse order. For example, the
following code fragment may produce a deadlock between threads A and B:
/* Thread A */
pthread_mutex_lock(&mutex1);
pthread_mutex_lock(&mutex2);
/* Thread B */
pthread_mutex_lock(&mutex2);
pthread_mutex_lock(&mutex1);
To avoid these kinds of deadlocks, you should ensure that successive mutexes are
always locked in the same order.
Using Condition Variables
Condition variables allow threads to wait until some event or condition has
occurred. Typically, a program will use three objects:
v A boolean variable, indicating whether the condition is met
v A mutex to serialize the access to the boolean variable
v A condition variable to wait for the condition.
Using a condition variable requires some effort from the programmer. However,
condition variables allow the implementation of powerful and efficient
synchronization mechanisms. See 250 for more information about implementing
long locks and semaphores with condition variables.
A condition variable has attributes, which specify the characteristics of the
condition. In the current version of AIX, the condition attributes are not used.
Therefore, the condition attributes object can be ignored when creating a condition
variable.
Condition Attributes Object
Like threads and mutexes, condition variables are created with the help of an
attributes object. The condition attributes object is an abstract object, containing at
most one attribute, depending on the implementation of POSIX options. It is
accessed through a variable of type pthread_condattr_t. In AIX, the
pthread_condattr_t data type is a pointer; on other systems, it may be a structure
or another data type.
Condition Attributes Object Creation and Destruction
The mutex attributes object is initialized to default values by the
pthread_condattr_init subroutine. The attribute is handled by subroutines. The
thread attributes object is destroyed by the pthread_condattr_destroy subroutine.
224
General Programming Concepts
This subroutine may free storage dynamically allocated by the
pthread_condattr_init subroutine, depending on the implementation of the threads
library.
In the following example, a mutex attributes object is created and initialized with
default values, then used and finally destroyed:
pthread_condattr_t attributes;
/* the attributes object is created */
...
if (!pthread_condattr_init(&attributes)) {
/* the attributes object is initialized */
...
/* using the attributes object */
...
pthread_condattr_destroy(&attributes);
/* the attributes object is destroyed */
}
The same attributes object can be used to create several condition variables. It can
also be modified between two condition variable creations. When the condition
variables are created, the attributes object can be destroyed without affecting the
condition variables created with it.
Condition Attribute
In AIX, no condition attribute is defined. Condition attributes depend on POSIX
options that are not implemented in AIX. However, the following attribute may be
defined on other systems:
Process-shared
Specifies the process sharing of a condition variable. This attribute
depends on the process sharing POSIX option.
See 248 for more information about the process-shared attribute.
Creating and Destroying Condition Variables
A condition variable is created by calling the pthread_cond_init subroutine. You
may specify a condition attributes object. If you specify a NULL pointer, the
condition variable will have the default attributes. Thus, the code fragment:
pthread_cond_t cond;
pthread_condattr_t attr;
...
pthread_condattr_init(&attr);
pthread_cond_init(&cond, &attr);
pthread_condattr_destroy(&attr);
is equivalent to:
pthread_cond_t cond;
...
pthread_cond_init(&cond, NULL);
The ID of the created condition variable is returned to the calling thread through
the condition parameter. The condition ID is an opaque object; its type is
pthread_cond_t. In AIX, the pthread_cond_t data type is a structure; on other
systems it may be a pointer or another data type.
A condition variable must be created once. Calling the pthread_cond_init
subroutine more than once with the same condition parameter (for example, in two
threads concurrently executing the same code) should be avoided. The second call
Chapter 11. Threads Programming Guidelines
225
will fail, returning an EBUSY error code. Ensuring the uniqueness of a condition
variable creation can be done in three ways:
v Calling the pthread_cond_init subroutine prior to the creation of other threads
that will use this variable; in the initial thread, for example.
v Calling the pthread_cond_init subroutine within a one-time initialization routine
(“One-Time Initializations” on page 243).
v Using a static condition variable initialized by the
PTHREAD_COND_INITIALIZER static initialization macro; the condition
variable will have default attributes.
Once the condition variable is no longer needed, it should be destroyed by calling
the pthread_cond_destroy subroutine. This subroutine may reclaim any storage
allocated by the pthread_cond_init subroutine. After having destroyed a condition
variable, the same pthread_cond_t variable can be reused for creating another
condition. For example, the following code fragment is legal, although not very
realistic:
pthread_cond_t cond;
...
for (i = 0; i < 10; i++) {
/* creates a condition variable */
pthread_cond_init(&cond, NULL);
/* uses the condition variable */
/* destroys the condition */
pthread_cond_destroy(&cond);
}
Like any system resource that can be shared among threads, a condition variable
allocated on a thread’s stack must be destroyed before the thread is terminated.
The threads library maintains a linked list of condition variables; thus if the stack
where a mutex is allocated is freed, the list will be corrupted.
Using Condition Variables
A condition variable must always be used together with a mutex. The same mutex
must be used for the same condition variable, even for different threads. It is
possible to bundle in a structure the condition, the mutex, and the condition
variable, as shown in the following code fragment:
struct condition_bundle_t {
int
condition_predicate;
pthread_mutex_t condition_lock;
pthread_cond_t condition_variable;
};
See “Synchronizing Threads with Condition Variables” on page 228 for more
information about using the condition predicate.
Waiting for a Condition
The mutex protecting the condition must be locked before waiting for the
condition. A thread can wait for a condition to be signaled by calling the
pthread_cond_wait or pthread_cond_timedwait subroutine. The subroutine
atomically unlocks the mutex and blocks the calling thread until the condition is
signaled. When the call returns, the mutex is locked again.
The pthread_cond_wait subroutine blocks the thread indefinitely. If the condition
is never signaled, the thread never wakes up. Because the pthread_cond_wait
226
General Programming Concepts
subroutine provides a cancellation point, the only way to get out of this deadlock
is to cancel (see “Canceling a Thread” on page 214) the blocked thread, if
cancelability is enabled.
The pthread_cond_timedwait subroutine blocks the thread only for a given period
of time. This subroutine has an extra parameter, timeout, specifying an absolute
date where the sleep must end. The timeout parameter is a pointer to a timespec
structure. This data type is also called timestruc_t. It contains two fields:
tv_sec
tv_nsec
A long unsigned integer, specifying seconds
A long integer, specifying nanoseconds.
Typically, the pthread_cond_timedwait subroutine is used in the following
manner:
struct timespec timeout;
...
time(&timeout.tv_sec);
timeout.tv_sec += MAXIMUM_SLEEP_DURATION;
pthread_cond_timedwait(&cond, &mutex, &timeout);
The timeout parameter specifies an absolute date. The previous code fragment
shows how to specify a duration rather than an absolute date.
To use pthread_cond_timedwait with an absolute date, you can use the mktime
subroutine to calculate the value of the tv_sec field of the timespec structure. In
the following example, the thread will wait for the condition until 08:00 January 1,
2001, local time.
struct tm
date;
time_t
seconds;
struct timespec timeout;
...
date.tm_sec = 0;
date.tm_min = 0;
date.tm_hour = 8;
date.tm_mday = 1;
date.tm_mon = 0;
date.tm_year = 101;
date.tm_wday = 1;
/* the range is 0-11 */
/* 0 is 1900 */
/* this field can be omitted but it will really be a Monday! */
date.tm_yday = 0;
/* first day of the year */
date.tm_isdst = daylight;
/* daylight is an external variable - we are assuming
that daylight savings time will still be used... */
seconds = mktime(&date);
timeout.tv_sec = (unsigned long)seconds;
timeout.tv_nsec = 0L;
pthread_cond_timedwait(&cond, &mutex, &timeout);
The pthread_cond_timedwait subroutine also provides a cancellation point,
although the sleep is not indefinite. Thus, a sleeping thread can be canceled,
whether the sleep has a timeout or not.
Signaling a Condition
A condition can be signaled by calling either the pthread_cond_signal or the
pthread_cond_broadcast subroutine.
The pthread_cond_signal subroutine wakes up at least one thread that is currently
blocked on the specified condition. The awoken thread is chosen according to the
Chapter 11. Threads Programming Guidelines
227
scheduling policy; it is the thread with the most-favored scheduling priority (see
“Scheduling Policy and Priority” on page 236). It may happen on multiprocessor
systems, or some non-AIX systems, that more than one thread is woken up. Do not
assume that this subroutine wakes up exactly one thread.
The pthread_cond_broadcast subroutine wakes up every thread that is currently
blocked on the specified condition. However, a thread can start waiting on the
same condition just after the call to the subroutine returns.
A call to these routines always succeeds, unless an invalid cond parameter is
specified. This does not mean that a thread has been awakened. Furthermore,
signaling a condition is not remembered by the library. For example, consider a
condition C. No thread is waiting on this condition. At time t, thread 1 signals the
condition C. The call is successful although no thread is woken up. At time t+1,
thread 2 calls the pthread_cond_wait subroutine with C as cond parameter. Thread
2 is blocked. If no other thread signals C, thread 2 may wait until the process
terminates.
A way to avoid this kind of deadlock is to check the EBUSY error code returned
by the pthread_cond_destroy subroutine when destroying the condition variable,
as in the following code fragment:
while (pthread_cond_destroy(&cond) == EBUSY) {
pthread_cond_broadcast(&cond);
pthread_yield();
}
The pthread_yield subroutine gives the opportunity to another thread to be
scheduled, one of the awoken threads for example. See Threads Scheduling for
more information about the pthread_yield subroutine.
The pthread_cond_wait and the pthread_cond_broadcast subroutines must not be
used within a signal handler. To provide a convenient way for a thread to await a
signal, the threads library provides the sigwait subroutine. See “Signal
Management” on page 255 for more information about the sigwait subroutine.
Synchronizing Threads with Condition Variables
Condition variables are used to wait until a particular predicate becomes true. This
predicate is set by another thread, usually the one that signals the condition.
Condition Wait Semantics
A predicate must be protected by a mutex. When waiting for a condition, the wait
subroutine (either pthread_cond_wait or pthread_cond_timedwait) atomically
unlocks the mutex and blocks the thread. When the condition is signaled, the
mutex is relocked and the wait subroutine returns. It is important to note that
when the subroutine returns without error, the predicate may still be false.
The reason is that more than one thread may be awoken: either a thread called the
pthread_cond_broadcast subroutine, or an unavoidable race between two
processors simultaneously woke two threads. The first thread locking the mutex
will block all other awoken threads in the wait subroutine until the mutex is
unlocked by the program. Thus, the predicate may have changed when the second
thread gets the mutex and returns from the wait subroutine.
228
General Programming Concepts
In general, whenever a condition wait returns, the thread should re-evaluate the
predicate to determine whether it can safely proceed, should wait again, or should
declare a timeout. A return from the wait subroutine does not imply that the
predicate is either true or false.
It is recommended that a condition wait be enclosed in a ″while loop″ that checks
the predicate. The following code fragment provides a basic implementation of a
condition wait.
pthread_mutex_lock(&condition_lock);
while (condition_predicate == 0)
pthread_cond_wait(&condition_variable, &condition_lock);
...
pthread_mutex_unlock(&condition_lock);
Timed Wait Semantics
When the pthread_cond_timedwait subroutine returns with the timeout error, the
predicate may be true. This is due to another unavoidable race between the
expiration of the timeout and the predicate state change.
Just as for non-timed wait, the thread should re-evaluate the predicate when a
timeout occurred to determine whether it should declare a timeout or should
proceed anyway. It is recommended to carefully check all possible cases when the
pthread_cond_timedwait subroutine returns. The following code fragment shows
how such checking could be implemented in a robust program:
int result = CONTINUE_LOOP;
pthread_mutex_lock(&condition_lock);
while (result == CONTINUE_LOOP) {
switch (pthread_cond_timedwait(&condition_variable,
&condition_lock, &timeout)) {
case 0:
if (condition_predicate)
result = PROCEED;
break;
case ETIMEDOUT:
result = condition_predicate ? PROCEED : TIMEOUT;
break;
default:
result = ERROR;
break;
}
}
...
pthread_mutex_unlock(&condition_lock);
The result variable can be used to choose an action. The statements preceding the
unlocking of the mutex should be as quick as possible, because a mutex should not
be held for long periods of time.
Specifying an absolute date in the timeout parameter allows easy implementation of
real-time behavior. An absolute timeout does not need to be recomputed if it is
used multiple times in a loop, such as that enclosing a condition wait. For cases
where the system clock is advanced discontinuously by an operator, using an
absolute timeout ensures that the timed wait will end as soon as the system time
specifies a date later than the timeout parameter.
Chapter 11. Threads Programming Guidelines
229
Condition Variables Usage Example
The following example provides the source code for a synchronization point
routine. A synchronization point is a given point in a program where different
threads must wait until all threads (or at least a certain number of threads) have
reached that point.
A synchronization point can simply be implemented by a counter, which is
protected by a lock, and a condition variable. Each thread takes the lock,
increments the counter, and waits for the condition to be signaled if the counter
did not reach its maximum. Otherwise, the condition is broadcast, and all threads
can proceed. The last thread calling the routine broadcasts the condition.
#define SYNC_MAX_COUNT 10
void SynchronizationPoint()
{
/* use static variables to ensure initialization */
static mutex_t sync_lock = PTHREAD_MUTEX_INITIALIZER;
static cond_t sync_cond = PTHREAD_COND_INITIALIZER;
static int sync_count = 0;
/* lock the access to the count */
pthread_mutex_lock(&sync_lock);
/* increment the counter */
sync_count++;
/* check if we should wait or not */
if (sync_count < SYNC_MAX_COUNT)
/* wait for the others */
pthread_cond_wait(&sync_cond, &sync_lock);
else
/* broadcast that everybody reached the point */
pthread_cond_broadcast(&sync_cond);
/* unlocks the mutex - otherwise only one thread
will be able to return from the routine! */
pthread_mutex_unlock(&sync_lock);
}
This routine has some limitations: it can be used only once, and the number of
threads that will call the routine is coded by a symbolic constant. However, this
example shows a basic usage of condition variables. More complex usage can be
found in 250.
Joining Threads
Joining a thread means waiting for it to terminate. It can be seen as a specific
usage of condition variables.
Waiting for a Thread
The pthread_join subroutine provides a simple mechanism allowing a thread to
wait for another thread to terminate. More complex conditions, such as waiting for
multiple threads to terminate, can be implemented by the programmer using
condition variables. See “Synchronizing Threads with Condition Variables” on
page 228 for more information.
Calling the pthread_join Subroutine
The pthread_join subroutine blocks the calling thread until the specified thread
terminates. The target thread (the thread whose termination is awaited) must not
be detached. If the target thread is already terminated, but not detached, the
230
General Programming Concepts
pthread_join subroutine returns immediately. Once a target thread has been joined,
it is automatically detached, and its storage can be reclaimed.
The following table indicates the two possible cases when a thread calls the
pthread_join subroutine, depending on the state and the detachstate attribute of
the target thread.
Undetached target
Detached target
Target is still running
The caller is blocked until
the target is terminated.
The call returns immediately
indicating an error.
Target is terminated
The call returns immediately
indicating a successful
completion.
A thread cannot join itself - a deadlock would occur and it is detected by the
library. However, two threads may try to join each other; they will deadlock. This
situation is not detected by the library.
Multiple Joins
It is possible for several threads to join the same target thread, if the target is not
detached. The success of this operation depends on the order of the calls to the
pthread_join subroutine and the moment when the target thread terminates.
v Any call to the pthread_join subroutine occurring before the target thread’s
termination blocks the calling thread.
v When the target thread terminates, all blocked threads are awoken, and the
target thread is automatically detached.
v Any call to the pthread_join subroutine occurring after the target thread’s
termination will fail, because the thread is detached by the previous join.
v If no thread called the pthread_join subroutine before the target thread’s
termination, the first call to the pthread_join subroutine will return immediately,
indicating a successful completion, and any further call will fail.
The following figure illustrates the two possible cases.
Chapter 11. Threads Programming Guidelines
231
First case: pthread_join called before target termination
Thread 1 (target)
0
Thread 2
0
Thread 3
ESRCH
Thread 4
Second case: pthread_join called after target termination
Thread 1 (target)
0
Thread 2
ESRCH
Thread 3
Keys:
Running (or runnable) thread
Blocked thread
Terminated but not detached thread
Call to the pthread_join subroutine
Return from the pthread_join subroutine, with the return value
Multiple Joins
Join Example
The following example is an enhanced version of the first multi-threaded program.
The program ends after exactly five messages in each language are displayed. This
is done by blocking the initial thread until the ″writer″ threads exit.
#include <pthread.h>
#include <stdio.h>
/* include file for pthreads - the 1st */
/* include file for printf()
*/
void *Thread(void *string)
{
int i;
/* writes five messages and exits */
for (i=0; i<5; i++)
printf("%s\n", (char *)string);
pthread_exit(NULL);
}
int main()
{
char *e_str = "Hello!";
char *f_str = "Bonjour !";
232
General Programming Concepts
pthread_attr_t attr;
pthread_t e_th;
pthread_t f_th;
int rc;
/* creates the right attribute */
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr,
PTHREAD_CREATE_UNDETACHED);
/* creates both threads */
rc = pthread_create(&e_th, &attr, Thread, (void *)e_str);
if (rc)
exit(-1);
rc = pthread_create(&f_th, &attr, Thread, (void *)f_str);
if (rc)
exit(-1);
pthread_attr_destroy(&attr);
/* joins the threads */
pthread_join(e_th, NULL);
pthread_join(f_th, NULL);
pthread_exit(NULL);
}
Returning Information from a Thread
The pthread_join subroutine also allows a thread to return information to another
thread. When a thread calls the pthread_exit subroutine or when it returns from its
entry-point routine, it returns a pointer (see “Exiting a Thread” on page 212). This
pointer is stored as long as the thread is not detached, and the pthread_join
subroutine can return it.
For example, a multi-threaded grep command may choose the following
implementation. The initial thread creates one thread per file to scan, each thread
having the same entry point routine. It then waits for all threads to be terminated.
Each ″scanning″ thread stores the found lines in a dynamically allocated buffer and
returns a pointer to this buffer. The initial thread prints out each buffer and frees it.
/* "scanning" thread */
...
buffer = malloc(...);
/* finds the search pattern in the file
and stores the lines in the buffer */
return (buffer);
/* initial thread */
...
for (/* each created thread */) {
void *buf;
pthread_join(thread, &buf);
if (buf != NULL) {
/* print all the lines in the buffer,
preceded by the filename of the thread */
free(buf);
}
}
...
If the target thread is canceled (see “Canceling a Thread” on page 214) the
pthread_join subroutine returns a value of -1 cast into a pointer. Because -1 cannot
be a pointer value, getting -1 as returned pointer from a thread means that the
thread was canceled.
The returned pointer can point to any kind of data. Care must be taken concerning
the storage class of the data the pointer refers to. The pointer must be still valid
Chapter 11. Threads Programming Guidelines
233
after the thread was terminated and its storage reclaimed. Therefore, returning a
thread-specific data value should be avoided, because the destructor routine is
called when the thread’s storage is reclaimed.
Returning a pointer to dynamically allocated storage to several threads should also
be handled with care. Consider the following code fragment:
void *returned_data;
...
pthread_join(target_thread, &returned_data);
/* retrieves information from returned_data */
free(returned_data);
When executed by only one thread, the returned_data pointer is freed as it should
be. If several threads execute this code fragment concurrently, the returned_data
pointer is freed several times; this must be avoided. A solution may consist in
using a flag, protected by a mutex, to signal that the returned_data pointer was
freed. The line:
free(returned_data);
would thus be replaced by the lines (assuming the flag variable is initially 0)
/* lock - entering a critical region, no other thread should
run this portion of code concurrently */
if (!flag) {
free(returned_data);
flag = 1;
}
/* unlock - exiting the critical region */
where a mutex can be used for locking the access to the critical region. This
ensures that the returned_data pointer is freed only once.
When returning a pointer to dynamically allocated storage to several threads all
executing different code, you must ensure that exactly one thread frees the pointer.
List of Synchronization Subroutines
pthread_mutex_destroy
pthread_mutex_init
PTHREAD_MUTEX_INITIALIZER
pthread_mutex_lock or
pthread_mutex_trylock
pthread_mutex_unlock
pthread_mutexattr_destroy
pthread_mutexattr_init
pthread_cond_destroy
pthread_cond_init
PTHREAD_COND_INITIALIZER
pthread_cond_signal or
pthread_cond_broadcast
pthread_cond_wait or
pthread_cond_timedwait
pthread_condattr_destroy
234
General Programming Concepts
Deletes a mutex.
Initializes a mutex and sets its attributes.
Initializes a static mutex with default
attributes.
Locks a mutex.
Unlocks a mutex.
Deletes a mutex attributes object.
Creates a mutex attributes object and
initializes it with default values.
Deletes a condition variable.
Initializes a condition variable and sets its
attributes.
Initializes a static condition variable with
default attributes.
Unblocks one or more threads blocked on a
condition.
Blocks the calling thread on a condition.
Deletes a condition attributes object.
pthread_condattr_init
Creates a condition attributes object and
initializes it with default values.
Scheduling Overview
Threads are the schedulable entity. The threads library provides several facilities to
handle and control the scheduling of threads. It also provides facilities to control
the scheduling of threads during synchronization operations such as locking a
mutex.
The following information will help you in using the scheduling facilities:
v “Threads Scheduling”
v “Synchronization Scheduling” on page 239
v List of Scheduling Subroutines
Threads Scheduling
Each thread has its own set of scheduling parameters. These parameters can be set
using the thread attributes object before the thread’s creation. They can also be
dynamically set during the thread’s execution.
Basic Scheduling Facilities
Controlling the scheduling of a thread is often a complicated task. Because the
scheduler handles all threads systemwide, the scheduling parameters of a thread
interact with those of all other threads in the process and in the other processes.
The following facilities are the first to be used if you want to control the
scheduling of a thread.
Inheritsched Attribute
The inheritsched attribute of the thread attributes object specifies how the thread’s
scheduling attributes will be defined. It may have one of the following values:
PTHREAD_INHERIT_SCHED
PTHREAD_EXPLICIT_SCHED
Specifies that the new thread will get the scheduling
attributes (schedpolicy and schedparam attributes) of its
creating thread. Scheduling attributes defined in the
attributes object are ignored.
Specifies that the new thread will get the scheduling
attributes defined in this attributes object.
The default value of the inheritsched attribute is PTHREAD_INHERIT_SCHED.
The attribute is set by calling the pthread_attr_setinheritsched subroutine. The
current value of the attribute is returned by calling the
pthread_attr_getinheritsched subroutine.
To set the scheduling attributes of a thread in the thread attributes object, the
inheritsched must first be set to PTHREAD_EXPLICIT_SCHED. Otherwise, the
attributes object scheduling attributes are ignored.
Chapter 11. Threads Programming Guidelines
235
Scheduling Policy and Priority
The threads library provides three scheduling policies:
First-in first-out (FIFO) scheduling. Each thread has a fixed priority; when
multiple threads have the same priority level, they run to completion in
FIFO order.
SCHED_RR
Round-robin (RR) scheduling. Each thread has a fixed priority; when
multiple threads have the same priority level, they run for a fixed time
slice in FIFO order.
SCHED_OTHER Default AIX scheduling. Each thread has a initial priority that is
dynamically modified by the scheduler, according to the thread’s activity;
thread execution is time-sliced. On other systems, this scheduling policy
may be different.
SCHED_FIFO
The default scheduling policy for threads is SCHED_OTHER.
The priority is an integer value, in the range from 1 to 127. 1 is the least-favored
priority, 127 is the most-favored. Priority level 0 cannot be used: it is reserved for
the system. Note that in AIX, the kernel inverts the priority levels. For the AIX
kernel, the priority is in the range from 0 to 127, where 0 is the most favored
priority and 127 the least-favored. Commands, such as the ps command, report the
kernel priority.
The threads library handles the priority through a sched_param structure, defined
in the sys/sched.h header file. Currently, this structure contains two fields:
sched_priority
sched_policy
Specifies the priority.
This field is ignored by the threads library and should not be used.
In the future, other fields may be defined for other scheduling characteristics.
Setting the Scheduling Policy and Priority at Creation Time
The scheduling policy can be set when creating a thread by setting the schedpolicy
attribute of the thread attributes object. The pthread_attr_setschedpolicy
subroutine sets the scheduling policy to one of the three previously defined
scheduling policies. The current value of the schedpolicy attribute of a thread
attributes object can be obtained by the pthread_attr_getschedpolicy subroutine.
The scheduling priority can be set at creation time of a thread by setting the
schedparam attribute of the thread attributes object. The
pthread_attr_setschedparam subroutine sets the value of the schedparam attribute,
copying the value of the specified structure. The pthread_attr_getschedparam
subroutine gets the schedparam attribute.
In the following code fragment, a thread is created with the round-robin
scheduling policy, using a priority level of 3:
sched_param schedparam;
schedparam.sched_priority = 3;
pthread_attr_init(&attr);
pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);
pthread_attr_setschedpolicy(&attr, SCHED_RR);
pthread_attr_setschedparam(&attr, &schedparam);
pthread_create(&thread, &attr, &start_routine, &args);
pthread_attr_destroy(&attr);
236
General Programming Concepts
See “Inheritsched Attribute” on page 235 to get more information about the
inheritsched attribute.
Setting the Scheduling Attributes at Execution Time
The current schedpolicy and schedparam attributes of a thread are returned by the
pthread_getschedparam subroutine. These attributes can be set by calling the
pthread_setschedparam subroutine. If the target thread is currently running on a
processor, the new scheduling policy and priority will be implemented the next
time the thread is scheduled. If the target thread is not running, it may be
scheduled immediately at the end of the subroutine call.
For example, consider a thread T that is currently running with RR policy at the
moment the schedpolicy attribute of T is changed to FIFO. T will run until the end
of its time slice, at which time its scheduling attributes are then re-evaluated. If no
threads have higher priority, T will be rescheduled, even before other threads
having the same priority. Consider a second example where a low-priority thread
is not running. If this thread’s priority is raised by another thread calling
pthread_setschedparam, the target thread will be scheduled immediately if it is the
highest priority runnable thread.
Note: Both subroutines use two parameters: a policy parameter and a
sched_param structure. Although this structure contains a sched_policy field,
programs should not use it. The subroutines use the policy parameter to pass
the scheduling policy and ignore the sched_policy field.
Considerations about Scheduling Policies
Applications should use the default scheduling policy, unless a specific application
requires the use of a fixed-priority scheduling policy.
Using the RR policy ensures that all threads having the same priority level will be
scheduled equally, regardless of their activity. This can be useful in programs
where threads have to read sensors or write actuators.
Using the FIFO policy should be done with great care. A thread running with FIFO
policy runs to completion, unless it is blocked by some calls, such as performing
input and output operations. A high-priority FIFO thread may not be preempted
and can affect the global performance of the system. For example, threads doing
intensive calculations, such as inverting a large matrix, should never run with
FIFO policy.
The setting of scheduling policy and priority is also influenced by the contention
scope of threads. Using the FIFO or the RR policy may not always be allowed. See
“Impacts of Contention Scope on Scheduling” on page 238 for more information.
Contention Scope
The threads library defines two possible contention scopes:
PTHREAD_SCOPE_PROCESS Process (or local) contention scope. Specifies that the
thread will be scheduled against all other local contention
scope threads in the process.
PTHREAD_SCOPE_SYSTEM
System (or global) contention scope. Specifies that the
thread will be scheduled against all other threads in the
system.
Chapter 11. Threads Programming Guidelines
237
See “Threads Implementation” on page 166 for more information about contention
scope.
Setting the Contention Scope
The contention scope can only be set before thread creation by setting the
contention-scope attribute of a thread attributes object. The pthread_attr_setscope
subroutine sets the value of the attribute; the pthread_attr_getscope returns it.
The contention scope is only meaningful in a mixed-scope M:N library
implementation. A single-scope 1:1 library implementation, as in Pre-AIX Version
4.3, always returns an error when trying to set the contention-scope attribute to
PTHREAD_SCOPE_PROCESS, because all threads have system contention scope.
This is the easiest way to test the implementation of a threads library. A
TestImplementation routine could be written as follows:
int TestImplementation()
{
pthread_attr_t a;
int result;
pthread_attr_init(&a);
switch (pthread_attr_setscope(&a,
{
case 0:
result =
case ENOTSUP:
result =
case ENOSYS:
result =
default:
result =
}
PTHREAD_SCOPE_PROCESS))
LIB_MN; break;
LIB_11; break;
NO_PRIO_OPTION; break;
ERROR; break;
pthread_attr_destroy(&a);
return result;
}
Prior to AIX Version 4.3, this routine would return LIB_11.
In AIX Version 4.3, this routine returns LIB_MN.
Impacts of Contention Scope on Scheduling
The contention scope of a thread influences its scheduling. Each system contention
scope thread is bound to one kernel thread. Thus changing the scheduling policy
and priority of a global user thread results in changing the scheduling policy and
priority of the underlying kernel thread.
In AIX, only kernel threads with root authority can use a fixed-priority scheduling
policy (FIFO or RR). The following code:
schedparam.sched_priority = 3;
pthread_setschedparam(pthread_self(), SCHED_FIFO, schedparam);
will always return the EPERM error code if the calling thread has system
contention scope but does not have root authority. It would not fail, if the calling
thread had process contention scope. One does not need to have root authority to
control the scheduling parameters of user threads with process contention scope.
Local user thread can set any scheduling policy and priority, within the valid range
of values. However, two threads having the same scheduling policy and priority
but having different contention scope will not be scheduled in the same way.
Threads having process contention scope are executed by kernel threads whose
scheduling parameters are set by the library.
238
General Programming Concepts
sched_yield Subroutine
The sched_yield subroutine is the equivalent for threads of the yield subroutine. It
forces the calling thread to relinquish the use of its processor. It gives other threads
a chance to be scheduled. The next scheduled thread may belong to the same
process as the calling thread or to another process. The yield subroutine must not
be used in a multi-threaded program.
The interface pthread_yield subroutine is not available in XOPEN VERSION 5.
Synchronization Scheduling
Programmers may want to control the execution scheduling of threads when there
are constraints, especially time constraints, that require certain threads to be
executed faster than other ones. Synchronization objects, such as mutexes, may
block even high-priority threads. In some cases, undesirable behavior, known as
priority inversion, may occur. The threads library provides a facility, the mutex
protocols, to avoid priority inversions.
Priority Inversion
Priority inversion occurs when a low-priority thread holds a mutex, blocking a
high-priority thread. Due to its low priority, the mutex owner may hold the mutex
for an unbounded duration. As a result, it becomes impossible to guarantee thread
deadlines.
The following example illustrates a typical priority inversion. To make the example
easier to understand, only the case of a uniprocessor system is considered. Priority
inversions also occur on multiprocessor systems in a similar way.
A mutex M is used to protect some common data. Thread A has a priority level of
100. It should be scheduled very often. Thread B has a priority level of 20. It is a
background thread. Other threads in the process have priority levels around 60. A
code fragment from thread A is:
pthread_mutex_lock(&M);
...
pthread_mutex_unlock(&M);
/* 1 */
A code fragment from thread B is:
pthread_mutex_lock(&M);
...
fprintf(...);
...
pthread_mutex_unlock(&M);
/* 2 */
/* 3 */
Consider the following execution chronology. Thread B is scheduled and executes
line 2. When executing line 3, thread B is preempted by thread A. Thread A
executes line 1 and is blocked, because the mutex M is held by thread B. Thus,
other threads in the process are scheduled. Because thread B has a very low
priority, it may not be rescheduled for a long period, blocking thread A although
thread A has a very high priority.
The following figure illustrates the execution of the example. Each thin vertical line
represents a thread. The number at the left side is the priority level. The thick line
Chapter 11. Threads Programming Guidelines
239
represents the execution path, that is, the instructions executed by the CPU in
chronological order.
Thread A
100
Thread B
20
Other Threads
60
lock()
lock()
Priority Inversion Example
Mutex Protocols
To avoid priority inversions, two mutex protocols are provided by the threads
library:
v The priority inheritance protocol (see “Priority Inheritance Protocol”), sometimes
called basic priority inheritance protocol
v The priority protection protocol (see “Priority Protection Protocol” on page 241),
sometimes called priority ceiling protocol emulation.
Both protocols increase the priority of a thread holding a specific mutex, so that
deadlines can be guaranteed. Furthermore, when correctly used, mutex protocols
can prevent mutual deadlocks. Mutex protocols are individually assigned to
mutexes.
Priority Inheritance Protocol
In the priority inheritance protocol, the mutex holder inherits the priority of the
highest priority blocked thread. When a thread tries to lock a mutex using this
protocol and is blocked, the mutex owner temporarily receives the blocked thread’s
priority, if that priority is higher than the owner’s. It recovers its original priority
when it unlocks the mutex.
In the preceding example, thread B would get a priority level of 100 when thread
A tries to lock mutex M. Thus, thread B will be scheduled next, instead of the
other threads, and will be able to unlock mutex M. The Priority inheritance
Protocol Example figure illustrates the execution chronology of this example.
240
General Programming Concepts
Thread A
100
Thread B
Other Threads
20
60
lock()
lock()
100
20
unlock()
Priority Inheritance Protocol Example
Priority Protection Protocol
In the priority protection protocol, each mutex has a priority ceiling. It is a priority
level within the valid range of priorities. When a thread owns a mutex, it
temporarily receives the mutex priority ceiling, if the ceiling is higher than its own
priority. It recovers its original priority when it unlocks the mutex. The priority
ceiling should have the value of the highest priority of all threads that may lock
the mutex. Otherwise, priority inversions or even deadlocks may occur, and the
protocol would be inefficient.
In the following example, mutex M has a priority ceiling of 120. Then, when
thread B locks mutex M, it gets a priority level of 120 until it unlocks mutex M.
Thus, thread B will not be preempted by other threads. The Priority Protection
Protocol Example figure illustrates the execution chronology of this example.
Thread A
100
Thread B
Other Threads
20
120
60
lock()
lock()
20
unlock()
Priority Protection Protocol Example
Choosing a Mutex Protocol
The choice of a mutex protocol is made by setting attributes when creating a
mutex (see “Protocol Attribute” on page 242). “Inheritance or Protection” on
page 242 provides guidelines for choosing a protocol.
Chapter 11. Threads Programming Guidelines
241
Protocol Attribute
The mutex protocol is controlled through an attribute: the protocol attribute. This
attribute can be set in the mutex attributes object using the
pthread_mutexattr_getprotocol and pthread_mutexattr_setprotocol subroutines.
The protocol attribute can have one of the following values:
PTHREAD_PRIO_NONE
Denotes no protocol. This is the default value.
PTHREAD_PRIO_INHERIT Denotes the priority inheritance protocol.
PTHREAD_PRIO_PROTECT Denotes the priority protection protocol.
The priority protection protocol uses one additional attribute: the prioceiling
attribute. This attribute contains the priority ceiling of the mutex. The prioceiling
attribute can be controlled in the mutex attributes object, using the
pthread_mutexattr_getprioceiling and pthread_mutexattr_setprioceiling
subroutines.
The prioceiling attribute of a mutex can also be dynamically controlled using the
pthread_mutex_getprioceiling and pthread_mutex_setprioceiling subroutines.
Note that when dynamically changing the priority ceiling of a mutex, the mutex is
locked by the library; it should not be held by the thread calling the
pthread_mutex_setprioceiling subroutine to avoid a deadlock. Dynamically setting
the priority ceiling of a mutex can be useful when increasing the priority of a
thread.
The implementation of mutex protocols is optional. Each protocol is a POSIX
option. See Threads Library Options for more information about the priority
inheritance and the priority protection POSIX options.
Inheritance or Protection
Both protocols are similar and result in promoting the priority of the thread
holding the mutex. If both protocols are available, a choice must be made. This
information will help the programmer in choosing a protocol.
The choice depends on whether the priorities of the threads that will lock the
mutex are available to the programmer creating the mutex. Typically, mutexes
defined by a library and used by application threads will use the inheritance
protocol, whereas mutexes created within the application program will use the
protection protocol.
In performance-critical programs, performance considerations may also influence
the choice. In most implementations, especially in AIX, changing the priority of a
thread results in making a system call. Therefore, the two mutex protocols differ in
the amount of system calls they generate.
v Using the inheritance protocol, a system call is made each time a thread is
blocked when trying to lock the mutex.
v Using the protection protocol, one system call is always made each time the
mutex is locked by a thread.
In most performance-critical programs, the inheritance protocol should be chosen,
because mutexes are low contention objects. Mutexes are not held for long periods
of time; thus, it is not likely that threads are blocked when trying to lock them.
242
General Programming Concepts
List of Scheduling Subroutines
pthread_att r_getschedparam
pthread_attr _setschedparam
pthread_getschedp aram
pthread_yield
Returns the value of the schedparam attribute of
a thread attributes object.
Sets the value of the schedparam attribute of a
thread attributes object.
Returns the value of the schedpolicy and
schedparam attributes of a thread.
Forces the calling thread to relinquish use of its
processor.
Threads Advanced Features
The threads library provides some advanced features to be used by trained
programmers. These features are helpful to perform special tasks.
One-Time Initializations
Some C libraries are designed for dynamic initialization. That is, the global
initialization for the library is performed when the first procedure in the library is
called. In a single-threaded program, this is usually implemented using a static
variable whose value is checked on entry to each routine, as in the following code
fragment:
static int isInitialized = 0;
extern void Initialize();
int function()
{
if (isInitialized == 0) {
Initialize();
isInitialized = 1;
}
...
}
For dynamic library initialization in a multi-threaded program a simple
initialization flag is not sufficient; this flag must be protected against modification
by multiple threads simultaneously calling a library function. Protecting the flag
requires the use of a mutex; however, mutexes must be initialized before they are
used. Ensuring that the mutex is only initialized once requires a recursive solution
to this problem.
To keep the same structure in a multi-threaded program a new subroutine,
pthread_once, is provided by the threads library. Otherwise, library initialization
must be accomplished by an explicit call to a library exported initialization
function prior to any use of the library. This subroutine also provides an
alternative for initializing mutexes and condition variables.
One-Time Initialization Object
The uniqueness of the initialization is ensured by an object, the one-time
initialization object, or once block. It is a variable having the pthread_once_t data
type. In AIX and most other implementations of the threads library, the
pthread_once_t data type is a structure.
Chapter 11. Threads Programming Guidelines
243
A one-time initialization object is typically a global variable. It must be initialized
with the PTHREAD_ONCE_INIT macro, as in the following example:
static pthread_once_t once_block = PTHREAD_ONCE_INIT;
The initialization can also be done in the initial thread or in any other thread.
Several one time initialization objects can be used in the same program. The only
requirement is that the one-time initialization object be initialized with the macro.
One-Time Initialization Routine
The pthread_once subroutine calls the specified initialization routine associated
with the specified one-time initialization object if it is the first time it is called;
otherwise, it does nothing. The same initialization routine must always be used
with the same one-time initialization object. The initialization routine must have
the following prototype:
void init_routine();
The pthread_once subroutine does not provide a cancellation point. However, the
initialization routine may provide cancellation points, and, if cancelability is
enabled, the first thread calling the pthread_once subroutine may be canceled
during the execution of the initialization routine. In this case, the routine is not
considered as executed, and the next call to the pthread_once subroutine would
result in recalling the initialization routine.
It is recommended to use cleanup handlers in one-time initialization routines,
especially when performing non-idempotent operations, such as opening a file,
locking a mutex, or allocating memory. See “Using Cleanup Handlers” on
page 218.
One-time initialization routines can be used for initializing mutexes or condition
variables or to perform dynamic initialization. The code fragment shown above
would be written in a multi-threaded library as follows:
static pthread_once_t once_block = PTHREAD_ONCE_INIT;
extern void Initialize();
int function()
{
pthread_once(&once_block, Initialize);
...
}
Thread-Specific Data
Many applications require that certain data be maintained on a per-thread basis
across function calls. For example, a multi-threaded grep command using one
thread for each file needs to have thread-specific file handlers and list of found
strings. The thread-specific data interface is provided by the threads library to
meet these needs.
Thread-specific data may be viewed as a two-dimensional array of values, with
keys serving as the row index and thread IDs as the column index. A
thread-specific data key is an opaque object, of type pthread_key_t. The same key
can be used by all threads in a process. Although all threads use the same key,
they set and access different thread-specific data values associated with that key.
Thread-specific data are void pointers. This allows referencing any kind of data,
such as dynamically allocated strings or structures.
244
General Programming Concepts
In the following figure, thread T2 has a thread-specific data value of 12 associated
with the key K3. Another thread T4 has the value 2 associated with the same key.
Threads
Keys
T1
T2
T3
T4
K1
6
56
4
1
K2
87
21
0
9
K3
23
12
61
2
K4
11
76
47
88
Thread-specific data value for thread T2, key K3 = 12
Thread-Specific Data Array
Creating and Destroying Keys
Thread-specific data keys must be created before being used. Their values can be
automatically destroyed when the corresponding threads terminate. A key can also
be destroyed upon request to reclaim its storage.
Key Creation
A thread-specific data key is created by calling the pthread_key_create subroutine.
This subroutine returns a key. The thread-specific data is set to a value of NULL
for all threads, including threads not yet created.
For example, consider two threads A and B. Thread A performs the following
operations in chronological order:
1. Create a thread-specific data key K.
Threads A and B can use the key K. The value for both threads is NULL.
2. Create a thread C.
Thread C can also use the key K. The value for thread C is NULL.
The number of thread-specific data keys is limited to 508 per process. This number
can be retrieved by the PTHREAD_KEYS_MAX symbolic constant.
The pthread_key_create subroutine must be called only once. Otherwise, two
different keys are created. For example, consider the following code fragment:
/* a global variable */
static pthread_key_t theKey;
/* thread A */
...
pthread_key_create(&theKey, NULL);
...
/* thread B */
...
pthread_key_create(&theKey, NULL);
...
/* call 1 */
/* call 2 */
Threads A and B run concurrently, but call 1 happens before call 2. Call 1 will
create a key K1 and store it in the theKey variable. Call 2 will create another key
Chapter 11. Threads Programming Guidelines
245
K2, and store it also in the theKey variable, thus overriding K1. As a result, thread
A will use K2, assuming it is K1. This situation should be avoided for two reasons:
v Key K1 is lost, thus its storage will never be reclaimed until the process
terminates. Because the number of keys is limited, you may run out of keys.
v If thread A stores a thread-specific data using the theKey variable before call 2,
the data will be bound to key K1. After call 2, the theKey variable contains K2;
if thread A then tries to fetch its thread-specific data, it would always get NULL.
Ensuring the uniqueness of key creation can be done in two ways:
v Using the one-time initialization facility. See “One-Time Initializations” on
page 243.
v Creating the key before the threads that will use it. This is often possible, for
example, when using a pool of threads with thread-specific data to perform
similar operations. This pool of threads is usually created by one thread, the
initial (or another ″driver″) thread.
It is the programmer’s responsibility to ensure the uniqueness of key creation. The
threads library provides no way to check if a key has been created more than once.
Destructor Routine
A destructor routine may be associated with each thread-specific data key.
Whenever a thread is terminated, if there is non-NULL, thread-specific data for
this thread bound to any key, the destructor routine associated with that key is
called. This allows dynamically allocated thread-specific data to be automatically
freed when the thread is terminated. The destructor routine has one parameter, the
value of the thread-specific data.
For example, a thread-specific data key may be used for dynamically allocated
buffers. A destructor routine should be provided to ensure that the buffer is freed
when the thread terminates, the free subroutine can be used:
pthread_key_create(&key, free);
More complex destructors may be used. If a multi-threaded grep command, using
a thread per file to scan, has thread-specific data to store a structure containing a
work buffer and the thread’s file descriptor, the destructor routine may be:
typedef struct {
FILE *stream;
char *buffer;
} data_t;
...
void destructor(void *data)
{
fclose(((data_t *)data)->stream);
free(((data_t *)data)->buffer);
free(data);
*data = NULL;
}
Although some implementations of the threads library may repeat destructor calls,
the destructor routine is called only once in AIX. Care must be taken when porting
code from other systems where a destructor routine can be called several times.
Key Destruction
A thread-specific data key can be destroyed by calling the pthread_key_delete
subroutine. This subroutine frees the key only if no thread-specific data is bound to
246
General Programming Concepts
it. Data is said to be bound to the key when at least one value is not NULL. The
pthread_key_delete subroutine does not actually call the destructor routine for
each thread having data. To destroy a thread-specific data key, the programmer
must ensure that no thread-specific data is bound to the key.
Once a data key is destroyed, it can be reused by another call to the
pthread_key_create subroutine. Thus, the pthread_key_delete is useful especially
when using many data keys. For example, in the following code fragment the loop
would never end:
/* bad example - do not write such code! */
pthread_key_t key;
while (pthread_key_create(&key, NULL))
pthread_key_delete(key);
Using Thread-Specific Data
Thread-specific data is accessed using the pthread_getspecific and
pthread_setspecific subroutines. The first one reads the value bound to the
specified key and specific to the calling thread; the second one sets the value.
Setting Successive Values
The value should be a pointer. The pointer may point to any kind of data.
Thread-specific data is typically used for dynamically allocated storage, as in the
following code fragment:
private_data = malloc(...);
pthread_setspecific(key, private_data);
When setting a value, the previous value is lost. For example, in the following
code fragment, the value of the old pointer is lost, and the storage it pointed to
may not be recoverable:
pthread_setspecific(key, old);
...
pthread_setspecific(key, new);
It is the programmer’s responsibility to retrieve the old thread-specific data value
to reclaim storage before setting the new value. For example, it is possible to
implement a swap_specific routine in the following manner:
int swap_specific(pthread_key_t key, void **old_pt, void *new)
{
*old_pt = pthread_getspecific(key);
if (*old_pt == NULL)
return -1;
else
return pthread_setspecific(key, new);
}
Such a routine does not exist in the threads library because it is not always
necessary to retrieve the previous value of thread-specific data. Such a case occurs,
for example, when thread-specific data are pointers to specific locations in a
memory pool allocated by the initial thread.
Taking Care about Destructor Routines
When using dynamically allocated thread-specific data, the programmer must
provide a destructor routine when calling the pthread_key_create subroutine. The
programmer must also ensure that, when freeing the storage allocated for
thread-specific data, the pointer is set to NULL. Otherwise, the destructor routine
may be called with an illegal parameter. For example:
Chapter 11. Threads Programming Guidelines
247
pthread_key_create(&key, free);
...
...
private_data = malloc(...);
pthread_setspecific(key, private_data);
...
/* bad example! */
...
pthread_getspecific(key, &data);
free(data);
...
When the thread terminates, the destructor routine is called for its thread-specific
data. Because the value is a pointer to already freed memory, an error may occur.
To correct this, the following code fragment should be substituted:
/* better example! */
...
pthread_getspecific(key, &data);
free(data);
pthread_setspecific(key, NULL);
...
When the thread terminates, the destructor routine is not called, because there is
no thread-specific data.
Using Non-Pointer Values
It is possible to store values that are not pointers, such as integers. It is not
recommended to do this for at least two reasons:
v Casting a pointer into a scalar type may not be portable
v The NULL pointer value is implementation-dependent; several systems assign
the NULL pointer a non-zero value.
If you are sure that your program will never be ported to another system, you may
use integer values for thread-specific data.
Advanced Attributes
This section describes special attributes of threads, mutexes, and condition
variables. The implementation of these attributes is optional; it depends on POSIX
options.
Stack Attributes
A stack is allocated for each thread. Stack management is implementationdependent; thus, the following information applies only to AIX, although similar
features may exist on other systems.
The stack is dynamically allocated when the thread is created. Using advanced
thread attributes, it is possible for the user to control the size (“Stack Size”) and
address (“Stack Address” on page 249) of the stack. The following information does
not apply to the initial thread, which is created by the system.
Stack Size
The stacksize attribute is defined in AIX. It depends on the stack size POSIX option
(“Stack Size POSIX Option” on page 260); this option may not be implemented on
other systems.
248
General Programming Concepts
The stacksize attribute specifies the minimum stack size that will be allocated for a
thread. The pthread_attr_getstacksize subroutine returns the value of the attribute,
and the pthread_attr_setstacksize subroutine sets the value.
The stacksize attribute is used as follows to calculate the size of the stack to
allocate:
v If the value of stacksize is less than 96KB, a stack of 96KB will be allocated
In the AIX implementation of the threads library, a chunk of data, called user thread
area, is allocated for each created thread. The allocation is always a multiple of
4KB. As shown in the following figure, the area is divided into:
High address
User stack
96KB
Cancellation Stack
24KB
Red zone
4KB
Low address
Stack Management
v A 4KB red zone, which is both read and write protected for stack overflow
detection
v A cancellation stack with a size equal to User stack size divided by 4 and
multiple of 4KB
v A default stack.
Note: The user thread area described here has nothing to do with the uthread
structure used in the AIX kernel. The user thread area is accessed only in user
mode and is exclusively handled by the threads library, whereas the uthread
structure only exists within the kernel environment.
Stack Address
The stackaddr attribute is not defined in AIX. It depends on the stack address
POSIX option (“Stack Address POSIX Option” on page 260); this option is not
implemented in AIX but may be implemented on other systems.
The stackaddr attribute specifies the address of the stack that will be allocated for
a thread. The pthread_attr_getstackaddr subroutine returns the value of the
attribute, and the pthread_attr_setstackaddr subroutine sets the value.
If no stack address is specified, the stack is allocated by the system at an arbitrary
address. There is no way to get this address. Usually you do not need to know the
stack address. However, if you really need to have the stack at a known location,
you can use the stackaddr attribute. For example, if you need a very large stack,
you may set its address to an unused segment, guaranteeing that the allocation
will succeed.
Chapter 11. Threads Programming Guidelines
249
If a stack address is specified when calling the pthread_create subroutine, the
system will try to allocate the stack at the given address. If it fails, the
pthread_create subroutine returns EINVAL. The pthread_attr_setstackaddr
subroutine never returns an error, unless the specified stack address exceeds the
addressing space, because it does not actually allocate the stack.
Process Sharing
Most UNIX systems allow several processes to share a common data space, known
as shared memory. AIX also provides this facility; see “Understanding Memory
Mapping” on page 569 for more information about the AIX shared memory facility.
The process sharing attributes for condition variables and mutexes are meant to
allow these objects to be allocated in shared memory to support synchronization
among threads belonging to different processes. However, there is no
industry-standard interface for shared memory management. For this reason, the
process sharing POSIX option is not implemented in the AIX threads library.
Making Complex Synchronization Objects
The subroutines provided in the threads library can be used as primitives to build
more complex synchronization objects. This article provides implementation
examples of some traditional synchronization objects:
v “Long Locks”, that can be held over long periods of time
v Inter-thread “Semaphores” on page 251
v “Write-Priority Read/Write Locks” on page 252
Long Locks
The mutexes provided by the threads library are low-contention objects and should
not be held for a very long time. Long locks are implemented with mutexes and
condition variables, so that a long lock may be held for long time without affecting
the performance of the program.
The following implementation is very basic. The lock owner is not checked, any
thread can unlock any lock. Error handling and cancellation handling are not
performed. As written hereafter, long locks should not be used with cancelability
enabled. The next example shows how to prevent data inconsistency using cleanup
handlers. However, this example shows a typical use of condition variables.
A long lock has the long_lock_t data type. It must be initialized by the
long_lock_init routine. The long_lock, long_trylock, and long_unlock subroutine
performs similar operations to the pthread_mutex_lock, pthread_mutex_trylock,
and pthread_mutex_unlock subroutine.
typedef struct {
pthread_mutex_t lock;
pthread_cond_t cond;
int free;
int wanted;
} long_lock_t;
void long_lock_init(long_lock_t *ll)
{
pthread_mutex_init(&ll->lock, NULL);
pthread_cond_init(&ll->cond);
ll->free = 1;
ll->wanted = 0;
}
250
General Programming Concepts
void long_lock_destroy(long_lock_t *ll)
{
pthread_mutex_destroy(&ll->lock);
pthread_cond_destroy(&ll->cond);
}
void long_lock(long_lock_t *ll)
{
pthread_mutex_lock(&ll->lock);
ll->wanted++;
while(!ll->free)
pthread_cond_wait(&ll->cond);
ll->wanted—;
ll->free = 0;
pthread_mutex_unlock(&ll->lock);
}
int long_trylock(long_lock_t *ll)
{
int got_the_lock;
pthread_mutex_lock(&ll->lock);
got_the_lock = ll->free;
if (got_the_lock)
ll->free = 0;
pthread_mutex_unlock(&ll->lock);
return got_the_lock;
}
void long_unlock(long_lock_t *ll)
{
pthread_mutex_lock(&ll->lock);
ll->free = 1;
if (ll->wanted)
pthread_cond_signal(&ll->cond);
pthread_mutex_unlock(&ll->lock);
}
Semaphores
Traditional semaphores in UNIX systems are interprocess synchronization facilities.
It is possible to implement interthread semaphores for specific usage.
The following implementation is very basic. Error handling is not performed, but
cancellations are properly handled with cleanup handlers whenever required.
A semaphore has the sema_t data type. It must be initialized by the sema_init
routine and destroyed with the sema_destroy routine. The P and V operations are
respectively performed by the sema_p and sema_v routines.
typedef struct {
pthread_mutex_t lock;
pthread_cond_t cond;
int count;
} sema_t;
void sema_init(sema_t *sem)
{
pthread_mutex_init(&sem->lock, NULL);
pthread_cond_init(&sem->cond, NULL);
sem->count = 1;
}
void sema_destroy(sema_t *sem)
{
pthread_mutex_destroy(&sem->lock);
pthread_cond_destroy(&sem->cond);
}
Chapter 11. Threads Programming Guidelines
251
void p_operation_cleanup(void *arg)
{
sema_t *sem;
sem = (sema_t *)arg;
pthread_mutex_unlock(&sem->lock);
}
void sema_p(sema_t *sem)
{
pthread_mutex_lock(&sem->lock);
pthread_cleanup_push(p_operation_cleanup, sem);
while (sem->count <= 0)
pthread_cond_wait(&sem->cond, &sem->lock);
sem->count—;
/*
* Note that the pthread_cleanup_pop subroutine will
* execute the p_operation_cleanup routine
*/
pthread_cleanup_pop(1);
}
void sema_v(sema_t *sem)
{
pthread_mutex_lock(&sem->lock);
sem->count++;
if (sem->count <= 0)
pthread_cond_signal(&sem->cond);
pthread_mutex_unlock(&sem->lock);
}
The counter specifies the number of users that are allowed to take the semaphore.
It is never strictly negative; thus, it does not specify the number of waiting users,
as for traditional semaphores. This implementation provides a typical solution to
the multiple wakeup problem on the pthread_cond_wait subroutine. Note that the
P operation is cancelable, because the pthread_cond_wait subroutine provides a
cancellation point.
Write-Priority Read/Write Locks
A write-priority read/write lock provides multiple threads simultaneous read-only
access to a protected resource, and a single thread write access to the resource
while excluding reads. When a writer releases a lock, other waiting writers will get
the lock before any waiting reader. Write-priority read/write locks are usually used
to protect resources that are more often read than written.
The following implementation is very basic. The lock owner is not checked, any
thread can unlock any lock. Routines similar to the pthread_mutex_trylock
subroutine are missing and error handling is not performed, but cancellations are
properly handled with cleanup handlers whenever required.
A write-priority read/write lock has the rwlock_t data type. It must be initialized
by the rwlock_init routine. The rwlock_lock_read routine locks the lock for a
reader (multiple readers are allowed), the rwlock_unlock_read routine unlocks it.
The rwlock_lock_write routine locks the lock for a writer, the
rwlock_unlock_write routine unlocks it. The proper unlocking routine (for the
reader or for the writer) must be called.
typedef struct {
pthread_mutex_t lock;
pthread_cond_t rcond;
pthread_cond_t wcond;
int lock_count; /* < 0 .. held by writer
252
General Programming Concepts
*/
/* > 0 .. held by lock_count readers */
/* = 0 .. held by nobody
*/
int waiting_writers; /* count of wating writers
*/
} rwlock_t;
void rwlock_init(rwlock_t *rwl)
{
pthread_mutex_init(&rwl->lock, NULL);
pthread_cond_init(&rwl->wcond, NULL);
pthread_cond_init(&rwl->rcond, NULL);
rwl->lock_count = 0;
rwl->waiting_writers = 0;
}
void waiting_reader_cleanup(void *arg)
{
rwlock_t *rwl;
rwl = (rwlock_t *)arg;
pthread_mutex_unlock(&rwl->lock);
}
void rwlock_lock_read(rwlock_t *rwl)
{
pthread_mutex_lock(&rwl->lock);
pthread_cleanup_push(waiting_reader_cleanup, rwl);
while ((rwl->lock_count < 0) && (rwl->waiting_writers))
pthread_cond_wait(&rwl->rcond, &rwl->lock);
rwl->lock_count++;
/*
* Note that the pthread_cleanup_pop subroutine will
* execute the waiting_reader_cleanup routine
*/
pthread_cleanup_pop(1);
}
void rwlock_unlock_read(rwlock_t *rwl)
{
pthread_mutex_lock(&rwl->lock);
rwl->lock_count—;
if (!rwl->lock_count)
pthread_cond_signal(&rwl->wcond);
pthread_mutex_unlock(&rwl->lock);
}
void waiting_writer_cleanup(void *arg)
{
rwlock_t *rwl;
rwl = (rwlock_t *)arg;
rwl->waiting_writers—;
if ((!rwl->waiting_writers) && (rwl->lock_count >= 0))
/*
* This only happens if we have been canceled
*/
pthread_cond_broadcast(&rwl->wcond);
pthread_mutex_unlock(&rwl->lock);
}
void rwlock_lock_write(rwlock_t *rwl)
{
pthread_mutex_lock(&rwl->lock);
rwl->waiting_writers++;
pthread_cleanup_push(waiting_writer_cleanup, rwl);
while (rwl->lock_count)
pthread_cond_wait(&rwl->wcond, &rwl->lock);
rwl->lock_count = -1;
/*
* Note that the pthread_cleanup_pop subroutine will
* execute the waiting_writer_cleanup routine
*/
pthread_cleanup_pop(1);
}
Chapter 11. Threads Programming Guidelines
253
void rwlock_unlock_write(rwlock_t *rwl)
{
pthread_mutex_lock(&rwl->lock);
l->lock_count = 0;
if (!rwl->wating_writers)
pthread_cond_broadcast(&rwl->rcond);
else
pthread_cond_signal(&rwl->wcond);
pthread_mutex_unlock(&rwl->lock);
}
Readers are just counted. When the count reaches zero, a waiting writer may take
the lock. Only one writer can hold the lock. When the lock is released by a writer,
another writer is awakened, if there is one. Otherwise, all waiting readers are
awakened.
Note that the locking routines are cancelable, because they call the
pthread_cond_wait subroutine. For this reason, cleanup handlers are registered
before calling the subroutine.
List of Threads Advanced-Feature Subroutines
pthread_attr_getstackaddr
pthread_attr_getstacksize
pthread_attr_setstackaddr
pthread_attr_setstacksize
pthread_condattr_getpshared
pthread_condattr_setpshared
pthread_getspecific
pthread_key_create
pthread_key_delete
pthread_mutexattr_getpshared
pthread_mutexattr_setpshared
pthread_once
PTHREAD_ONCE_INIT
pthread_setspecific
Returns the value of the stackaddr attribute of a
thread attributes object.
Returns the value of the stacksize attribute of a
thread attributes object.
Sets the value of the stackaddr attribute of a
thread attributes object.
Sets the value of the stacksize attribute of a
thread attributes object.
Returns the value of the process-shared attribute
of a condition attributes object.
Sets the value of the process-shared attribute of
a condition attributes object.
Returns the thread-specific data associated with
the specified key.
Creates a thread-specific data key.
Deletes a thread-specific data key.
Returns the value of the process-shared attribute
of a mutex attributes object.
Sets the value of the process-shared attribute of
a mutex attributes object.
Executes a routine exactly once in a process.
Initializes a once synchronization control
structure.
Sets the thread-specific data associated with the
specified key.
Threads-Processes Interactions Overview
Threads and processes interact in several ways, especially when performing the
following kind of tasks:
v “Signal Management” on page 255
v “Process Duplication and Termination” on page 258
v “Scheduling” on page 259
254
General Programming Concepts
Signal Management
Signal management in multi-threaded processes resulted from a compromise
among many and sometimes conflicting goals. The goal of compatibility is assured:
signals in multi-threaded processes are an extension of signals in traditional
single-threaded programs. Programs handling signals and written for
single-threaded systems will behave as expected in AIX Version 4.
Signal management in multi-threaded processes is shared by the process and
thread levels, and consists of:
v Per-process signal handlers
v Per-thread signal masks
v Single delivery of each signal
The threads library also provides a new subroutine and introduces new
programming practices for waiting for asynchronously generated signals.
Signal Handlers and Signal Masks
Signal handlers are maintained at process level. It is strongly recommended to use
only the sigaction subroutine to get and set signal handlers. Other subroutines
may not be supported in the future.
Because the list of signal handlers is maintained at process level, any thread within
the process may change it. If two threads set a signal handler on the same signal,
the last thread that called the sigaction subroutine will override the setting of the
previous thread call; and in most cases, it will be impossible to predict the order in
which threads are scheduled.
Signal masks are maintained at thread level. Each thread can have its own set of
signals that will be blocked from delivery. The sigthreadmask subroutine must be
used to get and set the calling thread’s signal mask. The sigprocmask subroutine
must not be used in multi-threaded programs; otherwise, unexpected behavior
may result.
The sigthreadmask subroutine is very similar to sigprocmask. The parameters and
usage of both subroutines are exactly the same. When porting existing code to
support the threads library, you may simply replace sigprocmask with
sigthreadmask.
Signal Generation
Signals generated by some action attributable to a particular thread, such as a
hardware fault, are sent to the thread that caused the signal to be generated.
Signals generated in association with a process ID, a process group ID, or an
asynchronous event (such as terminal activity) are sent to the process.
The pthread_kill subroutine sends a signal to a thread. Because thread IDs identify
threads within a process, this subroutine can only send signals to threads within
the same process.
The kill subroutine (and thus the kill command) sends a signal to a process. A
thread can send a signal Signal to its process by executing the following call:
kill(getpid(), Signal);
Chapter 11. Threads Programming Guidelines
255
The raise subroutine cannot be used to send a signal to the calling thread’s
process. The raise subroutine sends a signal to the calling thread, as in the
following call:
pthread_kill(pthread_self(), Signal);
This ensures that the signal is sent to the caller of the raise subroutine. Thus,
library routines written for single-threaded programs may easily be ported to a
multi-threaded system, because the raise subroutine is usually intended to send
the signal to the caller.
The alarm subroutine requests that a signal be sent later to the process, and alarm
states are maintained at process level. Thus, the last thread that called the alarm
subroutine overrides the settings of other threads in the process. In a
multi-threaded program, the SIGALRM signal is not necessarily delivered to the
thread that called the alarm subroutine. The calling thread may even be
terminated; and therefore, it cannot receive the signal.
Handling Signals
Signal handlers are called within the thread to which the signal is delivered. Signal
handlers may call the pthread_self subroutine to get their thread ID. Some
limitations to signal handlers are introduced by the threads library:
v Signal handlers may call the longjmp or siglongjmp subroutine only if the
corresponding call to the setjmp or sigsetjmp subroutine was performed in the
same thread.
Usually, a program that wants to wait for a signal installs a signal handler that
calls the longjmp subroutine to continue execution at the point where the
corresponding setjmp subroutine was called. This cannot be done in a
multi-threaded program, because the signal may be delivered to a thread other
than the one that called the setjmp subroutine, thus causing the handler to be
executed by the wrong thread.
v Signal handlers must not call the pthread_cond_signal or
pthread_cond_broadcast subroutine to signal a condition.
To allow a thread to wait for asynchronously generated signals, the threads library
provides the sigwait subroutine. The sigwait subroutine blocks the calling thread
until one of the awaited signals is sent to the process or to the thread. There must
not be a signal handler installed on a signal awaited using the sigwait subroutine.
Typically, programs may create a dedicated thread to wait for asynchronously
generated signals. Such a thread just loops on a sigwait subroutine call and
handles the signals. The following code fragment gives an example of such a signal
waiter thread:
sigset_t set;
int sig;
sigemptyset(&set);
sigaddset(&set, SIGINT);
sigaddset(&set, SIGQUIT);
sigaddset(&set, SIGTERM);
sigthreadmask(SIG_BLOCK, &set, NULL);
while (1) {
sigwait(&set, &sig);
switch (sig) {
case SIGINT:
/* handle interrupts */
break;
256
General Programming Concepts
case SIGQUIT:
/* handle quit */
break;
case SIGTERM:
/* handle termination */
break;
default:
/* unexpected signal */
pthread_exit((void *)-1);
}
}
If more than one thread called the sigwait subroutine, exactly one call returns
when a matching signal is sent. There is no way to predict which thread will be
awakened. Note that the sigwait subroutine provides a cancellation point.
Because a dedicated thread is not a real signal handler, it may signal a condition to
any other thread. It is possible to implement a sigwait_multiple routine that
would awaken all threads waiting for a specific signal. Each caller of the
sigwait_multiple routine would register a set of signals. The caller then waits on a
condition variable. A single thread calls the sigwait subroutine on the union of all
registered signals. When the call to the sigwait subroutine returns, the appropriate
state is set and condition variables are broadcasted. New callers to the
sigwait_multiple subroutine would cause the pending sigwait subroutine call to
be canceled and reissued to update the set of signals being waited for.
Signal Delivery
A signal is delivered to a thread, unless its action is set to ignore. The following
rules govern signal delivery in a multi-threaded process:
v A signal whose action is set to terminate, stop, or continue the target thread or
process respectively terminates, stops, or continues the entire process (and thus
all of its threads). This means that single-threaded programs may be rewritten as
multi-threaded programs without changing their externally visible signal
behavior.
Consider for example a multi-threaded user command, such as the grep
command. A user may start the command in his favorite shell and then decide
to stop it by sending a signal with the kill command. It is obvious that the
signal should stop the entire process running the grep command.
v Signals generated for a specific thread, using the pthread_kill or the raise
subroutines, are delivered to that thread. If the thread has blocked the signal
from delivery, the signal is set pending on the thread until the signal is
unblocked from delivery. If the thread is terminated before the signal delivery,
the signal will be ignored.
v Signals generated for a process, using the kill subroutine for example, are
delivered to exactly one thread in the process. If one or more threads called the
sigwait subroutine, the signal is delivered to exactly one of these threads.
Otherwise, the signal is delivered to exactly one thread that did not block the
signal from delivery. If no thread matches these conditions, the signal is set
pending on the process until a thread calls the sigwait subroutine specifying this
signal or a thread unblocks the signal from delivery.
If the action associated with a pending signal (on a thread or on a process) is set to
ignore, the signal is ignored.
Chapter 11. Threads Programming Guidelines
257
Process Duplication and Termination
Because all processes have at least one thread, creating (that is, duplicating) and
terminating a process implies the creation and the termination of threads. This
article describes the interactions between threads and processes when duplicating
and terminating a process.
Forking
There are two reasons why AIX programmers call the fork subroutine:
1. To create a new flow of control within the same program. AIX creates a new
process.
2. To create a new process running a different program. In this case, the call to the
fork subroutine is soon followed by a call to one of the exec subroutines.
In a multi-threaded program, the first use of the fork subroutine, creating new
flows of control, is provided by the pthread_create subroutine. The fork subroutine
should thus be used only to run new programs.
The fork subroutine duplicates the parent process, but duplicates only the calling
thread; the child process is a single-threaded process. The calling thread of the
parent process becomes the initial thread of the child process; it may not be the
initial thread of the parent process. Thus, if the initial thread of the child process
returns from its entry-point routine, the child process terminates.
When duplicating the parent process, the fork subroutine also duplicates all the
synchronization variables, including their state. Thus, for example, mutexes may be
held by threads that no longer exist in the child process and any associated
resource may be inconsistent.
It is strongly recommended to use the fork subroutine only to run new programs,
and to call one of the exec subroutines as soon as possible after the call to the fork
subroutine in the child process.
Fork Handlers
Unfortunately, the rule explained above does not address the needs of
multi-threaded libraries. Application programs may not be aware that a
multi-threaded library is in use and will feel free to call any number of library
routines between the fork and the exec subroutines, just as they always have.
Indeed, they may be old single-threaded programs and cannot, therefore, be
expected to obey new restrictions imposed by the threads library.
On the other hand, multi-threaded libraries need a way to protect their internal
state during a fork in case a routine is called later in the child process. The
problem arises especially in multi-threaded input/output libraries, which are
almost sure to be invoked between the fork and the exec subroutines to affect
input/output redirection.
The pthread_atfork subroutine provides a way for multi-threaded libraries to
protect themselves from innocent application programs which call the fork
subroutine. It also provides multi-threaded application programs with a standard
mechanism for protecting themselves from calls to the fork subroutine in a library
routine or the application itself.
258
General Programming Concepts
The pthread_atfork subroutine registers fork handlers to be called before and after
the call to the fork subroutine. The fork handlers are executed in the thread that
called the fork subroutine. There are three fork handlers:
Prepare
Parent
Child
The prepare fork handler is called just before the processing of the fork
subroutine begins.
The parent fork handler is called just after the processing of the fork subroutine
is completed in the parent process.
The child fork handler is called just after the processing of the fork subroutine
is completed in the child process.
The prepare fork handlers are called in last-in first-out (LIFO) order, whereas the
parent and child fork handlers are called in first-in first-out (FIFO) order. This
allows programs to preserve any desired locking order.
Process Termination
When a process terminates, by calling the _exit subroutine either explicitly or
implicitly, all threads within the process are terminated. Neither the cleanup
handlers nor the thread-specific data destructors are called.
The reason for this behavior is that there is no state to leave clean and no
thread-specific storage to reclaim, because the whole process terminates, including
all the threads, and all the process storage is reclaimed, including all
thread-specific storage.
Scheduling
In previous versions of AIX, up to version 3.2, the process was the schedulable
entity. Several commands and subroutines influenced on-process scheduling. The
introduction of threads changed the semantics of these commands and subroutines.
Process-Level Scheduling
In AIX Version 4, the scheduler allocates processor time to threads based on each
thread’s priority and scheduling policy. Previously, AIX scheduled processes and
did not support threads: process priority depended on nice values, which were
managed with the nice and renice commands and the getpriority, setpriority, and
nice subroutines. While these interfaces still exist, process nice values are only
used in the default scheduling policy, denoted SCHED_OTHER, which uses the
nice value and recent CPU usage to calculate priority. The other scheduling policies
are fixed priority. The getpri and setpri subroutines previously managed process
priority. They now manage thread priority and respectively return the priority of a
thread in the process, or set the priority of all threads in a process.
The yield subroutine previously caused a process to relinquish the processor,
allowing a higher priority process to be scheduled immediately, before the end of
its time slice. In a multi-threaded process, only the calling thread gives up its time
slice. Threads can use the pthread_yield service to yield the processor. If the
contention scope is global, the behavior is as with yield, and any thread can be
scheduled; if the scope is local, another local thread will be scheduled.
Timer and Sleep Subroutines
Timer routines now execute in the context of the calling thread, instead of the
calling process. Thus, if a timer expires, the watchdog timer function is called in
Chapter 11. Threads Programming Guidelines
259
the thread’s context. When a process or thread goes to sleep, it relinquishes the
processor. In a multi-threaded process, only the calling thread is put to sleep.
List of Threads-Processes Interactions Subroutines
alarm
kill or killpg
pthread_atfork
pthread_kill
raise
sigaction, sigvec, or signal
sigsuspend or sigpause
sigthreadmask
sigwait
Causes a signal to be sent to the calling process
after a specified timeout.
Sends a signal to a process or a group of
processes.
Registers fork cleanup handlers.
Sends a signal to the specified thread.
Sends a signal to the executing program.
Specifies the action to take upon delivery of a
signal.
Atomically changes the set of blocked signals
and waits for a signal.
Sets the signal mask of a thread.
Blocks the calling thread until a specified signal
is received.
Threads Library Options
The POSIX standard for the threads library specifies the implementation of some
parts as optional. All subroutines defined by the threads library API are always
available. Depending on the available options, some subroutines may not be
implemented. Unimplemented subroutines can be called by applications, but they
always return the ENOSYS error code.
List of Options
The following options are defined:
v Stack address (“Stack Address POSIX Option”)
v Stack size (“Stack Size POSIX Option”)
v Priority scheduling (“Priority Scheduling POSIX Option” on page 261)
v Priority inheritance
v Priority protection
v Process sharing.
The priority inheritance and priority protection options require the implementation
of the priority scheduling option.
Stack Address POSIX Option
The following attribute and subroutines are available when the option is
implemented:
v The stackaddr attribute of the thread attributes object
v The pthread_attr_getstackaddr and pthread_attr_setstackaddr subroutines.
Stack Size POSIX Option
The stack size option enables the control of the stacksize attribute of a thread
attributes object. This attribute specifies the minimum stack size to be used for the
created thread. See “Stack Size” on page 248 for more information about the
stacksize attribute.
260
General Programming Concepts
The following attribute and subroutines are available when the option is
implemented:
v The stacksize attribute of the thread attributes object
v The pthread_attr_getstacksize and pthread_attr_setstacksize subroutines.
Priority Scheduling POSIX Option
The priority scheduling option enables the control of execution scheduling at
thread level. When this option is disabled, all threads within a process share the
scheduling properties of the process. When this option is enabled, each thread has
its own scheduling properties. For local contention scope threads, the scheduling
properties are handled at process level by a library scheduler, while for global
contention scope threads, the scheduling properties are handled at system level by
the kernel scheduler. See “Scheduling” on page 259 to get more information about
the scheduling properties of a thread.
The attributes and subroutines shown below are available when the option is
implemented:
v The inheritsched attribute of the thread attributes object
v The schedparam attribute of the thread attributes object and the thread
v The schedpolicy attribute of the thread attributes objects and the thread
v The contention-scope attribute of the thread attributes objects and the thread
v The pthread_attr_getschedparam and pthread_attr_setschedparam subroutines
v The pthread_getschedparam subroutine.
Checking the Availability of an Option
Options can be checked at compile time (“Compile Time Checking”) or at run time
(“Run Time Checking” on page 262). Portable programs should check the
availability of options before using them, so that they do not need to be rewritten
when ported to other systems.
Compile Time Checking
Symbolic constants (symbols) can be used to get the availability of options on the
system where the program is compiled. The symbols are defined in the pthread.h
header file by the #define pre-processor command. For unimplemented options,
the corresponding symbol is undefined by the #undef pre-processor command.
Checking option symbols should be done in each program that may be ported to
another system.
The following list indicates the symbol associated with each option:
Stack address
Stack size
Priority scheduling
Priority inheritance
Priority protection
Process sharing
_POSIX_THREAD_ATTR_STACKADDR
_POSIX_THREAD_ATTR_STACKSIZE
_POSIX_THREAD_PRIORITY_SCHEDULING
_POSIX_THREAD_PRIO_INHERIT
_POSIX_THREAD_PRIO_PROTECT
_POSIX_THREAD_PROCESS_SHARED
The simplest action to take when an option is not available is to stop the
compilation, as in the following example:
#ifndef _POSIX_THREAD_ATTR_STACKSIZE
#error "The stack size POSIX option is required"
#endif
Chapter 11. Threads Programming Guidelines
261
The pthread.h header file also defines the following symbols that can be used by
other header files or by programs:
_POSIX_REENTRANT_FUNCTIONS
_POSIX_THREADS
Denotes that reentrant functions are required.
Denotes the implementation of the threads library.
Run Time Checking
The sysconf subroutine can be used to get the availability of options on the system
where the program is executed. This is useful when porting programs between
systems that have a binary compatibility, such as two versions of AIX.
The following list indicates the symbolic constant associated with each option and
that must be used for the Name parameter of the sysconf subroutine. The symbolic
constants are defined in the unistd.h header file.
Stack address
Stack size
Priority scheduling
Priority inheritance
Priority protection
Process sharing
_SC_THREAD_ATTR_STACKADDR
_SC_THREAD_ATTR_STACKSIZE
_SC_THREAD_PRIORITY_SCHEDULING
_SC_THREAD_PRIO_INHERIT
_SC_THREAD_PRIO_PROTECT
_SC_THREAD_PROCESS_SHARED
Two general options may also be checked using the sysconf subroutine with the
following Name parameter values:
_SC_REENTRANT_FUNCTIONS
_SC_THREADS
Denotes that reentrant functions are required.
Denotes the implementation of the threads library.
Threads Library Quick Reference
This section provides a summary of the threads library:
v “Supported Interfaces”
v “Threads Data Types” on page 265
v “Limits and Default Values” on page 265
Supported Interfaces
On AIX systems, _POSIX_THREADS, _POSIX_THREAD_ATTR_STACKADDR,
_POSIX_THREAD_ATTR_STACKSIZE and _POSIX_THREAD_PROCESS_SHARED
are always defined. Therefore, the following threads interfaces are supported:
POSIX Interfaces
pthread_atfork
pthread_attr_getdetachstate
pthread_attr_getstackaddr
pthread_attr_init
pthread_attr_setschedparam
pthread_attr_setstacksize
pthread_cleanup_pop
pthread_detach
pthread_exit
pthread_join
pthread_key_delete
pthread_mutex_destroy
262
General Programming Concepts
pthread_attr_destroy
pthread_attr_getschedparam
pthread_attr_getstacksize
pthread_attr_setdetachstate
pthread_attr_setstackaddr
pthread_cancel
pthread_cleanup_push
pthread_equal
pthread_getspecific
pthread_key_create
pthread_kill
pthread_mutex_init
pthread_mutex_lock
pthread_mutex_unlock
pthread_mutexattr_getpshared
pthread_mutexattr_setpshared
pthread_self
pthread_setcanceltype
pthread_sigmask
sigwait
pthread_cond_destroy
pthread_cond_signal
pthread_cond_wait
pthread_condattr_getpshared
pthread_condattr_setpshared
pthread_mutex_trylock
pthread_mutexattr_destroy
pthread_mutexattr_init
pthread_once
pthread_setcancelstate
pthread_setspecific
pthread_testcancel
pthread_cond_broadcast
pthread_cond_init
pthread_cond_timedwait
pthread_condattr_destroy
pthread_condattr_init
pthread_create
Single Unix Specification Interfaces
pthread_attr_getguardsize
pthread_getconcurrency
pthread_mutexattr_settype
pthread_rwlock_init
pthread_rwlock_tryrdlock
pthread_rwlock_unlock
pthread_rwlockattr_destroy
pthread_rwlockattr_init
pthread_setconcurrency
pthread_attr_setguardsize
pthread_mutexattr_gettype
pthread_rwlock_destroy
pthread_rwlock_rdlock
pthread_rwlock_trywrlock
pthread_rwlock_wrlock
pthread_rwlockattr_getpshared
pthread_rwlockattr_setpshared
On AIX systems, _POSIX_THREAD_SAFE_FUNCTIONS is always defined.
Therefore, the following interfaces are always supported:
asctime_r
flockfile
funlockfile
getchar_unlocked
getgrnam_r
getpwuid_r
localtime_r
putchar_unlocked
readdir_r
ctime_r
ftrylockfile
getc_unlocked
getgrgid_r
getpwnam_r
gmtime_r
putc_unlocked
rand_r
strtok_r
AIX does not support the following interfaces; the symbols are provided but they
always return an error and set the errno to ENOSYS:
pthread_attr_getinheritsched
pthread_attr_getscope
pthread_attr_setschedpolicy
pthread_getschedparam
pthread_mutex_setprioceiling
pthread_mutexattr_getprotocol
pthread_mutexattr_setprotocol
pthread_attr_getschedpolicy
pthread_attr_setinheritsched
pthread_attr_setscope
pthread_mutex_getprioceiling
pthread_mutexattr_getprioceiling
pthread_mutexattr_setprioceiling
pthread_setschedparam
Thread-safety
The following AIX interfaces are not thread-safe.
libc.a Library (Standard Functions):
advance
asctime
brk
catgets
Chapter 11. Threads Programming Guidelines
263
chroot
dbm_clearerr
dbm_fetch
dbm_store
encrypt
fcvt
getchar_unlocked
getgrent
getopt
getpwnam
getutxline
l64a
mrand48
putchar_unlocked
rand
re_exec
setgrent
setutxent
step
ttyslot
compile
dbm_close
dbm_firstkey
dirname
endgrent
gamma
getdate
getgrgid
getpagesize
getpwuid
getw
lgamma
nl_langinfo
putenv
random
regcmp
setkey
sigstack
strerror
wait3
ctime
dbm_delete
dbm_nextkey
drand48
endpwent
gcvt
getdtablesize
getgrnam
getpass
getutxent
getw
localtime
ptsname
pututxline
readdir
regex
setlocale
srand48
strtok
cuserid
dbm_error
dbm_open
ecvt
endutxent
getc_unlocked
getenv
getlogin
getpwent
getutxid
gmtime
lrand48
putc_unlocked
putw
re_comp
sbrk
setpwent
srandom
ttyname
libc.a Library (AIX Specific Functions):
endfsent
getfsent
getfstype
getutent
pututline
setutent
endttyent
getfsfile
getttyent
getutid
setfsent
utmpname
endutent
getfsspec
getttynam
getutline
setttyent
libbsd.a library:
timezone
libm.a and libmsaa.a libraries:
gamma
lgamma
None of the functions in the following libraries are thread safe:
v libPW.a
v libblas.a
v libcur.a
v libcurses.a
v libplot.a
v libprint.a
The interfaces ctermid and tmpnam are not thread-safe if passed a NULL
argument.
Note: Certain subroutines may be implemented as macros on some systems.
You should avoid using the address of threads subroutines.
264
General Programming Concepts
Threads Data Types
The following data types are defined for the threads library in the pthread.h
header file.
pthread_t
pthread_attr_t
pthread_cond_t
pthread_condattr_t
pthread_key_t
pthread_mutex_t
pthread_mutexattr_t
pthread_once_t
Identifies
Identifies
Identifies
Identifies
Identifies
Identifies
Identifies
Identifies
a
a
a
a
a
a
a
a
thread.
thread attributes object.
condition variable.
condition attributes object.
thread-specific data key.
mutex.
mutex attributes object.
one time initialization object.
The definition of these data types may vary from one system to another.
Limits and Default Values
The threads library has some implementation-dependent limits and default values.
These limits and default values can be retrieved by symbolic constants to enhance
the portability of programs.
Maximum Number of Threads per Process
The maximum number of threads per process is 512. The maximum number of
threads can be retrieved at compilation time using the
PTHREAD_THREADS_MAX symbolic constant defined in the pthread.h header
file.
Minimum Stack Size
The minimum stack size for a thread is 96KB. It is also the default stack size. This
number can be retrieved at compilation time using the PTHREAD_STACK_MIN
symbolic constant defined in the pthread.h header file.
Note that the maximum stack size is 256MB, the size of a segment. This limit is
indicated by the PTHREAD_STACK_MAX symbolic constant in the pthread.h
header file.
Maximum Number of Thread-Specific Data Keys
The number of thread-specific data keys is limited to 508. This number can be
retrieved at compilation time using the PTHREAD_KEYS_MAX symbolic constant
defined in the pthread.h header file.
Default Attribute Values
The default values for the thread attributes object are defined in the pthread.h
header file by the following symbolic constants:
DEFAULT_DETACHSTATE
DEFAULT_INHERIT
DEFAULT_PRIO
DEFAULT_SCHED
DEFAULT_SCOPE
PTHREAD_CREATE_DETACHED. Specifies the default
value for the detachstate attribute.
PTHREAD_INHERIT_SCHED. Specifies the default value for
the inheritsched attribute.
1 (one). Specifies the default value for the sched_prio field of
the schedparam attribute.
SCHED_OTHER. Specifies the default value for the
schedpolicy attribute of a thread attributes object.
PTHREAD_SCOPE_LOCAL. Specifies the default value for
the contention-scope attribute.
Chapter 11. Threads Programming Guidelines
265
The actual values shown may vary from one release to another.
266
General Programming Concepts
Chapter 12. lex and yacc Program Information
For a program to receive input, either interactively or in a batch environment, you
must provide another program or a routine to receive the input. Complicated input
requires additional code to break the input into pieces that mean something to the
program. You can use the lex and yacc commands to develop this type of input
program.
Creating an Input Language with the lex and yacc Commands
For a program to receive input, either interactively or in a batch environment, you
must provide another program or a routine to receive the input. Complicated input
requires additional code to break the input into pieces that mean something to the
program. You can use the lex and yacc commands to develop this type of input
program.
lex
yacc
Generates a lexical analyzer program that analyzes input and breaks it into tokens,
such as numbers, letters, or operators. The tokens are defined by grammar rules set
up in the lex specification file.
Generates a parser program that analyzes input using the tokens identified by the
lexical analyzer (generated by the lex command and stored in the lex specification
file) and performs specified actions, such as flagging improper syntax.
Writing a Lexical Analyzer Program with the lex Command
The lex command helps write a C language program that can receive and translate
character-stream input into program actions. To use the lex command, you must
supply or write a specification file that contains:
Extended regular expressions
Action statements
Character patterns that the generated lexical
analyzer recognizes.
C language program fragments that define how
the generated lexical analyzer reacts to extended
regular expressions it recognizes.
The format and logic allowed in this file are discussed in the lex Specification File
section of the lex command.
How the lex Command Operates
The lex command generates a C language program that can analyze an input
stream using information in the specification file. The lex command then stores the
output program in a lex.yy.c file. If the output program recognizes a simple,
one-word input structure, you can compile the lex.yy.c output file with the
following command to get an executable lexical analyzer:
cc lex.yy.c -ll
However, if the lexical analyzer must recognize more complex syntax, you can
create a parser program (“Creating a Parser with the yacc Program” on page 279)
to use with the output file to ensure proper handling of any input.
You can move a lex.yy.c output file to another system if it has C compiler that
supports the lex library functions.
The compiled lexical analyzer performs the following functions:
© Copyright IBM Corp. 1997, 1999
267
v Reads an input stream of characters.
v Copies the input stream to an output stream.
v Breaks the input stream into smaller strings that match the extended regular
expressions in the lex specification file.
v Executes an action for each extended regular expression that it recognizes. These
actions are C language program fragments in the lex specification file. Each
action fragment can call actions or subroutines outside of itself.
How the Lexical Analyzer Works
The lexical analyzer that the lex command generates uses an analysis method
called a deterministic finite-state automaton. This method provides for a limited
number of conditions that the lexical analyzer can exist in, along with the rules
that determine what state the lexical analyzer is in.
The automaton allows the generated lexical analyzer to look ahead more than one
or two characters in an input stream. For example, suppose you define two rules
in the lex specification file: one looks for the string ab and the other looks for the
string abcdefg. If the lexical analyzer receives an input string of abcdefh, it reads
characters to the end of input string before determining that it does not match the
string abcdefg. The lexical analyzer then returns to the rule that looks for the string
ab, decides that it matches part of the input, and begins trying to find another
match using the remaining input cdefh.
Extended Regular Expressions in the lex Command
Specifying extended regular expressions in a lex specification file is similar to
methods used in the sed or ed commands. An extended regular expression
specifies a set of strings to be matched. The expression contains both text
characters and operator characters. Text characters match the corresponding
characters in the strings being compared. Operator characters specify repetitions,
choices, and other features.
Numbers and letters of the alphabet are considered text characters. For example,
the extended regular expression integer matches the string integer, and the
expression a57D looks for the string a57D.
Operators
The following list describes how operators are used to specify extended regular
expressions:
Expression
Character
″String″
Use
Matches the character Character.
Example: a matches the literal character a; b matches
the literal character b, and c matches the literal
character c.
Matches the string enclosed within quotes, even if the
string includes an operator.
Example: to prevent the lex command from interpreting
$ (dollar sign) as an operator, enclose the symbol in
quotes.
268
General Programming Concepts
\Character or \Digits
Escape character. When preceding a character class
operator used in a string, the \ character indicates that
the operator symbol represents a literal character rather
than an operator. Valid escape sequences include:
\a
Alert
\b
Backspace
\f
Form-feed
\n
New-line character (Do not use the actual
new-line character in an expression.)
\r
Return
\t
Tab
\v
Vertical tab
\\
Backslash
\Digits The character whose encoding is represented
by the one-, two-, or three-digit octal integer
specified by the Digits string.
\xDigits
The character whose encoding is represented
by the sequence of hexadecimal characters
specified by the Digits string.
When the \ character precedes a character that
is not in the preceding list of escape sequences,
the lex command interprets the character
literally.
[List]
[:Class:]
Example: \c is interpreted as the c character
unchanged, and [\|abc] represents the class of
characters that includes the characters |abc.
Note: Never use \0 or \x0 in lex rules.
Matches any one character in the enclosed range ([x-y])
or the enclosed list ([xyz]) based on the locale in which
the lex command is invoked. All operator symbols, with
the exception of the following, lose their special
meaning within a bracket expression: - (dash), | (carat),
and \ (backslash).
Example: [abc-f] matches a, b, c, d, e, or f in the
En_US locale.
Matches any of the characters belonging to the character
class specified between the [::] delimiters as defined in
the LC_TYPE category in the current locale. The
following character class names are supported in all
locales:
alnum
alpha
blank
cntrl
digit
graph
lower
print
punct
space
upper
xdigit
The lex command also recognizes user-defined
character class names. The [::] operator is valid only in
a [] expression.
Example: [[:alpha:]] matches any character in the
alpha character class in the current locale, but
[:alpha:] matches only the characters :,a,l,p, and h.
Chapter 12. lex and yacc Program Information
269
[.CollatingSymbol.]
[=CollatingElement=]
[|Character]
CollatingElement-CollatingElement
Expression?
.
Expression*
Expression+
Matches the collating symbol specified within the [..]
delimiters as a single character. The [..] operator is
valid only in a [ ] expression. The collating symbol
must be a valid collating symbol for the current locale.
Example: [[.ch.]] matches c and h together while [ch]
matches c or h.
Matches the collating element specified within the [==]
delimiters and all collating elements belonging to its
equivalence class. The [==] operator is valid only in a
[] expression.
Example: If w and v belong to the same equivalence
class, [[=w=]] is the same as [wv] and matches w or v. If
w does not belong to an equivalence class, then [[=w=]]
matches w only.
Matches any character except the one following the |
(caret) symbol. The resultant character class consists
solely of single-byte characters. The character following
the | symbol can be a multibyte character, however for
this operator to match multibyte characters, you must
set %h and %m to greater than zero in the definitions
section.
Example: [|c] matches any character except c.
In a character class, indicates a range of characters
within the collating sequence defined for the current
locale. Ranges must be in ascending order. The ending
range point must collate equal to or higher than the
starting range point. Because the range is based on the
collating sequence of the current locale, a given range
may match different characters, depending on the locale
in which the lex command was invoked.
Matches either zero or one occurrence of the expression
immediately preceding the ? operator.
Example: ab?c matches either ac or abc.
Matches any character except the new-line character. In
order for . to match multi-byte characters, %z must be
set to greater than 0 in the definitions section of the lex
specification file. If %z is not set, . matches single-byte
characters only.
Matches zero or more occurrences of the expression
immediately preceding the * operator. For example, a*
is any number of consecutive a characters, including
zero. The usefulness of matching zero occurrences is
more obvious in complicated expressions.
Example: The expression, [A-Za-z][A-Za-z0-9]*
indicates all alphanumeric strings with a leading
alphabetic character, including strings that are only one
alphabetic character. You can use this expression for
recognizing identifiers in computer languages.
Matches one or more occurrences of the pattern
immediately preceding the + operator.
Example: a+ matches one or more instances of a. Also,
[a-z]+ matches all strings of lowercase letters.
270
General Programming Concepts
Expression|Expression
(Expression)
|Expression
Expression$
Expression1/Expression2
{DefinedName}
{Number1,Number2}
<StartCondition>
Indicates a match for the expression that precedes or
follows the | (pipe) operator.
Example: ab|cd matches either ab or cd.
Matches the expression in the parentheses. The ()
(parentheses) operator is used for grouping and causes
the expression within parentheses to be read into the
yytext array. A group in parentheses can be used in
place of any single character in any other pattern.
Example: (ab|cd+)?(ef)* matches such strings as
abefef, efefef, cdef, or cddd; but not abc, abcd, or
abcdef.
Indicates a match only when the | (carat) operator is at
the beginning of the line and the | is the first character
in an expression.
Example: |h matches an h at the beginning of a line.
Indicates a match only when the $ (dollar sign) is at the
end of the line and the $ is the last character in an
expression.
Example: The description of Expression/Expression.
Indicates a match only if Expression2 immediately
follows Expression1. The / (slash) operator reads only
the first expression into the yytext array.
Example: ab/cd matches the string ab, but only if
followed by cd, and then reads ab into the yytext array.
Note: Only one / trailing context operator can be
used in a single extended regular expression. The
| (carat) and $ (dollar sign) operators cannot be
used in the same expression with the / operator
as they indicate special cases of trailing context.
Matches the name as you defined it in the definitions
section.
Example: If you defined D to be numerical digits, {D}
matches all numerical digits.
Matches Number1 to Number2 occurrences of the pattern
immediately preceding it. The expressions {Number}
and {Number,} are also allowed and match exactly
Number occurrences of the pattern preceding the
expression.
Example: xyz{2,4} matches either xyzxyz, xyzxyzxyz,
or xyzxyzxyzxyz. This differs from the +, * and ?
operators in that these operators match only the
character immediately preceding them. To match only
the character preceding the interval expression, use the
grouping operator. For example, xy(z{2,4}) matches
xyzz, xyzzz or xyzzzz.
Executes the associated action only if the lexical
analyzer is in the indicated start condition (“lex Start
Conditions” on page 277).
Example: If being at the beginning of a line is start
condition ONE, then the | (caret) operator equals the
expression, <ONE>.
Chapter 12. lex and yacc Program Information
271
To use the operator characters as text characters, use one of the escape sequences: ″
″ (double quotation marks) or \ (backslash). The ″ ″ operator indicates what is
enclosed is text. Thus, the following example matches the string xyz++:
xyz"++"
Note that a part of a string may be quoted. Quoting an ordinary text character has
no effect. For example, the following expression is equivalent to the previous
example:
"xyz++"
Quoting all characters that are not letters or numbers ensures that text is
interpreted as text.
Another way to turn an operator character into a text character is to put a \
(backslash) character before the operator character. For example, the following
expression is equivalent to the preceding examples:
xyz\+\+
lex Actions
When the lexical analyzer matches one of the extended regular expressions in the
rules section of the specification file, it executes the action that corresponds to the
extended regular expression. Without sufficient rules to match all strings in the
input stream, the lexical analyzer copies the input to standard output. Therefore,
do not create a rule that only copies the input to the output. The default output
can help find gaps in the rules.
When using the lex command to process input for a parser that the yacc command
produces, provide rules to match all input strings. Those rules must generate
output that the yacc command can interpret.
Null Action
To ignore the input associated with an extended regular expression, use a ; (C
language null statement) as an action. The following example ignores the three
spacing characters (blank, tab, and new-line):
[ \t\n] ;
Same As Next Action
To avoid repeatedly writing the same action, use the | (pipe symbol). This
character indicates that the action for this rule is the same as the action for the next
rule. For instance, the previous example to ignore blank, tab, and new-line
characters can also be written as follows:
" "
|
"\t"
|
"\n"
;
The quotation marks around \n and \t are not required.
Printing a Matched String
To find out what text matched an expression in the rules section of the
specification file, you can include a C language printf subroutine call as one of the
actions for that expression. When the lexical analyzer finds a match in the input
stream, the program puts the matched string into the external character (char) and
wide character (wchar_t) arrays, called yytext and yywtext, respectively. For
example, you can use the following rule to print the matched string:
272
General Programming Concepts
[a-z]+
printf("%s",yytext);
The C language printf subroutine accepts a format argument and data to be
printed. In this example the arguments to the printf subroutine have the following
meanings:
%s
%S
yytext
yywtext
A symbol that converts the data to type string before printing
A symbol that converts the data to wide character string (wchar_t) before
printing
The name of the array containing the data to be printed
The name of the array containing the multibyte type (wchar_t) data to be
printed.
The lex command defines ECHO; as a special action to print out the contents of
yytext. For example, the following two rules are equivalent:
[a-z]+
[a-z]+
ECHO;
printf("%s",yytext);
You can change the representation of yytext by using either %array or %pointer in
the definitions section of the lex specification file.
%array
%pointer
Defines yytext as a null-terminated character array. This is the default action.
Defines yytext as a pointer to a null-terminated character string.
Finding the Length of a Matched String
To find the number of characters that the lexical analyzer matched for a particular
extended regular expression, use the yyleng or the yywleng external variables.
yyleng
yywleng
Tracks the number of bytes that are matched.
Tracks the number of wide characters in the matched string. Multibyte
characters have a length greater than 1.
To count both the number of words and the number of characters in words in the
input, use the following action:
[a-zA-Z]+
{words++;chars += yyleng;}
This action totals the number of characters in the words matched and puts that
number in chars.
The following expression finds the last character in the string matched:
yytext[yyleng-1]
Matching Strings within Strings
The lex command partitions the input stream and does not search for all possible
matches of each expression. Each character is accounted for only once. To override
this choice and search for items that may overlap or include each other, use the
REJECT directive. For example, to count all instance of she and he, including the
instances of he that are included in she, use the following action:
she
he
\n
.
{s++; REJECT;}
{h++}
|
;
After counting the occurrences of she, the lex command rejects the input string and
then counts the occurrences of he. Because he does not include she, a REJECT
action is not necessary on he.
Chapter 12. lex and yacc Program Information
273
Getting More Input
Normally, the next string from the input stream overwrites the current entry in the
yytext array. If you use the yymore subroutine, the next string from the input
stream is added to the end of the current entry in the yytext array.
For example, the following lexical analyzer looks for strings:
%s instring
%%
<INITIAL>\"
{ /* start of string */
BEGIN instring;
yymore();
}
<instring>\"
{ /* end of string */
printf("matched %s\n", yytext);
BEGIN INITIAL;
}
<instring>.
{
yymore();
}
<instring>\n
{
printf("Error, new line in string\n");
BEGIN INITIAL;
}
Even though a string may be recognized by matching several rules, repeated calls
to the yymore subroutine ensure that the yytext array will contain the entire string.
Putting Characters Back
To return characters to the input stream, use the call:
yyless(n)
where n is the number of characters of the current string to keep. Characters in the
string beyond this number are returned to the input stream. The yyless subroutine
provides the same type of look-ahead that the / (slash) operator uses, but it allows
more control over its usage.
Use the yyless subroutine to process text more than once. For example, when
parsing a C language program, an expression such asx=-a is difficult to
understand. Does it mean x is equal to minus a, or is it an older representation of x
-= a which means decrease x by the value of a? To treat this expression as x is equal to
minus a, but print a warning message, use a rule such as:
=-[a-zA-Z]
{
printf("Operator (=-) ambiguous\n");
yyless(yyleng-1);
... action for = ...
}
Input/Output Subroutines
The lex program allows a program to use the following input/output (I/O)
subroutines:
input()
output(c)
unput(c)
winput()
woutput(C)
wunput(C)
274
General Programming Concepts
Returns the next input character.
Writes the character c on the output.
Pushes the character c back onto the input stream to be read later by the
input subroutine.
Returns the next multibyte input character.
Writes the multibyte character C back onto the output stream.
Pushes the multibyte character C back onto the input stream to be read by
the winput subroutine.
lex provides these subroutines as macro definitions. The subroutines are coded in
the lex.yy.c file. You can override them and provide other versions.
The winput, wunput, and woutput macros are defined to use the yywinput,
yywunput, and yywoutput subroutines. For compatibility, the yy subroutines
subsequently use the input, unput, and output subroutine to read, replace, and
write the necessary number of bytes in a complete multibyte character.
These subroutines define the relationship between external files and internal
characters. If you change the subroutines, change them all in the same way. They
should follow these rules:
v All subroutines must use the same character set.
v The input subroutine must return a value of 0 to indicate end of file.
v Do not change the relationship of the unput subroutine to the input subroutine
or the look-ahead functions will not work.
The lex.yy.c file allows the lexical analyzer to back up a maximum of 200
characters.
To read a file containing nulls, create a different version of the input subroutine. In
the normal version of the input subroutine, the returned value of 0 (from the null
characters) indicates the end of file and ends the input.
Character Set
The lexical analyzers that the lex command generates process character I/O
through the input, output, and unput subroutines. Therefore, to return values in
the yytext subroutine, the lex command uses the character representation that these
subroutines use. Internally however, the lex command represents each character
with a small integer. When using the standard library, this integer is the value of
the bit pattern the computer uses to represent the character. Normally, the letter ’a’
is represented in the same form as the character constant ’a’. If you change this
interpretation with different I/O subroutines, put a translation table in the
definitions section of the specification file. The translation table begins and ends
with lines that contain only the entries:
%T
The translation table contains additional lines that indicate the value associated
with each character. For example:
%T
{integer}
{integer}
{integer}
%T
{character string}
{character string}
{character string}
End-of-File Processing
When the lexical analyzer reaches the end of a file, it calls the yywrap library
subroutine.
yywrap
Returns a value of 1 to indicate to the lexical analyzer that it should continue
with normal wrap-up at the end of input
However, if the lexical analyzer receives input from more than one source, change
the yywrap subroutine. The new function must get the new input and return a
value of 0 to the lexical analyzer. A return value of 0 indicates the program should
continue processing.
Chapter 12. lex and yacc Program Information
275
You can also include code to print summary reports and tables when the lexical
analyzer ends in a new version of the yywrap subroutine. The yywrap subroutine
is the only way to force the yylex subroutine to recognize the end of input.
Passing Code to the Generated lex Program
The lex command passes C code, unchanged, to the lexical analyzer in the
following circumstances:
v Lines beginning with a blank or tab in the definitions section, or at the start of
the rules section before the first rule, are copied into the lexical analyzer. If the
entry is in the definitions section, it is copied to the external declaration area of
the lex.yy.c file. If the entry is at the start of the rules section, the entry is copied
to the local declaration area of the yylex subroutine in the lex.yy.c file.
v Lines that lie between delimiter lines containing only %{ (percent sign, left brace)
and %} (percent sign, right brace) either in the definitions section or at the start
of the rules section are copied into the lexical analyzer in the same way as lines
beginning with a blank or tab.
v Any lines occurring after the second %% (percent sign, percent sign) delimiter are
copied to the lexical analyzer without format restrictions.
Defining lex Substitution Strings
You can define string macros that the lex program expands when it generates the
lexical analyzer. Define them before the first %% delimiter in the lex specification
file. Any line in this section that begins in column 1 and that does not lie between
%{ and %} defines a lex substitution string. Substitution string definitions have the
general format:
name
translation
where name and translation are separated by at least one blank or tab, and the
specified name begins with a letter. When the lex program finds the string defined
by name enclosed in {} (braces) in the rules part of the specification file, it changes
that name to the string defined in translation and deletes the braces.
For example, to define the names D and E, put the following definitions before the
first %% delimiter in the specification file:
D
E
[0-9]
[DEde][-+]{D}+
Then, use these names in the rules section of the specification file to make the rules
shorter:
{D}+
printf("integer");
{D}+"."{D}*({E})?
|
{D}*"."{D}+({E})?
|
{D}+{E}
printf("real");
You can also include the following items in the definitions section:
v Character set table
v List of start conditions
v Changes to size of arrays to accommodate larger source programs
276
General Programming Concepts
lex Start Conditions
A rule may be associated with any start condition. However, the lex program
recognizes the rule only when in that associated start condition. You can change
the current start condition at any time.
Define start conditions in the definitions section of the specification file by using a
line in the following form:
%Start name1 name2
where name1 and name2 define names that represent conditions. There is no limit to
the number of conditions, and they can appear in any order. You can also shorten
the word Start to s or S.
When using a start condition in the rules section of the specification file, enclose
the name of the start condition in <> (less than, greater than) symbols at the
beginning of the rule. The following example defines a rule, expression, that the
lex program recognizes only when the lex program is in start condition name1:
<name1>expression
To put the lex program in a particular start condition, execute the action statement
in the action part of a rule; for instance, BEGIN in the following line:
BEGIN name1;
This statement changes the start condition to name1.
To resume the normal state, enter:
BEGIN 0;
or
BEGIN INITIAL;
where INITIAL is defined to be 0 by the lex program. BEGIN 0; resets the lex
program to its initial condition.
The lex program also supports exclusive start conditions specified with %x
(percent sign, lowercase x) or %X (percent sign, uppercase X) operator followed by
a list of exclusive start names in the same format as regular start conditions.
Exclusive start conditions differ from regular start conditions in that rules that do
not begin with a start condition are not active when the lexical analyzer is in an
exclusive start state. For example:
%s
one
%x
two
%%
abc
{printf("matched ");ECHO;BEGIN one;}
<one>def
printf("matched ");ECHO;BEGIN two;}
<two>ghi
{printf("matched ");ECHO;BEGIN INITIAL;}
In start state one in the preceding example, both abc and def can be matched. In
start state two, only ghi can be matched.
Compiling the Lexical Analyzer
Compiling a lex program is a two-step process:
Chapter 12. lex and yacc Program Information
277
1. Use the lex program to change the specification file into a C language program.
The resulting program is in the lex.yy.c file.
2. Use the cc command with the -ll flag to compile and link the program with a
library of lex subroutines. The resulting executable program is in the a.out file.
For example, if the lex specification file is called lextest, enter the following
commands:
lex lextest
cc lex.yy.c -ll
lex Library
The lex library contains the following subroutines:
main()
yywrap()
yymore()
yyless(int n)
yyreject()
Invokes the lexical analyzer by calling the yylex subroutine.
Returns the value 1 when the end of input occurs.
Appends the next matched string to the current value of the yytext
array rather than replacing the contents of the yytext array.
Retains n initial characters in the yytext array and returns the
remaining characters to the input stream.
Allows the lexical analyzer to match multiple rules for the same input
string. (yyreject is called when the special action REJECT is used.)
Some of the lex subroutines can be substituted by user-supplied routines. For
example, lex supports user-supplied versions of the main and yywrap subroutines.
The library versions of these routines, provided as a base, are as follows:
main
#include <stdio.h>
#include <locale.h>
main() {
setlocale(LC_ALL, "");
yylex();
exit(0);
}
yywrap
yywrap() {
return(1);
}
yymore, yyless, and yyreject subroutines are available only through the lex library;
however, these subroutines are required only when used in lex actions.
Using the lex Program with the yacc Program
When used alone, the lex program generator makes a lexical analyzer that
recognizes simple, one-word input or receives statistical input. You can also use the
lex program with a parser generator, such as the yacc command. The yacc
command generates a program, called a parser, that analyzes the construction of
more than one-word input. This parser program operates well with the lexical
analyzers that the lex command generates. The parsers recognize many types of
grammar with no regard to context. These parsers need a preprocessor to recognize
input tokens such as the preprocessor that the lex command produces.
The lex program recognizes only extended regular expressions and formats them
into character packages called tokens, as specified by the input file. When using
278
General Programming Concepts
the lex program to make a lexical analyzer for a parser, the lexical analyzer
(created from the lex command) partitions the input stream. The parser (from the
yacc command) assigns structure to the resulting pieces. You can also use other
programs along with the programs generated by either the lex or yacc commands.
A token is the smallest independent unit of meaning as defined by either the
parser or the lexical analyzer. A token can contain data, a language keyword, an
identifier, or other parts of a language syntax.
The yacc program looks for a lexical analyzer subroutine named yylex, which is
generated by the lex command. Normally, the default main program in the lex
library calls the yylex subroutine. However, if the yacc command is loaded and its
main program is used, yacc calls the yylex subroutine. In this case, each lex rule
should end with:
return(token);
where the appropriate token value is returned.
The yacc command assigns an integer value to each token defined in the yacc
grammar file through a #define preprocessor statement. The lexical analyzer must
have access to these macros to return the tokens to the parser. Use the yacc -d
option to create a y.tab.h file, and include the y.tab.h file in the lex specification
file by adding the following lines to the definition section of the lex specification
file:
%{
#include "y.tab.h"
%}
Alternately, you can include the lex.yy.c file in the yacc output file by adding the
following line after the second %% (percent sign, percent sign) delimiter in the yacc
grammar file:
#include "lex.yy.c"
The yacc library should be loaded before the lex library to get a main program that
invokes the yacc parser. You can generate lex and yacc programs in either order.
Creating a Parser with the yacc Program
The yacc program creates parsers that define and enforce structure for character
input to a computer program. To use this program, you must supply the following
inputs:
Grammar file
main
yyerror
yylex
A source file that contains the specifications for the language to
recognize. This file also contains the main, yyerror, and yylex
subroutines. You must supply these subroutines.
A C language subroutine that, as a minimum, contains a call to the
yyparse subroutine generated by the yacc program. A limited form of
this subroutine is available in the yacc library.
A C language subroutine to handle errors that can occur during parser
operation. A limited form of this subroutine is available in the yacc
library.
A C language subroutine to perform lexical analysis on the input stream
and pass tokens to the parser. You can generate this lexical analyzer
subroutine using the lex command.
Chapter 12. lex and yacc Program Information
279
When the yacc command gets a specification, it generates a file of C language
functions called y.tab.c. When compiled using the cc command, these functions
form the yyparse subroutine and return an integer. When called, the yyparse
subroutine calls the yylex subroutine to get input tokens. The yylex subroutine
continues providing input until either the parser detects an error or the yylex
subroutine returns an end-marker token to indicate the end of operation. If an
error occurs and the yyparse subroutine cannot recover, it returns a value of 1 to
main. If it finds the end-marker token, the yyparse subroutine returns a value of 0
to main.
yacc Grammar File
To use the yacc command to generate a parser, give it a grammar file that
describes the input data stream and what the parser is to do with the data. The
grammar file includes rules describing the input structure, code to be invoked
when these rules are recognized, and a subroutine to do the basic input.
The yacc command uses the information in the grammar file to generate a parser
that controls the input process. This parser calls an input subroutine (the lexical
analyzer) to pick up the basic items (called tokens) from the input stream. The
parser organizes these tokens according to the structure rules in the grammar file.
The structure rules are called grammar rules. When the parser recognizes one of
these rules, it executes the user code supplied for that rule. The user code is called
an action. Actions return values and use the values returned by other actions.
Use the C programming language to write the action code and other subroutines.
The yacc command uses many of the C language syntax conventions for the
grammar file.
main and yyerror Subroutines
You must provide the main and yyerror subroutines for the parser. To ease the
initial effort of using the yacc command, the yacc library contains simple versions
of the main and yyerror subroutines. Include these subroutines by using the -ly
argument to the ld command (or to the cc command). The source code for the
main library program is:
#include <locale.h>
main()
{
setlocale(LC_ALL, "");
yyparse();
}
The source code for the yyerror library program is:
#include <stdio.h>
yyerror(s)
char *s;
{
fprintf( stderr, "%s\n" ,s);
}
The argument to the yyerror subroutine is a string containing an error message,
usually the string syntax error.
These are very limited programs. You should provide more function in these
subroutines. For example, keep track of the input line number and print it along
with the message when a syntax error is detected. You may also want to use the
280
General Programming Concepts
value in the external integer variable yychar. This variable contains the look-ahead
token number at the time the error was detected.
yylex Subroutine
The input subroutine that you supply must be able to:
v Read the input stream.
v Recognize basic patterns in the input stream.
v Pass the patterns to the parser along with tokens that define the pattern to the
parser.
A token is a symbol or name that tells the parser which pattern is being sent to it
by the input subroutine. A nonterminal symbol is the structure that the parser
recognizes.
For example, if the input subroutine separates an input stream into the tokens of
WORD, NUMBER, and PUNCTUATION, and it receives the following input:
I have 9 turkeys.
the program could choose to pass the following strings and tokens to the parser:
String
I
have
9
turkeys
.
Token
WORD
WORD
NUMBER
WORD
PUNCTUATION
The parser must contain definitions for the tokens passed to it by the input
subroutine. Using the -d option for the yacc command, it generates a list of tokens
in a file called y.tab.h. This list is a set of #define statements that allow the lexical
analyzer (yylex) to use the same tokens as the parser.
To avoid conflict with the parser, do not use names that begin with the letters yy.
You can use the lex command to generate the input subroutine or you can write
the routine in the C language.
Using the yacc Grammar File
A yacc grammar file consists of three sections:
v Declarations
v Rules
v Programs
Two adjacent %% (double percent signs) separate each section of the grammar file.
To make the file easier to read, put the %% on a line by themselves. A complete
grammar file looks like:
declarations
%%
rules
%%
programs
The declarations section may be empty. If you omit the programs section, omit the
second set of %%. Therefore, the smallest yacc grammar file is:
Chapter 12. lex and yacc Program Information
281
%%
rules
The yacc command ignores blanks, tabs and new-line characters in the grammar
file. Therefore, use these characters to make the grammar file easier to read. Do
not, however, use blanks, tabs or new-lines in names or reserved symbols.
Using Comments
Put comments in the grammar file to explain what the program is doing. You can
put comments anywhere in the grammar file you can put a name. However, to
make the file easier to read, put the comments on lines by themselves at the
beginning of functional blocks of rules. A comment in a yacc grammar file looks
the same as a comment in a C language program. The comment is enclosed
between /* (backslash, asterisk) and */ (asterisk, backslash). For example:
/* This is a comment on a line by itself. */
Using Literal Strings
A literal string is one or more characters enclosed in ’’ (single quotes). As in the C
language, the \ (backslash) is an escape character within literals, and all the C
language escape codes are recognized. Thus, the yacc command accepts the
symbols in the following table:
Symbol
’\a’
’\n’
’\r’
’\’’
’\″’
’\?’
’\\’
’\t’
’\v’
’\b’
’\f’
’\Digits’
’\xDigits’
Definition
Alert
New-line
Return
Single quote (’)
Double quote (″)
Question mark (?)
Backslash (\)
Tab
Vertical tab
Backspace
Form-feed
The character whose encoding is represented by the one-, two-, or
three-digit octal integer specified by the Digits string.
The character whose encoding is represented by the sequence of
hexadecimal characters specified by the Digits string.
Never use \0 or 0 (the null character) in grammar rules.
Formatting the Grammar File
Use the following guidelines to help make the yacc grammar file more readable:
v Use uppercase letters for token names and lowercase letters for nonterminal
symbol names.
v Put grammar rules and actions on separate lines to allow changing either one
without changing the other.
v Put all rules with the same left side together. Enter the left side once, and use
the vertical bar to begin the rest of the rules for that left side.
v For each set of rules with the same left side, enter the semicolon once on a line
by itself following the last rule for that left side. You can then add new rules
easily.
v Indent rule bodies by two tab stops and action bodies by three tab stops.
282
General Programming Concepts
Errors in the Grammar File
The yacc command cannot produce a parser for all sets of grammar specifications.
If the grammar rules contradict themselves or require matching techniques that are
different from what the yacc command provides, the yacc command will not
produce a parser. In most cases, the yacc command provides messages to indicate
the errors. To correct these errors, redesign the rules in the grammar file, or
provide a lexical analyzer (input program to the parser) to recognize the patterns
that the yacc command cannot.
yacc Declarations
The declarations section of the yacc grammar file contains:
v Declarations for any variables or constants used in other parts of the grammar
file
v #include statements to use other files as part of this file (used for library header
files)
v Statements that define processing conditions for the generated parser
It is also possible to keep semantic information associated with the tokens
currently on the parse stack in a user-defined C language union, if the members of
the union are associated with the various names in the grammar file.
A declaration for a variable or constant uses the syntax of the C programming
language:
TypeSpecifier Declarator ;
TypeSpecifier is a data type keyword and Declarator is the name of the variable or
constant. Names can be any length and consist of letters, dots, underscores, and
digits. A name cannot begin with a digit. Uppercase and lowercase letters are
distinct.
Terminal (or token) names can be declared using the %token declaration and
nonterminal names can be declared using the %type declaration. The %type
declaration is not required for nonterminal names. Nonterminal names are defined
automatically if they appear on the left side of at least one rule. Without declaring
a name in the declarations section, you can use that name only as a nonterminal
symbol. The #include statements are identical to C language syntax and perform
the same function.
The yacc program has a set of keywords that define processing conditions for the
generated parser. Each of the keywords begin with a % (percent sign) and is
followed by a list of tokens or nonterminal names. These keywords are:
%left
%nonassoc
%right
%start
%token
%type
%union
Identifies tokens that are left-associative with other tokens.
Identifies tokens that are not associative with other tokens.
Identifies tokens that are right-associative with other tokens.
Identifies a nonterminal name for the start symbol.
Identifies the token names that the yacc command accepts. Declares all token
names in the declarations section.
Identifies the type of nonterminals. Type checking is performed when this
construct is present.
Identifies the yacc value stack as the union of the various type of values
desired. By default, the values returned are integers. The effect of this
construct is to provide the declaration of YYSTYPE directly from the input.
Chapter 12. lex and yacc Program Information
283
%{
Code
%}
Copies the specified Code into the code file. This construct can be used to add C
language declarations and definitions to the declarations section.
Note: The %{ (percent sign, left bracket) and %} (percent sign, right bracket)
symbols must appear on lines by themselves.
The %token, %left, %right, and %nonassoc keywords optionally support the name of
a C union member (as defined by %union) called a <Tag> (literal angle brackets
surrounding a union member name). The %type keyword requires a <Tag>. The use
of <Tag> specifies that the tokens named on the line are to be of the same C type as
the union member referenced by <Tag>. For example, the declaration:
%token [<Tag>] Name [Number] [Name [Number]]...
declares the Name parameter to be a token. If <Tag> is present, the C type for all
tokens on this line are declared to be of the type referenced by <Tag>. If a positive
integer, Number, follows the Name parameter, that value is assigned to the token.
All of the tokens on the same line have the same precedence level and
associativity. The lines appear in the file in order of increasing precedence or
binding strength. For example:
%left '+' '-'
%left '*' '/'
describes the precedence and associativity of the four arithmetic operators. The +
(plus sign) and - (minus sign) are left associative and have lower precedence than
* (asterisk) and / (slash), which are also left associative.
Defining Global Variables
To define variables to be used by some or all actions, as well as by the lexical
analyzer, enclose the declarations for those variables between %{ (percent sign, left
bracket) and %} (percent sign, right bracket) symbols. Declarations enclosed in
these symbols are called global variables. For example, to make the var variable
available to all parts of the complete program, use the following entry in the
declarations section of the grammar file:
%{
int var = 0;
%}
Start Conditions
The parser recognizes a special symbol called the start symbol. The start symbol is
the name of the rule in the rules section of the grammar file that describes the
most general structure of the language to be parsed. Because it is the most general,
this is the structure where the parser starts in its top-down analysis of the input
stream. Declare the start symbol in the declarations section using the %start
keyword. If you do not declare the name of the start symbol, the parser uses the
name of the first grammar rule in the grammar file.
For example, when parsing a C language function, the most general structure for
the parser to recognize is:
main()
{
code_segment
}
284
General Programming Concepts
The start symbol should point to the rule that describes this structure. All
remaining rules in the file describe ways to identify lower-level structures within
the function.
Token Numbers
Token numbers are nonnegative integers that represent the names of tokens. If the
lexical analyzer passes the token number to the parser instead of the actual token
name, both programs must agree on the numbers assigned to the tokens.
You can assign numbers to the tokens used in the yacc grammar file. If you do not
assign numbers to the tokens, the yacc grammar file assigns numbers using the
following rules:
1. A literal character is the numerical value of the character in the ASCII character
set.
2. Other names are assigned token numbers starting at 257.
Note: Do not assign a token number of 0. This number is assigned to the
endmarker token. You cannot redefine it.
To assign a number to a token (including literals) in the declarations section of the
grammar file, put a positive integer (not 0) immediately following the token name
in the %token line. This integer is the token number of the name or literal. Each
token number must be unique. All lexical analyzers used with the yacc command
must return a 0 or a negative value for a token when they reach the end of their
input.
yacc Rules
The rules section contains one or more grammar rules. Each rule describes a
structure and gives it a name. A grammar rule has the form:
A : BODY;
where A is a nonterminal name, and BODY is a sequence of 0 or more names,
literals, and semantic actions that can optionally be followed by precedence rules.
Only the names and literals are required to form the grammar. Semantic actions
and precedence rules are optional. The colon and the semicolon are required yacc
punctuation.
Semantic actions allow you to associate actions to be performed each time a rule is
recognized in the input process. An action can be an arbitrary C statement, and as
such, perform input or output, call subprograms, or alter external variables.
Actions can also refer to the actions of the parser; for example, shift and reduce.
Precedence rules are defined by the %prec keyword and change the precedence
level associated with a particular grammar rule. The reserved symbol %prec can
appear immediately after the body of the grammar rule and can be followed by a
token name or a literal. The construct causes the precedence of the grammar rule
to become that of the token name or literal.
Repeating Nonterminal Names
If several grammar rules have the same nonterminal name, use the | (pipe symbol)
to avoid rewriting the left side. In addition, use the ; (semicolon) only at the end
of all rules joined by pipe symbols. For example, the grammar rules:
A : B C D ;
A : E F ;
A : G ;
Chapter 12. lex and yacc Program Information
285
can be given to the yacc command by using the pipe symbol as follows:
A : B C D
| E F
| G
;
Using Recursion in a Grammar File
Recursion is the process of using a function to define itself. In language definitions,
these rules normally take the form:
rule
:
|
EndCase
rule EndCase
Therefore, the simplest case of the rule is the EndCase, but rule can also consist of
more than one occurrence of EndCase. The entry in the second line that uses rule
in the definition of rule is the recursion. The parser cycles through the input until
the stream is reduced to the final EndCase.
When using recursion in a rule, always put the call to the name of the rule as the
leftmost entry in the rule (as it is in the preceding example). If the call to the name
of the rule occurs later in the line, such as in the following example, the parser
may run out of internal stack space and stop.:
rule
:
|
EndCase
EndCase rule
The following example defines the line rule as one or more combinations of a
string followed by a newline character (\n):
lines
:
|
;
line
line
lines line
:
string '\n'
;
Empty String
To indicate a nonterminal symbol that matches the empty string, use a ;
(semicolon) by itself in the body of the rule. To define a symbol empty that matches
the empty string, use a rule similar to the following rule:
empty
: ;
| x;
OR
empty
:
| x
;
End-of-Input Marker
When the lexical analyzer reaches the end of the input stream, it sends an
end-of-input marker to the parser. This marker is a special token called endmarker,
and has a token value of 0. When the parser receives an end-of-input marker, it
checks to see that it has assigned all input to defined grammar rules and that the
processed input forms a complete unit (as defined in the yacc grammar file). If the
input is a complete unit, the parser stops. If the input is not a complete unit, the
parser signals an error and stops.
The lexical analyzer must send the end-of-input marker at the appropriate time,
such as the end of a file, or the end of a record.
286
General Programming Concepts
yacc Actions
With each grammar rule, you can specify actions to be performed each time the
parser recognizes the rule in the input stream. An action is a C language statement
that does input and output, calls subprograms, and alters external vectors and
variables. Actions return values and obtain the values returned by previous
actions. The lexical analyzer can also return values for tokens.
Specify an action in the grammar file with one or more statements enclosed in {}
(braces). The following examples are grammer rules with actions:
A : '('B')'
{
hello(1, "abc" );
}
AND
XXX : YYY ZZZ
{
printf("a message\n");
flag = 25;
}
Passing Values between Actions
To get values generated by other actions, an action can use the yacc parameter
keywords that begin with a dollar sign ($1, $2, ... ). These keywords refer to the
values returned by the components of the right side of a rule, reading from left to
right. For example, if the rule is:
A : B C D ;
then $1 has the value returned by the rule that recognized B, $2 has the value
returned by the rule that recognized C, and $3 the value returned by the rule that
recognized D.
To return a value, the action sets the pseudo-variable $$ to some value. For
example, the following action returns a value of 1:
{ $$ = 1;}
By default, the value of a rule is the value of the first element in it ($1). Therefore,
you do not need to provide an action for rules that have the following form:
A : B ;
The following additional yacc parameter keywords beginning with a $ (dollar sign)
allow for type checking:
v $<Tag>$
v $<Tag>Number
$<Tag>Number imposes on the reference the type of the union member referenced
by <Tag>. This adds .tag to the reference so that the union member identified by
Tag is accessed. This construct is equivalent to specifying $$.Tag or $1.Tag. You
may use this construct when you use actions in the middle of rules where the
return type cannot be specified through a %type declaration. If a %type has been
declared for a nonterminal name, do not use the <Tag> construct; the union
reference will be done automatically
Chapter 12. lex and yacc Program Information
287
Putting Actions in the Middle of Rules
To get control of the parsing process before a rule is completed, write an action in
the middle of a rule. If this rule returns a value through the $ keywords, actions
that follow this rule can use that value. This rule can also use values returned by
actions that precede it. Therefore, the rule:
A : B
{
$$ =1;
}
C
{
x = $2;
y = $3;
}
;
sets x to 1 and y to the value returned by C. The value of rule A is the value
returned by B, following the default rule.
Internally, the yacc command creates a new nonterminal symbol name for the
action that occurs in the middle. It also creates a new rule matching this name to
the empty string. Therefore, the yacc command treats the preceding program as if
it were written in the following form:
$ACT : /* empty */
{
$$ = 1;
}
;
A
: B $ACT C
{
x = $2;
y = $3;
}
;
where $ACT is an empty action.
yacc Error Handling
When the parser reads an input stream, that input stream might not match the
rules in the grammar file. The parser detects the problem as early as possible. If
there is an error-handling subroutine in the grammar file, the parser can allow for
entering the data again, skipping over the bad data, or initiating a cleanup and
recovery action. When the parser finds an error, for example, it may need to
reclaim parse tree storage, delete or alter symbol table entries, and set switches to
avoid generating further output.
When an error occurs, the parser stops unless you provide error-handling
subroutines. To continue processing the input to find more errors, restart the parser
at a point in the input stream where the parser can try to recognize more input.
One way to restart the parser when an error occurs is to discard some of the
tokens following the error. Then try to restart the parser at that point in the input
stream.
The yacc command has a special token name, error, to use for error handling. Put
this token in the rules file at places that an input error might occur so that you can
provide a recovery subroutine. If an input error occurs in this position, the parser
executes the action for the error token, rather than the normal action.
288
General Programming Concepts
The following macros can be placed in yacc actions to assist in error handling:
YYERROR
YYABORT
YYACCEPT
YYRECOVERING()
Causes the parser to initiate error handling.
Causes the parser to return with a value of 1.
Causes the parser to return with a value of 0.
Returns a value of 1 if a syntax error has been detected and the
parser has not yet fully recovered.
To prevent a single error from producing many error messages, the parser remains
in error state until it processes 3 tokens following an error. If another error occurs
while the parser is in the error state, the parser discards the input token and does
not produce a message.
For example, a rule of the form:
stat : error ';'
tells the parser that when there is an error, it should skip over the token and all
following tokens until it finds the next semicolon. All tokens after the error and
before the next semicolon are discarded. After finding the semicolon, the parser
reduces this rule and performs any cleanup action associated with it.
Providing for Error Correction
You can also allow the person entering the input stream in an interactive
environment to correct any input errors by entering a line in the data stream again.
For example:
input : error '\n'
{
printf(" Reenter last line: " );
}
input
{
$$ = $4;
}
;
is one way to do this. However, in this example the parser stays in the error state
for 3 input tokens following the error. If the corrected line contains an error in the
first 3 tokens, the parser deletes the tokens and does not give a message. To allow
for this condition, use the yacc statement:
yyerrok;
When the parser finds this statement, it leaves the error state and begins
processing normally. The error-recovery example then becomes:
input : error '\n'
{
yyerrok;
printf(" Reenter last line: " );
}
input
{
$$ = $4;
}
;
Clearing the Look-Ahead Token
The look-ahead token is the next token that the parser examines. When an error
occurs, the look-ahead token becomes the token at which the error was detected.
However, if the error recovery action includes code to find the correct place to start
Chapter 12. lex and yacc Program Information
289
processing again, that code must also change the look-ahead token. To clear the
look-ahead token include the following statement in the error-recovery action:
yyclearin ;
Lexical Analysis for the yacc Command
You must provide a lexical analyzer to read the input stream and send tokens
(with values, if required) to the parser that the yacc command generates. To build
a lexical analyzer that works well with the parser that yacc generates, use the
lexlex command.
The lex command generates a lexical analyzer called yylex. The yylexprogram
must return an integer that represents the kind of token that was read. The integer
is called the token number. In addition, if a value is associated with the token, the
lexical analyzer must assign that value to the yylval external variable.
yacc-Generated Parser Operation
The yacc command turns the grammar file into a C language program. That
program, when compiled and executed, parses the input according to the grammar
specification given.
The parser is a finite state machine with a stack. The parser can read and
remember the look-ahead token. The current state is always the state at the top of
the stack. The states of the finite state machine are represented by small integers.
Initially, the machine is in state 0, the stack contains only 0, and no look-ahead
token has been read.
The machine can perform one of four actions:
shift State
reduce Rule
accept
error
The parser pushes the current state onto the stack, makes State the current
state, and clears the look-ahead token.
When the parser finds a string defined by Rule (a rule number) in the
input stream, the parser replaces that string with Rule in the output
stream.
The parser looks at all input, matches it to the grammar specification, and
recognizes the input as satisfying the highest-level structure (defined by
the start symbol). This action appears only when the look-ahead token is
the endmarker and indicates that the parser has successfully done its job.
The parser cannot continue processing the input stream and still
successfully match it with any rule defined in the grammar specification.
The input tokens the parser looked at, together with the look-ahead token,
cannot be followed by anything that would result in valid input. The
parser reports an error and attempts to recover the situation and resume
parsing.
The parser performs the following actions during one process step:
1. Based on its current state, the parser decides whether it needs a look-ahead
token to decide the action to be taken. If the parser needs a look-ahead token
and does not have one, it calls the yylex subroutine to obtain the next token.
2. Using the current state, and the look-ahead token if needed, the parser decides
on its next action and carries it out. As a result, states may be pushed onto or
popped off the stack, and the look-ahead token may be processed or left alone.
290
General Programming Concepts
Shift
The shift action is the most common action the parser takes. Whenever the parser
does a shift, there is always a look-ahead token. For example, consider the
following grammar specification rule:
IF shift 34
If the parser is in the state that contains this rule and the look-ahead token is IF,
the parser:
1. Pushes the current state down on the stack.
2. Makes state 34 the current state (puts it at the top of the stack).
3. Clears the look-ahead token.
Reduce
The reduce action keeps the stack from growing too large. The parser uses reduce
actions after matching the right side of a rule with the input stream. The parser is
then ready to replace the characters in the input stream with the left side of the
rule. The parser may have to use the look-ahead token to decide if the pattern is a
complete match.
Reduce actions are associated with individual grammar rules. Because grammar
rules also have small integer numbers, you can easily confuse the meanings of the
numbers in the two actions, shift and reduce. For example, the following action
refers to grammar rule 18:
. reduce 18
The following action refers to state 34:
IF shift 34
For example, to reduce the following rule, the parser pops off the top three states
from the stack:
A : x y z ;
The number of states popped equals the number of symbols on the right side of
the rule. These states are the ones put on the stack while recognizing x, y, and z.
After popping these states, a state is uncovered, which is the state the parser was
in before beginning to process the rule, that is, the state that needed to recognize
rule A to satisfy its rule. Using this uncovered state and the symbol on the left side
of the rule, the parser performs an action called goto, which is similar to a shift of
A. A new state is obtained, pushed onto the stack, and parsing continues.
The goto action is different from an ordinary shift of a token. The look-ahead token
is cleared by a shift but is not affected by a goto action. When the three states are
popped, the uncovered state contains an entry such as:
A goto 20
This entry causes state 20 to be pushed onto the stack and become the current
state.
The reduce action is also important in the treatment of user-supplied actions and
values. When a rule is reduced, the parser executes the code that you included in
the rule before adjusting the stack. Another stack running in parallel with the stack
holding the states holds the values returned from the lexical analyzer and the
actions. When a shift takes place, the yylval external variable is copied onto the
stack holding the values. After executing the code that you provide, the parser
Chapter 12. lex and yacc Program Information
291
performs the reduction. When the parser performs the goto action, it copies the
yylval external variable onto the value stack. The yacc keywords that begin with $
refer to the value stack.
Using Ambiguous Rules in the yacc Program
A set of grammar rules is ambiguous if any possible input string can be structured
in two or more different ways. For example, the grammar rule:
expr : expr '-' expr
states a rule that forms an arithmetic expression by putting two other expressions
together with a minus sign between them. Unfortunately, this grammar rule does
not specify how to structure all complex inputs. For example, if the input is:
expr - expr - expr
a program could structure this input as either left associative:
( expr - expr ) - expr
or as right associative:
expr - ( expr - expr )
and produce different results.
Parser Conflicts
When the parser tries to handle an ambiguous rule, confusion occurs over which
of its four actions to perform when processing the input. Two major types of
conflict develop:
shift/reduce conflict
reduce/reduce conflict
A rule can be evaluated correctly using either a shift
action or a reduce action, but the result is different.
A rule can be evaluated correctly using one of two
different reduce actions, producing two different actions.
A shift/shift conflict is not possible. The shift/reduce and reduce/reduce conflicts
result from a rule that is not completely stated. For example, using the ambiguous
rule stated in the previous section, if the parser receives the input:
expr - expr - expr
after reading the first three parts the parser has:
expr - expr
which matches the right side of the preceding grammar rule. The parser can
reduce the input by applying this rule. After applying the rule, the input becomes:
expr
which is the left side of the rule. The parser then reads the final part of the input:
- expr
and reduces it. This produces a left associative interpretation.
However, the parser can also look ahead in the input stream. If, when the parser
receives the first three parts:
expr - expr
292
General Programming Concepts
it reads the input stream until it has the next two parts, it then has the following
input:
expr - expr - expr
Applying the rule to the rightmost three parts reduces them to expr. The parser
then has the expression:
expr - expr
Reducing the expression once more produces a right associative interpretation.
Therefore, at the point where the parser has read only the first three parts, it can
take two valid actions: a shift or a reduce. If the parser has no rule to decide
between the two actions, a shift/reduce conflict results.
A similar situation occurs if the parser can choose between two valid reduce
actions. That situation is called a reduce/reduce conflict.
How the Parser Responds to Conflicts
When shift/reduce or reduce/reduce conflicts occur, the yacc command produces a
parser by selecting one of the valid steps wherever it has a choice. If you do not
provide a rule that makes the choice, yacc uses two rules:
1. In a shift/reduce conflict, choose the shift.
2. In a reduce/reduce conflict, reduce by the grammar rule that can be applied at
the earliest point in the input stream.
Using actions within rules can cause conflicts if the action must be performed
before the parser is sure which rule is being recognized. In such cases, the
preceding rules result in an incorrect parser. For this reason, the yacc program
reports the number of shift/reduce and reduce/reduce conflicts resolved using the
preceding rules.
Turning on Debug Mode for a yacc-Generated Parser
You can access the debugging code either by invoking the yacc command with the
-t option or compiling the y.tab.c file with -DYYDEBUG.
For normal operation, the yydebug external integer variable is set to 0. However, if
you set it to a nonzero value, the parser generates a description of:
v The input tokens it receives.
v The actions it takes for each token while parsing an input stream.
Set this variable in one of the following ways:
v Put the following C language statement in the declarations section of the yacc
grammar file:
int yydebug = 1;
v Use dbx to execute the final parser, and set the variable ON or OFF using dbx
commands.
Example Program for the lex and yacc Programs
This section describes example programs for the lex and yacc commands. Together,
these example programs create a simple, desk-calculator program that performs
addition, subtraction, multiplication, and division operations. This calculator
program also allows you to assign values to variables (each designated by a single,
Chapter 12. lex and yacc Program Information
293
lowercase letter) and then use the variables in calculations. The files that contain
the example lex and yacc programs are:
File
“Lexical
Analyzer
Source Code”
on page 297
“Parser
Source Code”
on page 295
Content
Specifies the lex command specification file that defines the lexical analysis
rules.
Specifies the yacc command grammar file that defines the parsing rules, and
calls the yylex subroutine created by the lex command to provide input.
The following descriptions assume that the calc.lex and calc.yacc example
programs are found in your current directory.
Compiling the Example Program
Perform the following steps, in order, to create the desk calculator example
program:
1. Process the yacc grammar file using the -d optional flag (which tells the yacc
command to create a file that defines the tokens used in addition to the C
language source code):
yacc -d calc.yacc
2. Use the li command to verify that the following files were created:
y.tab.c
y.tab.h
The C language source file that the yacc command created for the parser.
A header file containing define statements for the tokens used by the parser.
3. Process the lex specification file:
lex calc.lex
4. Use the li command to verify that the following file was created:
lex.yy.c
The C language source file that the lex command created for the lexical
analyzer.
5. Compile and link the two C language source files:
cc y.tab.c lex.yy.c
6. Use the li command to verify that the following files were created:
y.tab.o
lex.yy.o
a.out
The object file for the y.tab.c source file
The object file for the lex.yy.c source file
The executable program file
To then run the program directly from the a.out file, enter:
$ a.out
Or, to move the program to a file with a more descriptive name, as in the
following example, and run it, enter:
$ mv a.out calculate
$ calculate
In either case, after you start the program, the cursor moves to the line below
the $ (command prompt). Then enter numbers and operators in calculator
fashion. When you press the Enter key, the program displays the result of the
operation. After you assign a value to a variable:
m=4 <enter>
_
294
General Programming Concepts
the cursor moves to the next line. When you use the variable in subsequent
calculations, it will have the assigned value:
m+5 <enter>
9
_
Parser Source Code
The following example shows the contents of the calc.yacc file. This file has entries
in all three sections of a yacc grammar file: declarations, rules, and programs.
%{
#include <stdio.h>
int regs[26];
int base;
%}
%start list
%token DIGIT LETTER
%left
%left
%left
%left
%left
'|'
'&'
'+' '-'
'*' '/' '%'
UMINUS /*supplies precedence for unary minus */
%%
/* beginning of rules section */
list:
/*empty */
|
list stat '\n'
|
list error '\n'
{
yyerrok;
}
;
stat:
expr
{
printf("%d\n",$1);
}
|
LETTER '=' expr
{
regs[$1] = $3;
}
;
expr:
'(' expr ')'
{
$$ = $2;
}
|
expr
{
$$
}
|
expr
{
$$
}
|
expr
{
$$
}
|
expr
'*' expr
= $1 * $3;
'/' expr
= $1 / $3;
'%' expr
= $1 % $3;
'+' expr
Chapter 12. lex and yacc Program Information
295
{
$$ = $1 + $3;
}
|
expr '-' expr
{
$$
}
|
expr
{
$$
}
|
expr
{
$$
}
|
= $1 - $3;
'&' expr
= $1 & $3;
'|' expr
= $1 | $3;
'-' expr %prec UMINUS
{
$$ = -$2;
}
|
LETTER
{
$$ = regs[$1];
}
|
number
;
number: DIGIT
{
$$ = $1;
base = ($1==0) ? 8 : 10;
}
|
number DIGIT
{
$$ = base * $1 + $2;
}
;
%%
main()
{
return(yyparse());
}
yyerror(s)
char *s;
{
fprintf(stderr, "%s\n",s);
}
yywrap()
{
return(1);
}
Declarations Section: This section contains entries that:
v Include standard I/O header file.
v Define global variables.
v Define the list rule as the place to start processing.
v Define the tokens used by the parser.
v Define the operators and their precedence.
296
General Programming Concepts
Rules Section: The rules section defines the rules that parse the input stream.
Programs Section: The programs section contains the following subroutines.
Because these subroutines are included in this file, you do not need to use the yacc
library when processing this file.
main
yyerror(s)
yywrap
The required main program that calls the yyparse subroutine to start the
program.
This error-handling subroutine only prints a syntax error message.
The wrap-up subroutine that returns a value of 1 when the end of input
occurs.
Lexical Analyzer Source Code
Following are the contents of the calc.lex file. This file contains include statements
for standard input and output, as well as for the y.tab.h file. The yacc program
generates that file from the yacc grammar file information if you use the -d flag
with the yacc command. The y.tab.h file contains definitions for the tokens that the
parser program uses. In addition, calc.lex contains the rules to generate these
tokens from the input stream.
%{
#include <stdio.h>
#include "y.tab.h"
int c;
extern int yylval;
%}
%%
" "
;
[a-z]
{
c = yytext[0];
yylval = c - 'a';
return(LETTER);
}
[0-9]
{
c = yytext[0];
yylval = c - '0';
return(DIGIT);
}
[|a-z0-9\b]
{
c = yytext[0];
return(c);
}
Chapter 12. lex and yacc Program Information
297
298
General Programming Concepts
Chapter 13. Logical Volume Programming
This chapter provides an introduction to programming considerations for the
Logical Volume Manager (LVM), which consists of the library of LVM subroutines
and the logical volume device driver.
The Logical Volume Manager (LVM) consists of two main components.
The first is the library of LVM subroutines. These subroutines define volume
groups and maintain the logical and physical volumes of volume groups. They are
used by the system management commands to perform system management tasks
for the logical and physical volumes of a system. The programming interface for
the library of LVM subroutines is available to anyone who wishes to provide
alternatives or to expand the function of the system management commands for
logical volumes.
The other main component of LVM is the logical volume device driver. The logical
volume device driver is a pseudo-device driver that processes all logical I/O. It
exists as a layer between the file system and the disk device drivers. The logical
volume device driver converts a logical address to a physical address, handles
mirroring and bad-block relocation, and then sends the I/O request to the specific
disk device driver. Interfaces to the logical volume device driver are provided by
the open, close, read, write, and ioctl subroutines.
A description of the readx and writex extension parameters and those ioctl
operations specific to the logical volume device driver is found in Understanding
the Logical Volume Device Driver in AIX Kernel Extensions and Device Support
Programming Concepts.
See the Logical Volume Storage Overview in AIX Version 4.3 System Management
Guide: Operating System and Devices for more information on logical volumes.
Logical Volume Program Example
/*
This program is designed to demonstrate the use of Logical
*
Volume Manager subroutines.
*/
/*
The program objectives are:
*
- to create a volume group with two logical volumes and
*
two physical volumes
*
- to then delete the physical volumes, logical volumes
*
and volume group that were previously created.
*/
/*
The following subroutines are demonstrated:
*
- lvm_createvg
*
- lvm_createlv
*
- lvm_installpv
*
- lvm_extendlv
*
- lvm_reducelv
*
- lvm_deletelv
*
- lvm_deletepv
*
- lvm_varyonvg
*/
#include <lvm.h>
#define SUCCESS 0
/* Successful return code. */
#define VGNAME "rvg34"
/* Name of volume group, must be raw device. */
© Copyright IBM Corp. 1997, 1999
299
#define VGMAJOR 34
/* Major number for this volume group the major number
can be any unused major number in the system.*/
#define PVNAME1 "rhdisk1"
/* Name of physical volume, should define a raw device. */
#define PVNAME2 "rhdisk2"
/* Name of physical volume, should define a raw device. */
#define MAX_LVS 32
/* Max number of logical volumes desired in
the volume group, can be no more that 256. */
#define PPSIZE 22
/* Size of physical partitions in volume group,
2|22 or 4Meg. Possible values range from
LVM_MINPPSIZ and LVM_MAXPPSIZ both found in lvm.h. */
#define VGDASIZE 32
/* Size of volume group descriptor area.
Possible values are in the range of
LVM_MINVGDASIZ and LVM_MAXVGDASIZ
found in lvm.h. */
#define PVID1_W1 0x00000002
/* Physical volume id word one for pv1. */
#define PVID1_W2 0x00004321
/* Physical volume id word two for pv1. */
#define PVID2_W1 0x00000003
/* Physical volume id word one for pv2. */
#define PVID2_W2 0x00004322
/* Physical volume id word two for pv2. */
#define LVNAME1 "rhd34.1"
/* Logical volume name, must be raw device. */
#define LVNAME2 "rhd34.2"
/* Logical volume name, must be raw device. */
#define MAXSIZE 128
/* Max number of logical partitions on the
logical volumes in the volume group. */
#define MINOR1 0x1
/* Minor number for first logical volume. */
#define MINOR2 0x2
/* Minor number for second logical volume. */
#define LV1_PPS 2
/* Number of physical partitions being used to
extend the first logical volume created. */
#define LV2_PPS 4
/* Number of physical partitions being used to
extend the second logical volume created. */
struct createvg cvg;
/* Createvg structure */
struct createlv clv;
/* Createlv structure */
struct lv_id
lid;
/* LV ID structure */
struct ext_redlv e_r; /* Extend/Reduce LV structure */
struct changelv chlv; /* Change LV structure*/
struct varyonvg von;
/* Varyonvg structure*/
struct varyoffvg voff; /* Varyoffvg structure*/
struct changepv cpv;
/* Change PV structure*/
struct pp
pps1[LV1_PPS];
/* Structures for physical partition. */
struct pp
pps2[LV2_PPS];
/* Maps to be passed into extendlv and reducelv. */
struct unique_id pv1 = {PVID1_W1, PVID1_W2};
/* ID of first physical volume. */
struct unique_id pv2 = {PVID2_W1, PVID2_W2};
/* ID of second physical volume. */
int
sample()
{
int
rc;
/* Return code. */
/*
* Call the procedure to create a volume group with two
* logical volumes and two physical volumes.
300
General Programming Concepts
*/
rc = build();
if (rc < SUCCESS)
return(rc);
/*
* Call the procedure to delete everything created
* in the build routine
*/
rc = destroy();
if (rc < SUCCESS)
return(rc);
return(SUCCESS);
}
int
build()
{
int
rc;
/* Return code. */
/*
* Fill in the fields of the createvg structure to be
* passed into the lvm_createvg subroutine.
*/
cvg.kmid
= KMID;
cvg.vgname
= VGNAME;
cvg.vg_major
= VGMAJOR;
cvg.pvname
= PVNAME1;
cvg.maxlvs
= MAX_LVS;
cvg.ppsize
= PPSIZE;
cvg.vgda_size
= VGDASIZE;
/*
* Call lvm_createvg() to create a volume group.
*/
rc = lvm_createvg(&cvg);
if(rc < SUCCESS)
return(rc);
/*
* Fill in the fields of the varyonvg structure to be
* passed into lvm_varyonvg().
*/
von.kmid
= KMID;
von.vgname
= VGNAME;
von.vg_major
= VGMAJOR;
von.vg_id.word1
= cvg.vg_id.word1;
von.vg_id.word2
= cvg.vg_id.word2;
von.noopen_lvs
= FALSE;
von.auto_resync
= TRUE,
von.misspv_von
= TRUE;
von.missname_von
= TRUE;
von.vvg_in.pv[0].pv_id = pv1;
von.vvg_in.pv[0].pvname = PVNAME1;
von.vvg_in.num_pvs
= 1;
/*
* Call lvm_varyonvg() to vary the newly created volume
* group on-line.
*/
rc = lvm_varyonvg(&von);
if(rc < SUCCESS)
return(rc);
/*
* Install a second physical volume Into the volume group.
*/
rc = lvm_installpv(PVNAME2, &(cvg.vg_id));
if(rc < SUCCESS)
return(rc);
/*
* Fill in the fields of the createlv structure to be
* passed into lvm_createlv().
Chapter 13. Logical Volume Programming
301
*/
clv.vg_id.word1
clv.vg_id.word2
clv.lvname
clv.maxsize
clv.minor_num
clv.mirror_policy
clv.permissions
clv.bb_relocation
clv.write_verify
clv.mirwrt_consist
/*
= cvg.vg_id.word1;
= cvg.vg_id.word2;
= LVNAME1;
= MAXSIZE;
= MINOR1;
= LVM_PARALLEL;
= LVM_RDWR;
= LVM_RELOC;
= LVM_NOVERIFY;
= LVM_CONSIST;
* Call lvm_createlv() to create a logical volume.
*/
rc = lvm_createlv(&clv);
if(rc < SUCCESS)
return(rc);
/*
* Fill in the pp structs that will map physical partitions
* to logical partitions. There should be one entry for each
* physical partition being added.
* NOTE that all physical partitions for this logical volume
* are on the same (one) physical volume (hardfile)
*/
pps1[0].pv_id
= pv1;
pps1[0].lp_num = 1;
pps1[0].pp_num = 1;
pps1[1].pv_id
= pv1;
pps1[1].lp_num = 2;
pps1[1].pp_num = 2;
e_r.parts
= pps1;
/*
* e_r.size is two because there are two physical
* partitions being added.
*/
e_r.size = 2;
/*
* Set up the id of the logical volume being extended.
*/
lid.minor_num
= MINOR1;
lid.vg_id.word1 = cvg.vg_id.word1;
lid.vg_id.word2 = cvg.vg_id.word2;
/*
* Call lvm_extendlv() to extend the logical volume so that
* it has two logical partitions.
*/
rc = lvm_extendlv(&lid, &e_r);
if(rc < SUCCESS)
return(rc);
/*
* Fill in the fields of the createlv structure to be passed
* into lvm_createlv().
*/
clv.vg_id.word1
= cvg.vg_id.word1;
clv.vg_id.word2
= cvg.vg_id.word2;
clv.lvname
= LVNAME2;
clv.maxsize
= MAXSIZE;
clv.minor_num
= MINOR2;
clv.mirror_policy
= LVM_PARALLEL; /* from <lvm.h> */
clv.permissions = LVM_RDWR;
/* from <lvm.h> */
clv.bb_relocation
= LVM_RELOC;
/* from <lvm.h> */
clv.write_verify
= LVM_NOVERIFY; /* from <lvm.h> */
clv.mirwrt_consist
= LVM_CONSIST; /* from <lvm.h> */
/*
* Call lvm_createlv() to create a second logical volume.
302
General Programming Concepts
*/
rc = lvm_createlv(&clv);
if(rc < SUCCESS)
return(rc);
/*
* Fill in the pp structs that will map physical partitions
* to logical partitions. There should be one entry for each
* physical partition being added. NOTE that the physical
* partitions on this logical volume are spread across two
* physical volumes or hardfiles. Also, this logical volume
* is mirrored; it has two copies of each logical partition.
*/
pps2[0].pv_id = pv1;
/* First physical volume. */
pps2[0].lp_num = 1;
pps2[0].pp_num = 3;
pps2[1].pv_id = pv1;
pps2[1].lp_num = 2;
pps2[1].pp_num = 7;
pps2[2].pv_id = pv2;
/* Second physical volume. */
pps2[2].lp_num = 1;
pps2[2].pp_num = 5;
pps2[3].pv_id = pv2;
pps2[3].lp_num = 2;
pps2[3].pp_num = 9;
e_r.parts
= pps2;
/*
* e_r.size is four because there are four physical
* partitions being extended.
*/
e_r.size = 4;
/*
* Set up the id of the logical volume being extended.
*/
lid.minor_num
= MINOR2;
lid.vg_id.word1 = cvg.vg_id.word1;
lid.vg_id.word2 = cvg.vg_id.word2;
/*
* Call lvm_extendlv() to extend the logical volume.
*/
rc = lvm_extendlv(&lid, &e_r);
if(rc < SUCCESS)
return(rc);
return SUCCESS;
}
int
destroy()
{
int
rc;
/* Return code. */
/*
* Fill in the physical partition structures for the
* physical partitions to be deleted, or reduced.
*/
pps2[0].pv_id = pv1;
pps2[0].lp_num = 1;
pps2[0].pp_num = 3;
pps2[1].pv_id = pv1;
pps2[1].lp_num = 2;
pps2[1].pp_num = 7;
pps2[2].pv_id = pv2;
pps2[2].lp_num = 1;
pps2[2].pp_num = 5;
pps2[3].pv_id = pv2;
pps2[3].lp_num = 2;
pps2[3].pp_num = 9;
e_r.parts = pps2;
/*
* The size field should correspond to the number of
Chapter 13. Logical Volume Programming
303
* physical partitions being reduced.
*/
e_r.size = 4;
/* Fill in the id of the logical volume to be reduced. */
lid.minor_num
= MINOR2;
lid.vg_id.word1 = cvg.vg_id.word1;
lid.vg_id.word2 = cvg.vg_id.word2;
/*
* Call lvm_reducelv() to reduce the logical volume.
*/
rc = lvm_reducelv(&lid, &e_r);
if(rc < SUCCESS)
return(rc);
/*
* Fill in the id of the logical volume to be deleted and
* call lvm_deletelv() to delete the logical volume.
*/
lid.minor_num
= MINOR2;
lid.vg_id.word1 = cvg.vg_id.word1;
lid.vg_id.word2 = cvg.vg_id.word2;
rc = lvm_deletelv(&lid);
if(rc < SUCCESS)
return(rc);
/*
* Fill in the physical partition structure for the other
* logical volume that is being reduced.
*/
pps1[0].pv_id
= pv1;
pps1[0].lp_num = 1;
pps1[0].pp_num = 1;
pps1[1].pv_id
= pv1;
pps1[1].lp_num = 2;
pps1[1].pp_num = 2;
e_r.parts
= pps1;
/*
* The size field should correspond to the number of
* physical partitions being reduced, in this case two.
*/
e_r.size = 2;
lid.minor_num
= MINOR1;
lid.vg_id.word1 = cvg.vg_id.word1;
lid.vg_id.word2 = cvg.vg_id.word2;
/*
* Call lvm_reducelv() to reduce the logical volume.
*/
rc = lvm_reducelv(&lid, &e_r);
if(rc < SUCCESS)
return(rc);
/*
* Fill in the id for the logical volume to be
* deleted and call lvm_deletelv() to delete it from
* the volume group.
*/
lid.minor_num
= MINOR1;
lid.vg_id.word1 = cvg.vg_id.word1;
lid.vg_id.word2 = cvg.vg_id.word2;
rc = lvm_deletelv(&lid);
if(rc < SUCCESS)
return(rc);
/*
* Call lvm_deletepv() to delete a physical volume
* from the volume group.
*/
rc = lvm_deletepv(&(cvg.vg_id), &pv2);
if(rc < SUCCESS)
return(rc);
304
General Programming Concepts
/*
* Call lvm_deletepv() to delete the last
* physical volume from the volume group. Because we are
* deleting the last physical volume in the volume group,
* the volume group itself will also be deleted.
*/
rc = lvm_deletepv(&(cvg.vg_id), &pv1);
if(rc < SUCCESS)
return(rc);
return SUCCESS;
}
List of Logical Volume Subroutines
The library of LVM subroutines is a main component of the Logical Volume
Manager.
LVM subroutines define and maintain the logical and physical volumes of a
volume group. They are used by the system management commands to perform
system management for the logical and physical volumes of a system. The
programming interface for the library of LVM subroutines is available to anyone
who wishes to provide alternatives to or expand the function of the system
management commands for logical volumes.
Note: The LVM subroutines use the sysconfig system call, which requires
root user authority, to query and update kernel data structures describing a
volume group. You must have root user authority to use the services of the
LVM subroutine library.
The following services are available:
lvm_changelv
lvm_changepv
lvm_createlv
lvm_createvg
lvm_deletelv
lvm_deletepv
lvm_extendlv
lvm_installpv
lvm_migratepp
lvm_querylv
lvm_querypv
lvm_queryvg
lvm_queryvgs
lvm_reducelv
lvm_resynclp
lvm_resynclv
lvm_resyncpv
lvm_varyoffvg
lvm_varyonvg
Changes the attributes of a logical volume.
Changes the attributes of a physical volume in a volume group.
Creates an empty logical volume in a specified volume group.
Creates a new volume group and installs the first physical volume.
Deletes a logical volume from its volume group.
Deletes a physical volume from a volume group.
Extends a logical volume by a specified number of partitions.
Installs a physical volume into a volume group.
Moves a physical partition to a specified physical volume.
Queries a logical volume and returns all pertinent information.
Queries a physical volume and returns all pertinent information.
Queries a volume group and returns pertinent information.
Queries the volume groups of the system and returns information for
groups that are varied on-line.
Reduces a logical volume by a specified number of partitions.
Synchronizes all physical partition copies of a logical partition.
Synchronizes all physical copies of all the logical partitions for a logical
volume.
Synchronizes all physical partitions on a physical volume with the
related copies of the logical partition to which they correspond.
Varies a volume group off-line.
Varies a volume group on-line.
Chapter 13. Logical Volume Programming
305
306
General Programming Concepts
Chapter 14. make Command Overview
This chapter provides information about simplifying the recompiling and relinking
processes using the make command. It is a useful utility that can save you time
when managing projects.
The make program is most useful for medium-sized programming projects. It does
not solve the problems of maintaining more than one source version and of
describing large programs (see sccs command).
The make command assists you in maintaining a set of programs, usually
pertaining to a particular software project. It does this by building up-to-date
versions of programs.
In any project, you normally link programs from object files and libraries. Then,
after modifying a source file, you recompile some of the sources and relink the
program as often as required.
The make command simplifies the process of recompiling and relinking programs.
It allows you to record, once only, specific relationships among files. You can then
use the make command to automatically perform all updating tasks.
Using the make command to maintain programs, you can:
v Combine instructions for creating a large program in a single file.
v Define macros to use within the make command description file.
v Use shell commands to define the method of file creation, or use the make
program to create many of the basic types of files.
v Create libraries.
The make command requires a description file, file names, specified rules to tell
the make program how to build many standard types of files, and time stamps of
all system files.
Creating a Description File
The make program uses information from a description file that you create to
build a file containing the completed program, which is then called a target file.
The description file tells the make command how to build the target file, which
files are involved, and what their relationships are to the other files in the
procedure. The description file contains the following information:
v Target file name
v Parent file names that make up the target file
v Commands that create the target file from the parent files
v Definitions of macros in the description file
v User-specified rules for building target files
By checking the dates of the parent files, the make program determines which files
to create to get an up-to-date copy of the target file. If any parent file was changed
more recently than the target file, the make command creates the files affected by
the change, including the target file.
© Copyright IBM Corp. 1997, 1999
307
If you name the description file makefile or Makefile and are working in the
directory containing that description file, enter:
make
to update the first target file and its parent files. Updating occurs regardless of the
number of files changed since the last time the make command created the target
file. In most cases, the description file is easy to write and does not change often.
To keep many different description files in the same directory, name them
differently. Then, enter:
make -f Desc-File
substituting the name of the description file for the Desc-File variable.
Format of a make Description File Entry
The general form of an entry is:
target1 [target2..]:[:] [parent1..][; command]...
[(tab) commands]
Items inside brackets are optional. Targets and parents are file names (strings of
letters, numbers, periods, and slashes). The make command recognizes wildcard
characters such as * (asterisk) and ? (question mark). Each line in the description
file that contains a target file name is called a dependency line. Lines that contain
commands must begin with a tab character.
Note: The make command uses the $ (dollar sign) to designate a macro. Do
not use that symbol in file names of target or parent files, or in commands in
the description file unless you are using a predefined make command macro.
Begin comments in the description file with a # (pound sign). The make program
ignores the # and all characters that follow it. The make program also ignores
blank lines.
Except for comment lines, you can enter lines longer than the line width of the
input device. To continue a line on the next line, put a \ (backslash) at the end of
the line to be continued.
Using Commands in a make Description File
A command is any string of characters except a # (pound sign) or a new-line
character. A command can use a # only if it is in quotes. Commands can appear
either after a semicolon on a dependency line or on lines beginning with a tab that
immediately follows a dependency line.
When defining the command sequence for a particular target, specify one
command sequence for each target in the description file, or else separate
command sequences for special sets of dependencies. Do not do both.
To use one command sequence for every use of the target file, use a single : (colon)
following the target name on the dependency line. For example:
test:
test:
308
dependency list1...
command list...
.
.
.
dependency list2...
General Programming Concepts
defines a target name, test, with a set of parent files and a set of commands to
create the file. The target name, test, can appear in other places in the description
file with another dependency list.
However, that name cannot have another command list in the description file.
When one of the files that test depends on changes, the make command runs the
commands in that one command list to create the target file named test.
To specify more than one set of commands to create a particular target file, enter
more than one dependency definition. Each dependency line must have the target
name, followed by :: (two colons), a dependency list, and a command list that the
make command uses if any of the files in the dependency list changes. For
example:
test::
dependency list1...
command list1...
test::
dependency list2...
command list2...
defines two separate processes to create the target file, test. If any of the files in
dependency list1 changes, the make command runs command list1. If any of the
files in dependency list2 changes, the make command runs command list2. To
avoid conflicts, a parent file cannot appear in both dependency list1 and
dependency list2.
Note: The make command passes the commands from each command line to
a new shell. Be careful when using commands that have meaning only within
a single shell process; for example, cd and shell commands. The make
program forgets these results before running the commands on the next line.
To group commands together, use the \ (backslash) at the end of a command line.
The make program then continues that command line into the next line in the
description file. The shell sends both of these lines to a single new shell.
Calling the make Program from a Description File
To nest calls to the make program within a make command description file,
include the $(MAKE) macro in one of the command lines in the file. If this macro
is present, the make command calls another copy of the make command even if
the -n flag, which stops execution, is set (if .POSIX is not specified). The make
program then passes its current flags to the new copy of the make command. This
is true only in AIX Version 3.2.5. For versions 4.1.5 and beyond, the nested make
does not inherit the -n flag.
If the -n flag is set when the $(MAKE) macro is found, the new copy of the make
command does not execute any of its commands, except another $(MAKE) macro.
Use this characteristic to test a set of description files that describe a program.
Enter the command:
make -n
The make program does not perform any of the program operations. However, it
does write all of the steps needed to build the program, including output from
lower-level calls to the make command.
Preventing the make Program from Writing Commands
To prevent the make program from writing the commands while it runs, do any of
the following:
Chapter 14. make Command Overview
309
v Use the -s flag on the command line when using the make command.
v Put the fake target name .SILENT on a dependency line by itself in the
description file. Because .SILENT is not a real target file, it is called a fake
target. If .SILENT has prerequisites, the make command does not display any of
the commands associated with them.
v Put an @ (at sign) in the first character position of each line in the description
file that the make command should not write.
Preventing the make Program from Stopping on Errors
The make program normally stops if any program returns a nonzero error code.
Some programs return status that has no meaning.
To prevent the make command from stopping on errors, do any of the following:
v Use the -i flag with the make command on the command line.
v Put the fake target name .IGNORE on a dependency line by itself in the
description file. Because .IGNORE is not a real target file, it is called a fake
target. If .IGNORE has prerequisites, the make command ignores errors
associated with them.
v Put a - (minus sign) in the first character position of each line in the description
file where the make command should not stop on errors.
Example of a Description File
For example, a program named prog is made by compiling and linking three C
language files x.c, y.c, and z.c. The x.c and y.c files share some declarations in a
file named defs. The z.c file does not share those declarations. The following is an
example of a description file, which creates the prog program:
# Make prog from 3 object files
prog: x.o y.o z.o
# Use the cc program to make prog
cc x.o y.o z.o -o prog
# Make x.o from 2 other files
x.o: x.c defs
# Use the cc program to make x.o
cc -c x.c
# Make y.o from 2 other files
y.o: y.c defs
# Use the cc program to make y.o
cc -c y.c
# Make z.o from z.c
z.o: z.c
# Use the cc program to make z.o
cc -c z.c
If this file is called makefile, just enter the command:
make
to update the prog program after making changes to any of the four source files:
x.c, y.c, z.c, or defs.
Making the Description File Simpler
To make this file simpler, use the internal rules of the make program. Based on
file-system naming conventions, the make command recognizes three .c files
corresponding to the needed .o files. This command can also generate an object
from a source file, by issuing a cc -c command.
310
General Programming Concepts
Based on these internal rules, the description file becomes:
# Make prog from 3 object files
prog: x.o y.o z.o
# Use the cc program to make prog
cc x.o y.o z.o -o prog
# Use the file defs and the .c file
# when making x.o and y.o
x.o y.o: defs
Internal Rules for the make Program
The internal rules for the make program are in a file that looks like a description
file. When the -r flag is specified, the make program does not use the internal rules
file. You must supply the rules to create the files in your description file. The
internal-rules file contains a list of file-name suffixes (such as .o, or .a) that the
make command understands, plus rules that tell the make command how to create
a file with one suffix from a file with another suffix. If you do not change the list,
the make command understands the following suffixes:
.a
.C
.C\x
.c
.cx
.f
.fx
.h
.hx
.l
.lx
.o
.s
.sx
.sh
.shx
.y
.yx
Archive library.
C++ source file.
SCCS file containing C++ source file.
C source file.
Source Code Control System (SCCS) file containing C source file.
FORTRAN source file.
SCCS file containing FORTRAN source file.
C language header file.
SCCS file containing C language header file.
lex source grammar.
SCCS file containing lex source grammar.
Object file.
Assembler source file.
SCCS file containing assembler source file.
Shell-command source file.
SCCS file containing shell-command source file.
yacc-c source grammar.
SCCS file containing yacc-c source grammar.
The list of suffixes is similar to a dependency list in a description file, and follows
the fake target name .SUFFIXES. Because the make command looks at the suffixes
list in left-to-right order, the order of the entries is important.
The make program uses the first entry in the list that satisfies the following two
requirements:
v The entry matches input and output suffix requirements for the current target
and dependency files.
v The entry has a rule assigned to it.
The make program creates the name of the rule from the two suffixes of the files
that the rule defines. For example, the name of the rule to transform a .c file to an
.o file is .c.o.
To add more suffixes to the list, add an entry for the fake target name .SUFFIXES
in the description file. For a .SUFFIXES line without any suffixes following the
Chapter 14. make Command Overview
311
target name in the description file, the make command erases the current list. To
change the order of the names in the list, erase the current list and then assign a
new set of values to .SUFFIXES.
Example of Default Rules File
The following example shows a portion of the default rules file:
# Define suffixes that make knows.
.SUFFIXES: .o .C .C\x .c .cx .f .fx .y .yx .l .lx .s .sx .sh .shx .h .hx .a
#Begin macro definitions for
#internal macros
YACC=yacc
YFLAGS=
ASFLAGS=
LEX=lex
LFLAGS=
CC=cc
CCC=xlC
AS=as
CFLAGS=
CCFLAGS=
# End macro definitions for
# internal macros
# Create a .o file from a .c
# file with the cc program.
c.o:
$(CC) $(CFLAGS) -c $<
# Create a .o file from
# a .s file with the assembler.
s.o:
$(AS)$(ASFLAGS) -o $@ $<
.y.o:
# Use yacc to create an intermediate file
$(YACC) $(YFLAGS) $<
# Use cc compiler
$(CC) $(CFLAGS) -c y.tab.c
# Erase the intermediate file
rm y.tab.c
# Move to target file
mv y.tab.o $@.
.y.c:
# Use yacc to create an intermediate file
$(YACC) $(YFLAGS) $<
# Move to target file
mv y.tab.c $@
Single-Suffix Rules
The make program has a set of single-suffix rules to build source files directly into
a target file name that does not have a suffix (command files, for example). The
make program also has rules to change the following source files with suffix to
object files without a suffix:
.C:
.C\x:
.c:
.cx:
.sh:
.shx:
From
From
From
From
From
From
a C++ language source file.
an SCCS C++ language source file.
a C language source file.
an SCCS C language source file.
a shell file.
an SCCS shell file.
For example, to maintain the cat program, enter:
make cat
312
General Programming Concepts
if all of the needed source files are in the current directory.
Using the Make Command with Archive Libraries
Use the make command to build libraries and library files. The make program
recognizes the suffix .a as a library file. The internal rules for changing source files
to library files are:
.C.a
.C\x.a
.c.a
.cx.a
.sx.a
.f.a
.fx.a
C++ source to archive.
SCCS C++ source to archive.
C source to archive.
SCCS C source to archive.
SCCS assembler source to archive.
Fortran source to archive.
SCCS Fortran source to archive.
Changing Macros in the Rules File
The make program uses macro definitions in the rules file. To change these macro
definitions, enter new definitions for each macro on the command line or in the
description file. The make program uses the following macro names to represent
the language processors that it uses:
AS
CC
CCC
YACC
LEX
For
For
For
For
For
the
the
the
the
the
assembler.
C compiler.
C++ compiler.
yacc command.
lex command.
The make program uses the following macro names to represent the flags that it
uses:
CFLAGS
CCFLAGS
YFLAGS
LFLAGS
For
For
For
For
C compiler flags.
C++ compiler flags.
yacc command flags.
lex command flags.
Therefore, the command:
make "CC=NEWCC"
directs the make command to use the NEWCC program in place of the usual C
language compiler. Similarly, the command:
make "CFLAGS=-O"
directs the make command to optimize the final object code produced by the C
language compiler.
To review the internal rules that the make command uses, refer to the
/usr/ccs/lib/make.cfg file.
Defining Default Conditions in a Description File
When the make command creates a target file but cannot find commands in the
description file or internal rules to create a file, it looks at the description file for
default conditions. To define the commands that the make command performs in
this case, use the .DEFAULT target name in the description file:
Chapter 14. make Command Overview
313
.DEFAULT:
command
command
.
.
.
Because .DEFAULT is not a real target file, it is called a fake target. Use the
.DEFAULT fake target name for an error-recovery routine or for a general
procedure to create all files in the program that are not defined by an internal rule
of the make program.
Including Other Files in a Description File
Include files other than the current description file by using the word include as
the first word on any line in the description file. Follow the word with a blank or a
tab, and then the file name for the make command to include in the operation.
Note: Only one file is supported per include statement.
For example:
include /home/tom/temp
include /home/tom/sample
directs the make command to read the temp and sample files and the current
description file to build the target file.
Do not use more than 16 levels of nesting with the include files feature.
Defining and Using Macros in a Description File
A macro is a name (or label) to use in place of several other names. It is a way of
writing the longer string of characters in shorthand. To define a macro:
1. Start a new line with the name of the macro.
2. Follow the name with an = (equal sign).
3. To the right of the = (equal sign), enter the string of characters that the macro
name represents.
The macro definition can contain blanks before and after the = (equal sign) without
affecting the result. The macro definition cannot contain a : (colon) or a tab before
the = (equal sign).
The following are examples of macro definitions:
# Macro -"2" has a value of "xyz"
2 = xyz
# Macro "abc" has a value of "-ll -ly"
abc = -ll -ly
# Macro "LIBES" has a null value
LIBES =
A macro that is named, but not defined, has the same value as the null string.
Using Macros in a Description File
After defining a macro in a description file, use the macro in description file
commands by putting a $ (dollar sign) before the name of the macro. If the macro
314
General Programming Concepts
name is longer than one character, put ( ) (parentheses) or { } (braces) around it.
The following are examples of using macros:
$(CFLAGS)
$2
$(xy)
$Z
$(Z)
The last two examples in the previous list have the same effect.
The following fragment shows how to define and use some macros:
# OBJECTS is the 3 files x.o, y.o and
# z.o (previously compiled)
OBJECTS = x.o y.o z.o
# LIBES is the standard library
LIBES = -lc
# prog depends on x.o y.o and z.o
prog: $(OBJECTS)
# Link and load the 3 files with
# the standard library to make prog
cc $(OBJECTS) $(LIBES) -o prog
The make program using this description file links and loads the three object files
(x.o, y.o, and z.o) with the libc.a library.
A macro definition entered on the command line overrides any duplicate macro
definitions in the description file. Therefore, the command:
make "LIBES= -ll"
loads the files with the lex (-11) library.
Note: When entering macros with blanks in them on the command line, put ″
″ (double quotation marks) around the macro. Without the double quotation
marks, the shell interprets the blanks as parameter separators and not as part
of the macro.
The make command handles up to 10 levels of nested macro expansion. Based on
the definitions in the following example:
macro1=value1
macro2=macro1
the expression $($(macro2)) would evaluate to value1.
The evaluation of a macro occurs each time the macro is referenced. It is not
evaluated when it is defined. If a macro is defined but never used, it will never be
evaluated. This is especially important if the macro is assigned values that will be
interpreted by the shell, particularly if the value might change. A variable
declaration such as:
OBJS = 'ls *.o'
could change in value if referenced at different times during the process of
building or removing object files. It does not hold the value of the ls command at
the time the OBJS macro is defined.
Internal Macros
The make program has built-in macro definitions for use in the description file.
These macros help specify variables in the description file. The make program
Chapter 14. make Command Overview
315
replaces the macros with one of the following values:
$@
$$@
$?
$<
$*
$%
Name of the current target file.
Label name on the dependency line.
Names of the files that have changed more recently than the target.
Parent file name of the out-of-date file that caused a target file to be created.
Name of the current parent file without the suffix.
Name of an archive library member.
Target File Name
If the $@ macro is in the command sequence in the description file, the make
command replaces the symbol with the full name of the current target file before
passing the command to the shell to be run. The make program replaces the
symbol only when it runs commands from the description file to create the target
file.
Label Name
If the $$@ macro is on the dependency line in a description file, the make
command replaces this symbol with the label name that is on the left side of the
colon in the dependency line. For example, if the following is included on a
dependency line:
cat:
[email protected]
the make program translates it to:
cat:
cat.c
when the make command evaluates the expression. Use this macro to build a
group of files, each of which has only one source file. For example, to maintain a
directory of system commands, use a description file like:
# Define macro CMDS as a series
# of command names
CMDS = cat dd echo date cc cmp comm ar ld chown
# Each command depends on a .c file
$(CMDS):
[email protected]
# Create the new command set by compiling the out of
# date files ($?) to the target file name ($@)
$(CC) -O $? -o $@
The make program changes the $$(@F) macro to the file part of $@ when it runs.
For example, use this symbol when maintaining the usr/include directory while
using a description file in another directory. That description file is similar to the
following:
# Define directory name macro INCDIR
INCDIR = /usr/include
# Define a group of files in the directory
# with the macro name INCLUDES
INCLUDES = \
$(INCDIR)/stdio.h \
$(INCDIR)/pwd.h \
$(INCDIR)/dir.h \
$(INCDIR)/a.out.h \
# Each file in the list depends on a file
# of the same name in the current directory
$(INCLUDES):
$$(@F)
# Copy the younger files from the current
# directory to /usr/include
cp $? $@
# Set the target files to read only status
chmod 0444 $@
316
General Programming Concepts
This description file creates a file in the /usr/include directory when the
corresponding file in the current directory has been changed.
Younger Files
If the $? macro is in the command sequence in the description file, the make
command replaces the symbol with a list of parent files that have been changed
since the target file was last changed. The make program replaces the symbol only
when it runs commands from the description file to create the target file.
First Out-of-Date File
If the $< macro is in the command sequence in the description file, the make
command replaces the symbol with the name of the file that started the file
creation. The file name is the name of the parent file that was out-of-date with the
target file, and therefore caused the make command to create the target file again.
In addition, use a letter (D or F) after the < (less-than sign) to get either the
directory name (D) or the file name (F) of the first out-of-date file. For example, if
the first out-of-date file is:
/home/linda/sample.c
then the make command gives the following values:
$(<D)
$(<F)
$<
=
=
=
/home/linda
sample.c
/home/linda/sample.c
The make program replaces this symbol only when the program runs commands
from its internal rules or from the .DEFAULT list.
Current File-Name Prefix
If the $* macro is in the command sequence in the description file, the make
command replaces the symbol with the file-name part (without the suffix) of the
parent file that the make command is currently using to generate the target file.
For example, if the make command is using the file:
test.c
then the $* macro represents the file name test.
In addition, use a letter (D or F) after the * (asterisk) to get either the directory
name (D) or the file name (F) of the current file.
For example, the make command uses many files (specified either in the
description file or in the internal rules) to create a target file. Only one of those
files (the current file) is used at any moment. If that current file is:
/home/tom/sample.c
then the make command gives the following values for the macros:
$(*D) = /home/tom
$(*F) = sample
$*
= /home/tom/sample
The make program replaces this symbol only when running commands from its
internal rules (or from the .DEFAULT list), but not when running commands from
a description file.
Chapter 14. make Command Overview
317
Archive Library Member
If the $% macro is in a description file, and the target file is an archive library
member, the make command replaces the macro symbol with the name of the
library member. For example, if the target file is:
lib(file.o)
then the make command replaces the $% macro with the member name, file.o.
Changing Macro Definitions in a Command
When macros in the shell commands are defined in the description file, you can
change the values that the make command assigns to the macro. To change the
assignment of the macro, put a : (colon) after the macro name, followed by a
replacement string. The form is as follows:
$(macro:string1=string2)
When the make command reads the macro and begins to assign the values to the
macro based on the macro definition, the command replaces each string1 in the
macro definition with a value of string2. For example, if the description file
contains the macro definition:
FILES=test.o sample.o form.o defs
you can replace the form.o file with a new file, input.o, by using the macro in the
description-file commands, as follows:
cc -o $(FILES:form.o=input.o)
Changing the value of a macro in this manner is useful when maintaining archive
libraries For more information, see the ar command).
How the make Command Creates a Target File
The make command creates a file containing the completed program called a target
file, using a step-by-step procedure.
The make program:
1. Finds the name of the target file in the description file or in the make
command
2. Ensures that the files on which the target file depends exist and are up-to-date
3. Determines if the target file is up-to-date with the files it depends on.
If the target file or one of the parent files is out-of-date, the make program creates
the target file using one of the following:
v Commands from the description file
v Internal rules to create the file (if they apply)
v Default rules from the description file.
If all files in the procedure are up-to-date when running the make program, the
make command displays a message to indicate that the file is up-to-date, and then
stops. If some files have changed, the make command builds only those files that
are out-of-date. The command does not rebuild files that are already current.
When the make program runs commands to create a target file, it replaces macros
with their values, writes each command line, and then passes the command to a
new copy of the shell.
318
General Programming Concepts
Using the make Command with Source Code Control System (SCCS)
Files
The SCCS command and file system is primarily used to control access to a file,
track who altered the file, why it was altered, and what was altered. An SCCS file
is any text file controlled with SCCS commands. Using non-SCCS commands to
edit SCCS files can damage the SCCS files. See “Chapter 23. Source Code Control
System (SCCS)” on page 663 to learn more about SCCS.
All SCCS files use the prefix s. to set them apart from regular text files. The make
program does not recognize references to prefixes of file names. Therefore, do not
refer to SCCS files directly within the make command description file. The make
program uses a different suffix, the x (tilde), to represent SCCS files. Therefore,
.cx.o refers to the rule that transforms an SCCS C language source file into an
object file. The internal rule is:
.cx.o:
$(GET) $(GFLAGS) -p $<
$(CC) $(CFLAGS) -c $*.c
-rm -f $*.c
>$*.c
The x (tilde) added to any suffix changes the file search into an SCCS file-name
search, with the actual suffix named by the . (period) and all characters up to (but
not including) the x (tilde). The GFLAGS macro passes flags to the SCCS to
determine which SCCS file version to use.
The make program recognizes the following SCCS suffixes:
.C\x
.cx
.yx
.sx
.shx
.hx
.fx
.lx
C++ source
c source
yacc source grammar
Assembler source
Shell
Header
FORTRAN
lex source
The make program has internal rules for changing the following SCCS files:
.C\x.a:
.C\x.c:
.C\x.o:
.cx:
.cx.a:
.cx.c:
.cx.o:
.fx:
.fx.a:
.fx.o:
.fx.f:
.hx.h:
Chapter 14. make Command Overview
319
.lx.o:
.sx.a:
.shx:
.sx.o:
.yx.c:
.yx.o:
Description Files Stored in the Source Code Control System (SCCS)
If you specify a description file, or a file named makefile or Makefile is in the
current directory, the make command does not look for a description file within
SCCS. If a description file is not in the current directory and you enter the make
command, the make program looks for an SCCS file named either s.makefile or
s.Makefile. If either of these files are present, the make command uses a get
command to direct SCCS to build the description file from that source file. When
the SCCS generates the description file, the make command uses the file as a
normal description file. When the make command finishes executing, it removes
the created description file from the current directory.
Using the make Command with Non-Source Code Control System
(SCCS) Files
Start the make program from the directory that contains the description file for the
file to create. The variable name desc-file represents the name of that description
file. Then, enter the command:
make -f desc-file
on the command line. If the name of the description file is makefile or Makefile,
you do not have to use the -f flag. Enter macro definitions, flags, description file
names, and target file names along with the make command on the command line
as follows:
make [flags] [macro definitions] [targets]
The make program then examines the command-line entries to determine what to
do. First, it looks at all macro definitions on the command line (entries that are
enclosed in quotes and have equal signs in them) and assigns values to them. If
the make program finds a definition for a macro on the command line different
from the definition for that macro in the description file, it chooses the
command-line definition for the macro.
Next, the make program looks at the flags. For more information, see the make
command for a list of the flags that it recognizes.
The make program expects the remaining command-line entries to be the names of
target files to be created. Any shell commands enclosed in back quotes that
generate target names are performed by the make command. Then the make
program creates the target files in left-to-right order. Without a target file name, the
make program creates the first target file named in the description file that does
not begin with a period. With more than one description file specified, the make
command searches the first description file for the name of the target file.
320
General Programming Concepts
How the make Command Uses the Environment Variables
Each time the make command runs, it reads the current environment variables and
adds them to its defined macros. Using the MAKEFLAGS macro or the MFLAGS
macro, the user can specify flags to be passed to the make command. If both are
set, the MAKEFLAGS macro overrides the MFLAGS macro. The flags specified
using these variables are passed to the make command along with any
command-line options. In the case of recursive calls to the make command, using
the $(MAKE) macro in the description file, the make command passes all flags
with each invocation.
When the make command runs, it assigns macro definitions in the following order:
1. Reads the MAKEFLAGS environment variable.
If the MAKEFLAGS environment variable is not present or null, the make
command checks for a non-null value in the MFLAGS environment variable. If
one of these variables has a value, the make command assumes that each letter
in the value is an input flag. The make program uses these flags (except for the
-f, -p, and -d flags, which cannot be set from the MAKEFLAGS or MFLAGS
environment variable) to determine its operating conditions.
2. Reads and sets the input flags from the command line. The command line adds
to the previous settings from the MAKEFLAGS or MFLAGS environment
variable.
3. Reads macro definitions from the command line. The make command ignores
any further assignments to these names.
4. Reads the internal macro definitions.
5. Reads the environment. The make program treats the environment variables as
macro definitions and passes them to other shell programs.
Example of a Description File
The following example description file could maintain the make program. The
source code for the make command is spread over a number of C language source
files and a yacc grammar.
#
#
P
#
Description file for the Make program
Macro def: send to be printed
= qprt
Macro def: source filenames used
FILES = Makefile version.c defs main.c \
doname.c misc.c files.c \
dosy.c gram.y lex.c gcos.c
# Macro def: object filenames used
OBJECTS = version.o main.o doname.o \
misc.o files.o dosys.o \
gram.o
# Macro def: lint program and flags
LINT = lint -p
# Macro def: C compiler flags
CFLAGS = -O
# make depends on the files specified
# in the OBJECTS macro definition
make:
$(OBJECTS)
# Build make with the cc program
cc $(CFLAGS) $(OBJECTS) -o make
# Show the file sizes
@size make
# The object files depend on a file
# named defs
$(OBJECTS): defs
Chapter 14. make Command Overview
321
# The file gram.o depends on lex.c
# uses internal rules to build gram.o
gram.o: lex.c
# Clean up the intermediate files
clean:
-rm *.o gram.c
-du
# Copy the newly created program
# to /usr/bin and deletes the program
# from the current directory
install:
@size make /usr/bin/make
cp make /usr/bin/make ; rm make
# Empty file "print" depends on the
# files included in the macro FILES
print: $(FILES)
# Print the recently changed files
pr $? | $P
# Change the date on the empty file,
# print, to show the date of the last
# printing
touch print
# Check the date of the old
# file against the date
# of the newly created file
test:
make -dp | grep -v TIME >1zap
/usr/bin/make -dp | grep -v TIME >2zap
diff 1zap 2zap
rm 1zap 2zap
# The program, lint, depends on the
# files that are listed
lint:
dosys.c doname.c files.c main.c misc.c \
version.c gram.c
# Run lint on the files listed
# LINT is an internal macro
$(LINT) dosys. doname.c files.c main.c \
misc.c version.c gram.c
rm gram.c
# Archive the files that build make
arch:
ar uv /sys/source/s2/make.a $(FILES)
The make program usually writes out each command before issuing it.
The following output results from entering the simple make command in a
directory containing only the source and description file:
cc -O -c version.c
cc -O -c main.c
cc -O -c doname.c
cc -O -c misc.c
cc -O -c files.c
cc -O -c dosys.c
yacc gram.y
mv y.tab.c gram.c
cc -O -c gram.c
cc version.o main.o doname.o misc.o files.o dosys.o
gram.o -o make
make: 63620 + 13124 + 764 + 4951 = 82459
None of the source files or grammars are specified in the description file. However,
the make command uses its suffix rules to find them and then issues the needed
commands. The string of digits on the last line of the previous example results
322
General Programming Concepts
from the size make command. Because the @ (at sign) on the size command in the
description file prevented writing of the command, only the sizes are written.
The output can be sent to a different printer or to a file by changing the definition
of the P macro on the command line, as follows:
make print "P = print -sp"
OR
make print "P = cat >zap"
Chapter 14. make Command Overview
323
324
General Programming Concepts
Chapter 15. m4 Macro Processor Overview
This chapter provides information about the m4 macro processor, which is a
front-end processor for any programming language being used in the operating
system environment.
The m4 macro processor is useful in many ways. At the beginning of a program,
you can define a symbolic name or symbolic constant as a particular string of
characters. You can then use the m4 program to replace unquoted occurrences of
the symbolic name with the corresponding string. Besides replacing one string of
text with another, the m4 macro processor provides the following features:
v
v
v
v
Arithmetic capabilities
File manipulation
Conditional macro expansion
String and substring functions
The m4 macro processor processes strings of letters and digits called tokens. The
m4 program reads each alphanumeric token and determines if it is the name of a
macro. The program then replaces the name of the macro with its defining text,
and pushes the resulting string back onto the input to be rescanned. You can call
macros with arguments, in which case the arguments are collected and substituted
into the right places in the defining text before the defining text is rescanned.
The m4 program provides built-in macros such as define. You can also create new
macros. Built-in and user-defined macros work the same way.
Using the m4 Macro Processor
To use the m4 macro processor, enter the following command:
m4 [file]
The m4 program processes each argument in order. If there are no arguments or if
an argument is - (dash), m4 reads standard input as its input file. The m4 program
writes its results to standard output. Therefore, to redirect the output to a file for
later use, use a command such as:
m4 [file] >outputfile
Creating a User-Defined Macro
define (MacroName, Replacement)
Defines new macro MacroName with a value
of Replacement.
For example, if the following statement is in a program:
define(name, stuff)
The m4 program defines the string name as stuff. When the string name occurs
in a program file, the m4 program replaces it with the string stuff. The string
name must be ASCII alphanumeric and must begin with a letter or underscore. The
string stuff is any text, but if the text contains parentheses the number of open, or
© Copyright IBM Corp. 1997, 1999
325
left, parentheses must equal the number of closed, or right, parentheses. Use the /
(slash) character to spread the text for stuff over multiple lines.
The open (left) parenthesis must immediately follow the word define. For
example:
define(N, 100)
. . .
if (i > N)
defines N to be 100 and uses the symbolic constant N in a later if statement.
Macro calls in a program have the following form:
name(arg1,arg2, . . . argn)
A macro name is recognized only if it is surrounded by nonalphanumerics. Using
the following example:
define(N, 100)
. . .
if (NNN > 100)
the variable NNN is not related to the defined macro N.
You can define macros in terms of other names. For example:
define(N, 100)
define(M, N)
defines both M and N to be 100. If you later change the definition of N and assign it
a new value, M retains the value of 100, not N.
The m4 macro processor expands macro names into their defining text as soon as
possible. The string N is replaced by 100. Then the string M is also replaced by 100.
The overall result is the same as using the following input in the first place:
define(M, 100)
The order of the definitions can be interchanged as follows:
define(M, N)
define(N, 100)
Now M is defined to be the string N, so when the value of M is requested later, the
result is the value of N at that time (because the M is replaced by N, which is
replaced by 100).
Using the Quote Characters
To delay the expansion of the arguments of define, enclose them in quote
characters. If you do not change them, quote characters are ′ ’ (left and right single
quotes). Any text surrounded by quote characters is not expanded immediately,
but quote characters are removed. The value of a quoted string is the string with
the quote characters removed. If the input is:
define(N, 100)
define(M, 'N')
the quote characters around the N are removed as the argument is being collected.
The result of using quote characters is to define M as the string N, not 100. The
general rule is that the m4 program always strips off one level of quote characters
326
General Programming Concepts
whenever it evaluates something. This is true even outside of macros. To make the
word define appear in the output, enter the word in quote characters, as follows:
'define' = 1;
Another example of using quote characters is redefining N. To redefine N, delay the
evaluation by putting N in quote characters. For example:
define(N, 100)
. . .
define('N', 200)
To prevent problems from occurring, quote the first argument of a macro. For
example, the following fragment does not redefine N:
define(N, 100)
. . .
define(N, 200)
The N in the second definition is replaced by 100. The result is the same as the
following statement:
define(100, 200)
The m4 program ignores this statement because it can only define names, not
numbers.
Changing the Quote Characters
Quote characters are normally ′ ’ (left or right single quotes). If those characters are
not convenient, change the quote characters with the following built-in macro:
changequote (l, r )
Changes the left and right quote characters to the characters
represented by the l and r variables.
To restore the original quote characters, use changequote without arguments as
follows:
changequote
Arguments
The simplest form of macro processing is replacing one string by another (fixed)
string. However, macros can also have arguments, so that you can use the macro
in different places with different results. To indicate where an argument is to be
used within the replacement text for a macro (the second argument of its
definition), use the symbol $n to indicate the nth argument. When the macro is
used, the m4 macro processor replaces the symbol with the value of the indicated
argument. For example, the symbol:
$2
refers to the second argument of a macro. Therefore, if you define a macro called
bump as:
define(bump, $1 = $1 + 1)
the m4 program generates code to increment the first argument by 1. The bump(x)
statement is equivalent to x = x + 1.
A macro can have as many arguments as needed. However, you can access only
nine arguments using the $n symbol ($1 through $9). To access arguments past the
ninth argument, use the shift macro.
Chapter 15. m4 Macro Processor Overview
327
shift (ParameterList)
Returns all but the first element of ParameterList to perform
a destructive left shift of the list.
This macro drops the first argument and reassigns the remaining arguments to the
$n symbols (second argument to $1, third argument to $2. . . tenth argument to $9).
Using the shift macro more than once allows access to all arguments used with the
macro.
The $0 macro returns the name of the macro. Arguments that are not supplied are
replaced by null strings, so that you can define a macro that concatenates its
arguments like this:
define(cat, $1$2$3$4$5$6$7$8$9)
Thus:
cat(x, y, z)
is the same as:
xyz
Arguments $4 through $9 in this example are null since corresponding arguments
were not provided.
The m4 program discards leading unquoted blanks, tabs, or new-line characters in
arguments, but keeps all other white space. Thus:
define(a, b c)
defines a to be b c.
Arguments are separated by commas. Use parentheses to enclose arguments
containing commas, so that the comma does not end the argument. For example:
define(a, (b,c))
has only two arguments. The first argument is a, and the second is (b,c). To use a
comma or single parenthesis, enclose it in quote characters.
Using a Built-In m4 Macro
The m4 program provides a set of predefined macros. The subsequent sections
explain many of the macros and their uses.
Removing a Macro Definition
undefine (′MacroName’)
Removes the definition of a user-defined or built-in
macro (′MacroName’)
For example:
undefine(′N')
removes the definition of N. Once you remove a built-in macro with the undefine
macro, as follows:
undefine(′define')
then you cannot use its definition of the built-in macro again.
328
General Programming Concepts
Single quotes are required in this case to prevent substitution.
Checking for a Defined Macro
ifdef (′MacroName’, Argument1, Argument2)
If macro MacroName is defined and is not
defined to zero, returns the value of
Argument1. Otherwise, it returns Argument2.
The ifdef macro permits three arguments. If the first argument is defined, the
value of ifdef is the second argument. If the first argument is not defined, the
value of ifdef is the third argument. If there is no third argument, the value of ifdef
is null.
Using Integer Arithmetic
The m4 program provides the following built-in functions for doing arithmetic on
integers only:
incr (Number)
decr (Number )
eval
Returns the value of Number + 1.
Returns the value of Number - 1.
Evaluates an arithmetic expression.
Thus, to define a variable as one more than the Number value, use the following:
define(Number, 100)
define(Number1, ′incr(Number)')
This defines Number1 as one more than the current value of Number.
The eval function can evaluate expressions containing the following operators
(listed in decreasing order of precedence):
unary + and ** or | (exponentiation)
* / % (modulus)
+== != < <= > >=
!(not)
& or && (logical AND)
| or || (logical OR)
Use parentheses to group operations where needed. All operands of an expression
must be numeric. The numeric value of a true relation (for example, 1 > 0) is 1,
and false is 0. The precision of the eval function is 32 bits.
For example, define M to be 2==N+1 using the eval function as follows:
define(N, 3)
define(M, ′eval(2==N+1)')
Use quote characters around the text that defines a macro unless the text is very
simple.
Chapter 15. m4 Macro Processor Overview
329
Manipulating Files
To merge a new file in the input, use the built-in include function.
include (File)
Returns the contents of the file File.
For example:
include(FileName)
inserts the contents of FileName in place of the include command.
A fatal error occurs if the file named in the include macro cannot be accessed. To
avoid a fatal error, use the alternate form sinclude.
sinclude (File )
Returns the contents of the file File, but does not report an error if
it cannot access File.
The sinclude (silent include) macro does not write a message, but continues if the
file named cannot be accessed.
Redirecting Output
The output of the m4 program can be redirected again to temporary files during
processing, and the collected material can be output upon command. The m4
program maintains nine possible temporary files, numbered 1 through 9. If you use
the built-in divert macro.
divert (Number)
Changes output stream to the temporary file Number.
The m4 program writes all output from the program after the divert function at
the end of temporary file, Number. To return the output to the display screen, use
either the divert or divert(0) function, which resumes the normal output process.
The m4 program writes all redirected output to the temporary files in numerical
order at the end of processing. The m4 program discards the output if you redirect
the output to a temporary file other than 0 through 9.
To bring back the data from all temporary files in numerical order, use the built-in
undivert macro.
undivert (Number1, Number2... )
Appends the contents of the indicated
temporary files to the current temporary
file.
To bring back selected temporary files in a specified order, use the built-in
undivert macro with arguments. When using the undivert macro, the m4 program
discards the temporary files that are recovered and does not search the recovered
data for macros.
The value of the undivert macro is not the diverted text.
divnum
Returns the value of the currently active temporary file.
If you do not change the output file with the divert macro, the m4 program puts
all output in a temporary file named 0.
330
General Programming Concepts
Using System Programs in a Program
You can run any program in the operating system from a program by using the
built-in syscmd macro. For example, the following statement runs the date
program:
syscmd(date)
Using Unique File Names
Use the built-in maketemp macro to make a unique file name from a program.
maketemp (String...nnnnn...String)
Creates a unique file name by
replacing the characters nnnnn in the
argument string with the current
process ID.
For example, for the statement:
maketemp(myfilennnnn)
the m4 program returns a string that is myfile concatenated with the process ID.
Use this string to name a temporary file.
Using Conditional Expressions
ifelse (String1, String2, Argument1, Argument2)
If String1 matches String2, returns the value
of Argument1. Otherwise it returns
Argument2.
The built-in ifelse macro performs conditional testing. In the simplest form:
ifelse(a, b, c, d)
compares the two strings a and b.
If a and b are identical, the built-in ifelse macro returns the string c. If they are not
identical, it returns string d. For example, you can define a macro called compare to
compare two strings and return yes if they are the same, or no if they are different,
as follows:
define(compare, ′ifelse($1, $2, yes, no)′)
The quote characters prevent the evaluation of the ifelse macro from occurring too
early. If the fourth argument is missing, it is treated as empty.
The ifelse macro can have any number of arguments, and therefore, provides a
limited form of multiple-path decision capability. For example:
ifelse(a, b, c, d, e, f, g)
This statement is logically the same as the following fragment:
if(a == b) x = c;
else if(d == e) x = f;
else x = g;
return(x);
If the final argument is omitted, the result is null, so:
ifelse(a, b, c)
Chapter 15. m4 Macro Processor Overview
331
is c if a matches b, and null otherwise.
Manipulating Strings
len
Returns the byte length of the string that makes up its argument
Thus:
len(abcdef)
is 6, and:
len((a,b))
is 5.
dlen
Returns the length of the displayable characters in a string
Characters made up from 2-byte codes are displayed as one character. Thus, if the
string contains any 2-byte, international character-support characters, the results of
dlen will differ from the results of len.
substr (String, Position, Length)
Returns a substring of String that begins
at character number Position and is
Length characters long.
Using input, substr (s, i, n) returns the substring of s that starts at the ith position
(origin zero) and is n characters long. If n is omitted, the rest of the string is
returned. For example, the function:
substr('now is the time',1)
returns the following string:
now is the time
index (String1, String2)
Returns the character position in String1 where String2
starts (starting with character number 0), or -1 if
String1 does not contain String2.
As with the built-in substr macro, the origin for strings is 0.
translit (String, Set1, Set2)
Searches String for characters that are in Set1.
If it finds any, changes (transliterates) those
characters to corresponding characters in Set2.
It has the general form:
translit(s, f, t)
which modifies s by replacing any character found in f by the corresponding
character of t. For example, the function:
translit(s, aeiou, 12345)
replaces the vowels by the corresponding digits. If t is shorter than f, characters
that do not have an entry in t are deleted. If t is not present at all, characters from
f are deleted from s. So:
translit(s, aeiou)
deletes vowels from string s.
dnl
332
Deletes all characters that follow it, up to and including the new-line character.
General Programming Concepts
Use this macro to get rid of empty lines. For example, the function:
define(N, 100)
define(M, 200)
define(L, 300)
results in a new-line at the end of each line that is not part of the definition. These
new-line characters are passed to the output. To get rid of the new lines, add the
built-in dnl macro to each of the lines.
define(N, 100) dnl
define(M, 200) dnl
define(L, 300) dnl
Printing
errprint (String)
Writes its argument (String) to the standard error file
For example:
errprint ('error')
dumpdef (′MacroName’... )
Dumps the current names and definitions of items
named as arguments (′MacroName’...)
If you do not supply arguments, the dumpdef macro prints all current names and
definitions. Remember to quote the names.
List of Additional m4 Macros
A list of additional m4 macros, with a brief explanation of each, follows:
changecom (l, r )
defn (MacroName)
en (String)
eval (Expression)
m4exit (Code)
m4wrap (MacroName)
popdef (MacroName)
pushdef (MacroName, Replacement)
syscmd (Command)
sysval
traceoff (MacroList)
traceon (MacroName)
Changes the left and right comment
characters to the characters represented by
the l and r variables.
Returns the quoted definition of
MacroName
Returns the number of characters in
String.
Evaluates Expression as a 32-bit arithmetic
expression.
Exits m4 with a return code of Code.
Runs macro MacroName at the end of m4.
Replaces the current definition of
MacroName with the previous definition
saved with the pushdef macro.
Saves the current definition of MacroName
and then defines MacroName to be
Replacement.
Executes the system command Command
with no return value.
Gets the return code from the last use of
the syscmd macro.
Turns off trace for any macro in MacroList.
If MacroList is null, turns off all tracing.
Turns on trace for macro MacroName. If
MacroName is null, turns trace on for all
macros.
Chapter 15. m4 Macro Processor Overview
333
334
General Programming Concepts
Chapter 16. National Language Support
National Language Support (NLS)provides commands and library subroutines for
a single worldwide system base.
v Code sets
v Character classifications
v Character comparison rules
v Character collation order
v Numeric and monetary formatting
v Date and time formatting
v Message-text language
NLS Capabilities
An application that runs in an international environment must not have built-in
assumptions about:
v “Locale-Specific and Culture-Specific Conventions”
v “User Messages in Native Languages”
v “Code Set Support” on page 336
v “Input Method Support” on page 336
This information must be determined during application execution. NLS provides
these capabilities and a base upon which new languages and code sets can be
supported. As a result, programs can be ported across national language and locale
boundaries. The POSIX.1 standard, the POSIX.2 standard, the ANSI/ISO C
language standard, and the X/Open XPG specifications define standards for
providing NLS support.
Locale-Specific and Culture-Specific Conventions
An internationalized program can process information correctly for different
locations. (For example, the conventions for specifying date and time differ in the
United States and England.) Similarly, the decimal point (radix character) and
monetary symbols differ between the two countries. These types of language and
cultural conventions for handling information are defined in a locale. For more
information about locales, see “Locale Overview for Programming” on page 337.
User Messages in Native Languages
To facilitate translations of messagesinto various languages and make translated
messages available to the program based on a user’s locale, messages are kept
separate from the programs by providing them in the form of message catalogs
that a program can access at run time. To aid in this task, commands and
subroutines are provided by the Message Facility. For more information, see
“Message Facility Overview for Programming” on page 498.
© Copyright IBM Corp. 1997, 1999
335
Code Set Support
A characteris any symbol used for the organization, control, or representation of
data. A group of such symbols for describing a particular language make up a
character set.A code setcontains the encoding values for a character set. The
encoding values in a code set provide the interface between the system and its
input and output devices.
In the past,the effort was directed at encoding the English alphabet. A 7-bit
encoding method was adequate for this purpose because the number of English
characters is not large. The C language defined the char data type to indicate a
7-bit character. A byte is an 8-bit quantity and is therefore used to represent a char
data type value. The eighth bit was typically used for parity.
To support larger character sets, such as the Asian languages (for example,
Chinese, Japanese, and Korean), additional code sets were developed that
contained multibyte encodings. Because of multibyte encodings, the old concept of
the char data type is no longer sufficient to represent a character. The C standard
continues to refer to the char data type to mean a 7-bit character. However, the
char data type really means a byte, either signed or unsigned.
An internationalized program must accurately read data generated in different
code set environments and process the information accurately. You can use
nl_langinfo(CODESET)to obtain the current code set in a process. The return
value is a char pointer that is the name of the code set in the system. Because code
set names are not standard, programs should not depend on any specific value for
this string. Knowing the current code set can aid in code-set conversion. NLS
supplies convertersthat translate character encoding values found in different code
sets. For more information, see “Converters Overview for Programming” on
page 423.
Input Method Support
The input of characters becomes complicated for languages having large character
sets. For example, in Chinese, Korean, and Japanese, where the number of
characters is large, it is not possible to provide one-to-one key mapping for a
keystroke to a character. However, a special input method enables the user to enter
phonetic or stroke characters and have them converted into native-language
characters. A keyboard map associated with each keyboard matches sequences of
one or more keystrokes with the appropriate character encoding. For more
information, see “Input Method Overview” on page 465.
Overview of Chapter Contents
This NLS chapter contains the following information:
v “Locale Overview for Programming” on page 337
v “National Language Support Subroutines Overview” on page 348
v “Layout (Bidirectional Text and Character Shaping) Overview” on page 381
v “Use of the libcur Package” on page 385
v “Code Set Overview” on page 387
v “Converters Overview for Programming” on page 423
v “Writing Converters Using the iconv Interface” on page 451
v “Input Method Overview” on page 465
v “Message Facility Overview for Programming” on page 498
336
General Programming Concepts
v “Culture-Specific Data Processing” on page 508
v “NLS Sample Program” on page 510
v “National Language Support (NLS) Quick Reference” on page 516
Locale Overview for Programming
National Language Support (NLS)provides commands and library subroutines for
a single worldwide system base. An internationalized system has no built-in
assumptions or dependencies on language-specific or cultural-specific conventions.
All locale information is obtained at program run time.
The following concepts are needed to understand the internationalization of
programs:
v “Working with Code Sets”
v “Data Representation” on page 338
v “Character Properties” on page 339
v “Localization” on page 340
v “Multibyte Subroutines” on page 343
v “Wide Character Subroutines” on page 343
v “Bidirectionality and Character Shaping” on page 344
v “Code Set Independence” on page 344
v “File Name Matching” on page 345
v “Radix Character Handling” on page 345
v “Programming Model” on page 346
Working with Code Sets
ASCII is a code set containing 128 code points (0x00 through 0x7F). The ASCII
character set contains control characters, punctuation marks, digits, and the
uppercase and lowercase English alphabet. Several 8-bit code sets incorporate
ASCII as a proper subset. However, throughout this document, ASCII refers to
7-bit-only code sets. To emphasize this, it is referred to as 7-bit ASCII. The 7-bit
ASCII code set is a proper subset of all supported code sets and is referred to as
the portable character set. For more information, see “Code Set Overview” on
page 387.
Single-Byte and Multibyte Code Sets
A single-byte encoding method is sufficient for representing the English character
set because the number of characters is not large. To support larger alphabets, such
as Japanese and Chinese, additional code sets containing multibyte encodings are
necessary. All supported single-byte and multibyte code sets contain the single-byte
ASCII character set. Therefore, programs that handle multibyte code sets must
handle character encodings of one or more bytes.
Examples of single-byte code sets are the ISO 8859 family of code sets and the
IBM-850 code set. Examples of multibyte character sets are the IBM-eucJP and the
IBM-943 code sets. The single-byte code sets have at most 256 characters and the
multibyte code sets have more than 256 (without any theoretical limit).
Chapter 16. National Language Support
337
The Unique Code-Point Range
None of the supported code sets have bytes 0x00 through 0x3F in any byte of a
multibyte character. This group of code points is called the unique code-point range.
Furthermore, these code points always refer to the same characters as specified for
7-bit ASCII. This is a special property governing all supported code sets. “ASCII
Characters” on page 388 lists the characters in the unique code-point range.
For more information about code sets, see “Code Set Overview” on page 387.
Data Representation
Because the encoding for some characters requires more than one byte, a single
character may be represented by one or several bytes when data is created in files
or transferred between a computer and its I/O devices. This external
representation of data is referred to as the file code or multibyte character code
representation of a character.
For processing strings of such characters, it is more efficient to convert file codes
into a uniform representation. This converted form is intended for internal
processing of characters. This internal representation of data is referred to as the
process code or wide character code representation of the character. An understanding
of multibyte character and wide character codes is essential to the overall
internationalization strategy.
Multibyte Character Code Data Representation
A multibyte character code is an external representation of data, regardless of
whether it is character input from a keyboard or a file on a disk. Within the same
code set, the number of bytes that represent the multibyte code of a character can
vary. You must use NLS functions for character processing to ensure code set
independence.
For example, a code set may specify the following character encodings:
C = 0x43
* = 0x81 0x43
*C = 0x81 0x43& 0x43
A program searching for C, not accounting for multibyte characters, finds the
second byte of the *C string and assumes it found C when in fact it found the
second byte of the * (asterisk) character.
Wide Character Code Data Representation
The wide character code was developed so that multibyte characters could be
processed more efficiently internally in the system. A multibyte character
representation is converted into a uniform internal representation (wide character
code) so that internally all characters have the same length. Using this internal
form, character processing can be done in a code set-independent fashion. The
wide character code refers to this internal representation of characters.
The wchar_t data type is used to represent the wide character code of a character.
The size of the wchar_t data type is implementation-specific. It is a typedef
definition and can be found in the ctype.h, stddef.h, and stdlib.h files. No
program should assume a particular size for the wchar_t data type, enabling
programs to run under implementations that use different sizes for the wchar_t
data type.
338
General Programming Concepts
On AIX 4.3, the wchar_t datatype is implemented as an unsigned short value (16
bits). The locale methods in AIX have been standardized such that in most locales,
the value stored in the wchar_t for a particular character will always be its
Unicode data value. For applications which are intended to run only on AIX, this
allows certain applications handle the wchar_t datatype in a consistent fashion,
even if the underlying codeset is unknown. All locales on AIX 4.3 will use Unicode
for their wide character code values (process code), except the following:
1. Locales based on the IBM-850 codeset are provided strictly for compatibility
with previous releases of AIX. These locales have not been modified from
previous releases and will be removed in a future release of AIX. It is strongly
suggested that users use the industry standard ISO8859-1 codeset instead of
IBM-850. For IBM-850 locales, the wchar_t data value will be the same value as
the IBM-850 codepoint value.
2. The IBM-eucTW codeset (LANG =zh_TW) contains many characters that are
not contained in the Unicode standard. Because of this, it is impossible to
represent these characters with a Unicode wide character value. Applications
that need to have Unicode based wchar_t data for Traditional Chinese should
use the Zh_TW locale (big5 codeset) instead.
Character Properties
Every character has several language-dependent attributes or properties. These
properties are called class properties. For example, the lowercase letter a in U.S.
English has the following properties:
v
v
v
v
v
alphabetic
hexadecimal digit
printable
lowercase
graphic
Character class properties are specified by the LC_CTYPE category.
Collation-Order Properties
Character ordering or collation refers to the culture-specific ordering of characters.
This ordering differs from that based on the ordinal value of a character in a code
set. Collation-based ordering is dependent on the language. Character collation is
specified by the LC_COLLATE category. The term collating element refers to one or
more characters that have a collation value in a specific locale. The Spanish ll
character is an example of a multicharacter collating element.
To sort the characters in any given language in the proper order, a Weight is
assigned to each character so they sort as expected. However, a character’s sort
value and code-point value are not necessarily related.
One set of weights is not sufficient to sort strings for all languages. For example, in
the case of the German words b<a-umlaut>ch and bane, if there is only one set of
weights, and the weight of the letter a is less than that of <a-umlaut>, then bane
sorts before b<a-umlaut>ch. However, the opposite result is correct. To satisfy the
requirement of this example, two sets of weights, the Primary and Secondary
Weights, are given to each character in the language. In the case of the characters a
and <a-umlaut>, they have the same Primary Weights, but differ in their Secondary
Weights. In the German locale, the Secondary Weight of a is less than that of
<a-umlaut>.
Chapter 16. National Language Support
339
The sorting algorithm first compares the two strings based on the Primary Weights
of each character. If the Primary Weight values are the same, the two strings are
compared again based on their Secondary Weights. In this example, the Primary
Weights of the first two characters ba and b<a-umlaut> are the same, but the
Primary Weights of the characters that follow (c and n, respectively) differ. As a
result of this comparison, b<a-umlaut>ch is sorted before bane.
Here, the Secondary Weights are not used to collate the strings. However, as in the
case of the strings bach and b<a-umlaut>ch, Secondary Weights must be used to get
the proper order. When compared using Primary Weight values, these two strings
are found to be equivalent. To break the tie, the Secondary Weights of a and
<a-umlaut> are used. Because the Secondary Weight of a is less than that of
<a-umlaut>, the string bach sorts before b<a-umlaut>ch.
Characters having the same Primary Weights belong to the same equivalence class.
In this example, the characters a and <a-umlaut> are said to be members of the
same equivalence class.
In string collation, each pair of strings is first compared based on Primary Weight.
If the two strings are equal, they are compared again based on their Secondary
Weights. If still equal, they are compared again based on Tertiary Weights up to
the limit set by the COLL_WEIGHTS_MAX collating weight limit specified in the
sys/limits.h file.
Code-Set Width
Code-set width refers to the maximum number of bytes required to represent a
character as a file code. This information is specified by the LC_CTYPE category.
Code-Set Display Width
Code-set display width refers to the maximum number of columns required to
display a character on a terminal. This information is specified by the LC_CTYPE
category.
Localization
An internationalized program must process information correctly for different
locations. For example, in the United States, the date format 9/6/1990 is
interpreted to mean the sixth day of the ninth month of the year 1990. The United
Kingdom interprets the same date format to mean the ninth day of the sixth month
of the year 1990. The formatting of numerical and monetary data is also country
specific, as in the case of the U.S. dollar and the U.K. pound.
A locale is defined by language-specific and cultural-specific conventions for
processing information. All such information should be accessible to a program at
run time so that the same program can display or process data differently for
different countries. The process of providing a language interface to obtain and
process this information into a database containing the locale-specific data is
known as localization.
The setlocale subroutine establishes locale information. This subroutine uses the
values of certain environment variables to initialize locale information contained in
locale definition files. To deal with locale data in a logical manner, locale definition
source files are divided into six categories defining specific aspects of the locale
data.
340
General Programming Concepts
Locale Categories
A category is a group of language-specific and culture-specific data. For instance,
data referring to date and time formatting, the names of the days of the week, and
names of the months is grouped into the LC_TIME category. Each category uses a
set of keywords that describe a particular aspect of a locale. The following
standard categories can be defined in a locale definition source file:
Defines string-collation order information.
Defines character classification, case conversion, and other character
attributes.
LC_MESSAGES Defines the format for affirmative and negative responses.
LC_MONETARY Defines rules and symbols for formatting monetary numeric information.
LC_NUMERIC Defines rules and symbols for formatting nonmonetary numeric
information.
LC_TIME
Lists rules and symbols for formatting time and date information.
LC_COLLATE
LC_CTYPE
Understanding Locale
Locale information consists of data from these six categories. Each locale is
described by a locale definition file. These files are named by the language,
territory and code set information they describe. The format for naming a locale
definition file is:
language[_territory][.codeset][@modifier]
For example, the locale for the Danish language spoken in Denmark using the
ISO8859-1 code set is da_DK.ISO8859-1. The da stands for the Danish language and
the DK stands for Denmark. The short form of da_DK is sufficient to indicate this
locale. The same language and territory using the IBM-850 code set is indicated by
either Da_DK.IBM-850 or Da_DK for short.
The C or POSIX Locale
This locale refers to the ANSI C or POSIX-defined standard for the locale inherited
by all processes at startup time. The C or POSIX locale assumes the 7-bit ASCII
character set and defines information for the six previous categories.
The Installation Default Locale
The installation default locale refers to the locale selected at system installation
time as the systemwide locale. For example, a French user in Canada may define
the default locale to be fr_CA.ISO8859-1 (fr for French, CA for Canada, and
ISO8859-1 for the code set). Every process uses this locale unless the NLS
environment variables are changed.
NLS Environment Variables
For localization, NLS uses the following environment variables:
v LANG
v LC_ALL
v LC_COLLATE
v LC_CTYPE
v LC_MESSAGES
v LC_MONETARY
v LC_NUMERIC
Chapter 16. National Language Support
341
v LC_TIME
v LOCPATH
v NLSPATH
The LC_COLLATE, LC_CTYPE, LC_MONETARY, LC_NUMERIC, LC_TIME, and
LC_MESSAGES environment variables determine the current values for their
respective categories.
The LC_ALL and LANG environment variables also determine the current locale.
The NLSPATH environment variable specifies a colon-separated list of directory
names where the message catalog files are located. This environment variable is
used by the Message Facility component of the NLS subsystem.
The LOCPATH environment variable specifies the directories where localization
information such as locale database files, input method files, and iconv converters
are located. This variable specifies a colon-separated list of directory names. The
list is used for setting up the locale for a particular process.
Note: All setuid and setgid programs will ignore the LOCPATH environment
variable.
The environment variables that affect locale information can be grouped into three
priority classes:
Priority Class
Environment Variable
high
LC_ALL
medium
LC_COLLATE, LC_CTYPE,
LC_MESSAGES, LC_MONETARY,
LC_NUMERIC
low
LANG
When a locale is requested by the setlocale subroutine for a particular category or
for all categories, the environment variable settings are queried by their priority
level in the following manner:
v If the LC_ALL environment variable is set, all six categories use the locale it
specified. For example, if the LC_ALL environment variable is equal to en_US
and the LANG environment variable is equal to fr_FR, a call to the setlocale
subroutine sets each of the six categories to the en_US locale.
v If the LC_ALL environment variable is not set, each individual category uses the
locale specified by its corresponding environment variable. For example, if the
LC_ALL environment variable is not set, the LC_COLLATE environment
variable is set to de_DE, and the LC_TIME environment variable is set to fr_CA,
then a call to the setlocale subroutine sets the LC_COLLATE category to de_DE
and the LC_TIME category to fr_CA. Neither environment variable has
precedence over the other in this situation.
v If the LC_ALL environment variable is not set, and a value for a particular LC_*
environment variable is not set, the value of the LANG environment variable
determines the setting for that specific category. For example, if the LC_ALL
environment variable is not set, the LC_CTYPE environment variable is set to
en_US, the LC_NUMERIC environment variable is not set, and the LANG
environment variable is set to is_IS, then a call to the setlocale subroutine sets
the LC_CTYPE category to en_US and the LC_NUMERIC category to is_IS. The
342
General Programming Concepts
LANG environment variable specifies the locale for only those categories not
previously determined by an LC_* environment variable.
v If the LC_ALL environment variable is not set, a value for a particular LC_*
environment variable is not set, and the value of the LANG environment
variable is not set, the locale for that specific category defaults to the C locale.
For example, if the LC_ALL environment variable is not set, the
LC_MONETARY environment variable is set to sv_SE, the LC_TIME
environment variable is not set, and the LANG environment variable is not set,
then a call to the setlocale subroutine sets the LC_MONETARY category to
sv_SE and the LC_TIME category to C.
Environment Variables Precedence Example
The following table shows the current setting of the environment variables and the
effect of calling setlocale(LC_ALL,″″). After the setlocale subroutine is called, the
string sorting and character properties are done as in the German language, the
monetary formatting is done as in the US conventions, the numeric, time
formatting is done in Danish conventions, the date and time data formatting is
done in the Danish conventions, and the user messages are displayed in the
Danish language. The last column indicates the locale setting after
setlocale(LC_ALL,″″) is called.
Environment Variable and
Category Names
Value of Environment
Variables
Value of Category After Call
To setlocale(LC_ALL,″″)
LC_COLLATE
de_DE
de_DE
LC_CTYPE
de_DE
de_DE
LC_MONETARY
en_US
en_US
LC_NUMERIC
(unset)
da_DK
LC_TIME
(unset)
da_DK
LC_MESSAGES
(unset)
da_DK
LC_ALL
(unset)
(not applicable)
LANG
da_DK
(not applicable)
Multibyte Subroutines
Multibyte subroutines process characters in file-code form. The names of these
subroutines usually start with the prefix mb. However, some multibyte subroutines
do not have this prefix. For example, the strcoll and strxfrm subroutines process
characters in their multibyte form but do not have the mb prefix. The following
standard C subroutines operate on bytes and can be used in handling multibyte
data: strcmp, strcpy, strncmp, strncpy, strcat, and strncat. The standard C search
subroutines strchr, strrchr, strpbrk, strcspn, strrchr, strspn, strstr, and strtok can
be used in the following cases:
v Searching or scanning for characters in single-byte code sets
v Searching or scanning for unique code-point range characters in multibyte
strings.
For more information about multibyte character subroutines, see “National
Language Support Subroutines Overview” on page 348.
Wide Character Subroutines
Wide character subroutines process characters in process-code form. Wide character
subroutines usually start with a wc prefix. However, there are exceptions to this
Chapter 16. National Language Support
343
rule. For example, the wide character classification functions use an isw prefix. To
determine if a subroutine is a wide character subroutine, check if the subroutine
prototype defines characters as wchar_t data type or wchar_t data pointer, or else
check whether the subroutine returns a wchar_t data type. There are some
exceptions to this rule. For example, the wide character classification subroutines
accept wint_t data type values.
For more information about wide character subroutines, see “National Language
Support Subroutines Overview” on page 348.
Bidirectionality and Character Shaping
An internationalized program may need to handle bidirectionality of text and
character shaping.
Bidirectionality (BIDI) occurs when texts of different direction orientation appear
together. For example, English text is read from left to right. Hebrew text is read
from right to left. If both English and Hebrew texts appear on the same line, the
text is bidirectional.
Character shaping occurs when the shape of a character is dependent on its position
in a line of text. In some languages, such as Arabic, characters have different
shapes depending on their position in a string and on the surrounding characters.
For more information about bidirectionality and character shaping, see “Layout
(Bidirectional Text and Character Shaping) Overview” on page 381, “Character
Shaping” on page 384, and “Introducing Layout Library Subroutines” on page 385.
Code Set Independence
The system needs certain information about code sets to communicate with the
external environment. This information is hidden by the code set-independent
library subroutines (NLS library). These subroutines pass information to the code
set-dependent functions. Because NLS subroutines handle the necessary code set
information, you do not need explicit knowledge of any code set when you write
programs that process characters. This programming technique is called code set
independence.
Determining Maximum Number of Bytes in Code Sets
You can use the MB_CUR_MAX macro to determine the maximum number of
bytes in a multibyte character for the code set in the current locale. The value of
this macro is dependent on the current setting of the LC_CTYPE category. Because
the locale can differ between processes, running the MB_CUR_MAX macro in
different processes or at different times may produce different results. The
MB_CUR_MAX macro is defined in the stdlib.h header file.
You can use the MB_LEN_MAX macro to determine the maximum number of
bytes in any code set that is supported by the system. This macro is defined in the
limits.h header file.
Determining Character and String Display Widths
The _max_disp_width macro is operating-system-specific, and its use should be
avoided in portable applications. If portability is not important, you can use the
_max_disp_width macro to determine the maximum number of display columns
344
General Programming Concepts
required by a single character in the code set in the current locale. The value of
this macro is dependent on the current setting of the LC_CTYPE category. If the
value of this is 1 (one), all characters in the current code set require only one
display column width on output.
When both MB_CUR_MAX and _max_disp_width are set to 1 (one), you can use
the strlen subroutine to determine the display column width needed for a string.
When MB_CUR_MAX is greater than one, use the wcswidth subroutine to find
the display column width of the string.
The wcswidth and wcwidth wide character display width subroutines do not have
corresponding multibyte functions. The wcswidth subroutine does not indicate
how many characters can be displayed in the space available on a display. The
wcwidth subroutine is useful for this purpose. This subroutine must be called
repeatedly on a wide character string to find out how many characters can be
displayed in the available positions on the display.
Exceptions to Code Set Knowledge: Unique Code-Point Range
There is one exception to the statement: ″No knowledge of the underlying code set
can be assumed in a program.″ This exception arises due to the way the supported
code sets are organized.
When a multibyte character string is searched for any character within the unique
code-point range (for example, the . (period) character), it is not necessary to
convert the string to process code form. It is sufficient to just look for that
character (.) by examining each byte. This exception enables the kernel and utilities
to search for the special characters . and / while parsing file names. If a program
searches for any of the characters in the unique code-point range, the standard
string functions that operate on bytes (such as strchr), should be used. “ASCII
Characters” on page 388) lists the characters in the unique code-point range.
Note: This exception is not a property that is applicable to all code sets. It
should not be construed that this exception will remain valid in future
releases.
File Name Matching
POSIX.2 defines the fnmatch subroutine to be used for file name matching. An
application can use the fnmatch subroutine to read a directory and apply a pattern
against each entry. For example, the find utility can use the fnmatch subroutine.
The pax utility can use the fnmatch subroutine to process its pattern operands.
Applications that need to match strings in a similar fashion can use the fnmatch
subroutine.
Radix Character Handling
Note that the radix character, as obtained by nl_langinfo(RADIXCHAR), is a
pointer to a string. It is possible that a locale may specify this as a multibyte
character or as a string of characters. However, in AIX, a simplifying assumption is
made that the RADIXCHAR is a single-byte character.
Chapter 16. National Language Support
345
Programming Model
The programming model presented here highlights changes you need to make
when an existing program is internationalized or when a new program is
developed:
v Provide complete internationalization. Do not assume that characters have any
specific properties. Determine the properties dynamically by using the
appropriate interfaces. Do not assume properties of code sets, except for the
ASCII characters with code points in the unique code-point range.
v Make programs code set-independent. Programs should not assume single-byte,
double-byte, or multibyte encoding of any sort. Data can be processed in either
process-code or file-code form by using the appropriate subroutines.
v Provide interaction with the kernel in file-code form only. The kernel does not
handle process codes.
v The NLS subroutine library can handle processing based on file-code as well as
processing based on process-code.
Note: Several subroutines based on process-code do not have
corresponding subroutines based on file-code. Due to this asymmetry, it
may be necessary to convert strings to process-code form and invoke the
appropriate process-code subroutines.
v Some libraries may not provide processing in process-code form. An application
needing these libraries must use file-codes when invoking functions from them.
v Programs can process characters either in process-code form or file-code form. It
is possible to write code set-independent programs using both methods.
The Program Model for Internationalization of Programs diagram gives an overall
view of the flow of user text data. The model indicates when the text data is in the
multibyte character code (file-code) form and when it is in wide character code
(process-code) form.
346
General Programming Concepts
Program
File codes
user
Process codes
File codes
System calls
library
Code
set-independent,
Process code-based
functions
Code
set-independent,
File code-based
functions
(no character
recognition,
bytes only)
(widechar functions)
(multibyte functions)
NLS
library
Code set-dependent (CSD) functions
File codes
File codes
File-name parsing
kernel
TTY driver
File-code
form
LAN
File-code
form
DISPLAY
File-code
form
KEYBOARD
File-code
form
File-code
form
PRINTER
DISK FILE
Program Model for Internationalization of Programs
The Application Program Interface to the NLS Library diagram shows the
process-code and file-code subroutine categories. Subroutines that support process
codes may not have corresponding subroutines that support file codes. Also,
subroutines that support file codes may not have corresponding subroutines that
support process codes.
Chapter 16. National Language Support
347
user
Program
Process codes
NLS
Library
File codes
widechar subroutines
multibyte subroutines
widechar I/O
multibyte I/O
widechar string manipulation
multibyte string manipulation
multibyte regular expressions
widechar classification
multibyte classification
message catalog access
CSI
locale setup and access
code set conversion
widechar-to-multibyte
conversion and vice versa
multibyte-to-widechar
conversion and vice versa
multibyte string conversion
widechar display width
CSD
Code set-dependent functions
File codes
kernel
Application Program Interface to the NLS Library
National Language Support Subroutines Overview
When internationalizing programs using National Language Support (NLS), it is
important that there be some guidelines for providing this support. The intent of
this section is to guide programmers in developing portable internationalized
programs. An understanding of the concepts explained in “Locale Overview for
Programming” on page 337 is a prerequisite to this section.
Introducing Locale Subroutines
Programs that perform locale-dependent processing, including user messages, must
call the setlocalesubroutine at the beginning of the program. This call should be
the first executable statement in the main program. Programs that do not call the
setlocale subroutine in this way inherit the Cor POSIXlocale. Such programs
perform as in the C locale regardless of the setting of the LC_* and LANG
environment variables.
348
General Programming Concepts
Other subroutines are provided to determine the current settings for locale data
formatting. For more information about these subroutines, see “Locale
Subroutines” on page 350.
Introducing Time Formatting Subroutines
Programs that need to format or time into wide character code strings can use the
wcsftime subroutine. Programs that need to convert multibyte strings into an
internal time format can use the strptime subroutine. For more information about
these subroutines, see “Time Formatting Subroutines” on page 355.
Introducing Monetary Formatting Subroutines
Programs that need to specify or access monetary quantities can call the strfmon
subroutine. For more information about this subroutine, see “Monetary Formatting
Subroutines” on page 355.
Introducing Multibyte and Wide Character Subroutines
The external representation of data is referred to as the file coderepresentation of a
character. When file code data is created in files or transferred between a computer
and its I/O devices, a single character may be represented by one or several bytes.
For processing strings of such characters, it is more efficient to convert these codes
into a uniform-length representation. This converted form is intended for internal
processing of characters. The internal representation of data is referred to as the
process codeor wide character code representation of the character.
NLS internationalization of programs is a blend of multibyte and wide character
subroutines. A multibyte subroutineuses multibyte character sets. A wide character
subroutineuses wide character sets. Multibyte subroutines have an mb prefix. Wide
character subroutines have a wc prefix. The corresponding string-handling
subroutines are indicated by the mbs and wcs prefixes, respectively. Deciding
when to use multibyte or wide character subroutines can be made only after
careful analysis.
If a program primarily uses multibyte subroutines, it may be necessary to convert
the multibyte character codes to wide character codes to use certain wide character
subroutines. If a program uses wide character subroutines, data may need to be
converted to multibyte form for invoking subroutines. Both methods have
drawbacks, depending on the program and the availability of standard subroutines
to perform the required processing. For instance, there is no corresponding
standard multibyte subroutine for the wide character display-column-width
subroutine.
If a program can process its characters in multibyte code, this method should be
used instead of converting the characters to wide character code.
For more information about the subroutines provided for converting between
multibyte code and wide character code form, see “Multibyte Code and Wide
Character Code Conversion Subroutines” on page 358.
wchar.h Header File
The wchar.h header file declares information necessary for programming with
multibyte and wide character subroutines. The wchar.h header file declares the
wchar_t , wctype_t , and wint_t data types, as well as several functions for testing
wide characters. Because the number of characters implemented as wide characters
Chapter 16. National Language Support
349
exceeds that of basic characters, it is not possible to classify all wide characters into
the existing classes used for basic characters. Therefore, it is necessary to provide a
way of defining additional classes specific to some locale. The action of these
subroutines is affected by the current locale.
The wchar.h header file also declares subroutines for manipulating wide character
strings (that is, wchar_t data type arrays). Array length is always determined in
terms of the number of wchar_t elements in an array. A null wide character code
ends an array. A pointer to a wchar_t or void array always points to the initial
element of the array.
Note: If the number of wchar_telements in an array exceeds the defined array
length, unpredictable results can occur.
Introducing Internationalized Regular Expression Subroutines
Programs that contain internationalized regular expressions can use the regcomp,
regexec, regerror, regfree, and fnmatch subroutines. For more information about
these subroutines, see “Internationalized Regular Expression Subroutines” on
page 379.
Locale Subroutines
The locale of a process determines the way character collation, character
classification, date and time formatting, numeric punctuation, monetary
punctuation, and message output are handled. The following section describes how
to set and access information about the current locale in a program using National
Language Support (NLS).
Setting the Locale
Every internationalized program must set the current locale using the
setlocalesubroutine. This subroutine allows a process to change or query the
current locale by accessing locale databases.
When a process is started, its current locale is set to the C or POSIX locale. A
program that depends on locale data not defined in the C or POSIX locale must
invoke the setlocalesubroutine in the following manner before using any of the
locale-specific information:
setlocale(LC_ALL, "");
Accessing Locale Information
The following subroutines provide access to information defined in the current
locale as determined by the most recent call to the setlocalesubroutine:
localeconv
nl_langinfo
350
General Programming Concepts
Provides access to locale information defined in the LC_MONETARY
and LC_NUMERIC categories of the current locale. The localeconv
subroutine retrieves information about these categories, places the
information in a structure of type lconv as defined in the locale.hfile,
and returns a pointer to this structure.
Returns a pointer to a null-terminated string containing information
defined in the LC_CTYPE, LC_MESSAGES, LC_MONETARY,
LC_NUMERIC, and LC_TIME categories of the current locale.
rpmatch
Tests for positive and negative responses. These are specified in the
LC_MESSAGES category of the current locale. Responses can be
regular expressions as well as simple strings. The rpmatch subroutine
is not an industry-standard subroutine. Portable applications should
not assume that this subroutine is available.
The localeconv and nl_langinfo subroutines do not provide access to all
LC_*categories.
The current locale setting for a category can be obtained by: setlocale(Category,
(char*)0). The return value is a string specifying the current locale for Category.
The following example determines the current locale setting for the LC_CTYPE
category:
char *ctype_locale; ctype_locale =
setlocale(LC_CTYPE, (char*)0);
Examples
1. The following example uses the setlocalesubroutine to change the locale from
the default C localeto the locale specified by the environment variables,
consistent with the hierarchy of the locale environment variables:
#include <locale.h>
main()
{
char *p;
p = setlocale(LC_ALL, "");
/*
** The program will have the locale a
s set by the
** LC_* and LANG variables.
*/
}
2. The following exampleuses the setlocalesubroutine to obtain the current locale
setting for the LC_COLLATEcategory:
#include <stdio.h>
#include <locale.h>
main()
{
char *p;
/* set the current locale to what is specified */
p = setlocale(LC_ALL, "");
/* The current locale settings for all the
** categories is pointed to by p
*/
/*
** Find the current setting for the
** LC_COLLATE category
*/
p = setlocale(LC_COLLATE, NULL);
/*
** p points to a string containing the current locale
** setting for the LC_COLLATE category.
*/
}
3. The following exampleuses the setlocalesubroutine to obtain the current locale
setting and saves it for later use. This action allows the program to
temporarily change the locale to a new locale. After processing is complete,
the locale can be returned to its original state.
#include <stdio.h>
#include <locale.h>
#include <string.h>
Chapter 16. National Language Support
351
#define NEW_LOCALE "MY_LOCALE"
main()
{
char *p, *save_locale;
p = setlocale(LC_ALL, "");
/*
** Initiate locale. p points to the c
urrent locale
** setting for all the categories
*/
save_locale = (char *)malloc(strlen(p) +1);
strcpy(save_locale, p);
/* Save the current locale setting */
p = setlocale(LC_ALL, NEW_LOCALE);
/* Change to new locale */
/*
** Do processing ...
*/
/* Change back to old locale */
p = setlocale(LC_ALL, save_locale);
/* Restore old locale */
free(save_locale);
/* Free the memory */
}
4. The following exampleuses the setlocalesubroutine to set the
LC_MESSAGEScategory to the locale determined by the environment
variables. All other categories remain set to the C locale.
#include <locale.h>
main()
{
char *p;
/*
** The program starts in the C locale
for all categories.
*/
p = setlocale(LC_MESSAGES, "");
/*
** At this time the LC_COLLATE, LC_CTYPE, LC_NUMERIC,
** LC_MONETARY, LC_TIME will be in the C locale.
** LC_MESSAGES will be set to the cur
rent locale setting
** as determined by the environment variab
les.
*/
}
5. The following exampleuses the localeconvsubroutine to obtain the
decimal-point setting for the current locale:
#include <locale.h>
main()
{
struct lconv *ptr;
char *decimal;
(void)setlocale(LC_ALL, "");
ptr = localeconv();
/*
** Access the data obtained. For example,
** obtain the current decimal point setting.
*/
decimal = ptr->decimal_point;
}
6. The following exampleuses the nl_langinfosubroutine to obtain the date and
time format for the current locale:
#include <langinfo.h>
#include <locale.h>
main()
352
General Programming Concepts
{
char *ptr;
(void)setlocale(LC_ALL, "");
ptr = nl_langinfo(D_T_FMT);
}
7. The following exampleuses the nl_langinfo subroutine to obtain the radix
character for the current locale:
#include <langinfo.h>
#include <locale.h>
main()
{
char *ptr;
(void)setlocale(LC_ALL, "");
/* Set the program's locale */
ptr = nl_langinfo(RADIXCHAR);
/* Obtain the radix character*/
}
8. The following exampleuses the nl_langinfo subroutine to obtain the setting of
the currency symbol for the current locale:
#include <langinfo.h>
#include <locale.h>
main()
{
char *ptr;
(void)setlocale(LC_ALL, "");
/* Set the program's locale */
ptr = nl_langinfo(CRNCYSTR);
/* Obtain the currency string*/
/* The currency string will be "-$" in the U. S. locale. */
}
9. The following exampleuses the rpmatchsubroutine to obtain the setting of
affirmative and negative response strings for the current locale:
The affirmative and negative responses as specified in the locale database are
no longer simple strings; they can be regular expressions. For example, the
yesexpr can be the following regular expression, which will accept an upper
or lower case letter y, followed by zero or more alphabetic characters; or the
character O followed by K. Thus, yesexpr may be the following regular
expression:
([yY][:alpha:]*|OK)
The standards do not contain a subroutine to retrieve and compare this
information. You can use the AIX-specific rpmatch(const char *response)
subroutine.
#include <stdio.h>
#include <langinfo.h>
#include <locale.h>
#include <regex.h>
int rpmatch(const char *);
/*
** Returns 1 if yes response, 0 if no response,
** -1 otherwise
*/
main()
{
int ret;
char *resp;
(void)setlocale(LC_ALL, "");
do {
/*
** Obtain the response to the query for yes/no strings.
** The string pointer resp points to this response.
Chapter 16. National Language Support
353
** Check if the string is yes.
*/
ret = rpmatch(resp);
if(ret == 1){
/* Response was yes. */
/* Process accordingly. */
}else if(ret == 0){
/* Response was negative. */
/* Process negative response. */
}else if(ret<0){
/* No match with yes/no occurred. */
continue;
}
}while(ret <0);
}
10. The following example provides a method of implementing the rpmatch
subroutine. Note that most applications should use the rpmatch subroutine in
libc. The following implementation of rpmatch is just for illustration
purposes.
Note that nl_langinfo(YESEXPR) and nl_langinfo(NOEXPR) are used to
obtain the regular expressions for the affirmative and negative responses
respectively.
#include <langinfo.h>
#include <regex.h>
/*
** rpmatch() performs comparison of a string to a regular expression
** using the POSIX.2 defined regular expression compile and match
** functions. The first argument is the response from the user and the
** second string is the current locale setting of the regular expression.
*/
int rpmatch( const char *string)
{
int status;
int retval;
regex_t re;
char *pattern;
pattern = nl_langinfo(YESEXPR);
/* Compile the regular expression pointed to by pattern. */
if( ( status = regcomp( &re, pattern, REG_EXTENDED | REG_NOSUB )) !=
0 ){
retval = -2; /*-2 indicates yes expr compile error */
return(retval);
}
/* Match the string with the compiled regular expression. */
status = regexec( &re, string, (size_t)0, (regmatch_t *)NULL, 0);
if(status == 0){
retval = 1; /* Yes match found */
}else{
/* Check for negative response */
pattern = nl_langinfo(NOEXPR);
if( ( status = regcomp( &re, pattern,
REG_EXTENDED | REG_NOSUB )) != 0 ){
retval = -3;/*-3 indicates no compile error */
return(retval);
}
status = regexec( &re, string, (size_t)0,
(regmatch_t *)NULL, 0);
if(status == 0)
retval = 0;/* Negative response match found */
}else
retval = -1; /* The string did not match yes or no
response */
regfree(&re);
return(retval);
}
354
General Programming Concepts
Time Formatting Subroutines
In addition to the strftime subroutine defined in the C programming language
standard, XPG4 defines the following time formatting subroutines:
wcsftime
strptime
Formats time into wide character code strings.
Converts a multibyte string into an internal time format.
Examples
1. The following example uses the wcsftime subroutine to format time into a
wide character string:
#include <stdio.h>
#include <langinfo.h>
#include <locale.h>
#include <time.h>
main()
{
wchar_t timebuf[BUFSIZE];
time_t clock = time( (time_t*) NULL);
struct tim *tmptr = localetime(&clock);
(void)setlocale(LC_ALL, "");
wcsftime(
timebuf,
/* Time string output buffer */
BUFSIZ,
/*Maximum size of output string */
nl_langinfo(D_T_FMT),
/* Date/time format */
tmptr
/* Pointer to tm structure */
);
printf("%S\n", timebuf);
}
2. The following example uses the strptime subroutine to convert a formatted
time string to internal format:
#include <langinfo.h>
#include <locale.h>
#include <time.h>
main(int argc, char **argv)
{
struct tm tm;
(void)setlocale(LC_ALL, "");
if (argc != 2) {
...
/* Error handling */
}
if (strptime(
argv[1],
/* Formatted time string */
nl_langinfo(D_T_FMT),
/* Date/time format */
&tm
/* Address of tm structure */
) == NULL) {
...
/* Error handling */
}
else {
...
/* Other Processing */
}
}
Monetary Formatting Subroutines
Although the C programming language standard in conjunction with POSIX
provides a means of specifying and accessing monetary information, these
standards do not define a subroutine that formats monetary quantities. The XPG
strfmon subroutine provides the facilities to format monetary quantities. There is
no defined subroutine that converts a formatted monetary string into a numeric
Chapter 16. National Language Support
355
quantity suitable for arithmetic. Applications that need to do arithmetic on
monetary quantities may do so after processing the locale-dependent monetary
string into a number. The culture-specific monetary formatting information is
specified by the LC_MONETARY category. An application can obtain information
pertaining to the monetary format and the currency symbol by calling the
localeconv subroutine.
Euro Currency Support via the @euro Modifier
The strfmon subroutine uses the information from the locale’s LC_MONETARY
category to determine the correct monetary format for the given
language/territory. With the advent of the common European currency (Euro),
locales must be able to handle both the traditional national currencies as well as
the common European currency. This is accomplished via the @euro modifier. Each
European country that uses the Euro will have an additional LC_MONETARY
definition with the @euro modifier appended. This alternate format will be
invoked when specified via the locale environment variables, or with the setlocale
subroutine.
To use the French locale , UTF-8 codeset environment, and French francs as the
monetary unit, simply set:
LANG=FR_FR
To use the French locale, UTF-8 codeset environment,and Euros as the monetary
unit, set:
LANG=FR_FR
LC_MONETARY=FR_FR@euro
Users should NOT attempt to set LANG=FR_FR@euro, as the @euro variant for
locale categories other than LC_MONETARY is undefined.
Examples
1. The following example uses the strfmon subroutine and accepts a format
specification and an input value. The input value is formatted according to the
input format specification.
#include <monetary.h>
#include <locale.h>
#include <stdio.h>
main(int argc, char **argv)
{
char bfr[256], format[256];
int match; ssize_t size;
float value;
(void) setlocal(LC_ALL, "");
if (argc != 3){
...
/* Error handling */
}
match = sscanf(argv[1], "%f", &value);
if (!match) {
...
/* Error handling */
}
match = sscanf(argv[2], "%s", format);
if (!match) {
...
/*Error handling */
}
size = strfmon(bfr, 256, format, value);
if (size == -1) {
356
General Programming Concepts
...
/* Error handling */
}
printf ("Formatted monetary value is: %s\n", bfr);
}
The following table provides examples of some of the possible conversion
specifications and the outputs for 12345.67 and -12345.67 in a US English locale:
Conversion Specification
Output
Description
%n
$12,345.67 -$12,345.67
Default formatting
%15n
$12,345.67 -$12,345.67
Right justifies within a
15-character field.
%#6n
$ 12,345.67 -$ 12,345.67
Aligns columns for values up
to 999,999.
%=*#8n
$****12,345.67 -$****12,345.67
Specifies a fill character.
%=0#8n
$000012,345.67
-$000012,345.67
Fill characters do not use
grouping.
%|#6n
$ 12345.67 -$ 12345.67
Disables the thousands
separator.
%|#6.0n
$ 12346 -$ 12346
Rounds off to whole units.
%|#6.3n
$ 12345.670 -$ 12345.670
Increases the precision.
%(#6n
$ 12,345.67 ($ 12,345.67)
Uses an alternate positive or
negative style.
%!(#6n
12,345.67 ( 12,345.67)
Disables the currency
symbol.
2. The following example converts a monetary value into a numeric value. The
monetary string is pointed to by input and the result of converting it into
numeric form is stored in the string pointed to by output. Assume input and
output are initialized.
char *input; /* the input multibyte string containing the monetary string */
char *output; /* the numeric string obtained from the input string */
wchar_t src_string[SIZE], dest_string[SIZE];
wchar_t *monetary, *numeric;
wchar_t mon_decimal_point, radixchar;
wchar_t wc;
localeconv *lc;
/* Initialize input and output to point to valid buffers as appropriate. */
/* Convert the input string to process code form*/
retval = mbstowcs(src_string, input, SIZE);
/* Handle error returns */
monetary = src_string;
numeric = dest_string;
lc = localeconv();
/* obtain the LC_MONETARY and LC_NUMERIC info */
/* Convert the monetary decimal point to wide char form */
retval = mbtowc( &mon_decimal_point, lc->mon_decimal_point,
MB_CUR_MAX);
/* Handle any error case */
/* Convert the numeric decimal point to wide char form */
retval = mbtowc( &radixchar, lc->decimal_point, MB_CUR_MAX);
/* Handle error case */
/* Assuming the string is converted first into wide character
** code form via mbstowcs, monetary points to this string.
*/
/* Pick up the numeric information from the wide character
** string and copy it into a temp buffer.
*/
while(wc = *monetary++){
Chapter 16. National Language Support
357
if(iswdigit(wc))
*numeric++ = wc;
else if( wc == mon_decimal_point)
*numeric++=radixchar;
}
*numeric = 0;
/* dest_string has the numeric value of the monetary quantity. */
/* Convert the numeric quantity into multibyte form */
retval = wcstombs( output, dest_string, SIZE);
/* Handle any error returns */
/* Output contains a numeric value suitable for atof conversion. */
Multibyte and Wide Character Subroutines
This section contains information about multibyte and wide character code
subroutines.
Multibyte Code and Wide Character Code Conversion
Subroutines
The internationalized environment of National Language Support blends multibyte
and wide character subroutines. The decision of when to use wide character or
multibyte subroutines can be made only after careful analysis.
If a program primarily uses multibyte subroutines, it may be necessary to convert
the multibyte character codes to wide character codes before certain wide character
subroutines can be used. If a program uses wide character subroutines, data may
need to be converted to multibyte form when invoking subroutines. Both methods
have drawbacks, depending on the program in use and the availability of standard
subroutines to perform the required processing. For instance, the wide character
display-column-width subroutine has no corresponding standard multibyte
subroutine.
If a program can process its characters in multibyte form, this method should be
used instead of converting the characters to wide character form.
Attention: The conversion between multibyte and wide character code
depends on the current locale setting. Do not exchange wide character codes
between two processes, unless you have knowledge that each locale that
might used handles wide character codes in a consistent fashion. Most AIX
locales use the Unicode character value as a wide character code, except
locales based on the IBM-850 and IBM-eucTW codesets.
Multibyte Code to Wide Character Code Conversion Subroutines
The following subroutines are used when converting from multibyte code to wide
character code:
mblen
mbstowcs
mbtowc
Determines the length of a multibyte character.
Converts a multibyte string to a wide character string.
Converts a multibyte character to a wide character.
Wide Character Code to Multibyte Code Conversion Subroutines
The following subroutines are used when converting from wide character code to
multibyte character code:
wcslen
358
General Programming Concepts
Determines the number of wide characters in a wide character string.
wcstombs
wctomb
Converts a wide character string to a multibyte character string.
Converts a wide character to a multibyte character.
Examples
1.
The following example uses the mbtowc subroutine to convert a character in
multibyte character code to wide character code:
main()
{
char
*s;
wchar_t wc;
int
n;
(void)setlocale(LC_ALL,"");
/*
** s points to the character string that needs to be
** converted to a wide character to be stored in wc.
*/
n = mbtowc(&wc, s, MB_CUR_MAX);
if (n == -1){
/* Error handle */
}
if (n == 0){
/* case of name pointing to null */
}
/*
** wc contains the process code for the multibyte character
** pointed to by s.
*/
}
2. The following example uses the wctomb subroutine to convert a character in
wide character code to multibyte character code:
#include
#include
#include
main()
{
char
<stdlib.h>
<limits.h>
<stdlib.h>
s[MB_LEN_MAX};
/* for MB_LEN_MAX */
/* for wchar_t
*/
/* system wide maximum number of
** bytes in a multibyte character r. */
wchar_t wc;
int
n;
(void)setlocale(LC_ALL,"");
/*
** wc is the wide character code to be converted to
** multibyte character code.
*/
n = wctomb(s, wc);
if(n == -1){
/* pwcs does not point to a valid wide character */
}
/*
** n has the number of bytes contained in the multibyte
** character stored in s.
*/
}
3. The following example uses the mblen subroutine to find the byte length of a
character in multibyte character code:
#include <stdlib.h>
#include <locale.h>
main
{
char *name = "h";
int n;
(void)setlocale(LC_ALL,"");
n = mblen(name, MB_CUR_MAX);
Chapter 16. National Language Support
359
/*
** The count returned in n is the multibyte length.
** It is always less than or equal to the value of
** MB_CUR_MAX in stdlib.h
*/
if(n == -1){
/* Error Handling */
}
}
4. The following example obtains a previous character position in a multibyte
string. If you need to determine the previous character position, starting from a
current character position (not just some random byte position), step through
the buffer starting at the beginning. Use the mblen subroutine until the current
character position is reached and save the previous character position to obtain
the needed character position.
char buf[];
/* contains the multibyte string */
char *cur,
/* points to the current character position */
char *prev,
/* points to previous multibyte character */
char *p;
/* moving pointer */
/* initialize the buffer and pointers as needed */
/* loop through the buffer until the moving pointer reaches
** the current character position in the buffer, always
** saving the last character position in prev pointer */
p = prev = buf;
/* cur points to a valid character somewhere in buf */
while(p< cur){
prev = p;
if( (i=mblen(p, mbcurmax))<=0){
/* invalid multibyte character or null */
/* You can have a different error handling
** strategy */
p++;
/* skip it */
}else {
p += i;
}
}
/* prev will point to the previous character position */
/* Note that if( prev == cur), then it means that there was
** no previous character. Also, if all bytes up to the
** current character are invalid, it will treat them as
** all valid single-byte characters and this may not be what
** you want. One may change this to handle another method of
** error recovery. */
5. The following example uses of the mbstowcs subroutine to convert a multibyte
string to wide character string:
#include <stdlib.h>
#include <locale.h>
main()
{
char
*s;
wchar_t *pwcs;
size_t retval, n;
(void)setlocale(LC_ALL, "");
n = strlen(s) + 1;
/*string length + terminating null */
/* Allocate required wchar array
*/
pwcs = (wchar_t *)malloc(n * sizeof(wchar_t) );
retval = mbstowcs(pwcs, s, n);
if(retval == -1){
/* Error handle */
}
/*
** pwcs contains the wide character string.
*/
}
360
General Programming Concepts
6. The following example illustrates the problems with using the mbstowcs
subroutine on a large block of data for conversion to wide character form.
When it encounters an invalid multibyte, the mbstowcs subroutine returns a
value of -1 but does not specify where the error occurred. Therefore, the
mbtowc subroutine must be used repeatedly to convert one character at a time
to wide character code.
Note: Processing in this manner will considerably slow down program
performance.
During the conversion of single-byte code sets, there is no possibility for partial
multibytes. However, during the conversion of multibyte code sets, partial
multibytes are copied to a save buffer. During the next call to the read
subroutine, the partial multibyte is prefixed to the rest of the byte sequence.
Note: A null-terminated wide character string is obtained. Optional error
handling can be done if an instance of an invalid byte sequence is found.
#include <stdio.h>
#include <locale.h>
#include <stdlib.h>
main(int argc, char *argv[])
{
char
*curp, *cure;
int
bytesread, bytestoconvert, leftover;
int
invalid_multibyte, mbcnt, wcnt;
wchar_t *pwcs;
wchar_t wbuf[BUFSIZ+1];
char
buf[BUFSIZ+1];
char
savebuf[MB_LEN_MAX];
size_t mb_cur_max;
int
fd;
/*
** MB_LEN_MAX specifies the system wide constant for
** the maximum number of bytes in a multibyte character.
*/
(void)setlocale(LC_ALL, "");
mb_cur_max = MB_CUR_MAX;
fd = open(argv[1], 0);
if(fd < 0){
/* error handle */
}
leftover = 0;
if(mb_cur_max==1){
/* Single byte code sets case */
for(;;){
bytesread = read(fd, buf, BUSIZ);
if(bytesread <= 0)
break;
mbstowcs(wbuf, buf, bytesread+1);
/* Process using the wide character buffer */
}
/* File processed ... */
exit(0);
/* End of program
*/
}else{
/* Multibyte code sets */
leftover = 0;
for(;;) {
if(leftover)
strncpy(buf, savebuf ,leftover);
bytesread=read(fd,buf+leftover, BUFSIZ-leftover);
if(bytesread <= 0)
break;
buf[leftover+bytesread] = '\0';
/* Null terminate string */
invalid_multibyte = 0;
Chapter 16. National Language Support
361
bytestoconvert = leftover+bytesread;
cure= buf+bytestoconvert;
leftover=0;
pwcs = wbuf;
/* Stop processing when invalid mbyte found. */
curp= buf;
for(;curp<cure;){
mbcnt = mbtowc(pwcs,curp, mb_cur_max);
if(mbcnt>0){
curp += mbcnt;
pwcs++;
continue;
}else{
/* More data needed on next read*/
if ( cure-curp<mb_cur_max){
leftover=cure-curp;
strncpy(savebuf,curp,leftover);
/* Null terminate before partial mbyte */
*curp=0;
break;
}else{
/*Invalid multibyte found */
invalid_multibyte =1;
break;
}
}
}
if(invalid_multibyte){
/*error handle */
}
/* Process the wide char buffer */
}
}
}
7. The following example uses the wcstombs and wcslen subroutines to convert a
wide character string to multibyte form:
#include <stdlib.h>
#include <locale.h>
main()
{
wchar_t *pwcs; /* Source wide character string */
char *s;
/* Destination multibyte character string */
size_t n;
size_t retval;
(void)setlocale(LC_ALL, "");
/*
** Calculate the maximum number of bytes needed to
** store the wide character buffer in multibyte form in the
** current code page and malloc() the appropriate storage,
** including the terminating null.
*/
s = (char *) malloc( wcslen(pwcs) * MB_CUR_MAX + 1 );
retval= wcstombs( s, pwcs, n);
if( retval == -1) {
/* Error handle */
/* s points to the multibyte character string. */
}
Wide Character Classification Subroutines
The majority of wide character classification subroutines are similar to traditional
character classification subroutines, except that wide character classification
subroutines operate on a wchar_t data type argument passed as a wint_t data type
argument.
362
General Programming Concepts
Generic Wide Character Classification Subroutines
In the internationalized environment of National Language Support, the ability to
create new character class properties is essential. For example, several properties
are defined for Japanese characters that are not applicable to the English language.
As more languages are supported, a framework enabling applications to deal with
a varying number of character properties is needed. The wctype and iswctype
subroutines allow handling of character classes in a general fashion. These
subroutines are used to allow for both user-defined and language-specific character
classes.
The action of wide character classification subroutines is affected by the definitions
in the LC_CTYPE category for the current locale.
To create new character classifications for use with the wctype and iswctype
subroutines, create a new character class in the LC_CTYPE category and generate
the locale using the localedef command. A user application obtains this locale data
with the setlocale subroutine. The program can then access the new classification
subroutines by using the wctype subroutine to get the wctype_t property handle. It
then passes to the iswctype subroutine both the property handle and the wide
character code of the character to be tested.
wctype
iswctype
Obtains handle for character property classification.
Tests for character property.
Standard Wide Character Classification Subroutines
The isw* subroutines determine various aspects of a standard wide character
classification. The isw* subroutines also work with single-byte code sets. The isw*
subroutines should be used in preference to the wctype and iswctype subroutines.
The wctype and iswctype subroutines should be used only for extended character
class properties (for example, Japanese language properties).
When using the wide character functions to convert the case in several blocks of
data, the application must convert characters from multibyte to wide character
code form. Since this may affect performance in single-byte code set locales, you
should consider providing two conversion paths in your application. The
traditional path for single-byte code set locales would convert case using the
isupper,islower, toupper, and tolower subroutines. The alternate path for
multibyte code set locales would convert multibyte characters to wide character
code form and convert case using the iswupper, iswlower, towupper and
towlower subroutines. When converting multibyte characters to wide character
code form, an application needs to handle special cases where a multibyte
character may split across successive blocks.
iswalnum
iswalpha
iswcntrl
iswdigit
iswgraph
iswlower
iswprint
iswpunct
iswspace
iswupper
iswxdigit
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
for
for
for
for
for
for
for
for
for
for
for
alphanumeric character classification.
alphabetic character classification.
control character classification.
digit character classification.
graphic character classification.
lowercase character classification.
printable character classification.
punctuation character classification.
space character classification.
uppercase character classification.
hexadecimal-digit character classification.
Chapter 16. National Language Support
363
Wide Character Case Conversion Subroutines
The following subroutines convert cases for wide characters. The action of wide
character case conversion subroutines is affected by the definition in the
LC_CTYPE category for the current locale.
towlower
towupper
Converts an uppercase wide character to a lowercase wide character.
Converts a lowercase wide character to an uppercase wide character.
Example
The following example uses the wctype subroutine to test for the NEW_CLASS
character classification:
#include <ctype.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wint_t
wc;
int
retval;
wctype_t chandle;
(void)setlocale(LC_ALL,"");
/*
** Obtain the character property handle for the NEW_CLASS
** property.
*/
chandle = wctype("NEW_CLASS") ;
if(chandle == (wctype_t)0){
/* Invalid property. Error handle. */
}
/* Let wc be the wide character code for a character */
/* Test if wc has the property of NEW_CLASS */
retval = iswctype( wc, chandle );
if( retval > 0 ) {
/*
** wc has the property NEW_CLASS.
*/
}else if(retval == 0) {
/*
** The character represented by wc does not have the
** property NEW_CLASS.
*/
}
}
Wide Character Display Column Width Subroutines
When characters are displayed or printed, the number of columns occupied by a
character may differ. For example, a Kanji character (Japanese language) may
occupy more than one column position. The number of display columns required
by each character is part of the National Language Support locale database. The
LC_CTYPEcategory defines the number of columns needed to display a character.
There are no standard multibyte display-column-width subroutines. For portability,
convert multibyte codes to wide character codes and use the required wide
character display-width subroutines. However, if the __max_disp_width macro
(defined in the stdlib.h file) is set to 1 and a single-byte code set is in use, then the
display-column widths of all characters (except tabs) in the code set are the same,
and are equal to 1. In this case, the strlen (string) subroutine gives the display
column width of the specified string. This is demonstrated in the following
example:
364
General Programming Concepts
#include <stdlib.h>
int display_column_width; /* number of display columns */
char *s;
/* character string
*/
....
if((MB_CUR_MAX == 1) && (__max_disp_width == 1)){
display_column_width = strlen(s);
/* s is a string pointer
*/
}
The following subroutines find the display widths for wide character strings:
wcswidth
wcwidth
Determines the display width of a wide character string.
Determines the display width of a wide character.
Examples
1. The following example uses the wcwidth subroutine to find the display column
width of a wide character:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wint_t wc;
int
retval;
(void)setlocale(LC_ALL, "");
/*
**
Let wc be the wide character whose display width is
**
to be found.
*/
retval = wcwidth(wc);
if(retval == -1){
/*
** Error handling. Invalid or nonprintable
** wide character in wc.
*/
}
}
2. The following example uses the wcswidth subroutine to find the display
column width of a wide character string:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs;
int
retval;
size_t n;
(void)setlocale(LC_ALL, "");
/*
**
Let pwcs point to a wide character null
**
terminated string.
**
Let n be the number of wide characters
**
whose display column width is to be determined.
*/
retval = wcswidth(pwcs, n);
if(retval == -1){
/*
** Error handling. Invalid wide or nonprintable
** character ode encountered in the wide
** character string pwcs.
*/
}
}
Chapter 16. National Language Support
365
Multibyte and Wide Character String Collation Subroutines
Strings can be compared in two ways:
v Using the ordinal (binary) values of the characters.
v Using the weights associated with the characters for each locale, as determined
by the LC_COLLATE category.
National Language Support (NLS) uses the second method.
Collation is a locale-specific property of characters. A weight is assigned to each
character to indicate its relative order for sorting. A character may be assigned
more than one weight. Weights are prioritized as primary, secondary, tertiary, and
so forth. The maximum number of weights assigned each character is
system-defined.
A process inherits the C locale or POSIX locale at its startup time. When the
setlocale (LC_ALL, ″ ″) subroutine is called, a process obtains its locale based on
the LC_* and LANG environment variables. The following subroutines are affected
by the LC_COLLATE category and determine how two strings will be sorted in
any given locale.
Note: Collation-based string comparisons take a long time because of the
processing involved in obtaining the collation values. Such comparisons
should be used only when necessary. If you need to find whether two wide
character strings are equal, do not use the wcscoll and wcsxfrm subroutines.
Use the wcscmp subroutine instead.
The following subroutines compare multibyte character strings:
strcoll
strxfrm
Compares the collation weights of multibyte character strings.
Converts a multibyte character string to values representing character collation
weights.
The following subroutines compare wide character strings:
wcscoll
wcsxfrm
Compares the collation weights of wide character strings.
Converts a wide character string to values representing character collation
weights.
Examples
1. The following example uses the wcscoll subroutine to compare two wide
character strings based on their collation weights:
#include <stdio.h>
#include <string.h>
#include <locale.h>
#include <stdlib.h>
extern int errno;
main()
{
wchar_t *pwcs1, *pwcs2;
size_t n;
(void)setlocale(LC_ALL, "");
/*
set it to zero for checking errors on wcscoll
*/
errno = 0;
/*
**
Let pwcs1 and pwcs2 be two wide character strings to
**
compare.
*/
366
General Programming Concepts
n = wcscoll(pwcs1, pwcs2);
/*
**
If errno is set then it indicates some
**
collation error.
*/
if(errno != 0){
/* error has occurred... handle error ...*/
}
}
2. The following example uses the wcsxfrm subroutine to compare two wide
character strings based on collation weights:
Note: Determining the size n (where n is a number) of the transformed
string, when using the wcsxfrm subroutine, can be accomplished in one of
the following ways:
a. For each character in the wide character string, the number of bytes for
possible collation values cannot exceed the COLL_WEIGHTS_MAX *
sizeof(wchar_t) value. This value, multiplied by the number of wide
character codes, gives the buffer length needed. To the buffer length add 1
for the terminating wide character null. This strategy may slow down
performance.
b. Estimate the byte-length needed. If the previously obtained value is not
enough, increase it. This may not satisfy all strings but gives maximum
performance.
c. Call the wcsxfrm subroutine twice: once to find the value of n, and again to
transform the string using this n value. This strategy slows down
performance because the wcsxfrm subroutine is called twice. However, it
yields a precise value for the buffer size needed to store the transformed
string.
Which method to choose depends on the characteristics of the strings used in
the program and the performance objectives of the program.
#include <stdio.h>
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, *pwcs2, *pwcs3, *pwcs4;
size_t n, retval;
(void)setlocale(LC_ALL, "");
/*
** Let the string pointed to by pwcs1 and pwcs3 be the
** wide character arrays to store the transformed wide
** character strings. Let the strings pointed to by pwcs2
** and pwcs4 be the wide character strings to compare based
** on the collation values of the wide characters in these
** strings.
** Let n be large enough (say,BUFSIZ) to transform the two
** wide character strings specified by pwcs2 and pwcs4.
**
** Note:
** In practice, it is best to call wcsxfrm if the wide
** character string is to be compared several times to
** different wide character strings.
*/
do {
retval = wcsxfrm(pwcs1, pwcs2, n);
if(retval == (size_t)-1){
/* error has occurred. */
/* Process the error if needed */
Chapter 16. National Language Support
367
break;
}
if(retval >= n ){
/*
** Increase the value of n and use a bigger buffer pwcs1.
*/
}
}while (retval >= n);
do {
retval = wcsxfrm(pwcs3, pwcs4, n);
if (retval == (size_t)-1){
/* error has occurred. */
/* Process the error if needed */
break;
if(retval >= n){
/*Increase the value of n and use a bigger buffer pwcs3.*/
}
}while (retval >= n);
retval = wcscmp(pwcs1, pwcs3);
/* retval has the result */
}
Multibyte and Wide Character String Comparison Subroutines
The strcmp and strncmp subroutines determine if the contents of two multibyte
strings are equivalent. If your application needs to know how the two strings differ
lexically, use the multibyte and wide character string collation subroutines.
The following NLS subroutines compare wide character strings:
wcscmp
wcsncmp
Compares two wide character strings.
Compares a specific number of wide character strings.
Example
The following example uses the wcscmp subroutine to compare two wide
character strings:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, *pwcs2;
int retval;
(void)setlocale(LC_ALL, "");
/*
** pwcs1 and pwcs2 point to two wide character
** strings to compare.
*/
retval = wcscmp(pwcs1, pwcs2);
/* pwcs1 contains a copy of the wide character string
** in pwcs2
*/
}
Wide Character String Conversion Subroutines
The following NLS subroutines convert wide character strings to double, long, and
unsigned long integers:
wcstod
wcstol
wcstoul
368
General Programming Concepts
Converts a wide character string to a double-precision floating point.
Converts a wide character string to a signed long integer.
Converts a wide character string to an unsigned long integer.
Before calling the wcstod, wcstoul, or wcstol subroutine, the errno global variable
must be set to 0. Any error that occurs as a result of calling these subroutines can
then be handled correctly.
Examples
1. The following example uses the wcstod subroutine to convert a wide character
string to a double-precision floating point:
#include <stdlib.h>
#include <locale.h>
#include <errno.h>
extern int errno;
main()
{
wchar_t *pwcs, *endptr;
double retval;
(void)setlocale(LC_ALL, "");
/*
** Let pwcs point to a wide character null terminated
** string containing a floating point value.
*/
errno = 0;
/* set errno to zero */
retval = wcstod(pwcs, &endptr);
if(errno != 0){
/* errno has changed, so error has occurred */
if(errno == ERANGE){
/* correct value is outside range of
** representable values. Case of overflow
** error
*/
if((retval == HUGE_VAL) ||
(retval == -HUGE_VAL)){
/* Error case. Handle accordingly. */
}else if(retval == 0){
/* correct value causes underflow */
/* Handle appropriately
*/
}
}
}
/* retval contains the double. */
}
2. The following example uses the wcstol subroutine to convert a wide character
string to a signed long integer:
#include <stdlib.h>
#include <locale.h>
#include <errno.h>
#include <stdio.h>
extern int errno;
main()
{
wchar_t *pwcs, *endptr;
long int retval;
(void)setlocale(LC_ALL, "");
/*
** Let pwcs point to a wide character null terminated
** string containing a signed long integer value.
*/
errno = 0; /* set errno to zero */
retval = wcstol(pwcs, &endptr, 0);
if(errno != 0){
/* errno has changed, so error has occurred */
if(errno == ERANGE){
/* correct value is outside range of
** representable values. Case of overflow
** error
Chapter 16. National Language Support
369
*/
if((retval == LONG_MAX) || (retval == LONG_MIN)){
/* Error case. Handle accordingly.
*/
}else if(errno == EINVAL){
/* The value of base is not supported */
/* Handle appropriately
*/
}
}
}
/* retval contains the long integer.
*/
}
3. The following example uses the wcstoul subroutine to convert a wide character
string to an unsigned long integer:
#include <stdlib.h>
#include <locale.h>
#include <errno.h>
extern int errno;
main()
{
wchar_t
*pwcs, *endptr;
unsigned long int retval;
(void)setlocale(LC_ALL, "");
/*
** Let pwcs point to a wide character null terminated
** string containing an unsigned long integer value.
*/
errno = 0; /* set errno to zero */
retval = wcstoul(pwcs, &endptr, 0);
if(errno != 0){
/* error has occurred */
if(retval == ULONG_MAX || errno == ERANGE){
/*
** Correct value is outside of
** representable value. Handle appropriately
*/
}else if(errno == EINVAL){
/* The value of base is not representable */
/* Handle appropriately
*/
}
}
/* retval contains the unsigned long integer. */
}
Wide Character String Copy Subroutines
The following NLS subroutines copy wide character strings:
wcscpy
wcsncpy
wcscat
wcsncat
Copies a wide character string to another wide character string.
Copies a specific number of characters from a wide character string to another
wide character string.
Appends a wide character string to another wide character string.
Appends a specific number of characters from a wide character string to another
wide character string.
Example
The following example uses the wcscpy subroutine to copy a wide character string
into a wide character array:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, *pwcs2;
370
General Programming Concepts
size_t n;
(void)setlocale(LC_ALL, "");
/*
** Allocate the required wide character array.
*/
pwcs1 = (wchar_t *)malloc( (wcslen(pwcs2) +1)*sizeof(wchar_t));
wcscpy(pwcs1, pwcs2);
/*
** pwcs1 contains a copy of the wide character string in pwcs2
*/
}
Wide Character String Search Subroutines
The following NLS subroutines are used to search for wide character strings:
wcschr
wcsrchr
wcspbrk
wcsspn
wcscspn
wcswcs
wcstok
Searches for the first occurrence of a wide character in a wide character string.
Searches for the last occurrence of a wide character in a wide character string.
Searches for the first occurrence of a several wide characters in a wide character
string.
Determines the number of wide characters in the initial segment of a wide
character string.
Searches for a wide character string.
Searches for the first occurrence of a wide character string within another wide
character string.
Breaks a wide character string into a sequence of separate wide character
strings.
Examples
1. The following example uses the wcschr subroutine to locate the first occurrence
of a wide character in a wide character string:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, wc, *pws;
int
retval;
(void)setlocale(LC_ALL, "");
/*
** Let pwcs1 point to a wide character null terminated string.
** Let wc point to the wide character to search for.
**
*/
pws = wcschr(pwcs1, wc);
if (pws == (wchar_t )NULL ){
/* wc does not occur in pwcs1 */
}else{
/* pws points to the location where wc is found */
}
}
2. The following example uses the wcsrchr subroutine to locate the last occurrence
of a wide character in a wide character string:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, wc, *pws;
int
retval;
(void)setlocale(LC_ALL, "");
/*
Chapter 16. National Language Support
371
** Let pwcs1 point to a wide character null terminated string.
** Let wc point to the wide character to search for.
**
*/
pws = wcsrchr(pwcs1, wc);
if (pws == (wchar_t )NULL ){
/* wc does not occur in pwcs1 */
}else{
/* pws points to the location where wc is found */
}
}
3. The following example uses the wcspbrk subroutine to locate the first
occurrence of several wide characters in a wide character string:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, *pwcs2, *pws;
(void)setlocale(LC_ALL, "");
/*
** Let pwcs1 point to a wide character null terminated string.
** Let pwcs2 be initialized to the wide character string
** that contains wide characters to search for.
*/
pws = wcspbrk(pwcs1, pwcs2);
if (pws == (wchar_t )NULL ){
/* No wide character from pwcs2 is found in pwcs1
*/
}else{
/* pws points to the location where a match is found */
}
}
4. The following example uses the wcsspn subroutine to determine the number of
wide characters in the initial segment of a wide character string segment:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, *pwcs2;
size_t count;
(void)setlocale(LC_ALL, "");
/*
** Let pwcs1 point to a wide character null terminated string.
** Let pwcs2 be initialized to the wide character string
** that contains wide characters to search for.
*/
count = wcsspn(pwcs1, pwcs2);
/*
** count contains the length of the segment.
*/
}
5. The following example uses the wcscspn subroutine to determine the number
of wide characters not in a wide character string segment:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, *pwcs2;
size_t count;
(void)setlocale(LC_ALL, "");
/*
** Let pwcs1 point to a wide character null terminated string.
372
General Programming Concepts
** Let pwcs2 be initialized to the wide character string
** that contains wide characters to search for.
*/
count = wcscspn(pwcs1, pwcs2);
/*
** count contains the length of the segment consisting
** of characters not in pwcs2.
*/
}
6. The following example uses the wcswcs subroutine to locate the first
occurrence of a wide character string within another wide character string:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1, *pwcs2, *pws;
(void)setlocale(LC_ALL, "");
/*
** Let pwcs1 point to a wide character null terminated string.
** Let pwcs2 be initialized to the wide character string
** that contains wide characters sequence to locate.
*/
pws = wcswcs(pwcs1, pwcs2);
if (pws == (wchar_t)NULL){
/* wide character sequence pwcs2 is not found in pwcs1 */
}else{
/*
** pws points to the first occurrence of the sequence
** specified by pwcs2 in pwcs1.
*/
}
}
7. The following example uses the wcstok subroutine to tokenize a wide character
string:
#include <string.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wchar_t *pwcs1 = L"?a???b,,,#c";
wchar_t *pwcs;
(void)setlocale(LC_ALL, "");
pwcs = wcstok(pwcs1, L"?");
/* pws points to the token: L"a" */
pwcs = wcstok((wchar_t *)NULL, L",");
/* pws points to the token: L"??b" */
pwcs = wcstok((wchar_t *)NULL, L"#,");
/* pws points to the token: L"c" */
}
Wide Character Input/Output Subroutines
NLS provides subroutines for both formatted and unformatted I/O.
Formatted Wide Character I/O
Additions to the printf and scanf family of subroutines allow for the formatting of
wide characters. The printf and scanf subroutines have two additional format
specifiers for wide character handling: %C and %S. The %C and %S format
Chapter 16. National Language Support
373
specifiers allow I/O on a wide character and a wide character string, respectively.
They are similar to the %c and %s format specifiers, which allow I/O on a
multibyte character and string.
The multibyte subroutines accept a multibyte array and output a multibyte array.
To convert multibyte output from a multibyte subroutine to a wide character
string, use the mbstowcs subroutine.
Unformatted Wide Character I/O
Unformatted wide character I/O subroutines are used when a program requires
code set-independent I/O for characters from multibyte code sets. For example, the
fgetwc or getwc subroutine should be used to input a multibyte character. If the
program uses the getc subroutine to input a multibyte character, the program must
call the getc subroutine once for each byte in the multibyte character.
Wide character input subroutines read multibyte characters from a stream and
convert them to wide characters. The conversion is done as if the subroutines call
the mbtowc and mbstowcs subroutines.
Wide character output subroutines convert wide characters to multibyte characters
and write the result to the stream. The conversion is done as if the subroutines call
the wctomb and wcstombs subroutines.
The behavior of wide character I/O subroutines is affected by the LC_CTYPE
category of the current locale.
Reading and Processing an Entire File: If a program has to go through an entire
file that must be handled in wide character code form, it can be done in one of the
following ways:
v In the case of multibyte characters, use either the read or fread subroutine to
convert a block of text data into a buffer. Convert one character at a time in this
buffer using the mbtowc subroutine. Handle special cases of multibyte
characters crossing block boundaries. For multibyte code sets, do not use the
mbstowcs subroutine on this buffer. On an invalid or a partial multibyte
character sequence, the mbstowcs subroutine returns -1 without indicating how
far it successfully converted the data. You can use the mbstowcs subroutine with
single-byte code sets because you will not run into a partial-byte sequence
problem with single-byte code sets.
v Use the fgetws subroutine to obtain a line from the file. If the returned wide
character string contains a wide character <new-line>, then a complete line is
obtained. If there is no <new-line> wide character, it means that the line is
longer than expected, and more calls to the fgetws subroutine are needed to
obtain the complete line. If the program can efficiently process one line at a time,
this approach is recommended.
v If the fgets subroutine is used to read a multibyte file to obtain one line at a
time, a split multibyte character may result. This condition needs to be handled
just as in the case of the read subroutine breaking up a multibyte character
across successive reads. If you can guarantee that the input line length is not
more than a set limit, a buffer of that size (plus 1 for null) can be used, thereby
avoiding the possibility of a split multibyte character. If the program can
efficiently process one line at a time, this approach may be used. Because of the
possibility of split bytes in the buffer, you should use the fgetws subroutine in
preference to the fgets subroutine for multibyte characters.
374
General Programming Concepts
v Use the fgetwc subroutine on the file to read one wide character code at a time.
If a file is large, the function call overhead becomes large and reduces the value
of this method.
The decision of which one of these methods to use should be made on a per
program basis. The second option is recommended, as it is capable of high
performance and the program does not have to handle the special cases.
Input Subroutines: A new data type, wint_t,is required to represent the wide
character code value as well as the end-of-file (EOF) marker. For example, consider
the case of the fgetwc subroutine, which returns a wide character code value:
If the wchar_t data type is defined as a char value, the y-umlaut
symbol cannot be distinguished from the end-of-file (EOF)
marker in the ISO8859-1 code set. The 0xFF code point is a valid
character (y umlaut). Hence, the return value cannot be the
wchar_t data type. A data type is needed that can hold both the
EOF marker and all the code points in a code set.
On some machines, the int data type is defined to be 16 bits.
When the wchar_t data type is larger than 16 bits, the int value
cannot represent all the return values.
wchar_t fgetwc();
int fgetwc();
Due to these reasons, wint_t data type is needed to represent the fgetwc
subroutine return value. The wint_t data type is defined in the wchar.h file.
The following subroutines are used for wide character input:
fgetwc
fgetws
getwc
getwchar
getws
ungetwc
Gets next wide character from a stream.
Gets a string of wide characters from a stream.
Gets next wide character from a stream.
Gets next wide character from standard input.
Gets a string of wide characters from a standard input.
Pushes a wide character onto a stream.
Output Subroutines: The following subroutines are used for wide character
output:
fputwc
fputws
putwc
putwchar
putws
Writes
Writes
Writes
Writes
Writes
a
a
a
a
a
wide
wide
wide
wide
wide
character
character
character
character
character
to an output stream.
string to an output stream.
to an output stream.
to standard output.
string to standard output.
Examples
1. The following example uses the fgetwc subroutine to read wide character codes
from a file:
#include <stdio.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wint_t retval;
FILE
*fp;
wchar_t *pwcs;
(void)setlocale(LC_ALL, "");
/*
** Open a stream.
*/
fp = fopen("file", "r");
/*
Chapter 16. National Language Support
375
** Error Handling if fopen was not successful.
*/
if(fp == NULL){
/* Error handler */
}else{
/*
** pwcs points to a wide character buffer of BUFSIZ.
*/
while((retval = fgetwc(fp)) != WEOF){
*pwcs++ = (wchar_t)retval;
/* break when buffer is full */
}
}
/* Process the wide characters in the buffer */
}
2. The following example uses the getwchar subroutine to read wide characters
from standard input:
#include <stdio.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wint_t retval;
FILE
*fp;
wchar_t *pwcs;
(void)setlocale(LC_ALL, "");
index = 0;
while((retval = getwchar()) != WEOF){
/* pwcs points to a wide character buffer of BUFSIZ. */
*pwcs++ = (wchar_t)retval;
/* break on buffer full */
}
/* Process the wide characters in the buffer */
}
3. The following example uses the ungetwc subroutine to push a wide character
onto an input stream:
#include <stdio.h>
#include <locale.h>
#include <stdlib.h>
main()
{
wint_t retval;
FILE
*fp;
(void)setlocale(LC_ALL, "");
/*
** Open a stream.
*/
fp = fopen("file", "r");
/*
** Error Handling if fopen was not successful.
*/
if(fp == NULL){
/* Error handler */
else{
retval = fgetwc(fp);
if(retval != WEOF){
/*
** Peek at the character and return it to the stream.
*/
retval = ungetwc(retval, fp);
if(retval == EOF){
/* Error on ungetwc */
376
General Programming Concepts
}
}
}
}
4. The following example uses the fgetws subroutine to read a file one line at a
time:
#include <stdio.h>
#include <locale.h>
#include <stdlib.h>
main()
{
FILE
*fp;
wchar_t *pwcs;
(void)setlocale(LC_ALL, "");
/*
** Open a stream.
*/
fp = fopen("file", "r");
/*
** Error Handling if fopen was not successful.
*/
if(fp == NULL){
/* Error handler */
}else{
/* pwcs points to wide character buffer of BUFSIZ. */
while(fgetws(pwcs, BUFSIZ, fp) != (wchar_t *)NULL){
/*
** pwcs contains wide characters with null
** termination.
*/
}
}
}
5. The following example uses the fputwc subroutine to write wide characters to
an output stream:
#include <stdio.h>
#include <locale.h>
#include <stdlib.h>
main()
{
int
index, len;
wint_t retval;
FILE
*fp;
wchar_t *pwcs;
(void)setlocale(LC_ALL, "");
/*
** Open a stream.
*/
fp = fopen("file", "w");
/*
** Error Handling if fopen was not successful.
*/
if(fp == NULL){
/* Error handler */
}else{
/* Let len indicate number of wide chars to output.
** pwcs points to a wide character buffer of BUFSIZ.
*/
for(index=0; index < len; index++){
retval = fputwc(*pwcs++, fp);
if(retval == WEOF)
break; /* write error occurred */
Chapter 16. National Language Support
377
/*
errno is set to indicate the error. */
}
}
}
6. The following example uses the fputws subroutine to write a wide character
string to a file:
#include <stdio.h>
#include <locale.h>
#include <stdlib.h>
main()
{
int
retval;
FILE
*fp;
wchar_t *pwcs;
(void)setlocale(LC_ALL, "");
/*
** Open a stream.
*/
fp = fopen("file", "w");
/*
** Error Handling if fopen was not successful.
*/
if(fp == NULL){
/* Error handler */
}else{
/*
** pwcs points to a wide character string
** to output to fp.
*/
retval = fputws(pwcs, fp);
if(retval == -1){
/* Write error occurred
*/
/* errno is set to indicate the error */
}
}
}
Working with the Wide Character Constant
Use the L constant for ASCII characters only. For ASCII characters, the L constant
value is numerically the same as the code point value of the character. For
example, L’a’ is same as a. The reason for using the L constant is to obtain the
wchar_t value of an ASCII character for assignment purposes. A wide character
constant is introduced by the L specifier. For example:
wchar_t wc = L'x' ;
A wide character code corresponding to the character x is stored in wc. The C
compiler converts the character x using the mbtowc or mbstowcs subroutine as
appropriate. This conversion to wide characters is based on the current locale
setting at compile time. Because ASCII characters are part of all supported code
sets and the wide character representation of all ASCII characters is the same in all
locales, L’x’ results in the same value across all code sets. However, if the character
x is non-ASCII, the program may not work when it is run on a different code set
than used at compile time. This limitation impacts some programs that use switch
statements using the wide character constant representation.
See the following partial program example, compiled using the IBM-850 code set.
378
General Programming Concepts
wchar_t wc;
switch(wc){
case L'ä': /* a-umlaut */
/* Process */
break;
case L'ç': /* c-cedilla */
/* Process */
break;
default:
break;
}
If this program is compiled and executed on an IBM-850 code set system, it will
run correctly. However, if the same executable is run on an ISO8859-1 system, it
may not work correctly. The characters a-umlaut and c-cedilla may have different
process codes in IBM-850 and ISO8859-1 code sets.
Internationalized Regular Expression Subroutines
The following subroutines are available for use with internationalized regular
expressions.
regcomp
regexec
regerror
regfree
fnmatch
Compiles a specified basic or extended regular expression into an executable
string.
Compares a null-terminated string with a compiled basic or extended regular
expression that must have been previously compiled by a call to the regcomp
subroutine.
Provides a mapping from error codes returned by the regcomp and regexec
subroutines to printable strings.
Frees any memory allocated by the regcomp subroutine associated with the
compiled basic or extended regular expression. The expression is no longer
treated as a compiled basic or extended regular expression after it is given to
the regfree subroutine.
Checks a specified string to see if it matches a specified pattern. You can use
the fnmatch subroutine in an application that reads a dictionary to find which
entries match a given pattern. You also can use the fnmatch subroutine to
match pathnames to patterns.
Examples
1. The following example compiles an internationalized regular expression and
matches a string using this compiled expression. A match is found for the first
pattern, but no match is found for the second pattern.
#include
#include
#define
main()
{
char
char
<locale.h>
<regex.h>
BUFSIZE
256
*p;
*pattern[] = {
"hello[0-9]*",
"1234"
};
char
*string = "this is a test string hello112 and this is test";
/* This is the source string for matching */
int
retval;
regex_t
re;
Chapter 16. National Language Support
379
char
buf[BUFSIZE];
int
i;
setlocale(LC_ALL, "");
for(i = 0;i <2; i++){
retval = match(string, pattern[i], &re);
if(retval == 0){
printf("Match found \n");
}else{
regerror(retval, &re, buf, BUFSIZE);
printf("error = %s\n", buf);
}
}
regfree( &re);
}
int match(char *string, char *pattern, regex_t *re)
{
int
status;
if((status=regcomp( re, pattern, REG_EXTENDED))!= 0)
return(status);
status = regexec( re, string, 0, NULL, 0);
return(status);
}
2. The following example finds all substrings in a line that match a pattern. The
numbers 11 and 1992 are matched. Every digit that is matched counts as one
match. There are six such matches corresponding to the six digits supplied in
the string.
#include
<locale.h>
#include
<regex.h>
#define
BUFSIZE 256
main()
{
char
*p;
char
*pattern = "[0-9]";
char
*string = "Today is 11 Feb 1992 ";
int
retval;
regex_t re;
char
buf[BUFSIZE];
regmatch_t pmatch[100];
int
status;
char
*ps;
int
eflag;
setlocale(LC_ALL, "");
/* Compile the pattern */
if((status = regcomp( &re, pattern, REG_EXTENDED))!= 0){
regerror(status, &re, buf, 120);
exit(2);
}
ps = string;
printf("String to match=%s\n", ps);
eflag = 0;
/* extract all the matches */
while( status = regexec( &re, ps, 1, pmatch, eflag)== 0){
printf("match found at: %d, string=%s\n",
pmatch[0].rm_so, ps +pmatch[0].rm_so);
ps += pmatch[0].rm_eo;
printf("\nNEXTString to match=%s\n", ps);
eflag = REG_NOTBOL;
}
regfree( &re);
}
3. The following example uses the fnmatch subroutine to read a directory and
match file names with a pattern.
#include
#include
#include
380
General Programming Concepts
<locale.h>
<fnmatch.h>
<sys/dir.h>
main(int argc, char *argv[] )
{
char
*pattern;
DIR
*dir;
struct dirent
*entry;
int
ret;
setlocale(LC_ALL, "");
dir = opendir(".");
pattern = argv[1];
if(dir != NULL){
while( (entry = readdir(dir)) != NULL){
ret = fnmatch(pattern, entry->d_name,
FNM_PATHNAME|FNM_PERIOD);
if(ret == 0){
printf("%s\n", entry->d_name);
}else if(ret == FNM_NOMATCH){
continue ;
}else{
printf("error file=%s\n",
entry->d_name);
}
}
closedir(dir);
}
}
Layout (Bidirectional Text and Character Shaping) Overview
Bidirectional (BIDI) text results when texts of different direction orientation appear
together. For example, English text is read from left to right. Arabic and Hebrew
texts are read from right to left. If both English and Hebrew texts appear on the
same line, the text is bidirectional.
Write bidirectional text according to the following guidelines:
v Arabic and Hebrew words are written from right to left. (A character string is
considered a word for the purposes of sequencing in an alphanumeric
environment.)
v Numbers and English quotations are written from left to right.
v Digits and their punctuation are written marks from left to right.
Bidirectional script is read from right to left and from top to bottom.
If the embedded text is contained in one line, the text is written from left to right
and embedded in the bidirectional text. However, if the embedded text is split
between two or more lines, the correct order must be maintained in the left to right
portions to allow top to bottom reading.
For example, right-to-left text embedded in left-to-right text that is contained in
one line is written as follows:
THERE IS txet lanoitceridib deddebme IN THIS SENTENCE.
Right-to-left text embedded in left-to-right text that is split between two lines is
written as follows:
THERE IS senil owt neewteb tilps si taht txet lanoitceridib deddebme IN THIS SENTENCE.
Both texts maintain readability even though the embedded text is split.
Chapter 16. National Language Support
381
Data Streams
Bidirectional text environments use the following data streams:
Visual Data Streams
The system organizes characters in the sequence in which
they are presented on the screen.
If a visual data stream is presented from left to right, the
first character of the data stream is on the left side of the
viewport (screen, window, line, field, and so on). If the same
data stream is presented on a right-to-left viewport, the
initial character of the data stream is on the right.
If a language of opposite writing orientation is embedded in
the visual data stream, the sequence of each text is
preserved when the viewport orientation is reversed. For
example, (the lowercase text represents bidirectional text) if
the keystroke order is :
THERE IS bidirectional text IN THIS SENTENCE.
then the visual data stream is:
THERE IS txet lanoitceridib IN THIS SENTENCE.
This visual data stream’s presentation on a left-to-right
viewport is left-justified, as follows:
THERE IS txet lanoitceridib IN THIS SENTENCE.
———-> <————————- ————————>
The arrows indicate reading direction.
If you change the viewport orientation to right-to-left, the
visual data stream is reversed, right-justified, and
unreadable, as follows:
.ECNETNES SIHT NI bidirectional text SI EREHT
<———————— ————————-> <———Thus, if English text is embedded in Arabic or Hebrew text,
both texts are in proper reading order only on a left-to-right
viewport. The same is true for Arabic or Hebrew embedded
in English. Reversing the viewport orientation makes both
texts unreadable.
382
General Programming Concepts
Logical Data Streams
The system organizes characters in a readable sequence. The
bidirectional presentation-management functions arrange
text strings in a readable order.
If a logical data stream is presented on a left-to-right
viewport, the initial character of the data stream is
presented on the left side. If the same data stream is
presented on a right-to-left viewport, the initial character of
the data stream is presented on the right side, though it is
still presented in a readable order.
If a language of opposite writing orientation is embedded in
the logical data stream, the orientations of each text are
preserved by the bidirectional presentation-management
functions. For example, if the keystroke order is:
THERE IS bidirectional text IN THIS SENTENCE.
then the logical data stream is the same. For example:
THERE IS bidirectional text IN THIS SENTENCE.
This logical data stream’s presentation on a left to right
viewport (left-justified) is as follows:
THERE IS txet lanoitceridib IN THIS SENTENCE.
———-> <————————- ————————>
The logical data stream’s presentation on a right to left
viewport (right-justified) is as follows:
IN THIS SENTENCE. txet lanoitceridib THERE IS
————————> <————————- ———->
The logical data stream is readable on both viewport
orientations.
Cursor Movement
Cursor movement on a screen containing bidirectional text is as follows:
Visual
The cursor moves from its current position left or right to the next character, or
up or down to the next row. For example, if the cursor is located at the end of
the first left-to-right part of a mixed sentence:
THERE IS_txet lanoitceridib IN THIS SENTENCE.
then, moving the cursor visually to the right causes it to move one character to
the right, as follows:
THERE IS txet lanoitceridib IN THIS SENTENCE.
The cursor moves without regard to the contents of the text.
Chapter 16. National Language Support
383
Logical
The cursor moves from its current position to the next or previous character in
the data stream. The character may be adjacent to the cursor’s position,
elsewhere in the same line, or on another line on the screen. Logical cursor
movement requires scanning the data stream to find the next logical character.
For example, if the cursor is located at the end of the first left-to- right part of a
mixed sentence:
THERE IS_txet lanoitceridib IN THIS SENTENCE.
then, moving the cursor logically to the next character causes the data stream to
be scanned to find the next logical character. The cursor moves to the next
logical part of the sentence, as follows:
THERE IS txet lanoitceridib_IN THIS SENTENCE.
The cursor moves according to content.
Character Shaping
Character shaping occurs when the shape of a character is dependent on its
position in a line of text. In some languages, such as Arabic, characters have
different shapes depending on their position in a string and on the surrounding
characters.
The following characteristics determine character shaping in Arabic script:
v The written language has no equivalent to capital letters.
v The characters have different shapes, depending on their position in a string and
on the surrounding characters.
v The written language is cursive. Most characters of a word are connected, as in
English handwriting.
v Joined characters can form nonspacing characters. Additionally, a character can
have a vowel or diacritic mark written over or under it.
v Characters can vary in length, resulting in an output of two coded shapes.
Methods of Character Shaping
Implement character shaping separately from other system components. However,
character shaping should be accessible as a utility by other system components.
The system may use character shaping in the following ways:
v As the user enters data into the computer, the system uses character shaping to
shape the characters. The system stores these characters in their shaped format.
This method avoids the need to use character shaping every time these
characters are displayed. This method is meant for static data such as menus
and help. This method requires preprocessing for proper sorting, searching, or
indexing of the characters.
The characters may need reshaping after processing for proper presentation.
v As the user enters data into the computer, the system stores the characters in
their unshaped format.
This method allows for sorting, searching or indexing of the characters.
However, the system must use character shaping every time the characters are
displayed.
Base shapes are isolated shapes that were not generated by character shaping. Use
base shapes during editing, searching for character strings, or other text operations.
Use shaping only when the text is displayed or printed. If characters are stored in
their shaped form, the system must deshape them before sorting, collating,
searching, or indexing. Character shapes that are not shape determined according
384
General Programming Concepts
to their position in a string are needed for specific character-handling applications
as well as for communication with different coding environments.
Contextual Character Shaping
In general, contextual character shaping is the selection of the required shape of a
character in a given font depending on its position in a word and its surrounding
characters. The following shapes are possible:
Isolated
Final
Initial
Middle
A character that is connected to neither a preceding nor succeeding character.
A character that is connected to a preceding character but not with a
succeeding character.
A character connected to a succeeding character but not with a preceding
character.
A character connected to both a preceding and succeeding character.
A character may also have any of the following characteristics:
v Connecting to a preceding character.
v Connecting to a succeeding character.
v Allowing surrounding characters’ connections to pass through it.
Acronyms, part numbers, and graphic characters do not need contextual character
shaping. To properly enter these characters, turn off the contextual character
shaping and use a specific keyboard interface for exact selection of the desired
shape. Tag these characters by field, line, or control character for later presentation.
For more information about bidirectionality and character shaping, see “Layout
(Bidirectional Text and Character Shaping) Overview” on page 381 and
“Introducing Layout Library Subroutines”.
Introducing Layout Library Subroutines
For information on the layout library, please see website:
www.opengroup.org
Or order ″Portable Layout Services: Context-dependent and Directional Text″
Book# C616 ISBN 1-85912-142-X January 1997
From:
The Open Group,
Publications Department,
PO Box 96,
Witney,
Oxon OX8 6PG,
England
Tel: +44 (0)1993 708731, Fax: +44 (0)1993 708732
Use of the libcur Package
Programs that use the libcur package (extension to AT&T’s libcurses package) need
to make the following changes:
1. Remove the assumption that the number of bytes need to represent a
character in a code set also represents the display column width of the
character. Use the wcwidth subroutine to determine the number of display
columns needed by the wide character code of a character.
2. NLSCHAR is redefined to be wchar_t.
Chapter 16. National Language Support
385
3. The win->_y [y][x] has wchar_t encodings.
4. Programs should not assume any particular encodings on the wchar_t.
5. Programs should use the addstr, waddstr, mvaddstr, and mvwaddstr
subroutines rather than the addch family of subroutines. All string arguments
are in multibyte form.
6. The addch and waddch subroutines accept a wchar_t encoding of the
character. Programs that use these subroutines should ensure that wchar_t are
used in calling these functions. The (x,y) are incremented by the number of
columns occupied by the wchar_t passed to these subroutines.
7. The delch, wdelch, mvdelch, and mvwdelch subroutines support delete and
backspace on multibyte characters depending on the current position of (x,y).
If the current (x,y) column position points to either the first or second column
of a two-column character, the delch subroutine deletes both columns and
shifts the rest of the line by the number of columns deleted.
8. The insch, winsch, mvinsch, and mvwinsch subroutines can be used to insert
a wchar_t encoding of a character at the current (x,y) position. The line is
shifted by the number of columns needed by the wchar_t.
9. The libcur package is modified to support box drawing characters as defined
in the terminfo database and not assume the graphic characters in the
IBM-850 code set. The libcur package supports drawing of primary and
alternate box characters as defined in the box_chars_1 and box_chars_2 entries
in the terminfo database. To use this, programs should be modified in the
following fashion:
Drawing Primary box characters:
wcolorout(win, Bxa);
cbox(win);
wcolorend(win);
or,
wcolorout(win, Bxa);
drawbox(win, y,x, height, width);
wcolorend(win);
Drawing Alternate box characters:
wcolorout(win, Bya)
cboxalt(win);
wcolorend(win);
or,
wcolorout(win, Bya);
drawbox(win, y, x, height, width);
wcolorend(win);
Bxa and Bya refer to the primary and alternate attributes defined in the
terminfo database.
The following macros are added in the cur01.h file:
cboxalt(win)
drawboxalt(win, y,x, height, width)
10. Programs that need to support input of multibyte characters should not set
_extended to TRUE by a call to extended(TRUE). When the _extended flag is
true, the wgetch subroutine returns wchar_t encodings of the character. With
multibyte characters, this encoding of wchar_t may conflict with predefined
values for escape sequences or function keys. Avoid this conflict when using
multibyte code sets by setting extended to off (extended(FALSE)) before input.
(The default is TRUE.)
Programs that do multibyte character input should do the following:
386
General Programming Concepts
Input routine:
Example:
int c, count;
char buf[];
extended(FALSE); /* obtain one byte at a time */
count =0;
while(1){
c = wgetch(); /* get one byte at a time */
buf[count++] = c;
if(count <=MB_CUR_MAX)
if(mblen(buf, count) != -1)
break; /* character found* /
else
/*Error. No character can be found */
/* Handle this case appropriately */
break;
}
/* buf contains the input multibyte sequence */
/* Now handle PF keys, or any escape sequence here */
11. The inch, winch, mvinch, and mvwinch subroutines return the wchar_t at the
current (x,y) position. Note that in the case of a double column width
character, if the (x,y) point is at the first column, the wchar_t code of the
double column width character is returned. If the (x,y) point is at the second
column, WEOF is returned.
Code Set Overview
To understand code sets, it is necessary to first understand character sets. A
character set is a collection of predefined characters based on the specific needs of
one or more languages without regard to the encoding values used to represent the
characters. The choice of which code set to use depends on the user’s data
processing requirements. A particular character set can be encoded using different
encoding schemes. For example, the ASCII character set defines the set of
characters found in the English language. The Japanese Industrial Standard (JIS)
character set defines the set of characters used in the Japanese language. Both the
English and Japanese character sets can be encoded using different code sets.
The ISO2022 standard defines a coded character set as a set of precise rules that
defines a character set and the one-to-one relationship between each character and
it’s bit pattern. A code set defines the bit patterns that the system uses to identify
characters.
A code page is similar to a code set with the limitation that a code-page
specification is based on a 16-column by 16-row matrix. The intersection of each
column and row defines a coded character.
The following code sets are supported:
v Support for industry-standard code sets is provided. The ISO8859 family of code
sets provides a range of single-byte code set support that includes Latin-1,
Latin-2, Cyrillic, Arabic, Greek, Hebrew, and Turkish countries. The IBM-eucJP
code set is the industry-standard code set used to support the Japanese locale.
The IBM-eucKR code set is the industry-standard code set used to support
Korean countries. The IBM-eucTW code set is the industry-standard code set
used to support Traditional Chinese countries. The IBM-eucCN code set is the
industry-standard code set used to support countries using Simplified Chinese.
The UTF-8 code set is a Universal Transformation Format of Unicode/ISO10646
used to support multiple languages at once (including Simplified Chinese,
Traditional Chinese, and Chinese characters used in Japanese and Korean).
Chapter 16. National Language Support
387
v ISO8859-15 standard codeset is a replacement standard for the existing
ISO8859-1 codeset that is currently in use by the western European locales, the
United States, and Canada. The need for a new codeset came about as a result of
the introduction of the Euro currency unit, and the need for European countries
to be able to do business transactions using the Euro. In addition, ISO8859-15
contains 7 additional characters for the French and Finnish languages.
v Support is also provided for the personal computer (PC) based code sets
IBM-850, IBM-856, IBM-943, IBM-932, and IBM-1046. IBM-850 is a single-byte
code set used to support Latin-1 countries (U.S., Canada, and Western Europe).
IBM-856 is a single-byte code set used to support Hebrew countries. IBM-943
and IBM-932 are multibyte code set used to support the Japanese locale.
IBM-1046 is a single-byte code set used to support Arabic countries.
v IBM-1129 is a single-byte code set used to support Vietnamese.
v TIS-620 is a single-byte code set used to support Thai.
v IBM-1124 is a single-byte code set used to support Ukrainian.
v Full Unicode support is provided via the UTF-8 code set for ALL languages and
territories supported by AIX. The UTF-8 code set is a Universal Transformation
Format of Unicode/ISO10646 used to support multiple languages at once. The
UTF-8 code set provides the most complete solution for use in environments
where multiple languages and alphabets must be processed. The Unicode/UTF-8
codeset also provides full support for the common European currency (Euro).
v IBM-1252 codeset support is provided as a compatibility option for users who
require a single byte codeset environment containing the Euro currency symbol.
The structure of the IBM-1252 codeset is identical to the industry standard
codeset ISO8859-1, except that additional graphic characters are added in the
ISO control characters range from 0x80 through 0x9F. The Euro currency symbol
is located at hexadecimal value 0x80 in the IBM-1252 codeset.
For more information on code sets, refer to these articles:
v “ASCII Characters”
v “Code Set Strategy” on page 390
v “Code Set Structure” on page 391
v “ISO Code Sets” on page 392
v “IBM PC Code Sets” on page 407
ASCII Characters
The following sections describe the 7-bit ASCII characters.
ASCII Characters in the Unique Code-Point Range
The following table lists the ASCII characters in the unique code-point range.
These characters are in the range 0x00 through 0x3F.
ASCII Characters in the Unique Code-Point Range
388
Symbolic
Name
Hex Value
<nul>
Symbolic
Name
Hex Value
Glyph
00
<space>
20
blank
<soh>
01
<exclamation- 21
mark>
!
<stx>
02
<quotationmark>
″
General Programming Concepts
Glyph
22
<etx>
03
<numbersign>
23
#
<eot>
04
<dollar-sign>
24
$
<enq>
05
<percent>
25
%
<ack>
06
<ampersand> 26
&
<alert>
07
<apostrophe> 27
’
<backspace>
08
<leftparenthesis>
28
(
<tab>
09
<rightparenthesis>
29
)
<newline>
0A
<asterisk>
2A
*
<vertical-tab> 0B
<plus-sign>
2B
+
<form-feed>
0C
<comma>
2C
,
<carriagereturn>
0D
<hyphen>
2D
-
<so>
0E
<period>
2E
.
<si>
0F
<slash>
2F
/
<dle>
10
<zero>
30
0
<dc1>
11
<one>
31
1
<dc2>
12
<two>
32
2
<dc3>
13
<three>
33
3
<dc4>
14
<four>
34
4
<nak>
15
<five>
35
5
<syn>
16
<six>
36
6
<etb>
17
<seven>
37
7
<can>
18
<eight>
38
8
<em>
19
<nine>
39
9
<sub>
1A
<colon>
3A
:
<esc>
1B
<semicolon>
3B
;
<is1>
1C
<less-than>
3C
<
<is2>
1D
<equal-sign>
3D
=
<is3>
1E
<greaterthan>
3E
>
<is4>
1F
<questionmark>
3F
?
Other ASCII Characters
The following table lists the 7-bit ASCII characters that are not in the unique
code-point range. These characters are in the range 0x40 through 0x7F.
Other ASCII Characters
Symbolic
Name
Hex Value
<commercial- 40
at>
Glyph
Symbolic
Name
Hex Value
Glyph
@
<graveaccent>
60
′
Chapter 16. National Language Support
389
<A>
41
A
<a>
61
a
<B>
42
B
<b>
62
b
<C>
43
C
<c>
63
c
<D>
44
D
<d>
64
d
<E>
45
E
<e>
65
e
<F>
46
F
<f>
66
f
<G>
47
G
<g>
67
g
<H>
48
H
<h>
68
h
<I>
49
I
<i>
69
i
<J>
4A
J
<j>
6A
j
<K>
4B
K
<k>
6B
k
<L>
4C
L
<l>
6C
l
<M>
4D
M
<m>
6D
m
<N>
4E
N
<n>
6E
n
<O>
4F
O
<o>
6F
o
<P>
50
P
<p>
70
p
<Q>
51
Q
<q>
71
q
<R>
52
R
<r>
72
r
<S>
53
S
<s>
73
s
<T>
54
T
<t>
74
t
<U>
55
U
<u>
75
u
<V>
56
V
<v>
76
v
<W>
57
W
<w>
77
w
<X>
58
X
<x>
78
x
<Y>
59
Y
<y>
79
y
<Z>
5A
Z
<z>
7A
z
<left-bracket> 5B
[
<left-brace>
7B
{
<backslash>
5C
\
<verticalline>
7C
|
<rightbracket>
5D
]
<right-brace> 7D
}
<circumflex>
5E
|
<tilde>
7E
x
<underscore> 5F
_
<del>
7F
Code Set Strategy
Prior to Version 3.2, IBM-850 and IBM-932 were the only supported code sets.
Version 3.2 enhanced the system code set support by adding code sets that are
based on International Organization for Standardization (ISO) and
industry-standard code sets. It is suggested that users use to these new code sets.
The ultimate goal is to provide industry-standard code sets that satisfy the data
processing needs of users.
Support for the IBM-850 codeset will be removed in future releases of AIX. Users
who are currently using IBM-850 based locales should strongly consider use of the
390
General Programming Concepts
corresponding industry standard ISO8859-1 based locale. For example, users of the
French IBM-850 locale (Fr_FR) should use the French ISO8859-1 locale (fr_FR).
Each locale in the system defines which code set it uses and how the characters
within the code set are manipulated. Because multiple locales can be installed on
the system, multiple code sets can be used by different users on the system. While
the system can be configured with locales using different code sets, all system
utilities assume that the system is running under a single code set.
Most commands have no knowledge of the underlying code set being used by the
locale. The knowledge of code sets is hidden by the code set-independent library
subroutines (NLS library), which pass information to the code set-dependent
subroutines.
Because many programs rely on ASCII, all code sets include the 7-bit ASCII code
set as a proper subset. Since the 7-bit ASCII code set is common to all supported
code sets, its characters are sometimes referred to as the portable character set.
The 7-bit ASCII code set is based on the ISO646 definition and contains the control
characters, punctuation characters, digits (0-9), and the English alphabet in
uppercase and lowercase.
Code Set Structure
This code set table shows the general format for all code sets:
0
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
C0
C
o
n
t
r
o
l
s
1
2
3
4
5
6
7
8
9
(Graphic Left)
C1
7–Bit ASCII
C
o
n
t
r
o
l
s
A
B
C
D
E
F
(Graphic Right)
Code
Set
Unique
General Structure of a Code Set
Each code set is divided into two principal areas:
Graphic Left (GL)
Graphic Right (GR)
Columns 0-7
Columns 8-F
The first two columns of each code set are reserved by International Organization
for Standardization (ISO) standards for control characters. The terms C0 and C1 are
used to denote the control characters for the Graphic Left and Graphic Right areas,
respectively.
Note: The IBM PC code sets use the C1 control area to encode graphic
characters.
Chapter 16. National Language Support
391
The remaining 6 columns are used to encode graphic characters. Graphic
characters are considered to be printable characters, while the control characters
are used by devices and applications to indicate some special function.
Control Characters
Based on the ISO definition, a control character initiates, modifies, or stops a
control operation. A control character is not a graphic character, but can have
graphic representation in some instances. The control characters in the control
points table are present in all supported code sets and the encoded values of the
control characters are consistent throughout the code sets.
NUL
SOH
STX
ETX
EOT
ENQ
ACK
BEL
BS
HT
LF
VT
FF
CR
SO
SI
00
01
02
03
04
05
60
07
08
09
0A
0B
0C
0D
0E
0F
Null
Start of header
Start of text
End of text
End of transmission
Enquiry
Acknowledge
Bell
Backspace
Horizontal tab
Line feed
Vertical tab
Form feed
Carrier return
Shift Out
Shift In
DLE
DC1
DC2
DC3
DC4
NAK
SYN
ETB
CAN
EM
SUB
ESC
IS4
IS3
IS2
IS1
10
11
12
13
14
15
16
17
18
19
1A
1B
1C
1D
1E
1F
Data link escape
Device control 1
Device control 2
Device control 3
Device control 4
Not acknowledge
Synchronous idle
End of trans. block
Cancel
End of media
Substitute character
Escape character
Info Separator Four
Info Separator Three
Info Separator Two
Info Separator One
Code Set Control Points Table
Graphic Characters
Each code set can be considered to be divided into one or more character sets, such
that each character is given a unique coded value. The ISO standard reserves six
columns for encoding characters and does not allow graphic characters to be
encoded in the control character columns.
The internationalization of AIX is based on the assumption that all code sets can be
divided into any number of character sets.
Single-Byte and Multibyte Code Sets
Code sets that use all 8 bits of a byte can support European, Middle Eastern, and
other alphabetic languages. Such code sets are called single-byte code sets. This
provides a limit of encoding 191 characters, not including control characters.
Languages that require more than 191 characters use a mixture of single-byte
characters (8 bits) and multibyte characters (more than 8 bits). The system is
capable of supporting any number of bits to encode a character.
ISO Code Sets
These code sets are based on definitions set by the International Organization for
Standardization (ISO).
v “ISO646-IRV” on page 393
v “ISO8859 Family” on page 393
392
General Programming Concepts
v
v
v
v
v
v
v
v
v
v
v
v
“Code Set ISO8859-2” on page 396
“Code Set ISO8859-5” on page 397
“Code Set ISO8859-6” on page 398
“Code Set ISO8859-7” on page 399
“Code Set ISO8859-8” on page 400
“Code Set ISO8859-9” on page 401
“Code Set ISO8859-15” on page 402
“IBM-eucJP” on page 403
“IBM-eucTW” on page 405
“Big5” on page 405
“IBM-eucKR” on page 406
“UCS-2 and UTF-8” on page 420
ISO646-IRV
The 393 defines the code set used for information processing based on a 7-bit
encoding. The character set associated with this code set is derived from the ASCII
characters.
!
”
#
$
%
&
’
(
)
* :
+ ;
,
’
@
{
}
/ ?
ISO646–IRV Code Set
ISO8859 Family
ISO8859 is a family of single-byte encodings based on and compatible with other
ISO, American National Standards Institute (ANSI), and European Computer
Manufacturer’s Association (ECMA) code extension techniques. The ISO8859
Chapter 16. National Language Support
393
encoding defines a family of code sets with each member containing its own
unique character sets. The 7-bit ASCII code set is a proper subset of each of the
code sets in the ISO8859 family.
While the ASCII code set defines an order for the English alphabet, the Graphic
Right (GR) characters are not ordered according to any specific language. The
language-specific ordering is defined by the locale.
Each code set includes the ASCII character set plus its own unique character set.
The ISO8859 encoding table shows the ISO8859 general encoding scheme.
Character
Encoding
Code
Point
Description
Count
000xxxxx
00100000
0xxxxxxx
01111111
100xxxxx
10100000
1xxxxxxx
00–1f
20
21–7e
7f
80–9f
a0
a1–ff
Controls
Space
7–bit
Delete
Controls
No–Break Space
8–bit
(32)
(1)
(94)
(1)
(32)
(1)
(96)
0
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
C
o
n
t
r
o
l
s
1
2
3
4
ASCII
5
6
7
8
9
C
o
n
t
r
o
l
s
ISO8859 Encoding
394
General Programming Concepts
A
B
C
D
Extended
Characters
E
F
Code Set ISO8859-1
The following figure summarizes the available symbols and shows the layout of
Code Set ISO8859-1.
0
1
2
3
4
5
6
7
8
9
A
B
C
D E
F
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set ISO8859-1
Chapter 16. National Language Support
395
Code Set ISO8859-2
The following figure summarizes the available symbols and shows the layout of
Code Set ISO8859-2.
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set ISO8859-2
396
General Programming Concepts
A B
C D
E
F
Code Set ISO8859-5
The following figure summarizes the available symbols and shows the layout of
Code Set ISO8859-5.
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set ISO8859-5
Chapter 16. National Language Support
397
Code Set ISO8859-6
The following figure summarizes the available symbols and shows the layout of
Code Set ISO8859-6.
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set ISO8859-6
398
General Programming Concepts
A
B
C
D
E
F
Code Set ISO8859-7
The following figure summarizes the available symbols and layout of Code Set
ISO8859-7. This code set is made up of an ASCII character set plus its own unique
character set.
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set ISO8859-7
Chapter 16. National Language Support
399
Code Set ISO8859-8
The following figure summarizes the available symbols and shows the layout of
Code Set ISO8859-8.
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set ISO8859–8
400
General Programming Concepts
A
B
C
D
E
F
Code Set ISO8859-9
The following figure summarizes the available symbols and layout of Code Set
ISO8859-9. This code set is made up of an ASCII character set plus its own unique
character set.
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set ISO8859-9
Chapter 16. National Language Support
401
Code Set ISO8859-15
The following figure summarizes the available symbols and shows the layout of
Code Set ISO8859-15.
Code Set ISO8859-15
Extended UNIX Code (EUC) Encoding Scheme
The EUC encoding scheme defines a set of encoding rules that can support one to
four character sets. The encoding rules are based on the ISO2022 definition for the
encoding of 7-bit and 8-bit data. The EUC encoding scheme uses control characters
to identify some of the character sets. The EUC encoding table shows the basic
structure of all EUC encoding.
402
General Programming Concepts
CS0
0xxxxxxx
CS1
1xxxxxxx
1xxxxxxx 1xxxxxxx
1xxxxxxx 1xxxxxxx 1xxxxxxx
...
CS2
10001110 1xxxxxxx
10001110 1xxxxxxx 1xxxxxxx
10001110 1xxxxxxx 1xxxxxxx 1xxxxxxx
...
CS3
10001111 1xxxxxxx
10001111 1xxxxxxx 1xxxxxxx
10001111 1xxxxxxx 1xxxxxxx 1xxxxxxx
...
EUC Encoding Table
The term EUC denotes these general encoding rules. A code set based on EUC
conforms to the EUC encoding rules but also identifies the specific character sets
associated with the specific instances. For example, IBM-eucJP for Japanese refers
to the encoding of the Japanese Industrial Standard characters according to the
EUC encoding rules.
The first set (CS0) always contains an ISO646 character set. All of the other sets
must have the most significant bit (MSB) set to 1 and can use any number of bytes
to encode the characters. In addition, all characters within a set must have:
v Same number of bytes to encode all characters
v Same column display width (number of columns on a fixed-width terminal)
All characters in the third set (CS2) are always preceded with the control character
SS2 (single-shift 2, 0x8e). Code sets that conform to EUC do not use the SS2 control
character other than to identify the third set.
All characters in the fourth set (CS3) are always preceded with the control
character SS3 (single-shift 3, 0x8f). Code sets that conform to EUC do not use the
SS3 control character other than to identify the fourth set.
IBM-eucJP
The EUC for Japanese is an encoding consisting of single-byte and multibyte
characters. The encoding is based on ISO2022, Japanese Industrial Standard (JIS),
and EUC definitions.
The IBM-eucJP code set consists of the following character sets:
JISCII
JISX0201.1976
JISX0208.1983
IBM-udcJP
JISX0201 Graphic Left character set
Katakana/Hiragana Graphic Right character
set
Kanji level 1 and 2 character sets
IBM-user definable characters
The IBM-eucJP code set is also capable of supporting:
JISX0212.1990
Supplemental Kanji
The IBM-eucJP code set is encoded as follows:
v CS0 maps JISX0201 Graphic Left characters starting at the 0x00 position.
Chapter 16. National Language Support
403
v CS1 maps the JISX0208 character set starting at the 0xa1xa1 position.
The positions 0xf5a1 through 0xfefe (940 characters) in CS1 are reserved as primary
user definable character areas.
v CS2 maps the JISX0201 Graphic Right starting at the 0x8ea1 position.
v CS3 is capable of mapping JISX0212 starting at the 0x8fa1a1 position.
The positions 0x8ff5a1 through 0x8ffefe in CS3 (940 characters) are reserved as
secondary user definable character areas.
v The positions 0x8feea1 through 0x8ff4fe in CS3 (658 characters) are reserved for
future system use. Therefore, users should not use this area.
IBM-eucCN
The EUC for the Simplified Chinese language is an encoding consisting of
characters that contain 1 or 2 bytes. The EUC encoding is based on ISO2022,
GB2312 as defined by the People’s Republic of China, and multibyte character
definitions unique to the manufacturer.
The current GB2312 defines 6,763 Simplified Chinese characters and 682 symbols.
The IBM-eucCN is based upon a concept of one plane containing up to 94x94
characters. The encoding values of these characters range from 0xa1a1 to 0xfefe.
The GB2312 is mapped into the CS1 of EUC. Specifically, the IBM-eucCN consists
of the following character sets:
7-bit ASCII character set, Graphic Left.
Contains 7445 characters. It occupies
positions 0xa1a1 to 0xfedf (some user-defined
characters scattered in 0xa1a1 to 0xfedf).
Scattered in GB. It occupies positions Oxa1a1
to Oxfedf. The actual values are:
ISO0646-IRV
GB2312.1980
IBM-udcCN
a2a1 — a2b0
a1e3 — a2e4 a1ef — a2f0
a2fd — a1fe
a4f4 — a4fe a5f7 — a5fe
a6b9 — a6c0
a6d9 — a6fe a7c2 — a7d0
a7f2 — a7fe
a8bb — a8c4 a8ea — a9a3
a9f0 — affe
a7fa — d7fe f8a1 — fedf
Scattered in GB. It occupies positions 0xfee0
to 0xfefe.
IBM-sbdCN
GBK
GBK stands for Guo (national) Biao (Standard) Kuo (Extension). GBK expands the
national ″Industry GB″ definition to contain all 20, 902 Han Characters defined in
Unicode and additional DBCS symbols defined in Big-5 code (Traditional Chinese
PC defacto standard). Restated, GBK defined all DBCS characters and symbols in
use on both sides of the Taiwan Strait. Currently, GBK is a Normative Annex of
GB13000 (PRC Unicode Standard) and is being positioned as an interim step for
migration to Unicode.
404
Locale
Code Set
Description
Zh_CN
GBK
Simplified Chinese, GBK
Locale
Code Range
Words
Marks
A1A1-A9FE
846
GB2312, GB12345 (GBK/1)
A840-A9A0
192
Big5, Symbols (GBK/5)
General Programming Concepts
Code Range
Words
Marks
B0A1-F7FE
6768
GB2312 (GBK/2)
8140-A0FE
6080
GB13000 (GBK/3)
AA40-FEA0
8160
GB13000 (GBK/4)
AAA1-AFFE
564
User defined 1
F8A1-FEFE
658
User defined 2
A140-A7A0
672
User defined 3
IBM-eucTW
The EUC for the Traditional Chinese language is an encoding consisting of
characters that contain 1, 2 and 4 bytes. The EUC encoding is based on ISO2022,
the Chinese National Standard (CNS) as defined by the Republic of China and
multibyte character definitions unique to the manufacturer.
The current CNS defines 13,501 Chinese characters and 684 symbols. The
IBM-eucTW is based upon a concept of 15 planes, each containing up to 8836
(94x94) characters. The encoding values of these characters range from 0xa1a1 to
0xfefe. Characters have presently been defined for only 4 of the planes, with the
other planes being reserved for future expansion.
The 15 planes are mapped into the CS1 and CS2 of EUC, with the CS2 of EUC
consisting of 14 planes. Specifically, the IBM-eucTW consists of the following
character sets:
ISO646-IRV
CNS11643.1986-1
CNS11643.1986-2
CNS11643.1992-3
IBM-udcTW
IBM-sbdTW
7-bit ASCII character set, Graphic Left.
Plane 1, containing 6085 characters
(5401+684). This plane uses positions
0ax1a1-0xc2c1 and 0xc4a1-0xfdcb.
Plane 2, containing 7650 characters. This
plane occupies positions
0x8ea2a1a1-0x8ea2f2c4.
Plane 4, containing 7298 characters. This
plane occupies positions
0x8ea4a1a1-0x8ea4eedc.
Plane 12, containing 6204 characters. This
plane is reserved for the User Defined
Characters (udc) areas. It occupies the
positions 0x8eaca1a1-0x8ea2f2c4.
Plane 13, containing 325 characters. This
plane is reserved for symbols unique to the
manufacturer. It occupies positions
0xeada1a1-0x8eada4cb.
Planes 3-11 are expected to occupy positions 0x8ea3xxxx to 0x8eabxxxx. Planes
14-15 are expected to occupy positions 0x8eaexxxx to 0x8eafxxxx.
Big5
The Traditional Chinese big5 locale, Zh_TW, code set is the most commonly used
code set in the PC field which is used to support countries using Traditional
Chinese.
Big5 code set defines 13056 characters and 1004 symbols. It includes 684 symbols
in CNS11643.192, as well as 325 IBM unique symbols.
Chapter 16. National Language Support
405
Locale
Code Set
Description
Zh_TW
Big5 (IBM-950)
Traditional Chinese, Big5
Locale
Code Range for Big5 Locale::
Plan
Code Range
Description
1
A140H - A3E0H
Symbol and Chinese Control
Code
1
A440H - C67EH
Commonly Used Characters
2
C940H - F9D5H
Less Commonly Used
Characters
UDF
FA40H - FEFE
User-Defined Characters
8E40H - A0FEH
User-Defined Characters
8140H - 8DFEH
User-Defined Characters
8181H - 8C82H
User-Defined Characters
F9D6H - F9F1H
User-Defined Characters
Code Set
Words
Code Range
Commonly Used
Area
5841
A140-C67E
Less Commonly
Used Area
7652
C940-F9D5
ET Unique Area (1)
308
C6A1-C878
ET Unique Area (2)
7
C8CD-C8D3
IBM Unique Area
251
F286-F9A0
User-Defined Area
(1)
785
FA40-FEFE
User-Defined Area
(2)
2983
8E40-A0FE
User-Defined Area
(3)
2041
8140-8DFE
User-Defined Area
(4)
354
8181-8C82
User-Defined Area
(5)
41
F9D6-F9FE
Marks
Low-Byte Range
81-A0
Low-Byte Range
81-AQ
IBM-eucKR
The EUC for the Korean language is an encoding consisting of single-byte and
multibyte characters. The encoding is based on ISO2022, Korean Standard Code set
and EUC definitions.
The Korean EUC code set consists of two main character groups:
v ASCII (English)
v Hangul (Korean characters)
The Hangul code set includes Hangul and Hanja (Chinese) characters. One Hangul
character can be comprised of several consonants and vowels. However, most
406
General Programming Concepts
Hangul words can be expressed in Hanja. Each Hanja character has its own
meaning and is more specific than Hangul.
The IBM-eucKR consists of the following character sets:
7-bit ASCII character set, Graphic Left
Korean Graphic Character Set, Graphic Right
ISO646-IRV
KSC5601.1987-0
IBM PC Code Sets
IBM PC code sets are the code sets originally supported on the IBM PC systems
and AIX. The IBM PC code sets assign graphic characters to the Control One (C1)
control area. Applications that depend on these control characters can not support
these code sets.
The ASCII characters are encoded with the most significant bit (MSB) zero in
positions 0x20-0x7e. The extended Latin 1 combined with the IBM PC unique
characters sets make up the extended set of characters which are encoded in
positions 0x80-0xfe. The personal computer encoding table shows the location of
the control, ASCII, and extended characters for the IBM-850 code set.
Character
Encoding
Code
Points
Description
Count
000xxxxx
00100000
0xxxxxxx
01111111
1xxxxxxx
11111111
00–1f
20
21–7e
7f
80–fe
ff
Controls
Space
7–bit
Delete
8–bit
All Ones
(32)
(1)
(94)
(1)
(127)
(1)
0
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
1
C
o
n
t
r
o
l
s
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Extended
Characters
ASCII
Personal Computer Encoding
The IBM PC unique character set includes the following:
IBM PC Unique Character Set
Symbol
Return Code
Florin sign
0x9f
Quarter-hashed
0xb0
Half-hashed
0xb1
Chapter 16. National Language Support
407
408
Full-hashed
0xb2
Vertical bar
0xb3
Right-side middle
0xb4
Double right-side middle
0xb9
Double vertical bar
0xba
Double upper right-corner box
0xbb
Double lower right-corner box
0xbc
Upper right-corner box
0xbf
Lower left-corner box
0xc0
Bottom-side middle
0xc1
Top-side middle
0xc2
Left-side middle
0xc3
Center-box bar
0xc4
Intersection
0xc5
Double lower left-corner box
0xc8
Double upper left-corner box
0xc9
Double bottom-side middle
0xca
Double top-side middle
0xcb
Double left-side middle
0xcc
Double center-box bar
0xcd
Double intersection
0xce
Small i dotless
0xd5
Lower right-corner box
0xd9
Upper left-corner box
0xda
Bright character cell
0xdb
Bright character cell - lower half
0xde
Bright character cell - upper half
0xdf
Overbar
0xee
Middle dot, Product dot
0xfa
Vertical solid rectangle
0xfe
General Programming Concepts
IBM-850
The following figure summarizes the available symbols and shows the layout of
Code Set IBM-850.
+
_
=
/4
3
¶
§
P
{
/2
/4
1
£
1
!
!
”
#
$
%
&
’
(
)
* :
+ ;
,
’
@
}
¥
/ ?
Code Set IBM-850
Chapter 16. National Language Support
409
IBM-856
The following figure summarizes the available symbols and shows the layout of
Code Set IBM-856.
0
1
2
3
4
5
6
7
8
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set IBM-856
410
General Programming Concepts
9
A
B
C
D
E
F
IBM-921
The following figure summarizes the available symbols and shows the layout of
Code Set IBM-921.
0
1
2
3
4
5
6
7
9
8
A
B
C
D
E
F
0
”
1
2
3
”
4
5
”
6
7
8
9
A
B
C
D
E
F
Code Set IBM-921
Chapter 16. National Language Support
411
IBM-922
The following figure summarizes the available symbols and shows the layout of
Code Set IBM-922.
0
1
2
3
4
5
6
7
8
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set IBM-922
412
General Programming Concepts
9
A
B
C
D
E
F
IBM-943 and IBM-932
Each of the Japanese IBM PC code sets are an encoding consisting of single-byte
and multibyte coded characters. The encoding is based on the IBM PC code set
and places the JIS characters in shifted positions. This is referred to as Shift-JIS or
SJIS.
IBM-943 is newer code set for the Japanese locale than IBM-932. IBM-943 is a
compatible code set for the Japanese Microsoft Windows environment. This code
set is known as 1983 ordered shift-JIS. The difference between IBM-932 and
IBM-943 is as follows
v Old JIS sequence (1978 ordered) is applied for IBM-932 while new JIS sequence
(1983 ordered) is applied for IBM-943.
v NEC selected characters are added to IBM-943.
v NEC’s IBM selected characters are added to IBM-943.
The IBM-932 consists of the following character sets:
JISCII
JISX0201.1976
JISX0208.1983
IBM-udcJP
JISX0201 Graphic Left character set
Katakana/Hiragana Graphic Right character
set
Kanji level 1 and 2 character sets
IBM user-definable characters
The IBM-943 consists of the following character sets:
JISCII
JISX0201.1976
JISX0208.1990
IBM-udcJP
JISX0201 Graphic Left character set
Katakana/Hiragana Graphic Right character
set
Kanji level 1 and 2 character sets
IBM user-definable characters and NEC’s
IBM selected characters and NEC selected
characters
Chapter 16. National Language Support
413
The first byte of each character is used to determine the number of bytes for a
given character. The values 0x20-0x7e and 0xa1-oxdf are used to encode JISX0201
characters, with exceptions. The positions 0x81-0x9f and 0xe0-0xfc are reserved for
use as the first byte of a multibyte character. The JISX0208 characters are mapped
to the multibyte values starting at 0x8140. The second byte of a multibyte character
can have any value. The Shift-JIS table shows where these characters are located on
the code set.
414
General Programming Concepts
The following figure shows the DBCS part of IBM-943.
Chapter 16. National Language Support
415
The following figure shows the DBCS part of IBM-932.
416
General Programming Concepts
IBM-1046
The following figure summarizes the available symbols and shows the layout of
Code Set IBM-1046.
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
Code Set IBM-1046
Chapter 16. National Language Support
417
IBM-1124
The following figure summarizes the available symbols and shows the layout of
Code Set IBM-1124.
418
General Programming Concepts
IBM-1129
The following figure summarizes the available symbols and shows the layout of
Code Set IBM-1129.
Chapter 16. National Language Support
419
TIS-620
The following figure summarizes the available symbols and shows the layout of
Code Set TIS-620.
UCS-2 and UTF-8
AIX provides a set of codesets that address the needs of a particular language or a
language group. None of the codesets represented in the ISO8859 family of
codesets, the PC codesets, nor the Extended Unix Code (EUC) codesets allow the
mixing of characters from different scripts. With ISO8859-1, you can mix and
represent the Latin 1 characters (languages principally spoken in the U.S., Canada,
Western Europe, and Latin America). ISO8859-2 covers Eastern European
languages; ISO8859-5 covers Cyrillic, ISO8859-6 covers Arabic, ISO8859-7 covers
Greek, ISO8859-8 covers Hebrew, ISO8859-9 covers Turkish, IBM-eucJP covers
Japanese, IBM-eucKR covers Korean, IBM-eucTW covers Simplified Chinese. The
point is that none of the above codesets covers all of the languages.
The International Organization for Standardization (ISO) has addressed the limited
language coverage by codesets by adopting Unicode as the encoding for the 2-octet
form of the ISO10646 Universal Multiple-Octet Coded Character Set (UCS-2). The
32-bit form of ISO10646 is known as UCS-4 for 4-octet form. AIX has adopted the
16-bit form of ISO10646 and uses the standard label ″UCS-2″ to describe this
encoding.
420
General Programming Concepts
Although UCS-2 is ideal for an internal process code, it is not suitable for encoding
plain text on traditional byte-oriented systems, such as AIX. Therefore, the external
file code is X/Open’s File System Safe UCS Transformation Format (FSS-UTF). This
transformation format encoding is also known as UTF-8, and ″UTF-8″ is the label
that is used for this encoding on AIX.
ISO10646 UCS-2 (Unicode)
Universal Coded Character Set (UCS) is the name of the ISO10646 standard that
defines a single code for the representation, interchange, processing, storage, entry,
and presentation of the written form of all the major languages of the world.
UCS has the following key objectives:
v Improve interoperability between systems in an open environment.
v Simplify development of internationalized products.
v Provide a base for multilingual applications.
v Make more characters available.
ISO10646 defines canonical character codes with a length of 32 bits. This provides
code numbers for over 4 billion characters. When used in canonical form to
represent text, the coding is referred to as UCS-4 for Universal Coded Character
Set 4-byte form.
The code values from 0x0000 through 0xFFFF of ISO 10646 can be represented by a
uniform character encoding of 16 bits. When used in this form to represent text,
these codes are referred to as UCS-2, for Universal Character Set 2-octet form. This
range is also called the Basic Multilingual Plane (BMP) of ISO10646. The standard
is arranged so that the most useful characters, covering all major existing standards
worldwide, are assigned within this range.
The character code values of UCS-2 are identical to those of the Unicode character
encoding standard published by the Unicode Consortium.
UCS-2 defines codes for characters used in all major written languages. In addition
to a set of scientific, mathematic, and publishing symbols, UCS-2 covers the
following scripts:
v Arabic
v Armenian
v Bengali
v Bopomofo
v Cyrillic
v Devanagari
v Georgian
v Greek
v Gujarati
v Gurmukhi
v Hangul
v Chinese Hanzi
v Hebrew
v Hiragana
v International Phonetic Alphabet (IPA)
v Katakana
Chapter 16. National Language Support
421
v
v
v
v
v
v
v
v
v
v
v
Japanese Kanji
Kannada
Korean Hanja
Laotian
Latin
Malayalam
Oriya
Tamil
Teluga
Thai
Tibetan
The ability of AIX to display characters in the scripts mentioned above is limited to
the availability of fonts. AIX provides bitmap fonts for most of the major languages
of the world, as well as a Unicode based scalable TrueType font. Use of this font
requires the TrueType font rasterizer for AIX, which is a separately installable
feature.
UCS-2 encodes a number of combining characters, also known as non-spacing
marks for floating diacritics. These characters are necessary in several scripts
including Indic, Thai, Arabic, and Hebrew. The combining characters are used for
generating characters in Latin, Cyrillic, and Greek scripts. However, the presence of
combining characters creates the possibility for an alternative coding for the same
text. Although the coding is unambiguous and data integrity is preserved, the
processing of text that contains combining characters is more complex. To provide
conformance for applications that choose not to deal with the combining
characters, ISO10646 defines the following three implementation levels:
Level 1
Level 2
Level 3
Does not allow combining characters.
Allows combining marks from Thai, Indic, Hebrew, and Arabic scripts.
Allows combining marks, including ones for Latin, Cyrillic, and Greek.
UTF-8 (UCS Transformation Format)
X/Open has developed a transformation format for UCS designed for use in
existing file systems. The original name of this transformation is FSS-UTF, but it is
expected to be registered by ISO as UTF-8. UTF-8 is expected to become the
standard transformation method, for situations where UCS is not practical. The
intent is that UCS will be the process code for the transformation format, which is
usable as a file code.
UTF-8 has the following properties:
v It is a superset of ASCII, in which the ASCII characters are encoded as
single-byte characters with the same numeric value.
v No ASCII code values occur in multibyte characters, other than those which
represent the ASCII characters.
v Conversion to and from UCS is simple and efficient.
v The first byte of a character indicates the number of bytes to follow in the
multibyte character sequence and cannot occur anywhere else in the sequence.
The UTF-8 encodes UCS values in the 0 through 0x7FFFFFFF range using
multibyte characters with lengths of 1, 2, 3, 4, 5, and 6 bytes. Single-byte characters
are reserved for the ASCII characters in the 0 through 0x7f range. These all have
the high order bit set to 0. For all character encodings of more than one byte, the
initial byte determines the number of bytes used, and the high-order bit in each
422
General Programming Concepts
byte is set. Every byte that does not start with the bit combination of 10xxxxxx,
where x represents a bit that may be 0 or 1, is the start of a UCS character
sequence.
UTF-8 Multibyte Codes
Bytes
Bits
Hex Minimum
Hex Maximum
Byte Sequence
in Binary
1
7
00000000
0000007F
0xxxxxxx
2
11
00000080
000007FF
110xxxxx
10xxxxxx
3
16
00000800
0000FFFF
1110xxxx
10xxxxxx
10xxxxxx
4
21
00010000
001FFFFF
11110xxx
10xxxxxx
10xxxxxx
10xxxxxx
5
26
00200000
03FFFFFF
111110xx
10xxxxxx
10xxxxxx
10xxxxx
10xxxxxx
6
31
04000000
7FFFFFFF
1111110x
10xxxxxx
10xxxxxx
10xxxxxx
10xxxxxx
10xxxxxx
The UCS value is just the concatenation of the x bits in the multibyte encoding.
When there are multiple ways to encode a value (for example, UCS 0), only the
shortest encoding is legal.
The following subset of UTF-8 is used to encode UCS-2:
UTF-8 Multibyte Codes
Bytes
Bits
Hex Minimum
Hex Maximum
Byte Sequence
in Binary
1
7
00000000
0000007F
0xxxxxxx
2
11
00000080
000007FF
110xxxxx
10xxxxxx
3
16
00000800
0000FFFF
1110xxxx
10xxxxxx
10xxxxxx
This subset of UTF-8 requires a maximum of three (3) bytes.
Converters Overview for Programming
National Language Support (NLS) provides a base for internationalization in which
data often can be changed from one code set to another. Support of several
standard converters for this purpose is provided. This section discusses the
following aspects of conversion:
v “Converters Introduction” on page 424
Chapter 16. National Language Support
423
v
v
v
v
“Standard Converters”
“Understanding libiconv” on page 425
“Using Converters” on page 428
“List of Converters” on page 429
Converters Introduction
Data sent by one program to another program residing on a remote host may
require conversion from the code set of the source machine to that of the receiver.
For example, when communicating with a VM system, the workstation converts its
ISO8859-1 data to an EBCDIC form.
Code sets define graphic characters and control character assignments to code
points. These coded characters must also be converted when a program obtains
data in one code set but displays it in another code set.
Two interfaces for conversions are provided:
iconv command
libiconv command
Allows you to request a specific conversion by naming the
FromCode and ToCode code sets.
Allow applications to request converters by name.
The system provides ready-to-use libraries of converters. You supply the name of
the converter you want to use. The converter libraries are found in the
/usr/lib/nls/loc/iconv/* and /usr/lib/nls/loc/iconvTable/* directories.
In addition to code set converters, the converter library also provides a set of
network interchange converters. In a network environment, the code sets of the
communications systems and the protocols of communication determine how the
data should be converted.
Interchange converters are used to convert data sent from one system to another.
Conversions from one internal code set to another require code set converters.
When data must be converted from a sender’s code set to a receiver’s code set or
from 8-bit data to 7-bit data, a uniform interface is required. The iconv subroutines
provide this interface.
Standard Converters
The system supports standard converters for use with the iconv command and
subroutines. The following list describes the different types of converters:
Code Set Converter Types
Table converter (“List of PC, ISO, and
EBCDIC Code Set Converters” on
page 430)
Algorithmic converter (“List of
Multibyte Code Set Converters” on
page 434)
Interchange Converter Types
7-bit converter (“List of Interchange
Converters—7-bit” on page 438)
8-bit converter (“List of Interchange
Converters—8-bit” on page 440)
424
General Programming Concepts
Description
Converts single-byte stateless code sets. Performs a
table translation from one byte to another byte.
Performs a conversion that cannot be implemented
using a simple single-byte mapping table. All
multibyte converters are currently implemented in
this way.
Description
Converts between internal code sets and
ISO2022 standard interchange formats (7-bit).
Converts between internal code sets and
ISO2022 standard interchange formats (8-bit).
Interchange Converter Types
Compound Text converter (“List of
Interchange Converters—Compound Text”
on page 443)
uucode converter (“List of Interchange
Converters—uucode” on page 445)
UCS-2 converters (“List of UCS-2
Interchange Converters” on page 447)
UTF-8 converters (“List of UTF-8
Interchange Converters” on page 449)
Miscellaneous Converters
Miscellaneous converters (“List of
Miscellaneous Converters” on
page 450)
Description
Converts between compound text and internal
code sets.
Provides the same mapping as that defined in
the uuencode and uudecode command.
Converts between UCS-2 and other code sets.
Converts between UTF-8 and other code sets.
Description
Used by some of the converters listed above.
Understanding libiconv
The iconv application programming interface (API) consists of three subroutines
that accomplish conversion:
iconv_open
iconv
iconv_close
Performs the initialization required to convert characters from the code set
specified by the FromCode parameter to the code set specified by the
ToCode parameter. The strings specified are dependent on the converters
installed in the system. If initialization is successful, the converter
descriptor, iconv_t, is returned in its initial state.
Invokes the converter function using the descriptor obtained from the
iconv_open subroutine. The inbuf parameter points to the first character in
the input buffer, and the inbytesleft parameter indicates the number of
bytes to the end of the buffer being converted. The outbuf parameter
points to the first available byte in the output buffer, and the outbytesleft
parameter indicates the number of available bytes to the end of the buffer.
For state-dependent encoding, the subroutine is placed in its initial state
by a call for which the inbuf value is a null pointer. Subsequent calls with
the inbuf parameter as something other than a null pointer cause the
internal state of the function to be altered as necessary.
Closes the conversion descriptor specified by the cd variable and makes it
usable again.
In a network environment, two factors determine how data should be converted:
v Code sets of the sender and the receiver
v Communication protocol (8-bit or 7-bit data)
The following table outlines the conversion methods and recommends how you
should convert data in different situations. See the “List of Interchange
Converters—7-bit” on page 438 and the “List of Interchange Converters—8-bit” on
page 440 for more information.
Outline of Methods and Recommended Choices
Method to
choose
Communication with system using
the same code set
Communication with system using
different code set or receiver’s code
set is unknown
Protocol
Protocol
7-bit only
8-bit
7-bit only
8-bit
Chapter 16. National Language Support
425
as is
Not valid
Best choice
Not valid
Not valid if
remote code set
is unknown
fold7
OK
OK
Best choice
OK
fold8
Not valid
OK
Not valid
Best choice
uucode
Best choice
OK
Not valid
Not valid
If the sender uses the same code set as the receiver, there are two possibilities:
v When protocol allows 8-bit data, the data can be sent without conversions.
v When protocol allows only 7-bit data, the 8-bit code points must be mapped to
7-bit values. Use the iconv interface and one of the following methods:
uucode method
fold7 method
Provides the same mapping as the uuencode and uudecode
commands. This is the recommended method.
Converts internal code sets using 7-bit data. This method passes ASCII
without any change.
If the sender uses a code set different from the receiver, there are two possibilities:
v When protocol allows only 7-bit data, use the fold7 method.
v When protocol allows 8-bit data and you know the receiver’s code set, use the
iconv interface to convert the data. If you do not know the receiver’s code set,
use the following method:
fold8 method
Converts internal code sets to standard interchange formats. The 8-bit
data is transmitted and the information is preserved so that the receiver
can reconstruct the data in its code set.
Using the iconv_open Subroutine
The following examples illustrate how to use the iconv_open subroutine in
different situations:
v Sender and receiver use the same code sets:
If the protocol allows 8-bit data, you can send data without converting it.
If the protocol allows only 7-bit data, do the following:
Sender:
cd = iconv_open("uucode", nl_langinfo(CODESET));
Receiver:
cd = iconv_open(nl_langinfo(CODESET), "uucode");
v Sender and receiver use different code sets:
If the protocol allows 8-bit data and the receiver’s code set is unknown, do the
following:
Sender:
cd = iconv_open("fold8", nl_langinfo(CODESET));
Receiver:
cd = iconv_open(nl_langinfo(CODESET),"fold8" );
If the protocol allows only 7-bit data, do the following:
Sender:
cd = iconv_open("fold7", nl_langinfo(CODESET));
Receiver:
cd = iconv_open(nl_langinfo(CODESET), "fold7" );
How the iconv_open Subroutine Finds Converters
The iconv_open subroutine uses the LOCPATH environment variable to search for
a converter whose name is in the form:
iconv/FromCodeSet_ToCodeSet
426
General Programming Concepts
The FromCodeSet string represents the sender’s code set, and the ToCodeSet string
represents the receiver’s code set. The underscore character separates the two
strings.
Note: All setuid and setgid programs will ignore the LOCPATH environment
variable.
Since the iconv converter is a loadable object module, a different object is required
when running in the 64-bit environment. In the 64-bit environment, the iconv_open
routine will use the LOCPATH environment variable to search for a converter
whose name is in the form:
iconv/FromCodeSet_ToCodeSet__64.
The iconv library will automatically choose whether to load the standard converter
object or the 64-bit converter object.
If the iconv_open subroutine does not find the converter, it uses the from,to pair to
search for a file that defines a table-driven conversion. The file contains a
conversion table created by the genxlt command.
The iconvTable converter uses the LOCPATH environment variable to search for a
file whose name is in the form:
iconvTable/FromCodeSet_ToCodeSet
If the converter is found, it performs a load operation and is initialized. The
converter descriptor, iconv_t, is returned in its initial state.
Converter Programs versus Tables
Converter programs are executable functions that convert data according to a set of
rules. Converter tables are single-byte conversion tables that perform stateless
conversions. Programs and tables are in separate directories:
/usr/lib/nls/loc/iconv
/usr/lib/nls/loc/iconvTable
Converter programs
Converter tables.
After a converter program is compiled and linked with the libiconv.a library, the
program is placed in the /usr/lib/nls/loc/iconv directory.
To build a table converter, build a source converter table file. Use the genxlt
command to compile translation tables into a format understood by the table
converter. The output file is then placed in the /usr/lib/nls/loc/iconvTable directory.
Unicode and Universal Converters
Unicode (or UCS-2) conversion tables are found in:
$LOCPATH/uconvTable/*CodeSet*
The $LOCPATH/uconv/UCSTBL converter program is used to perform the
conversion to and from UCS-2 using the iconv utilities. For the iconv utilities to
use these uconvTable conversion tables, links must be set up within the
$LOCPATH/iconv directory, for example, for code set ″X.″
ln -s /usr/lib/nls/loc/uconv/UCSTBL /usr/lib/nls/loc/iconv/X_UCS-2
ln -s /usr/lib/nls/loc/uconv/UCSTBL /usr/lib/nls/loc/iconv/UCS-2_X
A ″Universal converter″ program is provided that can be used to convert between
any two code sets whose conversions to and from UCS-2 is defined. Given the
following uconvTables:
Chapter 16. National Language Support
427
X
-> UCS-2
UCS-2 -> Y
a universal conversion can be defined that maps
X -> UCS-2 -> Y
by use of the $LOCPATH/iconv/Universal_UCS_Conv. The conversion X->Y is set
by defining links to the universal converter, for example:
ln-s /usr/lib/nls/loc/iconv/Universal_UCS_Conv /usr/lib/nls/loc/iconv/X_Y
Using Converters
The iconv interface is a set of subroutines used to open, perform, and close
conversions:
v iconv_open
v iconv
v iconv_close
Code Set Conversion Filter Example
The following example shows how you can use these subroutines to create a code
set conversion filter that accepts the ToCode and FromCode parameters as input
arguments:
#include <stdio.h>
#include <nl_types.h>
#include <iconv.h>
#include <string.h>
#include <errno.h>
#include <locale.h>
#define ICONV_DONE() (r>=0)
#define ICONV_INVAL() (r<0) && (errno==EILSEQ))
#define ICONV_OVER() (r<0) && (errno==E2BIG))
#define ICONV_TRUNC() (r<0) && (errno==EINVAL))
#define USAGE 1
#define ERROR 2
#define INCOMP 3
char ibuf[BUFSIZ], obuf[BUFSIZ];
extern int errno;
main (argc,argv)
int argc;
char **argv;
{
size_t ileft,oleft;
nl_catd catd;
iconv_t cd;
int r;
char *ip,*op;
setlocale(LC_ALL,"");
catd = catopen (argv[0],0);
if(argc!=3){
fprintf(stderr,
catgets (catd,NL_SETD,USAGE,"usage;conv fromcode tocode\n"));
exit(1);
}
cd=iconv_open(argv[2],argv[1]);
ileft=0;
while(!feof(stdin)) {
/*
* After the next operation,ibuf will
* contain new data plus any truncated
* data left from the previous read.
*/
ileft+=fread(ibuf+ileft,1,BUFSIZ-ileft,stdin);
428
General Programming Concepts
do {
ip=ibuf;
op=obuf;
oleft=BUFSIZ;
r=iconv(cd,&ip,&ileft,&op,&oleft);
if(ICONV_INVAL()){
fprintf(stderr,
catgets(catd,NL_SETD,ERROR,"invalid input\n"));
exit(2);
}
fwrite(obuf,1,BUFSIZ-oleft,stdout);
if(ICONV_TRUNC() || ICONV_OVER())
/*
*Data remaining in buffer-copy
*it to the beginning
*/
memcpy(ibuf,ip,ileft);
/*
*loop until all characters in the input
*buffer have been converted.
*/
} while(ICONV_OVER());
}
if(ileft!=0){
/*
*This can only happen if the last call
*to iconv() returned ICONV_TRUNC, meaning
*the last data in the input stream was
*incomplete.
*/
fprintf(stderr,catgets(catd,NL_SETD,INCOMP,"input incomplete\n"));
exit(3);
}
iconv_close(cd);
exit(0);
}
Naming Converters
Code set names are in the form CodesetRegistry-CodesetEncoding where:
CodesetRegistry
CodesetEncoding
Identifies the registration authority for the encoding. The
CodesetRegistry must be made of characters from the portable code
set (usually A-Z and 0-9).
Identifies the coded character set defined by the registered
authority.
The from,to variable used by the iconv command and iconv_open subroutine
identifies a file whose name should be in the form /usr/lib/nls/loc/iconv/%f_%t or
/usr/lib/nls/loc/iconvTable/%f_%t, where:
%f
%t
Represents the FromCode set name.
Represents the ToCode set name.
List of Converters
Converters change data from one code set to another. The sets of converters
supported with the ICONV library are in the following sections. All converters
shipped with the BOS Runtime Environment are located in the
/usr/lib/nls/loc/iconv/* or /usr/lib/nls/loc/iconvTable/* directory.
These directories also contain private converters; that is, they are used by other
converters. However, users and programs should only depend on the converters in
the following lists.
Chapter 16. National Language Support
429
Any converter shipped with the BOS Runtime Environment and not listed here
should be considered private and subject to change or deletion. Converters
supplied by other products can be placed in the /usr/lib/nls/loc/iconv/* or
/usr/lib/nls/loc/iconvTable/* directory.
Programmers are encouraged to use registered code set names or code set names
associated with an application. The X Consortium maintains a registry of code set
names for reference. See the “Code Set Overview” on page 387 for more
information about code sets.
v “List of PC, ISO, and EBCDIC Code Set Converters”
v “List of Multibyte Code Set Converters” on page 434
v “List of Interchange Converters—7-bit” on page 438
v “List of Interchange Converters—8-bit” on page 440
v “List of Interchange Converters—Compound Text” on page 443)
v “List of Interchange Converters—uucode” on page 445
v “List of UCS-2 Interchange Converters” on page 447
v “List of UTF-8 Interchange Converters” on page 449
v “List of Miscellaneous Converters” on page 450
List of PC, ISO, and EBCDIC Code Set Converters
These converters provide conversion between PC, ISO, and EBCDIC single-byte
stateless code sets. The following types of conversions are supported: PC to/from
ISO, PC to/from EBCDIC, and ISO to/from EBCDIC.
Conversion is provided between compatible code sets such as Latin-1 to Latin-1
and Greek to Greek. However, conversion between different EBCDIC national code
sets is not supported. For information about converting between incompatible
character sets refer to the “List of Interchange Converters—7-bit” on page 438 and
the “List of Interchange Converters—8-bit” on page 440.
Conversion tables in the iconvTable directory are created by the genxlt command.
Compatible Code Set Names: The following table lists code set names that are
compatible. Each line defines to/from strings that may be used when requesting a
converter.
Note: The PC and ISO code sets are ASCII-based.
Code Set Compatibility
430
Character Set
Languages
Latin-1
ISO
EBCDIC
U.S. English,
IBM-850
Portuguese,
Canadian French
ISO8859-1
IBM-037
Latin-1
Danish,
Norwegian
IBM-850
ISO8859-1
IBM-277
Latin-1
Finnish, Swedish IBM-850
ISO8859-1
IBM-278
Latin-1
Italian
IBM-850
ISO8859-1
IBM-280
Latin-1
Japanese
IBM-850
ISO8859-1
IBM-281
Latin-1
Spanish
IBM-850
ISO8859-1
IBM-284
Latin-1
U.K. English
IBM-850
ISO8859-1
IBM-285
General Programming Concepts
PC
Latin-1
German
IBM-850
ISO8859-1
IBM-273
Latin-1
French
IBM-850
ISO8859-1
IBM-297
Latin-1
Belgian, Swiss
German
IBM-850
ISO8859-1
IBM-500
Latin-2
IBM-852
Croatian,
Czechoslovakian,
Hungarian,
Polish,
Romanian,
Serbian Latin,
Slovak, Slovene
ISO88859-2
IBM-870
Cyrillic
Bulgarian,
Macedonian,
Serbian Cyrillic,
Russian
IBM-855
ISO8859-5
IBM-880
IBM-1025
Cyrillic
Russian
IBM-866
ISO8859-5
IBM-1025
Hebrew
Hebrew
IBM-856
IBM-862
ISO8859-8
IBM-424
IBM-803
Turkish
Turkish
IBM-857
ISO8859-9
IBM-1026
Arabic
Arabic
IBM-864
IBM-1046
ISO8859-6
IBM-420
Greek
Greek
IBM-869
ISO8859-7
IBM-875
Greek
Greek
IBM-869
ISO8859-7
IBM-875
Baltic
Lithuanian,
Latvian,
Estonian
IBM-921
IBM-922
IBM-1112
IBM-1122
Note: A character that exists in the source code set but does not exist in the
target code set is converted to a converter-defined substitute character.
Files: The following table describes the inconvTable converters found in the
/usr/lib/nls/loc/iconvTable directory:
iconvTable Converters
Converter Table
Description
Language
IBM-037_IBM-850
IBM-037 to IBM-850
U.S. English, Portuguese,
Canadian-French
IBM-273_IBM-850
IBM-273 to IBM-850
German
IBM-277_IBM-850
IBM-277 to IBM-850
Danish, Norwegian
IBM-278_IBM-850
IBM-278 to IBM-850
Finnish, Swedish
IBM-280_IBM-850
IBM-280 to IBM-850
Italian
IBM-281_IBM-850
IBM-281 to IBM-850
Japanese-Latin
IBM-284_IBM-850
IBM-284 to IBM-850
Spanish
IBM-285_IBM-850
IBM-285 to IBM-850
U.K. English
IBM-297_IBM-850
IBM-297 to IBM-850
French
IBM-420_IBM_1046
IBM-420 to IBM-1046
Arabic
IBM-424_IBM-856
IBM-424 to IBM-856
Hebrew
IBM-424_IBM-862
IBM-424 to IBM-862
Hebrew
IBM-500_IBM-850
IBM-500 to IBM-850
Belgian, Swiss German
Chapter 16. National Language Support
431
432
IBM-803_IBM-856
IBM-803 to IBM-856
Hebrew
IBM-803_IBM-862
IBM-803 to IBM-862
Hebrew
IBM-850_IBM-037
IBM-850 to IBM-037
U.S. English, Portuguese,
Canadian-French
IBM-850_IBM-273
IBM-850 to IBM-273
German
IBM-850_IBM-277
IBM-850 to IBM-277
Danish, Norwegian
IBM-850_IBM-278
IBM-850 to IBM-278
Finnish, Swedish
IBM-850_IBM-280
IBM-850 to IBM-280
Italian
IBM-850_IBM-281
IBM-850 to IBM-281
Japanese-Latin
IBM-850_IBM-284
IBM-850 to IBM-284
Spanish
IBM-850_IBM-285
IBM-850 to IBM-285
U.K. English
IBM-850_IBM-297
IBM-850 to IBM-297
French
IBM-850_IBM-500
IBM-850 to IBM-500
Belgian, Swiss German
IBM-856_IBM-424
IBM-856 to IBM-424
Hebrew
IBM-856_IBM-803
IBM-856 to IBM-803
Hebrew
IBM-856_IBM-862
IBM-856 to IBM-862
Hebrew
IBM-862_IBM-424
IBM-862 to IBM-424
Hebrew
IBM-862_IBM-803
IBM-862 to IBM-803
Hebrew
IBM-862_IBM-856
IBM-862 to IBM-856
Hebrew
IBM-864_IBM-1046
IBM-864 to IBM-1046
Arabic
IBM-921_IBM-1112
IBM-921 to IBM-1112
Lithuanian, Latvian
IBM-922_IBM-1122
IBM-922 to IBM-1122
Estonian
IBM-1112_IBM-921
IBM-1121 to IBM-921
Lithuanian, Latvian
IBM-1122_IBM-922
IBM-1122 to IBM-922
Estonian
IBM-1046_IBM-420
IBM-1046 to IBM-420
Arabic
IBM-1046_IBM-864
IBM-1046 to IBM-864
Arabic
IBM-037_ISO8859-1
IBM-037 to ISO8859-1
U.S. English, Portuguese,
Canadian French
IBM-273_ISO8859-1
IBM-273 to ISO8859-1
German
IBM-277_ISO8859-1
IBM-277 to ISO8859-1
Danish, Norwegian
IBM-278_ISO8859-1
IBM-278 to ISO8859-1
Finnish, Swedish
IBM-280_ISO8859-1
IBM-280 to ISO8859-1
Italian
IBM-281_ISO8859-1
IBM-281 to ISO8859-1
Japanese-Latin
IBM-284_ISO8859-1
IBM-284 to ISO8859-1
Spanish
IBM-285_ISO8859-1
IBM-285 to ISO8859-1
U.K. English
IBM-297_ISO8859-1
IBM-297 to ISO8859-1
French
IBM-420_ISO8859-6
IBM-420 to ISO8859-6
Arabic
IBM-424_ISO8859-8
IBM-424 to ISO8859-8
Hebrew
IBM-500_ISO8859-1
IBM-500 to ISO8859-1
Belgian, Swiss German
IBM-803_ISO8859-8
IBM-803 to ISO8859-8
Hebrew
General Programming Concepts
IBM-852_ISO8859-2
IBM-852 to ISO8859-2
Croatian, Czechoslovakian,
Hungarian, Polish,
Romanian, Serbian Latin,
Slovak, Slovene
IBM-855_ISO8859-5
IBM-855 to ISO8859-5
Bulgarian, Macedonian,
Serbian Cyrillic, Russian
IBM-866_ISO8859-5
IBM-866 to ISO8859-5
Russian
IBM-869_ISO8859-7
IBM-869 to ISO8859-7
Greek
IBM-875_ISO8859-7
IBM-875 to ISO8859-7
Greek
IBM-870_ISO8859-2
IBM-870 to ISO8859-2
Croatian, Czechoslovakian,
Hungarian, Polish,
Romanian, Serbian, Slovak,
Slovene
IBM-880_ISO8859-5
IBM-880 to ISO8859-5
Bulgarian, Macedonian,
Serbian Cyrillic, Russian
IBM-1025_ISO8859-5
IBM-1025 to ISO8859-5
Bulgarian, Macedonian,
Serbian Cyrillic, Russian
IBM-857_ISO8859-9
IBM-857 to ISO8859-9
Turkish
IBM-1026_ISO8859-9
IBM-1026 to ISO8859-9
Turkish
IBM-850_ISO8859-1
IBM-850 to ISO8859-1
Latin
IBM-856_ISO8859-8
IBM-856 to ISO8859-8
Hebrew
IBM-862_ISO8859-8
IBM-862 to ISO8859-8
Hebrew
IBM-864_ISO8859-6
IBM-864 to ISO8859-6
Arabic
IBM-1046_ISO8859-6
IBM-1046 to ISO8859-6
Arabic
ISO8859-1_IBM-850
ISO8859-1 to IBM-850
Latin
ISO8859-6_IBM-864
ISO8859-6 to IBM-864
Arabic
ISO8859-6_IBM-1046
ISO8859-6 to IBM-1046
Arabic
ISO8859-8_IBM-856
ISO8859-8 to IBM-856
Hebrew
ISO8859-8_IBM-862
ISO8859-8 to IBM-862
Hebrew
ISO8859-1_IBM-037
ISO8859-1 to IBM-037
U.S. English, Portuguese,
Canadian French
ISO8859-1_IBM-273
ISO8859-1 to IBM-273
German
ISO8859-1_IBM-277
ISO8859-1 to IBM-277
Danish, Norwegian
ISO8859-1_IBM-278
ISO8859-1 to IBM-278
Finnish, Swedish
ISO8859-1_IBM-280
ISO8859-1 to IBM-280
Italian
ISO8859-1_IBM-281
ISO8859-1 to IBM-281
Japanese-Latin
ISO8859-1_IBM-284
ISO8859-1 to IBM-284
Spanish
ISO8859-1_IBM-285
ISO8859-1 to IBM-285
U.K. English
ISO8859-1_IBM-297
ISO8859-1 to IBM-297
French
ISO8859-1_IBM-500
ISO8859-1 to IBM-500
Belgian, Swiss German
ISO8859-2_IBM-852
ISO8859-2 to IBM-852
Croatian, Czechoslovakian,
Hungarian, Polish,
Romanian, Serbian Latin,
Slovak, Slovene
Chapter 16. National Language Support
433
ISO8859-2_IBM-870
ISO8859-2 to IBM-870
Croatian, Czechoslovakian,
Hungarian, Polish,
Romanian, Serbian Latin,
Slovak, Slovene
ISO8859-5_IBM-855
ISO8859-5 to IBM-855
Bulgarian, Macedonian,
Serbian Cyrillic, Russian
ISO8859-5_IBM-880
ISO8859-5 to IBM-880
Bulgarian, Macedonian,
Serbian Cyrillic, Russian
ISO8859-5_IBM-1025
ISO8859-5 to IBM-1025
Bulgarian, Macedonian,
Serbian Cyrillic, Russian
ISO8859-6_IBM-420
ISO8859-6 to IBM-420
Arabic
ISO8859-5_IBM-866
ISO8859-5 to IBM-866
Russian
ISO8859-7_IBM-869
ISO8859-7 to IBM-869
Greek
ISO8859-7_IBM-875
ISO8859-7 to IBM-875
Greek
ISO8859-8_IBM-424
ISO8859-8 to IBM-424
Hebrew
ISO8859-8_IBM-803
ISO8859-8 to IBM-803
Hebrew
ISO8859-9_IBM-857
ISO8859-9 to IBM-857
Turkish
ISO8859-9_IBM-1026
ISO8859-9 to IBM-1026
Turkish
List of Multibyte Code Set Converters
Multibyte code-set converters convert characters among the following code-sets:
v PC multibyte code sets
v EUC multibyte code sets (ISO-based)
v EBCDIC multibyte code sets
The following table lists code set names that are compatible. Each line defines
to/from strings that may be used when requesting a converter.
Code Set Compatibility
Language
PC
ISO
EBCDIC
Japanese
IBM-932
IBM-eucJP
IBM-930, IBM-939
Japanese
(MS compatible)
IBM-943
IBM-eucJP
IBM-930, IBM-939
Korean
IBM-934
IBM-eucKR
IBM-933
Traditional Chinese
IBM-938, big-5
IBM-eucTW
IBM-937
Simplified Chinese
IBM-1381
IBM-eucCN
IBM-935
1. Conversions between Simplified and Traditional Chinese are provided
(IBM-eucTW <—> IBM-eucCN and big5 <—> IBM-eucCN).
2. UTF-8 is an additional code set. See “List of UTF-8 Interchange Converters” on
page 449 for more information.
Files: The following list describes the Multibyte Code Set converters that are
found in the /usr/lib/nls/loc/iconv directory.
434
Converter
Description
IBM-eucJP_IBM-932
IBM-eucJP to IBM-932
IBM-eucJP_IBM-943
IBM-eucJP to IBM-943
IBM-eucJP_IBM-930
IBM-eucJP to IBM-930
General Programming Concepts
Converter
Description
IBM-eucCN_IBM-936(PC5550)
IBM-eucCN to IBM-936(PC5550)
IBM-eucCN_IBM-935
IBM-eucCN to IBM-935
IBM-eucJP_IBM-939
IBM-eucJP to IBM-939
IBM-eucCN_IBM-1381
IBM-eucCN to IBM-1381
IBM-943_IBM-932
IBM-943 to IBM-932
IBM-932_IBM-943
IBM-932 to IBM-943
IBM-930_IBM-932
IBM-930 to IBM-932
IBM-930_IBM-943
IBM-930 to IBM-943
IBM-930_IBM-eucJP
IBM-930 to IBM-eucJP
IBM-932_IBM-eucJP
IBM-932 to IBM-eucJP
IBM-932_IBM-930
IBM-932 to IBM-930
IBM-943_IBM-eucJP
IBM-943 to IBM-eucJP
IBM-943_IBM-930
IBM-943 to IBM-930
IBM-936(PC5550)_IBM-935
IBM-936(PC5550) to IBM-935
IBM-936_IBM-935
IBM-936 to IBM-935
IBM-932_IBM-939
IBM-932 to IBM-939
IBM-939_IBM-932
IBM-939 to IBM-932
IBM-943_IBM-939
IBM-943 to IBM-939
IBM-939_IBM-943
IBM-939 to IBM-943
IBM-935_IBM-936(PC5550)
IBM-935 to IBM-936(PC5550)
IBM-935_IBM-936
IBM-935 to IBM-936
IBM-1381_IBM-935
IBM-1381 to IBM-935
IBM-935_IBM-1381
IBM-935 to IBM-1381
IBM-935_IBM-eucCN
IBM-935 to IBM-eucCN
IBM-936(PC5550)_IBM-eucCN
IBM-936(PC5550) to IBM-eucCN
IBM-eucTW_IBM-eucCN
IBM-eucTW to IBM-eucCN
big5_IBM-eucCN
big5 to IBM-eucCN
IBM-1381_IBM-eucCN
IBM-1381 to IBM-eucCN
IBM-939_IBM-eucJP
IBM-939 to IBM-eucJP
IBM-eucKR_IBM-934
IBM-eucKR to IBM-934
IBM-934_IBM-eucKR
IBM-934 to IBM-eucKR
IBM-eucKR_IBM-933
IBM-eucKR to IBM-933
IBM-933_IBM-eucKR
IBM-933 to IBM-eucKR
IBM-eucTW_IBM-937
IBM-eucTW to IBM-937
IBM-938_IBM-937
IBM-938 to IBM-937
big-5_IBM-937
big-5 to IBM-937
IBM-eucCN_IBM-eucTW
IBM-eucCN to IBM-eucTW
IBM-937_IBM-eucTW
IBM-937 to IBM-eucTW
IBM-937_IBM-938
IBM-937 to IBM-938
IBM-eucTW_IBM-938
IBM_eucTW to IBM_938
IBM-eucCN_big5
IBM-eucCN to big5
Chapter 16. National Language Support
435
436
Converter
Description
IBM-eucTW_big-5
IBM_eucTW to big-5
IBM-937_big-5
IBM-937 to big-5
CNS11643.1992-3_IBM-eucTW
CNS11643.1992-3 to IBM_eucTW
CNS11643.1992-3-GL_IBM-eucTW
CNS11643.1992-3-GL to IBM_eucTW
CNS11643.1992-3-GR_IBM-eucTW
CNS11643.1992-3-GR to IBM_eucTW
CNS11643.1992-4_IBM-eucTW
CNS11643.1992-4 to IBM_eucTW
CNS11643.1992-4-GL_IBM-eucTW
CNS11643.1992-4-GL to IBM_eucTW
CNS11643.1992-4-GR_IBM-eucTW
CNS11643.1992-4-GR to IBM_eucTW
IBM-eucTW_CNS11643.1992-3
IBM_eucTW to CNS11643.1992-3
IBM-eucTW_CNS11643.1992-3-GL
IBM_eucTW to CNS11643.1992-3-GL
IBM-eucTW_CNS11643.1992-3-GR
IBM_eucTW to CNS11643.1992-3-GR
IBM-eucTW_CNS11643.1992-4
IBM_eucTW to CNS11643.1992-4
IBM-eucTW_CNS11643.1992-4-GL
IBM_eucTW to CNS11643.1992-4-GL
IBM-eucTW_CNS11643.1992-4-GR
IBM_eucTW to CNS11643.1992-4-GR
IBM-eucCN_GB2312.1980-1
IBM-eucCN to GB2312.1980-1
IBM-eucCN_GB2312.1980-1-GL
IBM-eucCN to GB2312.1980-1-GL
IBM-eucCN_GB2312.1980-1-GR
IBM-eucCN to GB2312.1980-1-GR
IBM-937_csic
IBM-937 to csic
csic_IBM-937
csic to IBM-937
IBM-938_csic
IBM-938 to csic
csic_IBM-938
csic to IBM-938
IBM-eucTW_ccdc
IBM-eucTW to ccdc
ccdc_IBM-eucTW
ccdc to IBM-eucTW
IBM-eucTW_cns
IBM-eucTW to cns
cns_IBM-eucTW
cnd to IBM-eucTW
IBM-eucTW_csic
IBM-eucTW to csic
csic_IBM-eucTW
csic to IBM-eucTW
IBM-eucTW_sops
IBM-ecuTW to sops
sops_IBM-eucTW
sops to IBM-eucTW
IBM-eucTW_tca
IBM-eucTW to tca
tca_IBM-eucTW
tca to IBM-eucTW
big5_cns
big5 to cns
cns_big5
cns to big5
big5_csic
big5 to csic
csic_big5
csic to big5
big5_ttc
big5 to ttc
ttc_big5
ttc to big5
big5_ttcmin
big5 to ttcmin
ttcmin_big5
ttcmin to big5
big5_unicode
big5 to unicode
unicode_big5
unicode to big5
General Programming Concepts
Converter
Description
big5_wang
big5 to wang
wang_big5
wang to big5
ccdc_csic
ccdc to csic
csic_ccdc
csic to_ccdc
csic_sops
csic to sops
sops_csic
sops to csic
CNS11643.1986-1_big5
CNS11643.1986-1 to big5
big5_CNS11643.1986-1
big5 to CNS11643.1986-1
CNS11643.1986-1-GR_big5
CNS11643.1986-1-GR to big5
big5_CNS11643.1986-1-GR
big5 to CNS11643.1986-1-GR
CNS11643.1986-2_big5
CNS11643.1986-2 to big5
big5_CNS11643.1986-2
big5 to CNS11643.1986-2
CNS11643.1986-2-GR_big5
CNS11643.1986-2-GR to big5
big5_CNS11643.1986-2-GR
big5 to CNS11643.1986-2-GR
CNS11643.CT-GR_big5
CNS11643.CT-GR to big5
big5_CNS11643.CT-GR
big5 to CNS11643.CT-GR
IBM-sbdTW-GR_big5
IBM-sbdTW-GR to big5
big5_IBM-sbdTW-GR
big5 to IBM-sbdTW-GR
IBM-sbdTW.CT-GR_big5
IBM-sbdTW.CT-GR to big5
big5_IBM-sbdTW.CT-GR
big5 to IBM-sbdTW.CT-GR
IBM-sbdTW_big5
IBM-sbdTW to big5
big5_IBM-sbdTW
big5 to IBM-sbdTW
IBM-udcTW-GR_big5
IBM-udcTW-GR to big5
big5_IBM-udcTW-GR
big5 to IBM-udcTW-GR
IBM-udcTW.CT-GR_big5
IBM-udcTW.CT-GR to big5
big5_IBM-udcTW.CT-GR
big5 to IBM-udcTW.CT-GR
ISO8859-1_big5
ISO8859 to big5
big5_ISO8859-1
big5 to ISO8859
IBM-sbdTW_big5
IBM-sbdTW to big5
big5_IBM-sbdTW
big5 to IBM-sbdTW
big5_ASCII-GR
big5 to ASCII-GR
ASCII-GR_big5
ASCII-GR to big5
GBK_big5
GBK to big5
big5_GBK
big5 to GBK
GBK_IBM-eucTW
GBK to IBM-eucTW
IBM-eucTW_GBK
IBM-eucTW to GBK
CNS11643.1986-1_GBK
CNS11643.1986-1 to GBK
GBK_CNS11643.1986-1
GBK to CNS11643.1986-1
CNS11643.1986-2_GBK
CNS11643.1986-2 to GBK
GBK_CNS11643.1986-2
GBK to CNS11643.1986-2
CNS11643.1986-1-GR_GBK
CNS11643.1986-1-GR to GBK
Chapter 16. National Language Support
437
Converter
Description
GBK_CNS11643.1986-1-GR
GBK to CNS11643.1986-1-GR
CNS11643.1986-2-GR_GBK
CNS11643.1986-2-GR to GBK
GBK_CNS11643.1986-2-GR
GBK to CNS11643.1986-2-GR
CNS11643.1986-1-GL_GBK
CNS11643.1986-1-GL to GBK
GBK_CNS11643.1986-1-GL
GBK to CNS11643.1986-1-GL
CNS11643.1986-2-GL_GBK
CNS11643.1986-2-GL to GBK
GBK_CNS11643.1986-2-GL
GBK to CNS11643.1986-2-GL
CNS11643.CT-GR_GBK
CNS11643.CT-GR to GBK
GBK_CNS11643.CT-GR
GBK to CNS11643.CT-GR
GB2312.1980.CT-GR_GBK
GB2312.1980.CT-GR to GBK
GBK_GB2312.1980.CT-GR
GBK to GB2312.1980.CT-GR
GB2312.1980-0_GBK
GBK2312.1980-0 to GBK
GBK_GB2312.1980-0
GBK to GBK2312.1980-0
GB2312.1980-0-GR_GBK
GB2312.1980-0-GR to GBK
GBK_GB2312.1980-0-GR
GBK to GB2312.1980-0-GR
GB2312.1980-0-GL_GBK
GB2312.1980-0-GL to GBK
GBK_GB2312.1980-0-GL
GBK to GB2312.1980-0-GL
ASCII-GR_GBK
ASCII-GR to GBK
GBK_ASCII-GR
GBK to ASCII-GR
ISO8859-1_GBK
ISO8859-1 to GBK
GBK_ISO8859-1
GBK to ISO8859-1
IBM-eucCN_GBK
IBM-eucCN to GBK
GBK_IBM-eucCN
GBK to IBM-eucCN
List of Interchange Converters—7-bit
This converter provides conversion between internal code and 7-bit standard
interchange formats (fold7). The fold7 name identifies encodings that can be used
to pass text data through 7-bit mail protocols. The encodings are based on ISO2022.
For more information about fold7 see “Understanding libiconv” on page 425.
The fold7 converters convert characters from a code set to a canonical 7-bit
encoding that identifies each character. This type of conversion is useful in
networks where clients communicate with different code sets but use the same
character sets. For example:
IBM-850 <—> ISO8859-1
IBM-932 <—>IBM-eucJP
Common Latin characters
Common Japanese characters
The following escape sequences designate standard code sets:
438
Escape Sequence
Standard Code Set
01/11 02/04 04/00
GL JIS X0208.1978-0.
01/11 02/04 02/08 04/01
GL left half of GB2312.1980-0.
01/11 02/08 04/02
GL 7-bit ASCII or left half of ISO8859-1.
01/11 02/14 04/01
GL right half of ISO8859-1.
General Programming Concepts
Escape Sequence
Standard Code Set
01/11 02/14 04/02
GL right half of ISO8859-2.
01/11 02/14 04/03
GL right half of ISO8859-3.
01/11 02/14 04/04
GL right half of ISO8859-4.
01/11 02/14 04/06
GL right half of ISO8859-7.
01/11 02/14 04/07
GL right half of ISO8859-6.
01/11 02/14 04/08
GL right half of ISO8859-8.
01/11 02/14 04/12
GL right half of ISO8859-5.
01/11 02/14 04/13
GL right half of ISO8859-9.
01/11 02/08 04/09
GL right half of JIS X0201.1976-0.
01/11 02/08 04/10
GL left half of JIS X0201.1976.
01/11 02/04 04/02
GL JIS X0208.1983-0.
01/11 02/04 02/08 04/02
GL JIS X0208.1983-0.
01/11 02/04 02/08 04/00
GL JISX0208.1978-0.
01/11 02/05 02/15 03/01 M L 06/09 06/02 06/13 GL right half of IBM-850 unique characters.
02/13 03/08 03/05 03/00 00/02
Characters common to ISO8859-1 do not use
this escape sequence.
01/11 02/05 02/15 03/02 M L 06/09 06/02 06/13 GL Japanese) IBM-udcJP) user-definable
02/13 07/05 06/04 06/03 04/10 05/00 00/02
characters.
01/11 02/04 02/08 04/03
GL KSC5601-1987.
01/11 02/04 02/09 03/00
GL CNS11643-1986-1.
01/11 02/04 02/10 03/01
GL CNS11643-1986-2.
01/11 02/05 02/15 03/00 M L 05/05 05/04 04/06 UCS-2 encoded as base64; used only for
02/13 03/07 00/02
those characters not encoded by any of the
other 7-bit escape sequences listed above.
When converting from a code set to fold7, the escape sequence used to designate
the code set is chosen according to the order listed. For example, the
JISX0208.1983-0 characters use 01/11 01/04 04/02 as the designation.
Files: The following list describes the fold7 converters that are found in the
/usr/lib/nls/loc/iconv directory:
Converter
Description
fold7_IBM-850
Interchange format to IBM-850
fold7_IBM-921
Interchange format to IBM-921
fold7_IBM-922
Interchange format to IBM-922
fold7_IBM-932
Interchange format to IBM-932
fold7_IBM-943
Interchange format to IBM-943
fold7_IBM_1124
Interchange format to IBM-1124
fold7_IBM_1129
Interchange format to IBM-1129
fold7_IBM_eucCN
Interchange format to IBM-eucCN
fold7_IBM-eucJP
Interchange format to IBM-eucJP
fold7_IBM-eucKR
Interchange format to IBM-eucKR
fold7_IBM-eucTW
Interchange format to IBM-eucTW
fold7_ISO8859-1
Interchange format to ISO8859-1
Chapter 16. National Language Support
439
Converter
Description
fold7_ISO8859-2
Interchange format to ISO8859-2
fold7_ISO8859-3
Interchange format to ISO8859-3
fold7_ISO8859-4
Interchange format to ISO8859-4
fold7_ISO8859-5
Interchange format to ISO8859-5
fold7_ISO8859-6
Interchange format to ISO8859-6
fold7_ISO8859-7
Interchange format to ISO8859-7
fold7_ISO8859-8
Interchange format to ISO8859-8
fold7_ISO8859-9
Interchange format to ISO8859-9
fold7_TIS-620
Interchange format to TIS-620
fold7_UTF-8
Interchange format to UTF-8
fold7_big5
Interchange format to big5
fold7_GBK
Interchange format to GBK
IBM-921_fold7
IBM-921 to interchange format
IBM-922_fold7
IBM-922 to interchange format
IBM-850_fold7
IBM-850 to interchange format
IBM-932_fold7
IBM-932 to interchange format
IBM-943_fold7
IBM-943 to interchange format
IBM-1124_fold7
IBM-1124 to interchange format
IBM-1129_fold7
IBM-1129 to interchange format
IBM-eucCN_fold7
IBM-eucCN to interchange format
IBM-eucJP_fold7
IBM-eucJP to interchange format
IBM-eucKR_fold7
IBM-eucKR to interchange format
IBM-eucTW_fold7
IBM-eucTW to interchange format
ISO8859-1_fold7
ISO8859-1 to interchange format
ISO8859-2_fold7
ISO8859-2 to interchange format
ISO8859-3_fold7
ISO8859-3 to interchange format
ISO8859-4_fold7
ISO8859-4 to interchange format
ISO8859-5_fold7
ISO8859-5 to interchange format
ISO8859-6_fold7
ISO8859-6 to interchange format
ISO8859-7_fold7
ISO8859-7 to interchange format
ISO8859-8_fold7
ISO8859-8 to interchange format
ISO8859-9_fold7
ISO8859-9 to interchange format
TIS-620_fold7
TIS-620 to interchange format
UTF-8_fold7
UTF-8 to interchange format
big5_fold7
big5 to interchange format
GBK_fold7
GBK to interchange format
List of Interchange Converters—8-bit
This converter provides conversions between internal code and 8-bit standard
interchange formats (fold8). The fold8 name identifies encodings that can be used
to pass text data through 8-bit mail protocols. The encodings are based on ISO2022.
For more information about fold8 see “Understanding libiconv” on page 425.
440
General Programming Concepts
The fold8 converters convert characters from a specific code set encoding to a
canonical 8-bit encoding that identifies each character. This type of conversion is
useful in networks where clients communicate with different code sets but use the
same character sets. For example:
IBM-850 <—> ISO8859-1
IBM-932 <—>IBM-eucJP
Common Latin characters
Common Japanese characters
The following escape sequences designate standard code sets.
Escape Sequence
Standard Code Set
01/11 02/04 02/09 04/01
GR right half of GB2312.1980-0.
01/11 02/13 04/01
GR right half of ISO8859-1.
01/11 02/13 04/02
GR right half of ISO8859-2.
01/11 02/13 04/03
GR right half of ISO8859-3.
01/11 02/13 04/04
GR right half of ISO8859-4.
01/11 02/13 04/06
GR right half of ISO8859-7.
01/11 02/13 04/07
GR right half of ISO8859-6.
01/11 02/13 04/08
GR right half of ISO8859-8.
01/11 02/13 04/13
GR right half of ISO8859-5.
01/11 02/13 04/13
GR right half of ISO8859-9.
01/11 02/09 04/09
GR right half of JIS X0201.1976-1.
01/11 02/04 02/09 04/02
GR JIS X0208.1983-1.
01/11 02/04 02/09 04/00
GR JISX0208.1978-1.
01/11 02/09 04/02
GR 7-bit ASCII or left half of ISO8859-1.
01/11 02/05 02/15 03/01 M L 04/09 04/02 04/13 GR right half of IBM-850 unique characters.
02/13 03/08 03/05 03/00 00/02
Characters common to ISO8859-1 should not
use this escape sequence.
01/11 02/05 02/15 03/02 M L 04/09 04/02 04/13 GR right half of Japanese user-definable
02/13 07/05 06/04 06/03 04/10 05/00 00/02
characters.
01/11 02/08 04/02
GL 7-bit ASCII or left half of ISO8859-1.
01/11 02/14 04/01
GL right half of ISO8859-1.
01/11 02/14 04/02
GL right half of ISO8859-2.
01/11 02/14 04/03
GL right half of ISO8859-3.
01/11 02/14 04/04
GL right half of ISO8859-4.
01/11 02/14 04/06
GL right half of ISO8859-7.
01/11 02/14 04/07
GL right half of ISO8859-6.
01/11 02/14 04/08
GL right half of ISO8859-8.
01/11 02/14 04/12
GL right half of ISO8859-5.
01/11 02/14 04/13
GL right half of ISO8859-9.
01/11 02/08 04/09
GL right half of JIS X0201.1976-0.
01/11 02/08 04/10
GL left half of JIS X0201.1976.
01/11 02/04 02/08 04/02
GL JIS X0208.1983-0.
01/11 02/04 04/02
GL JIS X0208.1983-0.
01/11 02/04 04/00
GL JIS X0208.1978-0.
Chapter 16. National Language Support
441
Escape Sequence
Standard Code Set
01/11 02/05 02/15 03/01 M L 06/09 06/02 06/13 GL right half of IBM-850 unique characters.
02/13 03/08 03/05 03/00 00/02
Characters common to ISO8859-1 do not use
this escape sequence.
01/11 02/05 02/15 03/02 M L 06/09 06/02 06/13 GL Japanese (IBM-udcJP) user-definable
02/13 07/05 06/04 06/03 04/10 05/00 00/02
characters.
01/11 02/04 02/09 04/03
GR KSC5601-1987.
01/11 02/04 02/09 03/00
GR CNS11643-1986-1.
01/11 02/04 02/10 03/01
GR CNS11643-1986-2.
01/11 02/05 02/15 03/02 M L 04/09 04/02 04/13 GR right half of Traditional Chinese
02/13 07/05 06/04 06/03 05/05 05/08 00/02
user-definable characters.
01/11 02/05 02/15 03/02 M L 04/09 04/02 04/13 GR right half of IBM-850 unique symbols.
02/13 07/03 06/02 06/04 05/05 05/08 00/02
GL KSC5601-1987.
01/11 02/04 02/08 04/03
01/11 02/05 02/15 03/02 M L 06/09 06/02 06/13 GL Traditional Chinese (IBM-udcTW)
02/13 07/05 06/04 06/03 05/05 05/08 00/02
user-definable characters.
01/11 02/05 02/15 03/02 M L 06/09 06/02 06/13 GL Traditional Chinese IBM-850 unique
02/13 07/03 06/02 06/04 05/05 05/08 00/02
symbols (IBM-shdTW) user-definable
characters.
01/11 02/05 02/15 03/00 M L 05/05 05/04 04/06 UCS-2 encoded as UTF-8; used only for
02/13 03/08 00/02
those characters not encoded by any of the
above escape sequences listed above.
When converting from a code set to fold8, the escape sequence used to designate
the code set is chosen according to the order listed. For example, the
JISX0208.1983-0 characters use 01/11 02/04 02/08 04/02 as the designation.
Files: The following list describes the fold8 converters found in the
/usr/lib/nls/loc/iconv directory:
442
Converter
Description
fold8_IBM-850
Interchange format to IBM-850
fold8_IBM-921
Interchange format to IBM-921
fold8_IBM-922
Interchange format to IBM-922
fold8_IBM-932
Interchange format to IBM-932
fold8_IBM-943
Interchange format to IBM-943
fold8_IBM-1124
Interchange format to IBM-1124
fold8_IBM-1129
Interchange format to IBM-1129
fold8_IBM-eucCN
Interchange format to IBM-eucCN
fold8_IBM-eucJP
Interchange format to IBM-eucJP
fold8_IBM-eucKR
Interchange format to IBM-eucKR
fold8_IBM-eucTW
Interchange format to IBM-eucTW
fold8_IBM-eucCN
Interchange fromat to IBM-eucCN
fold8_ISO8859-1
Interchange format to ISO8859-1
fold8_ISO8859-2
Interchange format to ISO8859-2
fold8_ISO8859-3
Interchange format to ISO8859-3
fold8_ISO8859-4
Interchange format to ISO8859-4
General Programming Concepts
Converter
Description
fold8_ISO8859-5
Interchange format to ISO8859-5
fold8_ISO8859-6
Interchange format to ISO8859-6
fold8_ISO8859-7
Interchange format to ISO8859-7
fold8_ISO8859-8
Interchange format to ISO8859-8
fold8_ISO8859-9
Interchange format to ISO8859-9
fold8_TIS-620
Interchange format to TIS-620
fold8_UTF-8
Interchange format to UTF-8
fold8_big5
Interchange format to big5
fold8_GBK
Interchange format to GBK
IBM-921_fold8
IBM-921 to interchange format
IBM-922_fold8
IBM-922 to interchange format
IBM-850_fold8
IBM-850 to interchange format
IBM-932_fold8
IBM-932 to interchange format
IBM-943_fold8
IBM-943 to interchange format
IBM-1124_fold8
IBM-1124 to interchange format
IBM-1129_fold8
IBM-1129 to interchange format
IBM-eucCN_fold8
IBM-eucCN to interchange format
IBM-eucJP_fold8
IBM-eucJP to interchange format
IBM-eucKR_fold8
IBM-eucKR to interchange format
IBM-eucTW_fold8
IBM-eucTW to interchange format
IBM-eucCN_fold8
IBM-eucCN to interchange format
ISO8859-1_fold8
ISO8859-1 to interchange format
ISO8859-2_fold8
ISO8859-2 to interchange format
ISO8859-3_fold8
ISO8859-3 to interchange format
ISO8859-4_fold8
ISO8859-4 to interchange format
ISO8859-5_fold8
ISO8859-5 to interchange format
ISO8859-6_fold8
ISO8859-6 to interchange format
ISO8859-7_fold8
ISO8859-7 to interchange format
ISO8859-8_fold8
ISO8859-8 to interchange format
ISO8859-9_fold8
ISO8859-9 to interchange format
TIS-620_fold8
TIS-620 to interchange format
UTF-8_fold8
UTF-8 to interchange format
big5_fold8
big5 to interchange format
GBK_fold8
GBK to interchange format
List of Interchange Converters—Compound Text
Compound text interchange converters convert between compound text and
internal code sets.
Compound text is an interchange encoding defined by the X Consortium. It is used
to communicate text between X clients. Compound text is based on ISO2022 and
can encode most character sets using standard escape sequences. It also provides
Chapter 16. National Language Support
443
extensions for encoding private character sets. The supported code sets provide a
converter to and from compound text. The name used to identify the compound
text encoding is ct.
The following escape sequences are used to designate standard code sets in the
order listed below.
01/11 02/05 02/15 03/01 M L 04/09 04/02 04/13 02/13 03/08 03/05 03/00 00/02
GR right half of IBM-850 unique characters.
Characters common to ISO8859-1 should not
use this escape sequence.
01/11 02/05 02/15 03/02 M L 04/09 04/02 04/13 02/13 07/05 06/04 06/03 04/10 05/00 00/02
GR right half of Japanese user-definable
characters.
01/11 02/05 02/15 03/01 M L 06/09 06/02 06/13 02/13 03/08 03/05 03/00 00/02
GL right half of IBM-850 unique characters.
Characters common to ISO8859-1 do not use
this escape sequence.
01/11 02/05 02/15 03/02 M L 06/09 06/02 06/13 02/13 07/05 06/04 06/03 04/10 05/00 00/02
GL Japanese (IBM-udcJP) user-definable
characters.
Files: The following list describes the compound text converters that are found in
the /usr/lib/nls/loc/iconv directory:
Converter
Description
ct_IBM-850
Interchange format to IBM-850
ct_IBM-921
Interchange format to IBM-921
ct_IBM-922
Interchange format to IBM-922
ct_IBM-932
Interchange format to IBM-932
ct_IBM-943
Interchange format to IBM-943
ct_IBM-1124
Interchange format to IBM-1124
ct_IBM-1129
Interchange format to IBM-1129
ct_IBM-eucCN
Interchange format to IBM-eucCN
ct_IBM-eucJP
Interchange format to IBM-eucJP
ct_IBM-eucKR
Interchange format to IBM-eucKR
ct_IBM-eucTW
Interchange format to IBM-eucTW
ct_ISO8859-1
Interchange format to ISO8859-1
ct_ISO8859-2
Interchange format to ISO8859-2
ct_ISO8859-3
Interchange format to ISO8859-3
ct_ISO8859-4
Interchange format to ISO8859-4
ct_ISO8859-5
Interchange format to ISO8859-5
ct_ISO8859-6
Interchange format to ISO8859-6
ct_ISO8859-7
Interchange format to ISO8859-7
ct_ISO8859-8
Interchange format to ISO8859-8
ct_ISO8859-9
Interchange format to ISO8859-9
ct_TIS-620
444
Interchange format to TIS-620
ct_big5
Interchange format to big5
ct_GBK
Interchange format to GBK
General Programming Concepts
Converter
Description
IBM-850_ct
IBM-850 to interchange format
IBM-921_ct
IBM-921 to interchange format
IBM-922_ct
IBM-922 to interchange format
IBM-932_ct
IBM-932 to interchange format
IBM-943_ct
IBM-943 to interchange format
IBM-1124_ct
IBM-1124 to interchange format
IBM-1129_ct
IBM-1129 to interchange format
IBM-eucCN_ct
IBM-eucCN to interchange format
IBM-eucJP_ct
IBM-eucJP to interchange format
IBM-eucKR_ct
IBM-eucKR to interchange format
IBM-eucTW_ct
IBM-eucTW to interchange format
ISO8859-1_ct
ISO8859-1 to interchange format
ISO8859-2_ct
ISO8859-2 to interchange format
ISO8859-3_ct
ISO8859-3 to interchange format
ISO8859-4_ct
ISO8859-4 to interchange format
ISO8859-5_ct
ISO8859-5 to interchange format
ISO8859-6_ct
ISO8859-6 to interchange format
ISO8859-7_ct
ISO8859-7 to interchange format
ISO8859-8_ct
ISO8859-8 to interchange format
ISO8859-9_ct
ISO8859-9 to interchange format
TIS-620_ct
TIS-620 to interchange format
big5_ct
big5 to interchange format
GBK_ct
GBK to interchange format
List of Interchange Converters—uucode
This converter provides the same mapping as the uuencode and uudecode
Command.
During conversion from uucode, 62 bytes at a time (including a new-line character
trailing the record) are converted, and generating 45 bytes in outbuf.
Files: The following list describes the uucode converters found in the
/usr/lib/nls/loc/iconv directory:
Converter
Description
IBM-850_uucode
IBM-850 to uucode
IBM-921_uucode
IBM-921 to uucode
IBM-922_uucode
IBM-922 to uucode
IBM-932_uucode
IBM-932 to uucode
IBM-943_uucode
IBM-943 to uucode
IBM-1124_uucode
IBM-1124 to uucode
IBM-1129_uucode
IBM-1129 to uucode
IBM-eucJP_uucode
IBM-eucJP to uucode
Chapter 16. National Language Support
445
Converter
Description
IBM-eucKR_uucode
IBM-eucKR to uucode
IBM-eucTW_uucode
IBM-eucTW to uucode
IBM-eucCN_uucode
IBM-eucCN to uucode
ISO8859-1_uucode
ISO8859-1 to uucode
ISO8859-2_uucode
ISO8859-2 to uucode
ISO8859-3_uucode
ISO8859-3 to uucode
ISO8859-4_uucode
ISO8859-4 to uucode
ISO8859-5_uucode
ISO8859-5 to uucode
ISO8859-6_uucode
ISO8859-6 to uucode
ISO8859-7_uucode
ISO8859-7 to uucode
ISO8859-8_uucode
ISO8859-8 to uucode
ISO8859-9_uucode
ISO8859-9 to uucode
TIS-620_uucode
big5_uucode
big5 to uucode
GBK_uucode
GBK to uucode
uucode_IBM-850
uucode to IBM-850
uucode_IBM-921
uucode to IBM-921
uucode_IBM-922
uucode to IBM-922
uucode_IBM-932
uucode to IBM-932
uucode_IBM-943
uucode to IBM-943
uucode_IBM-1124
uucode to IBM-1124
uucode_IBM-1129
uucode to IBM-1129
uucode_IBM-eucCN
uucode to IBM-eucCN
uucode_IBM-eucJP
uucode to IBM-eucJP
uucode_IBM-eucKR
uucode to IBM-eucKR
uucode_IBM-eucTW
uucode to IBM-eucTW
uucode_ISO8859-1
uucode to ISO8859-1
uucode_ISO8859-2
uucode to ISO8859-2
uucode_ISO8859-3
uucode to ISO8859-3
uucode_ISO8859-4
uucode to ISO8859-4
uucode_ISO8859-5
uucode to ISO8859-5
uucode_ISO8859-6
uucode to ISO8859-6
uucode_ISO8859-7
uucode to ISO8859-7
uucode_ISO8859-8
uucode to ISO8859-8
uucode_ISO8859-9
uucode to ISO8859-9
uucode_TIS-1124
446
TIS-620 to uucode
uucode to TIS-1129
uucode_big5
uucode to big5
uucode_GBK
uucode to GBK
General Programming Concepts
List of UCS-2 Interchange Converters
UCS-2 is a universal, 16-bit encoding described in the “Code Set Overview” on
page 387. Conversions for each code set are provided in both directions, between
the code set and UCS-2.
UCS-2 converters are found in /usr/lib/nls/loc/uconvTable and
/usr/lib/nls/loc/uconv directories. The uconvdef command is used to generate new
converters or to customize existing UCS-2 converters.
The /usr/lib/nls/loc/iconv/Universal_UCS_Conv converter is used to generate
conversions from any code set X to code set Y by setting the proper links:
cd /usr/lib/nls/loc/iconv
ln -s /usr/lib/nls/loc/uconv/Universal_UCS_Conv X_Y
ln -s /usr/lib/nls/loc/uconv/UCSTBL X_UCS-2
ln-s /usr/lib/nls/loc/uconv/UCSTBL UCS-2_Y
ln -s /usr/lib/nls/loc/uconv/UCSTBL X
ln -s /usr/lib/nls/loc/uconv/UCSTBL Y
Converter
Description
ISO8859-1
UCS-2 <—> ISO Latin-1
ISO8859-2
UCS-2 <—> ISO Latin-2
ISO8859-3
UCS-2 <—> ISO Latin-3
ISO8859-4
UCS-2 <—> ISO Latin-4
ISO8859-5
UCS-2 <—> ISO Cyrillic
ISO8859-6
UCS-2 <—> ISO Arabic
ISO8859-7
UCS-2 <—> ISO Greek
ISO8859-8
UCS-2 <—> ISO Hebrew
ISO8859-9
UCS-2 <—> ISO Turkish
JISX0201.1976-0
UCS-2 <—> Japanese JISX0201-0
JISX0208.1983-0
UCS-2 <—> Japanese JISX0208-0
CNS11643.1986-1
UCS-2 <—> Chinese CNS11643-1
CNS11643.1986-2
UCS-2 <—> Chinese CNS11643-2
KSC5601.1987-0
UCS-2 <—> Korean KSC5601-0
IBM-eucCN
UCS-2 <—> Simplified Chinese EUC
IBM-udcCN
UCS-2 <—> Simplified Chinese user-defined characters
IBM-sbdCN
UCS-2 <—> Simplified Chinese IBM-specific characters
GB2312.1980-0
UCS-2 <—> Simplified Chinese GB
IBM-1381
UCS-2 <—> Simplified Chinese PC data code
IBM-935
UCS-2 <—> Simplified Chinese EBCDIC
IBM-936
UCS-2 <—> Simplified Chinese PC5550
IBM-eucJP
UCS-2 <—> Japanese EUC
IBM-eucKR
UCS-2 <—> Korean EUC
IBM-eucTW
UCS-2 <—> Traditional Chinese EUC
IBM-udcJP
UCS-2 <—> Japanese user-defined characters
IBM-udcTW
UCS-2 <—> Traditional Chinese user-defined characters
IBM-sbdTW
UCS-2 <—> Traditional Chinese IBM-specific characters
UTF-8
UCS-2 <—> UTF-8
Chapter 16. National Language Support
447
Converter
Description
IBM-437
UCS-2 <—> USA PC data code
IBM-850
UCS-2 <—> Latin-1 PC data code
IBM-852
UCS-2 <—> Latin-2 PC data code
IBM-857
UCS-2 <—> Turkish PC data code
IBM-860
UCS-2 <—> Portuguese PC data code
IBM-861
UCS-2 <—> Icelandic PC data code
IBM-863
UCS-2 <—> French Canadian PC data code
IBM-865
UCS-2 <—> Nordic PC data code
IBM-869
UCS-2 <—> Greek PC data code
IBM-921
UCS-2 <—> Baltic Multilingual data code
IBM-922
UCS-2 <—> Estonian data code
IBM-932
UCS-2 <—> Japanese PC data code
IBM-943
UCS-2 <—> Japanese PC data code
IBM-934
UCS-2 <—> Korea PC data code
IBM-936
UCS-2 <—> People’s Republic of China PC data code
IBM-938
UCS-2 <—> Taiwanese PC data code
IBM-942
UCS-2 <—> Extended Japanese PC data code
IBM-944
UCS-2 <—> Korean PC data code
IBM-946
UCS-2 <—> People’s Republic of China SAA data code
IBM-948
UCS-2 <—> Traditional Chinese PC data code
IBM-1124
UCS-2 <—> Ukranian PC data code
IBM-1129
UCS-2 <—> Vietnamese PC data code
TIS-620
UCS-2 <—> Thailand PC data code
IBM-037
UCS-2 <—> USA, Canada EBCDIC
IBM-273
UCS-2 <—> Germany, Austria EBCDIC
IBM-277
UCS-2 <—> Denmark, Norway EBCDIC
IBM-278
UCS-2 <—> Finland, Sweden EBCDIC
IBM-280
UCS-2 <—> Italy EBCDIC
IBM-284
UCS-2 <—> Spain, Latin America EBCDIC
IBM-285
UCS-2 <—> United Kingdom EBCDIC
IBM-297
UCS-2 <—> France EBCDIC
IBM-500
UCS-2 <—> International EBCDIC
IBM-875
UCS-2 <—> Greek EBCDIC
IBM-930
UCS-2 <—> Japanese Katakana-Kanji EBCDIC
IBM-933
UCS-2 <—> Korean EBCDIC
IBM-937
UCS-2 <—> Traditional Chinese EBCDIC
IBM-939
UCS-2 <—> Japanese Latin-Kanji EBCDIC
IBM-1026
UCS-2 <—> Turkish EBCDIC
IBM-1112
UCS-2 <—> Baltic Multilingual EBCDIC
IBM-1122
UCS-2 <—> Estonian EBCDIC
IBM-1124
448
General Programming Concepts
UCS-2 <—> Ukranian EBCDIC
Converter
IBM-1129
GBK
TIS-620
Description
UCS-2 <—> Vietnamese EBCDIC
UCS-2<—> Simplified Chinese
UCS-2 <—>Thailand EBCDIC
List of UTF-8 Interchange Converters
UTF-8 is a universal, multibyte encoding described in the “UCS-2 and UTF-8” on
page 420. Conversions for each code set are provided in both directions, between
the code set and UTF-8.
UTF-8 converters are usually done by using the Universal_UCS_Conv (see “List of
UCS-2 Interchange Converters” on page 447) and /usr/lib/nls/loc/uconv/UTF-8
conversion.
Converter
Description
ISO8859-1
UTF-8 <—> ISO Latin-1
ISO8859-2
UTF-8 <—> ISO Latin-2
ISO8859-3
UTF-8 <—> ISO Latin-3
ISO8859-4
UTF-8 <—> ISO Latin-4
ISO8859-5
UTF-8 <—> ISO Cyrillic
ISO8859-6
UTF-8 <—> ISO Arabic
ISO8859-7
UTF-8 <—> ISO Greek
ISO8859-8
UTF-8 <—> ISO Hebrew
ISO8859-9
UTF-8 <—> ISO Turkish
JISX0201.1976-0
UTF-8 <—> Japanese JISX0201-0
JISX0208.1983-0
UTF-8 <—> Japanese JISX0208-0
CNS11643.1986-1
UTF-8 <—> Chinese CNS11643-1
CNS11643.1986-2
UTF-8 <—> Chinese CNS11643-2
KSC5601.1987-0
UTF-8 <—> Korean KSC5601-0
IBM-eucCN
UTF-8 <—> Simplified Chinese EUC
IBM-eucJP
UTF-8 <—> Japanese EUC
IBM-eucKR
UTF-8 <—> Korean EUC
IBM-eucTW
UTF-8 <—> Traditional Chinese EUC
IBM-udcJP
UTF-8 <—> Japanese user-defined characters
IBM-udcTW
UTF-8 <—> Traditional Chinese user-defined characters
IBM-sbdTW
UTF-8 <—> Traditional Chinese IBM-specific characters
UCS-2
UTF-8 <—> UCS-2
IBM-437
UTF-8 <—> USA PC data code
IBM-850
UTF-8 <—> Latin-1 PC data code
IBM-852
UTF-8 <—> Latin-2 PC data code
IBM-857
UTF-8 <—> Turkish PC data code
IBM-860
UTF-8 <—> Portuguese PC data code
IBM-861
UTF-8 <—> Icelandic PC data code
IBM-863
UTF-8 <—> French Canadian PC data code
IBM-865
UTF-8 <—> Nordic PC data code
Chapter 16. National Language Support
449
Converter
Description
IBM-869
UTF-8 <—> Greek PC data code
IBM-921
UTF-8 <—> Baltic Multilingual data code
IBM-922
UTF-8 <—> Estonian data code
IBM-932
UTF-8 <—> Japanese PC data code
IBM-943
UTF-8 <—> Japanese PC data code
IBM-934
UTF-8 <—> Korea PC data code
IBM-935
UTF-8 <—> Simplified Chinese EBCDIC
IBM-936
UTF-8 <—> People’s Republic of China PC data code
IBM-938
UTF-8 <—> Taiwanese PC data code
IBM-942
UTF-8 <—> Extended Japanese PC data code
IBM-944
UTF-8 <—> Korean PC data code
IBM-946
UTF-8 <—> People’s Republic of China SAA data code
IBM-948
UTF-8 <—> Traditional Chinese PC data code
IBM-1124
UTF-8 <—> Ukranian PC data code
IBM-1129
UTF-8 <—> Vietnamese PC data code
TIS-620
UTF-8 <—> Thailand PC data code
IBM-037
UTF-8 <—> USA, Canada EBCDIC
IBM-273
UTF-8 <—> Germany, Austria EBCDIC
IBM-277
UTF-8 <—> Denmark, Norway EBCDIC
IBM-278
UTF-8 <—> Finland, Sweden EBCDIC
IBM-280
UTF-8 <—> Italy EBCDIC
IBM-284
UTF-8 <—> Spain, Latin America EBCDIC
IBM-285
UTF-8 <—> United Kingdom EBCDIC
IBM-297
UTF-8 <—> France EBCDIC
IBM-500
UTF-8 <—> International EBCDIC
IBM-875
UTF-8 <—> Greek EBCDIC
IBM-930
UTF-8 <—> Japanese Katakana-Kanji EBCDIC
IBM-933
UTF-8 <—> Korean EBCDIC
IBM-937
UTF-8 <—> Traditional Chinese EBCDIC
IBM-939
UTF-8 <—> Japanese Latin-Kanji EBCDIC
IBM-1026
UTF-8 <—> Turkish EBCDIC
IBM-1112
UTF-8 <—> Baltic Multilingual EBCDIC
IBM-1122
UTF-8 <—> Estonian EBCDIC
IBM-1124
UTF-8 <—> Ukranian EBCDIC
IBM-1129
UTF-8 <—> Vietnamese EBCDIC
IBM-1381
UTF-8 <—> Simplified Chinese PC data code
GBK
UTF-8<—> Simplified Chinese
TIS-620
UTF-8 <—> Thailand EBCDIC
List of Miscellaneous Converters
A set of low level converters used by the code set and interchange converters is
provided. These converters are called miscellaneous converters. These low-level
450
General Programming Concepts
converters may be used by some of the interchange converters. However, the use
of these converters is discouraged because they are intended for support of other
converters.
Files: The following list describes the miscellaneous converters found in the
/usr/lib/nls/loc/iconv and /usr/lib/nls/loc/iconvTable directories:
Converter
Description
IBM-932_JISX0201.1976-0
IBM-932 to JISX0201.1976-0
IBM-932_JISX0208.1983-0
IBM-932 to JISX0208.1983-0
IBM-932_IBM-udcJP
IBM-932 to IBM-udcJP (Japanese user-defined
characters)
IBM-943_JISX0201.1976-0
IBM-943 to JISX0201.1976-0
IBM-943_JISX0208.1983-0
IBM-943 to JISX0208.1983-0
IBM-943_IBM-udcJP
IBM-943 to IBM-udcJP (Japanese user-defined
characters
IBM-eucJP_JISX0201.1976-0
IBM-eucJP to JISX0201.1976-0
IBM-eucJP_JISX0208.1983-0
IBM-eucJP to JISX0208.1983-0
IBM-eucJP_IBM-udcJP
IBM-eucJP to IBM-udcJP (Japanese user-defined
characters)
IBM-eucKR_KSC5601.1987-0
IBM_eucKR to KSC5601.1987-0
IBM-eucTW_CNS11643.1986-1
IBM-eucTW to CNS11643.1986.1
IBM-eucTW_CNS11643.1986-2
IBM-eucTW to CNS11643.1986-2
IBM-eucCN_GB2312.1980-0
IBM-eucCN to GB2312.1980-0
Writing Converters Using the iconv Interface
This section provides a general background on the iconv subroutines and
structures in preparation for writing code set converters. This section gives an
overview of the control flow and the order in which the framework operates,
details about writing code set converters, and an example including the code,
header file, and a makefile. This section applies to the iconv framework within
AIX.
Under the framework of the iconv_open, iconv and iconv_close subroutines, you
can create and use several different types of converters. Applications can call these
subroutines to convert characters in one code set into characters in a different code
set. The access and use of the iconv_open, iconv and iconv_close subroutines is
standardized by X/Open XPG4.
Code Sets and Converters
Code sets can be classified into two categories: stateful encodings and stateless
encodings.
Stateful Code Sets and Converters
The stateful encodings use shift-in and shift-out codes to change state. For instance,
the shift-out can be used to indicate the start of host double-byte data in a data
stream of characters, and shift-in can be used to indicate the end of this
double-byte character data. When the double-byte data is off, it signals the start of
single-byte character data. An example of such a stateful code set is IBM-930 used
mainly on mainframes (hosts).
Chapter 16. National Language Support
451
Converters written to do the conversion of stateful encodings to other code sets
tend to be complex due to the extra processing needed.
Stateless Code Sets and Converters
The stateless code sets are those that can be classified as one of two types:
v Single-byte code sets, such as ISO8859 family (ISO8859-1, ISO8859-2, and so on)
v Multibyte code sets, such as IBM-eucJP (Japanese), IBM-932 (Shift-JIS).
Note that conversions are meaningful only if the code sets represent the same
characters.
The simplest types of code set conversion can be found in single-byte code set
converters, such as the converter from ISO8859-1 to IBM-850. These single-byte
code set converters are based on simple table-based conversions. The conversion of
multibyte character encodings, such as IBM-eucJP to IBM-932, are in general based
on an algorithm and not on tables, because the tables can get lengthy.
iconv Framework - Overview of Structures
The iconv framework consists of the iconv_open, iconv and iconv_close
subroutines. It is based on a common core structure that is part of all converters.
The core structure is initialized at the load time of the converter object module.
After the loading of the converter is complete, the main entry point, which is
always the instantiate subroutine, is invoked. This initializes the core structure and
returns the core converter descriptor. This is further used during the call to the init
subroutine provided by the converter to allocate the converter-specific structures.
This init subroutine returns another converter descriptor that has a pointer to the
core converter descriptor. The init subroutine allocates memory as needed and
may invoke other converters if needed. The init subroutine is the place for any
converter-specific initialization whereas the instantiate subroutine is a generic
entry point.
Once the converter descriptor for this converter is allocated and initialized, the
next step is to provide the actual code needed for the exec part of the functionality.
If the converter is a table-based converter, the only need is to provide a source file
format that conforms to the input needs of the genxlt utility, which takes this
source table as the input and generates an output file format usable by the iconv
framework.
iconv.h File and Structures
The iconv.h file in /usr/include defines the following structures:
typedef struct __iconv_rec iconv_rec, *iconv_t;
struct __iconv_rec {
_LC_object_t hdr;
iconv_t (*open)(const char *tocode, const char *fromcode);
size_t (*exec)(iconv_t cd, char **inbuf, size_t *inbytesleft,
char **outbuf, size_t *outbytesleft);
void (*close)(iconv_t cd);
};
The common core structure is as follows (/usr/include/iconv.h):
typedef struct _LC_core_iconv_type
_LC_core_iconv_t;
struct _LC_core_iconv_type {
_LC_object_t hdr;
/* implementation initialization */
452
General Programming Concepts
_LC_core_iconv_t
size_t (*exec)();
void
(*close)();
*(*init)();
};
Every converter has a static memory area which contains the _LC_core_iconv_t
structure. It is initialized in the instantiate subroutine provided as part of the
converter program.
iconv Control Flow
The following sections describe the iconv control flow.
The iconv_open Subroutine: An application invokes a code set converter by the
following call:
iconv_open(char *to_codeset, char *from_codeset)
The to and from code sets are used in selecting the converter by way of the search
path defined by the LOCPATH environment variable. The iconv_open subroutine
uses the _lc_load subroutine to load the object module specified by concatenating
the from and to code set names to the iconv_open subroutine.
CONVERTER NAME= "from_codeset" + "_" +"to_codeset"
If the from_codeset is IBM-850 and the to_codeset is ISO8859-1, the converter name is
IBM-850_ISO8859-1.
After loading the converter, its entry point is invoked by the _lc_load loader
subroutine. This is the first call to the converter. The instantiate subroutine then
initializes the _LC_core_iconv_t core structure. The iconv_open subroutine then
calls the init subroutine associated with the core structure thus returned. The init
subroutine is responsible for allocating the converter specific descriptor structure
and initializing it as needed by the converter. The iconv_open subroutine returns
this converter-specific structure. However, the return value is typecast to iconv_t in
the user’s application. Thus, the application does not see the whole of the
converter-specific structure; it sees only the public iconv_t structure. The converter
code itself uses the private converter structure. Applications that use iconv
converters should not change the converter descriptor; the converter descriptor
should be used as an opaque structure.
Entry Point: An entry point is declared in every converter such that when the
converter is opened by a call to the iconv_open subroutine, that entry point is
automatically invoked. The entry point is the instantiate subroutine that should be
provided in all converters. The entry point is specified in the makefile as follows:
LDENTRY=-einstantiate
When the converter is loaded on a call to iconv_open(), the instantiate subroutine
is invoked. This subroutine knows how the converter works. It initializes a static
core conversion descriptor structure _LC_core_iconv_t cd.
The core conversion descriptor cd contains pointers to the init, _iconv_exec, and
_iconv_close subroutines supplied by the specific converter. The instantiate
subroutine returns the core conversion descriptor to be used later. The
_LC_core_iconv_t structure is defined in /usr/include/iconv.h.
When the iconv_open subroutine is called, the following actions occur:
1. The converter is found using LOCPATH, the converter is loaded, and the
instantiate subroutine is invoked. On success, it returns the core conversion
Chapter 16. National Language Support
453
descriptor. (_LC_core_iconv_t *cd). The instantiate subroutine provided by the
converter is responsible for initializing the header in the core structure.
2. The iconv_open subroutine then invokes the init subroutine specified in the
core conversion descriptor. The init subroutine provided by the converter is
responsible for allocation of memory needed to hold the converter descriptor
needed for this specific converter. For example, the following may be the
structure needed by a stateless converter:
typedef struct _LC_sample_iconv_rec {
LC_core_iconv_t
core;
} _LC_sample_iconv_t;
To initialize this, the converter has to do the following in the
init subroutine:
static _LC_sample_iconv_t*
init (_LC_core_iconv_t *core_cd, char* toname,char* fromname)
{
_LC_sample_iconv_t
*cd;
/* converter descriptor */
/*
**
Allocate a converter descriptor
**/
if(!(cd = ( _LC_sample_iconv_t *) malloc (
sizeof(_LC_sample_iconv_t ))))
return (NULL);
/*
** Copy the core part of converter descriptor which is
** passed in
*/
cd->core = *core_cd;
/*
**
Return the converter descriptor
*/
return cd;
}
The iconv Subroutine: An application invokes the iconv subroutine to do the
actual code set conversions. The iconv subroutine invokes the exec subroutine in
the core structure.
The iconv_close Subroutine: An application invokes the iconv_close subroutine
to free any memory allocated for conversions. The iconv_close subroutine invokes
the close subroutine in the core structure.
Writing a Code Set Converter
This section gives details on how to write a converter using the concepts explained
so far. This is done starting with a simple converter and proceeds to a more
complex one. The following procedures are discussed here:
v How to write an algorithmic converter.
v How to write a table lookup converter.
v How to write a stateful code set converter.
Every converter should define the following subroutines:
v instantiate()
v init()
v iconv_exec()
v iconv_close()
The converter-specific structure should have the core iconv structure as its first
element. For example:
454
General Programming Concepts
Example 1:
typedef struct _LC_example_rec {
/* Core should be the first element */
_LC_core_iconv_t
core;
/* The rest are converter specific data (optional) */
iconv_t
curcd;
iconv_t
sb_cd;
iconv_t
db_cd;
unsigned char *cntl;
} _LC_example_iconv_t;
Example 2: A simpler converter structure
typedef struct _LC_sample_iconv_rec {
_LC_core_iconv_t
core;
} _LC_sample_iconv_t;
Stateless Converters - Algorithm Based
Every converter should have the subroutines previously specified. Only the
subroutine headers are provided without details, except for the instantiate
subroutine that is common to all converters and should be coded the same.
The following example of an algorithm-based stateless converter is a sample
converter of the IBM-850 code set to the ISO8859-1 code set.
#include <stdlib.h>
#include <iconv.h>
#include "850_88591.h"
/*
*
Name : _iconv_exec()
*
*
This contains actual conversion method.
*/
static size_t _iconv_exec(_LC_sample_iconv_t *cd,
unsigned char** inbuf,
size_t *inbytesleft,
unsigned char** outbuf,
size_t *outbytesleft)
/*
*
cd
: converter descriptor
*
inbuf
: input buffer
*
outbuf
: output buffer
*
inbytesleft
: number of data(in bytes) in input buffer
*
outbytesleft
: number of data(in bytes) in output buffer
*/
{
}
/*
*
Name : _iconv_close()
*
*
Free the allocated converter descriptor
*/
static void
_iconv_close(iconv_t cd)
{
}
/*
*
Name : init()
*
*
This allocates and initializes the converter descriptor.
*/
static _LC_sample_iconv_t
*init (_LC_core_iconv_t *core_cd,
char* toname, char* fromname)
{
}
/*
Chapter 16. National Language Support
455
*
Name : instantiate()
*
*
Core part of a converter descriptor is initialized here.
*/
_LC_core_iconv_t
*instantiate(void)
{
static _LC_core_iconv_t cd;
/*
* * Initialize _LC_MAGIC and _LC_VERSION are
** defined in <lc_core.h>. _LC_ICONV and _LC_core_iconv_t
** are defined in <iconv.h>.
*/
cd.hdr.magic = _LC_MAGIC;
cd.hdr.version = _LC_VERSION;
cd.hdr.type_id = _LC_ICONV;
cd.hdr.size = sizeof (_LC_core_iconv_t);
/*
*
Set pointers to each method.
*/
cd.init = init;
cd.exec = _iconv_exec;
cd.close = _iconv_close;
/*
*
Returns the core part
*/
return &cd;
}
Stateful Converters
Here, only the subroutine headers are provided without details, except for the
instantiate subroutine that is common to all converters and should be coded the
same. Because stateful converters need more information, they provide additional
converter-dependent information as well.
The following example of a stateful converter is a sample converter of IBM-930 to
IBM-932 code set.
The host.h file contains the following structure:
typedef struct _LC_host_iconv_rec {
_LC_core_iconv_t
core;
iconv_t
curcd;
iconv_t
sb_cd;
iconv_t
db_cd;
unsigned char *cntl;
} _LC_host_iconv_t;
#include
#include
#include
#include
<stdlib.h>
<sys/types.h>
<iconv.h>
"host.h"
/*
** The _iconv_exec subroutine to be invoked via cd->exec()
*/
static int
_iconv_exec(_LC_host_iconv_t *cd,
unsigned char **inbuf, size_t *inbytesleft,
unsigned char **outbuf, size_t *outbytesleft)
{
unsigned char
*in, *out;
int
ret_value;
if (!cd){
errno = EBADF; return NULL;
}
456
General Programming Concepts
if (!inbuf) {
cd->curcd = cd->sb_cd;
return ICONV_DONE;
}
do {
if ((ret_value = iconv(cd->curcd, inbuf, inbytesleft, outbuf,
outbytesleft)) != ICONV_INVAL)
return ret_value;
in = *inbuf;
out = *outbuf;
if (in[0] == SO) {
if (cd->curcd == cd->db_cd){
errno = EILSEQ;
return ICONV_INVAL;
}
cd->curcd = cd->db_cd;
}
else if (in[0] == SI) {
if (cd->curcd == cd->sb_cd){
errno = EILSEQ;
return ICONV_INVAL;
}
cd->curcd = cd->sb_cd;
}else if (in[0] <= 0x3f &&
cd->curcd == cd->sb_cd) {
if (*outbytesleft < 1){
errno = E2BIG;
return ICONV_OVER;
}
out[0] = cd->cntl[in[0]];
*outbuf = ++out;
(*outbytesleft)—;
}
else {
errno = EILSEQ; return ICONV_INVAL;
}
*inbuf = ++in;
(*inbytesleft)—;
} while (1);
}
/*
** The iconv_close subroutine is a macro accessing this
** subroutine as set in the core iconv structure.
*/
static void _iconv_close(_LC_host_iconv_t *cd)
{
if (cd) {
if (cd->sb_cd)
iconv_close(cd->sb_cd);
if (cd->db_cd)
iconv_close(cd->db_cd);
free(cd);
}else{
errno = EBADF;
}
}
/*
** The init subroutine to be invoked when iconv_open() is called.
*/
static _LC_host_iconv_t *init(_LC_core_iconv_t *core_cd,
char* toname, char* fromname)
{
_LC_host_iconv_t* cd;
int i;
for (i = 0; 1; i++) {
if (!_iconv_host[i].local)
return NULL;
Chapter 16. National Language Support
457
if (strcmp(toname, _iconv_host[i].local) == 0 &&
strcmp(fromname, _iconv_host[i].host) == 0)
break;
}
if (!(cd = (_LC_host_iconv_t *)
malloc(sizeof(_LC_host_iconv_t))))
return (NULL);
if (!(cd->sb_cd = iconv_open(toname, _iconv_host[i].sbcs))) {
free(cd);
return NULL;
}
if (!(cd->db_cd = iconv_open(toname, _iconv_host[i].dbcs))) {
iconv_close(cd->sb_cd);
free(cd);
return NULL;
}
cd->core = *core_cd;
cd->cntl = _iconv_host[i].fcntl;
cd->curcd = cd->sb_cd;
return cd;
}
/*
** The instantiate() method is called when iconv_open() loads the
** converter by a call to __lc_load().
*/
_LC_core_iconv_t
*instantiate(void)
{
static _LC_core_iconv_t
cd;
cd.hdr.magic = _LC_MAGIC;
cd.hdr.version = _LC_VERSION;
cd.hdr.type_id = _LC_ICONV;
cd.hdr.size = sizeof (_LC_core_iconv_t);
cd.init = init;
cd.exec = _iconv_exec;
cd.close = _iconv_close;
return &cd;
}
Examples
1. The following example provides sample code for a stateless converter that
performs an algorithm-based convertion of the IBM-850 code set to the
ISO8859-1 code set. The file name for this example is 850_88591.c.
#include <stdlib.h>
#include <iconv.h>
#include "850_88591.h"
#define DONE
0
/*
* Name : _iconv_exec()
*
* This contains actual conversion method.
*/
static size_t _iconv_exec(_LC_sample_iconv_t *cd,
unsigned char** inbuf, size_t *inbytesleft,
unsigned char** outbuf, size_t *outbytesleft)
/*
* cd
: converter descriptor
* inbuf
: input buffer
* outbuf
: output buffer
* inbytesleft : number of data(in bytes) in input buffer
* outbytesleft
: number of data(in bytes) in output buffer
*/
{
unsigned char *in; /* point the input buffer */
unsigned char *out; /* point the output buffer */
458
General Programming Concepts
unsigned char
unsigned char
*e_in; /* point the end of input buffer*/
*e_out; /* point the end of output buffer*/
/*
* If given converter discripter is invalid,
* it sets the errno and returns the number
* of bytes left to be converted.
*/
if (!cd) {
errno = EBADF;
return *inbytesleft;
}
/*
*
If the input buffer does not exist or there
*
is no character to be converted, it returns
*
0 (no characters to be converted).
*/
if (!inbuf || !(*inbytesleft))
return DONE;
/*
*
Set up pointers and initialize other variables
*/
e_in = (in = *inbuf) + *inbytesleft;
e_out = (out = *outbuf) + *outbytesleft;
/*
*
Perform code point conversion until all input
*
is consumed.
*
When error occurs (i.e. buffer overflow), error
*
number is set and exit this loop.
*/
while (in < e_in) {
/*
* If there is not enough space left in output buffer
* to hold the converted data, it stops converting and
* sets the errno to E2BIG.
*/
if (e_out <= out) {
errno = E2BIG;
break;
}
/*
* Convert the input data and store it into the output
* buffer, then advance the pointers which point to the
* buffers.
*/
*out++ = table[*in++];
} /* while */
/*
* Update the pointers to the buffers and
* input /output byte counts
*/
*inbuf = in;
*outbuf = out;
*inbytesleft = e_in - in;
*outbytesleft = e_out - out;
/*
* Reurn the number of bytes left to be converted
* (0 for successful conversion completion)
*/
return *inbytesleft;
}
/*
* Name : _iconv_close()
*
* Free the allocated converter descriptor
*/
static void _iconv_close(iconv_t cd)
{
Chapter 16. National Language Support
459
if (!cd)
free(cd);
else
/*
* If given converter is not valid,
* it sets the errno to EBADF
*/
errno = EBADF;
}
/*
* Name : init()
*
* This allocates and initializes the converter descriptor.
*/
static _LC_sample_iconv_t*
init (_LC_core_iconv_t *core_cd, char* toname, char* fromname)
{
_LC_sample_iconv_t *cd; /* converter descriptor */
/*
* Allocate a converter descriptor
*/
if (!(cd = (_LC_sample_iconv_t *)
malloc(sizeof(_LC_sample_iconv_t))))
return (NULL);
/*
*Copy the core part of converter descriptor which is passed
*/
cd->core = *core_cd;
/*
* Return the converter descriptor
*/
return cd;
}
/*
* Name : instantiate()
*
* Core part of a converter descriptor is initialized here.
*/
_LC_core_iconv_t* instantiate(void)
{
static _LC_core_iconv_t cd;
/*
* Initialize
* _LC_MAGIC and _LC_VERSION are defined in <lc_core.h>.
* _LC_ICONV and _LC_core_iconv_t are defined in <iconv.h>.
*/
cd.hdr.magic = _LC_MAGIC;
cd.hdr.version = _LC_VERSION;
cd.hdr.type_id = _LC_ICONV;
cd.hdr.size = sizeof (_LC_core_iconv_t);
/*
* Set pointers to each method.
*/
cd.init = init;
cd.exec = _iconv_exec;
cd.close = _iconv_close;
/*
* Returns the core part
*/
return &cd;
}
*in
2. The following example contains a sample header file named 850_88591.h.
#ifndef _ICONV_SAMPLE_H
#define _ICONV_SAMPLE_H
/*
*
Define _LC_sample_iconv_t
460
General Programming Concepts
*/
typedef struct _LC_sample_iconv_rec {
_LC_core_iconv_t core;
} _LC_sample_iconv_t;
static unsigned char
table[] = { /*
_______________________________________
|
|
|
IBM-850
ISO8859-1 |
|_______________________________________|
/*
0x00
*/
0x00,
/*
0x01
*/
0x01,
/*
0x02
*/
0x02,
/*
0x03
*/
0x03,
/*
0x04
*/
0x04,
/*
0x05
*/
0x05,
/*
0x06
*/
0x06,
/*
0x07
*/
0x07,
/*
0x08
*/
0x08,
/*
0x09
*/
0x09,
/*
0x0A
*/
0x0A,
/*
0x0B
*/
0x0B,
/*
0x0C
*/
0x0C,
/*
0x0D
*/
0x0D,
/*
0x0E
*/
0x0E,
/*
0x0F
*/
0x0F,
/*
0x10
*/
0x10,
/*
0x11
*/
0x11,
/*
0x12
*/
0x12,
/*
0x13
*/
0x13,
/*
0x14
*/
0x14,
/*
0x15
*/
0x15,
/*
0x16
*/
0x16,
/*
0x17
*/
0x17,
/*
0x18
*/
0x18,
/*
0x19
*/
0x19,
/*
0x1A
*/
0x1A,
/*
0x1B
*/
0x1B,
/*
0x1C
*/
0x1C,
/*
0x1D
*/
0x1D,
/*
0x1E
*/
0x1E,
/*
0x1F
*/
0x1F,
/*
0x20
*/
0x20,
/*
0x21
*/
0x21,
/*
0x22
*/
0x22,
/*
0x23
*/
0x23,
/*
0x24
*/
0x24,
/*
0x25
*/
0x25,
/*
0x26
*/
0x26,
/*
0x27
*/
0x27,
/*
0x28
*/
0x28,
/*
0x29
*/
0x29,
/*
0x2A
*/
0x2A,
/*
0x2B
*/
0x2B,
/*
0x2C
*/
0x2C,
/*
0x2D
*/
0x2D,
/*
0x2E
*/
0x2E,
/*
0x2F
*/
0x2F,
/*
0x30
*/
0x30,
/*
0x31
*/
0x31,
/*
0x32
*/
0x32,
/*
0x33
*/
0x33,
/*
0x34
*/
0x34,
/*
0x35
*/
0x35,
/*
0x36
*/
0x36,
/*
0x37
*/
0x37,
/*
0x38
*/
0x38,
/*
0x39
*/
0x39,
Chapter 16. National Language Support
461
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
462
General Programming Concepts
0x3A
0x3B
0x3C
0x3D
0x3E
0x3F
0x40
0x41
0x42
0x43
0x44
0x45
0x46
0x47
Ox48
0x49
0x4A
0x4B
0x4C
0x4D
0x4E
0x4F
0x50
0x51
0x52
0x53
0x54
0x55
0x56
0x57
0x58
0x59
0x5A
0x5B
0x5C
0x5D
0x5E
0x5F
0x60
0x61
0x62
0x63
0x64
0x65
0x66
0x67
0x68
0x69
0x6A
0x6B
0x6C
0x6D
0x6E
0x6F
0x70
0x71
0x72
0x73
0x74
0x75
0x76
0x77
0x78
0x79
0x7A
0x7B
0x7C
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
0x3A,
0x3B,
0x3C,
0x3D,
0x3E,
0x3F,
0x40,
0x41,
0x42,
0x43,
0x44,
0x45,
0x46,
0x47,
0x48,
0x49,
0x4A,
0x4B,
0x4C,
0x4D,
0x4E,
0x4F,
0x50,
0x51,
0x52,
0x53,
0x54,
0x55,
0x56,
0x57,
0x58,
0x59,
0x5A,
0x5B,
0x5C,
0x5D,
0x5E,
0x5F,
0x60,
0x61,
0x62,
0x63,
0x64,
0x65,
0x66,
0x67,
0x68,
0x69,
0x6A,
0x6B,
0x6C,
0x6D,
0x6E,
0x6F,
0x70,
0x71,
0x72,
0x73,
0x74,
0x75,
0x76,
0x77,
0x78,
0x79,
0x7A,
0x7B,
0x7C,
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
0x7D
0x7E
0x7F
0x80
0x81
0x82
0x83
0x84
0x85
0x86
0x87
0x88
0x89
0x8A
0x8B
0x8C
0x8D
0x8E
0x8F
0x90
0x91
0x92
0x93
0x94
0x95
0x96
0x97
0x98
0x99
0x9A
0x9B
0x9C
0x9D
0x9E
0x9F
0xA0
0xA1
0xA2
0xA3
0xA4
0xA5
0xA6
0xA7
0xA8
0xA9
0xAA
0xAB
0xAC
0xAD
0xAE
0xAF
0xB0
0xB1
0xB2
0xB3
0xB4
0xB5
0xB6
0xB7
0xB8
0xB9
0xBA
0xBB
0xBC
0xBD
0xBE
0xBF
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
0x7D,
0x7E,
0x7F,
0xC7,
0xFC,
0xE9,
0xE2,
0xE4,
0xE0,
0xE5,
0xE7,
0xEA,
0xEB,
0xE8,
0xEF,
0xEE,
0xEC,
0xC4,
0xC5,
0xC9,
0xE6,
0xC6,
0xF4,
0xF6,
0xF2,
0xFB,
0xF9,
0xFF,
0xD6,
0xDC,
0xF8,
0xA3,
0xD8,
0xD7,
0x1A,
0xE1,
0xED,
0xF3,
0xFA,
0xF1,
0xD1,
0xAA,
0xBA,
0xBF,
0xAE,
0xAC,
0xBD,
0xBC,
0xA1,
0xAB,
0xBB,
0x1A,
0x1A,
0x1A,
0x1A,
0x1A,
0xC1,
0xC2,
0xC0,
0xA9,
0x1A,
0x1A,
0x1A,
0x1A,
0xA2,
0xA5,
0x1A,
Chapter 16. National Language Support
463
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
};
#endif
464
General Programming Concepts
0xC0
0xC1
0xC2
0xC3
0xC4
0xC5
0xC6
0xC7
0xC8
0xC9
0xCA
0xCB
0xCC
0xCD
0xCE
0xCF
0xD0
0xD1
0xD2
0xD3
0xD4
0xD5
0xD6
0xD7
0xD8
0xD9
0xDA
0xDB
0xDC
0xDD
0xDE
0xDF
0xE0
0xE1
0xE2
0xE3
0xE4
0xE5
0xE6
0xE7
0xE8
0xE9
0xEA
0xEB
0xEC
0xED
0xEE
0xEF
0xF0
0xF1
0xF2
0xF3
0xF4
0xF5
0xF6
0xF7
0xF8
0xF9
0xFA
0xFB
0xFC
0xFD
0xFE
0xFF
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
*/
0x1A,
0x1A,
0x1A,
0x1A,
0x1A,
0x1A,
0xE3,
0xC3,
0x1A,
0x1A,
0x1A,
0x1A,
0x1A,
0x1A,
0x1A,
0xA4,
0xF0,
0xD0,
0xCA,
0xCB,
0xC8,
0x1A,
0xCD,
0xCE,
0xCF,
0x1A,
0x1A,
0x1A,
0x1A,
0xA6,
0xCC,
0x1A,
0xD3,
0xDF,
0xD4,
0xD2,
0xF5,
0xD5,
0xB5,
0xFE,
0xDE,
0xDA,
0xDB,
0xD9,
0xFD,
0xDD,
0xAF,
0xB4,
0xAD,
0xB1,
0x1A,
0xBE,
0xB6,
0xA7,
0xF7,
0xB8,
0xB0,
0xA8,
0xB7,
0xB9,
0xB3,
0xB2,
0x1A,
0xA0,
3. The following example is a sample makefile.
SHELL = /bin/ksh
CFLAGS = $(COMPOPT) $(INCLUDE) $(DEFINES)
INCLUDE = -I.
COMPOPT =
DEFINES = -D_POSIX_SOURCE -D_XOPEN_SOURCE
CC
= /bin/xlc
LD
= /bin/ld
RM
= /bin/rm
SRC
= 850_88591.c
TARGET = 850_88591
ENTRY_POINT
= instantiate
$(TARGET) :
cc -e $(ENTRY_POINT) -o $(TARGET) $(SRC) -l iconv
clean :
$(RM) -f $(TARGET)
$(RM) -f *.o
Input Method Overview
For an application to run in the international environment for which National
Language Support (NLS) provides a base, input methods are needed. The Input
Method is an application programming interface (API) that allows you to develop
applications independent of a particular language, keyboard, or code set. Each type
of input method has the following features:
Keymaps
Keysyms
Modifiers
Set of input method keymaps (imkeymaps) that works with the input
method and determines the supported locales.
Set of key symbols (keysyms) that the input method can handle.
Set of modifiers or states, each having a mask value, that the input method
supports.
Input Method Introduction
An input method is a set of functions that translates key strokes into character
strings in the code set specified by your locale. Input method functions include
locale-specific input processing and keyboard controls (for example, Ctrl, Alt, Shift,
Lock, and Alt-Graphic). The input method allows various types of input, but only
keyboard events are dealt with here.
Your locale determines which input method should be loaded, how the input
method runs, and which devices are used. The input method then defines states
and their outcome.
When the input method translates a keystroke into a character string, the
translation process takes into account the keyboard and the code set you are using.
You may want to write your own input method if you do not have a standard
keyboard or if you customize your code set.
Many languages use a small set of symbols or letters to form words. To enter text
with a keyboard, you press keys that correspond to symbols of the alphabet. When
a character in your alphabet does not exist on the keyboard, you must press a
combination of keys. Input methods provide algorithms that allow you to compose
such characters.
Some languages use an ideographic writing system. They use a unique symbol,
rather than a group of letters, to represent a word. For instance, the character sets
used in China, Japan, Korea, and Taiwan have more than 5,000 characters.
Chapter 16. National Language Support
465
Consequently, more than one byte must be used to represent a character. Moreover,
a single keyboard cannot include all the required ideographic symbols. You need
input methods that can compose multibyte characters.
The /usr/lib/nls/loc directory contains the input methods installed on your system.
You can list the contents of this directory to determine which input methods are
available to you. Input method file names have the format Language_Territory.im.
For example, the fr_BE.im file is the input method file for the French language as
used in Belgium.
Through a well-structured protocol, input methods allow applications to support
different input without using locale-specific input processing.
In AIX, the input method is provided in the aixterm. When characters typed from
the AIXwindows interface reach the server, the characters are in the form of key
codes. These key codes are converted into keysyms as defined by the table
provided in the client. This table contains mappings for each of the key codes into
a predefined set of codes called the keysyms. Any key code generated by a
keyboard should have a keysym. These keysyms are maintained and allocated by
the MIT X Consortium. The keysyms are passed to the client aixterm terminal
emulator. In the aixterm, the input keysyms are converted into file codes by the
input method and are then sent to the application. The X server is designed to
work with the display adapter provided in the system hardware. The X server
communicates with the X client through sockets. Thus, the server and the client
can reside on different systems in a network, provided they can communicate with
each other. The data from the keyboard enters the X server, and from the server it
gets passed to the terminal emulator. The terminal emulator passes the data to the
application. When data comes from applications to the display device, it passes
through the terminal emulator by sockets to the server and from the server to the
display device. See the Input/Output Methods diagram for an illustration of this.
466
General Programming Concepts
Server
Network
Client
Application
aixterm
Input Method
Keysyms
X Server
ksh
vi
Key code
File codes
File codes
Key code
termio
Socket
lft
pty
Scan code
Scan code
lft
tty
Display
File code
Display
Key code to keysym
crt
crt
Keysym to file code
Keyboards
Key code to file code map
Input/Output Methods
Input Method Names
The set of input methods available depends on which locales have been installed
and what input methods those locales provide. The name of the input method
usually corresponds to the locale. For example, the Greek Input Method is named
el_GR, which is the same as the locale for the Greek language spoken in Greece.
When there is more than one input method for a locale, any secondary input
method is identified by a modifier that is part of the locale name. For example,
there are two input methods for the French locale as spoken in Canada, the default
and an alternative method that supports the earlier keyboard. The input method
names are:
fr_CA
fr_CA@im=alt
fr_CA.im__64
Default input method
Alternative input method
64-bit input method
The fr portion of the locale represents the language name (French), and the CA
represents the territory name (Canada). The @im=alt string is the modifier portion
of the locale that is used to identify the alternative input method. All modifier
strings are identified by the format @im=Modifier.
Since the input method is a loadable object module, a different object is required
when running in the 64-bit environment. In the 64-bit environment, the input
Chapter 16. National Language Support
467
method library will automatically append __64 to the name when searching for the
input method. In the example above, the name of the input method would be
fr_CA.im__64.
It is possible to name input methods without using the locale name. Since the
libIM library does not restrict names to locale names, the calling application must
ensure that the name passed to libIM can be found. However, applications should
request only modifier strings of the form @im=Modifier and that the user’s request
be concatenated with the return string from the setlocale (LC_CTYPE,NULL)
subroutine.
Input Method Areas
In order to compose, complex input methods require direct dialog with users. For
example, the Japanese Input Method may need to show a menu of candidate
strings based on the phonetic matches of the keys you enter. The feedback of the
key strokes appears in one or more areas on the display. The input method areas
are:
Status Text data and bitmaps can appear in the Status area. The Status area is an
extension of the light-emitting diodes (LEDs) on the keyboard.
Pre-edit
Intermediate text appears in the Pre-edit area for languages that compose
before the client handles the data.
A common feature of input methods is that you press a combination of
keys to represent a single character or set of characters. This process of
composing characters from keystrokes is called pre-editing.
Auxiliary
Pop-up menus and dialog that allow you to customize the input method
appear in the Auxiliary area. You can have multiple Auxiliary areas
managed by the input method and independent of the client.
Management for input method areas is based on the division of
responsibility between the application (or toolkit) and the input method.
The divisions of responsibility are:
v Applications are responsible for the size and position of the input
method area.
v Input methods are responsible for the contents of the input area. The
input method area cannot suggest a placement.
Programming Input Methods
The input method is a programming interface that allows applications to run in an
international environment provided through National Language Support (NLS).
The input method has the following characteristics:
v Localized input support (defined by locale)
v Multiple keyboard support
v Multibyte character-input processing
Refer to the diagram to see the relationship between the input method and the
application.
468
General Programming Concepts
Application / Toolkit
IM API
Japanese Input Method
R
a
w
French Input Method
English Input Method
IMInitialize
(IMCORE)
libIM
Input Method
Editor (IMED)
I
n
p
u
t
IM–CB API
Graphic Dependent
Functions
(CALLBACKS)
Graphic API
KEYBOARD
DISPLAY
Components of the Input Method
Initialization
You can use the IMQueryLanguage subroutine to determine if an input method is
available without initializing it. An application (toolkit) initializes a locale-specific
input method by calling the IMInitialize subroutine, which initializes a
locale-specific input method editor (IMED). The subroutine uses the LOCPATH
environment variable to search for the input method named by the LANG
environment variable. The LOCPATH variable specifies a set of directory names
used to search for input methods.
If the input method is found, the IMInitialize subroutine uses the load subroutine
to load the input method and attach the imkeymap file. When the input method is
accessed, an object of the type IMFep (input method front-end processor) is
returned. The IMFep should be treated as an opaque structure.
The IMInitialize subroutine links the converter function using the load subroutine.
The load subroutine is similar to the exec subroutine and links the converter
program at run-time. Since the IMInitialize subroutine is called as a library
function, it must preserve security for certain programs. When the IMInitialize
Chapter 16. National Language Support
469
subroutine is called from a set root ID program, it will ignore the LOCPATH
environment variable and search for converters only in the /usr/lib/nls/loc/iconv
and /etc/nls/loc/iconv directories.
Each IMFep inherits the locale’s code set when the IMInitialize subroutine is
called. Consequently, strings returned by the IMFilter and IMLookupString
subroutines are in the locale’s code set. Changing the locale after the IMInitialize
subroutine is called does not affect the code set of the IMFep.
For each IMFep, the application can use the IMCreate subroutine to create one or
more IMObject instances. The IMObject manages its own state and can manage
several “Input Method Areas” on page 468. How each IMObject defines input
processing depends on the code set and keyboard associated with the locale. In the
simplest case, a single IMObject is needed if the application is managing a single
dialog with the user. The input method also supports newer user interfaces where
the application allows multiple dialogs with the user, and each dialog requires one
IMObject.
The difference between an IMFep and IMObject is that the IMFep is a handle that
binds the application to the code of the input method, while the IMObject is a
handle that represents an instance of a state of an input device, such as a
keyboard. The IMFep does not represent a state of the input method. Each
IMObject is initialized to a specific input state and is changed according to the
sequence of events it receives.
Once the IMObject is created, the application can process key events. The
application should pass key events to the IMObject using the IMFilter and
IMLookupString subroutines. These subroutines are provided to isolate the
internal processing of the IMED from the customized key event mapping process.
Input Method Management
The input method provides the following subroutines for maintenance purposes:
IMInitialize
IMQueryLanguage
IMCreate
IMClose
IMDestroy
Initializes the standard input method for a specified language.
Returns a handle to an IMED associated with the locale. The handle
is an opaque structure of type IMFep.
Checks whether the specified language is supported.
Creates one instance of a particular input method. This subroutine
must be called before any key event processing is performed.
Closes the input method.
Destroys ">