HTCondor™ Version 8.6.4 Manual

HTCondor™ Version 8.6.4 Manual
HTCondor™ Version 8.6.4 Manual
Center for High Throughput Computing, University of Wisconsin–Madison
July 13, 2017
CONTENTS
1
2
Overview
1
1.1
High-Throughput Computing (HTC) and its Requirements . . . . . . . . . . . . . . . . . . . . . . .
1
1.2
HTCondor’s Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2
1.3
Exceptional Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
1.4
Current Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
1.5
Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
1.6
Contributions and Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
1.7
Contact Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
1.8
Privacy Notice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
Users’ Manual
9
2.1
Welcome to HTCondor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
2.2
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
2.3
Matchmaking with ClassAds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
2.3.1
Inspecting Machine ClassAds with condor_status . . . . . . . . . . . . . . . . . . . . . . . .
10
Running a Job: the Steps To Take . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
2.4.1
Choosing an HTCondor Universe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
Submitting a Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
2.5.1
17
2.4
2.5
Sample submit description files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
i
CONTENTS
2.6
2.7
2.8
ii
2.5.2
Using the Power and Flexibility of the Queue Command . . . . . . . . . . . . . . . . . . . .
20
2.5.3
Variables in the Submit Description File . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
2.5.4
Including Submit Commands Defined Elsewhere . . . . . . . . . . . . . . . . . . . . . . . .
23
2.5.5
Using Conditionals in the Submit Description File . . . . . . . . . . . . . . . . . . . . . . .
24
2.5.6
Function Macros in the Submit Description File . . . . . . . . . . . . . . . . . . . . . . . . .
26
2.5.7
About Requirements and Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
2.5.8
Submitting Jobs Using a Shared File System . . . . . . . . . . . . . . . . . . . . . . . . . .
31
2.5.9
Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism . . . .
32
2.5.10 Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
2.5.11 Heterogeneous Submit: Execution on Differing Architectures . . . . . . . . . . . . . . . . .
43
2.5.12 Jobs That Require GPUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
48
2.5.13 Interactive Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
48
Managing a Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
2.6.1
Checking on the progress of jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
2.6.2
Removing a job from the queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
2.6.3
Placing a job on hold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
2.6.4
Changing the priority of jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
53
2.6.5
Why is the job not running? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
53
2.6.6
Job in the Hold State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
56
2.6.7
In the Job Event Log File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
56
2.6.8
Job Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
59
Priorities and Preemption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60
2.7.1
Job Priority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60
2.7.2
User priority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
61
2.7.3
Details About How HTCondor Jobs Vacate Machines . . . . . . . . . . . . . . . . . . . . .
62
Java Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
62
2.8.1
A Simple Example Java Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
63
2.8.2
Less Simple Java Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
64
HTCondor Version 8.6.4 Manual
CONTENTS
2.8.3
iii
Chirp I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
66
Parallel Applications (Including MPI Applications) . . . . . . . . . . . . . . . . . . . . . . . . . . .
69
2.9.1
How Parallel Jobs Run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
69
2.9.2
Parallel Jobs and the Dedicated Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . .
69
2.9.3
Submission Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
70
2.9.4
MPI Applications Within HTCondor’s Vanilla Universe . . . . . . . . . . . . . . . . . . . .
75
2.10 DAGMan Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
2.10.1 DAGMan Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
2.10.2 The DAG Input File: Basic Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
77
2.10.3 Command Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
84
2.10.4 Node Job Submit File Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
84
2.10.5 DAG Submission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
85
2.10.6 File Paths in DAGs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
87
2.10.7 DAG Monitoring and DAG Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
88
2.10.8 Suspending a Running DAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
89
2.10.9 Advanced Features of DAGMan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
90
2.9
2.10.10 The Rescue DAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
2.10.11 DAG Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
2.10.12 Visualizing DAGs with dot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
2.10.13 Capturing the Status of Nodes in a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
2.10.14 A Machine-Readable Event History, the jobstate.log File . . . . . . . . . . . . . . . . . . . . 130
2.10.15 Status Information for the DAG in a ClassAd . . . . . . . . . . . . . . . . . . . . . . . . . . 134
2.10.16 Utilizing the Power of DAGMan for Large Numbers of Jobs . . . . . . . . . . . . . . . . . . 134
2.10.17 Workflow Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
2.10.18 DAGMan and Accounting Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
2.11 Virtual Machine Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
2.11.1 The Submit Description File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
2.11.2 Checkpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
HTCondor Version 8.6.4 Manual
CONTENTS
iv
2.11.3 Disk Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
2.11.4 Job Completion in the vm Universe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
2.11.5 Failures to Launch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
2.12 Docker Universe Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
2.13 Time Scheduling for Job Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
2.13.1 Job Deferral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
2.13.2 CronTab Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
2.14 Special Environment Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
2.14.1 AFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
2.14.2 NFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
2.14.3 HTCondor Daemons That Do Not Run as root . . . . . . . . . . . . . . . . . . . . . . . . . 156
2.14.4 Job Leases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
2.15 Potential Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
2.15.1 Renaming of argv[0] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
3
Administrators’ Manual
3.1
3.2
3.3
158
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
3.1.1
The Different Roles a Machine Can Play . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
3.1.2
The HTCondor Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Installation, Start Up, Shut Down, and Reconfiguration . . . . . . . . . . . . . . . . . . . . . . . . . 162
3.2.1
Obtaining the HTCondor Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
3.2.2
Installation on Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
3.2.3
Installation on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
3.2.4
Upgrading – Installing a New Version on an Existing Pool . . . . . . . . . . . . . . . . . . . 182
3.2.5
Shutting Down and Restarting an HTCondor Pool . . . . . . . . . . . . . . . . . . . . . . . 183
3.2.6
Reconfiguring an HTCondor Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Introduction to Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
3.3.1
HTCondor Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
HTCondor Version 8.6.4 Manual
CONTENTS
v
3.3.2
Ordered Evaluation to Set the Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
3.3.3
Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
3.3.4
Comments and Line Continuations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
3.3.5
Multi-Line Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
3.3.6
Executing a Program to Produce Configuration Macros . . . . . . . . . . . . . . . . . . . . . 192
3.3.7
Including Configuration from Elsewhere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
3.3.8
Reporting Errors and Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
3.3.9
Conditionals in Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
3.3.10 Function Macros in Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
3.3.11 Macros That Will Require a Restart When Changed . . . . . . . . . . . . . . . . . . . . . . . 199
3.3.12 Pre-Defined Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
3.4
3.5
Configuration Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
3.4.1
Configuration Templates: Using Predefined Sets of Configuration . . . . . . . . . . . . . . . 202
3.4.2
Available Configuration Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
3.4.3
Configuration Template Transition Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
3.4.4
Configuration Template Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Configuration Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
3.5.1
Introduction to Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
3.5.2
HTCondor-wide Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
3.5.3
Daemon Logging Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
3.5.4
DaemonCore Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
3.5.5
Network-Related Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
3.5.6
Shared File System Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . 251
3.5.7
Checkpoint Server Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . 255
3.5.8
condor_master Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
3.5.9
condor_startd Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
3.5.10 condor_schedd Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
3.5.11 condor_shadow Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
HTCondor Version 8.6.4 Manual
CONTENTS
vi
3.5.12 condor_starter Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
3.5.13 condor_submit Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
3.5.14 condor_preen Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
3.5.15 condor_collector Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
3.5.16 condor_negotiator Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . 309
3.5.17 condor_procd Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
3.5.18 condor_credd Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
3.5.19 condor_gridmanager Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . 318
3.5.20 condor_job_router Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . 321
3.5.21 condor_lease_manager Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . 324
3.5.22 Grid Monitor Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
3.5.23 Configuration File Entries Relating to Grid Usage . . . . . . . . . . . . . . . . . . . . . . . . 325
3.5.24 Configuration File Entries for DAGMan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
3.5.25 Configuration File Entries Relating to Security . . . . . . . . . . . . . . . . . . . . . . . . . 335
3.5.26 Configuration File Entries Relating to Virtual Machines . . . . . . . . . . . . . . . . . . . . 341
3.5.27 Configuration File Entries Relating to High Availability . . . . . . . . . . . . . . . . . . . . 343
3.5.28 MyProxy Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
3.5.29 Configuration File Macros Affecting APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
3.5.30 Configuration File Entries Relating to condor_ssh_to_job . . . . . . . . . . . . . . . . . . . . 348
3.5.31 condor_rooster Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
3.5.32 condor_shared_port Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . 350
3.5.33 Configuration File Entries Relating to Hooks . . . . . . . . . . . . . . . . . . . . . . . . . . 352
3.5.34 Configuration File Entries Only for Windows Platforms . . . . . . . . . . . . . . . . . . . . 357
3.5.35 condor_defrag Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
3.5.36 condor_gangliad Configuration File Macros . . . . . . . . . . . . . . . . . . . . . . . . . . 359
3.6
User Priorities and Negotiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
3.6.1
Real User Priority (RUP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
3.6.2
Effective User Priority (EUP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
HTCondor Version 8.6.4 Manual
CONTENTS
3.7
3.8
vii
3.6.3
Priorities in Negotiation and Preemption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
3.6.4
Priority Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
3.6.5
Negotiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
3.6.6
The Layperson’s Description of the Pie Spin and Pie Slice . . . . . . . . . . . . . . . . . . . 365
3.6.7
Group Accounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
3.6.8
Accounting Groups with Hierarchical Group Quotas . . . . . . . . . . . . . . . . . . . . . . 367
Policy Configuration for Execute Hosts and for Submit Hosts . . . . . . . . . . . . . . . . . . . . . . 370
3.7.1
condor_startd Policy Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
3.7.2
condor_schedd Policy Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
3.8.1
HTCondor’s Security Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
3.8.2
Security Negotiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
3.8.3
Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420
3.8.4
The Unified Map File for Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
3.8.5
Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
3.8.6
Integrity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
3.8.7
Authorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
3.8.8
Security Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
3.8.9
Host-Based Security in HTCondor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
3.8.10 Examples of Security Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
3.8.11 Changing the Security Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
3.8.12 Using HTCondor w/ Firewalls, Private Networks, and NATs . . . . . . . . . . . . . . . . . . 445
3.8.13 User Accounts in HTCondor on Unix Platforms . . . . . . . . . . . . . . . . . . . . . . . . . 445
3.9
Networking (includes sections on Port Usage and CCB) . . . . . . . . . . . . . . . . . . . . . . . . . 450
3.9.1
Port Usage in HTCondor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450
3.9.2
Reducing Port Usage with the condor_shared_port Daemon . . . . . . . . . . . . . . . . . . 454
3.9.3
Configuring HTCondor for Machines With Multiple Network Interfaces
3.9.4
HTCondor Connection Brokering (CCB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459
HTCondor Version 8.6.4 Manual
. . . . . . . . . . . 455
CONTENTS
viii
3.9.5
Using TCP to Send Updates to the condor_collector . . . . . . . . . . . . . . . . . . . . . . 461
3.9.6
Running HTCondor on an IPv6 Network Stack . . . . . . . . . . . . . . . . . . . . . . . . . 462
3.10 The Checkpoint Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
3.10.1 Preparing to Install a Checkpoint Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
3.10.2 Installing the Checkpoint Server Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
3.10.3 Configuring the Pool to Use Multiple Checkpoint Servers . . . . . . . . . . . . . . . . . . . 466
3.10.4 Checkpoint Server Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
3.11 DaemonCore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
3.11.1 DaemonCore and Unix signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
3.11.2 DaemonCore and Command-line Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . 470
3.12 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
3.12.1 Ganglia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
3.12.2 Absent ClassAds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474
3.13 The High Availability of Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
3.13.1 High Availability of the Job Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
3.13.2 High Availability of the Central Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
3.14 Setting Up for Special Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
3.14.1 Using HTCondor with AFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
3.14.2 Enabling the Transfer of Files Specified by a URL . . . . . . . . . . . . . . . . . . . . . . . 484
3.14.3 Configuring HTCondor for Multiple Platforms . . . . . . . . . . . . . . . . . . . . . . . . . 486
3.14.4 Full Installation of condor_compile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
3.14.5 The condor_kbdd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489
3.14.6 Configuring The HTCondorView Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
3.14.7 Running HTCondor Jobs within a Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . 493
3.14.8 HTCondor’s Dedicated Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
3.14.9 Configuring HTCondor for Running Backfill Jobs . . . . . . . . . . . . . . . . . . . . . . . . 498
3.14.10 Per Job PID Namespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
3.14.11 Group ID-Based Process Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
HTCondor Version 8.6.4 Manual
CONTENTS
ix
3.14.12 Cgroup-Based Process Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
3.14.13 Limiting Resource Usage with a User Job Wrapper . . . . . . . . . . . . . . . . . . . . . . . 506
3.14.14 Limiting Resource Usage Using Cgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
3.14.15 Concurrency Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509
3.15 Java Support Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
3.16 Setting Up the VM and Docker Universes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
3.16.1 The VM Universe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
3.16.2 The Docker Universe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516
3.17 Singularity Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
3.18 Power Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
3.18.1 Entering a Low Power State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
3.18.2 Returning From a Low Power State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
3.18.3 Keeping a ClassAd for a Hibernating Machine . . . . . . . . . . . . . . . . . . . . . . . . . 521
3.18.4 Linux Platform Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
3.18.5 Windows Platform Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
4
Miscellaneous Concepts
4.1
4.2
523
HTCondor’s ClassAd Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
4.1.1
ClassAds: Old and New . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
4.1.2
Old ClassAd Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
4.1.3
Old ClassAd Evaluation Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
4.1.4
Old ClassAds in the HTCondor System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
4.1.5
Extending ClassAds with User-written Functions . . . . . . . . . . . . . . . . . . . . . . . . 541
HTCondor’s Checkpoint Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542
4.2.1
Standalone Checkpoint Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
4.2.2
Checkpoint Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
4.2.3
Checkpoint Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
4.2.4
Checkpoint Library Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
HTCondor Version 8.6.4 Manual
CONTENTS
4.3
4.4
4.5
5
x
Computing On Demand (COD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546
4.3.1
Overview of How COD Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
4.3.2
Authorizing Users to Create and Manage COD Claims . . . . . . . . . . . . . . . . . . . . . 547
4.3.3
Defining a COD Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548
4.3.4
Managing COD Resource Claims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
4.3.5
Limitations of COD Support in HTCondor . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
Hooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
4.4.1
Job Hooks That Fetch Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
4.4.2
Hooks for a Job Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
4.4.3
Daemon ClassAd Hooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
Logging in HTCondor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
4.5.1
Job and Daemon Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
4.5.2
DAGMan Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
Grid Computing
574
5.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574
5.2
Connecting HTCondor Pools with Flocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
5.3
5.2.1
Flocking Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
5.2.2
Job Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
The Grid Universe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577
5.3.1
HTCondor-C, The condor Grid Type
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577
5.3.2
HTCondor-G, the gt2, and gt5 Grid Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
5.3.3
The nordugrid Grid Type
5.3.4
The unicore Grid Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
5.3.5
The batch Grid Type (for PBS, LSF, SGE, and SLURM) . . . . . . . . . . . . . . . . . . . . 589
5.3.6
The EC2 Grid Type
5.3.7
The GCE Grid Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
5.3.8
The cream Grid Type
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597
HTCondor Version 8.6.4 Manual
CONTENTS
5.3.9
xi
The BOINC Grid Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598
5.3.10 Matchmaking in the Grid Universe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
5.4
6
The HTCondor Job Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
5.4.1
Routing Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
5.4.2
Job Submission with Job Routing Capability . . . . . . . . . . . . . . . . . . . . . . . . . . 605
5.4.3
An Example Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607
5.4.4
Routing Table Entry ClassAd Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
5.4.5
Example: constructing the routing table from ReSS . . . . . . . . . . . . . . . . . . . . . . . 610
Application Programming Interfaces (APIs)
6.1
611
Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
6.1.1
Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
6.1.2
Job Submission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612
6.1.3
File Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
6.1.4
Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614
6.1.5
Get These Items Correct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
6.1.6
Methods for Transaction Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
6.1.7
Methods for Job Submission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
6.1.8
Methods for File Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
6.1.9
Methods for Job Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
6.1.10 Methods for ClassAd Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
6.1.11 Methods for Version Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622
6.1.12 Common Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622
6.2
The DRMAA API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
6.2.1
6.3
Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
The HTCondor User and Job Log Reader API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624
6.3.1
Constants and Enumerated Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625
6.3.2
Constructors and Destructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625
HTCondor Version 8.6.4 Manual
CONTENTS
xii
6.3.3
Initializers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627
6.3.4
Primary Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 628
6.3.5
Accessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
6.3.6
Methods for saving and restoring persistent reader state . . . . . . . . . . . . . . . . . . . . . 629
6.3.7
Save state to persistent storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
6.3.8
Restore state from persistent storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 630
6.3.9
API Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 630
6.3.10 Access to the persistent state data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
6.3.11 Future persistence API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
6.4
Chirp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
6.5
The Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
6.6
The HTCondor Perl Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
6.7
7
6.6.1
Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635
6.6.2
Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637
Python Bindings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642
6.7.1
htcondor Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642
6.7.2
Sample Code using the htcondor Python Module . . . . . . . . . . . . . . . . . . . . . . 653
6.7.3
ClassAd Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654
6.7.4
Sample Code using the classad Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
Platform-Specific Information
7.1
Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 661
7.1.1
7.2
661
Linux Address Space Randomization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 662
Microsoft Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 662
7.2.1
Limitations under Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 663
7.2.2
Supported Features under Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 663
7.2.3
Secure Password Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664
7.2.4
Executing Jobs as the Submitting User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664
HTCondor Version 8.6.4 Manual
CONTENTS
xiii
7.2.5
The condor_credd Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664
7.2.6
Executing Jobs with the User’s Profile Loaded . . . . . . . . . . . . . . . . . . . . . . . . . 666
7.2.7
Using Windows Scripts as Job Executables . . . . . . . . . . . . . . . . . . . . . . . . . . . 666
7.2.8
How HTCondor for Windows Starts and Stops a Job . . . . . . . . . . . . . . . . . . . . . . 668
7.2.9
Security Considerations in HTCondor for Windows . . . . . . . . . . . . . . . . . . . . . . . 669
7.2.10 Network files and HTCondor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 670
7.2.11 Interoperability between HTCondor for Unix and HTCondor for Windows . . . . . . . . . . 672
7.2.12 Some differences between HTCondor for Unix -vs- HTCondor for Windows . . . . . . . . . 672
7.3
Macintosh OS X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673
8
Frequently Asked Questions (FAQ)
674
9
Contrib and Source Modules
675
9.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675
9.2
Using HTCondor with the Hadoop File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675
9.2.1
9.3
9.4
Quill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677
9.3.1
Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 678
9.3.2
Four Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684
9.3.3
Quill and Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685
9.3.4
Quill and Its RDBMS Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685
The HTCondorView Client Contrib Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704
9.4.1
9.5
condor_hdfs Configuration File Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676
Step-by-Step Installation of the HTCondorView Client . . . . . . . . . . . . . . . . . . . . . 705
Job Monitor/Log Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707
9.5.1
Transition States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707
9.5.2
Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707
9.5.3
Selecting Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707
9.5.4
Zooming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 708
9.5.5
Keyboard and Mouse Shortcuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 708
HTCondor Version 8.6.4 Manual
CONTENTS
xiv
10 Version History and Release Notes
709
10.1 Introduction to HTCondor Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709
10.1.1 HTCondor Version Number Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709
10.1.2 The Stable Release Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 710
10.1.3 The Development Release Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 710
10.2 Upgrading from the 8.4 series to the 8.6 series of HTCondor . . . . . . . . . . . . . . . . . . . . . . 710
10.3 Stable Release Series 8.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 712
10.4 Development Release Series 8.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718
10.5 Stable Release Series 8.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 732
11 Command Reference Manual (man pages)
748
bosco_cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 749
bosco_findplatform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 751
bosco_install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 752
bosco_ssh_start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753
bosco_start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754
bosco_stop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 755
bosco_uninstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756
condor_advertise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 757
condor_check_userlogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 761
condor_checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 762
condor_chirp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 765
condor_cod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769
condor_compile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772
condor_config_val . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774
condor_configure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 779
condor_continue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 784
condor_convert_history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786
HTCondor Version 8.6.4 Manual
CONTENTS
xv
condor_dagman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 788
condor_dagman_metrics_reporter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794
condor_drain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 797
condor_fetchlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 799
condor_findhost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 802
condor_gather_info . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 804
condor_gpu_discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807
condor_history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 810
condor_hold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 814
condor_install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 817
condor_job_router_info . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 822
condor_master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824
condor_off
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825
condor_on . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828
condor_ping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 831
condor_pool_job_report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834
condor_power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835
condor_preen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 837
condor_prio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 839
condor_procd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 841
condor_q . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 844
condor_qedit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 859
condor_qsub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 861
condor_reconfig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 866
condor_release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 869
condor_reschedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 871
condor_restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 873
condor_rm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876
HTCondor Version 8.6.4 Manual
CONTENTS
xvi
condor_rmdir . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 879
condor_router_history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 881
condor_router_q . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 883
condor_router_rm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 885
condor_run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 887
condor_set_shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 890
condor_ssh_to_job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 892
condor_sos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896
condor_stats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 898
condor_status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 901
condor_store_cred . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 909
condor_submit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 911
condor_submit_dag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 953
condor_suspend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 960
condor_tail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 962
condor_transfer_data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 964
condor_transform_ads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 966
condor_update_machine_ad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 969
condor_updates_stats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 971
condor_urlfetch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 974
condor_userlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 976
condor_userprio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 979
condor_vacate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 984
condor_vacate_job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 986
condor_version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 989
condor_wait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 991
condor_who . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 994
gidd_alloc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 998
HTCondor Version 8.6.4 Manual
CONTENTS
xvii
procd_ctl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 999
12 Appendix A: ClassAd Attributes
1001
13 Appendix B: Codes and Other Needed Values
1057
LICENSING AND COPYRIGHT
HTCondor is released under the Apache License, Version 2.0.
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
Copyright © 1990-2015 Center for High Throughput Computing, Computer Sciences Department, University of WisconsinMadison, WI.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the
License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS"
BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
language governing permissions and limitations under the License.
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of
this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under
common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect,
to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent
(50outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including but not limited to software source code,
documentation source, and configuration files.
"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation, and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated
by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and
for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of
HTCondor Version 8.6.4 Manual
CONTENTS
xviii
authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely
link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or
additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work
by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the
purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor
or its representatives, including but not limited to communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving
the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright
owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received
by Licensor and subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a
perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative
Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or
Object form.
3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a
perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to
make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those
patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination
of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against
any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within
the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License
for that Work shall terminate as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or
without modifications, and in Source or Object form, provided that You meet the following conditions:
(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative
Works; and
(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works;
or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The
contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own
attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the
Work, provided that such additional attribution notices cannot be construed as modifying the License.
You may add Your own copyright statement to Your modifications and may provide additional or different license terms
and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole,
provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion
in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or
conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement
you may have executed with Licensor regarding such Contributions.
HTCondor Version 8.6.4 Manual
CONTENTS
xix
6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of
the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the
content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and
each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NONINFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible
for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise
of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor
be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character
arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss
of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if
such Contributor has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose
to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole
responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor
harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such
warranty or additional liability.
END OF TERMS AND CONDITIONS
HTCondor Version 8.6.4 Manual
CHAPTER
ONE
Overview
1.1 High-Throughput Computing (HTC) and its Requirements
For many research and engineering projects, the quality of the research or the product is heavily dependent upon
the quantity of computing cycles available. It is not uncommon to find problems that require weeks or months of
computation to solve. Scientists and engineers engaged in this sort of work need a computing environment that
delivers large amounts of computational power over a long period of time. Such an environment is called a HighThroughput Computing (HTC) environment. In contrast, High Performance Computing (HPC) environments deliver
a tremendous amount of compute power over a short period of time. HPC environments are often measured in terms of
FLoating point Operations Per Second (FLOPS). A growing community is not concerned about operations per second,
but operations per month or per year. Their problems are of a much larger scale. They are more interested in how
many jobs they can complete over a long period of time instead of how fast an individual job can complete.
The key to HTC is to efficiently harness the use of all available resources. Years ago, the engineering and scientific
community relied on a large, centralized mainframe or a supercomputer to do computational work. A large number of
individuals and groups needed to pool their financial resources to afford such a machine. Users had to wait for their
turn on the mainframe, and they had a limited amount of time allocated. While this environment was inconvenient for
users, the utilization of the mainframe was high; it was busy nearly all the time.
As computers became smaller, faster, and cheaper, users moved away from centralized mainframes and purchased
personal desktop workstations and PCs. An individual or small group could afford a computing resource that was
available whenever they wanted it. The personal computer is slower than the large centralized machine, but it provides
exclusive access. Now, instead of one giant computer for a large institution, there may be hundreds or thousands of
personal computers. This is an environment of distributed ownership, where individuals throughout an organization
own their own resources. The total computational power of the institution as a whole may rise dramatically as the result
of such a change, but because of distributed ownership, individuals have not been able to capitalize on the institutional
growth of computing power. And, while distributed ownership is more convenient for the users, the utilization of the
1
1.2. HTCondor’s Power
2
computing power is lower. Many personal desktop machines sit idle for very long periods of time while their owners
are busy doing other things (such as being away at lunch, in meetings, or at home sleeping).
1.2 HTCondor’s Power
HTCondor is a software system that creates a High-Throughput Computing (HTC) environment. It effectively utilizes
the computing power of workstations that communicate over a network. HTCondor can manage a dedicated cluster
of workstations. Its power comes from the ability to effectively harness non-dedicated, preexisting resources under
distributed ownership.
A user submits the job to HTCondor. HTCondor finds an available machine on the network and begins running
the job on that machine. HTCondor has the capability to detect that a machine running a HTCondor job is no longer
available (perhaps because the owner of the machine came back from lunch and started typing on the keyboard). It
can checkpoint the job and move (migrate) the jobs to a different machine which would otherwise be idle. HTCondor
continues the job on the new machine from precisely where it left off.
In those cases where HTCondor can checkpoint and migrate a job, HTCondor makes it easy to maximize the
number of machines which can run a job. In this case, there is no requirement for machines to share file systems (for
example, with NFS or AFS), so that machines across an entire enterprise can run a job, including machines in different
administrative domains.
HTCondor can be a real time saver when a job must be run many (hundreds of) different times, perhaps with
hundreds of different data sets. With one command, all of the hundreds of jobs are submitted to HTCondor. Depending
upon the number of machines in the HTCondor pool, dozens or even hundreds of otherwise idle machines can be
running the job at any given moment.
HTCondor does not require an account (login) on machines where it runs a job. HTCondor can do this because of
its remote system call technology, which traps library calls for such operations as reading or writing from disk files.
The calls are transmitted over the network to be performed on the machine where the job was submitted.
HTCondor provides powerful resource management by match-making resource owners with resource consumers.
This is the cornerstone of a successful HTC environment. Other compute cluster resource management systems attach
properties to the job queues themselves, resulting in user confusion over which queue to use as well as administrative
hassle in constantly adding and editing queue properties to satisfy user demands. HTCondor implements ClassAds, a
clean design that simplifies the user’s submission of jobs.
ClassAds work in a fashion similar to the newspaper classified advertising want-ads. All machines in the HTCondor pool advertise their resource properties, both static and dynamic, such as available RAM memory, CPU type, CPU
speed, virtual memory size, physical location, and current load average, in a resource offer ad. A user specifies a
resource request ad when submitting a job. The request defines both the required and a desired set of properties of the
resource to run the job. HTCondor acts as a broker by matching and ranking resource offer ads with resource request
ads, making certain that all requirements in both ads are satisfied. During this match-making process, HTCondor also
considers several layers of priority values: the priority the user assigned to the resource request ad, the priority of the
user which submitted the ad, and desire of machines in the pool to accept certain types of ads over others.
HTCondor Version 8.6.4 Manual
1.3. Exceptional Features
3
1.3 Exceptional Features
Checkpoint and Migration. Where programs can be linked with HTCondor libraries, users of HTCondor may be
assured that their jobs will eventually complete, even in the ever changing environment that HTCondor utilizes.
As a machine running a job submitted to HTCondor becomes unavailable, the job can be check pointed. The
job may continue after migrating to another machine. HTCondor’s checkpoint feature periodically checkpoints
a job even in lieu of migration in order to safeguard the accumulated computation time on a job from being lost
in the event of a system failure, such as the machine being shutdown or a crash.
Remote System Calls. Despite running jobs on remote machines, the HTCondor standard universe execution mode
preserves the local execution environment via remote system calls. Users do not have to worry about making
data files available to remote workstations or even obtaining a login account on remote workstations before
HTCondor executes their programs there. The program behaves under HTCondor as if it were running as the
user that submitted the job on the workstation where it was originally submitted, no matter on which machine it
really ends up executing on.
No Changes Necessary to User’s Source Code. No special programming is required to use HTCondor. HTCondor
is able to run non-interactive programs. The checkpoint and migration of programs by HTCondor is transparent
and automatic, as is the use of remote system calls. If these facilities are desired, the user only re-links the
program. The code is neither recompiled nor changed.
Pools of Machines can be Hooked Together. Flocking is a feature of HTCondor that allows jobs submitted within a
first pool of HTCondor machines to execute on a second pool. The mechanism is flexible, following requests
from the job submission, while allowing the second pool, or a subset of machines within the second pool to set
policies over the conditions under which jobs are executed.
Jobs can be Ordered. The ordering of job execution required by dependencies among jobs in a set is easily handled.
The set of jobs is specified using a directed acyclic graph, where each job is a node in the graph. Jobs are
submitted to HTCondor following the dependencies given by the graph.
HTCondor Enables Grid Computing. As grid computing becomes a reality, HTCondor is already there. The technique of glidein allows jobs submitted to HTCondor to be executed on grid machines in various locations worldwide. As the details of grid computing evolve, so does HTCondor’s ability, starting with Globus-controlled
resources.
Sensitive to the Desires of Machine Owners. The owner of a machine has complete priority over the use of the
machine. An owner is generally happy to let others compute on the machine while it is idle, but wants it back
promptly upon returning. The owner does not want to take special action to regain control. HTCondor handles
this automatically.
ClassAds. The ClassAd mechanism in HTCondor provides an extremely flexible, expressive framework for matchmaking resource requests with resource offers. Users can easily request both job requirements and job desires.
For example, a user can require that a job run on a machine with 64 Mbytes of RAM, but state a preference
for 128 Mbytes, if available. A workstation owner can state a preference that the workstation runs jobs from a
specified set of users. The owner can also require that there be no interactive workstation activity detectable at
certain hours before HTCondor could start a job. Job requirements/preferences and resource availability constraints can be described in terms of powerful expressions, resulting in HTCondor’s adaptation to nearly any
desired policy.
HTCondor Version 8.6.4 Manual
1.4. Current Limitations
4
1.4 Current Limitations
Limitations on Jobs which can Checkpointed Although HTCondor can schedule and run any type of process, HTCondor does have some limitations on jobs that it can transparently checkpoint and migrate:
1. Multi-process jobs are not allowed. This includes system calls such as fork(), exec(), and
system().
2. Interprocess communication is not allowed. This includes pipes, semaphores, and shared memory.
3. Network communication must be brief. A job may make network connections using system calls such as
socket(), but a network connection left open for long periods will delay checkpointing and migration.
4. Sending or receiving the SIGUSR2 or SIGTSTP signals is not allowed. HTCondor reserves these signals
for its own use. Sending or receiving all other signals is allowed.
5. Alarms, timers, and sleeping are not allowed. This includes system calls such as alarm(),
getitimer(), and sleep().
6. Multiple kernel-level threads are not allowed. However, multiple user-level threads are allowed.
7. Memory mapped files are not allowed. This includes system calls such as mmap() and munmap().
8. File locks are allowed, but not retained between checkpoints.
9. All files must be opened read-only or write-only. A file opened for both reading and writing will cause
trouble if a job must be rolled back to an old checkpoint image. For compatibility reasons, a file opened
for both reading and writing will result in a warning but not an error.
10. A fair amount of disk space must be available on the submitting machine for storing a job’s checkpoint
images. A checkpoint image is approximately equal to the virtual memory consumed by a job while it
runs. If disk space is short, a special checkpoint server can be designated for storing all the checkpoint
images for a pool.
11. On Linux, the job must be statically linked. condor_compile does this by default.
12. Reading to or writing from files larger than 2 GBytes is only supported when the submit side condor_shadow and the standard universe user job application itself are both 64-bit executables.
Note: these limitations only apply to jobs which HTCondor has been asked to transparently checkpoint. If job
checkpointing is not desired, the limitations above do not apply.
Security Implications. HTCondor does a significant amount of work to prevent security hazards, but loopholes are
known to exist. HTCondor can be instructed to run user programs only as the UNIX user nobody, a user
login which traditionally has very restricted access. But even with access solely as user nobody, a sufficiently
malicious individual could do such things as fill up /tmp (which is world writable) and/or gain read access to
world readable files. Furthermore, where the security of machines in the pool is a high concern, only machines
where the UNIX user root on that machine can be trusted should be admitted into the pool. HTCondor provides
the administrator with extensive security mechanisms to enforce desired policies.
Jobs Need to be Re-linked to get Checkpointing and Remote System Calls Although typically no source code
changes are required, HTCondor requires that the jobs be re-linked with the HTCondor libraries to take advantage of checkpointing and remote system calls. This often precludes commercial software binaries from taking
advantage of these services because commercial packages rarely make their object code available. HTCondor’s
other services are still available for these commercial packages.
HTCondor Version 8.6.4 Manual
1.5. Availability
5
1.5 Availability
HTCondor is currently available as a free download from the Internet via the World Wide Web at URL
http://htcondor.org/downloads/. Binary distributions of this HTCondor Version 8.6.4 release are available for the
platforms detailed in Table 1.1. A platform is an architecture/operating system combination.
In the table, clipped means that HTCondor does not support checkpointing or remote system calls on the given
platform. This means that standard universe jobs are not supported. Some clipped platforms will have further limitations with respect to supported universes. See section 2.4.1 on page 13 for more details on job universes within
HTCondor and their abilities and limitations.
The HTCondor source code is available for public download alongside the binary distributions.
Architecture
Intel x86
x86_64
Operating System
- RedHat Enterprise Linux 6
- All versions Windows Vista or greater (clipped)
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 7
- Debian Linux 7.0 (wheezy)
- Debian Linux 8.0 (jessie)
- Macintosh OS X 10.7 through 10.10 (clipped)
- Ubuntu 12.04; Precise Pangolin (clipped)
- Ubuntu 14.04; Trusty Tahr
Table 1.1: Supported platforms in HTCondor Version 8.6.4
NOTE: Other Linux distributions likely work, but are not tested or supported.
For more platform-specific information about HTCondor’s support for various operating systems, see Chapter 7
on page 661.
Jobs submitted to the standard universe utilize condor_compile to relink programs with libraries provided by
HTCondor. Table 1.2 lists supported compilers by platform for this Version 8.6.4 release. Other compilers may work,
but are not supported.
1.6 Contributions and Acknowledgments
The quality of the HTCondor project is enhanced by the contributions of external organizations. We gratefully acknowledge the following contributions.
HTCondor Version 8.6.4 Manual
1.6. Contributions and Acknowledgments
Platform
Red Hat Enterprise Linux 6 on x86_64
Red Hat Enterprise Linux 7 on x86_64
Debian Linux 7.0 (wheezy) on x86_64
Debian Linux 8.0 (jessie) on x86_64
Ubuntu 14.04 on x86_64
6
Compiler
gcc, g++, and g77
gcc, g++, and g77
gcc, g++, gfortran
gcc, g++, gfortran
gcc, g++, gfortran
Notes
as shipped
as shipped
as shipped
as shipped
as shipped
Table 1.2: Supported compilers in HTCondor Version 8.6.4
• The Globus Alliance (http://www.globus.org), for code and assistance in developing HTCondor-G and the Grid
Security Infrastructure (GSI) for authentication and authorization.
• The GOZAL Project from the Computer Science Department of the Technion Israel Institute of Technology
(http://www.technion.ac.il/), for their enhancements for HTCondor’s High Availability. The condor_had daemon allows one of multiple machines to function as the central manager for a HTCondor pool. Therefore, if an
acting central manager fails, another can take its place.
• Micron Corporation (http://www.micron.com/) for the MSI-based installer for HTCondor on Windows.
• Paradyn Project (http://www.paradyn.org/) and the Universitat
(http://www.caos.uab.es/) for work on the Tool Daemon Protocol (TDP).
Autònoma
de
Barcelona
Our Web Services API acknowledges the use of gSOAP with their requested wording:
• Part of the software embedded in this product is gSOAP software. Portions created by gSOAP are Copyright
(C) 2001-2004 Robert A. van Engelen, Genivia inc. All Rights Reserved.
THE SOFTWARE IN THIS PRODUCT WAS IN PART PROVIDED BY GENIVIA INC AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
• Some
distributions
of
HTCondor
include
the
Google
Coredumper
library
(http://goog-coredumper.sourceforge.net/). The Google Coredumper library is released under these terms:
Copyright (c) 2005, Google Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the
following conditions are met:
– Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
HTCondor Version 8.6.4 Manual
1.7. Contact Information
7
– Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the
following disclaimer in the documentation and/or other materials provided with the distribution.
– Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote
products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The HTCondor project wishes to acknowledge the following:
• This material is based upon work supported by the National Science Foundation under Grant Numbers MCS8105904, OCI-0437810, and OCI-0850745. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science
Foundation.
1.7 Contact Information
The latest software releases, publications/papers regarding HTCondor and other High-Throughput Computing research
can be found at the official web site for HTCondor at http://htcondor.org/.
In addition, there is an e-mail list at htcondor-world@cs.wisc.edu. The HTCondor Team uses this e-mail list to
announce new releases of HTCondor and other major HTCondor-related news items. To subscribe or unsubscribe from
the the list, follow the instructions at http://htcondor.org/mail-lists/. Because many of us receive too much e-mail as it
is, you will be happy to know that the HTCondor World e-mail list group is moderated, and only major announcements
of wide interest are distributed.
Our users support each other by belonging to an unmoderated mailing list (htcondor-users@cs.wisc.edu) targeted at solving problems with HTCondor. HTCondor team members attempt to monitor traffic to htcondor-users,
responding as they can. Follow the instructions at http://htcondor.org/mail-lists/.
Finally, you can reach the HTCondor Team directly. The HTCondor Team is comprised of the developers
and administrators of HTCondor at the University of Wisconsin-Madison. HTCondor questions, comments, pleas
for help, and requests for commercial contract consultation or support are all welcome; send Internet e-mail to
htcondor-admin@cs.wisc.edu. Please include your name, organization, and telephone number in your message. If
you are having trouble with HTCondor, please help us troubleshoot by including as much pertinent information as you
can, including snippets of HTCondor log files.
HTCondor Version 8.6.4 Manual
1.8. Privacy Notice
8
1.8 Privacy Notice
The HTCondor software periodically sends short messages to the HTCondor Project developers at the University of
Wisconsin, reporting totals of machines and jobs in each running HTCondor system. An example of such a message
is given below.
The HTCondor Project uses these collected reports to publish summary figures and tables, such as the total of
HTCondor systems worldwide, or the geographic distribution of HTCondor systems. This information helps the
HTCondor Project to understand the scale and composition of HTCondor in the real world and improve the software
accordingly.
The HTCondor Project will not use these reports to publicly identify any HTCondor system or user without permission. The HTCondor software does not collect or report any personal information about individual users.
We hope that you will contribute to the development of HTCondor through this reporting feature. However, you are free to disable it at any time by changing the configuration variables CONDOR_DEVELOPERS and
CONDOR_DEVELOPERS_COLLECTOR, both described in section 3.5.15 of this manual.
Example of data reported:
This is an automated email from the HTCondor system
on machine "your.condor.pool.com". Do not reply.
This Collector has the following IDs:
HTCondor: 6.6.0 Nov 12 2003
HTCondor: INTEL-LINUX-GLIBC22
Machines Owner Claimed Unclaimed Matched Preempting
INTEL/LINUX
INTEL/WINDOWS
SUN4u/SOLARIS28
SUN4x/SOLARIS28
Total
RunningJobs
920
810
120
114
5
1049
52
5
12
1
70
716
115
92
0
923
37
0
9
4
50
IdleJobs
3868
HTCondor Version 8.6.4 Manual
0
0
0
0
0
5
0
1
0
6
CHAPTER
TWO
Users’ Manual
2.1 Welcome to HTCondor
HTCondor is developed by the Center for High Throughput Computing at the University of Wisconsin-Madison (UWMadison), and was first installed as a production system in the UW-Madison Computer Sciences department more than
15 years ago. HTCondor pools have since served as a major source of computing cycles to UW faculty and students.
For many, it has revolutionized the role computing plays in their research. An increase of one, and sometimes even
two, orders of magnitude in the computing throughput of a research organization can have a profound impact on
research size, complexity, and scope. Over the years, the project, and now the Center for High Throughput Computing
have established collaborations with scientists from around the world, and have provided them with access to many
cycles. One scientist consumed 100 CPU years!
2.2 Introduction
In a nutshell, HTCondor is a specialized batch system for managing compute-intensive jobs. Like most batch systems,
HTCondor provides a queuing mechanism, scheduling policy, priority scheme, and resource classifications. Users
submit their compute jobs to HTCondor, HTCondor puts the jobs in a queue, runs them, and then informs the user as
to the result.
Batch systems normally operate only with dedicated machines. Often termed compute servers, these dedicated
machines are typically owned by one organization and dedicated to the sole purpose of running compute jobs. HTCondor can schedule jobs on dedicated machines. But unlike traditional batch systems, HTCondor is also designed to
effectively utilize non-dedicated machines to run jobs. By being told to only run compute jobs on machines which are
currently not being used (no keyboard activity, low load average, etc.), HTCondor can effectively harness otherwise
idle machines throughout a pool of machines. This is important because often times the amount of compute power
9
2.3. Matchmaking with ClassAds
10
represented by the aggregate total of all the non-dedicated desktop workstations sitting on people’s desks throughout
the organization is far greater than the compute power of a dedicated central resource.
HTCondor has several unique capabilities at its disposal which are geared toward effectively utilizing nondedicated resources that are not owned or managed by a centralized resource. These include transparent process
checkpoint and migration, remote system calls, and ClassAds. Read section 1.2 for a general discussion of these
features before reading any further.
2.3 Matchmaking with ClassAds
Before you learn about how to submit a job, it is important to understand how HTCondor allocates resources. Understanding the unique framework by which HTCondor matches submitted jobs with machines is the key to getting the
most from HTCondor’s scheduling algorithm.
HTCondor simplifies job submission by acting as a matchmaker of ClassAds. HTCondor’s ClassAds are analogous
to the classified advertising section of the newspaper. Sellers advertise specifics about what they have to sell, hoping
to attract a buyer. Buyers may advertise specifics about what they wish to purchase. Both buyers and sellers list
constraints that need to be satisfied. For instance, a buyer has a maximum spending limit, and a seller requires a
minimum purchase price. Furthermore, both want to rank requests to their own advantage. Certainly a seller would
rank one offer of $50 dollars higher than a different offer of $25. In HTCondor, users submitting jobs can be thought
of as buyers of compute resources and machine owners are sellers.
All machines in a HTCondor pool advertise their attributes, such as available memory, CPU type and speed,
virtual memory size, current load average, along with other static and dynamic properties. This machine ClassAd also
advertises under what conditions it is willing to run a HTCondor job and what type of job it would prefer. These policy
attributes can reflect the individual terms and preferences by which all the different owners have graciously allowed
their machine to be part of the HTCondor pool. You may advertise that your machine is only willing to run jobs at
night and when there is no keyboard activity on your machine. In addition, you may advertise a preference (rank) for
running jobs submitted by you or one of your co-workers.
Likewise, when submitting a job, you specify a ClassAd with your requirements and preferences. The ClassAd
includes the type of machine you wish to use. For instance, perhaps you are looking for the fastest floating point
performance available. You want HTCondor to rank available machines based upon floating point performance. Or,
perhaps you care only that the machine has a minimum of 128 MiB of RAM. Or, perhaps you will take any machine
you can get! These job attributes and requirements are bundled up into a job ClassAd.
HTCondor plays the role of a matchmaker by continuously reading all the job ClassAds and all the machine
ClassAds, matching and ranking job ads with machine ads. HTCondor makes certain that all requirements in both
ClassAds are satisfied.
2.3.1 Inspecting Machine ClassAds with condor_status
Once HTCondor is installed, you will get a feel for what a machine ClassAd does by trying the condor_status command. Try the condor_status command to get a summary of information from ClassAds about the resources available
in your pool. Type condor_status and hit enter to see a summary similar to the following:
HTCondor Version 8.6.4 Manual
2.3.1. Inspecting Machine ClassAds with condor_status
11
Name
OpSys
Arch
State
Activity LoadAv Mem
amul.cs.wisc.edu
slot1@amundsen.cs.
slot2@amundsen.cs.
angus.cs.wisc.edu
anhai.cs.wisc.edu
apollo.cs.wisc.edu
arragon.cs.wisc.ed
bamba.cs.wisc.edu
LINUX
LINUX
LINUX
LINUX
LINUX
LINUX
LINUX
LINUX
INTEL
INTEL
INTEL
INTEL
INTEL
INTEL
INTEL
INTEL
Claimed
Owner
Owner
Claimed
Claimed
Unclaimed
Claimed
Owner
Busy
Idle
Idle
Busy
Busy
Idle
Busy
Idle
0.990
0.000
0.110
0.940
1.400
1.000
0.980
0.040
ActvtyTime
1896 0+00:07:04
1456 0+00:21:58
1456 0+00:21:59
873 0+00:02:54
1896 0+00:03:03
3032 0+00:00:04
873 0+00:04:29
3032 15+20:10:19
...
The condor_status command has options that summarize machine ads in a variety of ways. For example,
condor_status -available shows only machines which are willing to run jobs now.
condor_status -run shows only machines which are currently running jobs.
condor_status -long lists the machine ClassAds for all machines in the pool.
Refer to the condor_status command reference page located on page 901 for a complete description of the condor_status command.
The following shows a portion of a machine ClassAd for a single machine: turunmaa.cs.wisc.edu. Some of the
listed attributes are used by HTCondor for scheduling. Other attributes are for information purposes. An important
point is that any of the attributes in a machine ClassAd can be utilized at job submission time as part of a request or
preference on what machine to use. Additional attributes can be easily added. For example, your site administrator
can add a physical location attribute to your machine ClassAds.
Machine = "turunmaa.cs.wisc.edu"
FileSystemDomain = "cs.wisc.edu"
Name = "turunmaa.cs.wisc.edu"
CondorPlatform = "$CondorPlatform: x86_rhap_5 $"
Cpus = 1
IsValidCheckpointPlatform = ( ( ( TARGET.JobUniverse == 1 ) == false ) ||
( ( MY.CheckpointPlatform =!= undefined ) &&
( ( TARGET.LastCheckpointPlatform =?= MY.CheckpointPlatform ) ||
( TARGET.NumCkpts == 0 ) ) ) )
CondorVersion = "$CondorVersion: 7.6.3 Aug 18 2011 BuildID: 361356 $"
Requirements = ( START ) && ( IsValidCheckpointPlatform )
EnteredCurrentActivity = 1316094896
MyAddress = "<128.105.175.125:58026>"
EnteredCurrentState = 1316094896
Memory = 1897
CkptServer = "pitcher.cs.wisc.edu"
OpSys = "LINUX"
State = "Owner"
START = true
Arch = "INTEL"
Mips = 2634
Activity = "Idle"
StartdIpAddr = "<128.105.175.125:58026>"
HTCondor Version 8.6.4 Manual
2.4. Running a Job: the Steps To Take
12
TargetType = "Job"
LoadAvg = 0.210000
CheckpointPlatform = "LINUX INTEL 2.6.x normal 0x40000000"
Disk = 92309744
VirtualMemory = 2069476
TotalSlots = 1
UidDomain = "cs.wisc.edu"
MyType = "Machine"
2.4 Running a Job: the Steps To Take
The road to using HTCondor effectively is a short one. The basics are quickly and easily learned.
Here are all the steps needed to run a job using HTCondor.
Code Preparation. A job run under HTCondor must be able to run as a background batch job. HTCondor runs
the program unattended and in the background. A program that runs in the background will not be able to do
interactive input and output. HTCondor can redirect console output (stdout and stderr) and keyboard input
(stdin) to and from files for the program. Create any needed files that contain the proper keystrokes needed
for program input. Make certain the program will run correctly with the files.
The HTCondor Universe. HTCondor has several runtime environments (called a universe) from which to choose.
Of the universes, two are likely choices when learning to submit a job to HTCondor: the standard universe
and the vanilla universe. The standard universe allows a job running under HTCondor to handle system calls by
returning them to the machine where the job was submitted. The standard universe also provides the mechanisms
necessary to take a checkpoint and migrate a partially completed job, should the machine on which the job is
executing become unavailable. To use the standard universe, it is necessary to relink the program with the
HTCondor library using the condor_compile command. The manual page for condor_compile on page 772 has
details.
The vanilla universe provides a way to run jobs that cannot be relinked. There is no way to take a checkpoint
or migrate a job executed under the vanilla universe. For access to input and output files, jobs must either use a
shared file system, or use HTCondor’s File Transfer mechanism.
Choose a universe under which to run the HTCondor program, and re-link the program if necessary.
Submit description file. Controlling the details of a job submission is a submit description file. The file contains
information about the job such as what executable to run, the files to use in place of stdin and stdout, and
the platform type required to run the program. The number of times to run a program may be included; it is
simple to run the same program multiple times with multiple data sets.
Write a submit description file to go with the job, using the examples provided in section 2.5 for guidance.
Submit the Job. Submit the program to HTCondor with the condor_submit command.
Once submitted, HTCondor does the rest toward running the job. Monitor the job’s progress with the condor_q
and condor_status commands. You may modify the order in which HTCondor will run your jobs with condor_prio. If
desired, HTCondor can even inform you in a log file every time your job is checkpointed and/or migrated to a different
machine.
HTCondor Version 8.6.4 Manual
2.4.1. Choosing an HTCondor Universe
13
When your program completes, HTCondor will tell you (by e-mail, if preferred) the exit status of your program
and various statistics about its performances, including time used and I/O performed. If you are using a log file for
the job (which is recommended) the exit status will be recorded in the log file. You can remove a job from the queue
prematurely with condor_rm.
2.4.1
Choosing an HTCondor Universe
A universe in HTCondor defines an execution environment. HTCondor Version 8.6.4 supports several different
universes for user jobs:
• standard
• vanilla
• grid
• java
• scheduler
• local
• parallel
• vm
• docker
The universe under which a job runs is specified in the submit description file. If a universe is not specified, the
default is vanilla, unless your HTCondor administrator has changed the default. However, we strongly encourage you
to specify the universe, since the default can be changed by your HTCondor administrator, and the default that ships
with HTCondor has changed.
The standard universe provides migration and reliability, but has some restrictions on the programs that can be run.
The vanilla universe provides fewer services, but has very few restrictions. The grid universe allows users to submit
jobs using HTCondor’s interface. These jobs are submitted for execution on grid resources.
The java universe
allows users to run jobs written for the Java Virtual Machine (JVM). The scheduler universe allows users to submit
lightweight jobs to be spawned by the program known as a daemon on the submit host itself. The parallel universe
is for programs that require multiple machines for one job. See section 2.9 for more about the Parallel universe. The
vm universe allows users to run jobs where the job is no longer a simple executable, but a disk image, facilitating the
execution of a virtual machine. The docker universe runs a Docker container as an HTCondor job.
Standard Universe
In the standard universe, HTCondor provides checkpointing and remote system calls. These features make a job more
reliable and allow it uniform access to resources from anywhere in the pool. To prepare a program as a standard
HTCondor Version 8.6.4 Manual
2.4.1. Choosing an HTCondor Universe
14
universe job, it must be relinked with condor_compile. Most programs can be prepared as a standard universe job, but
there are a few restrictions.
HTCondor checkpoints a job at regular intervals. A checkpoint image is essentially a snapshot of the current state
of a job. If a job must be migrated from one machine to another, HTCondor makes a checkpoint image, copies the
image to the new machine, and restarts the job continuing the job from where it left off. If a machine should crash or
fail while it is running a job, HTCondor can restart the job on a new machine using the most recent checkpoint image.
In this way, jobs can run for months or years even in the face of occasional computer failures.
Remote system calls make a job perceive that it is executing on its home machine, even though the job may
execute on many different machines over its lifetime. When a job runs on a remote machine, a second process, called
a condor_shadow runs on the machine where the job was submitted.
When the job attempts a system call, the
condor_shadow performs the system call instead and sends the results to the remote machine. For example, if a job
attempts to open a file that is stored on the submitting machine, the condor_shadow will find the file, and send the data
to the machine where the job is running.
To convert your program into a standard universe job, you must use condor_compile to relink it with the HTCondor
libraries. Put condor_compile in front of your usual link command. You do not need to modify the program’s source
code, but you do need access to the unlinked object files. A commercial program that is packaged as a single executable
file cannot be converted into a standard universe job.
For example, if you would have linked the job by executing:
% cc main.o tools.o -o program
Then, relink the job for HTCondor with:
% condor_compile cc main.o tools.o -o program
There are a few restrictions on standard universe jobs:
1. Multi-process jobs are not allowed. This includes system calls such as fork(), exec(), and system().
2. Interprocess communication is not allowed. This includes pipes, semaphores, and shared memory.
3. Network communication must be brief. A job may make network connections using system calls such as
socket(), but a network connection left open for long periods will delay checkpointing and migration.
4. Sending or receiving the SIGUSR2 or SIGTSTP signals is not allowed. HTCondor reserves these signals for its
own use. Sending or receiving all other signals is allowed.
5. Alarms, timers, and sleeping are not allowed. This includes system calls such as alarm(), getitimer(),
and sleep().
6. Multiple kernel-level threads are not allowed. However, multiple user-level threads are allowed.
7. Memory mapped files are not allowed. This includes system calls such as mmap() and munmap().
HTCondor Version 8.6.4 Manual
2.4.1. Choosing an HTCondor Universe
15
8. File locks are allowed, but not retained between checkpoints.
9. All files must be opened read-only or write-only. A file opened for both reading and writing will cause trouble if
a job must be rolled back to an old checkpoint image. For compatibility reasons, a file opened for both reading
and writing will result in a warning but not an error.
10. A fair amount of disk space must be available on the submitting machine for storing a job’s checkpoint images.
A checkpoint image is approximately equal to the virtual memory consumed by a job while it runs. If disk space
is short, a special checkpoint server can be designated for storing all the checkpoint images for a pool.
11. On Linux, the job must be statically linked. condor_compile does this by default.
12. Reading to or writing from files larger than 2 GBytes is only supported when the submit side condor_shadow
and the standard universe user job application itself are both 64-bit executables.
Vanilla Universe
The vanilla universe in HTCondor is intended for programs which cannot be successfully re-linked. Shell scripts are
another case where the vanilla universe is useful. Unfortunately, jobs run under the vanilla universe cannot checkpoint
or use remote system calls. This has unfortunate consequences for a job that is partially completed when the remote
machine running a job must be returned to its owner. HTCondor has only two choices. It can suspend the job, hoping
to complete it at a later time, or it can give up and restart the job from the beginning on another machine in the pool.
Since HTCondor’s remote system call features cannot be used with the vanilla universe, access to the job’s input
and output files becomes a concern. One option is for HTCondor to rely on a shared file system, such as NFS or
AFS. Alternatively, HTCondor has a mechanism for transferring files on behalf of the user. In this case, HTCondor
will transfer any files needed by a job to the execution site, run the job, and transfer the output back to the submitting
machine.
Under Unix, HTCondor presumes a shared file system for vanilla jobs. However, if a shared file system is unavailable, a user can enable the HTCondor File Transfer mechanism. On Windows platforms, the default is to use the File
Transfer mechanism. For details on running a job with a shared file system, see section 2.5.8 on page 31. For details
on using the HTCondor File Transfer mechanism, see section 2.5.9 on page 32.
Grid Universe
The Grid universe in HTCondor is intended to provide the standard HTCondor interface to users who wish to start
jobs intended for remote management systems. Section 5.3 on page 577 has details on using the Grid universe. The
manual page for condor_submit on page 911 has detailed descriptions of the grid-related attributes.
Java Universe
A program submitted to the Java universe may run on any sort of machine with a JVM regardless of its location, owner,
or JVM version. HTCondor will take care of all the details such as finding the JVM binary and setting the classpath.
HTCondor Version 8.6.4 Manual
2.5. Submitting a Job
16
Scheduler Universe
The scheduler universe allows users to submit lightweight jobs to be run immediately, alongside the condor_schedd
daemon on the submit host itself. Scheduler universe jobs are not matched with a remote machine, and will never be
preempted. The job’s requirements expression is evaluated against the condor_schedd’s ClassAd.
Originally intended for meta-schedulers such as condor_dagman, the scheduler universe can also be used to manage jobs of any sort that must run on the submit host.
However, unlike the local universe, the scheduler universe does not use a condor_starter daemon to manage the
job, and thus offers limited features and policy support. The local universe is a better choice for most jobs which must
run on the submit host, as it offers a richer set of job management features, and is more consistent with other universes
such as the vanilla universe. The scheduler universe may be retired in the future, in favor of the newer local universe.
Local Universe
The local universe allows an HTCondor job to be submitted and executed with different assumptions for the execution
conditions of the job. The job does not wait to be matched with a machine. It instead executes right away, on the
machine where the job is submitted. The job will never be preempted. The job’s requirements expression is evaluated
against the condor_schedd’s ClassAd.
Parallel Universe
The parallel universe allows parallel programs, such as MPI jobs, to be run within the opportunistic HTCondor environment. Please see section 2.9 for more details.
VM Universe
HTCondor facilitates the execution of VMware and Xen virtual machines with the vm universe.
Please see section 2.11 for details.
Docker Universe
The docker universe runs a docker container on an execute host as a job. Please see section 2.12 for details.
2.5 Submitting a Job
A job is submitted for execution to HTCondor using the condor_submit command. condor_submit takes as an argument the name of a file called a submit description file. This file contains commands and keywords to direct the
HTCondor Version 8.6.4 Manual
2.5.1. Sample submit description files
17
queuing of jobs. In the submit description file, HTCondor finds everything it needs to know about the job. Items such
as the name of the executable to run, the initial working directory, and command-line arguments to the program all
go into the submit description file. condor_submit creates a job ClassAd based upon the information, and HTCondor
works toward running the job.
The contents of a submit description file have been designed to save time for HTCondor users. It is easy to submit
multiple runs of a program to HTCondor with a single submit description file. To run the same program many times
on different input data sets, arrange the data files accordingly so that each run reads its own input, and each run writes
its own output. Each individual run may have its own initial working directory, files mapped for stdin, stdout,
stderr, command-line arguments, and shell environment; these are all specified in the submit description file. A
program that directly opens its own files will read the file names to use either from stdin or from the command line.
A program that opens a static file, given by file name, every time will need to use a separate subdirectory for the output
of each run.
The condor_submit manual page is on page 911 and contains a complete and full description of how to use condor_submit. It also includes descriptions 914 of all of the many commands that may be placed into a submit description
file. In addition, the index lists entries for each command under the heading of Submit Commands.
Note that job ClassAd attributes can be set directly in a submit file using the +<attribute> = <value> syntax
(see 943 for details.)
2.5.1 Sample submit description files
In addition to the examples of submit description files given here, there are more in the condor_submit manual page
(see 911).
Example 1
Example 1 is one of the simplest submit description files possible. It queues up the program myexe for execution
somewhere in the pool. Use of the vanilla universe is implied, as that is the default when not specified in the submit
description file.
An executable is compiled to run on a specific platform. Since this submit description file does not specify a
platform, HTCondor will use its default, which is to run the job on a machine which has the same architecture and
operating system as the machine where condor_submit is run to submit the job.
Standard input for this job will come from the file inputfile, as specified by the input command, and standard
output for this job will go to the file outputfile, as specified by the output command. HTCondor expects to find
inputfile in the current working directory when this job is submitted, and the system will take care of getting the
input file to where it needs to be when the job is executed, as well as bringing back the output results (to the current
working directory) after job execution.
A log file, myexe.log, will also be produced that contains events the job had during its lifetime inside of HTCondor. When the job finishes, its exit conditions will be noted in the log file. This file’s contents are an excellent way
to figure out what happened to submitted jobs.
HTCondor Version 8.6.4 Manual
2.5.1. Sample submit description files
18
####################
#
# Example 1
# Simple HTCondor submit description file
#
####################
Executable
Log
Input
Output
Queue
=
=
=
=
myexe
myexe.log
inputfile
outputfile
Example 2
Example 2 queues up one copy of the program foo (which had been created by condor_compile) for execution by
HTCondor. No input, output, or error commands are given in the submit description file, so stdin, stdout, and
stderr will all refer to /dev/null. The program may produce output by explicitly opening a file and writing to it.
####################
#
# Example 2
# Standard universe submit description file
#
####################
Executable
Universe
Log
Queue
= foo
= standard
= foo.log
Example 3
Example 3 queues two copies of the program mathematica. The first copy will run in directory run_1, and the
second will run in directory run_2 due to the initialdir command. For each copy, stdin will be test.data,
stdout will be loop.out, and stderr will be loop.error. Each run will read input and write output files
within its own directory. Placing data files in separate directories is a convenient way to organize data when a large
group of HTCondor jobs is to run. The example file shows program submission of mathematica as a vanilla universe
job. The vanilla universe is most often the right choice of universe when the source and/or object code is not available.
The request_memory command is included to ensure that the mathematica jobs match with and then execute on
pool machines that provide at least 1 GByte of memory.
####################
HTCondor Version 8.6.4 Manual
2.5.1. Sample submit description files
19
#
# Example 3: demonstrate use of multiple
# directories for data organization.
#
####################
executable
universe
input
output
error
log
request_memory
=
=
=
=
=
=
=
mathematica
vanilla
test.data
loop.out
loop.error
loop.log
1 GB
initialdir
queue
= run_1
initialdir
queue
= run_2
Example 4
The submit description file for Example 4 queues 150 runs of program foo which has been compiled and linked for
Linux running on a 32-bit Intel processor. This job requires HTCondor to run the program on machines which have
greater than 32 MiB of physical memory, and the rank command expresses a preference to run each instance of the
program on machines with more than 64 MiB. It also advises HTCondor that this standard universe job will use up to
28000 KiB of memory when running. Each of the 150 runs of the program is given its own process number, starting
with process number 0. So, files stdin, stdout, and stderr will refer to in.0, out.0, and err.0 for the first
run of the program, in.1, out.1, and err.1 for the second run of the program, and so forth. A log file containing
entries about when and where HTCondor runs, checkpoints, and migrates processes for all the 150 queued programs
will be written into the single file foo.log.
####################
#
# Example 4: Show off some fancy features including
# the use of pre-defined macros.
#
####################
Executable
Universe
requirements
rank
image_size
request_memory
=
=
=
=
=
=
foo
standard
OpSys == "LINUX" && Arch =="INTEL"
Memory >= 64
28000
32
HTCondor Version 8.6.4 Manual
2.5.2. Using the Power and Flexibility of the Queue Command
error
input
output
log
=
=
=
=
err.$(Process)
in.$(Process)
out.$(Process)
foo.log
queue 150
2.5.2 Using the Power and Flexibility of the Queue Command
A wide variety of job submissions can be specified with extra information to the queue submit command. This
flexibility eliminates the need for a job wrapper or Perl script for many submissions.
The form of the queue command defines variables and expands values, identifying a set of jobs. Square brackets
identify an optional item.
queue [<int expr>]
queue [<int expr>] [<varname>] in [slice] <list of items>
queue [<int expr>] [<varname>] matching [files | dirs] [slice] <list of items with file globbing>
queue [<int expr>] [<list of varnames>] from [slice] <file name> | <list of items>
All optional items have defaults:
• If <int expr> is not specified, it defaults to the value 1.
• If <varname> or <list of varnames> is not specified, it defaults to the single variable called ITEM.
• If slice is not specified, it defaults to all elements within the list. This is the Python slice [::], with a step
value of 1.
• If neither files nor dirs is specified in a specification using the from key word, then both files and directories
are considered when globbing.
The list of items uses syntax in one of two forms. One form is a comma and/or space separated list; the items are
placed on the same line as the queue command. The second form separates items by placing each list item on its own
line, and delimits the list with parentheses. The opening parenthesis goes on the same line as the queue command.
The closing parenthesis goes on its own line. The queue command specified with the key word from will always use
the second form of this syntax. Example 3 below uses this second form of syntax.
The optional slice specifies a subset of the list of items using the Python syntax for a slice. Negative step values
are not permitted.
Here are a set of examples.
Example 1
HTCondor Version 8.6.4 Manual
20
2.5.2. Using the Power and Flexibility of the Queue Command
transfer_input_files = $(filename)
arguments
= -infile $(filename)
queue filename matching files *.dat
The use of file globbing expands the list of items to be all files in the current directory that end in .dat. Only files,
and not directories are considered due to the specification of files. One job is queued for each file in the list of
items. For this example, assume that the three files initial.dat, middle.dat, and ending.dat form the
list of items after expansion; macro filename is assigned the value of one of these file names for each job queued.
That macro value is then substituted into the arguments and transfer_input_files commands. The queue command
expands to
transfer_input_files
arguments
queue
transfer_input_files
arguments
queue
transfer_input_files
arguments
queue
= initial.dat
= -infile initial.dat
= middle.dat
= -infile middle.dat
= ending.dat
= -infile ending.dat
Example 2
queue 1 input in A, B, C
Variable input is set to each of the 3 items in the list, and one job is queued for each. For this example the queue
command expands to
input = A
queue
input = B
queue
input = C
queue
Example 3
queue input,arguments from (
file1, -a -b 26
file2, -c -d 92
)
HTCondor Version 8.6.4 Manual
21
2.5.3. Variables in the Submit Description File
Using the from form of the options, each of the two variables specified is given a value from the list of items. For this
example the queue command expands to
input = file1
arguments = -a -b 26
queue
input = file2
arguments = -c -d 92
queue
2.5.3 Variables in the Submit Description File
There are automatic variables for use within the submit description file.
$(Cluster) or $(ClusterId) Each set of queued jobs from a specific user, submitted from a single submit
host, sharing an executable have the same value of $(Cluster) or $(ClusterId). The first cluster of jobs
are assigned to cluster 0, and the value is incremented by one for each new cluster of jobs. $(Cluster) or
$(ClusterId) will have the same value as the job ClassAd attribute ClusterId.
$(Process) or $(ProcId) Within a cluster of jobs, each takes on its own unique $(Process) or $(ProcId)
value. The first job has value 0. $(Process) or $(ProcId) will have the same value as the job ClassAd
attribute ProcId.
$(Item) The default name of the variable when no <varname> is provided in a queue command.
$(ItemIndex) Represents an index within a list of items. When no slice is specified, the first $(ItemIndex) is
0. When a slice is specified, $(ItemIndex) is the index of the item within the original list.
$(Step) For the <int expr> specified, $(Step) counts, starting at 0.
$(Row) When a list of items is specified by placing each item on its own line in the submit description file, $(Row)
identifies which line the item is on. The first item (first line of the list) is $(Row) 0. The second item (second
line of the list) is $(Row) 1. When a list of items are specified with all items on the same line, $(Row) is the
same as $(ItemIndex).
Here is an example of a queue command for which the values of these automatic variables are identified.
Example 1
This example queues six jobs.
queue 3 in (A, B)
• $(Process) takes on the six values 0, 1, 2, 3, 4, and 5.
HTCondor Version 8.6.4 Manual
22
2.5.4. Including Submit Commands Defined Elsewhere
• Because there is no specification for the <varname> within this queue command, variable $(Item) is defined. It has the value A for the first three jobs queued, and it has the value B for the second three jobs queued.
• $(Step) takes on the three values 0, 1, and 2 for the three jobs with $(Item)=A, and it takes on the same
three values 0, 1, and 2 for the three jobs with $(Item)=B.
• $(ItemIndex) is 0 for all three jobs with $(Item)=A, and it is 1 for all three jobs with $(Item)=B.
• $(Row) has the same value as $(ItemIndex) for this example.
2.5.4 Including Submit Commands Defined Elsewhere
Externally defined submit commands can be incorporated into the submit description file using the syntax
include : <what-to-include>
The <what-to-include> specification may specify a single file, where the contents of the file will be incorporated into the submit description file at the point within the file where the include is. Or, <what-to-include>
may cause a program to be executed, where the output of the program is incorporated into the submit description
file. The specification of <what-to-include> has the bar character (|) following the name of the program to be
executed.
The include key word is case insensitive. There are no requirements for white space characters surrounding the
colon character.
Included submit commands may contain further nested include specifications, which are also parsed, evaluated,
and incorporated. Levels of nesting on included files are limited, such that infinite nesting is discovered and thwarted,
while still permitting nesting.
Consider the example
include : list-infiles.sh |
In this example, the bar character at the end of the line causes the script list-infiles.sh to be invoked, and the
output of the script is parsed and incorporated into the submit description file. If this bash script contains
echo "transfer_input_files = `ls -m infiles/*.dat`"
then the output of this script has specified the set of input files to transfer to the execute host. For example, if directory
infiles contains the three files A.dat, B.dat, and C.dat, then the submit command
transfer_input_files = infiles/A.dat, infiles/B.dat, infiles/C.dat
is incorporated into the submit description file.
HTCondor Version 8.6.4 Manual
23
2.5.5. Using Conditionals in the Submit Description File
2.5.5 Using Conditionals in the Submit Description File
Conditional if/else semantics are available in a limited form. The syntax:
if <simple condition>
<statement>
. . .
<statement>
else
<statement>
. . .
<statement>
endif
An else key word and statements are not required, such that simple if semantics are implemented. The
<simple condition> does not permit compound conditions. It optionally contains the exclamation point character (!) to represent the not operation, followed by
• the defined keyword followed by the name of a variable. If the variable is defined, the statement(s) are
incorporated into the expanded input. If the variable is not defined, the statement(s) are not incorporated into
the expanded input. As an example,
if defined MY_UNDEFINED_VARIABLE
X = 12
else
X = -1
endif
results in X = -1, when MY_UNDEFINED_VARIABLE is not yet defined.
• the version keyword, representing the version number of of the daemon or tool currently reading this conditional. This keyword is followed by an HTCondor version number. That version number can be of the form
x.y.z or x.y. The version of the daemon or tool is compared to the specified version number. The comparison
operators are
– == for equality. Current version 8.2.3 is equal to 8.2.
– >= to see if the current version number is greater than or equal to. Current version 8.2.3 is greater than
8.2.2, and current version 8.2.3 is greater than or equal to 8.2.
– <= to see if the current version number is less than or equal to. Current version 8.2.0 is less than 8.2.2, and
current version 8.2.3 is less than or equal to 8.2.
As an example,
if version >= 8.1.6
DO_X = True
HTCondor Version 8.6.4 Manual
24
2.5.5. Using Conditionals in the Submit Description File
else
DO_Y = True
endif
results in defining DO_X as True if the current version of the daemon or tool reading this if statement is 8.1.6
or a more recent version.
• True or yes or the value 1. The statement(s) are incorporated.
• False or no or the value 0 The statement(s) are not incorporated.
• $(<variable>) may be used where the immediately evaluated value is a simple boolean value. A value that
evaluates to the empty string is considered False, otherwise a value that does not evaluate to a simple boolean
value is a syntax error.
The syntax
if <simple condition>
<statement>
. . .
<statement>
elif <simple condition>
<statement>
. . .
<statement>
endif
is the same as syntax
if <simple condition>
<statement>
. . .
<statement>
else
if <simple condition>
<statement>
. . .
<statement>
endif
endif
Here is an example use of a conditional in the submit description file. A portion of the sample.sub submit
description file uses the if/else syntax to define command line arguments in one of two ways:
if defined X
HTCondor Version 8.6.4 Manual
25
2.5.6. Function Macros in the Submit Description File
arguments = -n $(X)
else
arguments = -n 1 -debug
endif
Submit variable X is defined on the condor_submit command line with
condor_submit
X=3
sample.sub
This command line incorporates the submit command X = 3 into the submission before parsing the submit description file. For this submission, the command line arguments of the submitted job become
-n 3
If the job were instead submitted with the command line
condor_submit
sample.sub
then the command line arguments of the submitted job become
-n 1 -debug
2.5.6 Function Macros in the Submit Description File
A set of predefined functions increase flexibility. Both submit description files and configuration files are read using
the same parser, so these functions may be used in both submit description files and configuration files.
Case is significant in the function’s name, so use the same letter case as given in these definitions.
$CHOICE(index, listname) or $CHOICE(index, item1, item2, . . .) An item within the list is returned. The list is represented by a parameter name, or the list items are the parameters. The index parameter
determines which item. The first item in the list is at index 0. If the index is out of bounds for the list contents,
an error occurs.
$ENV(environment-variable-name[:default-value]) Evaluates to the value of environment variable
environment-variable-name. If there is no environment variable with that name, Evaluates to UNDEFINED unless the optional :default-value is used; in which case it evaluates to default-value. For
example,
A = $ENV(HOME)
binds A to the value of the HOME environment variable.
HTCondor Version 8.6.4 Manual
26
2.5.6. Function Macros in the Submit Description File
$F[fpduwnxbqa](filename) One or more of the lower case letters may be combined to form the function
name and thus, its functionality. Each letter operates on the filename in its own way.
• f convert relative path to full path by prefixing the current working directory to it. This option works only
in condor_submit files.
• p refers to the entire directory portion of filename, with a trailing slash or backslash character. Whether
a slash or backslash is used depends on the platform of the machine. The slash will be recognized on Linux
platforms; either a slash or backslash will be recognized on Windows platforms, and the parser will use
the same character specified.
• d refers to the last portion of the directory within the path, if specified. It will have a trailing slash or
backslash, as appropriate to the platform of the machine. The slash will be recognized on Linux platforms;
either a slash or backslash will be recognized on Windows platforms, and the parser will use the same
character specified unless u or w is used. if b is used the trailing slash or backslash will be omitted.
• u convert path separators to Unix style slash characters
• w convert path separators to Windows style backslash characters
• n refers to the file name at the end of any path, but without any file name extension. As an example, the
return value from $Fn(/tmp/simulate.exe) will be simulate (without the .exe extension).
• x refers to a file name extension, with the associated period (.). As an example, the return value from
$Fn(/tmp/simulate.exe) will be .exe.
• b when combined with the d option, causes the trailing slash or backslash to be omitted. When combined
with the x option, causes the leading period (.) to be omitted.
• q causes the return value to be enclosed within quotes. Double quote marks are used unless a is also
specified.
• a When combined with the q option, causes the return value to be enclosed within single quotes.
$DIRNAME(filename) is the same as $Fp(filename)
$BASENAME(filename) is the same as $Fnx(filename)
$INT(item-to-convert) or $INT(item-to-convert, format-specifier) Expands, evaluates,
and returns a string version of item-to-convert. The format-specifier has the same syntax as a
C language or Perl format specifier. If no format-specifier is specified, "%d" is used as the format
specifier.
$RANDOM_CHOICE(choice1, choice2, choice3, . . .) A random choice of one of the parameters in the
list of parameters is made. For example, if one of the integers 0-8 (inclusive) should be randomly chosen:
$RANDOM_CHOICE(0,1,2,3,4,5,6,7,8)
$RANDOM_INTEGER(min, max [, step]) A random integer within the range min and max, inclusive, is
selected. The optional step parameter controls the stride within the range, and it defaults to the value 1. For
example, to randomly chose an even integer in the range 0-8 (inclusive):
$RANDOM_INTEGER(0, 8, 2)
HTCondor Version 8.6.4 Manual
27
2.5.6. Function Macros in the Submit Description File
$REAL(item-to-convert) or $REAL(item-to-convert, format-specifier) Expands, evaluates,
and returns a string version of item-to-convert for a floating point type. The format-specifier is
a C language or Perl format specifier. If no format-specifier is specified, "%16G" is used as a format
specifier.
$SUBSTR(name, start-index) or $SUBSTR(name, start-index, length) Expands name and returns a substring of it. The first character of the string is at index 0. The first character of the substring is at
index start-index. If the optional length is not specified, then the substring includes characters up to the
end of the string. A negative value of start-index works back from the end of the string. A negative value
of length eliminates use of characters from the end of the string. Here are some examples that all assume
Name = abcdef
• $SUBSTR(Name, 2) is cdef.
• $SUBSTR(Name, 0, -2) is abcd.
• $SUBSTR(Name, 1, 3) is bcd.
• $SUBSTR(Name, -1) is f.
• $SUBSTR(Name, 4, -3) is the empty string, as there are no characters in the substring for this request.
Here are example uses of the function macros in a submit description file. Note that these are not complete submit
description files, but only the portions that promote understanding of use cases of the function macros.
Example 1
Generate a range of numerical values for a set of jobs, where values other than those given by $(Process) are
desired.
MyIndex
= $(Process) + 1
initial_dir = run-$INT(MyIndex, %04d)
Assuming that there are three jobs queued, such that $(Process) becomes 0, 1, and 2, initial_dir will evaluate
to the directories run-0001, run-0002, and run-0003.
Example 2
This variation on Example 1 generates a file name extension which is a 3-digit integer value.
Values
Extension
input
= $(Process) * 10
= $INT(Values, %03d)
= X.$(Extension)
HTCondor Version 8.6.4 Manual
28
2.5.7. About Requirements and Rank
29
Assuming that there are four jobs queued, such that $(Process) becomes 0, 1, 2, and 3, Extension will evaluate
to 000, 010, 020, and 030, leading to files defined for input of X.000, X.010, X.020, and X.030.
Example 3
This example uses both the file globbing of the queue command and a macro function to specify a job input file that
is within a subdirectory on the submit host, but will be placed into a single, flat directory on the execute host.
arguments
= $Fnx(FILE)
transfer_input_files = $(FILE)
queue FILE MATCHING (
samplerun/*.dat
)
Assume that two files that end in .dat, A.dat and B.dat, are within the directory samplerun. Macro FILE
expands to samplerun/A.dat and samplerun/B.dat for the two jobs queued. The input files transferred are
samplerun/A.dat and samplerun/B.dat on the submit host. The $Fnx() function macro expands to the
complete file name with any leading directory specification stripped, such that the command line argument for one of
the jobs will be A.dat and the command line argument for the other job will be B.dat.
2.5.7 About Requirements and Rank
The requirements and rank commands in the submit description file are powerful and flexible.
effectively requires care, and this section presents those details.
Using them
Both requirements and rank need to be specified as valid HTCondor ClassAd expressions, however, default
values are set by the condor_submit program if these are not defined in the submit description file. From the condor_submit manual page and the above examples, you see that writing ClassAd expressions is intuitive, especially
if you are familiar with the programming language C. There are some pretty nifty expressions you can write with
ClassAds. A complete description of ClassAds and their expressions can be found in section 4.1 on page 523.
All of the commands in the submit description file are case insensitive, except for the ClassAd attribute string
values. ClassAd attribute names are case insensitive, but ClassAd string values are case preserving.
Note that the comparison operators (<, >, <=, >=, and ==) compare strings case insensitively. The special comparison operators =?= and =!= compare strings case sensitively.
A requirements or rank command in the submit description file may utilize attributes that appear in a machine
or a job ClassAd. Within the submit description file (for a job) the prefix MY. (on a ClassAd attribute name) causes a
reference to the job ClassAd attribute, and the prefix TARGET. causes a reference to a potential machine or matched
machine ClassAd attribute.
The condor_status command displays statistics about machines within the pool. The -l option displays the machine
ClassAd attributes for all machines in the HTCondor pool. The job ClassAds, if there are jobs in the queue, can be
seen with the condor_q -l command. This shows all the defined attributes for current jobs in the queue.
HTCondor Version 8.6.4 Manual
2.5.7. About Requirements and Rank
30
A list of defined ClassAd attributes for job ClassAds is given in the unnumbered Appendix on page 1002. A list
of defined ClassAd attributes for machine ClassAds is given in the unnumbered Appendix on page 1020.
Rank Expression Examples
When considering the match between a job and a machine, rank is used to choose a match from among all machines
that satisfy the job’s requirements and are available to the user, after accounting for the user’s priority and the machine’s
rank of the job. The rank expressions, simple or complex, define a numerical value that expresses preferences.
The job’s Rank expression evaluates to one of three values. It can be UNDEFINED, ERROR, or a floating point
value. If Rank evaluates to a floating point value, the best match will be the one with the largest, positive value. If
no Rank is given in the submit description file, then HTCondor substitutes a default value of 0.0 when considering
machines to match. If the job’s Rank of a given machine evaluates to UNDEFINED or ERROR, this same value of
0.0 is used. Therefore, the machine is still considered for a match, but has no ranking above any other.
A boolean expression evaluates to the numerical value of 1.0 if true, and 0.0 if false.
The following Rank expressions provide examples to follow.
For a job that desires the machine with the most available memory:
Rank = memory
For a job that prefers to run on a friend’s machine on Saturdays and Sundays:
Rank = ( (clockday == 0) || (clockday == 6) )
&& (machine == "friend.cs.wisc.edu")
For a job that prefers to run on one of three specific machines:
Rank = (machine == "friend1.cs.wisc.edu") ||
(machine == "friend2.cs.wisc.edu") ||
(machine == "friend3.cs.wisc.edu")
For a job that wants the machine with the best floating point performance (on Linpack benchmarks):
Rank = kflops
This particular example highlights a difficulty with Rank expression evaluation as currently defined. While all machines have floating point processing ability, not all machines will have the kflops attribute defined. For machines
where this attribute is not defined, Rank will evaluate to the value UNDEFINED, and HTCondor will use a default
rank of the machine of 0.0. The Rank attribute will only rank machines where the attribute is defined. Therefore, the
machine with the highest floating point performance may not be the one given the highest rank.
HTCondor Version 8.6.4 Manual
2.5.8. Submitting Jobs Using a Shared File System
So, it is wise when writing a Rank expression to check if the expression’s evaluation will lead to the expected
resulting ranking of machines. This can be accomplished using the condor_status command with the -constraint
argument. This allows the user to see a list of machines that fit a constraint. To see which machines in the pool have
kflops defined, use
condor_status -constraint kflops
Alternatively, to see a list of machines where kflops is not defined, use
condor_status -constraint "kflops=?=undefined"
For a job that prefers specific machines in a specific order:
Rank = ((machine == "friend1.cs.wisc.edu")*3) +
((machine == "friend2.cs.wisc.edu")*2) +
(machine == "friend3.cs.wisc.edu")
If the machine being ranked is friend1.cs.wisc.edu, then the expression
(machine == "friend1.cs.wisc.edu")
is true, and gives the value 1.0. The expressions
(machine == "friend2.cs.wisc.edu")
and
(machine == "friend3.cs.wisc.edu")
are false, and give the value 0.0.
Therefore, Rank evaluates to the value 3.0.
In this way,
machine friend1.cs.wisc.edu is ranked higher than machine friend2.cs.wisc.edu, machine
friend2.cs.wisc.edu is ranked higher than machine friend3.cs.wisc.edu, and all three of these machines are ranked higher than others.
2.5.8
Submitting Jobs Using a Shared File System
If vanilla, java, or parallel universe jobs are submitted without using the File Transfer mechanism, HTCondor must
use a shared file system to access input and output files. In this case, the job must be able to access the data files from
any machine on which it could potentially run.
As an example, suppose a job is submitted from blackbird.cs.wisc.edu, and the job requires a particular data file called /u/p/s/psilord/data.txt. If the job were to run on cardinal.cs.wisc.edu, the file
/u/p/s/psilord/data.txt must be available through either NFS or AFS for the job to run correctly.
HTCondor Version 8.6.4 Manual
31
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
HTCondor allows users to ensure their jobs have access to the right shared files by using the
FileSystemDomain and UidDomain machine ClassAd attributes. These attributes specify which machines have
access to the same shared file systems. All machines that mount the same shared directories in the same locations are
considered to belong to the same file system domain. Similarly, all machines that share the same user information (in
particular, the same UID, which is important for file systems like NFS) are considered part of the same UID domain.
The default configuration for HTCondor places each machine in its own UID domain and file system domain, using
the full host name of the machine as the name of the domains. So, if a pool does have access to a shared file system,
the pool administrator must correctly configure HTCondor such that all the machines mounting the same files have
the same FileSystemDomain configuration. Similarly, all machines that share common user information must be
configured to have the same UidDomain configuration.
When a job relies on a shared file system, HTCondor uses the requirements expression to ensure that the job
runs on a machine in the correct UidDomain and FileSystemDomain. In this case, the default requirements
expression specifies that the job must run on a machine with the same UidDomain and FileSystemDomain as
the machine from which the job is submitted. This default is almost always correct. However, in a pool spanning
multiple UidDomains and/or FileSystemDomains, the user may need to specify a different requirements
expression to have the job run on the correct machines.
For example, imagine a pool made up of both desktop workstations and a dedicated compute cluster. Most of
the pool, including the compute cluster, has access to a shared file system, but some of the desktop machines do
not. In this case, the administrators would probably define the FileSystemDomain to be cs.wisc.edu for all
the machines that mounted the shared files, and to the full host name for each machine that did not. An example is
jimi.cs.wisc.edu.
In this example, a user wants to submit vanilla universe jobs from her own desktop machine (jimi.cs.wisc.edu)
which does not mount the shared file system (and is therefore in its own file system domain, in its own world). But,
she wants the jobs to be able to run on more than just her own machine (in particular, the compute cluster), so she puts
the program and input files onto the shared file system. When she submits the jobs, she needs to tell HTCondor to
send them to machines that have access to that shared data, so she specifies a different requirements expression
than the default:
Requirements = TARGET.UidDomain == "cs.wisc.edu" && \
TARGET.FileSystemDomain == "cs.wisc.edu"
WARNING: If there is no shared file system, or the HTCondor pool administrator does not configure the
FileSystemDomain setting correctly (the default is that each machine in a pool is in its own file system and
UID domain), a user submits a job that cannot use remote system calls (for example, a vanilla universe job), and the
user does not enable HTCondor’s File Transfer mechanism, the job will only run on the machine from which it was
submitted.
2.5.9
Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
HTCondor works well without a shared file system. The HTCondor file transfer mechanism permits the user to select
which files are transferred and under which circumstances. HTCondor can transfer any files needed by a job from the
HTCondor Version 8.6.4 Manual
32
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
machine where the job was submitted into a remote scratch directory on the machine where the job is to be executed.
HTCondor executes the job and transfers output back to the submitting machine. The user specifies which files and
directories to transfer, and at what point the output files should be copied back to the submitting machine. This
specification is done within the job’s submit description file.
Specifying If and When to Transfer Files
To enable the file transfer mechanism, place two commands in the job’s submit description file: should_transfer_files
and when_to_transfer_output. By default, they will be:
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
Setting the should_transfer_files command explicitly enables or disables the file transfer mechanism. The command takes on one of three possible values:
1. YES: HTCondor transfers both the executable and the file defined by the input command from the machine
where the job is submitted to the remote machine where the job is to be executed. The file defined by the output
command as well as any files created by the execution of the job are transferred back to the machine where
the job was submitted. When they are transferred and the directory location of the files is determined by the
command when_to_transfer_output.
2. IF_NEEDED: HTCondor transfers files if the job is matched with and to be executed on a machine in a different FileSystemDomain than the one the submit machine belongs to, the same
as if should_transfer_files = YES. If the job is matched with a machine in the local
FileSystemDomain, HTCondor will not transfer files and relies on the shared file system.
3. NO: HTCondor’s file transfer mechanism is disabled.
The when_to_transfer_output command tells HTCondor when output files are to be transferred back to the
submit machine. The command takes on one of two possible values:
1. ON_EXIT: HTCondor transfers the file defined by the output command, as well as any other files in the remote
scratch directory created by the job, back to the submit machine only when the job exits on its own.
2. ON_EXIT_OR_EVICT: HTCondor behaves the same as described for the value ON_EXIT when the job exits
on its own. However, if, and each time the job is evicted from a machine, files are transferred back at eviction
time. The files that are transferred back at eviction time may include intermediate files that are not part of the
final output of the job. When transfer_output_files is specified, its list governs which are transferred back at
eviction time. Before the job starts running again, all of the files that were stored when the job was last evicted
are copied to the job’s new remote scratch directory.
The purpose of saving files at eviction time is to allow the job to resume from where it left off. This is similar to
using the checkpoint feature of the standard universe, but just specifying ON_EXIT_OR_EVICT is not enough
HTCondor Version 8.6.4 Manual
33
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
to make a job capable of producing or utilizing checkpoints. The job must be designed to save and restore its
state using the files that are saved at eviction time.
The files that are transferred back at eviction time are not stored in the location where the job’s final output will
be written when the job exits. HTCondor manages these files automatically, so usually the only reason for a
user to worry about them is to make sure that there is enough space to store them. The files are stored on the
submit machine in a temporary directory within the directory defined by the configuration variable SPOOL. The
directory is named using the ClusterId and ProcId job ClassAd attributes. The directory name takes the
form:
<X mod 10000>/<Y mod 10000>/cluster<X>.proc<Y>.subproc0
where <X> is the value of ClusterId, and <Y> is the value of ProcId. As an example, if job 735.0 is
evicted, it will produce the directory
$(SPOOL)/735/0/cluster735.proc0.subproc0
The default values for these two submit commands make sense as used together. If only should_transfer_files
is set, and set to the value NO, then no output files will be transferred, and the value of when_to_transfer_output
is irrelevant. If only when_to_transfer_output is set, and set to the value ON_EXIT_OR_EVICT, then the default
value for an unspecified should_transfer_files will be YES.
Note that the combination of
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT_OR_EVICT
would produce undefined file access semantics. Therefore, this combination is prohibited by condor_submit.
Specifying What Files to Transfer
If the file transfer mechanism is enabled, HTCondor will transfer the following files before the job is run on a remote
machine.
1. the executable, as defined with the executable command
2. the input, as defined with the input command
3. any jar files, for the java universe, as defined with the jar_files command
If the job requires other input files, the submit description file should utilize the transfer_input_files command. This
comma-separated list specifies any other files or directories that HTCondor is to transfer to the remote scratch directory,
to set up the execution environment for the job before it is run. These files are placed in the same directory as the job’s
executable. For example:
HTCondor Version 8.6.4 Manual
34
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = file1,file2
This example explicitly enables the file transfer mechanism, and it transfers the executable, the file specified by the
input command, any jar files specified by the jar_files command, and files file1 and file2.
If the file transfer mechanism is enabled, HTCondor will transfer the following files from the execute machine
back to the submit machine after the job exits.
1. the output file, as defined with the output command
2. the error file, as defined with the error command
3. any files created by the job in the remote scratch directory; this only occurs for jobs other than grid universe,
and for HTCondor-C grid universe jobs; directories created by the job within the remote scratch directory are
ignored for this automatic detection of files to be transferred.
A path given for output and error commands represents a path on the submit machine. If no path is specified, the
directory specified with initialdir is used, and if that is not specified, the directory from which the job was submitted
is used. At the time the job is submitted, zero-length files are created on the submit machine, at the given path for the
files defined by the output and error commands. This permits job submission failure, if these files cannot be written
by HTCondor.
To restrict the output files or permit entire directory contents to be transferred, specify the exact list with transfer_output_files. Delimit the list of file names, directory names, or paths with commas. When this list is defined,
and any of the files or directories do not exist as the job exits, HTCondor considers this an error, and places the job
on hold. Setting transfer_output_files to the empty string ("") means no files are to be transferred. When this list is
defined, automatic detection of output files created by the job is disabled. Paths specified in this list refer to locations
on the execute machine. The naming and placement of files and directories relies on the term base name. By example,
the path a/b/c has the base name c. It is the file name or directory name with all directories leading up to that
name stripped off. On the submit machine, the transferred files or directories are named using only the base name.
Therefore, each output file or directory must have a different name, even if they originate from different paths.
For grid universe jobs other than than HTCondor-C grid jobs, files to be transferred (other than standard output
and standard error) must be specified using transfer_output_files in the submit description file, because automatic
detection of new files created by the job does not take place.
Here are examples to promote understanding of what files and directories are transferred, and how they are named
after transfer. Assume that the job produces the following structure within the remote scratch directory:
o1
o2
d1 (directory)
o3
o4
HTCondor Version 8.6.4 Manual
35
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
If the submit description file sets
transfer_output_files = o1,o2,d1
then transferred back to the submit machine will be
o1
o2
d1 (directory)
o3
o4
Note that the directory d1 and all its contents are specified, and therefore transferred. If the directory d1 is not created
by the job before exit, then the job is placed on hold. If the directory d1 is created by the job before exit, but is empty,
this is not an error.
If, instead, the submit description file sets
transfer_output_files = o1,o2,d1/o3
then transferred back to the submit machine will be
o1
o2
o3
Note that only the base name is used in the naming and placement of the file specified with d1/o3.
File Paths for File Transfer
The file transfer mechanism specifies file names and/or paths on both the file system of the submit machine and on the
file system of the execute machine. Care must be taken to know which machine, submit or execute, is utilizing the file
name and/or path.
Files in the transfer_input_files command are specified as they are accessed on the submit machine. The job, as
it executes, accesses files as they are found on the execute machine.
There are three ways to specify files and paths for transfer_input_files:
1. Relative to the current working directory as the job is submitted, if the submit command initialdir is not specified.
2. Relative to the initial directory, if the submit command initialdir is specified.
HTCondor Version 8.6.4 Manual
36
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
3. Absolute.
Before executing the program, HTCondor copies the executable, an input file as specified by the submit command
input, along with any input files specified by transfer_input_files. All these files are placed into a remote scratch
directory on the execute machine, in which the program runs. Therefore, the executing program must access input
files relative to its working directory. Because all files and directories listed for transfer are placed into a single, flat
directory, inputs must be uniquely named to avoid collision when transferred. A collision causes the last file in the list
to overwrite the earlier one.
Both relative and absolute paths may be used in transfer_output_files. Relative paths are relative to the job’s
remote scratch directory on the execute machine. When the files and directories are copied back to the submit machine,
they are placed in the job’s initial working directory as the base name of the original path. An alternate name or path
may be specified by using transfer_output_remaps.
A job may create files outside the remote scratch directory but within the file system of the execute machine, in
a directory such as /tmp, if this directory is guaranteed to exist and be accessible on all possible execute machines.
However, HTCondor will not automatically transfer such files back after execution completes, nor will it clean up
these files.
Here are several examples to illustrate the use of file transfer. The program executable is called my_program, and
it uses three command-line arguments as it executes: two input file names and an output file name. The program
executable and the submit description file for this job are located in directory /scratch/test.
Here is the directory tree as it exists on the submit machine, for all the examples:
/scratch/test (directory)
my_program.condor (the submit description file)
my_program (the executable)
files (directory)
logs2 (directory)
in1 (file)
in2 (file)
logs (directory)
Example 1 This first example explicitly transfers input files. These input files to be transferred are specified relative
to the directory where the job is submitted. An output file specified in the arguments command, out1, is
created when the job is executed. It will be transferred back into the directory /scratch/test.
# file name: my_program.condor
# HTCondor submit description file for my_program
Executable
= my_program
Universe
= vanilla
Error
= logs/err.$(cluster)
Output
= logs/out.$(cluster)
Log
= logs/log.$(cluster)
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = files/in1,files/in2
HTCondor Version 8.6.4 Manual
37
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
Arguments
Queue
= in1 in2 out1
The log file is written on the submit machine, and is not involved with the file transfer mechanism.
Example 2 This second example is identical to Example 1, except that absolute paths to the input files are specified,
instead of relative paths to the input files.
# file name: my_program.condor
# HTCondor submit description file for my_program
Executable
= my_program
Universe
= vanilla
Error
= logs/err.$(cluster)
Output
= logs/out.$(cluster)
Log
= logs/log.$(cluster)
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = /scratch/test/files/in1,/scratch/test/files/in2
Arguments
Queue
= in1 in2 out1
Example 3 This third example illustrates the use of the submit command initialdir, and its effect on the paths used
for the various files. The expected location of the executable is not affected by the initialdir command. All
other files (specified by input, output, error, transfer_input_files, as well as files modified or created by the
job and automatically transferred back) are located relative to the specified initialdir. Therefore, the output file,
out1, will be placed in the files directory. Note that the logs2 directory exists to make this example work
correctly.
# file name: my_program.condor
# HTCondor submit description file for my_program
Executable
= my_program
Universe
= vanilla
Error
= logs2/err.$(cluster)
Output
= logs2/out.$(cluster)
Log
= logs2/log.$(cluster)
initialdir
= files
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = in1,in2
Arguments
Queue
= in1 in2 out1
Example 4 – Illustrates an Error This example illustrates a job that will fail. The files specified using the transfer_input_files command work correctly (see Example 1). However, relative paths to files in the arguments
command cause the executing program to fail. The file system on the submission side may utilize relative paths
to files, however those files are placed into the single, flat, remote scratch directory on the execute machine.
HTCondor Version 8.6.4 Manual
38
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
# file name: my_program.condor
# HTCondor submit description file for my_program
Executable
= my_program
Universe
= vanilla
Error
= logs/err.$(cluster)
Output
= logs/out.$(cluster)
Log
= logs/log.$(cluster)
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = files/in1,files/in2
Arguments
Queue
= files/in1 files/in2 files/out1
This example fails with the following error:
err: files/out1: No such file or directory.
Example 5 – Illustrates an Error As with Example 4, this example illustrates a job that will fail. The executing
program’s use of absolute paths cannot work.
# file name: my_program.condor
# HTCondor submit description file for my_program
Executable
= my_program
Universe
= vanilla
Error
= logs/err.$(cluster)
Output
= logs/out.$(cluster)
Log
= logs/log.$(cluster)
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = /scratch/test/files/in1, /scratch/test/files/in2
Arguments = /scratch/test/files/in1 /scratch/test/files/in2 /scratch/test/files/out1
Queue
The job fails with the following error:
err: /scratch/test/files/out1: No such file or directory.
Example 6 This example illustrates a case where the executing program creates an output file in a directory other than
within the remote scratch directory that the program executes within. The file creation may or may not cause an
error, depending on the existence and permissions of the directories on the remote file system.
The output file /tmp/out1 is transferred back to the job’s initial working directory as
/scratch/test/out1.
# file name: my_program.condor
# HTCondor submit description file for my_program
Executable
= my_program
Universe
= vanilla
Error
= logs/err.$(cluster)
Output
= logs/out.$(cluster)
HTCondor Version 8.6.4 Manual
39
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
Log
= logs/log.$(cluster)
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = files/in1,files/in2
transfer_output_files = /tmp/out1
Arguments
Queue
= in1 in2 /tmp/out1
Behavior for Error Cases
This section describes HTCondor’s behavior for some error cases in dealing with the transfer of files.
Disk Full on Execute Machine When transferring any files from the submit machine to the remote scratch directory,
if the disk is full on the execute machine, then the job is place on hold.
Error Creating Zero-Length Files on Submit Machine As a job is submitted, HTCondor creates zero-length files
as placeholders on the submit machine for the files defined by output and error. If these files cannot be created,
then job submission fails.
This job submission failure avoids having the job run to completion, only to be unable to transfer the job’s output
due to permission errors.
Error When Transferring Files from Execute Machine to Submit Machine When a job exits, or potentially when
a job is evicted from an execute machine, one or more files may be transferred from the execute machine back
to the machine on which the job was submitted.
During transfer, if any of the following three similar types of errors occur, the job is put on hold as the error
occurs.
1. If the file cannot be opened on the submit machine, for example because the system is out of inodes.
2. If the file cannot be written on the submit machine, for example because the permissions do not permit it.
3. If the write of the file on the submit machine fails, for example because the system is out of disk space.
File Transfer Using a URL
Instead of file transfer that goes only between the submit machine and the execute machine, HTCondor has the ability
to transfer files from a location specified by a URL for a job’s input file, or from the execute machine to a location specified by a URL for a job’s output file(s). This capability requires administrative set up, as described in section 3.14.2.
The transfer of an input file is restricted to vanilla and vm universe jobs only. HTCondor’s file transfer mechanism must be enabled. Therefore, the submit description file for the job will define both should_transfer_files
and when_to_transfer_output. In addition, the URL for any files specified with a URL are given in the transfer_input_files command. An example portion of the submit description file for a job that has a single file specified
with a URL:
HTCondor Version 8.6.4 Manual
40
2.5.9. Submitting Jobs Without a Shared File System: HTCondor’s File Transfer Mechanism
41
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = http://www.full.url/path/to/filename
The destination file is given by the file name within the URL.
For the transfer of the entire contents of the output sandbox, which are all files that the job creates or modifies,
HTCondor’s file transfer mechanism must be enabled. In this sample portion of the submit description file, the first
two commands explicitly enable file transfer, and the added output_destination command provides both the protocol
to be used and the destination of the transfer.
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
output_destination = urltype://path/to/destination/directory
Note that with this feature, no files are transferred back to the submit machine. This does not interfere with the
streaming of output.
If only a subset of the output sandbox should be transferred, the subset is specified by further adding a submit
command of the form:
transfer_output_files = file1, file2
Requirements and Rank for File Transfer
The requirements expression for a job must depend on the should_transfer_files command. The job
must specify the correct logic to ensure that the job is matched with a resource that meets the file transfer needs.
If no requirements expression is in the submit description file, or if the expression specified does not refer to
the attributes listed below, condor_submit adds an appropriate clause to the requirements expression for the job.
condor_submit appends these clauses with a logical AND, &&, to ensure that the proper conditions are met. Here are
the default clauses corresponding to the different values of should_transfer_files:
1. should_transfer_files = YES results in the addition of the clause (HasFileTransfer). If the
job is always going to transfer files, it is required to match with a machine that has the capability to transfer
files.
2. should_transfer_files = NO results in the addition of (TARGET.FileSystemDomain == MY.FileSystem
In addition, HTCondor automatically adds the FileSystemDomain attribute to the job ClassAd, with whatever string is defined for the condor_schedd to which the job is submitted. If the job is not using the file
transfer mechanism, HTCondor assumes it will need a shared file system, and therefore, a machine in the same
FileSystemDomain as the submit machine.
3. should_transfer_files = IF_NEEDED results in the addition of
(HasFileTransfer || (TARGET.FileSystemDomain == MY.FileSystemDomain))
If HTCondor will optionally transfer files, it must require that the machine is either capable of transferring files
or in the same file system domain.
HTCondor Version 8.6.4 Manual
2.5.10. Environment Variables
42
To ensure that the job is matched to a machine with enough local disk space to hold all the transferred files, HTCondor automatically adds the DiskUsage job attribute. This attribute includes the total size of the job’s executable
and all input files to be transferred. HTCondor then adds an additional clause to the Requirements expression that
states that the remote machine must have at least enough available disk space to hold all these files:
&& (Disk >= DiskUsage)
If should_transfer_files = IF_NEEDED and the job prefers to run on a machine in the local file system
domain over transferring files, but is still willing to allow the job to run remotely and transfer files, the Rank expression
works well. Use:
rank = (TARGET.FileSystemDomain == MY.FileSystemDomain)
The Rank expression is a floating point value, so if other items are considered in ranking the possible machines
this job may run on, add the items:
Rank = kflops + (TARGET.FileSystemDomain == MY.FileSystemDomain)
The value of kflops can vary widely among machines, so this Rank expression will likely not do as it intends.
To place emphasis on the job running in the same file system domain, but still consider floating point speed among the
machines in the file system domain, weight the part of the expression that is matching the file system domains. For
example:
Rank = kflops + (10000 * (TARGET.FileSystemDomain == MY.FileSystemDomain))
2.5.10 Environment Variables
The environment under which a job executes often contains information that is potentially useful to the job. HTCondor
allows a user to both set and reference environment variables for a job or job cluster.
Within a submit description file, the user may define environment variables for the job’s environment by using
the environment command. See within the condor_submit manual page at section 11 for more details about this
command.
The submitter’s entire environment can be copied into the job ClassAd for the job at job submission. The getenv
command within the submit description file does this, as described at section 11.
If the environment is set with the environment command and getenv is also set to true, values specified with
environment override values in the submitter’s environment, regardless of the order of the environment and getenv
commands.
Commands within the submit description file may reference the environment variables of the submitter as a job is
submitted. Submit description file commands use $ENV(EnvironmentVariableName) to reference the value
of an environment variable.
HTCondor Version 8.6.4 Manual
2.5.11. Heterogeneous Submit: Execution on Differing Architectures
HTCondor sets several additional environment variables for each executing job that may be useful for the job to
reference.
• _CONDOR_SCRATCH_DIR gives the directory where the job may place temporary data files. This directory
is unique for every job that is run, and its contents are deleted by HTCondor when the job stops running on a
machine, no matter how the job completes.
• _CONDOR_SLOT gives the name of the slot (for SMP machines), on which the job is run. On machines with
only a single slot, the value of this variable will be 1, just like the SlotID attribute in the machine’s ClassAd.
This setting is available in all universes. See section 3.7.1 for more details about SMP machines and their
configuration.
• CONDOR_VM equivalent to _CONDOR_SLOT described above, except that it is only available in the standard
universe. NOTE: As of HTCondor version 6.9.3, this environment variable is no longer used. It will only be
defined if the ALLOW_VM_CRUFT configuration variable is set to True.
• X509_USER_PROXY gives the full path to the X.509 user proxy file if one is associated with the job. Typically,
a user will specify x509userproxy in the submit description file. This setting is currently available in the local,
java, and vanilla universes.
• _CONDOR_JOB_AD is the path to a file in the job’s scratch directory which contains the job ad for the currently
running job. The job ad is current as of the start of the job, but is not updated during the running of the job. The
job may read attributes and their values out of this file as it runs, but any changes will not be acted on in any way
by HTCondor. The format is the same as the output of the condor_q -l command. This environment variable
may be particularly useful in a USER_JOB_WRAPPER.
• _CONDOR_MACHINE_AD is the path to a file in the job’s scratch directory which contains the machine ad for
the slot the currently running job is using. The machine ad is current as of the start of the job, but is not updated
during the running of the job. The format is the same as the output of the condor_status -l command.
• _CONDOR_JOB_IWD is the path to the initial working directory the job was born with.
• _CONDOR_WRAPPER_ERROR_FILE
is only set when the administrator has installed a
USER_JOB_WRAPPER. If this file exists, HTCondor assumes that the job wrapper has failed and copies the
contents of the file to the StarterLog for the administrator to debug the problem.
• CONDOR_IDS overrides the value of configuration variable CONDOR_IDS, when set in the environment.
• CONDOR_ID is set for scheduler universe jobs to be the same as the ClusterId attribute.
2.5.11 Heterogeneous Submit: Execution on Differing Architectures
If executables are available for the different platforms of machines in the HTCondor pool, HTCondor can be allowed
the choice of a larger number of machines when allocating a machine for a job. Modifications to the submit description
file allow this choice of platforms.
A simplified example is a cross submission. An executable is available for one platform, but the submission is done
from a different platform. Given the correct executable, the requirements command in the submit description file
HTCondor Version 8.6.4 Manual
43
2.5.11. Heterogeneous Submit: Execution on Differing Architectures
specifies the target architecture. For example, an executable compiled for a 32-bit Intel processor running Windows
Vista, submitted from an Intel architecture running Linux would add the requirement
requirements = Arch == "INTEL" && OpSys == "WINDOWS"
Without this requirement, condor_submit will assume that the program is to be executed on a machine with the
same platform as the machine where the job is submitted.
Cross submission works for all universes except scheduler and local. See section 5.3.10 for how matchmaking works in the grid universe. The burden is on the user to both obtain and specify the correct executable for the
target architecture. To list the architecture and operating systems of the machines in a pool, run condor_status.
Vanilla Universe Example for Execution on Differing Architectures
A more complex example of a heterogeneous submission occurs when a job may be executed on many different
architectures to gain full use of a diverse architecture and operating system pool. If the executables are available
for the different architectures, then a modification to the submit description file will allow HTCondor to choose an
executable after an available machine is chosen.
A special-purpose Machine Ad substitution macro can be used in string attributes in the submit description file.
The macro has the form
$$(MachineAdAttribute)
The $$() informs HTCondor to substitute the requested MachineAdAttribute from the machine where the job
will be executed.
An example of the heterogeneous job submission has executables available for two platforms: RHEL 3 on both
32-bit and 64-bit Intel processors. This example uses povray to render images using a popular free rendering engine.
The substitution macro chooses a specific executable after a platform for running the job is chosen. These executables must therefore be named based on the machine attributes that describe a platform. The executables named
povray.LINUX.INTEL
povray.LINUX.X86_64
will work correctly for the macro
povray.$$(OpSys).$$(Arch)
The executables or links to executables with this name are placed into the initial working directory so that they
may be found by HTCondor. A submit description file that queues three jobs for this example:
####################
HTCondor Version 8.6.4 Manual
44
2.5.11. Heterogeneous Submit: Execution on Differing Architectures
#
# Example of heterogeneous submission
#
####################
universe
Executable
Log
Output
Error
=
=
=
=
=
vanilla
povray.$$(OpSys).$$(Arch)
povray.log
povray.out.$(Process)
povray.err.$(Process)
Requirements = (Arch == "INTEL" && OpSys == "LINUX") || \
(Arch == "X86_64" && OpSys =="LINUX")
Arguments
Queue
= +W1024 +H768 +Iimage1.pov
Arguments
Queue
= +W1024 +H768 +Iimage2.pov
Arguments
Queue
= +W1024 +H768 +Iimage3.pov
These jobs are submitted to the vanilla universe to assure that once a job is started on a specific platform, it will
finish running on that platform. Switching platforms in the middle of job execution cannot work correctly.
There are two common errors made with the substitution macro. The first is the use of a non-existent
MachineAdAttribute. If the specified MachineAdAttribute does not exist in the machine’s ClassAd, then
HTCondor will place the job in the held state until the problem is resolved.
The second common error occurs due to an incomplete job set up. For example, the submit description file given
above specifies three available executables. If one is missing, HTCondor reports back that an executable is missing
when it happens to match the job with a resource that requires the missing binary.
Standard Universe Example for Execution on Differing Architectures
Jobs submitted to the standard universe may produce checkpoints. A checkpoint can then be used to start up and
continue execution of a partially completed job. For a partially completed job, the checkpoint and the job are specific
to a platform. If migrated to a different machine, correct execution requires that the platform must remain the same.
In previous versions of HTCondor, the author of the heterogeneous submission file would need to write extra policy expressions in the requirements expression to force HTCondor to choose the same type of platform when
continuing a checkpointed job. However, since it is needed in the common case, this additional policy is now automatically added to the requirements expression. The additional expression is added provided the user does not use
CkptArch in the requirements expression. HTCondor will remain backward compatible for those users who
have explicitly specified CkptRequirements–implying use of CkptArch, in their requirements expression.
HTCondor Version 8.6.4 Manual
45
2.5.11. Heterogeneous Submit: Execution on Differing Architectures
The expression added when the attribute CkptArch is not specified will default to
# Added by HTCondor
CkptRequirements = ((CkptArch == Arch) || (CkptArch =?= UNDEFINED)) && \
((CkptOpSys == OpSys) || (CkptOpSys =?= UNDEFINED))
Requirements = (<user specified policy>) && $(CkptRequirements)
The behavior of the CkptRequirements expressions and its addition to requirements is as follows. The
CkptRequirements expression guarantees correct operation in the two possible cases for a job. In the first case,
the job has not produced a checkpoint. The ClassAd attributes CkptArch and CkptOpSys will be undefined, and
therefore the meta operator (=?=) evaluates to true. In the second case, the job has produced a checkpoint. The
Machine ClassAd is restricted to require further execution only on a machine of the same platform. The attributes
CkptArch and CkptOpSys will be defined, ensuring that the platform chosen for further execution will be the
same as the one used just before the checkpoint.
Note that this restriction of platforms also applies to platforms where the executables are binary compatible.
The complete submit description file for this example:
####################
#
# Example of heterogeneous submission
#
####################
universe
Executable
Log
Output
Error
=
=
=
=
=
standard
povray.$$(OpSys).$$(Arch)
povray.log
povray.out.$(Process)
povray.err.$(Process)
# HTCondor automatically adds the correct expressions to insure that the
# checkpointed jobs will restart on the correct platform types.
Requirements = ( (Arch == "INTEL" && OpSys == "LINUX") || \
(Arch == "X86_64" && OpSys == "LINUX") )
Arguments
Queue
= +W1024 +H768 +Iimage1.pov
Arguments
Queue
= +W1024 +H768 +Iimage2.pov
Arguments
Queue
= +W1024 +H768 +Iimage3.pov
HTCondor Version 8.6.4 Manual
46
2.5.11. Heterogeneous Submit: Execution on Differing Architectures
47
Vanilla Universe Example for Execution on Differing Operating Systems
The addition of several related OpSys attributes assists in selection of specific operating systems and versions in
heterogeneous pools.
####################
#
# Example of submission targeting RedHat platforms in a heterogeneous Linux pool
#
####################
universe
Executable
Log
Output
Error
=
=
=
=
=
vanilla
/bin/date
distro.log
distro.out
distro.err
Requirements = (OpSysName == "RedHat")
Queue
####################
#
# Example of submission targeting RedHat 6 platforms in a heterogeneous Linux pool
#
####################
universe
Executable
Log
Output
Error
=
=
=
=
=
vanilla
/bin/date
distro.log
distro.out
distro.err
Requirements = ( OpSysName == "RedHat" && OpSysMajorVersion == 6)
Queue
Here is a more compact way to specify a RedHat 6 platform.
####################
#
# Example of submission targeting RedHat 6 platforms in a heterogeneous Linux pool
#
####################
HTCondor Version 8.6.4 Manual
2.5.12. Jobs That Require GPUs
universe
Executable
Log
Output
Error
=
=
=
=
=
48
vanilla
/bin/date
distro.log
distro.out
distro.err
Requirements = ( OpSysAndVer == "RedHat6")
Queue
2.5.12 Jobs That Require GPUs
A job that needs GPUs to run identifies the number of GPUs needed in the submit description file by adding the submit
command
request_GPUs = <n>
where <n> is replaced by the integer quantity of GPUs required for the job. For example, a job that needs 1 GPU uses
request_GPUs = 1
Because there are different capabilities among GPUs, the job might need to further qualify which GPU of available
ones is required. Do this by specifying or adding a clause to an existing Requirements submit command. As an
example, assume that the job needs a speed and capacity of a CUDA GPU that meets or exceeds the value 1.2. In the
submit description file, place
request_GPUs = 1
requirements = (CUDACapability >= 1.2) && $(requirements:True)
Access to GPU resources by an HTCondor job needs special configuration of the machines that offer GPUs. Details
of how to set up the configuration are in section 3.7.1.
2.5.13 Interactive Jobs
An interactive job is a Condor job that is provisioned and scheduled like any other vanilla universe Condor job onto
an execute machine within the pool. The result of a running interactive job is a shell prompt issued on the execute
machine where the job runs. The user that submitted the interactive job may then use the shell as desired, perhaps
to interactively run an instance of what is to become a Condor job. This might aid in checking that the set up and
execution environment are correct, or it might provide information on the RAM or disk space needed. This job (shell)
continues until the user logs out or any other policy implementation causes the job to stop running. A useful feature
of the interactive job is that the users and jobs are accounted for within Condor’s scheduling and priority system.
HTCondor Version 8.6.4 Manual
2.5.13. Interactive Jobs
49
Neither the submit nor the execute host for interactive jobs may be on Windows platforms.
The current working directory of the shell will be the initial working directory of the running job. The shell type
will be the default for the user that submits the job. At the shell prompt, X11 forwarding is enabled.
Each interactive job will have a job ClassAd attribute of
InteractiveJob = True
Submission of an interactive job specifies the option -interactive on the condor_submit command line.
A submit description file may be specified for this interactive job. Within this submit description file, a specification
of these 5 commands will be either ignored or altered:
1. executable
2. transfer_executable
3. arguments
4. universe. The interactive job is a vanilla universe job.
5. queue <n>. In this case the value of <n> is ignored; exactly one interactive job is queued.
The submit description file may specify anything else needed for the interactive job, such as files to transfer.
If no submit description file is specified for the job, a default one is utilized as identified by the value of the
configuration variable INTERACTIVE_SUBMIT_FILE.
Here are examples of situations where interactive jobs may be of benefit.
• An application that cannot be batch processed might be run as an interactive job. Where input or output cannot
be captured in a file and the executable may not be modified, the interactive nature of the job may still be run on
a pool machine, and within the purview of Condor.
• A pool machine with specialized hardware that requires interactive handling can be scheduled with an interactive
job that utilizes the hardware.
• The debugging and set up of complex jobs or environments may benefit from an interactive session. This
interactive session provides the opportunity to run scripts or applications, and as errors are identified, they can
be corrected on the spot.
• Development may have an interactive nature, and proceed more quickly when done on a pool machine. It may
also be that the development platforms required reside within Condor’s purview as execute hosts.
HTCondor Version 8.6.4 Manual
2.6. Managing a Job
50
2.6 Managing a Job
This section provides a brief summary of what can be done once jobs are submitted. The basic mechanisms for
monitoring a job are introduced, but the commands are discussed briefly. You are encouraged to look at the man pages
of the commands referred to (located in Chapter 11 beginning on page 748) for more information.
When jobs are submitted, HTCondor will attempt to find resources to run the jobs. A list of all those with jobs
submitted may be obtained through condor_status with the -submitters option. An example of this would yield output
similar to:
%
condor_status -submitters
Name
Machine
ballard@cs.wisc.edu
nice-user.condor@cs.
wright@cs.wisc.edu
jbasney@cs.wisc.edu
bluebird.c
cardinal.c
finch.cs.w
perdita.cs
Running IdleJobs HeldJobs
0
6
1
0
11
504
1
0
0
0
0
5
RunningJobs
IdleJobs
HeldJobs
ballard@cs.wisc.edu
jbasney@cs.wisc.edu
nice-user.condor@cs.
wright@cs.wisc.edu
0
0
6
1
11
0
504
1
0
5
0
0
Total
7
516
5
2.6.1 Checking on the progress of jobs
At any time, you can check on the status of your jobs with the condor_q command. This command displays the status
of all queued jobs. An example of the output from condor_q is
%
condor_q
-- Submitter: submit.chtc.wisc.edu : <128.104.55.9:32772> : submit.chtc.wisc.edu
ID
OWNER
SUBMITTED
RUN_TIME ST PRI SIZE CMD
711197.0
aragorn
1/15 19:18
0+04:29:33 H 0
0.0 script.sh
894381.0
frodo
3/16 09:06 82+17:08:51 R 0
439.5 elk elk.in
894386.0
frodo
3/16 09:06 82+20:21:28 R 0
219.7 elk elk.in
894388.0
frodo
3/16 09:06 81+17:22:10 R 0
439.5 elk elk.in
1086870.0
gollum
4/27 09:07
0+00:10:14 I 0
7.3 condor_dagman
1086874.0
gollum
4/27 09:08
0+00:00:01 H 0
0.0 RunDC.bat
1297254.0
legolas
5/31 18:05 14+17:40:01 R 0
7.3 condor_dagman
1297255.0
legolas
5/31 18:05 14+17:39:55 R 0
7.3 condor_dagman
1297256.0
legolas
5/31 18:05 14+17:39:55 R 0
7.3 condor_dagman
1297259.0
legolas
5/31 18:05 14+17:39:55 R 0
7.3 condor_dagman
1297261.0
legolas
5/31 18:05 14+17:39:55 R 0
7.3 condor_dagman
1302278.0
legolas
6/4 12:22
1+00:05:37 I 0
390.6 mdrun_1.sh
1304740.0
legolas
6/5 00:14
1+00:03:43 I 0
390.6 mdrun_1.sh
1304967.0
legolas
6/5 05:08
0+00:00:00 I 0
0.0 mdrun_1.sh
HTCondor Version 8.6.4 Manual
2.6.1. Checking on the progress of jobs
51
14 jobs; 4 idle, 8 running, 2 held
This output contains many columns of information about the queued jobs.
status of current jobs in the queue:
The ST column (for status) shows the
R: The job is currently running.
I: The job is idle. It is not running right now, because it is waiting for a machine to become available.
H: The job is the hold state. In the hold state, the job will not be scheduled to run until it is released. See the
condor_hold manual page located on page 814 and the condor_release manual page located on page 869.
The RUN_TIME time reported for a job is the time that has been committed to the job.
Another useful method of tracking the progress of jobs is through the job event log. The specification of a log in
the submit description file causes the progress of the job to be logged in a file. Follow the events by viewing the job
event log file. Various events such as execution commencement, checkpoint, eviction and termination are logged in
the file. Also logged is the time at which the event occurred.
When a job begins to run, HTCondor starts up a condor_shadow process on the submit machine. The shadow process is the mechanism by which the remotely executing jobs can access the environment from which it was submitted,
such as input and output files.
It is normal for a machine which has submitted hundreds of jobs to have hundreds of condor_shadow processes
running on the machine. Since the text segments of all these processes is the same, the load on the submit machine
is usually not significant. If there is degraded performance, limit the number of jobs that can run simultaneously by
reducing the MAX_JOBS_RUNNING configuration variable.
You can also find all the machines that are running your job through the condor_status command. For example, to
find all the machines that are running jobs submitted by breach@cs.wisc.edu, type:
%
condor_status -constraint 'RemoteUser == "breach@cs.wisc.edu"'
Name
Arch
OpSys
State
Activity
LoadAv Mem
alfred.cs.
biron.cs.w
cambridge.
falcons.cs
happy.cs.w
istat03.st
istat04.st
istat09.st
...
INTEL
INTEL
INTEL
INTEL
INTEL
INTEL
INTEL
INTEL
LINUX
LINUX
LINUX
LINUX
LINUX
LINUX
LINUX
LINUX
Claimed
Claimed
Claimed
Claimed
Claimed
Claimed
Claimed
Claimed
Busy
Busy
Busy
Busy
Busy
Busy
Busy
Busy
0.980
1.000
0.988
0.996
0.988
0.883
0.988
0.301
64
128
64
32
128
64
64
64
ActvtyTime
0+07:10:02
0+01:10:00
0+00:15:00
0+02:05:03
0+03:05:00
0+06:45:01
0+00:10:00
0+03:45:00
To find all the machines that are running any job at all, type:
%
condor_status -run
Name
Arch
OpSys
LoadAv RemoteUser
ClientMachine
HTCondor Version 8.6.4 Manual
2.6.2. Removing a job from the queue
adriana.cs
alfred.cs.
amul.cs.wi
anfrom.cs.
anthrax.cs
astro.cs.w
aura.cs.wi
balder.cs.
bamba.cs.w
bardolph.c
...
INTEL
INTEL
X86_64
X86_64
INTEL
INTEL
X86_64
INTEL
INTEL
INTEL
LINUX
LINUX
LINUX
LINUX
LINUX
LINUX
WINDOWS
WINDOWS
LINUX
LINUX
52
0.980
0.980
1.000
1.023
0.285
1.000
0.996
1.000
1.574
1.000
hepcon@cs.wisc.edu
breach@cs.wisc.edu
nice-user.condor@cs.
ashoks@jules.ncsa.ui
hepcon@cs.wisc.edu
nice-user.condor@cs.
nice-user.condor@cs.
nice-user.condor@cs.
dmarino@cs.wisc.edu
nice-user.condor@cs.
chevre.cs.wisc.
neufchatel.cs.w
chevre.cs.wisc.
jules.ncsa.uiuc
chevre.cs.wisc.
chevre.cs.wisc.
chevre.cs.wisc.
chevre.cs.wisc.
riola.cs.wisc.e
chevre.cs.wisc.
2.6.2 Removing a job from the queue
A job can be removed from the queue at any time by using the condor_rm command. If the job that is being removed
is currently running, the job is killed without a checkpoint, and its queue entry is removed. The following example
shows the queue of jobs before and after a job is removed.
%
condor_q
-- Submitter: froth.cs.wisc.edu : <128.105.73.44:33847>
ID
OWNER
SUBMITTED
CPU_USAGE ST PRI
125.0
jbasney
4/10 15:35
0+00:00:00 I -10
132.0
raman
4/11 16:57
0+00:00:00 R 0
: froth.cs.wisc.edu
SIZE CMD
1.2 hello.remote
1.4 hello
2 jobs; 1 idle, 1 running, 0 held
% condor_rm 132.0
Job 132.0 removed.
%
condor_q
-- Submitter: froth.cs.wisc.edu : <128.105.73.44:33847> : froth.cs.wisc.edu
ID
OWNER
SUBMITTED
CPU_USAGE ST PRI SIZE CMD
125.0
jbasney
4/10 15:35
0+00:00:00 I -10 1.2 hello.remote
1 jobs; 1 idle, 0 running, 0 held
2.6.3 Placing a job on hold
A job in the queue may be placed on hold by running the command condor_hold. A job in the hold state remains in
the hold state until later released for execution by the command condor_release.
Use of the condor_hold command causes a hard kill signal to be sent to a currently running job (one in the running
state). For a standard universe job, this means that no checkpoint is generated before the job stops running and enters
the hold state. When released, this standard universe job continues its execution using the most recent checkpoint
available.
Jobs in universes other than the standard universe that are running when placed on hold will start over from the
beginning when released.
HTCondor Version 8.6.4 Manual
2.6.4. Changing the priority of jobs
53
The manual page for condor_hold on page 814 and the manual page for condor_release on page 869 contain usage
details.
2.6.4 Changing the priority of jobs
In addition to the priorities assigned to each user, HTCondor also provides each user with the capability of assigning
priorities to each submitted job. These job priorities are local to each queue and can be any integer value, with higher
values meaning better priority.
The default priority of a job is 0, but can be changed using the condor_prio command. For example, to change the
priority of a job to -15,
%
condor_q raman
-- Submitter: froth.cs.wisc.edu : <128.105.73.44:33847> : froth.cs.wisc.edu
ID
OWNER
SUBMITTED
CPU_USAGE ST PRI SIZE CMD
126.0
raman
4/11 15:06
0+00:00:00 I 0
0.3 hello
1 jobs; 1 idle, 0 running, 0 held
%
condor_prio -p -15 126.0
%
condor_q raman
-- Submitter: froth.cs.wisc.edu : <128.105.73.44:33847> : froth.cs.wisc.edu
ID
OWNER
SUBMITTED
CPU_USAGE ST PRI SIZE CMD
126.0
raman
4/11 15:06
0+00:00:00 I -15 0.3 hello
1 jobs; 1 idle, 0 running, 0 held
It is important to note that these job priorities are completely different from the user priorities assigned by HTCondor. Job priorities do not impact user priorities. They are only a mechanism for the user to identify the relative
importance of jobs among all the jobs submitted by the user to that specific queue.
2.6.5 Why is the job not running?
Users occasionally find that their jobs do not run. There are many possible reasons why a specific job is not running.
The following prose attempts to identify some of the potential issues behind why a job is not running.
At the most basic level, the user knows the status of a job by using condor_q to see that the job is not running. By
far, the most common reason (to the novice HTCondor job submitter) why the job is not running is that HTCondor
has not yet been through its periodic negotiation cycle, in which queued jobs are assigned to machines within the pool
and begin their execution. This periodic event occurs by default once every 5 minutes, implying that the user ought to
wait a few minutes before searching for reasons why the job is not running.
Further inquiries are dependent on whether the job has never run at all, or has run for at least a little bit.
For jobs that have never run, many problems can be diagnosed by using the -analyze option of the condor_q
command. Here is an example; running condor_q’s analyzer provided the following information:
HTCondor Version 8.6.4 Manual
2.6.5. Why is the job not running?
54
$ condor_q -analyze 27497829
-- Submitter: submit-1.chtc.wisc.edu : <128.104.100.43:9618?sock=5557_e660_3> : submit-1.chtc.wisc.edu
User priority for einstein@submit.chtc.wisc.edu is not available, attempting to analyze without it.
--27497829.000: Run analysis summary. Of 5257 machines,
5257 are rejected by your job's requirements
0 reject your job because of their own requirements
0 match and are already running your jobs
0 match but are serving other users
0 are available to run your job
No successful match recorded.
Last failed match: Tue Jun 18 14:36:25 2013
Reason for last match failure: no match found
WARNING: Be advised:
No resources matched request's constraints
The Requirements expression for your job is:
( OpSys == "OSX" ) && ( TARGET.Arch == "X86_64" ) &&
( TARGET.Disk >= RequestDisk ) && ( TARGET.Memory >= RequestMemory ) &&
( ( TARGET.HasFileTransfer ) || ( TARGET.FileSystemDomain == MY.FileSystemDomain ) )
Suggestions:
Condition
Machines Matched Suggestion
------------------------ ---------1
( target.OpSys == "OSX" )
0
MODIFY TO "LINUX"
2
( TARGET.Arch == "X86_64" )
5190
3
( TARGET.Disk >= 1 )
5257
4
( TARGET.Memory >= ifthenelse(MemoryUsage isnt undefined,MemoryUsage,1) )
5257
5
( ( TARGET.HasFileTransfer ) || ( TARGET.FileSystemDomain == "submit-1.chtc.wisc.edu" ) )
5257
This example also shows that the job does not run because the platform requested, Mac OS X, is not available on
any of the machines in the pool. Recall that unless informed otherwise in the Requirements expression in the submit
description file, the platform requested for an execute machine will be the same as the platform where condor_submit
is run to submit the job. And, while Mac OS X is a Unix-type operating system, it is not the same as Linux, and thus
will not match with machines running Linux.
While the analyzer can diagnose most common problems, there are some situations that it cannot reliably detect
due to the instantaneous and local nature of the information it uses to detect the problem. Thus, it may be that the
analyzer reports that resources are available to service the request, but the job still has not run. In most of these
situations, the delay is transient, and the job will run following the next negotiation cycle.
A second class of problems represents jobs that do or did run, for at least a short while, but are no longer running.
The first issue is identifying whether the job is in this category. The condor_q command is not enough; it only tells
the current state of the job. The needed information will be in the log file or the error file, as defined in the submit
description file for the job. If these files are not defined, then there is little hope of determining if the job ran at all. For
a job that ran, even for the briefest amount of time, the log file will contain an event of type 1, which will contain the
string Job executing on host.
HTCondor Version 8.6.4 Manual
2.6.5. Why is the job not running?
55
A job may run for a short time, before failing due to a file permission problem. The log file used by the condor_shadow daemon will contain more information if this is the problem. This log file is associated with the machine
on which the job was submitted. The location and name of this log file may be discovered on the submitting machine,
using the command
%
condor_config_val SHADOW_LOG
Memory and swap space problems may be identified by looking at the log file used by the condor_schedd daemon.
The location and name of this log file may be discovered on the submitting machine, using the command
%
condor_config_val SCHEDD_LOG
A swap space problem will show in the log with the following message:
2/3 17:46:53 Swap space estimate reached! No more jobs can be run!
12/3 17:46:53
Solution: get more swap space, or set RESERVED_SWAP = 0
12/3 17:46:53
0 jobs matched, 1 jobs idle
As an explanation, HTCondor computes the total swap space on the submit machine. It then tries to limit the total
number of jobs it will spawn based on an estimate of the size of the condor_shadow daemon’s memory footprint and
a configurable amount of swap space that should be reserved. This is done to avoid the situation within a very large
pool in which all the jobs are submitted from a single host. The huge number of condor_shadow processes would
overwhelm the submit machine, and it would run out of swap space and thrash.
Things can go wrong if a machine has a lot of physical memory and little or no swap space. HTCondor does not
consider the physical memory size, so the situation occurs where HTCondor thinks it has no swap space to work with,
and it will not run the submitted jobs.
To see how much swap space HTCondor thinks a given machine has, use the output of a condor_status command
of the following form:
% condor_status -schedd [hostname] -long | grep VirtualMemory
If the value listed is 0, then this is what is confusing HTCondor. There are two ways to fix the problem:
1. Configure the machine with some real swap space.
2. Disable this check within HTCondor. Define the amount of reserved swap space for the submit machine to 0.
Set RESERVED_SWAP to 0 in the configuration file:
RESERVED_SWAP = 0
and then send a condor_restart to the submit machine.
HTCondor Version 8.6.4 Manual
2.6.6. Job in the Hold State
56
2.6.6 Job in the Hold State
A variety of errors and unusual conditions may cause a job to be placed into the Hold state. The job will stay in this
state and in the job queue until conditions are corrected and condor_release is invoked.
A table listing the reasons why a job may be held is at section 12. A string identifying the reason that a particular
job is in the Hold state may be displayed by invoking condor_q. For the example job ID 16.0, use:
condor_q
-hold
16.0
This command prints information about the job, including the job ClassAd attribute HoldReason.
2.6.7 In the Job Event Log File
In a job event log file are a listing of events in chronological order that occurred during the life of one or more jobs.
The formatting of the events is always the same, so that they may be machine readable. Four fields are always present,
and they will most often be followed by other fields that give further information that is specific to the type of event.
The first field in an event is the numeric value assigned as the event type in a 3-digit format. The second field
identifies the job which generated the event. Within parentheses are the job ClassAd attributes of ClusterId value,
ProcId value, and the node number for parallel universe jobs or a set of zeros (for jobs run under all other universes),
separated by periods. The third field is the date and time of the event logging. The fourth field is a string that briefly
describes the event. Fields that follow the fourth field give further information for the specific event type.
These are all of the events that can show up in a job log file:
Event Number: 000
Event Name: Job submitted
Event Description: This event occurs when a user submits a job. It is the first event you will see for a job, and it
should only occur once.
Event Number: 001
Event Name: Job executing
Event Description: This shows up when a job is running. It might occur more than once.
Event Number: 002
Event Name: Error in executable
Event Description: The job could not be run because the executable was bad.
Event Number: 003
Event Name: Job was checkpointed
Event Description: The job’s complete state was written to a checkpoint file. This might happen without the job
being removed from a machine, because the checkpointing can happen periodically.
Event Number: 004
Event Name: Job evicted from machine
HTCondor Version 8.6.4 Manual
2.6.7. In the Job Event Log File
57
Event Description: A job was removed from a machine before it finished, usually for a policy reason. Perhaps an
interactive user has claimed the computer, or perhaps another job is higher priority.
Event Number: 005
Event Name: Job terminated
Event Description: The job has completed.
Event Number: 006
Event Name: Image size of job updated
Event Description: An informational event, to update the amount of memory that the job is using while running. It
does not reflect the state of the job.
Event Number: 007
Event Name: Shadow exception
Event Description: The condor_shadow, a program on the submit computer that watches over the job and performs
some services for the job, failed for some catastrophic reason. The job will leave the machine and go back into the
queue.
Event Number: 008
Event Name: Generic log event
Event Description: Not used.
Event Number: 009
Event Name: Job aborted
Event Description: The user canceled the job.
Event Number: 010
Event Name: Job was suspended
Event Description: The job is still on the computer, but it is no longer executing. This is usually for a policy reason,
such as an interactive user using the computer.
Event Number: 011
Event Name: Job was unsuspended
Event Description: The job has resumed execution, after being suspended earlier.
Event Number: 012
Event Name: Job was held
Event Description: The job has transitioned to the hold state. This might happen if the user applies the condor_hold
command to the job.
Event Number: 013
Event Name: Job was released
Event Description: The job was in the hold state and is to be re-run.
Event Number: 014
Event Name: Parallel node executed
Event Description: A parallel universe program is running on a node.
Event Number: 015
HTCondor Version 8.6.4 Manual
2.6.7. In the Job Event Log File
58
Event Name: Parallel node terminated
Event Description: A parallel universe program has completed on a node.
Event Number: 016
Event Name: POST script terminated
Event Description: A node in a DAGMan work flow has a script that should be run after a job. The script is run on
the submit host. This event signals that the post script has completed.
Event Number: 017
Event Name: Job submitted to Globus
Event Description: A grid job has been delegated to Globus (version 2, 3, or 4). This event is no longer used.
Event Number: 018
Event Name: Globus submit failed
Event Description: The attempt to delegate a job to Globus failed.
Event Number: 019
Event Name: Globus resource up
Event Description: The Globus resource that a job wants to run on was unavailable, but is now available. This event
is no longer used.
Event Number: 020
Event Name: Detected Down Globus Resource
Event Description: The Globus resource that a job wants to run on has become unavailable. This event is no longer
used.
Event Number: 021
Event Name: Remote error
Event Description: The condor_starter (which monitors the job on the execution machine) has failed.
Event Number: 022
Event Name: Remote system call socket lost
Event Description: The condor_shadow and condor_starter (which communicate while the job runs) have lost contact.
Event Number: 023
Event Name: Remote system call socket reestablished
Event Description: The condor_shadow and condor_starter (which communicate while the job runs) have been able
to resume contact before the job lease expired.
Event Number: 024
Event Name: Remote system call reconnect failure
Event Description: The condor_shadow and condor_starter (which communicate while the job runs) were unable to
resume contact before the job lease expired.
Event Number: 025
Event Name: Grid Resource Back Up
Event Description: A grid resource that was previously unavailable is now available.
HTCondor Version 8.6.4 Manual
2.6.8. Job Completion
59
Event Number: 026
Event Name: Detected Down Grid Resource
Event Description: The grid resource that a job is to run on is unavailable.
Event Number: 027
Event Name: Job submitted to grid resource
Event Description: A job has been submitted, and is under the auspices of the grid resource.
Event Number: 028
Event Name: Job ad information event triggered.
Event Description: Extra job ClassAd attributes are noted. This event is written as a supplement to other events when
the configuration parameter EVENT_LOG_JOB_AD_INFORMATION_ATTRS is set.
Event Number: 029
Event Name: The job’s remote status is unknown
Event Description: No updates of the job’s remote status have been received for 15 minutes.
Event Number: 030
Event Name: The job’s remote status is known again
Event Description: An update has been received for a job whose remote status was previous logged as unknown.
Event Number: 031
Event Name: Job stage in
Event Description: A grid universe job is doing the stage in of input files.
Event Number: 032
Event Name: Job stage out
Event Description: A grid universe job is doing the stage out of output files.
Event Number: 033
Event Name: Job ClassAd attribute update
Event Description: A Job ClassAd attribute is changed due to action by the condor_schedd daemon. This includes
changes by condor_prio.
Event Number: 034
Event Name: Pre Skip event
Event Description: For DAGMan, this event is logged if a PRE SCRIPT exits with the defined PRE_SKIP value in
the DAG input file. This makes it possible for DAGMan to do recovery in a workflow that has such an event, as it
would otherwise not have any event for the DAGMan node to which the script belongs, and in recovery, DAGMan’s
internal tables would become corrupted.
2.6.8 Job Completion
When an HTCondor job completes, either through normal means or by abnormal termination by signal, HTCondor
will remove it from the job queue. That is, the job will no longer appear in the output of condor_q, and the job will be
inserted into the job history file. Examine the job history file with the condor_history command. If there is a log file
specified in the submit description file for the job, then the job exit status will be recorded there as well.
HTCondor Version 8.6.4 Manual
2.7. Priorities and Preemption
60
By default, HTCondor does not send an email message when the job completes. Modify this behavior with the
notification command in the submit description file. The message will include the exit status of the job, which is the
argument that the job passed to the exit system call when it completed, or it will be notification that the job was killed
by a signal. Notification will also include the following statistics (as appropriate) about the job:
Submitted at: when the job was submitted with condor_submit
Completed at: when the job completed
Real Time: the elapsed time between when the job was submitted and when it completed, given in a form of
<days> <hours>:<minutes>:<seconds>
Virtual Image Size: memory size of the job, computed when the job checkpoints
Statistics about just the last time the job ran:
Run Time: total time the job was running, given in the form <days> <hours>:<minutes>:<seconds>
Remote User Time: total CPU time the job spent executing in user mode on remote machines; this does
not count time spent on run attempts that were evicted without a checkpoint. Given in the form
<days> <hours>:<minutes>:<seconds>
Remote System Time: total CPU time the job spent executing in system mode (the time spent at system calls);
this does not count time spent on run attempts that were evicted without a checkpoint. Given in the form
<days> <hours>:<minutes>:<seconds>
The Run Time accumulated by all run attempts are summarized with the time given in the form
<days> <hours>:<minutes>:<seconds>.
And, statistics about the bytes sent and received by the last run of the job and summed over all attempts at running
the job are given.
2.7 Priorities and Preemption
HTCondor has two independent priority controls: job priorities and user priorities.
2.7.1 Job Priority
Job priorities allow a user to assign a priority level to each of their own submitted HTCondor jobs, in order to control
the order of job execution. This handles the situation in which a user has more jobs queued, waiting to be executed,
than there are machines available. Setting a job priority identifies the ordering in which that user’s jobs are executed;
a higher job priority job is matched and executed before a lower priority job. A job priority can be any integer, and
larger values are of higher priority. So, 0 is a higher job priority than -3, and 6 is a higher job priority than 5.
HTCondor Version 8.6.4 Manual
2.7.2. User priority
61
For the simple case, each job can be given a distinct priority. For an already queued job, its priority may be set
with the condor_prio command; see the example in section 2.6.4, or the condor_prio manual page 839 for details.
This sets the value of job ClassAd attribute JobPrio.
A fine-grained categorization of jobs and their ordering is available for experts by using the job ClassAd attributes:
PreJobPrio1, PreJobPrio2, JobPrio, PostJobPrio1, or PostJobPrio2.
2.7.2 User priority
Machines are allocated to users based upon a user’s priority. A lower numerical value for user priority means higher
priority, so a user with priority 5 will get more resources than a user with priority 50. User priorities in HTCondor
can be examined with the condor_userprio command (see page 979). HTCondor administrators can set and change
individual user priorities with the same utility.
HTCondor continuously calculates the share of available machines that each user should be allocated. This share
is inversely related to the ratio between user priorities. For example, a user with a priority of 10 will get twice as many
machines as a user with a priority of 20. The priority of each individual user changes according to the number of
resources the individual is using. Each user starts out with the best possible priority: 0.5. If the number of machines a
user currently has is greater than the user priority, the user priority will worsen by numerically increasing over time.
If the number of machines is less then the priority, the priority will improve by numerically decreasing over time. The
long-term result is fair-share access across all users. The speed at which HTCondor adjusts the priorities is controlled
with the configuration variable PRIORITY_HALFLIFE, an exponential half-life value. The default is one day. If
a user that has user priority of 100 and is utilizing 100 machines removes all his/her jobs, one day later that user’s
priority will be 50, and two days later the priority will be 25.
HTCondor enforces that each user gets his/her fair share of machines according to user priority both when allocating machines which become available and by priority preemption of currently allocated machines. For instance,
if a low priority user is utilizing all available machines and suddenly a higher priority user submits jobs, HTCondor
will immediately take a checkpoint and vacate jobs belonging to the lower priority user. This will free up machines
that HTCondor will then give over to the higher priority user. HTCondor will not starve the lower priority user; it will
preempt only enough jobs so that the higher priority user’s fair share can be realized (based upon the ratio between
user priorities). To prevent thrashing of the system due to priority preemption, the HTCondor site administrator can
define a PREEMPTION_REQUIREMENTS expression in HTCondor’s configuration. The default expression that ships
with HTCondor is configured to only preempt lower priority jobs that have run for at least one hour. So in the previous
example, in the worse case it could take up to a maximum of one hour until the higher priority user receives a fair share
of machines. For a general discussion of limiting preemption, please see section ?? of the Administrator’s manual.
User priorities are keyed on <username>@<domain>, for example johndoe@cs.wisc.edu. The domain
name to use, if any, is configured by the HTCondor site administrator. Thus, user priority and therefore resource
allocation is not impacted by which machine the user submits from or even if the user submits jobs from multiple
machines.
An extra feature is the ability to submit a job as a nice job (see page ??). Nice jobs artificially boost the user
priority by ten million just for the nice job. This effectively means that nice jobs will only run on machines that no
other HTCondor job (that is, non-niced job) wants. In a similar fashion, an HTCondor administrator could set the user
priority of any specific HTCondor user very high. If done, for example, with a guest account, the guest could only use
HTCondor Version 8.6.4 Manual
2.7.3. Details About How HTCondor Jobs Vacate Machines
62
cycles not wanted by other users of the system.
2.7.3
Details About How HTCondor Jobs Vacate Machines
When HTCondor needs a job to vacate a machine for whatever reason, it sends the job an asynchronous signal specified
in the KillSig attribute of the job’s ClassAd. The value of this attribute can be specified by the user at submit time
by placing the kill_sig option in the HTCondor submit description file.
If a program wanted to do some special work when required to vacate a machine, the program may set up a signal
handler to use a trappable signal as an indication to clean up. When submitting this job, this clean up signal is specified
to be used with kill_sig. Note that the clean up work needs to be quick. If the job takes too long to go away, HTCondor
follows up with a SIGKILL signal which immediately terminates the process.
A job that is linked using condor_compile and is subsequently submitted into the standard universe, will checkpoint
and exit upon receipt of a SIGTSTP signal. Thus, SIGTSTP is the default value for KillSig when submitting to the
standard universe. The user’s code may still checkpoint itself at any time by calling one of the following functions
exported by the HTCondor libraries:
ckpt()() Performs a checkpoint and then returns.
ckpt_and_exit()() Checkpoints and exits; HTCondor will then restart the process again later, potentially on a
different machine.
For jobs submitted into the vanilla universe, the default value for KillSig is SIGTERM, the usual method to
nicely terminate a Unix program.
2.8 Java Applications
HTCondor allows users to access a wide variety of machines distributed around the world. The Java Virtual Machine
(JVM) provides a uniform platform on any machine, regardless of the machine’s architecture or operating system.
The HTCondor Java universe brings together these two features to create a distributed, homogeneous computing
environment.
Compiled Java programs can be submitted to HTCondor, and HTCondor can execute the programs on any machine
in the pool that will run the Java Virtual Machine.
The condor_status command can be used to see a list of machines in the pool for which HTCondor can use the
Java Virtual Machine.
% condor_status -java
Name
JavaVendor Ver
State
adelie01.cs.wisc.e Sun Micros 1.6.0_ Claimed
adelie02.cs.wisc.e Sun Micros 1.6.0_ Owner
Activity LoadAv
Mem
ActvtyTime
Busy
Idle
873
873
0+00:02:46
0+03:19:32
0.090
0.210
HTCondor Version 8.6.4 Manual
2.8.1. A Simple Example Java Application
63
slot10@bio.cs.wisc Sun Micros 1.6.0_ Unclaimed Idle
slot2@bio.cs.wisc. Sun Micros 1.6.0_ Unclaimed Idle
...
0.000
0.000
118
118
7+03:13:28
7+03:13:28
If there is no output from the condor_status command, then HTCondor does not know the location details of the
Java Virtual Machine on machines in the pool, or no machines have Java correctly installed. In this case, contact your
system administrator or see section 3.15 for more information on getting HTCondor to work together with Java.
2.8.1 A Simple Example Java Application
Here is a complete, if simple, example. Start with a simple Java program, Hello.java:
public class Hello {
public static void main( String [] args ) {
System.out.println("Hello, world!\n");
}
}
Build this program using your Java compiler. On most platforms, this is accomplished with the command
javac Hello.java
Submission to HTCondor requires a submit description file. If submitting where files are accessible using a shared
file system, this simple submit description file works:
####################
#
# Example 1
# Execute a single Java class
#
####################
universe
executable
arguments
output
error
queue
=
=
=
=
=
java
Hello.class
Hello
Hello.output
Hello.error
The Java universe must be explicitly selected.
The main class of the program is given in the executable statement. This is a file name which contains the entry
point of the program. The name of the main class (not a file name) must be specified as the first argument to the
program.
HTCondor Version 8.6.4 Manual
2.8.2. Less Simple Java Specifications
64
If submitting the job where a shared file system is not accessible, the submit description file becomes:
####################
#
# Example 2
# Execute a single Java class,
# not on a shared file system
#
####################
universe
= java
executable
= Hello.class
arguments
= Hello
output
= Hello.output
error
= Hello.error
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
queue
For more information about using HTCondor’s file transfer mechanisms, see section 2.5.9.
To submit the job, where the submit description file is named Hello.cmd, execute
condor_submit Hello.cmd
To monitor the job, the commands condor_q and condor_rm are used as with all jobs.
2.8.2 Less Simple Java Specifications
Specifying more than 1 class file. For programs that consist of more than one .class file, identify the files in the
submit description file:
executable = Stooges.class
transfer_input_files = Larry.class,Curly.class,Moe.class
The executable command does not change. It still identifies the class file that contains the program’s entry
point.
JAR files. If the program consists of a large number of class files, it may be easier to collect them all together into a
single Java Archive (JAR) file. A JAR can be created with:
% jar cvf Library.jar Larry.class Curly.class Moe.class Stooges.class
HTCondor must then be told where to find the JAR as well as to use the JAR. The JAR file that contains the
entry point is specified with the executable command. All JAR files are specified with the jar_files command.
For this example that collected all the class files into a single JAR file, the submit description file contains:
HTCondor Version 8.6.4 Manual
2.8.2. Less Simple Java Specifications
65
executable = Library.jar
jar_files = Library.jar
Note that the JVM must know whether it is receiving JAR files or class files. Therefore, HTCondor must also be
informed, in order to pass the information on to the JVM. That is why there is a difference in submit description
file commands for the two ways of specifying files (transfer_input_files and jar_files).
If there are multiple JAR files, the executable command specifies the JAR file that contains the program’s entry
point. This file is also listed with the jar_files command:
executable = sortmerge.jar
jar_files = sortmerge.jar,statemap.jar
Using a third-party JAR file. As HTCondor requires that all JAR files (third-party or not) be available, specification
of a third-party JAR file is no different than other JAR files. If the sortmerge example above also relies on version
2.1 from http://jakarta.apache.org/commons/lang/, and this JAR file has been placed in the same directory with
the other JAR files, then the submit description file contains
executable = sortmerge.jar
jar_files = sortmerge.jar,statemap.jar,commons-lang-2.1.jar
An executable JAR file. When the JAR file is an executable, specify the program’s entry point in the arguments
command:
executable = anexecutable.jar
jar_files = anexecutable.jar
arguments = some.main.ClassFile
Discovering the main class within a JAR file. As of Java version 1.4, Java virtual machines have a -jar option,
which takes a single JAR file as an argument. With this option, the Java virtual machine discovers the main
class to run from the contents of the Manifest file, which is bundled within the JAR file. HTCondor’s java
universe does not support this discovery, so before submitting the job, the name of the main class must be
identified.
For a Java application which is run on the command line with
java -jar OneJarFile.jar
the equivalent version after discovery might look like
java -classpath OneJarFile.jar TheMainClass
The specified value for TheMainClass can be discovered by unjarring the JAR file, and looking for the
MainClass definition in the Manifest file. Use that definition in the HTCondor submit description file. Partial
contents of that file Java universe submit file will appear as
universe
= java
executable = OneJarFile.jar
jar_files = OneJarFile.jar
Arguments = TheMainClass More-Arguments
queue
HTCondor Version 8.6.4 Manual
2.8.3. Chirp I/O
66
Packages. An example of a Java class that is declared in a non-default package is
package hpc;
public class CondorDriver
{
// class definition here
}
The JVM needs to know the location of this package. It is passed as a command-line argument, implying the
use of the naming convention and directory structure.
Therefore, the submit description file for this example will contain
arguments = hpc.CondorDriver
JVM-version specific features. If the program uses Java features found only in certain JVMs, then the Java application submitted to HTCondor must only run on those machines within the pool that run the needed JVM. Inform
HTCondor by adding a requirements statement to the submit description file. For example, to require
version 3.2, add to the submit description file:
requirements = (JavaVersion=="3.2")
Benchmark speeds. Each machine with Java capability in an HTCondor pool will execute a benchmark to determine its speed. The benchmark is taken when HTCondor is started on the machine, and it uses the SciMark2
(http://math.nist.gov/scimark2) benchmark. The result of the benchmark is held as an attribute within the machine ClassAd. The attribute is called JavaMFlops. Jobs that are run under the Java universe (as all other
HTCondor jobs) may prefer or require a machine of a specific speed by setting rank or requirements in
the submit description file. As an example, to execute only on machines of a minimum speed:
requirements = (JavaMFlops>4.5)
JVM options. Options to the JVM itself are specified in the submit description file:
java_vm_args = -DMyProperty=Value -verbose:gc -Xmx1024m
These options are those which go after the java command, but before the user’s main class. Do not use this to
set the classpath, as HTCondor handles that itself. Setting these options is useful for setting system properties,
system assertions and debugging certain kinds of problems.
2.8.3 Chirp I/O
If a job has more sophisticated I/O requirements that cannot be met by HTCondor’s file transfer mechanism, then the
Chirp facility may provide a solution. Chirp has two advantages over simple, whole-file transfers. First, it permits
the input files to be decided upon at run-time rather than submit time, and second, it permits partial-file I/O with
HTCondor Version 8.6.4 Manual
2.8.3. Chirp I/O
67
results than can be seen as the program executes. However, small changes to the program are required in order to take
advantage of Chirp. Depending on the style of the program, use either Chirp I/O streams or UNIX-like I/O functions.
Chirp I/O streams are the easiest way to get started. Modify the program to use the objects ChirpInputStream
and ChirpOutputStream instead of FileInputStream and FileOutputStream. These classes are completely documented in the HTCondor Software Developer’s Kit (SDK). Here is a simple code example:
import java.io.*;
import edu.wisc.cs.condor.chirp.*;
public class TestChirp {
public static void main( String args[] ) {
try {
BufferedReader in = new BufferedReader(
new InputStreamReader(
new ChirpInputStream("input")));
PrintWriter out = new PrintWriter(
new OutputStreamWriter(
new ChirpOutputStream("output")));
while(true) {
String line = in.readLine();
if(line==null) break;
out.println(line);
}
out.close();
} catch( IOException e ) {
System.out.println(e);
}
}
}
To perform UNIX-like I/O with Chirp, create a ChirpClient object. This object supports familiar operations
such as open, read, write, and close. Exhaustive detail of the methods may be found in the HTCondor SDK,
but here is a brief example:
import java.io.*;
import edu.wisc.cs.condor.chirp.*;
public class TestChirp {
public static void main( String args[] ) {
HTCondor Version 8.6.4 Manual
2.8.3. Chirp I/O
68
try {
ChirpClient client = new ChirpClient();
String message = "Hello, world!\n";
byte [] buffer = message.getBytes();
// Note that we should check that actual==length.
// However, skip it for clarity.
int fd = client.open("output","wct",0777);
int actual = client.write(fd,buffer,0,buffer.length);
client.close(fd);
client.rename("output","output.new");
client.unlink("output.new");
} catch( IOException e ) {
System.out.println(e);
}
}
}
Regardless of which I/O style, the Chirp library must be specified and included with the job. The Chirp JAR
(Chirp.jar) is found in the lib directory of the HTCondor installation. Copy it into your working directory in
order to compile the program after modification to use Chirp I/O.
% condor_config_val LIB
/usr/local/condor/lib
% cp /usr/local/condor/lib/Chirp.jar .
Rebuild the program with the Chirp JAR file in the class path.
% javac -classpath Chirp.jar:. TestChirp.java
The Chirp JAR file must be specified in the submit description file. Here is an example submit description file that
works for both of the given test programs:
universe = java
executable = TestChirp.class
arguments = TestChirp
jar_files = Chirp.jar
+WantIOProxy = True
queue
HTCondor Version 8.6.4 Manual
2.9. Parallel Applications (Including MPI Applications)
2.9 Parallel Applications (Including MPI Applications)
HTCondor’s parallel universe supports jobs that span multiple machines, where the multiple processes within a job
must be running concurrently on these multiple machines, perhaps communicating with each other. The parallel universe provides machine scheduling, but does not enforce a particular programming paradigm for the underlying applications. Thus, parallel universe jobs may run under various MPI implementations as well as under other programming
environments.
The parallel universe supersedes the mpi universe. The mpi universe eventually will be removed from HTCondor.
2.9.1 How Parallel Jobs Run
Parallel universe jobs are submitted from the machine running the dedicated scheduler. The dedicated scheduler
matches and claims a fixed number of machines (slots) for the parallel universe job, and when a sufficient number of
machines are claimed, the parallel job is started on each claimed slot.
Each invocation of condor_submit assigns a single ClusterId for what is considered the single parallel job
submitted. The machine_count submit command identifies how many machines (slots) are to be allocated. Each
instance of the queue submit command acquires and claims the number of slots specified by machine_count. Each
of these slots shares a common job ClassAd and will have the same ProcId job ClassAd attribute value.
Once the correct number of machines are claimed, the executable is started at more or less the same time on
all machines. If desired, a monotonically increasing integer value that starts at 0 may be provided to each of these
machines. The macro $(Node) is similar to the MPI rank construct. This macro may be used within the submit
description file in either the arguments or environment command. Thus, as the executable runs, it may discover its
own $(Node) value.
Node 0 has special meaning and consequences for the parallel job. The completion of a parallel job is implied and
taken to be when the Node 0 executable exits. All other nodes that are part of the parallel job and that have not yet
exited on their own are killed. This default behavior may be altered by placing the line
+ParallelShutdownPolicy = "WAIT_FOR_ALL"
in the submit description file. It causes HTCondor to wait until every node in the parallel job has completed to consider
the job finished.
2.9.2 Parallel Jobs and the Dedicated Scheduler
To run parallel universe jobs, HTCondor must be configured such that machines running parallel jobs are dedicated.
Note that dedicated has a very specific meaning in HTCondor: while dedicated machines can run serial jobs, they
prefer to run parallel jobs, and dedicated machines never preempt a parallel job once it starts running.
A machine becomes a dedicated machine when an administrator configures it to accept parallel jobs from one
specific dedicated scheduler. Note the difference between parallel and serial jobs. While any scheduler in a pool
HTCondor Version 8.6.4 Manual
69
2.9.3. Submission Examples
70
can send serial jobs to any machine, only the designated dedicated scheduler may send parallel universe jobs to a
dedicated machine. Dedicated machines must be specially configured. See section 3.14.8 for a description of the
necessary configuration, as well as examples. Usually, a single dedicated scheduler is configured for a pool which can
run parallel universe jobs, and this condor_schedd daemon becomes the single machine from which parallel universe
jobs are submitted.
The following command line will list the execute machines in the local pool which have been configured to use a
dedicated scheduler, also printing the name of that dedicated scheduler. In order to run parallel jobs, this name will be
defined to be the string "DedicatedScheduler@", prepended to the name of the scheduler host.
condor_status -const '!isUndefined(DedicatedScheduler)' \
-format "%s\t" Machine -format "%s\n" DedicatedScheduler
execute1.example.com DedicatedScheduler@submit.example.com
execute2.example.com DedicatedScheduler@submit.example.com
If this command emits no lines of output, then then pool is not correctly configured to run parallel jobs. Make sure
that the name of the scheduler is correct. The string after the @ sign should match the name of the condor_schedd
daemon, as returned by the command
condor_status -schedd
2.9.3 Submission Examples
Simplest Example
Here is a submit description file for a parallel universe job example that is as simple as possible:
#############################################
## submit description file for a parallel universe job
#############################################
universe = parallel
executable = /bin/sleep
arguments = 30
machine_count = 8
log = log
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
queue
This job specifies the universe as parallel, letting HTCondor know that dedicated resources are required. The
machine_count command identifies that eight machines are required for this job.
HTCondor Version 8.6.4 Manual
2.9.3. Submission Examples
71
Because no requirements are specified, the dedicated scheduler claims eight machines with the same architecture
and operating system as the submit machine. When all the machines are ready, it invokes the /bin/sleep command,
with a command line argument of 30 on each of the eight machines more or less simultaneously. Job events are written
to the log specified in the log command.
The file transfer mechanism is enabled for this parallel job, such that if any of the eight claimed execute machines does not share a file system with the submit machine, HTCondor will correctly transfer the executable. This
/bin/sleep example implies that the submit machine is running a Unix operating system, and the default assumption
for submission from a Unix machine would be that there is a shared file system.
Example with Operating System Requirements
Assume that the pool contains Linux machines installed with either a RedHat or an Ubuntu operating system. If
the job should run only on RedHat platforms, the requirements expression may specify this:
#############################################
## submit description file for a parallel program
## targeting RedHat machines
#############################################
universe = parallel
executable = /bin/sleep
arguments = 30
machine_count = 8
log = log
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
requirements = (OpSysName == "RedHat")
queue
The machine selection may be further narrowed, instead using the OpSysAndVer attribute.
#############################################
## submit description file for a parallel program
## targeting RedHat 6 machines
#############################################
universe = parallel
executable = /bin/sleep
arguments = 30
machine_count = 8
log = log
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
requirements = (OpSysAndVer == "RedHat6")
queue
Using the $(Node) Macro
######################################
## submit description file for a parallel program
HTCondor Version 8.6.4 Manual
2.9.3. Submission Examples
72
## showing the $(Node) macro
######################################
universe = parallel
executable = /bin/cat
log = logfile
input = infile.$(Node)
output = outfile.$(Node)
error = errfile.$(Node)
machine_count = 4
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
queue
The $(Node) macro is expanded to values of 0-3 as the job instances are about to be started. This assigns unique
names to the input and output files to be transferred or accessed from the shared file system. The $(Node) value is
fixed for the entire length of the job.
Differing Requirements for the Machines
Sometimes one machine’s part in a parallel job will have specialized needs. These can be handled with a Requirements submit command that also specifies the number of needed machines.
######################################
## Example submit description file
## with 4 total machines and differing requirements
######################################
universe = parallel
executable = special.exe
machine_count = 1
requirements = ( machine == "machine1@example.com")
queue
machine_count = 3
requirements = ( machine =!= "machine1@example.com")
queue
The dedicated scheduler acquires and claims four machines. All four share the same value of ClusterId,
as this value is associated with this single parallel job. The existence of a second queue command causes a total
of two ProcId values to be assigned for this parallel job. The ProcId values are assigned based on ordering
within the submit description file. Value 0 will be assigned for the single executable that must be executed on machine1@example.com, and the value 1 will be assigned for the other three that must be executed elsewhere.
Requesting multiple cores per slot
If the parallel program has a structure that benefits from running on multiple cores within the same slot, multi-core
slots may be specified.
######################################
HTCondor Version 8.6.4 Manual
2.9.3. Submission Examples
73
## submit description file for a parallel program
## that needs 8-core slots
######################################
universe = parallel
executable = foo.sh
log = logfile
input = infile.$(Node)
output = outfile.$(Node)
error = errfile.$(Node)
machine_count = 2
request_cpus = 8
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
queue
This parallel job causes the scheduler to match and claim two machines, where each of the machines (slots) has
eight cores. The parallel job is assigned a single ClusterId and a single ProcId, meaning that there is a single
job ClassAd for this job.
The executable, foo.sh, is started at the same time on a single core within each of the two machines (slots). It
is presumed that the executable will take care of invoking processes that are to run on the other seven CPUs (cores)
associated with the slot.
Potentially fewer machines are impacted with this specification, as compared with the request that contains
machine_count = 16
request_cpus = 1
The interaction of the eight cores within the single slot may be advantageous with respect to communication delay or
memory access. But, 8-core slots must be available within the pool.
MPI Applications
MPI applications use a single executable, invoked on one or more machines (slots), executing in parallel. The
various implementations of MPI such as Open MPI and MPICH require further framework. HTCondor supports this
necessary framework through a user-modified script. This implementation-dependent script becomes the HTCondor
executable. The script sets up the framework, and then it invokes the MPI application’s executable.
The scripts are located in the $(RELEASE_DIR)/etc/examples directory. The script for the Open MPI
implementation is openmpiscript. The scripts for MPICH implementations are mp1script and mp2script.
An MPICH3 script is not available at this time. These scripts rely on running ssh for communication between the
nodes of the MPI application. The ssh daemon on Unix platforms restricts connections to the approved shells listed in
the /etc/shells file.
Here is a sample submit description file for an MPICH MPI application:
######################################
## Example submit description file
## for MPICH 1 MPI
HTCondor Version 8.6.4 Manual
2.9.3. Submission Examples
74
## works with MPICH 1.2.4, 1.2.5 and 1.2.6
######################################
universe = parallel
executable = mp1script
arguments = my_mpich_linked_executable arg1 arg2
machine_count = 4
should_transfer_files = yes
when_to_transfer_output = on_exit
transfer_input_files = my_mpich_linked_executable
queue
The executable is the mp1script script that will have been modified for this MPI application. This script is
invoked on each slot or core. The script, in turn, is expected to invoke the MPI application’s executable. To know
the MPI application’s executable, it is the first in the list of arguments. And, since HTCondor must transfer this
executable to the machine where it will run, it is listed with the transfer_input_files command, and the file transfer
mechanism is enabled with the should_transfer_files command.
Here is the equivalent sample submit description file, but for an Open MPI application:
######################################
## Example submit description file
## for Open MPI
######################################
universe = parallel
executable = openmpiscript
arguments = my_openmpi_linked_executable arg1 arg2
machine_count = 4
should_transfer_files = yes
when_to_transfer_output = on_exit
transfer_input_files = my_openmpi_linked_executable
queue
Most MPI implementations require two system-wide prerequisites. The first prerequisite is the ability to run a
command on a remote machine without being prompted for a password. ssh is commonly used. The second prerequisite is an ASCII file containing the list of machines that may utilize ssh. These common prerequisites are implemented
in a further script called sshd.sh. sshd.sh generates ssh keys to enable password-less remote execution and starts
an sshd daemon. Use of the sshd.sh script requires the definition of two HTCondor configuration variables. Configuration variable CONDOR_SSHD is an absolute path to an implementation of sshd. sshd.sh has been tested with openssh
version 3.9, but should work with more recent versions. Configuration variable CONDOR_SSH_KEYGEN points to the
corresponding ssh-keygen executable.
mp1script and mp2script require the PATH to the MPICH installation to be set. The variable MPDIR may be
modified in the scripts to indicate its proper value. This directory contains the MPICH mpirun executable.
openmpiscript also requires the PATH to the Open MPI installation. Either the variable MPDIR can
be set manually in the script, or the administrator can define MPDIR using the configuration variable
OPENMPI_INSTALL_PATH. When using Open MPI on a multi-machine HTCondor cluster, the administrator may
also want to consider tweaking the OPENMPI_EXCLUDE_NETWORK_INTERFACES configuration variable as well
as set MOUNT_UNDER_SCRATCH = /tmp.
HTCondor Version 8.6.4 Manual
2.9.4. MPI Applications Within HTCondor’s Vanilla Universe
2.9.4 MPI Applications Within HTCondor’s Vanilla Universe
The vanilla universe may be preferred over the parallel universe for certain parallel applications such as MPI ones.
These applications are ones in which the allocated cores need to be within a single slot. The request_cpus command
causes a claimed slot to have the required number of CPUs (cores).
There are two ways to ensure that the MPI job can run on any machine that it lands on:
1. Statically build an MPI library and statically compile the MPI code.
2. Use CDE to create a directory tree that contains all of the libraries needed to execute the MPI code.
For Linux machines, our experience recommends using CDE, as building static MPI libraries can be difficult. CDE
can be found at http://www.pgbovine.net/cde.html.
Here is a submit description file example assuming that MPI is installed on all machines on which the MPI job
may run, or that the code was built using static libraries and a static version of mpirun is available.
############################################################
##
submit description file for
##
static build of MPI under the vanilla universe
############################################################
universe = vanilla
executable = /path/to/mpirun
request_cpus = 2
arguments = -np 2 my_mpi_linked_executable arg1 arg2 arg3
should_transfer_files = yes
when_to_transfer_output = on_exit
transfer_input_files = my_mpi_linked_executable
queue
If CDE is to be used, then CDE needs to be run first to create the directory tree. On the host machine which has
the original program, the command
prompt-> cde mpirun -n 2 my_mpi_linked_executable
creates a directory tree that will contain all libraries needed for the program. By creating a tarball of this directory,
the user can package up the executable itself, any files needed for the executable, and all necessary libraries. The
following example assumes that the user has created a tarball called cde_my_mpi_linked_executable.tar
which contains the directory tree created by CDE.
############################################################
##
submit description file for
##
MPI under the vanilla universe; CDE used
############################################################
universe = vanilla
executable = cde_script.sh
request_cpus = 2
HTCondor Version 8.6.4 Manual
75
2.10. DAGMan Applications
76
should_transfer_files = yes
when_to_transfer_output = on_exit
transfer_input_files = cde_my_mpi_linked_executable.tar
transfer_output_files = cde-package/cde-root/path/to/original/directory
queue
The executable is now a specialized shell script tailored to this job. In this example, cde_script.sh contains:
#!/bin/sh
# Untar the CDE package
tar xpf cde_my_mpi_linked_executable.tar
# cd to the subdirectory where I need to run
cd cde-package/cde-root/path/to/original/directory
# Run my command
./mpirun.cde -n 2 ./my_mpi_linked_executable
# Since HTCondor will transfer the contents of this directory
# back upon job completion.
# We do not want the .cde command and the executable transferred back.
# To prevent the transfer, remove both files.
rm -f mpirun.cde
rm -f my_mpi_linked_executable
Any additional input files that will be needed for the executable that are not already in the tarball should be included
in the list in transfer_input_files command. The corresponding script should then also be updated to move those files
into the directory where the executable will be run.
2.10 DAGMan Applications
A directed acyclic graph (DAG) can be used to represent a set of computations where the input, output, or execution
of one or more computations is dependent on one or more other computations. The computations are nodes (vertices)
in the graph, and the edges (arcs) identify the dependencies. HTCondor finds machines for the execution of programs,
but it does not schedule programs based on dependencies. The Directed Acyclic Graph Manager (DAGMan) is a
meta-scheduler for the execution of programs (computations). DAGMan submits the programs to HTCondor in an
order represented by a DAG and processes the results. A DAG input file describes the DAG.
DAGMan is itself executed as a scheduler universe job within HTCondor. It submits the HTCondor jobs within
nodes in such a way as to enforce the DAG’s dependencies. DAGMan also handles recovery and reporting on the
HTCondor jobs.
2.10.1 DAGMan Terminology
A node within a DAG may encompass more than a single program submitted to run under HTCondor. Figure 2.1
illustrates the elements of a node.
More than one HTCondor job may belong to a single node. All HTCondor jobs within a node must be within a
single cluster, as given by the job ClassAd attribute ClusterId.
HTCondor Version 8.6.4 Manual
2.10.2. The DAG Input File: Basic Commands
77
[ optional ]
PRE script
HTCondor job(s)
(with a single
cluster number)
or Stork job
[ optional ]
POST script
Figure 2.1: One Node within a DAG
DAGMan enforces the dependencies within a DAG using the events recorded in a separate file that is specified by
the default configuration. If the exact same DAG were to be submitted more than once, such that these DAGs were
running at the same time, expected them to fail in unpredictable and unexpected ways. They would all be using the
same single file to enforce dependencies.
As DAGMan schedules and submits jobs within nodes to HTCondor, these jobs are defined to succeed or fail based
on their return values. This success or failure is propagated in well-defined ways to the level of a node within a DAG.
Further progression of computation (towards completing the DAG) is based upon the success or failure of nodes.
The failure of a single job within a cluster of multiple jobs (within a single node) causes the entire cluster of jobs
to fail. Any other jobs within the failed cluster of jobs are immediately removed. Each node within a DAG may be
further constrained to succeed or fail based upon the return values of a PRE script and/or a POST script.
2.10.2 The DAG Input File: Basic Commands
The input file used by DAGMan is called a DAG input file. It specifies the nodes of the DAG as well as the dependencies that order the DAG. All items are optional, except that there must be at least one JOB item.
Comments may be placed in the DAG input file. The pound character (#) as the first character on a line identifies
the line as a comment. Comments do not span lines.
A simple diamond-shaped DAG, as shown in Figure 2.2 is presented as a starting point for examples. This DAG
contains 4 nodes.
HTCondor Version 8.6.4 Manual
2.10.2. The DAG Input File: Basic Commands
78
A
B
C
D
Figure 2.2: Diamond DAG
A very simple DAG input file for this diamond-shaped DAG is
# File name: diamond.dag
#
JOB A A.condor
JOB B B.condor
JOB C C.condor
JOB D D.condor
PARENT A CHILD B C
PARENT B C CHILD D
A set of basic commands appearing in a DAG input file is described below.
JOB
The JOB command specifies an HTCondor job. The syntax used for each JOB command is
JOB JobName SubmitDescriptionFileName [DIR directory] [NOOP] [DONE]
A JOB entry maps a JobName to an HTCondor submit description file. The JobName uniquely identifies nodes
within the DAG input file and in output messages. Each node name, given by JobName, within the DAG must be
unique. The JOB entry must appear within the DAG input file before other items that reference the node.
The keywords JOB, DIR, NOOP, and DONE are not case sensitive. Therefore, DONE, Done, and done are all
equivalent. The values defined for JobName and SubmitDescriptionFileName are case sensitive, as file names in a file
system are case sensitive. The JobName can be any string that contains no white space, except for the strings PARENT
and CHILD (in upper, lower, or mixed case).
Note that DIR, NOOP, and DONE, if used, must appear in the order shown above.
The optional DIR keyword specifies a working directory for this node, from which the HTCondor job will be
submitted, and from which a PRE and/or POST script will be run. If a relative directory is specified, it is relative to the
current working directory as the DAG is submitted. Note that a DAG containing DIR specifications cannot be run in
conjunction with the -usedagdir command-line argument to condor_submit_dag. A "full" rescue DAG generated by
a DAG run with the -usedagdir argument will contain DIR specifications, so such a rescue DAG must be run without
the -usedagdir argument. (Note that "full" rescue DAGs are no longer the default.)
The optional NOOP keyword identifies that the HTCondor job within the node is not to be submitted to HTCondor.
This optimization is useful in cases such as debugging a complex DAG structure, where some of the individual jobs
HTCondor Version 8.6.4 Manual
2.10.2. The DAG Input File: Basic Commands
are long-running. For this debugging of structure, some jobs are marked as NOOPs, and the DAG is initially run to
verify that the control flow through the DAG is correct. The NOOP keywords are then removed before submitting the
DAG. Any PRE and POST scripts for jobs specified with NOOP are executed; to avoid running the PRE and POST
scripts, comment them out. The job that is not submitted to HTCondor is given a return value that indicates success,
such that the node may also succeed. Return values of any PRE and POST scripts may still cause the node to fail.
Even though the job specified with NOOP is not submitted, its submit description file must exist; the log file for the
job is used, because DAGMan generates dummy submission and termination events for the job.
The optional DONE keyword identifies a node as being already completed. This is mainly used by Rescue DAGs
generated by DAGMan itself, in the event of a failure to complete the workflow. Nodes with the DONE keyword are
not executed when the Rescue DAG is run, allowing the workflow to pick up from the previous endpoint. Users should
generally not use the DONE keyword. The NOOP keyword is more flexible in avoiding the execution of a job within
a node. Note that, for any node marked DONE in a DAG, all of its parents must also be marked DONE; otherwise, a
fatal error will result. The DONE keyword applies to the entire node. A node marked with DONE will not have a PRE
or POST script run, and the HTCondor job will not be submitted.
DATA
As of version 8.3.5, condor_dagman no longer supports DATA nodes.
PARENT . . .CHILD
The PARENT CHILD command specifies the dependencies within the DAG. Nodes are parents and/or children within
the DAG. A parent node must be completed successfully before any of its children may be started. A child node may
only be started once all its parents have successfully completed.
The syntax used for each dependency (PARENT/CHILD) command is
PARENT ParentJobName. . . CHILD ChildJobName. . .
The PARENT keyword is followed by one or more ParentJobNames. The CHILD keyword is followed by one or
more ChildJobNames. Each child job depends on every parent job within the line. A single line in the input file can
specify the dependencies from one or more parents to one or more children. The diamond-shaped DAG example may
specify the dependencies with
PARENT A CHILD B C
PARENT B C CHILD D
An alternative specification for the diamond-shaped DAG may specify some or all of the dependencies on separate
lines:
PARENT A CHILD B C
PARENT B CHILD D
PARENT C CHILD D
HTCondor Version 8.6.4 Manual
79
2.10.2. The DAG Input File: Basic Commands
As a further example, the line
PARENT p1 p2 CHILD c1 c2
produces four dependencies:
1. p1 to c1
2. p1 to c2
3. p2 to c1
4. p2 to c2
SCRIPT
The optional SCRIPT command specifies processing that is done either before a job within a node is submitted or
after a job within a node completes its execution. Processing done before a job is submitted is called a PRE script.
Processing done after a job completes its execution is called a POST script. Note that the executable specified does
not necessarily have to be a shell script (Unix) or batch file (Windows); but it should be relatively light weight because
it will be run directly on the submit machine, not submitted as an HTCondor job.
The syntax used for each PRE or POST command is
SCRIPT [DEFER status time] PRE JobName|ALL_NODES ExecutableName [arguments]
SCRIPT [DEFER status time] POST JobName|ALL_NODES ExecutableName [arguments]
The SCRIPT command uses the PRE or POST keyword, which specifies the relative timing of when the script is to
be run. The JobName identifies the node to which the script is attached. The ExecutableName specifies the executable
(e.g., shell script or batch file) to be executed, and may not contain spaces. The optional arguments are command line
arguments to the script, and spaces delimit the arguments. Both ExecutableName and optional arguments are case
sensitive.
Scripts are executed on the submit machine; the submit machine is not necessarily the same machine upon which
the node’s job is run. Further, a single cluster of HTCondor jobs may be spread across several machines.
The optional DEFER feature causes a retry of only the script, if the execution of the script exits with the exit code
given by status. The retry occurs after at least time seconds, rather than being considered failed. While waiting for
the retry, the script does not count against a maxpre or maxpost limit. The ordering of the DEFER feature within
the SCRIPT specification is fixed. It must come directly after the SCRIPT keyword; this is done to avoid backward
compatibility issues for any DAG with a JobName of DEFER.
A PRE script is commonly used to place files in a staging area for the jobs to use. A POST script is commonly
used to clean up or remove files once jobs are finished running. An example uses PRE and POST scripts to stage files
that are stored on tape. The PRE script reads compressed input files from the tape drive, uncompresses them, and
places the resulting files in the current directory. The HTCondor jobs can then use these files, producing output files.
HTCondor Version 8.6.4 Manual
80
2.10.2. The DAG Input File: Basic Commands
The POST script compresses the output files, writes them out to the tape, and then removes both the staged files and
the output files.
If the PRE script fails, then the HTCondor job associated with the node is not submitted, and (as of version
8.5.4) the POST script is not run either (by default). However, if the job is submitted, and there is a POST script,
the POST script is always run once the job finishes. (The behavior when the PRE script fails may may be changed
to run the POST script by setting configuration variable DAGMAN_ALWAYS_RUN_POST to True or by passing the
-AlwaysRunPost argument to condor_submit_dag.)
Progress towards completion of the DAG is based upon the success of the nodes within the DAG. The success
of a node is based upon the success of the job(s), PRE script, and POST script. A job, PRE script, or POST script
with an exit value not equal to 0 is considered failed. The exit value of whatever component of the node was run
last determines the success or failure of the node. Table 2.1 lists the definition of node success and failure for all
variations of script and job success and failure, when DAGMAN_ALWAYS_RUN_POST is set to False. In this table,
a dash (-) represents the case where a script does not exist for the DAG, S represents success, and F represents failure.
Table 2.2 lists the definition of node success and failure only for the cases where the PRE script fails, when
DAGMAN_ALWAYS_RUN_POST is set to True.
PRE
S
S
S
S
S
S
F
F
JOB
S
F
S
S
F
F
S
F
S
S
F
F
not run
not run
POST
S
F
S
F
S
F
S
F
not run
Node
S
F
S
F
S
F
S
F
S
F
S
F
F
F
Table 2.1: Node success or failure definition with DAGMAN_ALWAYS_RUN_POST = False (the default)
PRE
F
F
F
JOB
not run
not run
not run
POST
S
F
Node
F
S
F
Table 2.2: Node Success or Failure definition with DAGMAN_ALWAYS_RUN_POST = True
HTCondor Version 8.6.4 Manual
81
2.10.2. The DAG Input File: Basic Commands
Special script argument macros
The five macros $JOB, $RETRY, $MAX_RETRIES, $DAG_STATUS and $FAILED_COUNT can be used within
the DAG input file as arguments passed to a PRE or POST script. The three macros $JOBID, $RETURN, and
$PRE_SCRIPT_RETURN can be used as arguments to POST scripts. The use of these variables is limited to being used as an individual command line argument to the script, surrounded by spaces, in order to cause the substitution
of the variable’s value.
The special macros are as follows:
• $JOB evaluates to the (case sensitive) string defined for JobName.
• $RETRY evaluates to an integer value set to 0 the first time a node is run, and is incremented each time the node
is retried. See section 2.10.9 for the description of how to cause nodes to be retried.
• $MAX_RETRIES evaluates to an integer value set to the maximum number of retries for the node. See
section 2.10.9 for the description of how to cause nodes to be retried. If no retries are set for the node,
$MAX_RETRIES will be set to 0.
• $JOBID (for POST scripts only) evaluates to a representation of the HTCondor job ID of the node job. It is the
value of the job ClassAd attribute ClusterId, followed by a period, and then followed by the value of the job
ClassAd attribute ProcId. An example of a job ID might be 1234.0. For nodes with multiple jobs in the same
cluster, the ProcId value is the one of the last job within the cluster.
• $RETURN (for POST scripts only) variable evaluates to the return value of the HTCondor job, if there is a single
job within a cluster. With multiple jobs within the same cluster, there are two cases to consider. In the first case,
all jobs within the cluster are successful; the value of $RETURN will be 0, indicating success. In the second
case, one or more jobs from the cluster fail. When condor_dagman sees the first terminated event for a job
that failed, it assigns that job’s return value as the value of $RETURN, and it attempts to remove all remaining
jobs within the cluster. Therefore, if multiple jobs in the cluster fail with different exit codes, a race condition
determines which exit code gets assigned to $RETURN.
A job that dies due to a signal is reported with a $RETURN value representing the additive inverse of the signal
number. For example, SIGKILL (signal 9) is reported as -9. A job whose batch system submission fails is
reported as -1001. A job that is externally removed from the batch system queue (by something other than
condor_dagman) is reported as -1002.
• $PRE_SCRIPT_RETURN (for POST scripts only) variable evaluates to the return value of the PRE script of
a node, if there is one. If there is no PRE script, this value will be -1. If the node job was skipped because of
failure of the PRE script, the value of $RETURN will be -1004 and the value of $PRE_SCRIPT_RETURN will
be the exit value of the PRE script; the POST script can use this to see if the PRE script exited with an error
condition, and assign success or failure to the node, as appropriate.
• $DAG_STATUS is the status of the DAG. Note that this macro’s value and definition is unrelated to the attribute
named DagStatus as defined for use in a node status file. This macro’s value is the same as the job ClassAd
attribute DAG_Status that is defined within the condor_dagman job’s ClassAd. This macro may have the
following values:
– 0: OK
HTCondor Version 8.6.4 Manual
82
2.10.2. The DAG Input File: Basic Commands
– 1: error; an error condition different than those listed here
– 2: one or more nodes in the DAG have failed
– 3: the DAG has been aborted by an ABORT-DAG-ON specification
– 4: removed; the DAG has been removed by condor_rm
– 5: cycle; a cycle was found in the DAG
– 6: halted; the DAG has been halted (see section 2.10.8)
• $FAILED_COUNT is defined by the number of nodes that have failed in the DAG.
Examples that use PRE or POST scripts
Examples use the diamond-shaped DAG. A first example uses a PRE script to expand a compressed file needed as
input to each of the HTCondor jobs of nodes B and C. The DAG input file:
# File
#
JOB A
JOB B
JOB C
JOB D
SCRIPT
SCRIPT
PARENT
PARENT
name: diamond.dag
A.condor
B.condor
C.condor
D.condor
PRE B pre.csh $JOB .gz
PRE C pre.csh $JOB .gz
A CHILD B C
B C CHILD D
The script pre.csh uses its command line arguments to form the file name of the compressed file. The script
contains
#!/bin/csh
gunzip $argv[1]$argv[2]
Therefore, the PRE script invokes
gunzip B.gz
for node B, which uncompresses file B.gz, placing the result in file B.
A second example uses the $RETURN macro. The DAG input file contains the POST script specification:
SCRIPT POST A stage-out job_status $RETURN
If the HTCondor job of node A exits with the value -1, the POST script is invoked as
stage-out job_status -1
HTCondor Version 8.6.4 Manual
83
2.10.3. Command Order
84
The slightly different example POST script specification in the DAG input file
SCRIPT POST A stage-out job_status=$RETURN
invokes the POST script with
stage-out job_status=$RETURN
This example shows that when there is no space between the = sign and the variable $RETURN, there is no
substitution of the macro’s value.
PRE_SKIP
The behavior of DAGMan with respect to node success or failure can be changed with the addition of a PRE_SKIP
command. A PRE_SKIP line within the DAG input file uses the syntax:
PRE_SKIP JobName|ALL_NODES non-zero-exit-code
The PRE script of a node identified by JobName that exits with the value given by non-zero-exit-code skips the
remainder of the node entirely. Neither the job associated with the node nor the POST script will be executed, and the
node will be marked as successful.
2.10.3 Command Order
As of version 8.5.6, commands referencing a JobName can come before the JOB command defining that JobName.
For example, the command sequence
SCRIPT PRE NodeA foo.pl
VARS NodeA state="Wisconsin"
JOB NodeA bar.sub
is now legal (it would have been illegal in 8.5.5 and all previous versions).
2.10.4 Node Job Submit File Contents
Each node in a DAG may use a unique submit description file. A key limitation is that each HTCondor submit
description file must submit jobs described by a single cluster number; DAGMan cannot deal with a submit description
file producing multiple job clusters.
Consider again the diamond-shaped DAG example, where each node job uses the same submit description file.
HTCondor Version 8.6.4 Manual
2.10.5. DAG Submission
85
# File name: diamond.dag
#
JOB A diamond_job.condor
JOB B diamond_job.condor
JOB C diamond_job.condor
JOB D diamond_job.condor
PARENT A CHILD B C
PARENT B C CHILD D
Here is a sample HTCondor submit description file for this DAG:
# File name:
#
executable
output
error
log
universe
queue
diamond_job.condor
=
=
=
=
=
/path/diamond.exe
diamond.out.$(cluster)
diamond.err.$(cluster)
diamond_condor.log
vanilla
Since each node uses the same HTCondor submit description file, this implies that each node within the DAG runs
the same job. The $(Cluster) macro produces unique file names for each job’s output.
The job ClassAd attribute DAGParentNodeNames is also available for use within the submit description file.
It defines a comma separated list of each JobName which is a parent node of this job’s node. This attribute may be
used in the arguments command for all but scheduler universe jobs. For example, if the job has two parents, with
JobNames B and C, the submit description file command
arguments = $$([DAGParentNodeNames])
will pass the string "B,C" as the command line argument when invoking the job.
2.10.5 DAG Submission
A DAG is submitted using the tool condor_submit_dag. The manual page 953 details the command. The simplest of
DAG submissions has the syntax
condor_submit_dag DAGInputFileName
and the current working directory contains the DAG input file.
The diamond-shaped DAG example may be submitted with
condor_submit_dag diamond.dag
HTCondor Version 8.6.4 Manual
2.10.5. DAG Submission
86
Do not submit the same DAG, with same DAG input file, from within the same directory, such that more than one
of this same DAG is running at the same time. It will fail in an unpredictable manner, as each instance of this same
DAG will attempt to use the same file to enforce dependencies.
To increase robustness and guarantee recoverability, the condor_dagman process is run as an HTCondor job. As
such, it needs a submit description file. condor_submit_dag generates this needed submit description file, naming it
by appending .condor.sub to the name of the DAG input file. This submit description file may be edited if the
DAG is submitted with
condor_submit_dag -no_submit diamond.dag
causing condor_submit_dag to create the submit description file, but not submit condor_dagman to HTCondor. To
submit the DAG, once the submit description file is edited, use
condor_submit diamond.dag.condor.sub
Submit machines with limited resources are supported by command line options that place limits on the submission
and handling of HTCondor jobs and PRE and POST scripts. Presented here are descriptions of the command line
options to condor_submit_dag. These same limits can be set in configuration. Each limit is applied within a single
DAG.
DAG Throttling
Total nodes/clusters: The -maxjobs option specifies the maximum number of clusters that condor_dagman can
submit at one time. Since each node corresponds to a single cluster, this limit restricts the number of nodes that can be
submitted (in the HTCondor queue) at a time. It is commonly used when there is a limited amount of input file staging
capacity. As a specific example, consider a case where each node represents a single HTCondor proc that requires 4
MB of input files, and the proc will run in a directory with a volume of 100 MB of free space. Using the argument
-maxjobs 25 guarantees that a maximum of 25 clusters, using a maximum of 100 MB of space, will be submitted to
HTCondor at one time. (See the condor_submit_dag man page ( 11) for more information. Also see the equivalent
DAGMAN_MAX_JOBS_SUBMITTED configuration option ( 3.5.24).)
Idle procs: The number of idle procs within a given DAG can be limited with the optional command line argument
-maxidle. condor_dagman will not submit any more node jobs until the number of idle procs in the DAG goes
below this specified value, even if there are ready nodes in the DAG. This allows condor_dagman to submit jobs in
a way that adapts to the load on the HTCondor pool at any given time. If the pool is lightly loaded, condor_dagman
will end up submitting more jobs; if the pool is heavily loaded, condor_dagman will submit fewer jobs. (See the
condor_submit_dag man page ( 11) for more information. Also see the equivalent DAGMAN_MAX_JOBS_IDLE
configuration option ( 3.5.24).)
Note that the -maxjobs option applies to counts of clusters, whereas the -maxidle option applies to counts of
procs. Unfortunately, this can be a bit confusing. Of course, if none of your submit files create more than one proc,
the distinction doesn’t matter. For example, though, a node job submit file that queues 5 procs will count as one for
-maxjobs, but five for -maxidle (if all of the procs are idle).
HTCondor Version 8.6.4 Manual
2.10.6. File Paths in DAGs
87
Subsets of nodes: Node submission can also be throttled in a finer-grained manner by grouping nodes into categories. See section 2.10.9 for more details.
PRE/POST scripts: Since PRE and POST scripts run on the submit machine, it may be desirable to limit the
number of PRE or POST scripts running at one time. The optional -maxpre command line argument limits the
number of PRE scripts that may be running at one time, and the optional -maxpost command line argument limits
the number of POST scripts that may be running at one time. (See the condor_submit_dag man page ( 11) for more
information. Also see the equivalent DAGMAN_MAX_PRE_SCRIPTS ( 3.5.24) and DAGMAN_MAX_POST_SCRIPTS
( 3.5.24) configuration options.)
2.10.6 File Paths in DAGs
condor_dagman assumes that all relative paths in a DAG input file and the associated HTCondor submit description
files are relative to the current working directory when condor_submit_dag is run. This works well for submitting a
single DAG. It presents problems when multiple independent DAGs are submitted with a single invocation of condor_submit_dag. Each of these independent DAGs would logically be in its own directory, such that it could be run
or tested independent of other DAGs. Thus, all references to files will be designed to be relative to the DAG’s own
directory.
Consider an example DAG within a directory named dag1. There would be a DAG input file, named one.dag
for this example. Assume the contents of this DAG input file specify a node job with
JOB A
A.submit
Further assume that partial contents of submit description file A.submit specify
executable = programA
input
= A.input
Directory contents are
dag1 (directory)
one.dag
A.submit
programA
A.input
All file paths are correct relative to the dag1 directory. Submission of this example DAG sets the current working
directory to dag1 and invokes condor_submit_dag:
cd dag1
condor_submit_dag one.dag
HTCondor Version 8.6.4 Manual
2.10.7. DAG Monitoring and DAG Removal
88
Expand this example such that there are now two independent DAGs, and each is contained within its own directory. For simplicity, assume that the DAG in dag2 has remarkably similar files and file naming as the DAG in dag1.
Assume that the directory contents are
parent (directory)
dag1 (directory)
one.dag
A.submit
programA
A.input
dag2 (directory)
two.dag
B.submit
programB
B.input
The goal is to use a single invocation of condor_submit_dag to run both dag1 and dag2. The invocation
cd parent
condor_submit_dag dag1/one.dag dag2/two.dag
does not work. Path names are now relative to parent, which is not the desired behavior.
The solution is the -usedagdir command line argument to condor_submit_dag. This feature runs each DAG as if
condor_submit_dag had been run in the directory in which the relevant DAG file exists. A working invocation is
cd parent
condor_submit_dag -usedagdir dag1/one.dag dag2/two.dag
Output files will be placed in the correct directory, and the .dagman.out file will also be in the correct directory.
A Rescue DAG file will be written to the current working directory, which is the directory when condor_submit_dag
is invoked. The Rescue DAG should be run from that same current working directory. The Rescue DAG includes all
the path information necessary to run each node job in the proper directory.
Use of -usedagdir does not work in conjunction with a JOB node specification within the DAG input file using the
DIR keyword. Using both will be detected and generate an error.
2.10.7 DAG Monitoring and DAG Removal
After submission, the progress of the DAG can be monitored by looking at the job event log file(s), observing the
e-mail that job submission to HTCondor causes, or by using condor_q -dag.
There is also a large amount of information logged in an extra file. The name of this extra file is produced by
appending .dagman.out to the name of the DAG input file; for example, if the DAG input file is diamond.dag,
HTCondor Version 8.6.4 Manual
2.10.8. Suspending a Running DAG
89
this extra file is named diamond.dag.dagman.out. If this extra file grows too large, limit its size with the configuration variable MAX_DAGMAN_LOG, as defined in section 3.5.3. The dagman.out file is an important resource
for debugging; save this file if a problem occurs. The dagman.out is appended to, rather than overwritten, with each
new DAGMan run.
To remove an entire DAG, consisting of the condor_dagman job, plus any jobs submitted to HTCondor, remove
the condor_dagman job by running condor_rm. For example,
% condor_q
-- Submitter: turunmaa.cs.wisc.edu : <128.105.175.125:36165> : turunmaa.cs.wisc.edu
ID
OWNER
SUBMITTED
RUN_TIME ST PRI SIZE CMD
9.0
taylor
10/12 11:47
0+00:01:32 R 0
8.7 condor_dagman -f 11.0
taylor
10/12 11:48
0+00:00:00 I 0
3.6 B.out
12.0
taylor
10/12 11:48
0+00:00:00 I 0
3.6 C.out
3 jobs; 2 idle, 1 running, 0 held
% condor_rm 9.0
When a condor_dagman job is removed, all node jobs (including sub-DAGs) of that condor_dagman will be
removed by the condor_schedd. As of version 8.5.8, the default is that condor_dagman itself also removes the node
jobs (to fix a race condition that could result in "orphaned" node jobs). (The condor_schedd has to remove the node
jobs to deal with the case of removing a condor_dagman job that has been held.)
The previous behavior of condor_dagman itself not removing the node jobs can be restored by setting the
DAGMAN_REMOVE_NODE_JOBS configuration macro (see 3.5.24) to False. This will decrease the load on the
condor_schedd, at the cost of allowing the possibility of "orphaned" node jobs.
A removed DAG will be considered failed unless the DAG has a FINAL node that succeeds.
In the case where a machine is scheduled to go down, DAGMan will clean up memory and exit. However, it will
leave any submitted jobs in the HTCondor queue.
2.10.8 Suspending a Running DAG
It may be desired to temporarily suspend a running DAG. For example, the load may be high on the submit machine,
and therefore it is desired to prevent DAGMan from submitting any more jobs until the load goes down. There are two
ways to suspend (and resume) a running DAG.
• Use condor_hold/condor_release on the condor_dagman job.
After placing the condor_dagman job on hold, no new node jobs will be submitted, and no PRE or POST
scripts will be run. Any node jobs already in the HTCondor queue will continue undisturbed. Any running
PRE or POST scripts will be killed. If the condor_dagman job is left on hold, it will remain in the HTCondor
queue after all of the currently running node jobs are finished. To resume the DAG, use condor_release on the
condor_dagman job.
Note that while the condor_dagman job is on hold, no updates will be made to the dagman.out file.
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
90
• Use a DAG halt file.
The second way of suspending a DAG uses the existence of a specially-named file to change the state of the
DAG. When in this halted state, no PRE scripts will be run, and no node jobs will be submitted. Running
node jobs will continue undisturbed. A halted DAG will still run POST scripts, and it will still update the
dagman.out file. This differs from behavior of a DAG that is held. Furthermore, a halted DAG will not
remain in the queue indefinitely; when all of the running node jobs have finished, DAGMan will create a Rescue
DAG and exit.
To resume a halted DAG, remove the halt file.
The specially-named file must be placed in the same directory as the DAG input file. The naming is the same
as the DAG input file concatenated with the string .halt. For example, if the DAG input file is test1.dag,
then test1.dag.halt will be the required name of the halt file.
As any DAG is first submitted with condor_submit_dag, a check is made for a halt file. If one exists, it is
removed.
Note that neither condor_hold nor a DAG halt is propagated to sub-DAGs. In other words, if you condor_hold
or create a halt file for a DAG that has sub-DAGs, any sub-DAGs that are already in the queue will continue to submit
node jobs.
A condor_hold or DAG halt does, however, apply to splices, because they are merged into the parent DAG and
controlled by a single condor_dagman instance.
2.10.9 Advanced Features of DAGMan
Retrying Failed Nodes
DAGMan can retry any failed node in a DAG by specifying the node in the DAG input file with the RETRY command.
The use of retry is optional. The syntax for retry is
RETRY JobName|ALL_NODES NumberOfRetries [UNLESS-EXIT value]
where JobName identifies the node. NumberOfRetries is an integer number of times to retry the node after failure.
The implied number of retries for any node is 0, the same as not having a retry line in the file. Retry is implemented
on nodes, not parts of a node.
The diamond-shaped DAG example may be modified to retry node C:
# File
#
JOB A
JOB B
JOB C
JOB D
PARENT
PARENT
Retry
name: diamond.dag
A.condor
B.condor
C.condor
D.condor
A CHILD B C
B C CHILD D
C 3
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
91
If node C is marked as failed for any reason, then it is started over as a first retry. The node will be tried a second
and third time, if it continues to fail. If the node is marked as successful, then further retries do not occur.
Retry of a node may be short circuited using the optional keyword UNLESS-EXIT, followed by an integer exit
value. If the node exits with the specified integer exit value, then no further processing will be done on the node.
The macro $RETRY evaluates to an integer value, set to 0 first time a node is run, and is incremented each time for
each time the node is retried. The macro $MAX_RETRIES is the value set for NumberOfRetries. These macros may
be used as arguments passed to a PRE or POST script.
Stopping the Entire DAG
The ABORT-DAG-ON command provides a way to abort the entire DAG if a given node returns a specific exit code.
The syntax for ABORT-DAG-ON is
ABORT-DAG-ON JobName|ALL_NODES AbortExitValue [RETURN DAGReturnValue]
If the return value of the node specified by JobName matches AbortExitValue, the DAG is immediately aborted. A
DAG abort differs from a node failure, in that a DAG abort causes all nodes within the DAG to be stopped immediately.
This includes removing the jobs in nodes that are currently running. A node failure differs, as it would allow the DAG
to continue running, until no more progress can be made due to dependencies.
The behavior differs based on the existence of PRE and/or POST scripts. If a PRE script returns the AbortExitValue
value, the DAG is immediately aborted. If the HTCondor job within a node returns the AbortExitValue value, the DAG
is aborted if the node has no POST script. If the POST script returns the AbortExitValue value, the DAG is aborted.
An abort overrides node retries. If a node returns the abort exit value, the DAG is aborted, even if the node has
retry specified.
When a DAG aborts, by default it exits with the node return value that caused the abort. This can be changed by
using the optional RETURN keyword along with specifying the desired DAGReturnValue. The DAG abort return value
can be used for DAGs within DAGs, allowing an inner DAG to cause an abort of an outer DAG.
A DAG return value other than 0, 1, or 2 will cause the condor_dagman job to stay in the queue after it exits and
get retried, unless the on_exit_remove expression in the .condor.sub file is manually modified.
Adding ABORT-DAG-ON for node C in the diamond-shaped DAG
# File name: diamond.dag
#
JOB A A.condor
JOB B B.condor
JOB C C.condor
JOB D D.condor
PARENT A CHILD B C
PARENT B C CHILD D
Retry C 3
ABORT-DAG-ON C 10 RETURN 1
causes the DAG to be aborted, if node C exits with a return value of 10. Any other currently running nodes, of
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
92
which only node B is a possibility for this particular example, are stopped and removed. If this abort occurs, the return
value for the DAG is 1.
Variable Values Associated with Nodes
Macros defined for DAG nodes can be used within the submit description file of the node job. The VARS command
provides a method for defining a macro. Macros are defined on a per-node basis, using the syntax
VARS JobName|ALL_NODES macroname="string" [macroname="string". . .]
The macro may be used within the submit description file of the relevant node. A macroname may contain alphanumeric characters (a-z, A-Z, and 0-9) and the underscore character. The space character delimits macros, such
that there may be more than one macro defined on a single line. Multiple lines defining macros for the same node are
permitted.
Correct syntax requires that the string must be enclosed in double quotes. To use a double quote mark within a
string, escape the double quote mark with the backslash character (\). To add the backslash character itself, use two
backslashes (\\).
A restriction is that the macroname itself cannot begin with the string queue, in any combination of upper or
lower case letters.
Examples
If the DAG input file contains
# File
#
JOB A
JOB B
JOB C
JOB D
VARS A
PARENT
PARENT
name: diamond.dag
A.submit
B.submit
C.submit
D.submit
state="Wisconsin"
A CHILD B C
B C CHILD D
then the submit description file A.submit may use the macro state. Consider this submit description file
A.submit:
# file name:
executable =
log
=
arguments =
queue
A.submit
A.exe
A.log
"$(state)"
The macro value expands to become a command-line argument in the invocation of the job. The job is invoked with
A.exe Wisconsin
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
93
The use of macros may allow a reduction in the number of distinct submit description files. A separate example
shows this intended use of VARS. In the case where the submit description file for each node varies only in file naming,
macros reduce the number of submit description files to one.
This example references a single submit description file for each of the nodes in the DAG input file, and it uses the
VARS entry to name files used by each job.
The relevant portion of the DAG input file appears as
JOB A theonefile.sub
JOB B theonefile.sub
JOB C theonefile.sub
VARS A filename="A"
VARS B filename="B"
VARS C filename="C"
The submit description file appears as
# submit description file called:
executable
= progX
output
= $(filename)
error
= error.$(filename)
log
= $(filename).log
queue
theonefile.sub
For a DAG such as this one, but with thousands of nodes, the ability to write and maintain a single submit description file together with a single, yet more complex, DAG input file is worthwhile.
Multiple macroname definitions
If a macro name for a specific node in a DAG is defined more than once, as it would be with the partial file contents
JOB job1 job1.submit
VARS job1 a="foo"
VARS job1 a="bar"
a warning is written to the log, of the format
Warning: VAR <macroname> is already defined in job <JobName>
Discovered at file "<DAG input file name>", line <line number>
The behavior of DAGMan is such that all definitions for the macro exist, but only the last one defined is used as
the variable’s value. Using this example, if the job1.submit submit description file contains
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
94
arguments = "$(a)"
then the argument will be bar.
Special characters within VARS string definitions
The value defined for a macro may contain spaces and tabs. It is also possible to have double quote marks and
backslashes within a value. In order to have spaces or tabs within a value specified for a command line argument, use
the New Syntax format for the arguments submit command, as described in section 11. Escapes for double quote
marks depend on whether the New Syntax or Old Syntax format is used for the arguments submit command. Note
that in both syntaxes, double quote marks require two levels of escaping: one level is for the parsing of the DAG input
file, and the other level is for passing the resulting value through condor_submit.
As of HTCondor version 8.3.7, single quotes are permitted within the value specification. For the specification of
command line arguments, single quotes can be used in three ways:
• in Old Syntax, within a macro’s value specification
• in New Syntax, within a macro’s value specification
• in New Syntax only, to delimit an argument containing white space
There are examples of all three cases below. In New Syntax, to pass a single quote as part of an argument, escape it
with another single quote for condor_submit parsing as in the example’s NodeA fourth macro.
As an example that shows uses of all special characters, here are only the relevant parts of a DAG input file. Note
that the NodeA value for the macro second contains a tab.
VARS
VARS
VARS
VARS
VARS
NodeA
NodeA
NodeA
NodeA
NodeA
first="Alberto Contador"
second="\"\"Andy Schleck\"\""
third="Lance\\ Armstrong"
fourth="Vincenzo ''The Shark'' Nibali"
misc="!@#$%^&*()_-=+=[]{}?/"
VARS
VARS
VARS
VARS
VARS
NodeB
NodeB
NodeB
NodeB
NodeB
first="Lance_Armstrong"
second="\\\"Andreas_Kloden\\\""
third="Ivan\\_Basso"
fourth="Bernard_'The_Badger'_Hinault"
misc="!@#$%^&*()_-=+=[]{}?/"
VARS NodeC args="'Nairo Quintana' 'Chris Froome'"
Consider an example in which the submit description file for NodeA uses the New Syntax for the arguments
command:
arguments = "'$(first)' '$(second)' '$(third)' '($fourth)' '$(misc)'"
The single quotes around each variable reference are only necessary if the variable value may contain spaces or tabs.
The resulting values passed to the NodeA executable are:
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
95
Alberto Contador
"Andy Schleck"
Lance\ Armstrong
Vincenzo 'The Shark' Nibali
!@#$%^&*()_-=+=[]{}?/
Consider an example in which the submit description file for NodeB uses the Old Syntax for the arguments
command:
arguments = $(first) $(second) $(third) $(fourth) $(misc)
The resulting values passed to the NodeB executable are:
Lance_Armstrong
"Andreas_Kloden"
Ivan\_Basso
Bernard_'The_Badger'_Hinault
!@#$%^&*()_-=+=[]{}?/
Consider an example in which the submit description file for NodeC uses the New Syntax for the arguments
command:
arguments = "$(args)"
The resulting values passed to the NodeC executable are:
Nairo Quintana
Chris Froome
Using special macros within a definition
The $(JOB) and $(RETRY) macros may be used within a definition of the string that defines a variable. This
usage requires parentheses, such that proper macro substitution may take place when the macro’s value is only a
portion of the string.
• $(JOB) expands to the node JobName. If the VARS line appears in a DAG file used as a splice file, then
$(JOB) will be the fully scoped name of the node.
For example, the DAG input file lines
JOB NodeC NodeC.submit
VARS NodeC nodename="$(JOB)"
set nodename to NodeC, and the DAG input file lines
JOB NodeD NodeD.submit
VARS NodeD outfilename="$(JOB)-output"
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
96
set outfilename to NodeD-output.
• $(RETRY) expands to 0 the first time a node is run; the value is incremented each time the node is retried. For
example:
VARS NodeE noderetry="$(RETRY)"
Using VARS to define ClassAd attributes
The macroname may also begin with a + character, in which case it names a ClassAd attribute. For example, the
VARS specification
VARS NodeF +A="\"bob\""
results in the job ClassAd attribute
A = "bob"
Note that ClassAd string values must be quoted, hence there are escaped quotes in the example above. The outer
quotes are consumed in the parsing of the DAG input file, so the escaped inner quotes remain in the definition of the
attribute value.
Continuing this example, it allows the HTCondor submit description file for NodeF to use the following line:
arguments = "$$([A])"
The special macros may also be used. For example
VARS NodeG +B="$(RETRY)"
places the numerical attribute
B = 1
into the ClassAd when the NodeG job is run for a second time, which is the first retry and the value 1.
Setting Priorities for Nodes
The PRIORITY command assigns a priority to a DAG node (and to the HTCondor job(s) associated with the node).
The syntax for PRIORITY is
PRIORITY JobName|ALL_NODES PriorityValue
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
97
The priority value is an integer (which can be negative). A larger numerical priority is better. The default priority
is 0.
The node priority affects the order in which nodes that are ready (all of their parent nodes have finished successfully) at the same time will be submitted. The node priority also sets the node job’s priority in the queue (that is, its
JobPrio attribute), which affects the order in which jobs will be run once they are submitted (see 2.7.1 for more
information about job priority). The node priority only affects the order of job submission within a given DAG; but
once jobs are submitted, their JobPrio value affects the order in which they will be run relative to all jobs submitted
by the same user.
Sub-DAGs can have priorities, just as "regular" nodes can. (The priority of a sub-DAG will affect the priorities of
its nodes: see "effective node priorities" below.) Splices cannot be assigned a priority, but individual nodes within a
splice can be assigned priorities.
Note that node priority does not override the DAG dependencies. Also note that node priorities are not guarantees
of the relative order in which nodes will be run, even among nodes that become ready at the same time – so node
priorities should not be used as a substitute for parent/child dependencies. In other words, priorities should be used
when it is preferable, but not required, that some jobs run before others. (The order in which jobs are run once they
are submitted can be affected by many things other than the job’s priority; for example, whether there are machines
available in the pool that match the job’s requirements.)
PRE scripts can affect the order in which jobs run, so DAGs containing PRE scripts may not submit the nodes in
exact priority order, even if doing so would satisfy the DAG dependencies.
Node priority is most relevant if node submission is throttled (via the -maxjobs or -maxidle command-line arguments or the DAGMAN_MAX_JOBS_SUBMITTED or DAGMAN_MAX_JOBS_IDLE configuration variables), or if
there are not enough resources in the pool to immediately run all submitted node jobs. This is often the case for DAGs
with large numbers of "sibling" nodes, or DAGs running on heavily-loaded pools.
Example
Adding PRIORITY for node C in the diamond-shaped DAG:
# File name: diamond.dag
#
JOB A A.condor
JOB B B.condor
JOB C C.condor
JOB D D.condor
PARENT A CHILD B C
PARENT B C CHILD D
Retry C 3
PRIORITY C 1
This will cause node C to be submitted (and, mostly likely, run) before node B. Without this priority setting for
node C, node B would be submitted first because the "JOB" statement for node B comes earlier in the DAG file than
the "JOB" statement for node C.
Effective node priorities
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
98
The "effective" priority for a node (the priority controlling the order in which nodes are actually submitted,
and which is assigned to JobPrio) is the sum of the explicit priority (specified in the DAG file) and the priority
of the DAG itself. DAG priorities also default to 0, so they are most relevant for sub-DAGs (although a top-level
DAG can be submitted with a non-zero priority by specifying a -priority value on the condor_submit_dag command
line). This algorithm for calculating effective priorities is a simplification introduced in version 8.5.7 (a node’s
effective priority is no longer dependent on the priorities of its parents).
Here is an example to clarify:
# File name: priorities.dag
#
JOB A A.sub
SUBDAG EXTERNAL B SD.dag
PARENT A CHILD B
PRIORITY A 60
PRIORITY B 100
# File name: SD.dag
#
JOB SA SA.sub
JOB SB SB.sub
PARENT SA CHILD SB
PRIORITY SA 10
PRIORITY SB 20
In this example (assuming that priorities.dag is submitted with the default priority of 0), the effective priority of
node A will be 60, and the effective priority of sub-DAG B will be 100. Therefore, the effective priority of node SA
will be 110 and the effective priority of node SB will be 120.
The effective priorities listed above are assigned by DAGMan. There is no way to change the priority in the submit
description file for a job, as DAGMan will override any priority command placed in a submit description file (unless
the effective node priority is 0; in this case, any priority specified in the submit file will take effect).
Throttling Nodes by Category
In order to limit the number of submitted job clusters within a DAG, the nodes may be placed into categories by
assignment of a name. Then, a maximum number of submitted clusters may be specified for each category.
The CATEGORY command assigns a category name to a DAG node. The syntax for CATEGORY is
CATEGORY JobName|ALL_NODES CategoryName
Category names cannot contain white space.
The MAXJOBS command limits the number of submitted job clusters on a per category basis. The syntax for
MAXJOBS is
MAXJOBS CategoryName MaxJobsValue
If the number of submitted job clusters for a given category reaches the limit, no further job clusters in that category
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
99
will be submitted until other job clusters within the category terminate. If MAXJOBS is not set for a defined category,
then there is no limit placed on the number of submissions within that category.
Note that a single invocation of condor_submit results in one job cluster. The number of HTCondor jobs within a
cluster may be greater than 1.
The configuration variable DAGMAN_MAX_JOBS_SUBMITTED and the condor_submit_dag -maxjobs commandline option are still enforced if these CATEGORY and MAXJOBS throttles are used.
Please see the end of section 2.10.9 on DAG Splicing for a description of the interaction between categories and
splices.
Configuration Specific to a DAG
All configuration variables and their definitions that relate to DAGMan may be found in section 3.5.24.
Configuration variables for condor_dagman can be specified in several ways, as given within the ordered list:
1. In an HTCondor configuration file.
2. With an environment variable. Prepend the string _CONDOR_ to the configuration variable’s name.
3. With a line in the DAG input file using the keyword CONFIG, such that there is a configuration file specified
that is specific to an instance of condor_dagman. The configuration file specification may instead be specified
on the condor_submit_dag command line using the -config option.
4. For some configuration variables, condor_submit_dag command line argument specifies a configuration variable. For example, the configuration variable DAGMAN_MAX_JOBS_SUBMITTED has the corresponding command line argument -maxjobs.
For this ordered list, configuration values specified or parsed later in the list override ones specified earlier. For example, a value specified on the condor_submit_dag command line overrides corresponding values in any configuration
file. And, a value specified in a DAGMan-specific configuration file overrides values specified in a general HTCondor
configuration file.
The CONFIG command within the DAG input file specifies a configuration file to be used to set configuration
variables related to condor_dagman when running this DAG. The syntax for CONFIG is
CONFIG ConfigFileName
As an example, if the DAG input file contains:
CONFIG dagman.config
then the configuration values in file dagman.config will be used for this DAG. If the contents of file
dagman.config is
DAGMAN_MAX_JOBS_IDLE = 10
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
100
then this configuration is defined for this DAG.
Only a single configuration file can be specified for a given condor_dagman run. For example, if one file is
specified within a DAG input file, and a different file is specified on the condor_submit_dag command line, this is a
fatal error at submit time. The same is true if different configuration files are specified in multiple DAG input files and
referenced in a single condor_submit_dag command.
If multiple DAGs are run in a single condor_dagman run, the configuration options specified in the condor_dagman
configuration file, if any, apply to all DAGs, even if some of the DAGs specify no configuration file.
Configuration variables that are not for condor_dagman and not utilized by DaemonCore, yet are specified in a
condor_dagman-specific configuration file are ignored.
Setting ClassAd attributes in the DAG file
The SET_JOB_ATTR keyword within the DAG input file specifies an attribute/value pair to be set in the DAGMan
job’s ClassAd. The syntax for SET_JOB_ATTR is
SET_JOB_ATTR AttributeName=AttributeValue
As an example, if the DAG input file contains:
SET_JOB_ATTR TestNumber = 17
the ClassAd of the DAGMan job itself will have an attribute TestNumber with the value 17.
The attribute set by the SET_JOB_ATTR command is set only in the ClassAd of the DAGMan job itself – it is not
propagated to node jobs of the DAG.
Values with spaces can be set by surrounding the string containing a space with single or double quotes. (Note that
the quote marks themselves will be part of the value.)
Only a single attribute/value pair can be specified per SET_JOB_ATTR command. If the same attribute is specified
multiple times in the DAG (or in multiple DAGs run by the same DAGMan instance) the last-specified value is the one
that will be utilized. An attribute set in the DAG file can be overridden by specifying
-append '+<attribute> = <value>'
on the condor_submit_dag command line.
Optimization of Submission Time
condor_dagman works by watching log files for events, such as submission, termination, and going on hold. When
a new job is ready to be run, it is submitted to the condor_schedd, which needs to acquire a computing resource.
Acquisition requires the condor_schedd to contact the central manager and get a claim on a machine, and this claim
cycle can take many minutes.
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
101
Configuration variable DAGMAN_HOLD_CLAIM_TIME avoids the wait for a negotiation cycle. When set to a non
zero value, the condor_schedd keeps a claim idle, such that the condor_startd delays in shifting from the Claimed
to the Preempting state (see Figure 3.1). Thus, if another job appears that is suitable for the claimed resource, then
the condor_schedd will submit the job directly to the condor_startd, avoiding the wait and overhead of a negotiation
cycle. This results in a speed up of job completion, especially for linear DAGs in pools that have lengthy negotiation
cycle times.
By default, DAGMAN_HOLD_CLAIM_TIME is 20, causing a claim to remain idle for 20 seconds, during which
time a new job can be submitted directly to the already-claimed condor_startd. A value of 0 means that claims are
not held idle for a running DAG. If a DAG node has no children, the value of DAGMAN_HOLD_CLAIM_TIME will be
ignored; the KeepClaimIdle attribute will not be defined in the job ClassAd of the node job, unless the job requests
it using the submit command keep_claim_idle.
Single Submission of Multiple, Independent DAGs
A single use of condor_submit_dag may execute multiple, independent DAGs. Each independent DAG has its own,
distinct DAG input file. These DAG input files are command-line arguments to condor_submit_dag.
Internally, all of the independent DAGs are combined into a single, larger DAG, with no dependencies between
the original independent DAGs. As a result, any generated Rescue DAG file represents all of the original independent
DAGs with a single DAG. The file name of this Rescue DAG is based on the DAG input file listed first within the
command-line arguments. For example, assume that three independent DAGs are submitted with
condor_submit_dag A.dag B.dag C.dag
The first listed is A.dag. The remainder of the specialized file name adds a suffix onto this first DAG input file name,
A.dag. The suffix is _multi.rescue<XXX>, where <XXX> is substituted by the 3-digit number of the Rescue
DAG created as defined in section 2.10.10. The first time a Rescue DAG is created for the example, it will have the
file name A.dag_multi.rescue001.
Other files such as dagman.out and the lock file also have names based on this first DAG input file.
The success or failure of the independent DAGs is well defined. When multiple, independent DAGs are submitted
with a single command, the success of the composite DAG is defined as the logical AND of the success of each
independent DAG. This implies that failure is defined as the logical OR of the failure of any of the independent DAGs.
By default, DAGMan internally renames the nodes to avoid node name collisions. If all node names are unique, the
renaming of nodes may be disabled by setting the configuration variable DAGMAN_MUNGE_NODE_NAMES to False
(see 3.5.24).
INCLUDE
The INCLUDE command allows the contents of one DAG file to be parsed as if they were physically included in the
referencing DAG file. The syntax for INCLUDE is
INCLUDE FileName
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
102
For example, if we have two DAG files like this:
# File name: foo.dag
#
JOB A A.sub
INCLUDE bar.dag
# File name: bar.dag
#
JOB B B.sub
JOB C C.sub
this is equivalent to the single DAG file:
JOB
JOB
JOB
A
B
C
A.sub
B.sub
C.sub
Note that the included file must be in proper DAG syntax. Also, there are many cases where a valid included DAG
file will cause a parse error, such as the including and included files defining nodes with the same name.
INCLUDEs can be nested to any depth (be sure not to create a cycle of includes!).
Example: Using INCLUDE to simplify multiple similar workflows
One use of the INCLUDE command is to simplify the DAG files when we have a single workflow that we want to
run on a number of data sets. In that case, we can do something like this:
# File name: workflow.dag
# Defines the structure of the workflow
JOB Split split.sub
JOB Process00 process.sub
...
JOB Process99 process.sub
JOB Combine combine.sub
PARENT Split CHILD Process00 ... Process99
PARENT Process00 ... Process99 CHILD Combine
# File name: split.sub
executable = my_split
input = $(dataset).phase1
output = $(dataset).phase2
...
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
103
# File name: data57.vars
VARS Split dataset="data57"
VARS Process00 dataset="data57"
...
VARS Process99 dataset="data57"
VARS Combine dataset="data57"
# File name: run_dataset57.dag
INCLUDE workflow.dag
INCLUDE data57.vars
Then, to run our workflow on dataset 57, we run the following command:
condor_submit_dag run_dataset57.dag
This avoids having to duplicate the JOB and PARENT/CHILD commands for every dataset – we can just re-use the
workflow.dag file, in combination with a dataset-specific vars file.
Composing workflows from multiple DAG files
The organization and dependencies of the jobs within a DAG are the keys to its utility. Some workflows are naturally
constructed hierarchically, such that a node within a DAG is also a DAG (instead of a "simple" HTCondor job).
HTCondor DAGMan handles this situation easily, and allows DAGs to be nested to any depth.
There are two ways that DAGs can be nested within other DAGs: sub-DAGs (see 2.10.9) and splices (see 2.10.9).
With sub-DAGs, each DAG has its own condor_dagman job, which then becomes a node job within the higherlevel DAG. With splices, on the other hand, the nodes of the spliced DAG are directly incorporated into the higher-level
DAG. Therefore, splices do not result in additional condor_dagman instances.
A weakness in scalability exists when submitting external sub-DAGs, because each executing independent DAG
requires its own instance of condor_dagman to be running. The outer DAG has an instance of condor_dagman, and
each named SUBDAG has an instance of condor_dagman while it is in the HTCondor queue. The scaling issue
presents itself when a workflow contains hundreds or thousands of sub-DAGs that are queued at the same time. (In
this case, the resources (especially memory) consumed by the multiple condor_dagman instances can be a problem.)
Further, there may be many Rescue DAGs created if a problem occurs. (Note that the scaling issue depends only on
how many sub-DAGs are queued at any given time, not the total number of sub-DAGs in a given workflow; division
of a large workflow into sequential sub-DAGs can actually enhance scalability.) To alleviate these concerns, the
DAGMan language introduces the concept of graph splicing.
Because splices are simpler in some ways than sub-DAGs, they are generally preferred unless certain features are needed that are only available with sub-DAGs.
This document:
https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=SubDagsVsSplices explains the pros and cons of splices
and external sub-DAGs, and should help users decide which alternative is better for their application.
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
104
Note that sub-DAGs and splices can be combined in a single workflow, and can be nested to any depth (but be sure
to avoid recursion, which will cause problems!).
A DAG Within a DAG Is a SUBDAG
As stated above, the SUBDAG EXTERNAL command causes the specified DAG file to be run by a separate instance
of condor_dagman, with the condor_dagman job becoming a node job within the higher-level DAG.
The syntax for the SUBDAG command is
SUBDAG EXTERNAL JobName DagFileName [DIR directory] [NOOP] [DONE]
The optional specifications of DIR, NOOP, and DONE, if used, must appear in this order within the entry. NOOP
and DONE for SUBDAG nodes have the same effect that they do for JOB nodes.
A SUBDAG node is essentially the same as any other node, except that the DAG input file for the inner DAG is
specified, instead of the HTCondor submit file. The keyword EXTERNAL means that the SUBDAG is run within its
own instance of condor_dagman.
Since more than one DAG is being discussed, here is terminology introduced to clarify which DAG is which.
Reuse the example diamond-shaped DAG as given in Figure 2.2. Assume that node B of this diamond-shaped DAG
will itself be a DAG. The DAG of node B is called a SUBDAG, inner DAG, or lower-level DAG. The diamond-shaped
DAG is called the outer or top-level DAG.
Work on the inner DAG first. Here is a very simple linear DAG input file used as an example of the inner DAG.
# File name: inner.dag
#
JOB X X.submit
JOB Y Y.submit
JOB Z Z.submit
PARENT X CHILD Y
PARENT Y CHILD Z
The HTCondor submit description file, used by condor_dagman, corresponding to inner.dag will be
named inner.dag.condor.sub. The DAGMan submit description file is always named <DAG file
name>.condor.sub. Each DAG or SUBDAG results in the submission of condor_dagman as an HTCondor job,
and condor_submit_dag creates this submit description file.
The preferred specification of the DAG input file for the outer DAG is
# File name: diamond.dag
#
JOB A A.submit
SUBDAG EXTERNAL B inner.dag
JOB C C.submit
JOB D D.submit
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
105
PARENT A CHILD B C
PARENT B C CHILD D
Within the outer DAG’s input file, the SUBDAG command specifies a special case of a JOB node, where the job
is itself a DAG.
One of the benefits of using the SUBDAG feature is that portions of the overall workflow can be constructed
and modified during the execution of the DAG (a SUBDAG file doesn’t have to exist until just before it is submitted). A drawback can be that each SUBDAG causes its own distinct job submission of condor_dagman, leading to a
larger number of jobs, together with their potential need of carefully constructed policy configuration to throttle node
submission or execution (because each SUBDAG has its own throttles).
Here are details that affect SUBDAGs:
• Nested DAG Submit Description File Generation
There are three ways to generate the <DAG file name>.condor.sub file of a SUBDAG:
– Lazily (the default in HTCondor version 7.5.2 and later versions)
– Eagerly (the default in HTCondor versions 7.4.1 through 7.5.1)
– Manually (the only way prior to version HTCondor version 7.4.1)
When the <DAG file name>.condor.sub file is generated lazily, this file is generated immediately before the SUBDAG job is submitted. Generation is accomplished by running
condor_submit_dag -no_submit
on the DAG input file specified in the SUBDAG entry. This is the default behavior. There are advantages to this
lazy mode of submit description file creation for the SUBDAG:
– The DAG input file for a SUBDAG does not have to exist until the SUBDAG is ready to run, so this file
can be dynamically created by earlier parts of the outer DAG or by the PRE script of the node containing
the SUBDAG.
– It is now possible to have SUBDAGs within splices. That is not possible with eager submit description file
creation, because condor_submit_dag does not understand splices.
The main disadvantage of lazy submit file generation is that a syntax error in the DAG input file of a SUBDAG
will not be discovered until the outer DAG tries to run the inner DAG.
When <DAG file name>.condor.sub files are generated eagerly, condor_submit_dag runs itself recursively (with the -no_submit option) on each SUBDAG, so all of the <DAG file name>.condor.sub
files are generated before the top-level DAG is actually submitted. To generate the <DAG file
name>.condor.sub files eagerly, pass the -do_recurse flag to condor_submit_dag; also set the
DAGMAN_GENERATE_SUBDAG_SUBMITS configuration variable to False, so that condor_dagman does
not re-run condor_submit_dag at run time thereby regenerating the submit description files.
To generate the .condor.sub files manually, run
condor_submit_dag -no_submit
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
106
on each lower-level DAG file, before running condor_submit_dag on the top-level DAG file; also set
the DAGMAN_GENERATE_SUBDAG_SUBMITS configuration variable to False, so that condor_dagman
does not re-run condor_submit_dag at run time. The main reason for generating the <DAG file
name>.condor.sub files manually is to set options for the lower-level DAG that one would not otherwise
be able to set An example of this is the -insert_sub_file option. For instance, using the given example do the
following to manually generate HTCondor submit description files:
condor_submit_dag -no_submit -insert_sub_file fragment.sub inner.dag
condor_submit_dag diamond.dag
Note that most condor_submit_dag command-line flags have corresponding configuration variables, so we encourage the use of per-DAG configuration files, especially in the case of nested DAGs. This is the easiest way
to set different options for different DAGs in an overall workflow.
It is possible to combine more than one method of generating the <DAG file name>.condor.sub
files.
For example, one might pass the -do_recurse flag to condor_submit_dag, but leave the
DAGMAN_GENERATE_SUBDAG_SUBMITS configuration variable set to the default of True. Doing this
would provide the benefit of an immediate error message at submit time, if there is a syntax error in one of
the inner DAG input files, but the lower-level <DAG file name>.condor.sub files would still be regenerated before each nested DAG is submitted.
The values of the following command-line flags are passed from the top-level condor_submit_dag instance to
any lower-level condor_submit_dag instances. This occurs whether the lower-level submit description files are
generated lazily or eagerly:
– -verbose
– -force
– -notification
– -allowlogerror
– -dagman
– -usedagdir
– -outfile_dir
– -oldrescue
– -autorescue
– -dorescuefrom
– -allowversionmismatch
– -no_recurse/do_recurse
– -update_submit
– -import_env
– -suppress_notification
– -priority
– -dont_use_default_node_log
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
107
The values of the following command-line flags are preserved in any already-existing lower-level DAG submit
description files:
– -maxjobs
– -maxidle
– -maxpre
– -maxpost
– -debug
Other command-line arguments are set to their defaults in any lower-level invocations of condor_submit_dag.
The -force option will cause existing DAG submit description files to be overwritten without preserving any
existing values.
• Submission of the outer DAG
The outer DAG is submitted as before, with the command
condor_submit_dag diamond.dag
• Interaction with Rescue DAGs
The use of new-style Rescue DAGs is now the default. With new-style rescue DAGs, the appropriate rescue DAG(s) will be run automatically if there is a failure somewhere in the workflow. For example (given
the DAGs in the example at the beginning of the SUBDAG section), if one of the nodes in inner.dag
fails, this will produce a Rescue DAG for inner.dag (named inner.dag.rescue.001). Then, since
inner.dag failed, node B of diamond.dag will fail, producing a Rescue DAG for diamond.dag (named
diamond.dag.rescue.001, etc.). If the command
condor_submit_dag diamond.dag
is re-run, the most recent outer Rescue DAG will be run, and this will re-run the inner DAG, which will in turn
run the most recent inner Rescue DAG.
• File Paths
Remember that, unless the DIR keyword is used in the outer DAG, the inner DAG utilizes the current working
directory when the outer DAG is submitted. Therefore, all paths utilized by the inner DAG file must be specified
accordingly.
DAG Splicing
As stated above, the SPLICE command causes the nodes of the spliced DAG to be directly incorporated into the
higher-level DAG (the DAG containing the SPLICE command).
The syntax for the SPLICE command is
SPLICE SpliceName DagFileName [DIR directory]
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
108
A splice is a named instance of a subgraph which is specified in a separate DAG file. The splice is treated as an
entity for dependency specification in the including DAG. (Conceptually, a splice is treated as a node within the DAG
containing the SPLICE command, although there are some limitations, which are discussed below. This means, for
example, that splices can have parents and children.) A splice can also be incorporated into an including DAG without
any dependencies; it is then considered a disjoint DAG within the including DAG.
The same DAG file can be reused as differently named splices, each one incorporating a copy of the dependency
graph (and nodes therein) into the including DAG.
The nodes within a splice are scoped according to a hierarchy of names associated with the splices, as the splices
are parsed from the top level DAG file. The scoping character to describe the inclusion hierarchy of nodes into the
top level dag is '+'. (In other words, if a splice named "SpliceX" contains a node named "NodeY", the full node
name once the DAGs are parsed is "SpliceX+NodeY". This character is chosen due to a restriction in the allowable
characters which may be in a file name across the variety of platforms that HTCondor supports. In any DAG input file,
all splices must have unique names, but the same splice name may be reused in different DAG input files.
HTCondor does not detect nor support splices that form a cycle within the DAG. A DAGMan job that causes a
cyclic inclusion of splices will eventually exhaust available memory and crash.
The SPLICE command in a DAG input file creates a named instance of a DAG as specified in another file as an
entity which may have PARENT and CHILD dependencies associated with other splice names or node names in the
including DAG file.
The following series of examples illustrate potential uses of splicing. To simplify the examples, presume that each
and every job uses the same, simple HTCondor submit description file:
# BEGIN SUBMIT FILE submit.condor
executable
= /bin/echo
arguments
= OK
universe
= vanilla
output
= $(jobname).out
error
= $(jobname).err
log
= submit.log
notification = NEVER
queue
# END SUBMIT FILE submit.condor
This first simple example splices a diamond-shaped DAG in between the two nodes of a top level DAG. Here is
the DAG input file for the diamond-shaped DAG:
# BEGIN DAG FILE diamond.dag
JOB A submit.condor
VARS A jobname="$(JOB)"
JOB B submit.condor
VARS B jobname="$(JOB)"
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
109
JOB C submit.condor
VARS C jobname="$(JOB)"
JOB D submit.condor
VARS D jobname="$(JOB)"
PARENT A CHILD B C
PARENT B C CHILD D
# END DAG FILE diamond.dag
The top level DAG incorporates the diamond-shaped splice:
# BEGIN DAG FILE toplevel.dag
JOB X submit.condor
VARS X jobname="$(JOB)"
JOB Y submit.condor
VARS Y jobname="$(JOB)"
# This is an instance of diamond.dag, given the symbolic name DIAMOND
SPLICE DIAMOND diamond.dag
# Set up a relationship between the nodes in this dag and the splice
PARENT X CHILD DIAMOND
PARENT DIAMOND CHILD Y
# END DAG FILE toplevel.dag
Figure 2.3 illustrates the resulting top level DAG and the dependencies produced. Notice the naming of nodes
scoped with the splice name. This hierarchy of splice names assures unique names associated with all nodes.
Figure 2.4 illustrates the starting point for a more complex example. The DAG input file X.dag describes this
X-shaped DAG. The completed example displays more of the spatial constructs provided by splices. Pay particular
attention to the notion that each named splice creates a new graph, even when the same DAG input file is specified.
# BEGIN DAG FILE X.dag
JOB A submit.condor
VARS A jobname="$(JOB)"
JOB B submit.condor
VARS B jobname="$(JOB)"
JOB C submit.condor
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
110
X
DIAMOND+A
DIAMOND+B
DIAMOND+C
DIAMOND+D
Y
Figure 2.3: The diamond-shaped DAG spliced between two nodes.
VARS C jobname="$(JOB)"
JOB D submit.condor
VARS D jobname="$(JOB)"
JOB E submit.condor
VARS E jobname="$(JOB)"
JOB F submit.condor
VARS F jobname="$(JOB)"
JOB G submit.condor
VARS G jobname="$(JOB)"
# Make an X-shaped dependency graph
PARENT A B C CHILD D
PARENT D CHILD E F G
# END DAG FILE X.dag
File s1.dag continues the example, presenting the DAG input file that incorporates two separate splices of the
X-shaped DAG. Figure 2.5 illustrates the resulting DAG.
# BEGIN DAG FILE s1.dag
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
111
A
B
C
D
E
F
G
Figure 2.4: The X-shaped DAG.
JOB A submit.condor
VARS A jobname="$(JOB)"
JOB B submit.condor
VARS B jobname="$(JOB)"
# name two individual splices of the X-shaped DAG
SPLICE X1 X.dag
SPLICE X2 X.dag
# Define dependencies
# A must complete before the initial nodes in X1 can start
PARENT A CHILD X1
# All final nodes in X1 must finish before
# the initial nodes in X2 can begin
PARENT X1 CHILD X2
# All final nodes in X2 must finish before B may begin.
PARENT X2 CHILD B
# END DAG FILE s1.dag
The top level DAG in the hierarchy of this complex example is described by the DAG input file toplevel.dag.
Figure 2.6 illustrates the final DAG. Notice that the DAG has two disjoint graphs in it as a result of splice S3 not having
any dependencies associated with it in this top level DAG.
# BEGIN DAG FILE toplevel.dag
JOB A submit.condor
VARS A jobname="$(JOB)"
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
112
A
X1+A
X1+B
X1+C
X1+D
X1+E
X1+F
X1+G
X2+A
X2+B
X2+C
X2+D
X2+E
X2+F
X2+G
B
Figure 2.5: The DAG described by s1.dag.
JOB B submit.condor
VARS B jobname="$(JOB)"
JOB C submit.condor
VARS C jobname="$(JOB)"
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
113
JOB D submit.condor
VARS D jobname="$(JOB)"
# a diamond-shaped DAG
PARENT A CHILD B C
PARENT B C CHILD D
# This splice of the X-shaped DAG can only run after
# the diamond dag finishes
SPLICE S2 X.dag
PARENT D CHILD S2
# Since there are no dependencies for S3,
# the following splice is disjoint
SPLICE S3 s1.dag
# END DAG FILE toplevel.dag
Splices and rescue DAGs
Because the nodes of a splice are directly incorporated into the DAG containing the SPLICE command, splices do
not generate their own rescue DAGs, unlike SUBDAG EXTERNALs.
The DIR option with splices
The DIR option specifies a working directory for a splice, from which the splice will be parsed and the jobs within
the splice submitted. The directory associated with the splice’s DIR specification will be propagated as a prefix to
all nodes in the splice and any included splices. If a node already has a DIR specification, then the splice’s DIR
specification will be a prefix to the node’s, separated by a directory separator character. Jobs in included splices with
an absolute path for their DIR specification will have their DIR specification untouched. Note that a DAG containing
DIR specifications cannot be run in conjunction with the -usedagdir command-line argument to condor_submit_dag.
A "full" rescue DAG generated by a DAG run with the -usedagdir argument will contain DIR specifications, so
such a rescue DAG must be run without the -usedagdir argument. (Note that "full" rescue DAGs are no longer the
default.)
Limitation: splice DAGs must exist at submit time
Unlike the DAG files referenced in a SUBDAG EXTERNAL command, DAG files referenced in a SPLICE command
must exist when the DAG containing the SPLICE command is submitted. (Note that, if a SPLICE is contained within
a sub-DAG, the splice DAG must exist at the time that the sub-DAG is submitted, not when the top-most DAG is
submitted, so the splice DAG can be created by a part of the workflow that runs before the relevant sub-DAG.)
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
114
A
S3+A
B
C
S3+X1+A
D
S2+A
S3+X1+C
S3+X1+D
S2+B
S2+C
S2+D
S2+E
S3+X1+B
S2+F
S3+X1+E
S3+X1+F
S3+X1+G
S3+X2+A
S3+X2+B
S3+X2+C
S2+G
S3+X2+D
S3+X2+E
S3+X2+F
S3+X2+G
S3+B
Figure 2.6: The complex splice example DAG.
Limitation: Splices and PRE or POST Scripts
A PRE or POST script may not be specified for a splice (however, nodes within a spliced DAG can have PRE and
POST scripts). (The reason for this is that, when the DAG is parsed, the splices are also parsed and the splice nodes
are directly incorporated into the DAG containing the SPLICE command. Therefore, once parsing is complete, there
are no actual nodes corresponding to the splice itself to which to "attach" the PRE or POST scripts.)
To achieve the desired effect of having a PRE script associated with a splice, introduce a new NOOP node into the
DAG with the splice as a dependency. Attach the PRE script to the NOOP node.
# BEGIN DAG FILE example1.dag
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
115
# Names a node with no associated node job, a NOOP node
# Note that the file noop.submit does not need to exist
JOB OnlyPreNode noop.submit NOOP
# Attach a PRE script to the NOOP node
SCRIPT PRE OnlyPreNode prescript.sh
# Define the splice
SPLICE TheSplice thenode.dag
# Define the dependency
PARENT OnlyPreNode CHILD TheSplice
# END DAG FILE example1.dag
The same technique is used to achieve the effect of having a POST script associated with a splice. Introduce a new
NOOP node into the DAG as a child of the splice, and attach the POST script to the NOOP node.
# BEGIN DAG FILE example2.dag
# Names a node with no associated node job, a NOOP node
# Note that the file noop.submit does not need to exist.
JOB OnlyPostNode noop.submit NOOP
# Attach a POST script to the NOOP node
SCRIPT POST OnlyPostNode postscript.sh
# Define the splice
SPLICE TheSplice thenode.dag
# Define the dependency
PARENT TheSplice CHILD OnlyPostNode
# END DAG FILE example2.dag
Limitation: Splices and the RETRY of a Node, use of VARS, or use of PRIORITY
A RETRY, VARS or PRIORITY command cannot be specified for a SPLICE; however, individual nodes within a
spliced DAG can have a RETRY, VARS or PRIORITY specified.
Here is an example showing a DAG that will not be parsed successfully:
# top level DAG input file
JOB
A a.sub
SPLICE B b.dag
PARENT A CHILD B
# cannot work, as B is not a node in the DAG once
# splice B is incorporated
RETRY B 3
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
116
VARS B dataset="10"
PRIORITY B 20
The following example will work:
# top level DAG input file
JOB
A a.sub
SPLICE B b.dag
PARENT A CHILD B
# file: b.dag
JOB
X x.sub
RETRY X 3
VARS X dataset="10"
PRIORITY X 20
When RETRY is desired on an entire subgraph of a workflow, sub-DAGs (see above) must be used instead of
splices.
Here is the same example, now defining job B as a SUBDAG, and effecting RETRY on that SUBDAG.
# top level DAG input file
JOB
A a.sub
SUBDAG EXTERNAL B b.dag
PARENT A CHILD B
RETRY B 3
Limitation: The Interaction of Categories and MAXJOBS with Splices
Categories normally refer only to nodes within a given splice. All of the assignments of nodes to a category, and
the setting of the category throttle, should be done within a single DAG file. However, it is now possible to have
categories include nodes from within more than one splice. To do this, the category name is prefixed with the ’+’
(plus) character. This tells DAGMan that the category is a cross-splice category. Towards deeper understanding, what
this really does is prevent renaming of the category when the splice is incorporated into the upper-level DAG. The
MAXJOBS specification for the category can appear in either the upper-level DAG file or one of the splice DAG files.
It probably makes the most sense to put it in the upper-level DAG file.
Here is an example which applies a single limitation on submitted jobs, identifying the category with +init.
# relevant portion of file name: upper.dag
SPLICE A splice1.dag
SPLICE B splice2.dag
MAXJOBS +init 2
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
117
# relevant portion of file name: splice1.dag
JOB C C.sub
CATEGORY C +init
JOB D D.sub
CATEGORY D +init
# relevant portion of file name: splice2.dag
JOB X X.sub
CATEGORY X +init
JOB Y Y.sub
CATEGORY Y +init
For both global and non-global category throttles, settings at a higher level in the DAG override settings at a lower
level. In this example:
# relevant portion of file name: upper.dag
SPLICE A lower.dag
MAXJOBS A+catX 10
MAXJOBS +catY 2
# relevant portion of file name: lower.dag
MAXJOBS catX 5
MAXJOBS +catY 1
the resulting throttle settings are 2 for the +catY category and 10 for the A+catX category in splice. Note that
non-global category names are prefixed with their splice name(s), so to refer to a non-global category at a higher level,
the splice name must be included.
DAG Splice Connections
In the "default" usage of splices described above, when one splice is the parent of another splice, all "terminal" nodes
(nodes with no children) of the parent splice become parents of all "initial" nodes (nodes with no parents) of the child
splice. The CONNECT, PIN_IN, and PIN_OUT commands (added in version 8.5.7) allow more flexible dependencies
between splices. (The terms PIN_IN and PIN_OUT were chosen because of the hardware analogy.)
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
118
The syntax for CONNECT is
CONNECT OutputSpliceName InputSpliceName
The syntax for PIN_IN is
PIN_IN NodeName PinNumber
The syntax for PIN_OUT is
PIN_OUT NodeName PinNumber
All output splice nodes connected to a given pin_out will become parents of all input splice nodes connected to the
corresponding pin_in. (The pin_ins and pin_outs exist only to create the correct parent/child dependencies between
nodes. Once the DAG is parsed, there are no actual DAG objects corresponding to the pin_ins and pin_outs.)
Any given splice can contain both PIN_IN and PIN_OUT definitions, and can be both an input and output splice
in different CONNECT commands. Furthermore, a splice can appear in any number of CONNECT commands (for
example, a given splice could be the output splice in two CONNECT commands that have different input splices). It is
not an error for a splice to have PIN_IN or PIN_OUT definitions that are not associated with a CONNECT command
– such PIN_IN and PIN_OUT commands are simply ignored.
Note that the pin_ins and pin_outs must be defined within the relevant splices (this can be done with INCLUDE
commands), not in the DAG that connects the splices.
There are a number of restrictions on splice connections:
• Connections can be made only between two splices; "regular" nodes or sub-DAGs cannot be used in a CONNECT command.
• Pin_ins and pin_outs must be numbered consecutively starting at 1.
• The pin_outs of the output splice in a connect command must match the pin_ins of the input splice in the
command.
• All "initial" nodes (nodes with no parents) of an input splice used in a CONNECT command must be connected
to a pin_in.
Violating any of these restrictions will result in an error during the parsing of the DAG files.
Note: it is probably desireable for any "terminal" node (a node with no children) in the output splice to be connected
to a pin_out – but this is not required.
Here is a simple example:
# File: top.dag
SPLICE A spliceA.dag
SPLICE B spliceB.dag
SPLICE C spliceC.dag
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
119
CONNECT A B
CONNECT B C
# File: spliceA.dag
JOB A1 A1.sub
JOB A2 A2.sub
PIN_OUT A1 1
PIN_OUT A2 2
# File:
JOB
JOB
JOB
JOB
spliceB.dag
B1 B1.sub
B2 B2.sub
B3 B3.sub
B4 B4.sub
PIN_IN
PIN_IN
PIN_IN
PIN_IN
B1
B2
B3
B4
PIN_OUT
PIN_OUT
PIN_OUT
PIN_OUT
1
1
2
2
B1
B2
B3
B4
1
2
3
4
# File: spliceC.dag
JOB C1 C1.sub
PIN_IN
PIN_IN
PIN_IN
PIN_IN
C1
C1
C1
C1
1
2
3
4
In this example, node A1 will be the parent of B1 and B2; node A2 will be the parent of B3 and B4; and nodes B1,
B2, B3 and B4 will all be parents of C1.
A diagram of the above example:
FINAL node
A FINAL node is a single and special node that is always run at the end of the DAG, even if previous nodes in the
DAG have failed. A FINAL node can be used for tasks such as cleaning up intermediate files and checking the output
of previous nodes. The FINAL command in the DAG input file specifies a node job to be run at the end of the DAG.
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
120
A1
po1
po2
pi1
B1
pi2
B2
po1
pi1
Splice A
A2
B3
B4
po2
po3
po4
pi2
pi3
pi4
C1
Splice B
Splice C
Figure 2.7: Diagram of the splice connect example
The syntax used for the FINAL command is
FINAL JobName SubmitDescriptionFileName [DIR directory] [NOOP]
The FINAL node within the DAG is identified by JobName, and the HTCondor job is described by the contents of
the HTCondor submit description file given by SubmitDescriptionFileName.
The keywords DIR and NOOP are as detailed in section 2.10.2. If both DIR and NOOP are used, they must appear
in the order shown within the syntax specification.
There may only be one FINAL node in a DAG. A parse error will be logged by the condor_dagman job in the
dagman.out file, if more than one FINAL node is specified.
The FINAL node is virtually always run. It is run if the condor_dagman job is removed with condor_rm. The only
case in which a FINAL node is not run is if the configuration variable DAGMAN_STARTUP_CYCLE_DETECT is set
to True, and a cycle is detected at start up time. If DAGMAN_STARTUP_CYCLE_DETECT is set to False and a
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
121
cycle is detected during the course of the run, the FINAL node will be run.
The success or failure of the FINAL node determines the success or failure of the entire DAG, overriding the status
of all previous nodes. This includes any status specified by any ABORT-DAG-ON specification that has taken effect.
If some nodes of a DAG fail, but the FINAL node succeeds, the DAG will be considered successful. Therefore, it is
important to be careful about setting the exit status of the FINAL node.
The $DAG_STATUS and $FAILED_COUNT macros can be used both as PRE and POST script arguments, and in
node job submit description files. As an example of this, here are the partial contents of the DAG input file,
FINAL final_node final_node.sub
SCRIPT PRE final_node final_pre.pl $DAG_STATUS $FAILED_COUNT
and here are the partial contents of the submit description file, final_node.sub
arguments = "$(DAG_STATUS) $(FAILED_COUNT)"
If there is a FINAL node specified for a DAG, it will be run at the end of the workflow. If this FINAL node must
not do anything in certain cases, use the $DAG_STATUS and $FAILED_COUNT macros to take appropriate actions.
Here is an example of that behavior. It uses a PRE script that aborts if the DAG has been removed with condor_rm,
which, in turn, causes the FINAL node to be considered failed without actually submitting the HTCondor job specified
for the node. Partial contents of the DAG input file:
FINAL final_node final_node.sub
SCRIPT PRE final_node final_pre.pl $DAG_STATUS
and partial contents of the Perl PRE script, final_pre.pl:
#! /usr/bin/env perl
if ($ARGV[0] eq 4) {
exit(1);
}
There are restrictions on the use of a FINAL node. The DONE option is not allowed for a FINAL node. And, a
FINAL node may not be referenced in any of the following specifications:
• PARENT, CHILD
• RETRY
• ABORT-DAG-ON
• PRIORITY
HTCondor Version 8.6.4 Manual
2.10.9. Advanced Features of DAGMan
122
• CATEGORY
As of HTCondor version 8.3.7, DAGMan allows at most two submit attempts of a FINAL node, if the DAG has
been removed from the queue with condor_rm.
The ALL_NODES option
In the following commands, a specific node name can be replaced by the option ALL_NODES:
• SCRIPT
• PRE_SKIP
• RETRY
• ABORT-DAG-ON
• VARS
• PRIORITY
• CATEGORY
This will cause the given command to apply to all nodes (except any FINAL node) in that DAG.
The ALL_NODES never applies to a FINAL node. (If the ALL_NODES option is used in a DAG that has a FINAL
node, the dagman.out file will contain messages noting that the FINAL node is skipped when parsing the relevant
commands.)
The ALL_NODES option is case-insensitive.
It is important to note that the ALL_NODES option does not apply across splices and sub-DAGs. In other words,
an ALL_NODES option within a splice or sub-DAG will apply only to nodes within that splice or sub-DAG; also, an
ALL_NODES option in a parent DAG will not apply to any splices or sub-DAGs referenced by the parent DAG.
The ALL_NODES option does work in combination with the INCLUDE command. In other words, a command
within an included file that uses the ALL_NODES option will apply to all nodes in the including DAG (again, except
any FINAL node).
As of version 8.5.8, the ALL_NODES option cannot be used when multiple DAG files are specified on the condor_submit_dag command line. Hopefully this limitation will be fixed in a future release.
When multiple commands (whether using the ALL_NODES option or not) set a given property of a DAG node, the
last relevant command overrides earlier commands, as shown in the following examples:
For example, in this DAG:
JOB A node.sub
VARS A name="A"
VARS ALL_NODES name="X"
HTCondor Version 8.6.4 Manual
2.10.10. The Rescue DAG
123
the value of name for node A will be "X".
In this DAG:
JOB A node.sub
VARS A name="A"
VARS ALL_NODES name="X"
VARS A name="foo"
the value of name for node A will be "foo".
Here is an example DAG using the ALL_NODES option:
# File:
JOB
JOB
JOB
all_ex.dag
A node.sub
B node.sub
C node.sub
SCRIPT PRE ALL_NODES my_script $JOB
VARS ALL_NODES name="$(JOB)"
# This overrides the above VARS command for node B.
VARS B name="nodeB"
RETRY all_nodes 3
2.10.10 The Rescue DAG
Any time a DAG exits unsuccessfully, DAGMan generates a Rescue DAG. The Rescue DAG records the state of the
DAG, with information such as which nodes completed successfully, and the Rescue DAG will be used when the DAG
is again submitted. With the Rescue DAG, nodes that have already successfully completed are not re-run.
There are a variety of circumstances under which a Rescue DAG is generated. If a node in the DAG fails, the DAG
does not exit immediately; the remainder of the DAG is continued until no more forward progress can be made based
on the DAG’s dependencies. At this point, DAGMan produces the Rescue DAG and exits. A Rescue DAG is produced
on Unix platforms if the condor_dagman job itself is removed with condor_rm. On Windows, a Rescue DAG is not
generated in this situation, but re-submitting the original DAG will invoke a lower-level recovery functionality, and it
will produce similar behavior to using a Rescue DAG. A Rescue DAG is produced when a node sets and triggers an
ABORT-DAG-ON event with a non-zero return value. A zero return value constitutes successful DAG completion, and
therefore a Rescue DAG is not generated.
By default, if a Rescue DAG exists, it will be used when the DAG is submitted specifying the original DAG
input file. If more than one Rescue DAG exists, the newest one will be used. By using the Rescue DAG, DAGMan
will avoid re-running nodes that completed successfully in the previous run. Note that passing the -force option
HTCondor Version 8.6.4 Manual
2.10.10. The Rescue DAG
124
to condor_submit_dag or condor_dagman will cause condor_dagman to not use any existing rescue DAG. This
means that previously-completed node jobs will be re-run.
The granularity defining success or failure in the Rescue DAG is the node. For a node that fails, all parts of the
node will be re-run, even if some parts were successful the first time. For example, if a node’s PRE script succeeds,
but then the node’s HTCondor job cluster fails, the entire node, including the PRE script, will be re-run. A job cluster
may result in the submission of multiple HTCondor jobs. If one of the jobs within the cluster fails, the node fails.
Therefore, the Rescue DAG will re-run the entire node, implying the submission of the entire cluster of jobs, not just
the one(s) that failed.
Statistics about the failed DAG execution are presented as comments at the beginning of the Rescue DAG input
file.
Rescue DAG Naming
The file name of the Rescue DAG is obtained by appending the string .rescue<XXX> to the original DAG
input file name. Values for <XXX> start at 001 and continue to 002, 003, and beyond. The configuration variable
DAGMAN_MAX_RESCUE_NUM sets a maximum value for <XXX>; see section 3.5.24 for the complete definition of
this configuration variable. If you hit the DAGMAN_MAX_RESCUE_NUM limit, the last Rescue DAG file is overwritten
if the DAG fails again.
If a Rescue DAG exists when the original DAG is re-submitted, the Rescue DAG with the largest magnitude value
for <XXX> will be used, and its usage is implied.
Example
Here is an example showing file naming and DAG submission for the case of a failed DAG. The initial DAG is
submitted with
condor_submit_dag
my.dag
A failure of this DAG results in the Rescue DAG named my.dag.rescue001. The DAG is resubmitted using the
same command:
condor_submit_dag
my.dag
This resubmission of the DAG uses the Rescue DAG file my.dag.rescue001, because it exists. Failure of this
Rescue DAG results in another Rescue DAG called my.dag.rescue002. If the DAG is again submitted, using the
same command as with the first two submissions, but not repeated here, then this third submission uses the Rescue
DAG file my.dag.rescue002, because it exists, and because the value 002 is larger in magnitude than 001.
Backtracking to an Older Rescue DAG
HTCondor Version 8.6.4 Manual
2.10.10. The Rescue DAG
125
To explicitly specify a particular Rescue DAG, use the optional command-line argument -dorescuefrom with condor_submit_dag. Note that this will have the side effect of renaming existing Rescue DAG files with larger magnitude
values of <XXX>. Each renamed file has its existing name appended with the string .old. For example, assume that
my.dag has failed 4 times, resulting in the Rescue DAGs named my.dag.rescue001, my.dag.rescue002,
my.dag.rescue003, and my.dag.rescue004. A decision is made to re-run using my.dag.rescue002.
The submit command is
condor_submit_dag
-dorescuefrom 2
my.dag
The DAG specified by the DAG input file my.dag.rescue002 is submitted. And, the existing Rescue
DAG my.dag.rescue003 is renamed to be my.dag.rescue003.old, while the existing Rescue DAG
my.dag.rescue004 is renamed to be my.dag.rescue004.old.
Special Cases
Note that if multiple DAG input files are specified on the condor_submit_dag command line, a single Rescue DAG
encompassing all of the input DAGs is generated. A DAG file containing splices also produces a single Rescue DAG
file. On the other hand, a DAG containing sub-DAGs will produce a separate Rescue DAG for each sub-DAG that is
queued (and for the top-level DAG).
If the Rescue DAG file is generated before all retries of a node are completed, then the Rescue DAG file will
also contain Retry entries. The number of retries will be set to the appropriate remaining number of retries. The
configuration variable DAGMAN_RESET_RETRIES_UPON_RESCUE, section 3.5.24, controls whether or not node
retries are reset in a Rescue DAG.
Partial versus Full Rescue DAGs
As of HTCondor version 7.7.2, the Rescue DAG file is a partial DAG file, not a complete DAG input file as in the
past.
A partial Rescue DAG file contains only information about which nodes are done, and the number of retries
remaining for nodes with retries. It does not contain information such as the actual DAG structure and the specification
of the submit description file for each node job. Partial Rescue DAGs are automatically parsed in combination with
the original DAG input file, which contains information about the DAG structure. This updated implementation means
that a change in the original DAG input file, such as specifying a different submit description file for a node job, will
take effect when running the partial Rescue DAG. In other words, you can fix mistakes in the original DAG file while
still gaining the benefit of using the Rescue DAG.
To use a partial Rescue DAG, you must re-run condor_submit_dag on the original DAG file, not the Rescue DAG
file.
Note that the existence of a DONE specification in a partial Rescue DAG for a node that no longer exists in the
original DAG input file is a warning, as opposed to an error, unless the DAGMAN_USE_STRICT configuration variable
is set to a value of 1 or higher (which is now the default). Comment out the line with DONE in the partial Rescue
DAG file to avoid a warning or error.
HTCondor Version 8.6.4 Manual
2.10.11. DAG Recovery
126
The previous (prior to version 7.7.2) behavior of producing full DAG input file as the Rescue DAG is obtained by
setting the configuration variable DAGMAN_WRITE_PARTIAL_RESCUE to the non-default value of False. Note
that the option to generate full Rescue DAGs is likely to disappear some time during the 8.3 series.
To run a full Rescue DAG, either one left over from an older version of DAGMan, or one produced by setting
DAGMAN_WRITE_PARTIAL_RESCUE to False, directly specify the full Rescue DAG file on the command line
instead of the original DAG file. For example:
condor_submit_dag my.dag.rescue002
Attempting to re-submit the original DAG file, if the Rescue DAG file is a complete DAG, will result in a parse
failure.
Rescue DAG Generated When There Are Parse Errors
Starting in HTCondor version 7.5.5, passing the -DumpRescue option to either condor_dagman or condor_submit_dag causes condor_dagman to output a Rescue DAG file, even if the parsing of a DAG input file fails. In
this parse failure case, condor_dagman produces a specially named Rescue DAG containing whatever it had successfully parsed up until the point of the parse error. This Rescue DAG may be useful in debugging parse errors in complex
DAGs, especially ones using splices. This incomplete Rescue DAG is not meant to be used when resubmitting a failed
DAG. Note that this incomplete Rescue DAG generated by the -DumpRescue option is a full DAG input file, as produced by versions of HTCondor prior to HTCondor version 7.7.2. It is not a partial Rescue DAG file, regardless of the
value of the configuration variable DAGMAN_WRITE_PARTIAL_RESCUE.
To avoid confusion between this incomplete Rescue DAG generated in the case of a parse failure and a usable Rescue DAG, a different name is given to the incomplete Rescue DAG. The name appends the string .parse_failed
to the original DAG input file name. Therefore, if the submission of a DAG with
condor_submit_dag
my.dag
has a parse failure, the resulting incomplete Rescue DAG will be named my.dag.parse_failed.
To further prevent one of these incomplete Rescue DAG files from being used, a line within the file contains the
single command REJECT. This causes condor_dagman to reject the DAG, if used as a DAG input file. This is done
because the incomplete Rescue DAG may be a syntactically correct DAG input file. It will be incomplete relative to
the original DAG, such that if the incomplete Rescue DAG could be run, it could erroneously be perceived as having
successfully executed the desired workflow, when, in fact, it did not.
2.10.11 DAG Recovery
DAG recovery restores the state of a DAG upon resubmission. Recovery is accomplished by reading the .nodes.log
file that is used to enforce the dependencies of the DAG. The DAG can then continue towards completion.
HTCondor Version 8.6.4 Manual
2.10.12. Visualizing DAGs with dot
127
Recovery is different than a Rescue DAG. Recovery is appropriate when no Rescue DAG has been created. There
will be no Rescue DAG if the machine running the condor_dagman job crashes, or if the condor_schedd daemon
crashes, or if the condor_dagman job crashes, or if the condor_dagman job is placed on hold.
Much of the time, when a not-completed DAG is re-submitted, it will automatically be placed into recovery mode
due to the existence and contents of a lock file created as the DAG is first run. In recovery mode, the .nodes.log is
used to identify nodes that have completed and should not be re-submitted.
DAGMan can be told to work in recovery mode by including the -DoRecovery option on the command line, as in
the example
condor_submit_dag diamond.dag -DoRecovery
where diamond.dag is the name of the DAG input file.
When debugging a DAG in which something has gone wrong, a first determination is whether a resubmission
will use a Rescue DAG or benefit from recovery. The existence of a Rescue DAG means that recovery would be
inappropriate. A Rescue DAG is has a file name ending in .rescue<XXX>, where <XXX> is replaced by a 3-digit
number.
Determine if a DAG ever completed (independent of whether it was successful or not) by looking at the last lines
of the .dagman.out file. If there is a line similar to
(condor_DAGMAN) pid 445 EXITING WITH STATUS 0
then the DAG completed. This line explains that the condor_dagman job finished normally. If there is no line similar
to this at the end of the .dagman.out file, and output from condor_q shows that the condor_dagman job for the
DAG being debugged is not in the queue, then recovery is indicated.
2.10.12 Visualizing DAGs with dot
It can be helpful to see a picture of a DAG. DAGMan can assist you in visualizing a DAG by creating the input
files used by the AT&T Research Labs graphviz package. dot is a program within this package, available from
http://www.graphviz.org/, and it is used to draw pictures of DAGs.
DAGMan produces one or more dot files as the result of an extra line in a DAG input file. The line appears as
DOT dag.dot
This creates a file called dag.dot. which contains a specification of the DAG before any jobs within the DAG
are submitted to HTCondor. The dag.dot file is used to create a visualization of the DAG by using this file as input
to dot. This example creates a Postscript file, with a visualization of the DAG:
dot -Tps dag.dot -o dag.ps
HTCondor Version 8.6.4 Manual
2.10.13. Capturing the Status of Nodes in a File
Within the DAG input file, the DOT command can take several optional parameters:
• UPDATE This will update the dot file every time a significant update happens.
• DONT-UPDATE Creates a single dot file, when the DAGMan begins executing. This is the default if the
parameter UPDATE is not used.
• OVERWRITE Overwrites the dot file each time it is created. This is the default, unless DONT-OVERWRITE
is specified.
• DONT-OVERWRITE Used to create multiple dot files, instead of overwriting the single one specified. To
create file names, DAGMan uses the name of the file concatenated with a period and an integer. For example,
the DAG input file line
DOT dag.dot DONT-OVERWRITE
causes files dag.dot.0, dag.dot.1, dag.dot.2, etc. to be created. This option is most useful when
combined with the UPDATE option to visualize the history of the DAG after it has finished executing.
• INCLUDE path-to-filename Includes the contents of a file given by path-to-filename in the file produced
by the DOT command. The include file contents are always placed after the line of the form label=. This
may be useful if further editing of the created files would be necessary, perhaps because you are automatically
visualizing the DAG as it progresses.
If conflicting parameters are used in a DOT command, the last one listed is used.
2.10.13 Capturing the Status of Nodes in a File
DAGMan can capture the status of the overall DAG and all DAG nodes in a node status file, such that the user or a
script can monitor this status. This file is periodically rewritten while the DAG runs. To enable this feature, the DAG
input file must contain a line with the NODE_STATUS_FILE command.
The syntax for a NODE_STATUS_FILE command is
NODE_STATUS_FILE statusFileName [minimumUpdateTime] [ALWAYS-UPDATE]
The status file is written on the machine on which the DAG is submitted; its location is given by statusFileName,
and it may be a full path and file name.
The optional minimumUpdateTime specifies the minimum number of seconds that must elapse between updates to
the node status file. This setting exists to avoid having DAGMan spend too much time writing the node status file for
very large DAGs. If no value is specified, this value defaults to 60 seconds (as of version 8.5.8; previously, it defaulted
to 0). The node status file can be updated at most once per DAGMAN_USER_LOG_SCAN_INTERVAL, as defined at
section 3.5.24, no matter how small the minimumUpdateTime value. Also, the node status file will be updated when
the DAG finishes, whether successfully or not, even if minimumUpdateTime seconds have not elapsed since the last
update.
HTCondor Version 8.6.4 Manual
128
2.10.13. Capturing the Status of Nodes in a File
Normally, the node status file is only updated if the status of some nodes has changed since the last time the file
was written. However, the optional ALWAYS-UPDATE keyword specifies that the node status file should be updated
every time the minimum update time (and DAGMAN_USER_LOG_SCAN_INTERVAL), has passed, even if no nodes
have changed status since the last time the file was updated. (The file will change slightly, because timestamps will
be updated.) For performance reasons, large DAGs with approximately 10,000 or more nodes are poor candidates for
using the ALWAYS-UPDATE option.
As an example, if the DAG input file contains the line
NODE_STATUS_FILE my.dag.status 30
the file my.dag.status will be rewritten at intervals of 30 seconds or more.
This node status file is overwritten each time it is updated. Therefore, it only holds information about the current
status of each node; it does not provide a history of the node status.
NOTE: HTCondor version 8.1.6 changes the format of the node status file.
The node status file is a collection of ClassAds in New ClassAd format. There is one ClassAd for the overall status
of the DAG, one ClassAd for the status of each node, and one ClassAd with the time at which the node status file was
completed as well as the time of the next update.
Here is an example portion of a node status file:
[
Type = "DagStatus";
DagFiles = {
"job_dagman_node_status.dag"
};
Timestamp = 1399674138; /* "Fri May 9 17:22:18 2014" */
DagStatus = 3; /* "STATUS_SUBMITTED ()" */
NodesTotal = 12;
NodesDone = 11;
NodesPre = 0;
NodesQueued = 1;
NodesPost = 0;
NodesReady = 0;
NodesUnready = 0;
NodesFailed = 0;
JobProcsHeld = 0;
JobProcsIdle = 1;
]
[
Type = "NodeStatus";
Node = "A";
NodeStatus = 5; /* "STATUS_DONE" */
StatusDetails = "";
HTCondor Version 8.6.4 Manual
129
2.10.14. A Machine-Readable Event History, the jobstate.log File
RetryCount = 0;
JobProcsQueued = 0;
JobProcsHeld = 0;
]
...
[
Type = "NodeStatus";
Node = "C";
NodeStatus = 3; /* "STATUS_SUBMITTED" */
StatusDetails = "idle";
RetryCount = 0;
JobProcsQueued = 1;
JobProcsHeld = 0;
]
[
Type = "StatusEnd";
EndTime = 1399674138; /* "Fri May 9 17:22:18 2014" */
NextUpdate = 1399674141; /* "Fri May 9 17:22:21 2014" */
]
Possible DagStatus and NodeStatus attribute values are:
• 0 (STATUS_NOT_READY): At least one parent has not yet finished or the node is a FINAL node.
• 1 (STATUS_READY): All parents have finished, but the node is not yet running.
• 2 (STATUS_PRERUN): The node’s PRE script is running.
• 3 (STATUS_SUBMITTED): The node’s HTCondor job(s) are in the queue.
• 4 (STATUS_POSTRUN): The node’s POST script is running.
• 5 (STATUS_DONE): The node has completed successfully.
• 6 (STATUS_ERROR): The node has failed.
A NODE_STATUS_FILE command inside any splice is ignored. If multiple DAG files are specified on the condor_submit_dag command line, and more than one specifies a node status file, the first specification takes precedence.
2.10.14 A Machine-Readable Event History, the jobstate.log File
DAGMan can produce a machine-readable history of events. The jobstate.log file is designed for use by
the Pegasus Workflow Management System, which operates as a layer on top of DAGMan. Pegasus uses the
jobstate.log file to monitor the state of a workflow. The jobstate.log file can used by any automated
tool for the monitoring of workflows.
HTCondor Version 8.6.4 Manual
130
2.10.14. A Machine-Readable Event History, the jobstate.log File
DAGMan produces this file when the command JOBSTATE_LOG is in the DAG input file. The syntax for
JOBSTATE_LOG is
JOBSTATE_LOG JobstateLogFileName
No more than one jobstate.log file can be created by a single instance of condor_dagman. If more than one
jobstate.log file is specified, the first file name specified will take effect, and a warning will be printed in the
dagman.out file when subsequent JOBSTATE_LOG specifications are parsed. Multiple specifications may exist in
the same DAG file, within splices, or within multiple, independent DAGs run with a single condor_dagman instance.
The jobstate.log file can be considered a filtered version of the dagman.out file, in a machine-readable
format. It contains the actual node job events that from condor_dagman, plus some additional meta-events.
The jobstate.log file is different from the node status file, in that the jobstate.log file is appended to,
rather than being overwritten as the DAG runs. Therefore, it contains a history of the DAG, rather than a snapshot of
the current state of the DAG.
There are 5 line types in the jobstate.log file. Each line begins with a Unix timestamp in the form of seconds
since the Epoch. Fields within each line are separated by a single space character.
DAGMan start This line identifies the condor_dagman job. The formatting of the line is
timestamp INTERNAL *** DAGMAN_STARTED dagmanCondorID ***
The dagmanCondorID field is the condor_dagman job’s ClusterId attribute, a period, and the ProcId
attribute.
DAGMan exit This line identifies the completion of the condor_dagman job. The formatting of the line is
timestamp INTERNAL *** DAGMAN_FINISHED exitCode ***
The exitCode field is value the condor_dagman job returns upon exit.
Recovery started If the condor_dagman job goes into recovery mode, this meta-event is printed. During recovery
mode, events will only be printed in the file if they were not already printed before recovery mode started. The
formatting of the line is
timestamp INTERNAL *** RECOVERY_STARTED ***
Recovery finished or Recovery failure At the end of recovery mode, either a RECOVERY_FINISHED or RECOVERY_FAILURE meta-event will be printed, as appropriate.
The formatting of the line is
timestamp INTERNAL *** RECOVERY_FINISHED ***
or
timestamp INTERNAL *** RECOVERY_FAILURE ***
Normal This line is used for all other event and meta-event types. The formatting of the line is
timestamp JobName eventName condorID jobTag - sequenceNumber
The JobName is the name given to the node job as defined in the DAG input file with the command JOB. It
identifies the node within the DAG.
HTCondor Version 8.6.4 Manual
131
2.10.14. A Machine-Readable Event History, the jobstate.log File
The eventName is one of the many defined event or meta-events given in the lists below.
The condorID field is the job’s ClusterId attribute, a period, and the ProcId attribute. There is no condorID
assigned yet for some meta-events, such as PRE_SCRIPT_STARTED. For these, the dash character (’-’) is
printed.
The jobTag field is defined for the Pegasus workflow manager. Its usage is generalized to be useful to other
workflow managers. Pegasus-managed jobs add a line of the following form to their HTCondor submit description file:
+pegasus_site = "local"
This defines the string local as the jobTag field.
Generalized usage adds a set of 2 commands to the HTCondor submit description file to define a string as the
jobTag field:
+job_tag_name = "+job_tag_value"
+job_tag_value = "viz"
This defines the string viz as the jobTag field. Without any of these added lines within the HTCondor submit
description file, the dash character (’-’) is printed for the jobTag field.
The sequenceNumber is a monotonically-increasing number that starts at one. It is associated with each attempt
at running a node. If a node is retried, it gets a new sequence number; a submit failure does not result in a new
sequence number. When a Rescue DAG is run, the sequence numbers pick up from where they left off within
the previous attempt at running the DAG. Note that this only applies if the Rescue DAG is run automatically or
with the -dorescuefrom command-line option.
Here is an example of a very simple Pegasus jobstate.log file, assuming the example jobTag field of local:
1292620511
1292620523
1292620523
1292620525
1292620525
1292620526
1292620526
1292620526
1292620531
1292620531
1292620535
INTERNAL *** DAGMAN_STARTED 4972.0 ***
NodeA PRE_SCRIPT_STARTED - local - 1
NodeA PRE_SCRIPT_SUCCESS - local - 1
NodeA SUBMIT 4973.0 local - 1
NodeA EXECUTE 4973.0 local - 1
NodeA JOB_TERMINATED 4973.0 local - 1
NodeA JOB_SUCCESS 0 local - 1
NodeA POST_SCRIPT_STARTED 4973.0 local - 1
NodeA POST_SCRIPT_TERMINATED 4973.0 local - 1
NodeA POST_SCRIPT_SUCCESS 4973.0 local - 1
INTERNAL *** DAGMAN_FINISHED 0 ***
Events defining the eventName field
• SUBMIT
• EXECUTE
• EXECUTABLE_ERROR
• CHECKPOINTED
HTCondor Version 8.6.4 Manual
132
2.10.14. A Machine-Readable Event History, the jobstate.log File
• JOB_EVICTED
• JOB_TERMINATED
• IMAGE_SIZE
• SHADOW_EXCEPTION
• GENERIC
• JOB_ABORTED
• JOB_SUSPENDED
• JOB_UNSUSPENDED
• JOB_HELD
• JOB_RELEASED
• NODE_EXECUTE
• NODE_TERMINATED
• POST_SCRIPT_TERMINATED
• GLOBUS_SUBMIT
• GLOBUS_SUBMIT_FAILED
• GLOBUS_RESOURCE_UP
• GLOBUS_RESOURCE_DOWN
• REMOTE_ERROR
• JOB_DISCONNECTED
• JOB_RECONNECTED
• JOB_RECONNECT_FAILED
• GRID_RESOURCE_UP
• GRID_RESOURCE_DOWN
• GRID_SUBMIT
• JOB_AD_INFORMATION
• JOB_STATUS_UNKNOWN
• JOB_STATUS_KNOWN
• JOB_STAGE_IN
• JOB_STAGE_OUT
Meta-Events defining the eventName field
• SUBMIT_FAILURE
• JOB_SUCCESS
• JOB_FAILURE
• PRE_SCRIPT_STARTED
• PRE_SCRIPT_SUCCESS
• PRE_SCRIPT_FAILURE
HTCondor Version 8.6.4 Manual
133
2.10.15. Status Information for the DAG in a ClassAd
• POST_SCRIPT_STARTED
• POST_SCRIPT_SUCCESS
• POST_SCRIPT_FAILURE
• DAGMAN_STARTED
• DAGMAN_FINISHED
• RECOVERY_STARTED
• RECOVERY_FINISHED
• RECOVERY_FAILURE
2.10.15 Status Information for the DAG in a ClassAd
The condor_dagman job places information about the status of the DAG into its own job ClassAd. The attributes are
fully described at section 12. The attributes are
• DAG_NodesTotal
• DAG_NodesDone
• DAG_NodesPrerun
• DAG_NodesQueued
• DAG_NodesPostrun
• DAG_NodesReady
• DAG_NodesFailed
• DAG_NodesUnready
• DAG_Status
• DAG_InRecovery
Note that most of this information is also available in the dagman.out file as described in section 2.10.7.
2.10.16 Utilizing the Power of DAGMan for Large Numbers of Jobs
Using DAGMan is recommended when submitting large numbers of jobs. The recommendation holds whether the
jobs are represented by a DAG due to dependencies, or all the jobs are independent of each other, such as they might
be in a parameter sweep. DAGMan offers:
Throttling Throttling limits the number of submitted jobs at any point in time.
HTCondor Version 8.6.4 Manual
134
2.10.16. Utilizing the Power of DAGMan for Large Numbers of Jobs
Retry of jobs that fail This is a useful tool when an intermittent error may cause a job to fail or may cause a job to
fail to run to completion when attempted at one point in time, but not at another point in time. The conditions
under which retry occurs are user-defined. In addition, the administrative support that facilitates the rerunning
of only those jobs that fail is automatically generated.
Scripts associated with node jobs PRE and POST scripts run on the submit host before and/or after the execution of
specified node jobs.
Each of these capabilities is described in detail within this manual section about DAGMan. To make effective use
of DAGMan, there is no way around reading the appropriate subsections.
To run DAGMan with large numbers of independent jobs, there are generally two ways of organizing and specifying the files that control the jobs. Both ways presume that programs or scripts will generate needed files, because
the file contents are either large and repetitive, or because there are a large number of similar files to be generated
representing the large numbers of jobs. The two file types needed are the DAG input file and the submit description
file(s) for the HTCondor jobs represented. Each of the two ways is presented separately:
A unique submit description file for each of the many jobs. A single DAG input file lists each of the jobs and specifies a distinct submit description file for each job. The DAG input file is simple to generate, as it chooses an
identifier for each job and names the submit description file. For example, the simplest DAG input file for a set
of 1000 independent jobs, as might be part of a parameter sweep, appears as
# file sweep.dag
JOB job0 job0.submit
JOB job1 job1.submit
JOB job2 job2.submit
.
.
.
JOB job999 job999.submit
There are 1000 submit description files, with a unique one for each of the job<N> jobs. Assuming that all files
associated with this set of jobs are in the same directory, and that files continue the same naming and numbering
scheme, the submit description file for job6.submit might appear as
# file job6.submit
universe = vanilla
executable = /path/to/executable
log = job6.log
input = job6.in
output = job6.out
arguments = "-file job6.out"
queue
Submission of the entire set of jobs uses the command line
condor_submit_dag sweep.dag
HTCondor Version 8.6.4 Manual
135
2.10.16. Utilizing the Power of DAGMan for Large Numbers of Jobs
A benefit to having unique submit description files for each of the jobs is that they are available if one of the
jobs needs to be submitted individually. A drawback to having unique submit description files for each of the
jobs is that there are lots of submit description files.
Single submit description file. A single HTCondor submit description file might be used for all the many jobs of the
parameter sweep. To distinguish the jobs and their associated distinct input and output files, the DAG input file
assigns a unique identifier with the VARS command.
# file sweep.dag
JOB job0 common.submit
VARS job0 runnumber="0"
JOB job1 common.submit
VARS job1 runnumber="1"
JOB job2 common.submit
VARS job2 runnumber="2"
.
.
.
JOB job999 common.submit
VARS job999 runnumber="999"
The single submit description file for all these jobs utilizes the runnumber variable value in its identification
of the job’s files. This submit description file might appear as
# file common.submit
universe = vanilla
executable = /path/to/executable
log = wholeDAG.log
input = job$(runnumber).in
output = job$(runnumber).out
arguments = "-$(runnumber)"
queue
The job with runnumber="8" expects to find its input file job8.in in the single, common directory, and it
sends its output to job8.out. The single log for all job events of the entire DAG is wholeDAG.log. Using
one file for the entire DAG meets the limitation that no macro substitution may be specified for the job log file,
and it is likely more efficient as well. This node’s executable is invoked with
/path/to/executable -8
These examples work well with respect to file naming and file location when there are less than several thousand
jobs submitted as part of a DAG. The large numbers of files per directory becomes an issue when there are greater
than several thousand jobs submitted as part of a DAG. In this case, consider a more hierarchical structure for the files
instead of a single directory. Introduce a separate directory for each run. For example, if there were 10,000 jobs, there
would be 10,000 directories, one for each of these jobs. The directories are presumed to be generated and populated
by programs or scripts that, like the previous examples, utilize a run number. Each of these directories named utilizing
the run number will be used for the input, output, and log files for one of the many jobs.
HTCondor Version 8.6.4 Manual
136
2.10.17. Workflow Metrics
137
As an example, for this set of 10,000 jobs and directories, assume that there is a run number of 600. The directory
will be named dir600, and it will hold the 3 files called in, out, and log, representing the input, output, and
HTCondor job log files associated with run number 600.
The DAG input file sets a variable representing the run number, as in the previous example:
# file biggersweep.dag
JOB job0 bigger.submit
VARS job0 runnumber="0"
JOB job1 bigger.submit
VARS job1 runnumber="1"
JOB job2 bigger.submit
VARS job2 runnumber="2"
.
.
.
JOB job9999 bigger.submit
VARS job9999 runnumber="9999"
A single HTCondor submit description file may be written. It resides in the same directory as the DAG input file.
# file bigger.submit
universe = vanilla
executable = /path/to/executable
log = log
input = in
output = out
arguments = "-$(runnumber)"
initialdir = dir$(runnumber)
queue
One item to care about with this set up is the underlying file system for the pool. The transfer of files (or not) when
using initialdir differs based upon the job universe and whether or not there is a shared file system. See section 11
for the details on the submit command initialdir.
Submission of this set of jobs is no different than the previous examples. With the current working directory the
same as the one containing the submit description file, the DAG input file, and the subdirectories,
condor_submit_dag biggersweep.dag
2.10.17 Workflow Metrics
condor_dagman may report workflow metrics to one or more HTTP servers. This capability is currently only used for
workflows run under Pegasus. The reporting is disabled by setting the CONDOR_DEVELOPERS configuration variable
HTCondor Version 8.6.4 Manual
2.10.17. Workflow Metrics
138
to NONE, or by setting the PEGASUS_METRICS environment variable to any value other than True (case-insensitive)
or 1. The dagman.out file will indicate whether or not metrics were reported.
For every DAG, a metrics file is created independent of the reporting of those metrics. This metrics file is named
<dag_file_name>.metrics, where <dag_file_name> is the name of the DAG input file. In a workflow
with nested DAGs, each nested DAG will create its own metrics file.
Here is an example metrics output file:
{
"client":"condor_dagman",
"version":"8.1.0",
"planner":"/lfs1/devel/Pegasus/pegasus/bin/pegasus-plan",
"planner_version":"4.3.0cvs",
"type":"metrics",
"wf_uuid":"htcondor-test-job_dagman_metrics-A-subdag",
"root_wf_uuid":"htcondor-test-job_dagman_metrics-A",
"start_time":1375313459.603,
"end_time":1375313491.498,
"duration":31.895,
"exitcode":1,
"dagman_id":"26",
"parent_dagman_id":"11",
"rescue_dag_number":0,
"jobs":4,
"jobs_failed":1,
"jobs_succeeded":3,
"dag_jobs":0,
"dag_jobs_failed":0,
"dag_jobs_succeeded":0,
"total_jobs":4,
"total_jobs_run":4,
"total_job_time":0.000,
"dag_status":2
}
Here is an explanation of each of the items in the file:
• client: the name of the client workflow software; in the example, it is "condor_dagman"
• version: the version of the client workflow software
• planner: the workflow planner, as read from the braindump.txt file
• planner_version: the planner software version, as read from the braindump.txt file
• type: the type of data, "metrics"
HTCondor Version 8.6.4 Manual
2.10.17. Workflow Metrics
139
• wf_uuid: the workflow ID, generated by pegasus-plan, as read from the braindump.txt file
• root_wf_uuid: the root workflow ID, which is relevant for nested workflows. It is generated by pegasusplan, as read from the braindump.txt file.
• start_time: the start time of the client, in epoch seconds, with millisecond precision
• end_time: the end time of the client, in epoch seconds, with millisecond precision
• duration: the duration of the client, in seconds, with millisecond precision
• exitcode: the condor_dagman exit code
• dagman_id: the value of the ClusterId attribute of the condor_dagman instance
• parent_dagman_id: the value of the ClusterId attribute of the parent condor_dagman instance of this
DAG; empty if this DAG is not a SUBDAG
• rescue_dag_number: the number of the Rescue DAG being run, or 0 if not running a Rescue DAG
• jobs: the number of nodes in the DAG input file, not including SUBDAG nodes
• jobs_failed: the number of failed nodes in the workflow, not including SUBDAG nodes
• jobs_succeeded: the number of successful nodes in the workflow, not including SUBDAG nodes; this
includes jobs that succeeded after retries
• dag_jobs: the number of SUBDAG nodes in the DAG input file
• dag_jobs_failed: the number of SUBDAG nodes that failed
• dag_jobs_succeeded: the number of SUBDAG nodes that succeeded
• total_jobs: the total number of jobs in the DAG input file
• total_jobs_run: the total number of nodes executed in a DAG. It should be equal to
jobs_succeeded + jobs_failed + dag_jobs_succeeded + dag_jobs_failed
• total_job_time: the sum of the time between the first execute event and the terminated event for all jobs
that are not SUBDAGs
• dag_status: the final status of the DAG, with values
– 0: OK
– 1: error; an error condition different than those listed here
– 2: one or more nodes in the DAG have failed
– 3: the DAG has been aborted by an ABORT-DAG-ON specification
– 4: removed; the DAG has been removed by condor_rm
– 5: a cycle was found in the DAG
– 6: the DAG has been halted; see section 2.10.8 for an explanation of halting a DAG
HTCondor Version 8.6.4 Manual
2.10.18. DAGMan and Accounting Groups
140
Note that any dag_status other than 0 corresponds to a non-zero exit code.
The braindump.txt file is generated by pegasus-plan; the name of the braindump.txt file is specified with the PEGASUS_BRAINDUMP_FILE environment variable. If not specified, the file name defaults to
braindump.txt, and it is placed in the current directory.
Note that the total_job_time value is always zero, because the calculation of that value has not yet been
implemented.
If a DAG succeeds, but the metrics reporting fails, the DAG is still considered successful.
The metrics are reported only at the end of a DAG run. This includes reporting the metrics if the condor_dagman
job is removed, or if the DAG drains from the queue because of being halted by a halt file.
The metrics are reported by the condor_dagman_metrics_reporter executable as described in the manual page at
794.
2.10.18 DAGMan and Accounting Groups
As of version 8.5.6, condor_dagman propagates accounting_group and accounting_group_user values specified for
condor_dagman itself to all jobs within the DAG (including sub-DAGs).
The accounting_group and accounting_group_user values can be specified using the -append flag to condor_submit_dag, for example:
condor_submit_dag -append accounting_group=group_physics -append accounting_group_user=a
See section 3.6.7 for a discussion of group accounting and section 3.6.8 for a discussion of accounting groups with
hierarchical group quotas.
2.11 Virtual Machine Applications
The vm universe facilitates an HTCondor job that matches and then lands a disk image on an execute machine within
an HTCondor pool. This disk image is intended to be a virtual machine. In this manner, the virtual machine is the job
to be executed.
This section describes this type of HTCondor job. See section 3.5.26 for details of configuration variables.
2.11.1 The Submit Description File
Different than all other universe jobs, the vm universe job specifies a disk image, not an executable. Therefore, the
submit commands input, output, and error do not apply. If specified, condor_submit rejects the job with an error.
HTCondor Version 8.6.4 Manual
2.11.1. The Submit Description File
141
The executable command changes definition within a vm universe job. It no longer specifies an executable file, but
instead provides a string that identifies the job for tools such as condor_q. Other commands specific to the type of
virtual machine software identify the disk image.
VMware, Xen, and KVM virtual machine software are supported. As these differ from each other, the submit
description file specifies one of
vm_type = vmware
or
vm_type = xen
or
vm_type = kvm
The job is required to specify its memory needs for the disk image with vm_memory, which is given in Mbytes.
HTCondor uses this number to assure a match with a machine that can provide the needed memory space.
Virtual machine networking is enabled with the command
vm_networking = true
And, when networking is enabled, a definition of vm_networking_type as bridge matches the job only with a machine
that is configured to use bridge networking. A definition of vm_networking_type as nat matches the job only with a
machine that is configured to use NAT networking. When no definition of vm_networking_type is given, HTCondor
may match the job with a machine that enables networking, and further, the choice of bridge or NAT networking is
determined by the machine’s configuration.
Modified disk images are transferred back to the machine from which the job was submitted as the vm universe job
completes. Job completion for a vm universe job occurs when the virtual machine is shut down, and HTCondor notices
(as the result of a periodic check on the state of the virtual machine). Should the job not want any files transferred back
(modified or not), for example because the job explicitly transferred its own files, the submit command to prevent the
transfer is
vm_no_output_vm = true
The required disk image must be identified for a virtual machine. This vm_disk command specifies a list of
comma-separated files. Each disk file is specified by colon-separated fields. The first field is the path and file name
of the disk file. The second field specifies the device. The third field specifies permissions, and the optional fourth
specifies the format. Here is an example that identifies a single file:
vm_disk = swap.img:sda2:w:raw
HTCondor Version 8.6.4 Manual
2.11.1. The Submit Description File
142
If HTCondor will be transferring the disk file, then the file name given in vm_disk should not contain any path
information. Otherwise, the full path to the file should be given.
Setting values in the submit description file for some commands have consequences for the virtual machine description file. These commands are
• vm_memory
• vm_macaddr
• vm_networking
• vm_networking_type
• vm_disk
For VMware virtual machines, setting values for these commands causes HTCondor to modify the .vmx file, overwriting existing values. For KVM and Xen virtual machines, HTCondor uses these values when it produces the description
file.
For Xen and KVM jobs, if any files need to be transferred from the submit machine to the machine where the vm
universe job will execute, HTCondor must be explicitly told to do so with the standard file transfer attributes:
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = /myxen/diskfile.img,/myxen/swap.img
Any and all needed files that will not accessible directly from the machines where the job may execute must be listed.
Further commands specify information that is specific to the virtual machine type targeted.
VMware-Specific Submit Commands
Specific to VMware, the submit description file command vmware_dir gives the path and directory (on the machine
from which the job is submitted) to where VMware-specific files and applications reside. One example of a VMwarespecific application is the VMDK files, which form a virtual hard drive (disk image) for the virtual machine. VMX
files containing the primary configuration for the virtual machine would also be in this directory.
HTCondor must be told whether or not the contents of the vmware_dir directory must be transferred to
the machine where the job is to be executed. This required information is given with the submit command
vmware_should_transfer_files. With a value of True, HTCondor does transfer the contents of the directory. With
a value of False, HTCondor does not transfer the contents of the directory, and instead presumes that access to this
directory is available through a shared file system.
By default, HTCondor uses a snapshot disk for new and modified files. They may also be utilized for checkpoints. The snapshot disk is initially quite small, growing only as new files are created or files are modified. When
vmware_should_transfer_files is True, a job may specify that a snapshot disk is not to be used with the command
HTCondor Version 8.6.4 Manual
2.11.1. The Submit Description File
143
vmware_snapshot_disk = False
In this case, HTCondor will utilize original disk files in producing checkpoints. Note that condor_submit issues an
error message and does not submit the job if both vmware_should_transfer_files and vmware_snapshot_disk are
False.
Because VMware Player does not support snapshots, machines using VMware Player may only run vm jobs that
set vmware_snapshot_disk to False. These jobs will also set vmware_should_transfer_files to True. A job
using VMware Player will go on hold if it attempts to use a snapshot. The pool administrator should have configured
the pool such that machines will not start jobs they can not run.
Note that if snapshot disks are requested and file transfer is not being used, the vmware_dir setting given
in the submit description file should not contain any symbolic link path components, as described on the
https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToAdminRecipes page under the answer to why VMware
jobs with symbolic links fail.
Here is a sample submit description file for a VMware virtual machine:
universe
executable
log
vm_type
vm_memory
vmware_dir
vmware_should_transfer_files
queue
=
=
=
=
=
=
=
vm
vmware_sample_job
simple.vm.log.txt
vmware
64
C:\condor-test
True
This sample uses the vmware_dir command to identify the location of the disk image to be executed as an HTCondor
job. The contents of this directory are transferred to the machine assigned to execute the HTCondor job.
Xen-Specific Submit Commands
A Xen vm universe job requires specification of the guest kernel. The xen_kernel command accomplishes this,
utilizing one of the following definitions.
1. xen_kernel = included implies that the kernel is to be found in disk image given by the definition of the
single file specified in vm_disk.
2. xen_kernel = path-to-kernel gives the file name of the required kernel. If this kernel must be transferred to machine on which the vm universe job will execute, it must also be included in the transfer_input_files
command.
This form of the xen_kernel command also requires further definition of the xen_root command. xen_root
defines the device containing files needed by root.
HTCondor Version 8.6.4 Manual
2.11.2. Checkpoints
144
2.11.2 Checkpoints
Creating a checkpoint is straightforward for a virtual machine, as a checkpoint is a set of files that represent a snapshot
of both disk image and memory. The checkpoint is created and all files are transferred back to the $(SPOOL) directory
on the machine from which the job was submitted. The submit command to create checkpoints is
vm_checkpoint = true
Without this command, no checkpoints are created (by default). With the command, a checkpoint is created any time
the vm universe jobs is evicted from the machine upon which it is executing. This occurs as a result of the machine
configuration indicating that it will no longer execute this job.
vm universe jobs can not use a checkpoint server.
Periodic creation of checkpoints is not supported at this time.
Enabling both networking and checkpointing for a vm universe job can cause networking problems when the job
restarts, particularly if the job migrates to a different machine. condor_submit will normally reject such jobs. To
enable both, then add the command
when_to_transfer_output = ON_EXIT_OR_EVICT
Take care with respect to the use of network connections within the virtual machine and their interaction with
checkpoints. Open network connections at the time of the checkpoint will likely be lost when the checkpoint is
subsequently used to resume execution of the virtual machine. This occurs whether or not the execution resumes on
the same machine or a different one within the HTCondor pool.
2.11.3 Disk Images
VMware on Windows and Linux
Following
the
platform-specific
guest
OS
installation
http://partnerweb.vmware.com/GOSIG/home.html, creates a VMware disk image.
instructions
found
at
Xen and KVM
While the following web page contains instructions specific to Fedora on how to create a virtual guest image, it should
provide a good starting point for other platforms as well.
http://fedoraproject.org/wiki/Virtualization_Quick_Start
HTCondor Version 8.6.4 Manual
2.11.4. Job Completion in the vm Universe
145
2.11.4 Job Completion in the vm Universe
Job completion for a vm universe job occurs when the virtual machine is shut down, and HTCondor notices (as
the result of a periodic check on the state of the virtual machine). This is different from jobs executed under the
environment of other universes.
Shut down of a virtual machine occurs from within the virtual machine environment. A script, executed with the
proper authorization level, is the likely source of the shut down commands.
Under a Windows 2000, Windows XP, or Vista virtual machine, an administrator issues the command
shutdown -s -t 01
Under a Linux virtual machine, the root user executes
/sbin/poweroff
The command /sbin/halt will not completely shut down some Linux distributions, and instead causes the job to
hang.
Since the successful completion of the vm universe job requires the successful shut down of the virtual machine,
it is good advice to try the shut down procedure outside of HTCondor, before a vm universe job is submitted.
2.11.5 Failures to Launch
It is not uncommon for a vm universe job to fail to launch because of a problem with the execute machine. In these
cases, HTCondor will reschedule the job and note, in its user event log (if requested), the reason for the failure and
that the job will be rescheduled. The reason is unlikely to be directly useful to you as an HTCondor user, but may help
your HTCondor administrator understand the problem.
If the VM fails to launch for other reasons, the job will be placed on hold and the reason placed in the job ClassAd’s
HoldReason attribute. The following table may help in understanding such reasons.
VMGAHP_ERR_JOBCLASSAD_NO_VM_MEMORY_PARAM
The attribute JobVMMemory was not set in the job ad sent to the
VM GAHP. HTCondor will usually prevent you from submitting a VM universe job
without JobVMMemory set. Examine your job and verify that JobVMMemory is set.
If it is, please contact your administrator.
VMGAHP_ERR_JOBCLASSAD_NO_VMWARE_VMX_PARAM
The attribute VMPARAM_VMware_Dir was not set in the job ad sent to the
VM GAHP. HTCondor will usually set this attribute when you submit a valid
HTCondor Version 8.6.4 Manual
2.11.5. Failures to Launch
146
VMWare job (it is derived from vmware_dir). If you used condor_submit to
submit this job, contact your administrator. Otherwise, examine your job
and verify that VMPARAM_VMware_Dir is set. If it is, contact your
administrator.
VMGAHP_ERR_JOBCLASSAD_KVM_NO_DISK_PARAM
The attribute VMPARAM_vm_Disk was not set in the job ad sent to the
VM GAHP. HTCondor will usually set this attribute when you submit a valid
KVM job (it is derived from vm_disk). Examine your job and verify that
VMPARAM_vm_Disk is set. If it is, please contact your administrator.
VMGAHP_ERR_JOBCLASSAD_KVM_INVALID_DISK_PARAM
The attribute vm_disk was invalid. Please consult the manual,
or the condor_submit man page, for information about the syntax of
vm_disk. A syntactically correct value may be invalid if the
on-disk permissions of a file specified in it do not match the requested
permissions. Presently, files not transferred to the root of the working
directory must be specified with full paths.
VMGAHP_ERR_JOBCLASSAD_KVM_MISMATCHED_CHECKPOINT
KVM jobs can not presently checkpoint if any of their disk files are not
on a shared filesystem. Files on a shared filesystem must be specified in
vm_disk with full paths.
VMGAHP_ERR_JOBCLASSAD_XEN_NO_KERNEL_PARAM
The attribute VMPARAM_Xen_Kernel was not set in the job ad sent to the
VM GAHP. HTCondor will usually set this attribute when you submit a valid
Xen job (it is derived from xen_kernel). Examine your job and verify that
VMPARAM_Xen_Kernel is set. If it is, please contact your administrator.
VMGAHP_ERR_JOBCLASSAD_MISMATCHED_HARDWARE_VT
Don't use 'vmx' as the name of your kernel image.
change xen_kernel to match.
Pick something else and
VMGAHP_ERR_JOBCLASSAD_XEN_KERNEL_NOT_FOUND
HTCondor could not read from the file specified by xen_kernel.
Check the path and the file's permissions. If it's on a shared filesystem,
you may need to alter your job's requirements expression to ensure the
filesystem's availability.
HTCondor Version 8.6.4 Manual
2.12. Docker Universe Applications
147
VMGAHP_ERR_JOBCLASSAD_XEN_INITRD_NOT_FOUND
HTCondor could not read from the file specified by xen_initrd.
Check the path and the file's permissions. If it's on a shared filesystem,
you may need to alter your job's requirements expression to ensure the
filesystem's availability.
VMGAHP_ERR_JOBCLASSAD_XEN_NO_ROOT_DEVICE_PARAM
The attribute VMPARAM_Xen_Root was not set in the job ad sent to the
VM GAHP. HTCondor will usually set this attribute when you submit a valid
Xen job (it is derived from xen_root). Examine your job and verify that
VMPARAM_Xen_Root is set. If it is, please contact your administrator.
VMGAHP_ERR_JOBCLASSAD_XEN_NO_DISK_PARAM
The attribute VMPARAM_vm_Disk was not set in the job ad sent to the
VM GAHP. HTCondor will usually set this attribute when you submit a valid
Xen job (it is derived from vm_disk). Examine your job and verify that
VMPARAM_vm_Disk is set. If it is, please contact your administrator.
VMGAHP_ERR_JOBCLASSAD_XEN_INVALID_DISK_PARAM
The attribute vm_disk was invalid. Please consult the manual,
or the condor_submit man page, for information about the syntax of
vm_disk. A syntactically correct value may be invalid if the
on-disk permissions of a file specified in it do not match the requested
permissions. Presently, files not transferred to the root of the working
directory must be specified with full paths.
VMGAHP_ERR_JOBCLASSAD_XEN_MISMATCHED_CHECKPOINT
Xen jobs can not presently checkpoint if any of their disk files are not
on a shared filesystem. Files on a shared filesystem must be specified in
vm_disk with full paths.
2.12 Docker Universe Applications
A docker universe job instantiates a Docker container from a Docker image, and HTCondor manages the running
of that container as an HTCondor job, on an execute machine. This running container can then be managed as any
HTCondor job. For example, it can be scheduled, removed, put on hold, or be part of a workflow managed by
DAGMan.
HTCondor Version 8.6.4 Manual
2.12. Docker Universe Applications
148
The docker universe job will only be matched with an execute host that advertises its capability to run docker
universe jobs. When an execute machine with docker support starts, the machine checks to see if the docker command
is available and has the correct settings for HTCondor. Docker support is advertised if available and if it has the correct
settings.
The image from which the container is instantiated is defined by specifying a Docker image with the submit
command docker_image. This image must be pre-staged on a docker hub that the execute machine can access.
After submission, the job is treated much the same way as a vanilla universe job. Details of file transfer are the same
as applied to the vanilla universe. One of the benefits of Docker containers is the file system isolation they provide.
Each container has a distinct file system, from the root on down, and this file system is completely independent of the
file system on the host machine. The container does not share a file system with either the execute host or the submit
host, with the exception of the scratch directory, which is volume mounted to the host, and is the initial working
directory of the job. Optionally, the administrator may configure other directories from the host machine to be volume
mounted, and thus visible inside the container. See the docker section of the administrator’s manual for details.
Therefore, the submit description file should contain the submit command
should_transfer_files = YES
With this command, all input and output files will be transferred as required to and from the scratch directory mounted
as a Docker volume.
If no executable is specified in the submit description file, it is presumed that the Docker container has a default
command to run.
When the job completes, is held, evicted, or is otherwise removed from the machine, the container will be removed.
Here is a complete submit description file for a sample docker universe job:
universe
docker_image
executable
arguments
should_transfer_files
when_to_transfer_output
output
error
log
request_memory
queue 1
=
=
=
=
=
=
=
=
=
=
docker
debian
/bin/cat
/etc/hosts
YES
ON_EXIT
out.$(Process)
err.$(Process)
log.$(Process)
100M
A debian container is the HTCondor job, and it runs the /bin/cat program on the /etc/hosts file before exiting.
HTCondor Version 8.6.4 Manual
2.13. Time Scheduling for Job Execution
149
2.13 Time Scheduling for Job Execution
Jobs may be scheduled to begin execution at a specified time in the future with HTCondor’s job deferral functionality.
All specifications are in a job’s submit description file. Job deferral functionality is expanded to provide for the
periodic execution of a job, known as the CronTab scheduling.
2.13.1 Job Deferral
Job deferral allows the specification of the exact date and time at which a job is to begin executing. HTCondor attempts
to match the job to an execution machine just like any other job, however, the job will wait until the exact time to begin
execution. A user can define the job to allow some flexibility in the execution of jobs that miss their execution time.
Deferred Execution Time
A job’s deferral time is the exact time that HTCondor should attempt to execute the job. The deferral time attribute is
defined as an expression that evaluates to a Unix Epoch timestamp (the number of seconds elapsed since 00:00:00 on
January 1, 1970, Coordinated Universal Time). This is the time that HTCondor will begin to execute the job.
After a job is matched and all of its files have been transferred to an execution machine, HTCondor checks to
see if the job’s ClassAd contains a deferral time. If it does, HTCondor calculates the number of seconds between the
execution machine’s current system time and the job’s deferral time. If the deferral time is in the future, the job waits
to begin execution. While a job waits, its job ClassAd attribute JobStatus indicates the job is in the Running state.
As the deferral time arrives, the job begins to execute. If a job misses its execution time, that is, if the deferral time is
in the past, the job is evicted from the execution machine and put on hold in the queue.
The specification of a deferral time does not interfere with HTCondor’s behavior. For example, if a job is waiting
to begin execution when a condor_hold command is issued, the job is removed from the execution machine and is put
on hold. If a job is waiting to begin execution when a condor_suspend command is issued, the job continues to wait.
When the deferral time arrives, HTCondor begins execution for the job, but immediately suspends it.
The deferral time is specified in the job’s submit description file with the command deferral_time.
Deferral Window
If a job arrives at its execution machine after the deferral time has passed, the job is evicted from the machine and put
on hold in the job queue. This may occur, for example, because the transfer of needed files took too long due to a slow
network connection. A deferral window permits the execution of a job that misses its deferral time by specifying a
window of time within which the job may begin.
The deferral window is the number of seconds after the deferral time, within which the job may begin. When a job
arrives too late, HTCondor calculates the difference in seconds between the execution machine’s current time and the
job’s deferral time. If this difference is less than or equal to the deferral window, the job immediately begins execution.
HTCondor Version 8.6.4 Manual
2.13.1. Job Deferral
150
If this difference is greater than the deferral window, the job is evicted from the execution machine and is put on hold
in the job queue.
The deferral window is specified in the job’s submit description file with the command deferral_window.
Preparation Time
When a job defines a deferral time far in the future and then is matched to an execution machine, potential computation
cycles are lost because the deferred job has claimed the machine, but is not actually executing. Other jobs could
execute during the interval when the job waits for its deferral time. To make use of the wasted time, a job defines
a deferral_prep_time with an integer expression that evaluates to a number of seconds. At this number of seconds
before the deferral time, the job may be matched with a machine.
Usage Examples
Here are examples of how the job deferral time, deferral window, and the preparation time may be used.
The job’s submit description file specifies that the job is to begin execution on January 1st, 2006 at 12:00 pm:
deferral_time = 1136138400
The Unix date program may be used to calculate a Unix epoch time. The syntax of the command to do this depends
on the options provided within that flavor of Unix. In some, it appears as
%
date --date "MM/DD/YYYY HH:MM:SS" +%s
and in others, it appears as
%
date -d "YYYY-MM-DD HH:MM:SS" +%s
MM is a 2-digit month number, DD is a 2-digit day of the month number, and YYYY is a 4-digit year. HH is the
2-digit hour of the day, MM is the 2-digit minute of the hour, and SS are the 2-digit seconds within the minute. The
characters +%s tell the date program to give the output as a Unix epoch time.
The job always waits 60 seconds before beginning execution:
deferral_time = (time() + 60)
In this example, assume that the deferral time is 45 seconds in the past as the job is available. The job begins
execution, because 75 seconds remain in the deferral window:
deferral_window = 120
HTCondor Version 8.6.4 Manual
2.13.2. CronTab Scheduling
151
In this example, a job is scheduled to execute far in the future, on January 1st, 2010 at 12:00 pm. The deferral_prep_time attribute delays the job from being matched until 60 seconds before the job is to begin execution.
deferral_time
= 1262368800
deferral_prep_time = 60
Limitations
There are some limitations to HTCondor’s job deferral feature.
• Job deferral is not available for scheduler universe jobs.
deferral_time produces a fatal error when submitted.
A scheduler universe job defining the
• The time that the job begins to execute is based on the execution machine’s system clock, and not the submission
machine’s system clock. Be mindful of the ramifications when the two clocks show dramatically different times.
• A job’s JobStatus attribute is always in the Running state when job deferral is used. There is currently no
way to distinguish between a job that is executing and a job that is waiting for its deferral time.
2.13.2 CronTab Scheduling
HTCondor’s CronTab scheduling functionality allows jobs to be scheduled to execute periodically. A job’s execution
schedule is defined by commands within the submit description file. The notation is much like that used by the Unix
cron daemon. As such, HTCondor developers are fond of referring to CronTab scheduling as Crondor. The scheduling
of jobs using HTCondor’s CronTab feature calculates and utilizes the DeferralTime ClassAd attribute.
Also, unlike the Unix cron daemon, HTCondor never runs more than one instance of a job at the same time.
The capability for repetitive or periodic execution of the job is enabled by specifying an on_exit_remove command
for the job, such that the job does not leave the queue until desired.
Semantics for CronTab Specification
A job’s execution schedule is defined by a set of specifications within the submit description file. HTCondor uses
these to calculate a DeferralTime for the job.
Table 2.3 lists the submit commands and acceptable values for these commands. At least one of these must be
defined in order for HTCondor to calculate a DeferralTime for the job. Once one CronTab value is defined, the
default for all the others uses all the values in the allowed values ranges.
The day of a job’s execution can be specified by both the cron_day_of_month and the cron_day_of_week attributes. The day will be the logical or of both.
The semantics allow more than one value to be specified by using the * operator, ranges, lists, and steps (strides)
within ranges.
HTCondor Version 8.6.4 Manual
2.13.2. CronTab Scheduling
152
Submit Command
cron_minute
cron_hour
cron_day_of_month
cron_month
cron_day_of_week
Allowed Values
0 - 59
0 - 23
1 - 31
1 - 12
0 - 7 (Sunday is 0 or 7)
Table 2.3: The list of submit commands and their value ranges.
The asterisk operator The * (asterisk) operator specifies that all of the allowed values are used for scheduling. For
example,
cron_month = *
becomes any and all of the list of possible months: (1,2,3,4,5,6,7,8,9,10,11,12). Thus, a job runs any month in
the year.
Ranges A range creates a set of integers from all the allowed values between two integers separated by a hyphen. The
specified range is inclusive, and the integer to the left of the hyphen must be less than the right hand integer. For
example,
cron_hour = 0-4
represents the set of hours from 12:00 am (midnight) to 4:00 am, or (0,1,2,3,4).
Lists A list is the union of the values or ranges separated by commas. Multiple entries of the same value are ignored.
For example,
cron_minute = 15,20,25,30
cron_hour
= 0-3,9-12,15
where this cron_minute example represents (15,20,25,30) and cron_hour represents (0,1,2,3,9,10,11,12,15).
Steps Steps select specific numbers from a range, based on an interval. A step is specified by appending a range or
the asterisk operator with a slash character (/), followed by an integer value. For example,
cron_minute = 10-30/5
cron_hour = */3
where this cron_minute example specifies every five minutes within the specified range to represent
(10,15,20,25,30), and cron_hour specifies every three hours of the day to represent (0,3,6,9,12,15,18,21).
HTCondor Version 8.6.4 Manual
2.13.2. CronTab Scheduling
153
Preparation Time and Execution Window
The cron_prep_time command is analogous to the deferral time’s deferral_prep_time command. It specifies the
number of seconds before the deferral time that the job is to be matched and sent to the execution machine. This
permits HTCondor to make necessary preparations before the deferral time occurs.
Consider the submit description file example that includes
cron_minute = 0
cron_hour = *
cron_prep_time = 300
The job is scheduled to begin execution at the top of every hour. Note that the setting of cron_hour in this example is
not required, as the default value will be *, specifying any and every hour of the day. The job will be matched and sent
to an execution machine no more than five minutes before the next deferral time. For example, if a job is submitted
at 9:30am, then the next deferral time will be calculated to be 10:00am. HTCondor may attempt to match the job to a
machine and send the job once it is 9:55am.
As the CronTab scheduling calculates and uses deferral time, jobs may also make use of the deferral window.
The submit command cron_window is analogous to the submit command deferral_window. Consider the submit
description file example that includes
cron_minute = 0
cron_hour = *
cron_window = 360
As the previous example, the job is scheduled to begin execution at the top of every hour. Yet with no preparation
time, the job is likely to miss its deferral time. The 6-minute window allows the job to begin execution, as long as it
arrives and can begin within 6 minutes of the deferral time, as seen by the time kept on the execution machine.
Scheduling
When a job using the CronTab functionality is submitted to HTCondor, use of at least one of the submit description file
commands beginning with cron_ causes HTCondor to calculate and set a deferral time for when the job should run. A
deferral time is determined based on the current time rounded later in time to the next minute. The deferral time is the
job’s DeferralTime attribute. A new deferral time is calculated when the job first enters the job queue, when the
job is re-queued, or when the job is released from the hold state. New deferral times for all jobs in the job queue using
the CronTab functionality are recalculated when a condor_reconfig or a condor_restart command that affects the job
queue is issued.
A job’s deferral time is not always the same time that a job will receive a match and be sent to the execution
machine. This is because HTCondor operates on the job queue at times that are independent of job events, such
as when job execution completes. Therefore, HTCondor may operate on the job queue just after a job’s deferral
time states that it is to begin execution. HTCondor attempts to start a job when the following pseudo-code boolean
expression evaluates to True:
HTCondor Version 8.6.4 Manual
2.13.2. CronTab Scheduling
154
( time() + SCHEDD_INTERVAL ) >= ( DeferralTime - CronPrepTime )
If the time() plus the number of seconds until the next time HTCondor checks the job queue is greater than or
equal to the time that the job should be submitted to the execution machine, then the job is to be matched and sent
now.
Jobs using the CronTab functionality are not automatically re-queued by HTCondor after their execution is complete. The submit description file for a job must specify an appropriate on_exit_remove command to ensure that a job
remains in the queue. This job maintains its original ClusterId and ProcId.
Usage Examples
Here are some examples of the submit commands necessary to schedule jobs to run at multifarious times. Please note
that it is not necessary to explicitly define each attribute; the default value is *.
Run 23 minutes after every two hours, every day of the week:
on_exit_remove = false
cron_minute = 23
cron_hour = 0-23/2
cron_day_of_month = *
cron_month = *
cron_day_of_week = *
Run at 10:30pm on each of May 10th to May 20th, as well as every remaining Monday within the month of May:
on_exit_remove = false
cron_minute = 30
cron_hour = 20
cron_day_of_month = 10-20
cron_month = 5
cron_day_of_week = 2
Run every 10 minutes and every 6 minutes before noon on January 18th with a 2-minute preparation time:
on_exit_remove = false
cron_minute = */10,*/6
cron_hour = 0-11
cron_day_of_month = 18
cron_month = 1
cron_day_of_week = *
cron_prep_time = 120
HTCondor Version 8.6.4 Manual
2.14. Special Environment Considerations
155
Limitations
The use of the CronTab functionality has all of the same limitations of deferral times, because the mechanism is based
upon deferral times.
• It is impossible to schedule vanilla and standard universe jobs at intervals that are smaller than the interval at which HTCondor evaluates jobs. This interval is determined by the configuration variable
SCHEDD_INTERVAL. As a vanilla or standard universe job completes execution and is placed back into the job
queue, it may not be placed in the idle state in time. This problem does not afflict local universe jobs.
• HTCondor cannot guarantee that a job will be matched in order to make its scheduled deferral time. A job
must be matched with an execution machine just as any other HTCondor job; if HTCondor is unable to find a
match, then the job will miss its chance for executing and must wait for the next execution time specified by the
CronTab schedule.
2.14 Special Environment Considerations
2.14.1 AFS
The HTCondor daemons do not run authenticated to AFS; they do not possess AFS tokens. Therefore, no child process
of HTCondor will be AFS authenticated. The implication of this is that you must set file permissions so that your job
can access any necessary files residing on an AFS volume without relying on having your AFS permissions.
If a job you submit to HTCondor needs to access files residing in AFS, you have the following choices:
1. Copy the needed files from AFS to either a local hard disk where HTCondor can access them using remote
system calls (if this is a standard universe job), or copy them to an NFS volume.
2. If the files must be kept on AFS, then set a host ACL (using the AFS fs setacl command) on the subdirectory to
serve as the current working directory for the job. If this is a standard universe job, then the host ACL needs to
give read/write permission to any process on the submit machine. If this is a vanilla universe job, then set the
ACL such that any host in the pool can access the files without being authenticated. If you do not know how to
use an AFS host ACL, ask the person at your site responsible for the AFS configuration.
The Center for High Throughput Computing hopes to improve upon how HTCondor deals with AFS authentication
in a subsequent release.
Please see section 3.14.1 for further discussion of this problem.
2.14.2 NFS
If the current working directory when a job is submitted is accessed via an NFS automounter, HTCondor may have
problems if the automounter later decides to unmount the volume before the job has completed. This is because
HTCondor Version 8.6.4 Manual
2.14.3. HTCondor Daemons That Do Not Run as root
condor_submit likely has stored the dynamic mount point as the job’s initial current working directory, and this mount
point could become automatically unmounted by the automounter.
There is a simple work around. When submitting the job, use the submit command initialdir to point to the
stable access point. For example, suppose the NFS automounter is configured to mount a volume at mount point
/a/myserver.company.com/vol1/johndoe whenever the directory /home/johndoe is accessed. Adding
the following line to the submit description file solves the problem.
initialdir = /home/johndoe
HTCondor attempts to flush the NFS cache on a submit machine in order to refresh a job’s initial working directory.
This allows files written by the job into an NFS mounted initial working directory to be immediately visible on the
submit machine. Since the flush operation can require multiple round trips to the NFS server, it is expensive. Therefore,
a job may disable the flushing by setting
+IwdFlushNFSCache = False
in the job’s submit description file. See page 1010 for a definition of the job ClassAd attribute.
2.14.3 HTCondor Daemons That Do Not Run as root
HTCondor is normally installed such that the HTCondor daemons have root permission. This allows HTCondor to
run the condor_shadow daemon and the job with the submitting user’s UID and file access rights. When HTCondor
is started as root, HTCondor jobs can access whatever files the user that submits the jobs can.
However, it is possible that the HTCondor installation does not have root access, or has decided not to run the
daemons as root. That is unfortunate, since HTCondor is designed to be run as root. To see if HTCondor is running as
root on a specific machine, use the command
condor_status -master -l <machine-name>
where <machine-name> is the name of the specified machine. This command displays the full condor_master
ClassAd; if the attribute RealUid equals zero, then the HTCondor daemons are indeed running with root access. If
the RealUid attribute is not zero, then the HTCondor daemons do not have root access.
NOTE: The Unix program ps is not an effective method of determining if HTCondor is running with root access.
When using ps, it may often appear that the daemons are running as the condor user instead of root. However, note that
the ps command shows the current effective owner of the process, not the real owner. (See the getuid(2) and geteuid(2)
Unix man pages for details.) In Unix, a process running under the real UID of root may switch its effective UID.
(See the seteuid(2) man page.) For security reasons, the daemons only set the effective UID to root when absolutely
necessary, as it will be to perform a privileged operation.
If daemons are not running with root access, make any and all files and/or directories that the job will touch
readable and/or writable by the UID (user id) specified by the RealUid attribute. Often this may mean using the
Unix command chmod 777 on the directory from which the HTCondor job is submitted.
HTCondor Version 8.6.4 Manual
156
2.14.4. Job Leases
2.14.4
157
Job Leases
A job lease specifies how long a given job will attempt to run on a remote resource, even if that resource loses contact
with the submitting machine. Similarly, it is the length of time the submitting machine will spend trying to reconnect
to the (now disconnected) execution host, before the submitting machine gives up and tries to claim another resource
to run the job. The goal aims at run only once semantics, so that the condor_schedd daemon does not allow the same
job to run on multiple sites simultaneously.
If the submitting machine is alive, it periodically renews the job lease, and all is well. If the submitting machine is
dead, or the network goes down, the job lease will no longer be renewed. Eventually the lease expires. While the lease
has not expired, the execute host continues to try to run the job, in the hope that the submit machine will come back
to life and reconnect. If the job completes and the lease has not expired, yet the submitting machine is still dead, the
condor_starter daemon will wait for a condor_shadow daemon to reconnect, before sending final information on the
job, and its output files. Should the lease expire, the condor_startd daemon kills off the condor_starter daemon and
user job.
A default value equal to 40 minutes exists for a job’s ClassAd attribute JobLeaseDuration, or this attribute
may be set in the submit description file, using job_lease_duration, to keep a job running in the case that the submit
side no longer renews the lease. There is a trade off in setting the value of job_lease_duration. Too small a value,
and the job might get killed before the submitting machine has a chance to recover. Forward progress on the job will
be lost. Too large a value, and an execute resource will be tied up waiting for the job lease to expire. The value should
be chosen based on how long the user is willing to tie up the execute machines, how quickly submit machines come
back up, and how much work would be lost if the lease expires, the job is killed, and the job must start over from its
beginning.
As a special case, a submit description file setting of
job_lease_duration = 0
as well as utilizing submission other than condor_submit that do not set JobLeaseDuration (such as using the
web services interface) results in the corresponding job ClassAd attribute to be explicitly undefined. This has the
further effect of changing the duration of a claim lease, the amount of time that the execution machine waits before
dropping a claim due to missing keep alive messages.
2.15 Potential Problems
2.15.1 Renaming of argv[0]
When HTCondor starts up your job, it renames argv[0] (which usually contains the name of the program) to condor_exec. This is convenient when examining a machine’s processes with the Unix command ps; the process is easily
identified as an HTCondor job.
Unfortunately, some programs read argv[0] expecting their own program name and get confused if they find
something unexpected like condor_exec.
HTCondor Version 8.6.4 Manual
CHAPTER
THREE
Administrators’ Manual
3.1 Introduction
This is the HTCondor Administrator’s Manual. Its purpose is to aid in the installation and administration of an
HTCondor pool. For help on using HTCondor, see the HTCondor User’s Manual.
An HTCondor pool is comprised of a single machine which serves as the central manager, and an arbitrary
number of other machines that have joined the pool. Conceptually, the pool is a collection of resources (machines)
and resource requests (jobs). The role of HTCondor is to match waiting requests with available resources. Every part
of HTCondor sends periodic updates to the central manager, the centralized repository of information about the state
of the pool. Periodically, the central manager assesses the current state of the pool and tries to match pending requests
with the appropriate resources.
Each resource has an owner, the one who sets the policy for the use of the machine. This person has absolute
power over the use of the machine, and HTCondor goes out of its way to minimize the impact on this owner caused
by HTCondor. It is up to the resource owner to define a policy for when HTCondor requests will serviced and when
they will be denied.
Each resource request has an owner as well: the user who submitted the job. These people want HTCondor to
provide as many CPU cycles as possible for their work. Often the interests of the resource owners are in conflict with
the interests of the resource requesters. The job of the HTCondor administrator is to configure the HTCondor pool to
find the happy medium that keeps both resource owners and users of resources satisfied. The purpose of this manual
is to relate the mechanisms that HTCondor provides to enable the administrator to find this happy medium.
158
3.1.1. The Different Roles a Machine Can Play
3.1.1 The Different Roles a Machine Can Play
Every machine in an HTCondor pool can serve a variety of roles. Most machines serve more than one role simultaneously. Certain roles can only be performed by a single machine in the pool. The following list describes what these
roles are and what resources are required on the machine that is providing that service:
Central Manager There can be only one central manager for the pool. This machine is the collector of information,
and the negotiator between resources and resource requests. These two halves of the central manager’s responsibility are performed by separate daemons, so it would be possible to have different machines providing those
two services. However, normally they both live on the same machine. This machine plays a very important part
in the HTCondor pool and should be reliable. If this machine crashes, no further matchmaking can be performed
within the HTCondor system, although all current matches remain in effect until they are broken by either party
involved in the match. Therefore, choose for central manager a machine that is likely to be up and running all
the time, or at least one that will be rebooted quickly if something goes wrong. The central manager will ideally
have a good network connection to all the machines in the pool, since these pool machines all send updates over
the network to the central manager.
Execute Any machine in the pool, including the central manager, can be configured as to whether or not it should
execute HTCondor jobs. Obviously, some of the machines will have to serve this function, or the pool will not
be useful. Being an execute machine does not require lots of resources. About the only resource that might
matter is disk space. In general the more resources a machine has in terms of swap space, memory, number of
CPUs, the larger variety of resource requests it can serve.
Submit Any machine in the pool, including the central manager, can be configured as to whether or not it should
allow HTCondor jobs to be submitted. The resource requirements for a submit machine are actually much
greater than the resource requirements for an execute machine. First, every submitted job that is currently
running on a remote machine runs a process on the submit machine. As a result, lots of running jobs will need
a fair amount of swap space and/or real memory. In addition, the checkpoint files from standard universe jobs
are stored on the local disk of the submit machine. If these jobs have a large memory image and there are a lot
of them, the submit machine will need a lot of disk space to hold these files. This disk space requirement can be
somewhat alleviated by using a checkpoint server, however the binaries of the jobs are still stored on the submit
machine.
Checkpoint Server Machines in the pool can be configured to act as checkpoint servers. This is optional, and is not
part of the standard HTCondor binary distribution. A checkpoint server is a machine that stores checkpoint files
for sets of jobs. A machine with this role should have lots of disk space and a good network connection to the
rest of the pool, as the traffic can be quite heavy.
3.1.2 The HTCondor Daemons
The following list describes all the daemons and programs that could be started under HTCondor and what they do:
condor_master This daemon is responsible for keeping all the rest of the HTCondor daemons running on each machine in the pool. It spawns the other daemons, and it periodically checks to see if there are new binaries
HTCondor Version 8.6.4 Manual
159
3.1.2. The HTCondor Daemons
160
installed for any of them. If there are, the condor_master daemon will restart the affected daemons. In addition, if any daemon crashes, the condor_master will send e-mail to the HTCondor administrator of the pool and
restart the daemon. The condor_master also supports various administrative commands that enable the administrator to start, stop or reconfigure daemons remotely. The condor_master will run on every machine in the pool,
regardless of the functions that each machine is performing.
condor_startd This daemon represents a given resource to the HTCondor pool, as a machine capable of running
jobs. It advertises certain attributes about machine that are used to match it with pending resource requests.
The condor_startd will run on any machine in the pool that is to be able to execute jobs. It is responsible for
enforcing the policy that the resource owner configures, which determines under what conditions jobs will be
started, suspended, resumed, vacated, or killed. When the condor_startd is ready to execute an HTCondor job,
it spawns the condor_starter.
condor_starter This daemon is the entity that actually spawns the HTCondor job on a given machine. It sets up
the execution environment and monitors the job once it is running. When a job completes, the condor_starter
notices this, sends back any status information to the submitting machine, and exits.
condor_schedd This daemon represents resource requests to the HTCondor pool. Any machine that is to be a submit
machine needs to have a condor_schedd running. When users submit jobs, the jobs go to the condor_schedd,
where they are stored in the job queue. The condor_schedd manages the job queue. Various tools to view
and manipulate the job queue, such as condor_submit, condor_q, and condor_rm, all must connect to the condor_schedd to do their work. If the condor_schedd is not running on a given machine, none of these commands
will work.
The condor_schedd advertises the number of waiting jobs in its job queue and is responsible for claiming available resources to serve those requests. Once a job has been matched with a given resource, the condor_schedd
spawns a condor_shadow daemon to serve that particular request.
condor_shadow This daemon runs on the machine where a given request was submitted and acts as the resource
manager for the request. Jobs that are linked for HTCondor’s standard universe, which perform remote system
calls, do so via the condor_shadow. Any system call performed on the remote execute machine is sent over the
network, back to the condor_shadow which performs the system call on the submit machine, and the result is
sent back over the network to the job on the execute machine. In addition, the condor_shadow is responsible
for making decisions about the request, such as where checkpoint files should be stored, and how certain files
should be accessed.
condor_collector This daemon is responsible for collecting all the information about the status of an HTCondor pool.
All other daemons periodically send ClassAd updates to the condor_collector. These ClassAds contain all
the information about the state of the daemons, the resources they represent or resource requests in the pool.
The condor_status command can be used to query the condor_collector for specific information about various
parts of HTCondor. In addition, the HTCondor daemons themselves query the condor_collector for important
information, such as what address to use for sending commands to a remote machine.
condor_negotiator This daemon is responsible for all the match making within the HTCondor system. Periodically,
the condor_negotiator begins a negotiation cycle, where it queries the condor_collector for the current state of
all the resources in the pool. It contacts each condor_schedd that has waiting resource requests in priority order,
and tries to match available resources with those requests. The condor_negotiator is responsible for enforcing
user priorities in the system, where the more resources a given user has claimed, the less priority they have to
acquire more resources. If a user with a better priority has jobs that are waiting to run, and resources are claimed
HTCondor Version 8.6.4 Manual
3.1.2. The HTCondor Daemons
161
by a user with a worse priority, the condor_negotiator can preempt that resource and match it with the user with
better priority.
NOTE: A higher numerical value of the user priority in HTCondor translate into worse priority for that user.
The best priority is 0.5, the lowest numerical value, and this priority gets worse as this number grows.
condor_kbdd This daemon is used on both Linux and Windows platforms. On those platforms, the condor_startd
frequently cannot determine console (keyboard or mouse) activity directly from the system, and requires a
separate process to do so. On Linux, the condor_kbdd connects to the X Server and periodically checks to
see if there has been any activity. On Windows, the condor_kbdd runs as the logged-in user and registers with
the system to receive keyboard and mouse events. When it detects console activity, the condor_kbdd sends a
command to the condor_startd. That way, the condor_startd knows the machine owner is using the machine
again and can perform whatever actions are necessary, given the policy it has been configured to enforce.
condor_ckpt_server The checkpoint server services requests to store and retrieve checkpoint files. If the pool is
configured to use a checkpoint server, but that machine or the server itself is down, HTCondor will revert to
sending the checkpoint files for a given job back to the submit machine.
condor_gridmanager This daemon handles management and execution of all grid universe jobs. The condor_schedd
invokes the condor_gridmanager when there are grid universe jobs in the queue, and the condor_gridmanager
exits when there are no more grid universe jobs in the queue.
condor_credd This daemon runs on Windows platforms to manage password storage in a secure manner.
condor_had This daemon implements the high availability of a pool’s central manager through monitoring the communication of necessary daemons. If the current, functioning, central manager machine stops working, then this
daemon ensures that another machine takes its place, and becomes the central manager of the pool.
condor_replication This daemon assists the condor_had daemon by keeping an updated copy of the pool’s state. This
state provides a better transition from one machine to the next, in the event that the central manager machine
stops working.
condor_transferer This short lived daemon is invoked by the condor_replication daemon to accomplish the task of
transferring a state file before exiting.
condor_procd This daemon controls and monitors process families within HTCondor. Its use is optional in general,
but it must be used if group-ID based tracking (see Section 3.14.11) is enabled.
condor_job_router This daemon transforms vanilla universe jobs into grid universe jobs, such that the transformed
jobs are capable of running elsewhere, as appropriate.
condor_lease_manager This daemon manages leases in a persistent manner. Leases are represented by ClassAds.
condor_rooster This daemon wakes hibernating machines based upon configuration details.
condor_defrag This daemon manages the draining of machines with fragmented partitionable slots, so that they become available for jobs requiring a whole machine or larger fraction of a machine.
condor_shared_port This daemon listens for incoming TCP packets on behalf of HTCondor daemons, thereby reducing the number of required ports that must be opened when HTCondor is accessible through a firewall.
HTCondor Version 8.6.4 Manual
3.2. Installation, Start Up, Shut Down, and Reconfiguration
When compiled from source code, the following daemons may be compiled in to provide optional functionality.
condor_hdfs This daemon manages the configuration of a Hadoop file system as well as the invocation of a properly
configured Hadoop file system.
3.2 Installation, Start Up, Shut Down, and Reconfiguration
This section contains the instructions for installing HTCondor. The installation will have a default configuration that
can be customized. Sections of the manual below explain customization.
Please read this entire section before starting installation.
Please read the copyright and disclaimer information in section . Installation and use of HTCondor is acknowledgment that you have read and agree to the terms.
Before installing HTCondor, please consider joining the htcondor-world mailing list. Traffic on this list is
kept to an absolute minimum; it is only used to announce new releases of HTCondor. To subscribe, go to
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-world, and fill out the online form.
You might also want to consider joining the htcondor-users mailing list.
This list is meant to be
a forum for HTCondor users to learn from each other and discuss using HTCondor.
It is an excellent place to ask the HTCondor community about using and configuring HTCondor. To subscribe, go to
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users, and fill out the online form.
Note that forward and reverse DNS lookup must be enabled for HTCondor to work properly.
3.2.1
Obtaining the HTCondor Software
The first step to installing HTCondor is to download it from the HTCondor web site, http://htcondor.org/. The downloads are available from the downloads page, at http://htcondor.org/downloads/.
3.2.2
Installation on Unix
The HTCondor binary distribution is packaged in the following files and directories:
LICENSE-2.0.txt the licensing agreement. By installing HTCondor, you agree to the contents of this file
README general information
bin directory which contains the distribution HTCondor user programs.
bosco_install the Perl script used to install Bosco.
condor_configure the Perl script used to install and configure HTCondor.
HTCondor Version 8.6.4 Manual
162
3.2.2. Installation on Unix
163
condor_install the Perl script used to install HTCondor.
etc directory which contains the distribution HTCondor configuration data.
examples directory containing C, Fortran and C++ example programs to run with HTCondor.
include directory containing HTCondor header files.
lib directory which contains the distribution HTCondor libraries.
libexec directory which contains the distribution HTCondor auxiliary programs for use internally by HTCondor.
man directory which contains the distribution HTCondor manual pages.
sbin directory containing HTCondor daemon binaries and admin tools.
src directory containing source for some interfaces.
Preparation
Before installation, you need to make a few important decisions about the basic layout of your pool. These decisions
answer the following questions:
1. What machine will be the central manager?
2. What machines should be allowed to submit jobs?
3. Will HTCondor run as root or not?
4. Who will be administering HTCondor on the machines in your pool?
5. Will you have a Unix user named condor and will its home directory be shared?
6. Where should the machine-specific directories for HTCondor go?
7. Where should the parts of the HTCondor system be installed?
• Configuration files
• Release directory
–
–
–
–
user binaries
system binaries
lib directory
etc directory
• Documentation
8. Am I using AFS?
9. Do I have enough disk space for HTCondor?
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
164
1. What machine will be the central manager? One machine in your pool must be the central manager. Install
HTCondor on this machine first. This is the centralized information repository for the HTCondor pool, and
it is also the machine that does match-making between available machines and submitted jobs. If the central
manager machine crashes, any currently active matches in the system will keep running, but no new matches
will be made. Moreover, most HTCondor tools will stop working. Because of the importance of this machine
for the proper functioning of HTCondor, install the central manager on a machine that is likely to stay up all the
time, or on one that will be rebooted quickly if it does crash.
Also consider network traffic and your network layout when choosing your central manager. All the daemons
send updates (by default, every 5 minutes) to this machine. Memory requirements for the central manager differ
by the number of machines in the pool: a pool with up to about 100 machines will require approximately 25
Mbytes of memory for the central manager’s tasks, and a pool with about 1000 machines will require approximately 100 Mbytes of memory for the central manager’s tasks.
A faster CPU will speed up matchmaking.
Generally jobs should not be either submitted or run on the central manager machine.
2. Which machines should be allowed to submit jobs? HTCondor can restrict the machines allowed to submit jobs.
Alternatively, it can allow any machine the network allows to connect to a submit machine to submit jobs. If
the HTCondor pool is behind a firewall, and all machines inside the firewall are trusted, the ALLOW_WRITE
configuration entry can be set to */*. Otherwise, it should be set to reflect the set of machines permitted to
submit jobs to this pool. HTCondor tries to be secure by default: it is shipped with an invalid value that allows
no machine to connect and submit jobs.
3. Will HTCondor run as root or not? We strongly recommend that the HTCondor daemons be installed and run
as the Unix user root. Without this, HTCondor can do very little to enforce security and policy decisions. You
can install HTCondor as any user; however there are serious security and performance consequences do doing a
non-root installation. Please see section 3.8.13 in the manual for the details and ramifications of installing and
running HTCondor as a Unix user other than root.
4. Who will administer HTCondor? Either root will be administering HTCondor directly, or someone else will be
acting as the HTCondor administrator. If root has delegated the responsibility to another person, keep in mind
that as long as HTCondor is started up as root, it should be clearly understood that whoever has the ability to
edit the condor configuration files can effectively run arbitrary programs as root.
The HTCondor administrator will be regularly updating HTCondor by following these instructions or by using
the system-specific installation methods below. The administrator will also customize policies of the HTCondor
submit and execute nodes. This person will also receive information from HTCondor if something goes wrong
with the pool, as described in the documentation of the CONDOR_ADMIN configuration variable.
5. Will you have a Unix user named condor, and will its home directory be shared? To simplify installation of
HTCondor, you should create a Unix user named condor on all machines in the pool. The HTCondor daemons will create files (such as the log files) owned by this user, and the home directory can be used to specify
the location of files and directories needed by HTCondor. The home directory of this user can either be shared
among all machines in your pool, or could be a separate home directory on the local partition of each machine.
Both approaches have advantages and disadvantages. Having the directories centralized can make administration easier, but also concentrates the resource usage such that you potentially need a lot of space for a single
shared home directory. See the section below on machine-specific directories for more details.
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
165
Note that the user condor must not be an account into which a person can log in. If a person can log in as user
condor, it permits a major security breach, in that the user condor could submit jobs that run as any other user,
providing complete access to the user’s data by the jobs. A standard way of not allowing log in to an account on
Unix platforms is to enter an invalid shell in the password file.
If you choose not to create a user named condor, then you must specify either via the CONDOR_IDS environment variable or the CONDOR_IDS config file setting which uid.gid pair should be used for the ownership of
various HTCondor files. See section 3.8.13 on UIDs in HTCondor in the Administrator’s Manual for details.
6. Where should the machine-specific directories for HTCondor go? HTCondor needs a few directories that are
unique on every machine in your pool. These are execute, spool, log, (and possibly lock). Generally,
all of them are subdirectories of a single machine specific directory called the local directory (specified by the
LOCAL_DIR macro in the configuration file). Each should be owned by the user that HTCondor is to be run as.
Do not stage other files in any of these directories; any files not created by HTCondor in these directories are
subject to removal.
If you have a Unix user named condor with a local home directory on each machine, the LOCAL_DIR could
just be user condor’s home directory (LOCAL_DIR = $(TILDE) in the configuration file). If this user’s
home directory is shared among all machines in your pool, you would want to create a directory for each host
(named by host name) for the local directory (for example, LOCAL_DIR = $(TILDE)/hosts/$(HOSTNAME)).
If you do not have a condor account on your machines, you can put these directories wherever you’d like.
However, where to place the directories will require some thought, as each one has its own resource needs:
execute This is the directory that acts as the current working directory for any HTCondor jobs that run on a
given execute machine. The binary for the remote job is copied into this directory, so there must be enough
space for it. (HTCondor will not send a job to a machine that does not have enough disk space to hold the
initial binary..) In addition, if the remote job dumps core for some reason, it is first dumped to the execute
directory before it is sent back to the submit machine. So, put the execute directory on a partition with
enough space to hold a possible core file from the jobs submitted to your pool.
spool The spool directory holds the job queue and history files, and the checkpoint files for all jobs submitted from a given machine. As a result, disk space requirements for the spool directory can be quite large,
particularly if users are submitting jobs with very large executables or image sizes. By using a checkpoint
server (see section 3.10 on Installing a Checkpoint Server on for details), you can ease the disk space
requirements, since all checkpoint files are stored on the server instead of the spool directories for each
machine. However, the initial checkpoint files (the executables for all the clusters you submit) are still
stored in the spool directory, so you will need some space, even with a checkpoint server. The amount of
space will depend on how many executables, and what size they are, that need to be stored in the spool
directory.
log Each HTCondor daemon writes its own log file, and each log file is placed in the log directory. You
can specify what size you want these files to grow to before they are rotated, so the disk space requirements of the directory are configurable. The larger the log files, the more historical information they
will hold if there is a problem, but the more disk space they use up. If you have a network file system installed at your pool, you might want to place the log directories in a shared location (such as
/usr/local/condor/logs/$(HOSTNAME)), so that you can view the log files from all your machines in a single location. However, if you take this approach, you will have to specify a local partition
for the lock directory (see below).
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
166
lock HTCondor uses a small number of lock files to synchronize access to certain files that are shared between multiple daemons. Because of problems encountered with file locking and network file systems
(particularly NFS), these lock files should be placed on a local partition on each machine. By default,
they are placed in the log directory. If you place your log directory on a network file system partition, specify a local partition for the lock files with the LOCK parameter in the configuration file (such as
/var/lock/condor).
Generally speaking, it is recommended that you do not put these directories (except lock) on the same partition
as /var, since if the partition fills up, you will fill up /var as well. This will cause lots of problems for your
machines. Ideally, you will have a separate partition for the HTCondor directories. Then, the only consequence
of filling up the directories will be HTCondor’s malfunction, not your whole machine.
7. Where should the parts of the HTCondor system be installed?
• Configuration Files
• Release directory
–
–
–
–
User Binaries
System Binaries
lib Directory
etc Directory
• Documentation
Configuration Files There can be more than one configuration file. They allow different levels of control over
how HTCondor is configured on each machine in the pool. The global configuration file is shared by all
machines in the pool. For ease of administration, this file should be located on a shared file system, if
possible. Local configuration files override settings in the global file permitting different daemons to run,
different policies for when to start and stop HTCondor jobs, and so on. There may be configuration files
specific to each platform in the pool. See section 3.14.3 on about Configuring HTCondor for Multiple
Platforms for details.
The location of configuration files is described in section 3.5.1.
Release Directory Every binary distribution contains a contains five subdirectories: bin, etc, lib, sbin,
and libexec. Wherever you choose to install these five directories we call the release directory (specified by the RELEASE_DIR macro in the configuration file). Each release directory contains platformdependent binaries and libraries, so you will need to install a separate one for each kind of machine in your
pool. For ease of administration, these directories should be located on a shared file system, if possible.
• User Binaries:
All of the files in the bin directory are programs that HTCondor users should expect to have in their
path. You could either put them in a well known location (such as /usr/local/condor/bin)
which you have HTCondor users add to their PATH environment variable, or copy those files directly
into a well known place already in the user’s PATHs (such as /usr/local/bin). With the above
examples, you could also leave the binaries in /usr/local/condor/bin and put in soft links
from /usr/local/bin to point to each program.
• System Binaries:
All of the files in the sbin directory are HTCondor daemons and agents, or programs that only the
HTCondor administrator would need to run. Therefore, add these programs only to the PATH of the
HTCondor administrator.
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
167
• Private HTCondor Binaries:
All of the files in the libexec directory are HTCondor programs that should never be run by hand,
but are only used internally by HTCondor.
• lib Directory:
The files in the lib directory are the HTCondor libraries that must be linked in with user jobs for all
of HTCondor’s checkpointing and migration features to be used. lib also contains scripts used by
the condor_compile program to help re-link jobs with the HTCondor libraries. These files should be
placed in a location that is world-readable, but they do not need to be placed in anyone’s PATH. The
condor_compile script checks the configuration file for the location of the lib directory.
• etc Directory:
etc contains an examples subdirectory which holds various example configuration files and other
files used for installing HTCondor. etc is the recommended location to keep the master copy of your
configuration files. You can put in soft links from one of the places mentioned above that HTCondor
checks automatically to find its global configuration file.
Documentation The documentation provided with HTCondor is currently available in HTML, Postscript and
PDF (Adobe Acrobat). It can be locally installed wherever is customary at your site. You can also find the
HTCondor documentation on the web at: http://htcondor.org/manual.
8. Am I using AFS? If you are using AFS at your site, be sure to read the section 3.14.1 in the manual. HTCondor
does not currently have a way to authenticate itself to AFS. A solution is not ready for Version 8.6.4. This
implies that you are probably not going to want to have the LOCAL_DIR for HTCondor on AFS. However, you
can (and probably should) have the HTCondor RELEASE_DIR on AFS, so that you can share one copy of those
files and upgrade them in a centralized location. You will also have to do something special if you submit jobs
to HTCondor from a directory on AFS. Again, read manual section 3.14.1 for all the details.
9. Do I have enough disk space for HTCondor? The compressed downloads of HTCondor currently range from a
low of about 13 Mbytes for 64-bit Ubuntu 12/Linux to about 115 Mbytes for Windows. The compressed source
code takes approximately 17 Mbytes.
In addition, you will need a lot of disk space in the local directory of any machines that are submitting jobs to
HTCondor. See question 6 above for details on this.
Unix Installation from an RPM
RPMs are available for HTCondor Version 8.6.4. We provide a Yum repository, as well as installation and configuration
in one easy step. This RPM installation is currently available for Red Hat-compatible systems only. As of HTCondor
version 7.5.1, the HTCondor RPM installs into File Hierachy Standard locations.
Yum repositories and instructions are at http://htcondor.org/yum/ . The repositories are named to distinguish stable
releases from development releases and by Red Hat version number. The 4 repositories are:
• condor-stable-rhel6.repo
• condor-stable-rhel7.repo
• condor-development-rhel6.repo
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
168
• condor-development-rhel7.repo
Here is an ordered set of steps that get HTCondor running using the RPM.
1. The HTCondor package will automatically add a condor user/group, if it does not exist already. Sites wishing
to control the attributes of this user/group should add the condor user/group manually before installation.
2. Download and install the meta-data that describes the appropriate YUM repository. This example is for the
stable series, on RHEL 7.
cd /etc/yum.repos.d
wget http://htcondor.org/yum/repo.d/condor-stable-rhel7.repo
Note that this step need be done only once; do not get the same repository more than once.
3. Import signing key The RPMs are signed in the Redhat 6 and RedHat 7 repositories.
wget http://htcondor.org/yum/RPM-GPG-KEY-HTCondor
rpm --import RPM-GPG-KEY-HTCondor
4. Install HTCondor.
yum install condor-all
5. As needed, edit the HTCondor configuration files to customize. The configuration files are in the directory
/etc/condor/ . Do not use condor_configure or condor_install for configuration. The installation will
be able to find configuration files without additional administrative intervention, as the configuration files are
placed in /etc, and HTCondor searches this directory.
6. Start HTCondor daemons:
/sbin/service condor start
Unix Installation from a Debian Package
Debian packages are available in HTCondor Version 8.6.4. We provide an APT repository, as well as installation and
configuration in one easy step. These Debian packages of HTCondor are currently available for Debian 7 (wheezy) and
Debian 8 (jessie). As of HTCondor version 7.5.1, the HTCondor Debian package installs into File Hierachy Standard
locations.
The HTCondor APT repositories are specified at http://htcondor.org/debian/ . See this web page for repository
information.
Here is an ordered set of steps that get HTCondor running.
1. The HTCondor package will automatically add a condor user/group, if it does not exist already. Sites wishing
to control the attributes of this user/group should add the condor user/group manually before installation.
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
169
2. If not already present, set up access to the appropriate APT repository; they are distinguished as stable or
development release, and by operating system. Ensure that the correct one of the following release and operating
system-specific lines is in the file /etc/apt/sources.list .
deb
deb
deb
deb
http://htcondor.org/debian/stable/ wheezy contrib
http://htcondor.org/debian/development/ wheezy contrib
http://htcondor.org/debian/stable/ jessie contrib
http://htcondor.org/debian/development/ jessie contrib
Note that this step need be done only once; do not add the same repository more than once.
3. Install and start HTCondor services:
apt-get update
apt-get install condor
4. As needed, edit the HTCondor configuration files to customize. The configuration files are in the directory
/etc/condor/ . Do not use condor_configure or condor_install for configuration. The installation will
be able to find configuration files without additional administrative intervention, as the configuration files are
placed in /etc, and HTCondor searches this directory.
Then, if any configuration changes are made, restart HTCondor with
/etc/init.d/condor restart
Unix Installation from a Tarball
Note that installation from a tarball is no longer the preferred method for installing HTCondor on Unix systems.
Installation via RPM or Debian package is recommended if available for your Unix version.
An overview of the tarball-based installation process is as follows:
1. Untar the HTCondor software.
2. Run condor_install or condor_configure to install the software.
Details are given below.
After download, all the files are in a compressed, tar format. They need to be untarred, as
tar xzf <completename>.tar.gz
After untarring, the directory will have the Perl scripts condor_configure and condor_install (and bosco_install), as
well as bin, etc, examples, include, lib, libexec, man, sbin, sql and src subdirectories.
The Perl script condor_configure installs HTCondor. Command-line arguments specify all needed information to
this script. The script can be executed multiple times, to modify or further set the configuration. condor_configure has
been tested using Perl 5.003. Use this or a more recent version of Perl.
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
170
condor_configure and condor_install are the same program, but have different default behaviors. condor_install is
identical to running
condor_configure --install=.
condor_configure and condor_install work on the named directories. As the names imply, condor_install is used to
install HTCondor, whereas condor_configure is used to modify the configuration of an existing HTCondor install.
condor_configure and condor_install are completely command-line driven and are not interactive. Several
command-line arguments are always needed with condor_configure and condor_install. The argument
--install=/path/to/release
specifies the path to the HTCondor release directories. The default command-line argument for condor_install is
--install=.
The argument
--install-dir=<directory>
or
--prefix=<directory>
specifies the path to the install directory.
The argument
--local-dir=<directory>
specifies the path to the local directory.
The --type option to condor_configure specifies one or more of the roles that a machine can take on within the
HTCondor pool: central manager, submit or execute. These options are given in a comma separated list. So, if a
machine is both a submit and execute machine, the proper command-line option is
--type=submit,execute
Install HTCondor on the central manager machine first. If HTCondor will run as root in this pool (Item 3 above),
run condor_install as root, and it will install and set the file permissions correctly. On the central manager machine,
run condor_install as follows.
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
171
% condor_install --prefix=~condor \
--local-dir=/scratch/condor --type=manager
To update the above HTCondor installation, for example, to also be submit machine:
% condor_configure --prefix=~condor \
--local-dir=/scratch/condor --type=manager,submit
As in the above example, the central manager can also be a submit point or an execute machine, but this is
only recommended for very small pools. If this is the case, the --type option changes to manager,execute or
manager,submit or manager,submit,execute.
After the central manager is installed, the execute and submit machines should then be configured. Decisions about
whether to run HTCondor as root should be consistent throughout the pool. For each machine in the pool, run
% condor_install --prefix=~condor \
--local-dir=/scratch/condor --type=execute,submit
See the condor_configure manual page 779 for details.
Starting HTCondor Under Unix After Installation
Now that HTCondor has been installed on the machine(s), there are a few things to check before starting up HTCondor.
1. Read through the <release_dir>/etc/condor_config file. There are a lot of possible settings and
you should at least take a look at the first two main sections to make sure everything looks okay. In particular,
you might want to set up security for HTCondor. See the section 3.8.1 to learn how to do this.
2. For Linux platforms, run the condor_kbdd to monitor keyboard and mouse activity on all machines within the
pool that will run a condor_startd; these are machines that execute jobs. To do this, the subsystem KBDD will
need to be added to the DAEMON_LIST configuration variable definition.
For Unix platforms other than Linux, HTCondor can monitor the activity of your mouse and keyboard, provided that you tell it where to look. You do this with the CONSOLE_DEVICES entry in the condor_startd
section of the configuration file. On most platforms, reasonable defaults are provided. For example, the default
device for the mouse is ’mouse’, since most installations have a soft link from /dev/mouse that points to
the right device (such as tty00 if you have a serial mouse, psaux if you have a PS/2 bus mouse, etc). If
you do not have a /dev/mouse link, you should either create one (you will be glad you did), or change the
CONSOLE_DEVICES entry in HTCondor’s configuration file. This entry is a comma separated list, so you
can have any devices in /dev count as ’console devices’ and activity will be reported in the condor_startd’s
ClassAd as ConsoleIdleTime.
3. (Linux only) HTCondor needs to be able to find the utmp file. According to the Linux File System Standard,
this file should be /var/run/utmp. If HTCondor cannot find it there, it looks in /var/adm/utmp. If it
still cannot find it, it gives up. So, if your Linux distribution places this file somewhere else, be sure to put a soft
link from /var/run/utmp to point to the real location.
HTCondor Version 8.6.4 Manual
3.2.2. Installation on Unix
172
To start up the HTCondor daemons, execute the command <release_dir>/sbin/condor_master. This
is the HTCondor master, whose only job in life is to make sure the other HTCondor daemons are running. The master
keeps track of the daemons, restarts them if they crash, and periodically checks to see if you have installed new binaries
(and, if so, restarts the affected daemons).
If you are setting up your own pool, you should start HTCondor on your central manager machine first. If you
have done a submit-only installation and are adding machines to an existing pool, the start order does not matter.
To ensure that HTCondor is running, you can run either:
ps -ef | egrep condor_
or
ps -aux | egrep condor_
depending on your flavor of Unix. On a central manager machine that can submit jobs as well as execute them, there
will be processes for:
• condor_master
• condor_collector
• condor_negotiator
• condor_startd
• condor_schedd
On a central manager machine that does not submit jobs nor execute them, there will be processes for:
• condor_master
• condor_collector
• condor_negotiator
For a machine that only submits jobs, there will be processes for:
• condor_master
• condor_schedd
For a machine that only executes jobs, there will be processes for:
• condor_master
HTCondor Version 8.6.4 Manual
3.2.3. Installation on Windows
173
• condor_startd
Once you are sure the HTCondor daemons are running, check to make sure that they are communicating with each
other. You can run condor_status to get a one line summary of the status of each machine in your pool.
Once you are sure HTCondor is working properly, you should add condor_master into your startup/bootup scripts
(i.e. /etc/rc ) so that your machine runs condor_master upon bootup. condor_master will then fire up the necessary
HTCondor daemons whenever your machine is rebooted.
If
your
system
uses
System-V
style
init
scripts,
you
can
look
in
<release_dir>/etc/examples/condor.boot for a script that can be used to start and stop HTCondor automatically by init. Normally, you would install this script as /etc/init.d/condor and put in soft link
from various directories (for example, /etc/rc2.d) that point back to /etc/init.d/condor. The exact
location of these scripts and links will vary on different platforms.
If your system uses BSD style boot scripts, you probably have an /etc/rc.local file. Add a line to start up
<release_dir>/sbin/condor_master.
Now that the HTCondor daemons are running, there are a few things you can and should do:
1. (Optional) Do a full install for the condor_compile script. condor_compile assists in linking jobs with the
HTCondor libraries to take advantage of all of HTCondor’s features. As it is currently installed, it will work by
placing it in front of any of the following commands that you would normally use to link your code: gcc, g++,
g77, cc, acc, c89, CC, f77, fort77 and ld. If you complete the full install, you will be able to use condor_compile
with any command whatsoever, in particular, make. See section 3.14.4 in the manual for directions.
2. Try building and submitting some test jobs. See examples/README for details.
3. If your site uses the AFS network file system, see section 3.14.1 in the manual.
4. We strongly recommend that you start up HTCondor (run the condor_master daemon) as user root. If you must
start HTCondor as some user other than root, see section 3.8.13.
3.2.3 Installation on Windows
This section contains the instructions for installing the Windows version of HTCondor. The install program will set
up a slightly customized configuration file that can be further customized after the installation has completed.
Be sure that the HTCondor tools are of the same version as the daemons installed. The HTCondor executable for
distribution is packaged in a single file named similarly to:
condor-8.4.11-390598-Windows-x86.msi
This file is approximately 107 Mbytes in size, and it can be removed once HTCondor is fully installed.
For any installation, HTCondor services are installed and run as the Local System account. Running the HTCondor
services as any other account (such as a domain user) is not supported and could be problematic.
HTCondor Version 8.6.4 Manual
3.2.3. Installation on Windows
174
Installation Requirements
• HTCondor for Windows is supported for Windows Vista or a more recent version.
• 300 megabytes of free disk space is recommended. Significantly more disk space could be necessary to be able
to run jobs with large data files.
• HTCondor for Windows will operate on either an NTFS or FAT32 file system. However, for security purposes,
NTFS is preferred.
• HTCondor for Windows uses the Visual C++ 2012 C runtime library.
Preparing to Install HTCondor under Windows
Before installing the Windows version of HTCondor, there are two major decisions to make about the basic layout of
the pool.
1. What machine will be the central manager?
2. Is there enough disk space for HTCondor?
If the answers to these questions are already known, skip to the Windows Installation Procedure section below,
section 3.2.3. If unsure, read on.
• What machine will be the central manager?
One machine in your pool must be the central manager. This is the centralized information repository for
the HTCondor pool and is also the machine that matches available machines with waiting jobs. If the central
manager machine crashes, any currently active matches in the system will keep running, but no new matches
will be made. Moreover, most HTCondor tools will stop working. Because of the importance of this machine
for the proper functioning of HTCondor, we recommend installing it on a machine that is likely to stay up all
the time, or at the very least, one that will be rebooted quickly if it does crash. Also, because all the services
will send updates (by default every 5 minutes) to this machine, it is advisable to consider network traffic and
network layout when choosing the central manager.
Install HTCondor on the central manager before installing on the other machines within the pool.
Generally jobs should not be either submitted or run on the central manager machine.
• Is there enough disk space for HTCondor?
The HTCondor release directory takes up a fair amount of space. The size requirement for the release directory
is approximately 250 Mbytes. HTCondor itself, however, needs space to store all of the jobs and their input
files. If there will be large numbers of jobs, consider installing HTCondor on a volume with a large amount of
free space.
HTCondor Version 8.6.4 Manual
3.2.3. Installation on Windows
175
Installation Procedure Using the MSI Program
Installation of HTCondor must be done by a user with administrator privileges. After installation, the HTCondor
services will be run under the local system account. When HTCondor is running a user job, however, it will run that
user job with normal user permissions.
Download HTCondor, and start the installation process by running the installer. The HTCondor installation is
completed by answering questions and choosing options within the following steps.
If HTCondor is already installed. If HTCondor has been previously installed, a dialog box will appear before the
installation of HTCondor proceeds. The question asks if you wish to preserve your current HTCondor configuration files. Answer yes or no, as appropriate.
If you answer yes, your configuration files will not be changed, and you will proceed to the point where the new
binaries will be installed.
If you answer no, then there will be a second question that asks if you want to use answers given during the
previous installation as default answers.
STEP 1: License Agreement. The first step in installing HTCondor is a welcome screen and license agreement. You
are reminded that it is best to run the installation when no other Windows programs are running. If you need to
close other Windows programs, it is safe to cancel the installation and close them. You are asked to agree to the
license. Answer yes or no. If you should disagree with the License, the installation will not continue.
Also fill in name and company information, or use the defaults as given.
STEP 2: HTCondor Pool Configuration. The HTCondor configuration needs to be set based upon if this is a new
pool or to join an existing one. Choose the appropriate radio button.
For a new pool, enter a chosen name for the pool. To join an existing pool, enter the host name of the central
manager of the pool.
STEP 3: This Machine’s Roles. Each machine within an HTCondor pool can either submit jobs or execute submitted
jobs, or both submit and execute jobs. A check box determines if this machine will be a submit point for the
pool.
A set of radio buttons determines the ability and configuration of the ability to execute jobs. There are four
choices:
Do not run jobs on this machine. This machine will not execute HTCondor jobs.
Always run jobs and never suspend them.
Run jobs when the keyboard has been idle for 15 minutes.
Run jobs when the keyboard has been idle for 15 minutes, and the CPU is idle.
For testing purposes, it is often helpful to use the always run HTCondor jobs option.
For a machine that is to execute jobs and the choice is one of the last two in the list, HTCondor needs to further
know what to do with the currently running jobs. There are two choices:
Keep the job in memory and continue when the machine meets the condition chosen for when to run jobs.
HTCondor Version 8.6.4 Manual
3.2.3. Installation on Windows
176
Restart the job on a different machine.
This choice involves a trade off. Restarting the job on a different machine is less intrusive on the workstation
owner than leaving the job in memory for a later time. A suspended job left in memory will require swap space,
which could be a scarce resource. Leaving a job in memory, however, has the benefit that accumulated run time
is not lost for a partially completed job.
STEP 4: The Account Domain. Enter the machine’s accounting (or UID) domain. On this version of HTCondor for
Windows, this setting is only used for user priorities (see section 3.6) and to form a default e-mail address for
the user.
STEP 5: E-mail Settings. Various parts of HTCondor will send e-mail to an HTCondor administrator if something
goes wrong and requires human attention. Specify the e-mail address and the SMTP relay host of this administrator. Please pay close attention to this e-mail, since it will indicate problems in the HTCondor pool.
STEP 6: Java Settings. In order to run jobs in the java universe, HTCondor must have the path to the jvm executable
on the machine. The installer will search for and list the jvm path, if it finds one. If not, enter the path. To
disable use of the java universe, leave the field blank.
STEP 7: Host Permission Settings. Machines within the HTCondor pool will need various types of access permission. The three categories of permission are read, write, and administrator. Enter the machines or domain to be
given access permissions, or use the defaults provided. Wild cards and macros are permitted.
Read Read access allows a machine to obtain information about HTCondor such as the status of machines in
the pool and the job queues. All machines in the pool should be given read access. In addition, giving read
access to *.cs.wisc.edu will allow the HTCondor team to obtain information about the HTCondor pool, in
the event that debugging is needed.
Write All machines in the pool should be given write access. It allows the machines you specify to send
information to your local HTCondor daemons, for example, to start an HTCondor job. Note that for a
machine to join the HTCondor pool, it must have both read and write access to all of the machines in the
pool.
Administrator A machine with administrator access will be allowed more extended permission to do things
such as change other user’s priorities, modify the job queue, turn HTCondor services on and off, and restart
HTCondor. The central manager should be given administrator access and is the default listed. This setting
is granted to the entire machine, so care should be taken not to make this too open.
For more details on these access permissions, and others that can be manually changed in your configuration
file, please see the section titled Setting Up IP/Host-Based Security in HTCondor in section section 3.8.9.
STEP 8: VM Universe Setting. A radio button determines whether this machine will be configured to run vm universe jobs utilizing VMware. In addition to having the VMware Server installed, HTCondor also needs Perl
installed. The resources available for vm universe jobs can be tuned with these settings, or the defaults listed
can be used.
Version Use the default value, as only one version is currently supported.
Maximum Memory The maximum memory that each virtual machine is permitted to use on the target machine.
Maximum Number of VMs The number of virtual machines that can be run in parallel on the target machine.
HTCondor Version 8.6.4 Manual
3.2.3. Installation on Windows
177
Networking Support The VMware instances can be configured to use network support. There are four options
in the pull-down menu.
•
•
•
•
None: No networking support.
NAT: Network address translation.
Bridged: Bridged mode.
NAT and Bridged: Allow both methods.
Path to Perl Executable The path to the Perl executable.
STEP 9: HDFS Settings. A radio button enables support for the Hadoop Distributed File System (HDFS). When
enabled, a further radio button specifies either name node or data node mode.
Running HDFS requires Java to be installed, and HTCondor must know where the installation is. Running
HDFS in data node mode also requires the installation of Cygwin, and the path to the Cygwin directory must be
added to the global PATH environment variable.
HDFS has several configuration options that must be filled in to be used.
Primary Name Node The full host name of the primary name node.
Name Node Port The port that the name node is listening on.
Name Node Web Port The port the name node’s web interface is bound to. It should be different from the
name node’s main port.
STEP 10: Choose Setup Type The next step is where the destination of the HTCondor files will be decided. We
recommend that HTCondor be installed in the location shown as the default in the install choice: C:\Condor.
This is due to several hard coded paths in scripts and configuration files. Clicking on the Custom choice permits
changing the installation directory.
Installation on the local disk is chosen for several reasons. The HTCondor services run as local system, and
within Microsoft Windows, local system has no network privileges. Therefore, for HTCondor to operate, HTCondor should be installed on a local hard drive, as opposed to a network drive (file server).
The second reason for installation on the local disk is that the Windows usage of drive letters has implications
for where HTCondor is placed. The drive letter used must be not change, even when different users are logged
in. Local drive letters do not change under normal operation of Windows.
While it is strongly discouraged, it may be possible to place HTCondor on a hard drive that is not local, if a
dependency is added to the service control manager such that HTCondor starts after the required file services
are available.
Unattended Installation Procedure Using the Included Setup Program
This section details how to run the HTCondor for Windows installer in an unattended batch mode. This mode is one
that occurs completely from the command prompt, without the GUI interface.
The HTCondor for Windows installer uses the Microsoft Installer (MSI) technology, and it can be configured for
unattended installs analogous to any other ordinary MSI installer.
The following is a sample batch file that is used to set all the properties necessary for an unattended install.
HTCondor Version 8.6.4 Manual
3.2.3. Installation on Windows
178
@echo on
set ARGS=
set ARGS=NEWPOOL="N"
set ARGS=%ARGS% POOLNAME=""
set ARGS=%ARGS% RUNJOBS="C"
set ARGS=%ARGS% VACATEJOBS="Y"
set ARGS=%ARGS% SUBMITJOBS="Y"
set ARGS=%ARGS% CONDOREMAIL="you@yours.com"
set ARGS=%ARGS% SMTPSERVER="smtp.localhost"
set ARGS=%ARGS% HOSTALLOWREAD="*"
set ARGS=%ARGS% HOSTALLOWWRITE="*"
set ARGS=%ARGS% HOSTALLOWADMINISTRATOR="$(IP_ADDRESS)"
set ARGS=%ARGS% INSTALLDIR="C:\Condor"
set ARGS=%ARGS% POOLHOSTNAME="$(IP_ADDRESS)"
set ARGS=%ARGS% ACCOUNTINGDOMAIN="none"
set ARGS=%ARGS% JVMLOCATION="C:\Windows\system32\java.exe"
set ARGS=%ARGS% USEVMUNIVERSE="N"
set ARGS=%ARGS% VMMEMORY="128"
set ARGS=%ARGS% VMMAXNUMBER="$(NUM_CPUS)"
set ARGS=%ARGS% VMNETWORKING="N"
REM set ARGS=%ARGS% LOCALCONFIG="http://my.example.com/condor_config.$(FULL_HOSTNAME)"
msiexec /qb /l* condor-install-log.txt /i condor-8.0.0-133173-Windows-x86.msi %ARGS%
Each property corresponds to answers that would have been supplied while running an interactive installer. The
following is a brief explanation of each property as it applies to unattended installations:
NEWPOOL = < Y | N > determines whether the installer will create a new pool with the target machine as the
central manager.
POOLNAME sets the name of the pool, if a new pool is to be created. Possible values are either the name or the
empty string "".
RUNJOBS = < N | A | I | C > determines when HTCondor will run jobs. This can be set to:
• Never run jobs (N)
• Always run jobs (A)
• Only run jobs when the keyboard and mouse are Idle (I)
• Only run jobs when the keyboard and mouse are idle and the CPU usage is low (C)
VACATEJOBS = < Y | N > determines what HTCondor should do when it has to stop the execution of a user job.
When set to Y, HTCondor will vacate the job and start it somewhere else if possible. When set to N, HTCondor
will merely suspend the job in memory and wait for the machine to become available again.
SUBMITJOBS = < Y | N > will cause the installer to configure the machine as a submit node when set to Y.
HTCondor Version 8.6.4 Manual
3.2.3. Installation on Windows
179
CONDOREMAIL sets the e-mail address of the HTCondor administrator. Possible values are an e-mail address or
the empty string "".
HOSTALLOWREAD is a list of host names that are allowed to issue READ commands to HTCondor daemons. This
value should be set in accordance with the HOSTALLOW_READ setting in the configuration file, as described in
section 3.8.9.
HOSTALLOWWRITE is a list of host names that are allowed to issue WRITE commands to HTCondor daemons.
This value should be set in accordance with the HOSTALLOW_WRITE setting in the configuration file, as described in section 3.8.9.
HOSTALLOWADMINISTRATOR is a list of host names that are allowed to issue ADMINISTRATOR commands
to HTCondor daemons. This value should be set in accordance with the HOSTALLOW_ADMINISTRATOR
setting in the configuration file, as described in section 3.8.9.
INSTALLDIR defines the path to the directory where HTCondor will be installed.
POOLHOSTNAME defines the host name of the pool’s central manager.
ACCOUNTINGDOMAIN defines the accounting (or UID) domain the target machine will be in.
JVMLOCATION defines the path to Java virtual machine on the target machine.
SMTPSERVER defines the host name of the SMTP server that the target machine is to use to send e-mail.
VMMEMORY an integer value that defines the maximum memory each VM run on the target machine.
VMMAXNUMBER an integer value that defines the number of VMs that can be run in parallel on the target machine.
VMNETWORKING = < N | A | B | C > determines if VM Universe can use networking. This can be set to:
• None (N)
• NAT (A)
• Bridged (B)
• NAT and Bridged (C)
USEVMUNIVERSE = < Y | N > will cause the installer to enable VM Universe jobs on the target machine.
LOCALCONFIG defines the location of the local configuration file. The value can be the path to a file on the local
machine, or it can be a URL beginning with http. If the value is a URL, then the condor_urlfetch tool is
invoked to fetch configuration whenever the configuration is read.
PERLLOCATION defines the path to Perl on the target machine. This is required in order to use the vm universe.
After defining each of these properties for the MSI installer, the installer can be started with the msiexec command.
The following command starts the installer in unattended mode, and it dumps a journal of the installer’s progress to a
log file:
msiexec /qb /lxv* condor-install-log.txt /i condor-8.0.0-173133-Windows-x86.msi [property=value] ...
More information on the features of msiexec can be found at Microsoft’s
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/msiexec.mspx.
HTCondor Version 8.6.4 Manual
website
at
3.2.3. Installation on Windows
180
Manual Installation HTCondor on Windows
If you are to install HTCondor on many different machines, you may wish to use some other mechanism to install
HTCondor on additional machines rather than running the Setup program described above on each machine.
WARNING: This is for advanced users only! All others should use the Setup program described above.
Here is a brief overview of how to install HTCondor manually without using the provided GUI-based setup program:
The Service The service that HTCondor will install is called "Condor". The Startup Type is Automatic. The service
should log on as System Account, but do not enable "Allow Service to Interact with Desktop". The program
that is run is condor_master.exe.
The HTCondor service can be installed and removed using the sc.exe tool, which is included in Windows XP
and Windows 2003 Server. The tool is also available as part of the Windows 2000 Resource Kit.
Installation can be done as follows:
sc create Condor binpath= c:\condor\bin\condor_master.exe
To remove the service, use:
sc delete Condor
The Registry HTCondor uses a few registry entries in its operation.
The key that HTCondor uses is
HKEY_LOCAL_MACHINE/Software/Condor. The values that HTCondor puts in this registry key serve two
purposes.
1. The values of CONDOR_CONFIG and RELEASE_DIR are used for HTCondor to start its service.
CONDOR_CONFIG should point to the condor_config file. In this version of HTCondor, it must
reside on the local disk.
RELEASE_DIR should point to the directory where HTCondor is installed. This is typically C:\Condor,
and again, this must reside on the local disk.
2. The other purpose is storing the entries from the last installation so that they can be used for the next one.
The File System The files that are needed for HTCondor to operate are identical to the Unix version of HTCondor,
except that executable files end in .exe. For example the on Unix one of the files is condor_master and on
HTCondor the corresponding file is condor_master.exe.
These files currently must reside on the local disk for a variety of reasons. Advanced Windows users might be
able to put the files on remote resources. The main concern is twofold. First, the files must be there when the
service is started. Second, the files must always be in the same spot (including drive letter), no matter who is
logged into the machine.
Note also that when installing manually, you will need to create the directories that HTCondor will expect to be
present given your configuration. This normally is simply a matter of creating the log, spool, and execute
directories. Do not stage other files in any of these directories; any files not created by HTCondor in these
directories are subject to removal.
HTCondor Version 8.6.4 Manual
3.2.3. Installation on Windows
181
Starting HTCondor Under Windows After Installation
After the installation of HTCondor is completed, the HTCondor service must be started. If you used the GUI-based
setup program to install HTCondor, the HTCondor service should already be started. If you installed manually,
HTCondor must be started by hand, or you can simply reboot. NOTE: The HTCondor service will start automatically
whenever you reboot your machine.
To start HTCondor by hand:
1. From the Start menu, choose Settings.
2. From the Settings menu, choose Control Panel.
3. From the Control Panel, choose Services.
4. From Services, choose Condor, and Start.
Or, alternatively you can enter the following command from a command prompt:
net start condor
Run the Task Manager (Control-Shift-Escape) to check that HTCondor services are running. The following tasks
should be running:
• condor_master.exe
• condor_negotiator.exe, if this machine is a central manager.
• condor_collector.exe, if this machine is a central manager.
• condor_startd.exe, if you indicated that this HTCondor node should start jobs
• condor_schedd.exe, if you indicated that this HTCondor node should submit jobs to the HTCondor pool.
Also, you should now be able to open up a new cmd (DOS prompt) window, and the HTCondor bin directory
should be in your path, so you can issue the normal HTCondor commands, such as condor_q and condor_status.
HTCondor is Running Under Windows ... Now What?
Once HTCondor services are running, try submitting test jobs. Example 2 within section 2.5.1 presents a vanilla
universe job.
HTCondor Version 8.6.4 Manual
3.2.4. Upgrading – Installing a New Version on an Existing Pool
3.2.4
Upgrading – Installing a New Version on an Existing Pool
An upgrade changes the running version of HTCondor from the current installation to a newer version. The safe
method to install and start running a newer version of HTCondor in essence is: shut down the current installation of
HTCondor, install the newer version, and then restart HTCondor using the newer version. To allow for falling back to
the current version, place the new version in a separate directory. Copy the existing configuration files, and modify the
copy to point to and use the new version, as well as incorporate any configuration variables that are new or changed
in the new version. Set the CONDOR_CONFIG environment variable to point to the new copy of the configuration, so
the new version of HTCondor will use the new configuration when restarted.
As of HTCondor version 8.2.0, the default configuration file has been substantially reduced in size by defining
compile-time default values for most configuration variables. Therefore, when upgrading from a version of HTCondor earlier than 8.2.0 to a more recent version, the option of reducing the size of the configuration file is an option.
The goal is to identify and use only the configuration variable values that differ from the compile-time default values. This is facilitated by using condor_config_val with the -writeconfig:upgrade argument, to create a file that
behaves the same as the current configuration, but is much smaller, because values matching the default values (as
well as some obsolete variables) have been removed. Items in the file created by running condor_config_val with the
-writeconfig:upgrade argument will be in the order that they were read from the original configuration files. This file
is a convenient guide to stripping the cruft from old configuration files.
When upgrading from a version of HTCondor earlier than 6.8 to more recent version, note that the configuration
settings must be modified for security reasons. Specifically, the HOSTALLOW_WRITE configuration variable must be
explicitly changed, or no jobs can be submitted, and error messages will be issued by HTCondor tools.
Another way to upgrade leaves HTCondor running. HTCondor will automatically restart itself if the condor_master binary is updated, and this method takes advantage of this. Download the newer version, placing it such
that it does not overwrite the currently running version. With the download will be a new set of configuration files;
update this new set with any specializations implemented in the currently running version of HTCondor. Then, modify
the currently running installation by changing its configuration such that the path to binaries points instead to the new
binaries. One way to do that (under Unix) is to use a symbolic link that points to the current HTCondor installation
directory (for example, /opt/condor). Change the symbolic link to point to the new directory. If HTCondor is
configured to locate its binaries via the symbolic link, then after the symbolic link changes, the condor_master daemon notices the new binaries and restarts itself. How frequently it checks is controlled by the configuration variable
MASTER_CHECK_NEW_EXEC_INTERVAL, which defaults 5 minutes.
When the condor_master notices new binaries, it begins a graceful restart. On an execute machine, a graceful
restart means that running jobs are preempted. Standard universe jobs will attempt to take a checkpoint. This could be
a bottleneck if all machines in a large pool attempt to do this at the same time. If they do not complete within the cutoff
time specified by the KILL policy expression (defaults to 10 minutes), then the jobs are killed without producing a
checkpoint. It may be appropriate to increase this cutoff time, and a better approach may be to upgrade the pool in
stages rather than all at once.
For universes other than the standard universe, jobs are preempted. If jobs have been guaranteed a certain amount
of uninterrupted run time with MaxJobRetirementTime, then the job is not killed until the specified amount of
retirement time has been exceeded (which is 0 by default). The first step of killing the job is a soft kill signal, which
can be intercepted by the job so that it can exit gracefully, perhaps saving its state. If the job has not gone away once
the KILL expression fires (10 minutes by default), then the job is forcibly hard-killed. Since the graceful shutdown of
HTCondor Version 8.6.4 Manual
182
3.2.5. Shutting Down and Restarting an HTCondor Pool
jobs may rely on shared resources such as disks where state is saved, the same reasoning applies as for the standard
universe: it may be appropriate to increase the cutoff time for large pools, and a better approach may be to upgrade
the pool in stages to avoid jobs running out of time.
Another time limit to be aware of is the configuration variable SHUTDOWN_GRACEFUL_TIMEOUT. This defaults
to 30 minutes. If the graceful restart is not completed within this time, a fast restart ensues. This causes jobs to be
hard-killed.
3.2.5
Shutting Down and Restarting an HTCondor Pool
All of the commands described in this section are subject to the security policy chosen for the HTCondor pool. As
such, the commands must be either run from a machine that has the proper authorization, or run by a user that is
authorized to issue the commands. Section 3.8 details the implementation of security in HTCondor.
Shutting Down HTCondor There are a variety of ways to shut down all or parts of an HTCondor pool. All utilize
the condor_off tool.
To stop a single execute machine from running jobs, the condor_off command specifies the machine by host
name.
condor_off -startd <hostname>
A running standard universe job will be allowed to take a checkpoint before the job is killed. A running job
under another universe will be killed. If it is instead desired that the machine stops running jobs only after the
currently executing job completes, the command is
condor_off -startd -peaceful <hostname>
Note that this waits indefinitely for the running job to finish, before the condor_startd daemon exits.
Th shut down all execution machines within the pool,
condor_off -all -startd
To wait indefinitely for each machine in the pool to finish its current HTCondor job, shutting down all of the
execute machines as they no longer have a running job,
condor_off -all -startd -peaceful
To shut down HTCondor on a machine from which jobs are submitted,
condor_off -schedd <hostname>
If it is instead desired that the submit machine shuts down only after all jobs that are currently in the queue are
finished, first disable new submissions to the queue by setting the configuration variable
MAX_JOBS_SUBMITTED = 0
HTCondor Version 8.6.4 Manual
183
3.2.6. Reconfiguring an HTCondor Pool
184
See instructions below in section 3.2.6 for how to reconfigure a pool. After the reconfiguration, the command to
wait for all jobs to complete and shut down the submission of jobs is
condor_off -schedd -peaceful <hostname>
Substitute the option -all for the host name, if all submit machines in the pool are to be shut down.
Restarting HTCondor, If HTCondor Daemons Are Not Running If HTCondor is not running, perhaps because
one of the condor_off commands was used, then starting HTCondor daemons back up depends on which part
of HTCondor is currently not running.
If no HTCondor daemons are running, then starting HTCondor is a matter of executing the condor_master
daemon. The condor_master daemon will then invoke all other specified daemons on that machine. The condor_master daemon executes on every machine that is to run HTCondor.
If a specific daemon needs to be started up, and the condor_master daemon is already running, then issue the
command on the specific machine with
condor_on -subsystem <subsystemname>
where <subsystemname> is replaced by the daemon’s subsystem name. Or, this command might be issued
from another machine in the pool (which has administrative authority) with
condor_on <hostname> -subsystem <subsystemname>
where <subsystemname> is replaced by the daemon’s subsystem name, and <hostname> is replaced by
the host name of the machine where this condor_on command is to be directed.
Restarting HTCondor, If HTCondor Daemons Are Running If HTCondor daemons are currently running, but
need to be killed and newly invoked, the condor_restart tool does this. This would be the case for a new
value of a configuration variable for which using condor_reconfig is inadequate.
To restart all daemons on all machines in the pool,
condor_restart -all
To restart all daemons on a single machine in the pool,
condor_restart <hostname>
where <hostname> is replaced by the host name of the machine to be restarted.
3.2.6 Reconfiguring an HTCondor Pool
To change a global configuration variable and have all the machines start to use the new setting, change the value
within the file, and send a condor_reconfig command to each host. Do this with a single command,
condor_reconfig -all
HTCondor Version 8.6.4 Manual
3.3. Introduction to Configuration
185
If the global configuration file is not shared among all the machines, as it will be if using a shared file system, the
change must be made to each copy of the global configuration file before issuing the condor_reconfig command.
Issuing a condor_reconfig command is inadequate for some configuration variables. For those, a restart of HTCondor is required. Those configuration variables that require a restart are listed in section 3.5.1. The manual page for
condor_restart is at 873.
3.3 Introduction to Configuration
This section of the manual contains general information about HTCondor configuration, relating to all parts of the
HTCondor system. If you’re setting up an HTCondor pool, you should read this section before you read the other
configuration-related sections:
• Section 3.4 contains information about configuration templates, which are now the preferred way to set many
configuration macros.
• Section 3.5 contains information about the hundreds of individual configuration macros. In general, it is best
to try to achieve your desired configuration using configuration templates before resorting to setting individual
configuration macros, but it is sometimes necessary to set individual configuration macros.
• The settings that control the policy under which HTCondor will start, suspend, resume, vacate or kill jobs are
described in section 3.7 on Policy Configuration for the condor_startd.
3.3.1 HTCondor Configuration Files
The HTCondor configuration files are used to customize how HTCondor operates at a given site. The basic configuration as shipped with HTCondor can be used as a starting point, but most likely you will want to modify that
configuration to some extent.
Each HTCondor program will, as part of its initialization process, configure itself by calling a library routine which
parses the various configuration files that might be used, including pool-wide, platform-specific, and machine-specific
configuration files. Environment variables may also contribute to the configuration.
The result of configuration is a list of key/value pairs. Each key is a configuration variable name, and each value is
a string literal that may utilize macro substitution (as defined below). Some configuration variables are evaluated by
HTCondor as ClassAd expressions; some are not. Consult the documentation for each specific case. Unless otherwise
noted, configuration values that are expected to be numeric or boolean constants can be any valid ClassAd expression
of operators on constants. Example:
MINUTE
= 60
HOUR
= (60 * $(MINUTE))
SHUTDOWN_GRACEFUL_TIMEOUT = ($(HOUR)*24)
HTCondor Version 8.6.4 Manual
3.3.2. Ordered Evaluation to Set the Configuration
3.3.2 Ordered Evaluation to Set the Configuration
Multiple files, as well as a program’s environment variables, determine the configuration. The order in which attributes
are defined is important, as later definitions override earlier definitions. The order in which the (multiple) configuration
files are parsed is designed to ensure the security of the system. Attributes which must be set a specific way must appear
in the last file to be parsed. This prevents both the naive and the malicious HTCondor user from subverting the system
through its configuration. The order in which items are parsed is:
1. a single initial configuration file, which has historically been known as the global configuration file (see below);
2. other configuration files that are referenced and parsed due to specification within the single initial configuration
file (these files have historically been known as local configuration files);
3. if HTCondor daemons are not running as root on Unix platforms,
the file
$(HOME)/.condor/user_config if it exists, or the file defined by configuration variable
USER_CONFIG_FILE;
if HTCondor daemons are not running as Local System on Windows platforms, the file
%USERPROFILE\.condor\user_config if it exists, or the file defined by configuration variable
USER_CONFIG_FILE;
4. specific environment variables whose names are prefixed with _CONDOR_ (note that these environment variables directly define macro name/value pairs, not the names of configuration files).
Some HTCondor tools utilize environment variables to set their configuration; these tools search for specificallynamed environment variables. The variable names are prefixed by the string _CONDOR_ or _condor_. The tools
strip off the prefix, and utilize what remains as configuration. As the use of environment variables is the last within
the ordered evaluation, the environment variable definition is used. The security of the system is not compromised, as
only specific variables are considered for definition in this manner, not any environment variables with the _CONDOR_
prefix.
The location of the single initial configuration file differs on Windows from Unix platforms. For Unix platforms,
the location of the single initial configuration file starts at the top of the following list. The first file that exists is used,
and then remaining possible file locations from this list become irrelevant.
1. the file specified by the CONDOR_CONFIG environment variable. If there is a problem reading that file, HTCondor will print an error message and exit right away.
2. /etc/condor/condor_config
3. /usr/local/etc/condor_config
4. ˜condor/condor_config
For Windows platforms, the location of the single initial configuration file is determined by the contents of the
environment variable CONDOR_CONFIG. If this environment variable is not defined, then the location is the registry
value of HKEY_LOCAL_MACHINE/Software/Condor/CONDOR_CONFIG.
HTCondor Version 8.6.4 Manual
186
3.3.3. Configuration File Macros
187
The single, initial configuration file may contain the specification of one or more other configuration files, referred
to here as local configuration files. Since more than one file may contain a definition of the same variable, and since
the last definition of a variable sets the value, the parse order of these local configuration files is fully specified here.
In order:
1. The value of configuration variable LOCAL_CONFIG_DIR lists one or more directories which contain configuration files. The list is parsed from left to right. The leftmost (first) in the list is parsed first. Within each
directory, a lexicographical ordering by file name determines the ordering of file consideration.
2. The value of configuration variable LOCAL_CONFIG_FILE lists one or more configuration files. These listed
files are parsed from left to right. The leftmost (first) in the list is parsed first.
3. If one of these steps changes the value (right hand side) of LOCAL_CONFIG_DIR, then LOCAL_CONFIG_DIR
is processed for a second time, using the changed list of directories.
The parsing and use of configuration files may be bypassed by setting environment variable CONDOR_CONFIG
with the string ONLY_ENV. With this setting, there is no attempt to locate or read configuration files. This may be
useful for testing where the environment contains all needed information.
3.3.3 Configuration File Macros
Macro definitions are of the form:
<macro_name> = <macro_definition>
The macro name given on the left hand side of the definition is a case insensitive identifier. There may be white
space between the macro name, the equals sign (=), and the macro definition. The macro definition is a string literal
that may utilize macro substitution.
Macro invocations are of the form:
$(macro_name[:<default if macro_name not defined>])
The colon and default are optional in a macro invocation. Macro definitions may contain references to other
macros, even ones that are not yet defined, as long as they are eventually defined in the configuration files. All macro
expansion is done after all configuration files have been parsed, with the exception of macros that reference themselves.
A = xxx
C = $(A)
is a legal set of macro definitions, and the resulting value of C is xxx. Note that C is actually bound to $(A), not its
value.
As a further example,
HTCondor Version 8.6.4 Manual
3.3.3. Configuration File Macros
188
A = xxx
C = $(A)
A = yyy
is also a legal set of macro definitions, and the resulting value of C is yyy.
A macro may be incrementally defined by invoking itself in its definition. For example,
A
B
A
A
=
=
=
=
xxx
$(A)
$(A)yyy
$(A)zzz
is a legal set of macro definitions, and the resulting value of A is xxxyyyzzz. Note that invocations of a macro in
its own definition are immediately expanded. $(A) is immediately expanded in line 3 of the example. If it were not,
then the definition would be impossible to evaluate.
Recursively defined macros such as
A = $(B)
B = $(A)
are not allowed. They create definitions that HTCondor refuses to parse.
A macro invocation where the macro name is not defined results in a substitution of the empty string. Consider the
example
MAX_ALLOC_CPUS = $(NUMCPUS)-1
If NUMCPUS is not defined, then this macro substitution becomes
MAX_ALLOC_CPUS = -1
The default value may help to avoid this situation. The default value may be a literal
MAX_ALLOC_CPUS = $(NUMCPUS:4)-1
such that if NUMCPUS is not defined, the result of macro substitution becomes
MAX_ALLOC_CPUS = 4-1
The default may be another macro invocation:
MAX_ALLOC_CPUS = $(NUMCPUS:$(DETECTED_CPUS))-1
HTCondor Version 8.6.4 Manual
3.3.3. Configuration File Macros
189
These default specifications are restricted such that a macro invocation with a default can not be nested inside of
another default. An alternative way of stating this restriction is that there can only be one colon character per line. The
effect of nested defaults can be achieved by placing the macro definitions on separate lines of the configuration.
All entries in a configuration file must have an operator, which will be an equals sign (=). Identifiers are alphanumerics combined with the underscore character, optionally with a subsystem name and a period as a prefix. As a
special case, a line without an operator that begins with a left square bracket will be ignored. The following two-line
example treats the first line as a comment, and correctly handles the second line.
[HTCondor Settings]
my_classad = [ foo=bar ]
To simplify pool administration, any configuration variable name may be prefixed by a subsystem (see the
$(SUBSYSTEM) macro in section 3.5.1 for the list of subsystems) and the period (.) character. For configuration variables defined this way, the value is applied to the specific subsystem. For example, the ports that HTCondor
may use can be restricted to a range using the HIGHPORT and LOWPORT configuration variables.
MASTER.LOWPORT
MASTER.HIGHPORT
= 20000
= 20100
Note that all configuration variables may utilize this syntax, but nonsense configuration variables may result. For
example, it makes no sense to define
NEGOTIATOR.MASTER_UPDATE_INTERVAL = 60
since the condor_negotiator daemon does not use the MASTER_UPDATE_INTERVAL variable.
It makes little sense to do so, but HTCondor will configure correctly with a definition such as
MASTER.MASTER_UPDATE_INTERVAL = 60
The condor_master uses this configuration variable, and the prefix of MASTER. causes this configuration to be specific
to the condor_master daemon.
As of HTCondor version 8.1.1, evaluation works in the expected manner when combining the definition of a macro
with use of a prefix that gives the subsystem name and a period. Consider the example
FILESPEC = A
MASTER.FILESPEC = B
combined with a later definition that incorporates FILESPEC in a macro:
USEFILE = mydir/$(FILESPEC)
HTCondor Version 8.6.4 Manual
3.3.3. Configuration File Macros
190
When the condor_master evaluates variable USEFILE, it evaluates to mydir/B. Previous to HTCondor version
8.1.1, it evaluated to mydir/A. When any other subsystem evaluates variable USEFILE, it evaluates to mydir/A.
This syntax has been further expanded to allow for the specification of a local name on the command line using
the command line option
-local-name <local-name>
This allows multiple instances of a daemon to be run by the same condor_master daemon, each instance with its own
local configuration variable.
The ordering used to look up a variable, called <parameter name>:
1. <subsystem name>.<local name>.<parameter name>
2. <local name>.<parameter name>
3. <subsystem name>.<parameter name>
4. <parameter name>
If this local name is not specified on the command line, numbers 1 and 2 are skipped. As soon as the first match is
found, the search is completed, and the corresponding value is used.
This example configures a condor_master to run 2 condor_schedd daemons. The condor_master daemon needs
the configuration:
XYZZY
XYZZY_ARGS
DAEMON_LIST
DC_DAEMON_LIST
XYZZY_LOG
=
=
=
=
=
$(SCHEDD)
-local-name xyzzy
$(DAEMON_LIST) XYZZY
+ XYZZY
$(LOG)/SchedLog.xyzzy
Using this example configuration, the condor_master starts up a second condor_schedd daemon, where this second
condor_schedd daemon is passed -local-name xyzzy on the command line.
Continuing the example, configure the condor_schedd daemon named xyzzy. This condor_schedd daemon will
share all configuration variable definitions with the other condor_schedd daemon, except for those specified separately.
SCHEDD.XYZZY.SCHEDD_NAME = XYZZY
SCHEDD.XYZZY.SCHEDD_LOG = $(XYZZY_LOG)
SCHEDD.XYZZY.SPOOL
= $(SPOOL).XYZZY
Note that the example SCHEDD_NAME and SPOOL are specific to the condor_schedd daemon, as opposed to a
different daemon such as the condor_startd. Other HTCondor daemons using this feature will have different requirements for which parameters need to be specified individually. This example works for the condor_schedd, and more
local configuration can, and likely would be specified.
HTCondor Version 8.6.4 Manual
3.3.4. Comments and Line Continuations
191
Also note that each daemon’s log file must be specified individually, and in two places: one specification is for use
by the condor_master, and the other is for use by the daemon itself. In the example, the XYZZY condor_schedd configuration variable SCHEDD.XYZZY.SCHEDD_LOG definition references the condor_master daemon’s XYZZY_LOG.
3.3.4 Comments and Line Continuations
An HTCondor configuration file may contain comments and line continuations. A comment is any line beginning
with a pound character (#). A continuation is any entry that continues across multiples lines. Line continuation
is accomplished by placing the backslash character (\) at the end of any line to be continued onto another. Valid
examples of line continuation are
START = (KeyboardIdle > 15 * $(MINUTE)) && \
((LoadAvg - CondorLoadAvg) <= 0.3)
and
ADMIN_MACHINES = condor.cs.wisc.edu, raven.cs.wisc.edu, \
stork.cs.wisc.edu, ostrich.cs.wisc.edu, \
bigbird.cs.wisc.edu
HOSTALLOW_ADMINISTRATOR = $(ADMIN_MACHINES)
Where a line continuation character directly precedes a comment, the entire comment line is ignored, and the
following line is used in the continuation. Line continuation characters within comments are ignored.
Both this example
A = $(B) \
# $(C)
$(D)
and this example
A = $(B) \
# $(C) \
$(D)
result in the same value for A:
A = $(B) $(D)
3.3.5 Multi-Line Values
As of version 8.5.6, the value for a macro can comprise multiple lines of text. The syntax for this is as follows:
HTCondor Version 8.6.4 Manual
3.3.6. Executing a Program to Produce Configuration Macros
<macro_name> @=<tag>
<macro_definition lines>
@<tag>
For example:
JOB_ROUTER_DEFAULTS @=jrd
[
requirements=target.WantJobRouter is True;
MaxIdleJobs = 10;
MaxJobs = 200;
/* now modify routed job attributes */
/* remove routed job if it goes on hold or stays idle for over 6 hours */
set_PeriodicRemove = JobStatus == 5 ||
(JobStatus == 1 && (time() - QDate) > 3600*6);
delete_WantJobRouter = true;
set_requirements = true;
]
@jrd
Note that in this example, the square brackets are part of the JOB_ROUTER_DEFAULTS value.
3.3.6 Executing a Program to Produce Configuration Macros
Instead of reading from a file, HTCondor can run a program to obtain configuration macros. The vertical bar character
(| ) as the last character defining a file name provides the syntax necessary to tell HTCondor to run a program. This syntax may only be used in the definition of the CONDOR_CONFIG environment variable, or the LOCAL_CONFIG_FILE
configuration variable.
The command line for the program is formed by the characters preceding the vertical bar character. The standard
output of the program is parsed as a configuration file would be.
An example:
LOCAL_CONFIG_FILE = /bin/make_the_config|
Program /bin/make_the_config is executed, and its output is the set of configuration macros.
Note that either a program is executed to generate the configuration macros or the configuration is read from one
or more files. The syntax uses space characters to separate command line elements, if an executed program produces
the configuration macros. Space characters would otherwise separate the list of files. This syntax does not permit
distinguishing one from the other, so only one may be specified.
(Note that the include command syntax (see below) is now the preferred way to execute a program to generate
configuration macros.)
HTCondor Version 8.6.4 Manual
192
3.3.7. Including Configuration from Elsewhere
3.3.7 Including Configuration from Elsewhere
Externally defined configuration can be incorporated using the following syntax:
include [ifexist] : <file>
include : <cmdline>|
include [ifexist] command [into <cache-file>] : <cmdline>
(Note that the ifexist and into options were added in version 8.5.7. Also note that the command option must
be specified in order to use the into option – just using the bar after <cmdline> will not work.)
In the file form of the include command, the <file> specification must describe a single file, the contents
of which will be parsed and incorporated into the configuration. Unless the ifexist option is specified, the nonexistence of the file is a fatal error.
In the command line form of the include command (specified with either the command option or by appending
a bar (|) character after the <cmdline> specification), the <cmdline> specification must describe a command line
(program and arguments); the command line will be executed, and the output will be parsed and incorporated into the
configuration.
If the into option is not used, the command line will be executed every time the configuration file is referenced.
This may well be undesirable, and can be avoided by using the into option. The into keyword must be followed
by the full pathname of a file into which to write the output of the command line. If that file exists, it will be read and
the command line will not be executed. If that file does not exist, the output of the command line will be written into it
and then the cache file will be read and incorporated into the configuration. If the command line produces no output,
a zero length file will be created. If the command line returns a non-zero exit code, configuration will abort and the
cache file will not be created unless the ifexist keyword is also specified.
The include key word is case insensitive. There are no requirements for white space characters surrounding the
colon character.
Consider the example
FILE = config.$(FULL_HOSTNAME)
include : $(LOCAL_DIR)/$(FILE)
Values are acquired for configuration variables FILE, and LOCAL_DIR by immediate evaluation, causing variable
FULL_HOSTNAME to also be immediately evaluated. The resulting value forms a full path and file name. This file is
read and parsed. The resulting configuration is incorporated into the current configuration. This resulting configuration
may contain further nested include specifications, which are also parsed, evaluated, and incorporated. Levels of
nested includes are limited, such that infinite nesting is discovered and thwarted, while still permitting nesting.
Consider the further example
SCRIPT_FILE = script.$(IP_ADDRESS)
include : $(RELEASE_DIR)/$(SCRIPT_FILE) |
HTCondor Version 8.6.4 Manual
193
3.3.8. Reporting Errors and Warnings
194
In this example, the bar character at the end of the line causes a script to be invoked, and the output of the script is
incorporated into the current configuration. The same immediate parsing and evaluation occurs in this case as when a
file’s contents are included.
For pools that are transitioning to using this new syntax in configuration, while still having some tools and daemons
with HTCondor versions earlier than 8.1.6, special syntax in the configuration will cause those daemons to fail upon
startup, rather than continuing, but incorrectly parsing the new syntax. Newer daemons will ignore the extra syntax.
Placing the @ character before the include key word causes the older daemons to fail when they attempt to parse
this syntax.
Here is the same example, but with the syntax that causes older daemons to fail when reading it.
FILE = config.$(FULL_HOSTNAME)
@include : $(LOCAL_DIR)/$(FILE)
A daemon older than version 8.1.6 will fail to start. Running an older condor_config_val identifies the @include
line as being bad. A daemon of HTCondor version 8.1.6 or more recent sees:
FILE = config.$(FULL_HOSTNAME)
include : $(LOCAL_DIR)/$(FILE)
and starts up successfully.
Here is an example using the new ifexist and into options:
# stuff.pl writes "STUFF=1" to stdout
include ifexist command into $(LOCAL_DIR)/stuff.config : perl $(LOCAL_DIR)/stuff.pl
3.3.8 Reporting Errors and Warnings
As of version 8.5.7, warning and error messages can be included in HTCondor configuration files.
The syntax for warning and error messages is as follows:
warning : <warning message>
error : <error message>
The warning and error messages will be printed when the configuration file is used (when almost any HTCondor
command is run, for example). Error messages (unlike warnings) will prevent the successful use of the configuration
file. This will, for example, prevent a daemon from starting, and prevent condor_config_val from returning a value.
Here’s an example of using an error message in a configuration file (combined with some of the new include
features documented above):
HTCondor Version 8.6.4 Manual
3.3.9. Conditionals in Configuration
195
# stuff.pl writes "STUFF=1" to stdout
include command into $(LOCAL_DIR)/stuff.config : perl $(LOCAL_DIR)/stuff.pl
if ! defined stuff
error : stuff is needed!
endif
3.3.9 Conditionals in Configuration
Conditional if/else semantics are available in a limited form. The syntax:
if <simple condition>
<statement>
. . .
<statement>
else
<statement>
. . .
<statement>
endif
An else key word and statements are not required, such that simple if semantics are implemented. The
<simple condition> does not permit compound conditions. It optionally contains the exclamation point character (!) to represent the not operation, followed by
• the defined keyword followed by the name of a variable. If the variable is defined, the statement(s) are
incorporated into the expanded input. If the variable is not defined, the statement(s) are not incorporated into
the expanded input. As an example,
if defined MY_UNDEFINED_VARIABLE
X = 12
else
X = -1
endif
results in X = -1, when MY_UNDEFINED_VARIABLE is not yet defined.
• the version keyword, representing the version number of of the daemon or tool currently reading this conditional. This keyword is followed by an HTCondor version number. That version number can be of the form
x.y.z or x.y. The version of the daemon or tool is compared to the specified version number. The comparison
operators are
– == for equality. Current version 8.2.3 is equal to 8.2.
– >= to see if the current version number is greater than or equal to. Current version 8.2.3 is greater than
8.2.2, and current version 8.2.3 is greater than or equal to 8.2.
HTCondor Version 8.6.4 Manual
3.3.9. Conditionals in Configuration
196
– <= to see if the current version number is less than or equal to. Current version 8.2.0 is less than 8.2.2, and
current version 8.2.3 is less than or equal to 8.2.
As an example,
if version >= 8.1.6
DO_X = True
else
DO_Y = True
endif
results in defining DO_X as True if the current version of the daemon or tool reading this if statement is 8.1.6
or a more recent version.
• True or yes or the value 1. The statement(s) are incorporated.
• False or no or the value 0 The statement(s) are not incorporated.
• $(<variable>) may be used where the immediately evaluated value is a simple boolean value. A value that
evaluates to the empty string is considered False, otherwise a value that does not evaluate to a simple boolean
value is a syntax error.
The syntax
if <simple condition>
<statement>
. . .
<statement>
elif <simple condition>
<statement>
. . .
<statement>
endif
is the same as syntax
if <simple condition>
<statement>
. . .
<statement>
else
if <simple condition>
<statement>
. . .
<statement>
endif
endif
HTCondor Version 8.6.4 Manual
3.3.10. Function Macros in Configuration
197
3.3.10 Function Macros in Configuration
A set of predefined functions increase flexibility. Both submit description files and configuration files are read using
the same parser, so these functions may be used in both submit description files and configuration files.
Case is significant in the function’s name, so use the same letter case as given in these definitions.
$CHOICE(index, listname) or $CHOICE(index, item1, item2, . . .) An item within the list is returned. The list is represented by a parameter name, or the list items are the parameters. The index parameter
determines which item. The first item in the list is at index 0. If the index is out of bounds for the list contents,
an error occurs.
$ENV(environment-variable-name[:default-value]) Evaluates to the value of environment variable
environment-variable-name. If there is no environment variable with that name, Evaluates to UNDEFINED unless the optional :default-value is used; in which case it evaluates to default-value. For
example,
A = $ENV(HOME)
binds A to the value of the HOME environment variable.
$F[fpduwnxbqa](filename) One or more of the lower case letters may be combined to form the function
name and thus, its functionality. Each letter operates on the filename in its own way.
• f convert relative path to full path by prefixing the current working directory to it. This option works only
in condor_submit files.
• p refers to the entire directory portion of filename, with a trailing slash or backslash character. Whether
a slash or backslash is used depends on the platform of the machine. The slash will be recognized on Linux
platforms; either a slash or backslash will be recognized on Windows platforms, and the parser will use
the same character specified.
• d refers to the last portion of the directory within the path, if specified. It will have a trailing slash or
backslash, as appropriate to the platform of the machine. The slash will be recognized on Linux platforms;
either a slash or backslash will be recognized on Windows platforms, and the parser will use the same
character specified unless u or w is used. if b is used the trailing slash or backslash will be omitted.
• u convert path separators to Unix style slash characters
• w convert path separators to Windows style backslash characters
• n refers to the file name at the end of any path, but without any file name extension. As an example, the
return value from $Fn(/tmp/simulate.exe) will be simulate (without the .exe extension).
• x refers to a file name extension, with the associated period (.). As an example, the return value from
$Fn(/tmp/simulate.exe) will be .exe.
• b when combined with the d option, causes the trailing slash or backslash to be omitted. When combined
with the x option, causes the leading period (.) to be omitted.
• q causes the return value to be enclosed within quotes. Double quote marks are used unless a is also
specified.
HTCondor Version 8.6.4 Manual
3.3.10. Function Macros in Configuration
198
• a When combined with the q option, causes the return value to be enclosed within single quotes.
$DIRNAME(filename) is the same as $Fp(filename)
$BASENAME(filename) is the same as $Fnx(filename)
$INT(item-to-convert) or $INT(item-to-convert, format-specifier) Expands, evaluates,
and returns a string version of item-to-convert. The format-specifier has the same syntax as a
C language or Perl format specifier. If no format-specifier is specified, "%d" is used as the format
specifier.
$RANDOM_CHOICE(choice1, choice2, choice3, . . .) A random choice of one of the parameters in the
list of parameters is made. For example, if one of the integers 0-8 (inclusive) should be randomly chosen:
$RANDOM_CHOICE(0,1,2,3,4,5,6,7,8)
$RANDOM_INTEGER(min, max [, step]) A random integer within the range min and max, inclusive, is
selected. The optional step parameter controls the stride within the range, and it defaults to the value 1. For
example, to randomly chose an even integer in the range 0-8 (inclusive):
$RANDOM_INTEGER(0, 8, 2)
$REAL(item-to-convert) or $REAL(item-to-convert, format-specifier) Expands, evaluates,
and returns a string version of item-to-convert for a floating point type. The format-specifier is
a C language or Perl format specifier. If no format-specifier is specified, "%16G" is used as a format
specifier.
$SUBSTR(name, start-index) or $SUBSTR(name, start-index, length) Expands name and returns a substring of it. The first character of the string is at index 0. The first character of the substring is at
index start-index. If the optional length is not specified, then the substring includes characters up to the
end of the string. A negative value of start-index works back from the end of the string. A negative value
of length eliminates use of characters from the end of the string. Here are some examples that all assume
Name = abcdef
• $SUBSTR(Name, 2) is cdef.
• $SUBSTR(Name, 0, -2) is abcd.
• $SUBSTR(Name, 1, 3) is bcd.
• $SUBSTR(Name, -1) is f.
• $SUBSTR(Name, 4, -3) is the empty string, as there are no characters in the substring for this request.
Environment references are not currently used in standard HTCondor configurations. However, they can sometimes
be useful in custom configurations.
HTCondor Version 8.6.4 Manual
3.3.11. Macros That Will Require a Restart When Changed
199
3.3.11 Macros That Will Require a Restart When Changed
When any of the following listed configuration variables are changed, HTCondor must be restarted. Reconfiguration
using condor_reconfig will not be enough.
• BIND_ALL_INTERFACES
• FetchWorkDelay
• MAX_NUM_CPUS
• MAX_TRACKING_GID
• MEMORY
• MIN_TRACKING_GID
• NETWORK_HOSTNAME
• NETWORK_INTERFACE
• NUM_CPUS
• PREEMPTION_REQUIREMENTS_STABLE
• PRIVSEP_ENABLED
• PROCD_ADDRESS
• SLOT_TYPE_<N>
• OFFLINE_MACHINE_RESOURCE_<name>
3.3.12 Pre-Defined Macros
HTCondor provides pre-defined macros that help configure HTCondor.
$(macro_name).
Pre-defined macros are listed as
This first set are entries whose values are determined at run time and cannot be overwritten. These are inserted
automatically by the library routine which parses the configuration files. This implies that a change to the underlying
value of any of these variables will require a full restart of HTCondor in order to use the changed value.
$(FULL_HOSTNAME) The fully qualified host name of the local machine, which is host name plus domain name.
$(HOSTNAME) The host name of the local machine, without a domain name.
HTCondor Version 8.6.4 Manual
3.3.12. Pre-Defined Macros
200
$(IP_ADDRESS) The ASCII string version of the local machine’s “most public” IP address. This address may be
IPv4 or IPv6, but the macro will always be set.
HTCondor selects the “most public” address heuristically. Your configuration should not depend on HTCondor
picking any particular IP address for this macro; this macro’s value may not even be one of the IP addresses
HTCondor is configured to advertise.
labelparam:IPv4Address
$(IPV4_ADDRESS) The ASCII string version of the local machine’s “most public” IPv4 address; unset if the local
machine has no IPv4 address.
See IP_ADDRESS about “most public”.
$(IPV6_ADDRESS) The ASCII string version of the local machine’s “most public” IPv6 address; unset if the local
machine has no IPv6 address.
See IP_ADDRESS about “most public”.
$(IP_ADDRESS_IS_V6) A boolean which is true if and only if IP_ADDRESS is an IPv6 address. Useful for
conditonal configuration.
$(TILDE) The full path to the home directory of the Unix user condor, if such a user exists on the local machine.
$(SUBSYSTEM) The subsystem name of the daemon or tool that is evaluating the macro. This is a unique string
which identifies a given daemon within the HTCondor system. The possible subsystem names are:
• C_GAHP
• C_GAHP_WORKER_THREAD
• CKPT_SERVER
• COLLECTOR
• DBMSD
• DEFRAG
• EC2_GAHP
• GANGLIAD
• GCE_GAHP
• GRIDMANAGER
• HAD
• HDFS
• JOB_ROUTER
• KBDD
• LEASEMANAGER
• MASTER
• NEGOTIATOR
• QUILL
HTCondor Version 8.6.4 Manual
3.3.12. Pre-Defined Macros
201
• REPLICATION
• ROOSTER
• SCHEDD
• SHADOW
• SHARED_PORT
• STARTD
• STARTER
• SUBMIT
• TOOL
• TRANSFERER
$(DETECTED_CPUS) The integer number of hyper-threaded CPUs, as given by $(DETECTED_CORES), when
COUNT_HYPERTHREAD_CPUS is True. The integer number of physical (non hyper-threaded) CPUs,
as given by $(DETECTED_PHYSICAL_CPUS), when COUNT_HYPERTHREAD_CPUS is False. When
COUNT_HYPERTHREAD_CPUS is True.
$(DETECTED_PHYSICAL_CPUS) The integer number of physical (non hyper-threaded) CPUs. This will be equal
the number of unique CPU IDs.
This second set of macros are entries whose default values are determined automatically at run time but which can
be overwritten.
$(ARCH) Defines the string used to identify the architecture of the local machine to HTCondor. The condor_startd
will advertise itself with this attribute so that users can submit binaries compiled for a given platform and force
them to run on the correct machines. condor_submit will append a requirement to the job ClassAd that it must
run on the same ARCH and OPSYS of the machine where it was submitted, unless the user specifies ARCH and/or
OPSYS explicitly in their submit file. See the condor_submit manual page on page 911 for details.
$(OPSYS) Defines the string used to identify the operating system of the local machine to HTCondor. If it is not
defined in the configuration file, HTCondor will automatically insert the operating system of this machine as
determined by uname.
$(OPSYS_VER) Defines the integer used to identify the operating system version number.
$(OPSYS_AND_VER) Defines the string used prior to HTCondor version 7.7.2 as $(OPSYS).
$(UNAME_ARCH) The architecture as reported by uname(2)’s machine field. Always the same as ARCH on Windows.
$(UNAME_OPSYS) The operating system as reported by uname(2)’s sysname field. Always the same as OPSYS
on Windows.
$(DETECTED_MEMORY) The amount of detected physical memory (RAM) in MiB.
$(DETECTED_CORES) The number of CPU cores that the operating system schedules. On machines that support
hyper-threading, this will be the number of hyper-threads.
HTCondor Version 8.6.4 Manual
3.4. Configuration Templates
202
$(PID) The process ID for the daemon or tool.
$(PPID) The process ID of the parent process for the daemon or tool.
$(USERNAME) The user name of the UID of the daemon or tool. For daemons started as root, but running under
another UID (typically the user condor), this will be the other UID.
$(FILESYSTEM_DOMAIN) Defaults to the fully qualified host name of the machine it is evaluated on. See section 3.5.6, Shared File System Configuration File Entries for the full description of its use and under what
conditions it could be desirable to change it.
$(UID_DOMAIN) Defaults to the fully qualified host name of the machine it is evaluated on. See section 3.5.6 for
the full description of this configuration variable.
Since $(ARCH) and $(OPSYS) will automatically be set to the correct values, we recommend that you do not
overwrite them.
3.4 Configuration Templates
Achieving certain behaviors in an HTCondor pool often requires setting the values of a number of configuration macros
in concert with each other. We have added configuration templates as a way to do this more easily, at a higher level,
without having to explicitly set each individual configuration macro.
Configuration templates are pre-defined; users cannot define their own templates.
Note that the value of an individual configuration macro that is set by a configuration template can be overridden
by setting that configuration macro later in the configuration.
Detailed information about configuration templates (such as the macros they set) can be obtained using the condor_config_val use option (see 11). (This document does not contain such information because the condor_config_val
command is a better way to obtain it.)
3.4.1 Configuration Templates: Using Predefined Sets of Configuration
Predefined sets of configuration can be identified and incorporated into the configuration using the syntax
use <category name> : <template name>
The use key word is case insensitive. There are no requirements for white space characters surrounding the colon
character. More than one <template name> identifier may be placed within a single use line. Separate the names
by a space character. There is no mechanism by which the administrator may define their own custom <category
name> or <template name>.
Each predefined <category name> has a fixed, case insensitive name for the sets of configuration that are
predefined. Placement of a use line in the configuration brings in the predefined configuration it identifies.
HTCondor Version 8.6.4 Manual
3.4.2. Available Configuration Templates
203
As of version 8.5.6, some of the configuration templates take arguments (as described below).
3.4.2 Available Configuration Templates
There are four <category name> values. Within a category, a predefined, case insensitive name identifies the set
of configuration it incorporates.
ROLE category Describes configuration for the various roles that a machine might play within an HTCondor pool.
The configuration will identify which daemons are running on a machine.
• Personal
Settings needed for when a single machine is the entire pool.
• Submit
Settings needed to allow this machine to submit jobs to the pool. May be combined with Execute and
CentralManager roles.
• Execute
Settings needed to allow this machine to execute jobs.
CentralManager roles.
May be combined with Submit and
• CentralManager
Settings needed to allow this machine to act as the central manager for the pool. May be combined with
Submit and Execute roles.
FEATURE category Describes configuration for implemented features.
• Remote_Runtime_Config
Enables the use of condor_config_val -rset to the machine with this configuration. Note that there are
security implications for use of this configuration, as it potentially permits the arbitrary modification of
configuration. Variable SETTABLE_ATTRS_CONFIG must also be defined.
• Remote_Config
Enables the use of condor_config_val -set to the machine with this configuration. Note that there are
security implications for use of this configuration, as it potentially permits the arbitrary modification of
configuration. Variable SETTABLE_ATTRS_CONFIG must also be defined.
• VMware
Enables use of the vm universe with VMware virtual machines. Note that this feature depends on Perl.
• GPUs
Sets configuration based on detection with the condor_gpu_discovery tool, and defines a custom resource
using the name GPUs. Supports both OpenCL and CUDA if detected.
• PartitionableSlot( slot_type_num [, allocation] )
Sets up a partitionable slot of the specified slot type number and allocation (defaults for slot_type_num
and allocation are 1 and 100% respectively). See 3.7.1 for information on partitionalble slot policies.
HTCondor Version 8.6.4 Manual
3.4.2. Available Configuration Templates
204
• AssignAccountingGroup( map_filename ) Sets up a condor_schedd job transform that assigns an accounting group to each job as it is submitted. The accounting is determined by mapping the
Owner attribute of the job using the given map file.
• ScheddUserMapFile( map_name, map_filename ) Defines a condor_schedd usermap
named map_name using the given map file.
• SetJobAttrFromUserMap( dst_attr, src_attr, map_name [, map_filename] )
Sets up a condor_schedd job transform that sets the dst_attr attribute of each job as it is submitted. The
value of dst_attr is determined by mapping the src_attr of the job using the usermap named map_name.
If the optional map_filename argument is specifed, then this metaknob also defines a condor_schedd
usermap named map_Name using the given map file.
• StartdCronOneShot( job_name, exe [, hook_args] )
Create a one-shot condor_startd job hook. (See 4.4.3 for more information about job hooks.)
• StartdCronPeriodic( job_name, period, exe [, hook_args] )
Create a periodic-shot condor_startd job hook. (See 4.4.3 for more information about job hooks.)
• StartdCronContinuous( job_name, exe [, hook_args] )
Create a (nearly) continuous condor_startd job hook. (See 4.4.3 for more information about job hooks.)
• ScheddCronOneShot( job_name, exe [, hook_args] )
Create a one-shot condor_schedd job hook. (See 4.4.3 for more information about job hooks.)
• ScheddCronPeriodic( job_name, period, exe [, hook_args] )
Create a periodic-shot condor_schedd job hook. (See 4.4.3 for more information about job hooks.)
• ScheddCronContinuous( job_name, exe [, hook_args] )
Create a (nearly) continuous condor_schedd job hook. (See 4.4.3 for more information about job hooks.)
• OneShotCronHook( STARTD_CRON | SCHEDD_CRON, job_name, hook_exe
[,hook_args] )
Create a one-shot job hook. (See 4.4.3 for more information about job hooks.)
• PeriodicCronHook( STARTD_CRON | SCHEDD_CRON , job_name, period,
hook_exe [,hook_args] )
Create a periodic job hook. (See 4.4.3 for more information about job hooks.)
• ContinuousCronHook( STARTD_CRON | SCHEDD_CRON , job_name, hook_exe
[,hook_args] )
Create a (nearly) continuous job hook. (See 4.4.3 for more information about job hooks.)
• UWCS_Desktop_Policy_Values
Configuration values used in the UWCS_DESKTOP policy. (Note that these values were previously in the parameter table; configuration that uses these values will have to use the
UWCS_Desktop_Policy_Values template. For example, POLICY : UWCS_Desktop uses the
FEATURE : UWCS_Desktop_Policy_Values template.)
POLICY category Describes configuration for the circumstances under which machines choose to run jobs.
HTCondor Version 8.6.4 Manual
3.4.2. Available Configuration Templates
205
• Always_Run_Jobs
Always start jobs and run them to completion, without consideration of condor_negotiator generated preemption or suspension. This is the default policy, and it is intended to be used with dedicated resources. If
this policy is used together with the Limit_Job_Runtimes policy, order the specification by placing
this Always_Run_Jobs policy first.
• UWCS_Desktop
This was the default policy before HTCondor version 8.1.6. It is intended to be used with desktop machines
not exclusively running HTCondor jobs. It injects UWCS into the name of some configuration variables.
• Desktop
An updated and reimplementation of the UWCS_Desktop policy, but without the UWCS naming of some
configuration variables.
• Limit_Job_Runtimes( limit_in_seconds )
Limits running jobs to a maximum of the specified time using preemption. (The default limit is 24 hours.)
If this policy is used together with the Always_Run_Jobs policy, order the specification by placing this
Limit_Job_Runtimes policy second.
• Preempt_If_Cpus_Exceeded
If the startd observes the number of CPU cores used by the job exceed the number of cores in the slot by
more than 0.8 on average over the past minute, preempt the job immediately ignoring any job retirement
time.
• Hold_If_Cpus_Exceeded
If the startd observes the number of CPU cores used by the job exceed the number of cores in the slot
by more than 0.8 on average over the past minute, immediately place the job on hold ignoring any job
retirement time. The job will go on hold with a reasonable hold reason in job attribute HoldReason
and a value of 101 in job attribute HoldReasonCode. The hold reason and code can be customized by
specifying HOLD_REASON_CPU_EXCEEDED and HOLD_SUBCODE_CPU_EXCEEDED respectively.
• Preempt_If_Memory_Exceeded
If the startd observes the memory usage of the job exceed the memory provisioned in the slot, preempt the
job immediately ignoring any job retirement time.
• Hold_If_Memory_Exceeded
If the startd observes the memory usage of the job exceed the memory provisioned in the slot, immediately place the job on hold ignoring any job retirement time. The job will go on hold with a reasonable hold reason in job attribute HoldReason and a value of 102 in job attribute HoldReasonCode.
The hold reason and code can be customized by specifying HOLD_REASON_MEMORY_EXCEEDED and
HOLD_SUBCODE_MEMORY_EXCEEDED respectively.
• Preempt_If( policy_variable )
Preempt jobs according to the specified policy. policy_variable must be the name of a configuration
macro containing an expression that evaluates to True if the job should be preempted.
See an example here: 3.4.4.
• Want_Hold_If( policy_variable, subcode, reason_text )
Add the given policy to the WANT_HOLD expression; if the WANT_HOLD expression is defined,
policy_variable is prepended to the existing expression; otherwise WANT_HOLD is simply set to
the value of the textttpolicy_variable macro.
See an example here: 3.4.4.
HTCondor Version 8.6.4 Manual
3.4.3. Configuration Template Transition Syntax
• Startd_Publish_CpusUsage
Publish the number of CPU cores being used by the job into to slot ad as attribute CpusUsage. This
value will be the average number of cores used by the job over the past minute, sampling every 5 seconds.
SECURITY category Describes configuration for an implemented security model.
• Host_Based
The default security model (based on IPs and DNS names). Do not combine with User_Based security.
• User_Based
Grants permissions to an administrator and uses With_Authentication. Do not combine with
Host_Based security.
• With_Authentication
Requires both authentication and integrity checks.
• Strong
Requires authentication, encryption, and integrity checks.
3.4.3 Configuration Template Transition Syntax
For pools that are transitioning to using this new syntax in configuration, while still having some tools and daemons
with HTCondor versions earlier than 8.1.6, special syntax in the configuration will cause those daemons to fail upon
start up, rather than use the new, but misinterpreted, syntax. Newer daemons will ignore the extra syntax. Placing the
@ character before the use key word causes the older daemons to fail when they attempt to parse this syntax.
As an example, consider the condor_startd as it starts up. A condor_startd previous to HTCondor version 8.1.6
fails to start when it sees:
@use feature : GPUs
Running an older condor_config_val also identifies the @use line as being bad. A condor_startd of HTCondor version
8.1.6 or more recent sees
use feature : GPUs
3.4.4 Configuration Template Examples
• Preempt a job if its memory usage exceeds the requested memory:
MEMORY_EXCEEDED = (isDefined(MemoryUsage) && MemoryUsage > RequestMemory)
use POLICY : PREEMPT_IF(MEMORY_EXCEEDED)
• Put a job on hold if its memory usage exceeds the requested memory:
HTCondor Version 8.6.4 Manual
206
3.5. Configuration Macros
207
MEMORY_EXCEEDED = (isDefined(MemoryUsage) && MemoryUsage > RequestMemory)
use POLICY : WANT_HOLD_IF(MEMORY_EXCEEDED, 102, memory usage exceeded request_memory
• Update dynamic GPU information every 15 minutes:
use FEATURE : StartdCronPeriodic(DYNGPU, 15*60, $(LOCAL_DIR)\dynamic_gpu_info.pl, $(
where dynamic_gpu_info.pl is a simple perl script that strips off the DetectedGPUs line from textttcondor_gpu_discovery:
#!/usr/bin/env perl
my @attrs = `@ARGV`;
for (@attrs) {
next if ($_ =~ /^Detected/i);
print $_;
}
3.5 Configuration Macros
The section contains a list of the individual configuraton macros for HTCondor. Before attempting to set up HTCondor
configuration, you should probably read the introduction to configuration section ( 3.3) and possibly the configuration
template section ( 3.4).
The settings that control the policy under which HTCondor will start, suspend, resume, vacate or kill jobs are
described in section 3.7 on Policy Configuration for the condor_startd, not in this section.
3.5.1 Introduction to Configuration Files
The HTCondor configuration files are used to customize how HTCondor operates at a given site. The basic configuration as shipped with HTCondor works well for most sites.
Each HTCondor program will, as part of its initialization process, configure itself by calling a library routine which
parses the various configuration files that might be used, including pool-wide, platform-specific, and machine-specific
configuration files. Environment variables may also contribute to the configuration.
The result of configuration is a list of key/value pairs. Each key is a configuration variable name, and each value is
a string literal that may utilize macro substitution (as defined below). Some configuration variables are evaluated by
HTCondor as ClassAd expressions; some are not. Consult the documentation for each specific case. Unless otherwise
noted, configuration values that are expected to be numeric or boolean constants may be any valid ClassAd expression
of operators on constants. Example:
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
208
MINUTE
= 60
HOUR
= (60 * $(MINUTE))
SHUTDOWN_GRACEFUL_TIMEOUT = ($(HOUR)*24)
Ordered Evaluation to Set the Configuration
Multiple files, as well as a program’s environment variables determine the configuration. The order in which attributes
are defined is important, as later definitions override existing definitions. The order in which the (multiple) configuration files are parsed is designed to ensure the security of the system. Attributes which must be set a specific way must
appear in the last file to be parsed. This prevents both the naive and the malicious HTCondor user from subverting the
system through its configuration. The order in which items are parsed is
1. a single initial configuration file, which has historically been known as the global configuration file
2. other configuration files that are referenced and parsed due to specification within the single initial configuration
file; these files have historically been known as local configuration files
3. If HTCondor daemons are not running as root on Unix platforms,
parse file
$(HOME)/.condor/user_config if it exists, or the file defined by configuration variable
USER_CONFIG_FILE.
If HTCondor daemons are not running as Local System on Windows platforms, parse file
%USERPROFILE\.condor\user_config if it exists, or the file defined by configuration variable
USER_CONFIG_FILE.
4. specific environment variables whose names are prefixed with _CONDOR_
Some HTCondor tools utilize environment variables to set their configuration. These tools search for specificallynamed environment variables. The variables are prefixed by the string _CONDOR_ or _condor_. The tools strip off
the prefix, and utilize what remains as configuration. As the use of environment variables is the last within the ordered
evaluation, the environment variable definition is used. The security of the system is not compromised, as only specific
variables are considered for definition in this manner, not any environment variables with the _CONDOR_ prefix.
The location of the single initial configuration file differs on Windows from Unix platforms. For Unix platforms,
the location of the single initial configuration file starts at the top of the following list. The first file that exists is used,
and then remaining possible file locations from this list become irrelevant.
1. the file specified by the CONDOR_CONFIG environment variable. If there is a problem reading that file, HTCondor will print an error message and exit right away.
2. /etc/condor/condor_config
3. /usr/local/etc/condor_config
4. ˜condor/condor_config
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
209
For Windows platforms, the location of the single initial configuration file is determined by the contents of the
environment variable CONDOR_CONFIG. If this environment variable is not defined, then the location is the registry
value of HKEY_LOCAL_MACHINE/Software/Condor/CONDOR_CONFIG.
The single, initial configuration file may contain the specification of one or more other configuration files, referred
to here as local configuration files. Since more than one file may contain a definition of the same variable, and since
the last definition of a variable sets the value, the parse order of these local configuration files is fully specified here.
In order:
1. The value of configuration variable LOCAL_CONFIG_DIR lists one or more directories which contain configuration files. The list is parsed from left to right. The leftmost (first) in the list is parsed first. Within each
directory, a lexicographical ordering by file name determines the ordering of file consideration.
2. The value of configuration variable LOCAL_CONFIG_FILE lists one or more configuration files. These listed
files are parsed from left to right. The leftmost (first) in the list is parsed first.
3. If one of these steps changes the value (right hand side) of LOCAL_CONFIG_DIR, then LOCAL_CONFIG_DIR
is processed for a second time, using the changed list of directories.
The parsing and use of configuration files may be bypassed by setting environment variable CONDOR_CONFIG
with the string ONLY_ENV. With this setting, there is no attempt to locate or read configuration files. This may be
useful for testing where the environment contains all needed information.
Configuration File Macros
Macro definitions are of the form:
<macro_name> = <macro_definition>
The macro name given on the left hand side of the definition is a case insensitive identifier. There may be white
space between the macro name, the equals sign (=), and the macro definition. The macro definition is a string literal
that may utilize macro substitution.
Macro invocations are of the form:
$(macro_name:default if macro_name not defined)
The colon and default are optional in a macro invocation. Macro definitions may contain references to other
macros, even ones that are not yet defined, as long as they are eventually defined in the configuration files. All macro
expansion is done after all configuration files have been parsed, with the exception of macros that reference themselves.
A = xxx
C = $(A)
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
210
is a legal set of macro definitions, and the resulting value of C is xxx. Note that C is actually bound to $(A), not its
value.
As a further example,
A = xxx
C = $(A)
A = yyy
is also a legal set of macro definitions, and the resulting value of C is yyy.
A macro may be incrementally defined by invoking itself in its definition. For example,
A
B
A
A
=
=
=
=
xxx
$(A)
$(A)yyy
$(A)zzz
is a legal set of macro definitions, and the resulting value of A is xxxyyyzzz. Note that invocations of a macro in
its own definition are immediately expanded. $(A) is immediately expanded in line 3 of the example. If it were not,
then the definition would be impossible to evaluate.
Recursively defined macros such as
A = $(B)
B = $(A)
are not allowed. They create definitions that HTCondor refuses to parse.
A macro invocation where the macro name is not defined results in a substitution of the empty string. Consider the
example
MAX_ALLOC_CPUS = $(NUMCPUS)-1
If NUMCPUS is not defined, then this macro substitution becomes
MAX_ALLOC_CPUS = -1
The default value may help to avoid this situation. The default value may be a literal
MAX_ALLOC_CPUS = $(NUMCPUS:4)-1
such that if NUMCPUS is not defined, the result of macro substitution becomes
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
211
MAX_ALLOC_CPUS = 4-1
The default may be another macro invocation:
MAX_ALLOC_CPUS = $(NUMCPUS:$(DETECTED_CPUS))-1
These default specifications are restricted such that a macro invocation with a default can not be nested inside of
another default. An alternative way of stating this restriction is that there can only be one colon character per line. The
effect of nested defaults can be achieved by placing the macro definitions on separate lines of the configuration.
All entries in a configuration file must have an operator, which will be an equals sign (=). Identifiers are alphanumerics combined with the underscore character, optionally with a subsystem name and a period as a prefix. As a
special case, a line without an operator that begins with a left square bracket will be ignored. The following two-line
example treats the first line as a comment, and correctly handles the second line.
[HTCondor Settings]
my_classad = [ foo=bar ]
To simplify pool administration, any configuration variable name may be prefixed by a subsystem (see the
$(SUBSYSTEM) macro in section 3.5.1 for the list of subsystems) and the period (.) character. For configuration variables defined this way, the value is applied to the specific subsystem. For example, the ports that HTCondor
may use can be restricted to a range using the HIGHPORT and LOWPORT configuration variables.
MASTER.LOWPORT
MASTER.HIGHPORT
= 20000
= 20100
Note that all configuration variables may utilize this syntax, but nonsense configuration variables may result. For
example, it makes no sense to define
NEGOTIATOR.MASTER_UPDATE_INTERVAL = 60
since the condor_negotiator daemon does not use the MASTER_UPDATE_INTERVAL variable.
It makes little sense to do so, but HTCondor will configure correctly with a definition such as
MASTER.MASTER_UPDATE_INTERVAL = 60
The condor_master uses this configuration variable, and the prefix of MASTER. causes this configuration to be specific
to the condor_master daemon.
As of HTCondor version 8.1.1, evaluation works in the expected manner when combining the definition of a macro
with use of a prefix that gives the subsystem name and a period. Consider the example
FILESPEC = A
MASTER.FILESPEC = B
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
212
combined with a later definition that incorporates FILESPEC in a macro:
USEFILE = mydir/$(FILESPEC)
When the condor_master evaluates variable USEFILE, it evaluates to mydir/B. Previous to HTCondor version
8.1.1, it evaluated to mydir/A. When any other subsystem evaluates variable USEFILE, it evaluates to mydir/A.
This syntax has been further expanded to allow for the specification of a local name on the command line using
the command line option
-local-name <local-name>
This allows multiple instances of a daemon to be run by the same condor_master daemon, each instance with its own
local configuration variable.
The ordering used to look up a variable, called <parameter name>:
1. <subsystem name>.<local name>.<parameter name>
2. <local name>.<parameter name>
3. <subsystem name>.<parameter name>
4. <parameter name>
If this local name is not specified on the command line, numbers 1 and 2 are skipped. As soon as the first match is
found, the search is completed, and the corresponding value is used.
This example configures a condor_master to run 2 condor_schedd daemons. The condor_master daemon needs
the configuration:
XYZZY
XYZZY_ARGS
DAEMON_LIST
DC_DAEMON_LIST
XYZZY_LOG
=
=
=
=
=
$(SCHEDD)
-local-name xyzzy
$(DAEMON_LIST) XYZZY
+ XYZZY
$(LOG)/SchedLog.xyzzy
Using this example configuration, the condor_master starts up a second condor_schedd daemon, where this second
condor_schedd daemon is passed -local-name xyzzy on the command line.
Continuing the example, configure the condor_schedd daemon named xyzzy. This condor_schedd daemon will
share all configuration variable definitions with the other condor_schedd daemon, except for those specified separately.
SCHEDD.XYZZY.SCHEDD_NAME = XYZZY
SCHEDD.XYZZY.SCHEDD_LOG = $(XYZZY_LOG)
SCHEDD.XYZZY.SPOOL
= $(SPOOL).XYZZY
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
213
Note that the example SCHEDD_NAME and SPOOL are specific to the condor_schedd daemon, as opposed to a
different daemon such as the condor_startd. Other HTCondor daemons using this feature will have different requirements for which parameters need to be specified individually. This example works for the condor_schedd, and more
local configuration can, and likely would be specified.
Also note that each daemon’s log file must be specified individually, and in two places: one specification is for use
by the condor_master, and the other is for use by the daemon itself. In the example, the XYZZY condor_schedd configuration variable SCHEDD.XYZZY.SCHEDD_LOG definition references the condor_master daemon’s XYZZY_LOG.
Comments and Line Continuations
An HTCondor configuration file may contain comments and line continuations. A comment is any line beginning
with a pound character (#). A continuation is any entry that continues across multiples lines. Line continuation
is accomplished by placing the backslash character (\) at the end of any line to be continued onto another. Valid
examples of line continuation are
START = (KeyboardIdle > 15 * $(MINUTE)) && \
((LoadAvg - CondorLoadAvg) <= 0.3)
and
ADMIN_MACHINES = condor.cs.wisc.edu, raven.cs.wisc.edu, \
stork.cs.wisc.edu, ostrich.cs.wisc.edu, \
bigbird.cs.wisc.edu
HOSTALLOW_ADMINISTRATOR = $(ADMIN_MACHINES)
Where a line continuation character directly precedes a comment, the entire comment line is ignored, and the
following line is used in the continuation. Line continuation characters within comments are ignored.
Both this example
A = $(B) \
# $(C)
$(D)
and this example
A = $(B) \
# $(C) \
$(D)
result in the same value for A:
A = $(B) $(D)
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
214
Executing a Program to Produce Configuration Macros
Instead of reading from a file, HTCondor may run a program to obtain configuration macros. The vertical bar character
(| ) as the last character defining a file name provides the syntax necessary to tell HTCondor to run a program. This syntax may only be used in the definition of the CONDOR_CONFIG environment variable, or the LOCAL_CONFIG_FILE
configuration variable.
The command line for the program is formed by the characters preceding the vertical bar character. The standard
output of the program is parsed as a configuration file would be.
An example:
LOCAL_CONFIG_FILE = /bin/make_the_config|
Program /bin/make_the_config is executed, and its output is the set of configuration macros.
Note that either a program is executed to generate the configuration macros or the configuration is read from one
or more files. The syntax uses space characters to separate command line elements, if an executed program produces
the configuration macros. Space characters would otherwise separate the list of files. This syntax does not permit
distinguishing one from the other, so only one may be specified.
Including Configuration from Elsewhere
Externally defined configuration can be incorporated using the syntax
include : <what-to-include>
The <what-to-include> specification may describe a single file, where the contents of the file will be parsed
and incorporated into the configuration. Or, <what-to-include> may cause a program to be executed, where the
output of the program is parsed and incorporated into the configuration.
The include key word is case insensitive. There are no requirements for white space characters surrounding the
colon character.
Consider the example
FILE = config.$(FULL_HOSTNAME)
include : $(LOCAL_DIR)/$(FILE)
Values are acquired for configuration variables FILE, and LOCAL_DIR by immediate evaluation, causing variable
FULL_HOSTNAME to also be immediately evaluated. The resulting value forms a full path and file name. This file is
read and parsed. The resulting configuration is incorporated into the current configuration. This resulting configuration
may contain further nested include specifications, which are also parsed, evaluated, and incorporated. Levels of
nested includes are limited, such that infinite nesting is discovered and thwarted, while still permitting nesting.
Consider the further example
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
215
SCRIPT_FILE = script.$(IP_ADDRESS)
include : $(RELEASE_DIR)/$(SCRIPT_FILE) |
In this example, the bar character at the end of the line causes a script to be invoked, and the output of the script is
incorporated into the current configuration. The same immediate parsing and evaluation occurs in this case as when a
file’s contents are included.
For pools that are transitioning to using this new syntax in configuration, while still having some tools and daemons
with HTCondor versions earlier than 8.1.6, special syntax in the configuration will cause those daemons to fail upon
start up, rather than use the new, but misinterpreted, syntax. Newer daemons will ignore the extra syntax. Placing the
@ character before the include key word causes the older daemons to fail when they attempt to parse this syntax.
Here is the same example, but with the syntax that causes older daemons to fail when reading it.
FILE = config.$(FULL_HOSTNAME)
@include : $(LOCAL_DIR)/$(FILE)
A daemon previous to HTCondor version 8.1.6 fails to start. Running an older condor_config_val identifies the
@include line as being bad. A daemon of HTCondor version 8.1.6 or more recent sees
FILE = config.$(FULL_HOSTNAME)
include : $(LOCAL_DIR)/$(FILE)
Metaknobs: Using Predefined Sets of Configuration
Predefined sets of configuration may be identified and incorporated into the configuration using the syntax
use <category id> : <name of set>
The use key word is case insensitive. There are no requirements for white space characters surrounding the colon
character. More than one <name of set> identifier may be placed within a single use line. Separate the names
by a space character. There is no mechanism by which the administrator may define their own custom <category
id> or <name of set>.
Each predefined <category id> has a fixed, case insensitive name for the sets of configuration that are predefined. Placement of a use line in the configuration brings in the predefined configuration it identifies.
There are four <category id> values. Within a category, a predefined, case insensitive name identifies the set
of configuration it incorporates.
ROLE Describes configuration for the various roles that a machine might play within an HTCondor pool. The configuration will identify which daemons are running on a machine.
• Personal Settings needed for when a single machine is the entire pool.
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
216
• Submit Settings needed to allow this machine to submit jobs to the pool. May be combined with
Execute and CentralManager roles.
• Execute Settings needed to allow this machine to execute jobs. May be combined with Submit and
CentralManager roles.
• CentralManager Settings needed to allow this machine to act as the central manager for the pool. May
be combined with Submit and Execute roles.
FEATURE Describes configuration for implemented features.
• Remote_Runtime_Config Enables the use of condor_config_val -rset to the machine with this configuration. Note that there are security implications for use of this configuration, as it potentially permits
the arbitrary modification of configuration. Variable SETTABLE_ATTRS_CONFIG must also be defined.
• Remote_Config Enables the use of condor_config_val -set to the machine with this configuration.
Note that there are security implications for use of this configuration, as it potentially permits the arbitrary
modification of configuration. Variable SETTABLE_ATTRS_CONFIG must also be defined.
• VMware Enables use of the vm universe with VMware virtual machines. Note that this feature depends
on Perl.
• GPUs Sets configuration based on detection with the condor_gpu_discovery tool, and defines a custom
resource using the name GPUs. Supports both OpenCL and CUDA if detected.
POLICY Describes configuration for the circumstances under which machines choose to run jobs.
• Always_Run_Jobs Always start jobs and run them to completion, without consideration of condor_negotiator generated preemption or suspension. This is the default policy, and it is intended to be
used with dedicated resources. If this policy is used together with the Limit_Job_Runtimes policy,
order the specification by placing this Always_Run_Jobs policy first.
• UWCS_Desktop This was the default policy before HTCondor version 8.1.6. It is intended to be used
with desktop machines not exclusively running HTCondor jobs. It injects UWCS into the name of some
configuration variables.
• Desktop An updated and reimplementation of the UWCS_Desktop policy, but without the UWCS naming of some configuration variables.
• Limit_Job_Runtimes Limits running jobs to a maximum of 24 hours using preemption. To
set the limit to a different amount of time, define configuration variable MAX_JOB_RUNTIME
with the desired limit in seconds; place this definition of MAX_JOB_RUNTIME after the
use POLICY : Limit_Job_Runtimes line within the configuration file.
If this policy
is used together with the Always_Run_Jobs policy, order the specification by placing this
Limit_Job_Runtimes policy second.
• Preempt_If_Cpus_Exceeded If the startd observes the number of CPU cores used by the job exceed the number of cores in the slot by more than 0.8 on average over the past minute, preempt the job
immediately ignoring any job retirement time.
• Hold_If_Cpus_Exceeded If the startd observes the number of CPU cores used by the job exceed the
number of cores in the slot by more than 0.8 on average over the past minute, immediately place the job on
hold ignoring any job retirement time. The job will go on hold with a reasonable hold reason in job attribute
HoldReason and a value of 101 in job attribute HoldReasonCode. The hold reason and code can be
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
217
customized by specifying HOLD_REASON_CPU_EXCEEDED and HOLD_SUBCODE_CPU_EXCEEDED
respectively.
• Preempt_If_Memory_Exceeded If the startd observes the memory usage of the job exceed the memory provisioned in the slot, preempt the job immediately ignoring any job retirement time.
• Hold_If_Memory_Exceeded If the startd observes the memory usage of the job exceed the memory provisioned in the slot, immediately place the job on hold ignoring any job retirement time. The
job will go on hold with a reasonable hold reason in job attribute HoldReason and a value of 102
in job attribute HoldReasonCode. The hold reason and code can be customized by specifying
HOLD_REASON_MEMORY_EXCEEDED and HOLD_SUBCODE_MEMORY_EXCEEDED respectively.
SECURITY Describes configuration for an implemented security model.
• Host_Based The default security model used. Do not combine with User_Based security.
• User_Based Grants permissions to an administrator and uses With_Authentication. Do not combine with Host_Based security.
• With_Authentication Requires both authentication and integrity checks.
• Strong Requires authentication, encryption, and integrity checks.
For pools that are transitioning to using this new syntax in configuration, while still having some tools and daemons
with HTCondor versions earlier than 8.1.6, special syntax in the configuration will cause those daemons to fail upon
start up, rather than use the new, but misinterpreted, syntax. Newer daemons will ignore the extra syntax. Placing the
@ character before the use key word causes the older daemons to fail when they attempt to parse this syntax.
As an example, consider the condor_startd as it starts up. A condor_startd previous to HTCondor version 8.1.6
fails to start when it sees:
@use feature : GPUs
Running an older condor_config_val also identifies the @use line as being bad. A condor_startd of HTCondor version
8.1.6 or more recent sees
use feature : GPUs
Conditionals in Configuration
Conditional if/else semantics are available in a limited form. The syntax:
if <simple condition>
<statement>
. . .
<statement>
else
<statement>
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
218
. . .
<statement>
endif
An else key word and statements are not required, such that simple if semantics are implemented. The
<simple condition> does not permit compound conditions. It optionally contains the exclamation point character (!) to represent the not operation, followed by
• the defined keyword followed by the name of a variable. If the variable is defined, the statement(s) are
incorporated into the expanded input. If the variable is not defined, the statement(s) are not incorporated into
the expanded input. As an example,
if defined MY_UNDEFINED_VARIABLE
X = 12
else
X = -1
endif
results in X = -1, when MY_UNDEFINED_VARIABLE is not yet defined.
• the version keyword, representing the version number of of the daemon or tool currently reading this conditional. This keyword is followed by an HTCondor version number. That version number can be of the form
x.y.z or x.y. The version of the daemon or tool is compared to the specified version number. The comparison
operators are
– == for equality. Current version 8.2.3 is equal to 8.2.
– >= to see if the current version number is greater than or equal to. Current version 8.2.3 is greater than
8.2.2, and current version 8.2.3 is greater than or equal to 8.2.
– <= to see if the current version number is less than or equal to. Current version 8.2.0 is less than 8.2.2, and
current version 8.2.3 is less than or equal to 8.2.
As an example,
if version >= 8.1.6
DO_X = True
else
DO_Y = True
endif
results in defining DO_X as True if the current version of the daemon or tool reading this if statement is 8.1.6
or a more recent version.
• True or yes or the value 1. The statement(s) are incorporated.
• False or no or the value 0 The statement(s) are not incorporated.
• $(<variable>) may be used where the immediately evaluated value is a simple boolean value. A value that
evaluates to the empty string is considered False, otherwise a value that does not evaluate to a simple boolean
value is a syntax error.
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
219
The syntax
if <simple condition>
<statement>
. . .
<statement>
elif <simple condition>
<statement>
. . .
<statement>
endif
is the same as syntax
if <simple condition>
<statement>
. . .
<statement>
else
if <simple condition>
<statement>
. . .
<statement>
endif
endif
Function Macros in Configuration
A set of predefined functions increase flexibility. Both submit description files and configuration files are read using
the same parser, so these functions may be used in both submit description files and configuration files.
Case is significant in the function’s name, so use the same letter case as given in these definitions.
$CHOICE(index, listname) or $CHOICE(index, item1, item2, . . .) An item within the list is returned. The list is represented by a parameter name, or the list items are the parameters. The index parameter
determines which item. The first item in the list is at index 0. If the index is out of bounds for the list contents,
an error occurs.
$ENV(environment-variable-name[:default-value]) Evaluates to the value of environment variable
environment-variable-name. If there is no environment variable with that name, Evaluates to UNDEFINED unless the optional :default-value is used; in which case it evaluates to default-value. For
example,
A = $ENV(HOME)
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
220
binds A to the value of the HOME environment variable.
$F[fpduwnxbqa](filename) One or more of the lower case letters may be combined to form the function
name and thus, its functionality. Each letter operates on the filename in its own way.
• f convert relative path to full path by prefixing the current working directory to it. This option works only
in condor_submit files.
• p refers to the entire directory portion of filename, with a trailing slash or backslash character. Whether
a slash or backslash is used depends on the platform of the machine. The slash will be recognized on Linux
platforms; either a slash or backslash will be recognized on Windows platforms, and the parser will use
the same character specified.
• d refers to the last portion of the directory within the path, if specified. It will have a trailing slash or
backslash, as appropriate to the platform of the machine. The slash will be recognized on Linux platforms;
either a slash or backslash will be recognized on Windows platforms, and the parser will use the same
character specified unless u or w is used. if b is used the trailing slash or backslash will be omitted.
• u convert path separators to Unix style slash characters
• w convert path separators to Windows style backslash characters
• n refers to the file name at the end of any path, but without any file name extension. As an example, the
return value from $Fn(/tmp/simulate.exe) will be simulate (without the .exe extension).
• x refers to a file name extension, with the associated period (.). As an example, the return value from
$Fn(/tmp/simulate.exe) will be .exe.
• b when combined with the d option, causes the trailing slash or backslash to be omitted. When combined
with the x option, causes the leading period (.) to be omitted.
• q causes the return value to be enclosed within quotes. Double quote marks are used unless a is also
specified.
• a When combined with the q option, causes the return value to be enclosed within single quotes.
$DIRNAME(filename) is the same as $Fp(filename)
$BASENAME(filename) is the same as $Fnx(filename)
$INT(item-to-convert) or $INT(item-to-convert, format-specifier) Expands, evaluates,
and returns a string version of item-to-convert. The format-specifier has the same syntax as a
C language or Perl format specifier. If no format-specifier is specified, "%d" is used as the format
specifier.
$RANDOM_CHOICE(choice1, choice2, choice3, . . .) A random choice of one of the parameters in the
list of parameters is made. For example, if one of the integers 0-8 (inclusive) should be randomly chosen:
$RANDOM_CHOICE(0,1,2,3,4,5,6,7,8)
$RANDOM_INTEGER(min, max [, step]) A random integer within the range min and max, inclusive, is
selected. The optional step parameter controls the stride within the range, and it defaults to the value 1. For
example, to randomly chose an even integer in the range 0-8 (inclusive):
$RANDOM_INTEGER(0, 8, 2)
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
221
$REAL(item-to-convert) or $REAL(item-to-convert, format-specifier) Expands, evaluates,
and returns a string version of item-to-convert for a floating point type. The format-specifier is
a C language or Perl format specifier. If no format-specifier is specified, "%16G" is used as a format
specifier.
$SUBSTR(name, start-index) or $SUBSTR(name, start-index, length) Expands name and returns a substring of it. The first character of the string is at index 0. The first character of the substring is at
index start-index. If the optional length is not specified, then the substring includes characters up to the
end of the string. A negative value of start-index works back from the end of the string. A negative value
of length eliminates use of characters from the end of the string. Here are some examples that all assume
Name = abcdef
• $SUBSTR(Name, 2) is cdef.
• $SUBSTR(Name, 0, -2) is abcd.
• $SUBSTR(Name, 1, 3) is bcd.
• $SUBSTR(Name, -1) is f.
• $SUBSTR(Name, 4, -3) is the empty string, as there are no characters in the substring for this request.
Environment references are not currently used in standard HTCondor configurations. However, they can sometimes
be useful in custom configurations.
Macros That Will Require a Restart When Changed
When any of the following listed configuration variables are changed, HTCondor must be restarted; reconfiguration
using condor_reconfig will not be enough to cause the new values to take effect.
• BIND_ALL_INTERFACES
• FetchWorkDelay
• MAX_NUM_CPUS
• MAX_TRACKING_GID
• MEMORY
• MIN_TRACKING_GID
• NETWORK_HOSTNAME
• NETWORK_INTERFACE
• NUM_CPUS
• PREEMPTION_REQUIREMENTS_STABLE
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
222
• PRIVSEP_ENABLED
• PROCD_ADDRESS
• SLOT_TYPE_<N>
• OFFLINE_MACHINE_RESOURCE_<name>
Pre-Defined Macros
HTCondor provides pre-defined macros that help configure HTCondor.
$(macro_name).
Pre-defined macros are listed as
This first set are entries whose values are determined at run time and cannot be overwritten. These are inserted
automatically by the library routine which parses the configuration files. This implies that a change to the underlying
value of any of these variables will require a full restart of HTCondor in order to use the changed value.
$(FULL_HOSTNAME) The fully qualified host name of the local machine, which is host name plus domain name.
$(HOSTNAME) The host name of the local machine, without a domain name.
$(IP_ADDRESS) The ASCII string version of the local machine’s IP address.
$(TILDE) The full path to the home directory of the Unix user condor, if such a user exists on the local machine.
$(SUBSYSTEM) The subsystem name of the daemon or tool that is evaluating the macro. This is a unique string
which identifies a given daemon within the HTCondor system. The possible subsystem names are:
• C_GAHP
• C_GAHP_WORKER_THREAD
• CKPT_SERVER
• COLLECTOR
• DBMSD
• DEFRAG
• EC2_GAHP
• GANGLIAD
• GCE_GAHP
• GRIDMANAGER
• HAD
• HDFS
• JOB_ROUTER
• KBDD
• LEASEMANAGER
HTCondor Version 8.6.4 Manual
3.5.1. Introduction to Configuration Files
223
• MASTER
• NEGOTIATOR
• QUILL
• REPLICATION
• ROOSTER
• SCHEDD
• SHADOW
• SHARED_PORT
• STARTD
• STARTER
• SUBMIT
• TOOL
• TRANSFERER
$(DETECTED_CPUS) The integer number of hyper-threaded CPUs, as given by $(DETECTED_CORES), when
COUNT_HYPERTHREAD_CPUS is True. The integer number of physical (non hyper-threaded) CPUs,
as given by $(DETECTED_PHYSICAL_CPUS), when COUNT_HYPERTHREAD_CPUS is False. When
COUNT_HYPERTHREAD_CPUS is True.
$(DETECTED_PHYSICAL_CPUS) The integer number of physical (non hyper-threaded) CPUs. This will be equal
the number of unique CPU IDs.
This second set of macros are entries whose default values are determined automatically at run time but which can
be overwritten.
$(ARCH) Defines the string used to identify the architecture of the local machine to HTCondor. The condor_startd
will advertise itself with this attribute so that users can submit binaries compiled for a given platform and force
them to run on the correct machines. condor_submit will append a requirement to the job ClassAd that it must
run on the same ARCH and OPSYS of the machine where it was submitted, unless the user specifies ARCH and/or
OPSYS explicitly in their submit file. See the condor_submit manual page on page 911 for details.
$(OPSYS) Defines the string used to identify the operating system of the local machine to HTCondor. If it is not
defined in the configuration file, HTCondor will automatically insert the operating system of this machine as
determined by uname.
$(OPSYS_VER) Defines the integer used to identify the operating system version number.
$(OPSYS_AND_VER) Defines the string used prior to HTCondor version 7.7.2 as $(OPSYS).
$(UNAME_ARCH) The architecture as reported by uname(2)’s machine field. Always the same as ARCH on Windows.
$(UNAME_OPSYS) The operating system as reported by uname(2)’s sysname field. Always the same as OPSYS
on Windows.
HTCondor Version 8.6.4 Manual
3.5.2. HTCondor-wide Configuration File Entries
$(DETECTED_MEMORY) The amount of detected physical memory (RAM) in MiB.
$(DETECTED_CORES) The number of CPU cores that the operating system schedules. On machines that support
hyper-threading, this will be the number of hyper-threads.
$(PID) The process ID for the daemon or tool.
$(PPID) The process ID of the parent process for the daemon or tool.
$(USERNAME) The user name of the UID of the daemon or tool. For daemons started as root, but running under
another UID (typically the user condor), this will be the other UID.
$(FILESYSTEM_DOMAIN) Defaults to the fully qualified host name of the machine it is evaluated on. See section 3.5.6, Shared File System Configuration File Entries for the full description of its use and under what
conditions it could be desirable to change it.
$(UID_DOMAIN) Defaults to the fully qualified host name of the machine it is evaluated on. See section 3.5.6 for
the full description of this configuration variable.
Since $(ARCH) and $(OPSYS) will automatically be set to the correct values, we recommend that you do not
overwrite them.
3.5.2 HTCondor-wide Configuration File Entries
This section describes settings which affect all parts of the HTCondor system. Other system-wide settings can be
found in section 3.5.5 on “Network-Related Configuration File Entries”, and section 3.5.6 on “Shared File System
Configuration File Entries”.
CONDOR_HOST This macro is used to define the $(COLLECTOR_HOST) macro. Normally the condor_collector
and condor_negotiator would run on the same machine. If for some reason they were not run on the
same machine, $(CONDOR_HOST) would not be needed. Some of the host-based security macros use
$(CONDOR_HOST) by default. See section 3.8.9, on Setting up IP/host-based security in HTCondor for details.
COLLECTOR_HOST The host name of the machine where the condor_collector is running for your pool. Normally, it is defined relative to the $(CONDOR_HOST) macro. There is no default value for this macro;
COLLECTOR_HOST must be defined for the pool to work properly.
In addition to defining the host name, this setting can optionally be used to specify the network port of the
condor_collector. The port is separated from the host name by a colon (’:’). For example,
COLLECTOR_HOST = $(CONDOR_HOST):1234
If no port is specified, the default port of 9618 is used. Using the default port is recommended for most sites. It is
only changed if there is a conflict with another service listening on the same network port. For more information
about specifying a non-standard port for the condor_collector daemon, see section 3.9.1 on page 451.
HTCondor Version 8.6.4 Manual
224
3.5.2. HTCondor-wide Configuration File Entries
Multiple condor_collector daemons may be running simultaneously, if COLLECTOR_HOST is defined with a
comma separated list of hosts. Multiple condor_collector daemons may run for the implementation of high
availability; see section 3.13 for details. With more than one running, updates are sent to all. With more than
one running, queries are sent to one of the condor_collector daemons, chosen at random.
COLLECTOR_PORT The default port used when contacting the condor_collector and the default port the condor_collector listens on if no port is specified. This variable is referenced if no port is given and there is no
other means to find the condor_collector port. The default value is 9618.
NEGOTIATOR_HOST This configuration variable is no longer used. It previously defined the host name of the machine where the condor_negotiator is running. At present, the port where the condor_negotiator is listening is
dynamically allocated.
CONDOR_VIEW_HOST A list of HTCondorView servers, separated by commas and/or spaces. Each HTCondorView
server is denoted by the host name of the machine it is running on, optionally appended by a colon and the port
number. This service is optional, and requires additional configuration to enable it. There is no default value
for CONDOR_VIEW_HOST. If CONDOR_VIEW_HOST is not defined, no HTCondorView server is used. See
section 3.14.6 on page 491 for more details.
SCHEDD_HOST The host name of the machine where the condor_schedd is running for your pool. This is the host
that queues submitted jobs. If the host specifies SCHEDD_NAME or MASTER_NAME, that name must be included
in the form name@hostname. In most condor installations, there is a condor_schedd running on each host from
which jobs are submitted. The default value of SCHEDD_HOST is the current host with the optional name
included. For most pools, this macro is not defined, nor does it need to be defined..
RELEASE_DIR The full path to the HTCondor release directory, which holds the bin, etc, lib, and sbin directories. Other macros are defined relative to this one. There is no default value for RELEASE_DIR.
BIN This directory points to the HTCondor directory where user-level programs are installed. The default value is
$(RELEASE_DIR)/bin.
LIB This directory points to the HTCondor directory where libraries used to link jobs for HTCondor’s standard
universe are stored. The condor_compile program uses this macro to find these libraries, so it must be defined
for condor_compile to function. The default value is $(RELEASE_DIR)/lib.
LIBEXEC This directory points to the HTCondor directory where support commands that HTCondor needs will be
placed. Do not add this directory to a user or system-wide path.
INCLUDE This directory points to the HTCondor directory where header files reside. The default value is
$(RELEASE_DIR)/include. It can make inclusion of necessary header files for compilation of programs
(such as those programs that use libcondorapi.a) easier through the use of condor_config_val.
SBIN This directory points to the HTCondor directory where HTCondor’s system binaries (such as the binaries for
the HTCondor daemons) and administrative tools are installed. Whatever directory $(SBIN) points to ought
to be in the PATH of users acting as HTCondor administrators. The default value is $(BIN) in Windows and
$(RELEASE_DIR)/sbin on all other platforms.
LOCAL_DIR The location of the local HTCondor directory on each machine in your pool. The default value is
$(RELEASE_DIR) on Windows and $(RELEASE_DIR)/hosts/$(HOSTNAME) on all other platforms.
Another possibility is to use the condor user’s home directory, which may be specified with $(TILDE). For
example:
HTCondor Version 8.6.4 Manual
225
3.5.2. HTCondor-wide Configuration File Entries
LOCAL_DIR = $(tilde)
LOG Used to specify the directory where each HTCondor daemon writes its log files. The names of the log files
themselves are defined with other macros, which use the $(LOG) macro by default. The log directory also
acts as the current working directory of the HTCondor daemons as the run, so if one of them should produce
a core file for any reason, it would be placed in the directory defined by this macro. The default value is
$(LOCAL_DIR)/log.
Do not stage other files in this directory; any files not created by HTCondor in this directory are subject to
removal.
RUN A path and directory name to be used by the HTCondor init script to specify the directory where the condor_master should write its process ID (PID) file. The default if not defined is $(LOG).
SPOOL The spool directory is where certain files used by the condor_schedd are stored, such as the job queue file
and the initial executables of any jobs that have been submitted. In addition, for systems not using a checkpoint
server, all the checkpoint files from jobs that have been submitted from a given machine will be store in that
machine’s spool directory. Therefore, you will want to ensure that the spool directory is located on a partition
with enough disk space. If a given machine is only set up to execute HTCondor jobs and not submit them, it
would not need a spool directory (or this macro defined). The default value is $(LOCAL_DIR)/spool. The
condor_schedd will not function if SPOOL is not defined.
Do not stage other files in this directory; any files not created by HTCondor in this directory are subject to
removal.
EXECUTE This directory acts as a place to create the scratch directory of any HTCondor job that is executing on
the local machine. The scratch directory is the destination of any input files that were specified for transfer. It
also serves as the job’s working directory if the job is using file transfer mode and no other working directory
was specified. If a given machine is set up to only submit jobs and not execute them, it would not need an
execute directory, and this macro need not be defined. The default value is $(LOCAL_DIR)/execute. The
condor_startd will not function if EXECUTE is undefined. To customize the execute directory independently
for each batch slot, use SLOT<N>_EXECUTE.
Do not stage other files in this directory; any files not created by HTCondor in this directory are subject to
removal.
TMP_DIR A directory path to a directory where temporary files are placed by various portions of the HTCondor system. The daemons and tools that use this directory are the condor_gridmanager, condor_config_val when using
the -rset option, systems that use lock files when configuration variable CREATE_LOCKS_ON_LOCAL_DISK
is True, the Web Service API, and the condor_credd daemon. There is no default value.
If both TMP_DIR and TEMP_DIR are defined, the value set for TMP_DIR is used and TEMP_DIR is ignored.
TEMP_DIR A directory path to a directory where temporary files are placed by various portions of the HTCondor system. The daemons and tools that use this directory are the condor_gridmanager, condor_config_val when using
the -rset option, systems that use lock files when configuration variable CREATE_LOCKS_ON_LOCAL_DISK
is True, the Web Service API, and the condor_credd daemon. There is no default value.
If both TMP_DIR and TEMP_DIR are defined, the value set for TMP_DIR is used and TEMP_DIR is ignored.
HTCondor Version 8.6.4 Manual
226
3.5.2. HTCondor-wide Configuration File Entries
SLOT<N>_EXECUTE Specifies an execute directory for use by a specific batch slot. <N> represents the number of
the batch slot, such as 1, 2, 3, etc. This execute directory serves the same purpose as EXECUTE, but it allows
the configuration of the directory independently for each batch slot. Having slots each using a different partition
would be useful, for example, in preventing one job from filling up the same disk that other jobs are trying to
write to. If this parameter is undefined for a given batch slot, it will use EXECUTE as the default. Note that each
slot will advertise TotalDisk and Disk for the partition containing its execute directory.
LOCAL_CONFIG_FILE Identifies the location of the local, machine-specific configuration file for each machine in
the pool. The two most common choices would be putting this file in the $(LOCAL_DIR), or putting all local
configuration files for the pool in a shared directory, each one named by host name. For example,
LOCAL_CONFIG_FILE = $(LOCAL_DIR)/condor_config.local
or,
LOCAL_CONFIG_FILE = $(release_dir)/etc/$(hostname).local
or, not using the release directory
LOCAL_CONFIG_FILE = /full/path/to/configs/$(hostname).local
The value of LOCAL_CONFIG_FILE is treated as a list of files, not a single file. The items in the list are
delimited by either commas or space characters. This allows the specification of multiple files as the local
configuration file, each one processed in the order given (with parameters set in later files overriding values from
previous files). This allows the use of one global configuration file for multiple platforms in the pool, defines
a platform-specific configuration file for each platform, and uses a local configuration file for each machine.
If the list of files is changed in one of the later read files, the new list replaces the old list, but any files that
have already been processed remain processed, and are removed from the new list if they are present to prevent
cycles. See section 3.5.1 on page 214 for directions on using a program to generate the configuration macros
that would otherwise reside in one or more files as described here. If LOCAL_CONFIG_FILE is not defined,
no local configuration files are processed. For more information on this, see section 3.14.3 about Configuring
HTCondor for Multiple Platforms on page 486.
If all files in a directory are local configuration files to be processed, then consider using LOCAL_CONFIG_DIR,
defined at section 3.5.2.
REQUIRE_LOCAL_CONFIG_FILE A boolean value that defaults to True. When True, HTCondor exits with an
error, if any file listed in LOCAL_CONFIG_FILE cannot be read. A value of False allows local configuration
files to be missing. This is most useful for sites that have both large numbers of machines in the pool and a local
configuration file that uses the $(HOSTNAME) macro in its definition. Instead of having an empty file for every
host in the pool, files can simply be omitted.
LOCAL_CONFIG_DIR A directory may be used as a container for local configuration files. The files found
in the directory are sorted into lexicographical order by file name, and then each file is treated as though
it was listed in LOCAL_CONFIG_FILE. LOCAL_CONFIG_DIR is processed before any files listed in
HTCondor Version 8.6.4 Manual
227
3.5.2. HTCondor-wide Configuration File Entries
LOCAL_CONFIG_FILE, and is checked again after processing the LOCAL_CONFIG_FILE list. It is
a list of directories, and each directory is processed in the order it appears in the list. The process
is not recursive, so any directories found inside the directory being processed are ignored. See also
LOCAL_CONFIG_DIR_EXCLUDE_REGEXP.
USER_CONFIG_FILE The file name of a configuration file to be parsed after other local configuration files and
before environment variables set configuration. Relevant only if HTCondor daemons are not run as root on
Unix platforms or Local System on Windows platforms. The default is $(HOME)/.condor/user_config
on Unix platforms. The default is %USERPROFILE\.condor\user_config on Windows platforms. If a fully qualified path is given, that is used. If a fully qualified path is not given, then the
Unix path $(HOME)/.condor/ prefixes the file name given on Unix platforms, or the Windows path
%USERPROFILE\.condor\ prefixes the file name given on Windows platforms.
The ability of a user to use this user-specified configuration file can be disabled by setting this variable to the
empty string:
USER_CONFIG_FILE =
LOCAL_CONFIG_DIR_EXCLUDE_REGEXP A regular expression that specifies file names to be ignored when looking for configuration files within the directories specified via LOCAL_CONFIG_DIR. The default expression
ignores files with names beginning with a ‘.’ or a ‘#’, as well as files with names ending in ‘˜’. This avoids
accidents that can be caused by treating temporary files created by text editors as configuration files.
CONDOR_IDS The User ID (UID) and Group ID (GID) pair that the HTCondor daemons should run as, if the daemons are spawned as root. This value can also be specified in the CONDOR_IDS environment variable. If
the HTCondor daemons are not started as root, then neither this CONDOR_IDS configuration macro nor the
CONDOR_IDS environment variable are used. The value is given by two integers, separated by a period. For
example, CONDOR_IDS = 1234.1234. If this pair is not specified in either the configuration file or in the
environment, and the HTCondor daemons are spawned as root, then HTCondor will search for a condor user
on the system, and run as that user’s UID and GID. See section 3.8.13 on UIDs in HTCondor for more details.
CONDOR_ADMIN The email address that HTCondor will send mail to if something goes wrong in the pool. For
example, if a daemon crashes, the condor_master can send an obituary to this address with the last few lines of
that daemon’s log file and a brief message that describes what signal or exit status that daemon exited with. The
default value is root@$(FULL_HOSTNAME).
<SUBSYS>_ADMIN_EMAIL The email address that HTCondor will send mail to if something goes wrong with the
named <SUBSYS>. Identical to CONDOR_ADMIN, but done on a per subsystem basis. There is no default value.
CONDOR_SUPPORT_EMAIL The email address to be included at the bottom of all email HTCondor sends out under
the label “Email address of the local HTCondor administrator:”. This is the address where HTCondor users at
your site should send their questions about HTCondor and get technical support. If this setting is not defined,
HTCondor will use the address specified in CONDOR_ADMIN (described above).
EMAIL_SIGNATURE Every e-mail sent by HTCondor includes a short signature line appended to the body. By
default, this signature includes the URL to the global HTCondor project website. When set, this variable defines
an alternative signature line to be used instead of the default. Note that the value can only be one line in length.
This variable could be used to direct users to look at local web site with information specific to the installation
of HTCondor.
HTCondor Version 8.6.4 Manual
228
3.5.2. HTCondor-wide Configuration File Entries
MAIL The full path to a mail sending program that uses -s to specify a subject for the message. On all platforms,
the default shipped with HTCondor should work. Only if you installed things in a non-standard location on
your system would you need to change this setting. The default value is $(BIN)/condor_mail.exe on
Windows and /usr/bin/mail on all other platforms. The condor_schedd will not function unless MAIL is
defined. For security reasons, non-Windows platforms should not use this setting and should use SENDMAIL
instead.
SENDMAIL The full path to the sendmail executable. If defined, which it is by default on non-Windows platforms,
sendmail is used instead of the mail program defined by MAIL.
MAIL_FROM The e-mail address that notification e-mails appear to come from. Contents is that of the From header.
There is no default value; if undefined, the From header may be nonsensical.
SMTP_SERVER For Windows platforms only, the host name of the server through which to route notification e-mail.
There is no default value; if undefined and the debug level is at FULLDEBUG, an error message will be generated.
RESERVED_SWAP The amount of swap space in MiB to reserve for this machine. HTCondor will not start up more
condor_shadow processes if the amount of free swap space on this machine falls below this level. The default
value is 0, which disables this check. It is anticipated that this configuration variable will no longer be used in
the near future. If RESERVED_SWAP is not set to 0, the value of SHADOW_SIZE_ESTIMATE is used.
RESERVED_DISK Determines how much disk space you want to reserve for your own machine. When HTCondor is
reporting the amount of free disk space in a given partition on your machine, it will always subtract this amount.
An example is the condor_startd, which advertises the amount of free space in the $(EXECUTE) directory.
The default value of RESERVED_DISK is zero.
LOCK HTCondor needs to create lock files to synchronize access to various log files. Because of problems with
network file systems and file locking over the years, we highly recommend that you put these lock files on a
local partition on each machine. If you do not have your $(LOCAL_DIR) on a local partition, be sure to change
this entry.
Whatever user or group HTCondor is running as needs to have write access to this directory. If you are not
running as root, this is whatever user you started up the condor_master as. If you are running as root, and
there is a condor account, it is most likely condor. Otherwise, it is whatever you set in the CONDOR_IDS
environment variable, or whatever you define in the CONDOR_IDS setting in the HTCondor config files. See
section 3.8.13 on UIDs in HTCondor for details.
If no value for LOCK is provided, the value of LOG is used.
HISTORY Defines the location of the HTCondor history file, which stores information about all HTCondor jobs
that have completed on a given machine. This macro is used by both the condor_schedd which appends the
information and condor_history, the user-level program used to view the history file. This configuration macro
is given the default value of $(SPOOL)/history in the default configuration. If not defined, no history file
is kept.
ENABLE_HISTORY_ROTATION If this is defined to be true, then the history file will be rotated. If it is false, then
it will not be rotated, and it will grow indefinitely, to the limits allowed by the operating system. If this is not
defined, it is assumed to be true. The rotated files will be stored in the same directory as the history file.
MAX_HISTORY_LOG Defines the maximum size for the history file, in bytes. It defaults to 20MB. This parameter is
only used if history file rotation is enabled.
HTCondor Version 8.6.4 Manual
229
3.5.2. HTCondor-wide Configuration File Entries
MAX_HISTORY_ROTATIONS When history file rotation is turned on, this controls how many backup files there are.
It default to 2, which means that there may be up to three history files (two backups, plus the history file that is
being currently written to). When the history file is rotated, and this rotation would cause the number of backups
to be too large, the oldest file is removed.
HISTORY_HELPER_MAX_CONCURRENCY Specifies the maximum number of concurrent remote condor_history
queries allowed at a time; defaults to 2. When this maximum is exceeded, further queries will be queued in a
non-blocking manner. Setting this option to 0 disables remote history access. A remote history access is defined
as an invocation of condor_history that specifies a -name option to query a condor_schedd running on a remote
machine.
HISTORY_HELPER_MAX_HISTORY Specifies the maximum number of ClassAds to parse on behalf of remote
history clients. The default is 10,000. This allows the system administrator to indirectly manage the maximum
amount of CPU time spent on each client. Setting this option to 0 disables remote history access.
MAX_JOB_QUEUE_LOG_ROTATIONS The condor_schedd daemon periodically rotates the job queue database file,
in order to save disk space. This option controls how many rotated files are saved. It defaults to 1, which means
there may be up to two history files (the previous one, which was rotated out of use, and the current one that is
being written to). When the job queue file is rotated, and this rotation would cause the number of backups to be
larger the the maximum specified, the oldest file is removed.
CLASSAD_LOG_STRICT_PARSING A boolean value that defaults to True. When True, ClassAd log files will
be read using a strict syntax checking for ClassAd expressions. ClassAd log files include the job queue log and
the accountant log. When False, ClassAd log files are read without strict expression syntax checking, which
allows some legacy ClassAd log data to be read in a backward compatible manner. This configuration variable
may no longer be supported in future releases, eventually requiring all ClassAd log files to pass strict ClassAd
syntax checking.
DEFAULT_DOMAIN_NAME The value to be appended to a machine’s host name, representing a domain name, which
HTCondor then uses to form a fully qualified host name. This is required if there is no fully qualified host name
in file /etc/hosts or in NIS. Set the value in the global configuration file, as HTCondor may depend on
knowing this value in order to locate the local configuration file(s). The default value as given in the sample
configuration file of the HTCondor download is bogus, and must be changed. If this variable is removed from
the global configuration file, or if the definition is empty, then HTCondor attempts to discover the value.
NO_DNS A boolean value that defaults to False. When True, HTCondor constructs host names using the host’s IP
address together with the value defined for DEFAULT_DOMAIN_NAME.
CM_IP_ADDR If neither COLLECTOR_HOST nor COLLECTOR_IP_ADDR macros are defined, then this macro will
be used to determine the IP address of the central manager (collector daemon). This macro is defined by an IP
address.
EMAIL_DOMAIN By default, if a user does not specify notify_user in the submit description file, any email
HTCondor sends about that job will go to "username@UID_DOMAIN". If your machines all share a common
UID domain (so that you would set UID_DOMAIN to be the same across all machines in your pool), but email
to user@UID_DOMAIN is not the right place for HTCondor to send email for your site, you can define the
default domain to use for email. A common example would be to set EMAIL_DOMAIN to the fully qualified
host name of each machine in your pool, so users submitting jobs from a specific machine would get email sent
to user@machine.your.domain, instead of user@your.domain. You would do this by setting EMAIL_DOMAIN
HTCondor Version 8.6.4 Manual
230
3.5.2. HTCondor-wide Configuration File Entries
to $(FULL_HOSTNAME). In general, you should leave this setting commented out unless two things are true:
1) UID_DOMAIN is set to your domain, not $(FULL_HOSTNAME), and 2) email to user@UID_DOMAIN will
not work.
CREATE_CORE_FILES Defines whether or not HTCondor daemons are to create a core file in the LOG directory if
something really bad happens. It is used to set the resource limit for the size of a core file. If not defined, it
leaves in place whatever limit was in effect when the HTCondor daemons (normally the condor_master) were
started. This allows HTCondor to inherit the default system core file generation behavior at start up. For Unix
operating systems, this behavior can be inherited from the parent shell, or specified in a shell script that starts
HTCondor. If this parameter is set and True, the limit is increased to the maximum. If it is set to False, the
limit is set at 0 (which means that no core files are created). Core files greatly help the HTCondor developers
debug any problems you might be having. By using the parameter, you do not have to worry about tracking
down where in your boot scripts you need to set the core limit before starting HTCondor. You set the parameter
to whatever behavior you want HTCondor to enforce. This parameter defaults to undefined to allow the initial
operating system default value to take precedence, and is commented out in the default configuration file.
CKPT_PROBE Defines the path and executable name of the helper process HTCondor will use to determine
information for the CheckpointPlatform attribute in the machine’s ClassAd. The default value is
$(LIBEXEC)/condor_ckpt_probe.
ABORT_ON_EXCEPTION When HTCondor programs detect a fatal internal exception, they normally log an error
message and exit. If you have turned on CREATE_CORE_FILES, in some cases you may also want to turn on
ABORT_ON_EXCEPTION so that core files are generated when an exception occurs. Set the following to True
if that is what you want.
Q_QUERY_TIMEOUT Defines the timeout (in seconds) that condor_q uses when trying to connect to the condor_schedd. Defaults to 20 seconds.
DEAD_COLLECTOR_MAX_AVOIDANCE_TIME Defines the interval of time (in seconds) between checks for a failed
primary condor_collector daemon. If connections to the dead primary condor_collector take very little time to
fail, new attempts to query the primary condor_collector may be more frequent than the specified maximum
avoidance time. The default value equals one hour. This variable has relevance to flocked jobs, as it defines the
maximum time they may be reporting to the primary condor_collector without the condor_negotiator noticing.
PASSWD_CACHE_REFRESH HTCondor can cause NIS servers to become overwhelmed by queries for uid and group
information in large pools. In order to avoid this problem, HTCondor caches UID and group information
internally. This integer value allows pool administrators to specify (in seconds) how long HTCondor should
wait until refreshes a cache entry. The default is set to 72000 seconds, or 20 hours, plus a random number of
seconds between 0 and 60 to avoid having lots of processes refreshing at the same time. This means that if a
pool administrator updates the user or group database (for example, /etc/passwd or /etc/group), it can
take up to 6 minutes before HTCondor will have the updated information. This caching feature can be disabled
by setting the refresh interval to 0. In addition, the cache can also be flushed explicitly by running the command
condor_reconfig. This configuration variable has no effect on Windows.
SYSAPI_GET_LOADAVG If set to False, then HTCondor will not attempt to compute the load average on the system,
and instead will always report the system load average to be 0.0. Defaults to True.
NETWORK_MAX_PENDING_CONNECTS This specifies a limit to the maximum number of simultaneous network
connection attempts. This is primarily relevant to condor_schedd, which may try to connect to large numbers
HTCondor Version 8.6.4 Manual
231
3.5.2. HTCondor-wide Configuration File Entries
of startds when claiming them. The negotiator may also connect to large numbers of startds when initiating
security sessions used for sending MATCH messages. On Unix, the default for this parameter is eighty percent
of the process file descriptor limit. On windows, the default is 1600.
WANT_UDP_COMMAND_SOCKET This setting, added in version 6.9.5, controls if HTCondor daemons should create
a UDP command socket in addition to the TCP command socket (which is required). The default is True, and
modifying it requires restarting all HTCondor daemons, not just a condor_reconfig or SIGHUP.
Normally, updates sent to the condor_collector use UDP, in addition to certain keep alive messages and other
non-essential communication. However, in certain situations, it might be desirable to disable the UDP command
port.
Unfortunately, due to a limitation in how these command sockets are created, it is not possible to define this
setting on a per-daemon basis, for example, by trying to set STARTD.WANT_UDP_COMMAND_SOCKET. At
least for now, this setting must be defined machine wide to function correctly.
If this setting is set to true on a machine running a condor_collector, the pool should be configured to use TCP
updates to that collector (see section 3.9.5 on page 461 for more information).
ALLOW_SCRIPTS_TO_RUN_AS_EXECUTABLES A boolean value that, when True, permits scripts on Windows
platforms to be used in place of the executable in a job submit description file, in place of a condor_dagman
pre or post script, or in producing the configuration, for example. Allows a script to be used in any circumstance
previously limited to a Windows executable or a batch file. The default value is True. See section 7.2.7 on
page 666 for further description.
OPEN_VERB_FOR_<EXT>_FILES A string that defines a Windows verb for use in a root hive registry look up.
<EXT> defines the file name extension, which represents a scripting language, also needed for the look up. See
section 7.2.7 on page 666 for a more complete description.
ENABLE_CLASSAD_CACHING A boolean value that controls the caching of ClassAds. Caching saves memory
when an HTCondor process contains many ClassAds with the same expressions. The default value is True
for all daemons other than the condor_shadow, condor_starter, and condor_master. A value of True enables
caching.
STRICT_CLASSAD_EVALUATION A boolean value that controls how ClassAd expressions are evaluated. If set to
True, then New ClassAd evaluation semantics are used. This means that attribute references without a MY. or
TARGET. prefix are only looked up in the local ClassAd. If set to the default value of False, Old ClassAd
evaluation semantics are used. See section 4.1.1 on page 524 for details.
CLASSAD_USER_LIBS A comma separated list of paths to shared libraries that contain additional ClassAd functions to be used during ClassAd evaluation.
CLASSAD_USER_PYTHON_MODULES A comma separated list of python modules to load, which are to be used
during ClassAd evaluation. If module foo is in this list, then function bar can be invoked in ClassAds
via the expression python_invoke("foo", "bar", ...). Any further arguments are converted from
ClassAd expressions to python; the function return value is converted back to ClassAds. The python modules are loaded at configuration time, so any module-level statements are executed. Module writers can invoke
classad.register at the module-level in order to use python functions directly.
Functions executed by ClassAds should be non-blocking and have no side-effects; otherwise, unpredictable
HTCondor behavior may occur.
HTCondor Version 8.6.4 Manual
232
3.5.2. HTCondor-wide Configuration File Entries
CLASSAD_USER_PYTHON_LIB Specifies
the
path
to
the
python
libraries,
which
is
needed
when
CLASSAD_USER_PYTHON_MODULES
is
set.
Defaults
to
$(LIBEXEC)/libclassad_python_user.so, and would rarely be changed from the default value.
CONDOR_FSYNC A boolean value that controls whether HTCondor calls fsync() when writing the user job and
transaction logs. Setting this value to False will disable calls to fsync(), which can help performance for
condor_schedd log writes at the cost of some durability of the log contents, should there be a power or hardware
failure. The default value is True.
STATISTICS_TO_PUBLISH A comma and/or space separated list that identifies which statistics collections are to
place attributes in ClassAds. Additional information specifies a level of verbosity and other identification of
which attributes to include and which to omit from ClassAds. The special value NONE disables all publishing,
so no statistics will be published; no option is included. For other list items that define this variable, the syntax
defines the two aspects by separating them with a colon. The first aspect defines a collection, which may
specify which daemon is to publish the statistics, and the second aspect qualifies and refines the details of which
attributes to publish for the collection, including a verbosity level. If the first aspect is ALL, the option is applied
to all collections. If the first aspect is DEFAULT, the option is applied to all collections, with the intent that
further list items will specify publishing that is to be different than the default. This first aspect may be SCHEDD
or SCHEDULER to publish Statistics attributes in the ClassAd of the condor_schedd. It may be TRANSFER
to publish file transfer statistics. It may be STARTER to publish Statistics attributes in the ClassAd of the
condor_starter. Or, it may be DC or DAEMONCORE to publish DaemonCore statistics. One or more options are
specified after the colon.
Option
0
1
2
3
R
!R
D
!D
Z
!Z
L
!L
Description
turns off the publishing of any statistics attributes
the default level, where some statistics attributes are published and others are omitted
the verbose level, where all statistics attributes are published
the super verbose level, which is currently unused, but intended to be all statistics
attributes published at the verbose level plus extra information
include attributes from the most recent time interval; the default
omit attributes from the most recent time interval
include attributes for debugging
omit attributes for debugging; the default
include attributes even if the attribute’s value is 0
omit attributes when the attribute’s value is 0
include attributes that represent the lifetime value; the default
omit attributes that represent the lifetime value
If this variable is not defined, then the default for each collection is used. If this variable is defined, and the
definition does not specify each possible collection, then no statistics are published for those collections not
defined. If an option specifies conflicting possibilities, such as R!R, then the last one takes precedence and is
applied.
As an example, to cause a verbose setting of the publication of Statistics attributes only for the condor_schedd,
and do not publish any other Statistics attributes:
HTCondor Version 8.6.4 Manual
233
3.5.2. HTCondor-wide Configuration File Entries
STATISTICS_TO_PUBLISH = SCHEDD:2
As a second example, to cause all collections other than those for DAEMONCORE to publish at a verbosity setting
of 1, and omit lifetime values, where the DAEMONCORE includes all statistics at the verbose level:
STATISTICS_TO_PUBLISH = DEFAULT:1!L, DC:2RDZL
STATISTICS_TO_PUBLISH_LIST A comma and/or space separated list of statistics attribute names that
should be published in updates to the condor_collector daemon, even though the verbosity specified in
STATISTICS_TO_PUBLISH would not normally send them. This setting has the effect of redefining the
verbosity level of the statistics attributes that it mentions, so that they will always match the current statistics
publication level as specified in STATISTICS_TO_PUBLISH.
STATISTICS_WINDOW_SECONDS An integer value that controls the time window size, in seconds, for collecting
windowed daemon statistics. These statistics are, by convention, those attributes with names that are of the form
Recent<attrname>. Any data contributing to a windowed statistic that is older than this number of seconds
is dropped from the statistic. For example, if STATISTICS_WINDOW_SECONDS = 300, then any jobs
submitted more than 300 seconds ago are not counted in the windowed statistic RecentJobsSubmitted.
Defaults to 1200 seconds, which is 20 minutes.
The window is broken into smaller time pieces called quantum. The window advances one quantum at a time.
STATISTICS_WINDOW_SECONDS_<collection> The same as STATISTICS_WINDOW_SECONDS, but
used to override the global setting for a particular statistic collection. Collection names currently implemented
are DC or DAEMONCORE and SCHEDD or SCHEDULER.
STATISTICS_WINDOW_QUANTUM For experts only, an integer value that controls the time quantization that form
a time window, in seconds, for the data structures that maintain windowed statistics. Defaults to 240 seconds, which is 6 minutes. This default is purposely set to be slightly smaller than the update rate to the condor_collector. Setting a smaller value than the default increases the memory requirement for the statistics.
Graphing of statistics at the level of the quantum expects to see counts that appear like a saw tooth.
STATISTICS_WINDOW_QUANTUM_<collection> The same as STATISTICS_WINDOW_QUANTUM, but
used to override the global setting for a particular statistic collection. Collection names currently implemented
are DC or DAEMONCORE and SCHEDD or SCHEDULER.
TCP_KEEPALIVE_INTERVAL The number of seconds specifying a keep alive interval to use for any HTCondor
TCP connection. The default keep alive interval is 360 (6 minutes); this value is chosen to minimize the likelihood that keep alive packets are sent, while still detecting dead TCP connections before job leases expire. A
smaller value will consume more operating system and network resources, while a larger value may cause jobs
to fail unnecessarily due to network disconnects. Most users will not need to tune this configuration variable. A
value of 0 will use the operating system default, and a value of -1 will disable HTCondor’s use of a TCP keep
alive.
ENABLE_IPV4 A boolean with the additional special value of auto. If true, HTCondor will use IPv4 if available,
and fail otherwise. If false, HTCondor will not use IPv4. If auto, HTCondor will use IPv4 if it can find an
interface with an IPv4 address; this is the default.
HTCondor Version 8.6.4 Manual
234
3.5.2. HTCondor-wide Configuration File Entries
ENABLE_IPV6 A boolean with the additional special value of auto. If true, HTCondor will use IPv6 if available,
and fail otherwise. If false, HTCondor will not use IPv6. If auto, HTCondor will use IPv6 if it can find an
interface with an IPv6 address; this is the default.
PREFER_IPV4 A boolean which will cause HTCondor to prefer IPv4 when it is able to choose. HTCondor will
otherwise prefer IPv6. The default is True.
ADVERTISE_IPV4_FIRST A string (treated as a boolean). If ADVERTISE_IPV4_FIRST evaluates to True,
HTCondor will advertise its IPv4 addresses before its IPv6 addresses; otherwise the IPv6 addresses will come
first. Defaults to $(PREFER_IPV4).
IGNORE_TARGET_PROTOCOL_PREFERENCE A
string
(treated
as
a
boolean).
If
IGNORE_TARGET_PROTOCOL_PREFERENCE evaluates to True, the target’s listed protocol preferences
will be ignored; othwerwise they will not. Defaults to $(PREFER_IPV4).
IGNORE_DNS_PROTOCOL_PREFERENCE A
string
(treated
as
a
boolean).
IGNORE_DNS_PROTOCOL_PREFERENCE evaluates to True, the protocol order returned by the DNS
will be ignored; otherwise it will not. Defaults to $(PREFER_IPV4).
PREFER_OUTBOUND_IPV4 A string (treated as a boolean). PREFER_OUTBOUND_IPV4 evaluates to True, HTCondor will prefer IPv4; otherwise it will not. Defaults to $(PREFER_IPV4).
<SUBSYS>_CLASSAD_USER_MAP_NAMES A string defining a list of names for username-to-accounting group
mappings for the specified daemon. Names must be separated by spaces or commas.
CLASSAD_USER_MAPFILE_<name> A string giving the name of a file to parse to initialize the map for the given
username. Note that this macro is only used if <SUBSYS>_CLASSAD_USER_MAP_NAMES is defined for the
relevant daemon.
CLASSAD_USER_MAPDATA_<name> A string containing data to be used to initialize the map for the given username. Note that this macro is only used if <SUBSYS>_CLASSAD_USER_MAP_NAMES is defined for the
relevant daemon, and CLASSAD_USER_MAPFILE_<name> is not defined for the given name.
The format for the map file and map data is the same as the format for the security unified map file (see 3.8.4
for details).
The first field must be * (or a subset name - see below), the second field is a regex that we will match against
the input, and the third field will be the output if the regex matches, the 3 and 4 argument form of the ClassAd
userMap() function (see 4.1.2) expect that the third field will be a comma separated list of values. for example:
#
*
*
*
*
file:
John
Juan
Bob
Alice
groups.mapdata
chemistry,physics,glassblowing
physics,chemistry
security
security,math
Optional submaps: If the first field of the mapfile contains something other than *, then a submap is defined.
To select a submap for lookup, the first argument for userMap() should be "mapname.submap". For example:
HTCondor Version 8.6.4 Manual
235
3.5.3. Daemon Logging Configuration File Entries
# mapdata
Bob
*
Alice
*
alt Alice
'groups' with submaps
security
security,math
math,hacking
IGNORE_LEAF_OOM A boolean value that, when True, tells HTCondor not to kill and hold a job that is within its
memory allocation, even if other processes within the same cgroup have exceeded theirs. The default value is
True. (Note that this represents a change in behavior compared to versions of HTCondor older than 8.6.0; this
configuration macro first appeared in version 8.4.11. To restore the previous behavior, set this value to False.)
3.5.3 Daemon Logging Configuration File Entries
These entries control how and where the HTCondor daemons write to log files. Many of the entries in this section
represents multiple macros. There is one for each subsystem (listed in section 3.5.1). The macro name for each
substitutes <SUBSYS> with the name of the subsystem corresponding to the daemon.
<SUBSYS>_LOG Defines the path and file name of the log file for a given subsystem. For example,
$(STARTD_LOG) gives the location of the log file for the condor_startd daemon. The default value for
most daemons is the daemon’s name in camel case, concatenated with Log. For example, the default log
defined for the condor_master daemon is $(LOG)/MasterLog. The default value for other subsystems is
$(LOG)/<SUBSYS>LOG. If the log file cannot be written to, then the daemon will attempt to log this into a
new file of the name $(LOG)/dprintf_failure.<SUBSYS> before the daemon exits.
MAX_<SUBSYS>_LOG Controls the maximum size in bytes or amount of time that a log will be allowed to grow.
For any log not specified, the default is $(MAX_DEFAULT_LOG), which currently defaults to 10 MiB in size.
Values are specified with the same syntax as MAX_DEFAULT_LOG.
Note that a log file for the condor_procd does not use this configuration variable definition. Its implementation
is separate. See section 3.5.17 for the definition of MAX_PROCD_LOG.
MAX_DEFAULT_LOG Controls the maximum size in bytes or amount of time that any log not explicitly specified
using MAX_<SUBSYS>_LOG will be allowed to grow. When it is time to rotate a log file, it will be
saved to a file with an ISO timestamp suffix. The oldest rotated file receives the ending .old. The
.old files are overwritten each time the maximum number of rotated files (determined by the value of
MAX_NUM_<SUBSYS>_LOG) is exceeded. The default value is 10 MiB in size. A value of 0 specifies
that the file may grow without bounds. A single integer value is specified; without a suffix, it defaults to
specifying a size in bytes. A suffix is case insensitive, except for Mb and Min; these both start with the
same letter, and the implementation attaches meaning to the letter case when only the first letter is present.
Therefore, use the following suffixes to qualify the integer:
Bytes for bytes
Kb for KiB, 210 numbers of bytes
Mb for MiB, 220 numbers of bytes
Gb for GiB, 230 numbers of bytes
Tb for TiB, 240 numbers of bytes
HTCondor Version 8.6.4 Manual
236
3.5.3. Daemon Logging Configuration File Entries
Sec for seconds
Min for minutes
Hr for hours
Day for days
Wk for weeks
MAX_NUM_<SUBSYS>_LOG An integer that controls the maximum number of rotations a log file is allowed to
perform before the oldest one will be rotated away. Thus, at most MAX_NUM_<SUBSYS>_LOG + 1 log files
of the same program coexist at a given time. The default value is 1.
TRUNC_<SUBSYS>_LOG_ON_OPEN If this macro is defined and set to True, the affected log will be truncated and
started from an empty file with each invocation of the program. Otherwise, new invocations of the program will
append to the previous log file. By default this setting is False for all daemons.
<SUBSYS>_LOG_KEEP_OPEN A boolean value that controls whether or not the log file is kept open between writes.
When True, the daemon will not open and close the log file between writes. Instead the daemon will hold the
log file open until the log needs to be rotated. When False, the daemon reverts to the previous behavior
of opening and closing the log file between writes. When the $(<SUBSYS>_LOCK) macro is defined, setting $(<SUBSYS>_LOG_KEEP_OPEN) has no effect, as the daemon will unconditionally revert back to the
open/close between writes behavior. On Windows platforms, the value defaults to True for all daemons. On
Linux platforms, the value defaults to True for all daemons, except the condor_shadow, due to a global file
descriptor limit.
<SUBSYS>_LOCK This macro specifies the lock file used to synchronize append operations to the log file for this
subsystem. It must be a separate file from the $(<SUBSYS>_LOG) file, since the $(<SUBSYS>_LOG) file
may be rotated and you want to be able to synchronize access across log file rotations. A lock file is only
required for log files which are accessed by more than one process. Currently, this includes only the SHADOW
subsystem. This macro is defined relative to the $(LOCK) macro.
JOB_QUEUE_LOG A full path and file name, specifying the job queue log. The default value, when not defined is
$(SPOOL)/job_queue.log. This specification can be useful, if there is a solid state drive which is big
enough to hold the frequently written to job_queue.log, but not big enough to hold the whole contents of
the spool directory.
FILE_LOCK_VIA_MUTEX This macro setting only works on Win32 – it is ignored on Unix. If set to be True, then
log locking is implemented via a kernel mutex instead of via file locking. On Win32, mutex access is FIFO,
while obtaining a file lock is non-deterministic. Thus setting to True fixes problems on Win32 where processes
(usually shadows) could starve waiting for a lock on a log file. Defaults to True on Win32, and is always
False on Unix.
LOCK_DEBUG_LOG_TO_APPEND A boolean value that defaults to False. This variable controls whether a daemon’s debug lock is used when appending to the log. When False, the debug lock is only used when rotating
the log file. This is more efficient, especially when many processes share the same log file. When True, the
debug lock is used when writing to the log, as well as when rotating the log file. This setting is ignored under
Windows, and the behavior of Windows platforms is as though this variable were True. Under Unix, the default
value of False is appropriate when logging to file systems that support the POSIX semantics of O_APPEND.
On non-POSIX-compliant file systems, it is possible for the characters in log messages from multiple processes
HTCondor Version 8.6.4 Manual
237
3.5.3. Daemon Logging Configuration File Entries
sharing the same log to be interleaved, unless locking is used. Since HTCondor does not support sharing of
debug logs between processes running on different machines, many non-POSIX-compliant file systems will still
avoid interleaved messages without requiring HTCondor to use a lock. Tests of AFS and NFS have not revealed
any problems when appending to the log without locking.
ENABLE_USERLOG_LOCKING A boolean value that defaults to False on Unix platforms and True on Windows
platforms. When True, a user’s job event log will be locked before being written to. If False, HTCondor will
not lock the file before writing.
ENABLE_USERLOG_FSYNC A boolean value that is True by default. When True, writes to the user’s job event
log are sync-ed to disk before releasing the lock.
USERLOG_FILE_CACHE_MAX The integer number of job event log files that the condor_schedd will keep open for
writing during an interval of time (specified by USERLOG_FILE_CACHE_CLEAR_INTERVAL). The default
value is 0, causing no files to remain open; when 0, each job event log is opened, the event is written, and then
the file is closed. Individual file descriptors are removed from this count when the condor_schedd detects that
no jobs are currently using them. Opening a file is a relatively time consuming operation on a networked file
system (NFS), and therefore, allowing a set of files to remain open can improve performance. The value of
this variable needs to be set low enough such that the condor_schedd daemon process does not run out of file
descriptors by leaving these job event log files open. The Linux operating system defaults to permitting 1024
assigned file descriptors per process; the condor_schedd will have one file descriptor per running job for the
condor_shadow.
USERLOG_FILE_CACHE_CLEAR_INTERVAL The integer number of seconds that forms the time interval within
which job event logs will be permitted to remain open when USERLOG_FILE_CACHE_MAX is greater than
zero. The default is 60 seconds. When the interval has passed, all job event logs that the condor_schedd has
permitted to stay open will be closed, and the interval within which job event logs may remain open between
writes of events begins anew. This time interval may be set to a longer duration if the administrator determines
that the condor_schedd will not exceed the maximum number of file descriptors; a longer interval may yield
higher performance due to fewer files being opened and closed.
EVENT_LOG_COUNT_EVENTS A boolean value that is False by default. When True, upon rotation of the user’s
job event log, a count of the number of job events is taken by scanning the log, such that the newly created,
post-rotation user job event log will have this count in its header. This configuration variable is relevant when
rotation of the user’s job event log is enabled.
CREATE_LOCKS_ON_LOCAL_DISK A boolean value utilized only for Unix operating systems, that defaults to
True. This variable is only relevant if ENABLE_USERLOG_LOCKING is True. When True, lock files are
written to a directory named condorLocks, thereby using a local drive to avoid known problems with locking
on NFS. The location of the condorLocks directory is determined by
1. The value of TEMP_DIR, if defined.
2. The value of TMP_DIR, if defined and TEMP_DIR is not defined.
3. The default value of /tmp, if neither TEMP_DIR nor TMP_DIR is defined.
TOUCH_LOG_INTERVAL The time interval in seconds between when daemons touch their log files. The change
in last modification time for the log file is useful when a daemon restarts after failure or shut down. The last
modification date is printed, and it provides an upper bound on the length of time that the daemon was not
running. Defaults to 60 seconds.
HTCondor Version 8.6.4 Manual
238
3.5.3. Daemon Logging Configuration File Entries
LOGS_USE_TIMESTAMP This macro controls how the current time is formatted at the start of each
line in the daemon log files.
When True, the Unix time is printed (number of seconds since
00:00:00 UTC, January 1, 1970). When False (the default value), the time is printed like so:
<Month>/<Day> <Hour>:<Minute>:<Second> in the local timezone.
DEBUG_TIME_FORMAT This string defines how to format the current time printed at the start of each line in the
daemon log files. The value is a format string is passed to the C strftime() function, so see that manual
page for platform-specific details. If not defined, the default value is
"%m/%d/%y %H:%M:%S"
<SUBSYS>_DEBUG All of the HTCondor daemons can produce different levels of output depending on how much
information is desired. The various levels of verbosity for a given daemon are determined by this macro. All
daemons have the default level D_ALWAYS, and log messages for that level will be printed to the daemon’s log,
regardless of this macro’s setting. Settings are a comma- or space-separated list of the following values:
D_ALL This flag turns on all debugging output by enabling all of the debug levels at once. There is no need
to list any other debug levels in addition to D_ALL; doing so would be redundant. Be warned: this will
generate about a HUGE amount of output. To obtain a higher level of output than the default, consider
using D_FULLDEBUG before using this option.
D_FULLDEBUG This level provides verbose output of a general nature into the log files. Frequent log messages
for very specific debugging purposes would be excluded. In those cases, the messages would be viewed
by having that another flag and D_FULLDEBUG both listed in the configuration file.
D_DAEMONCORE Provides log file entries specific to DaemonCore, such as timers the daemons have set and the
commands that are registered. If both D_FULLDEBUG and D_DAEMONCORE are set, expect very verbose
output.
D_PRIV This flag provides log messages about the privilege state switching that the daemons do. See section 3.8.13 on UIDs in HTCondor for details.
D_COMMAND With this flag set, any daemon that uses DaemonCore will print out a log message whenever a
command comes in. The name and integer of the command, whether the command was sent via UDP or
TCP, and where the command was sent from are all logged. Because the messages about the command
used by condor_kbdd to communicate with the condor_startd whenever there is activity on the X server,
and the command used for keep-alives are both only printed with D_FULLDEBUG enabled, it is best if this
setting is used for all daemons.
D_LOAD The condor_startd keeps track of the load average on the machine where it is running. Both the
general system load average, and the load average being generated by HTCondor’s activity there are determined. With this flag set, the condor_startd will log a message with the current state of both of these load
averages whenever it computes them. This flag only affects the condor_startd.
D_KEYBOARD With this flag set, the condor_startd will print out a log message with the current values for
remote and local keyboard idle time. This flag affects only the condor_startd.
D_JOB When this flag is set, the condor_startd will send to its log file the contents of any job ClassAd that the
condor_schedd sends to claim the condor_startd for its use. This flag affects only the condor_startd.
D_MACHINE When this flag is set, the condor_startd will send to its log file the contents of its resource
ClassAd when the condor_schedd tries to claim the condor_startd for its use. This flag affects only the
condor_startd.
HTCondor Version 8.6.4 Manual
239
3.5.3. Daemon Logging Configuration File Entries
D_SYSCALLS This flag is used to make the condor_shadow log remote syscall requests and return values. This
can help track down problems a user is having with a particular job by providing the system calls the job
is performing. If any are failing, the reason for the failure is given. The condor_schedd also uses this flag
for the server portion of the queue management code. With D_SYSCALLS defined in SCHEDD_DEBUG
there will be verbose logging of all queue management operations the condor_schedd performs.
D_MATCH When this flag is set, the condor_negotiator logs a message for every match.
D_NETWORK When this flag is set, all HTCondor daemons will log a message on every TCP accept, connect,
and close, and on every UDP send and receive. This flag is not yet fully supported in the condor_shadow.
D_HOSTNAME When this flag is set, the HTCondor daemons and/or tools will print verbose messages explaining how they resolve host names, domain names, and IP addresses. This is useful for sites that are having
trouble getting HTCondor to work because of problems with DNS, NIS or other host name resolving
systems in use.
D_CKPT When this flag is set, the HTCondor process checkpoint support code, which is linked into a STANDARD universe user job, will output some low-level details about the checkpoint procedure into the
$(SHADOW_LOG).
D_SECURITY This flag will enable debug messages pertaining to the setup of secure network communication,
including messages for the negotiation of a socket authentication mechanism, the management of a session
key cache. and messages about the authentication process itself. See section 3.8.1 for more information
about secure communication configuration.
D_PROCFAMILY HTCondor often times needs to manage an entire family of processes, (that is, a process and
all descendants of that process). This debug flag will turn on debugging output for the management of
families of processes.
D_ACCOUNTANT When this flag is set, the condor_negotiator will output debug messages relating to the computation of user priorities (see section 3.6).
D_PROTOCOL Enable debug messages relating to the protocol for HTCondor’s matchmaking and resource
claiming framework.
D_STATS Enable debug messages relating to the TCP statistics for file transfers. Note that the shadow and
starter, by default, log these statistics to special log files (see SHADOW_STATS_LOG section 3.5.11 and
STARTER_STATS_LOG, section 3.5.12). Note that, as of version 8.5.6, C_GAHP_DEBUG defaults to
D_STATS.
D_PID This flag is different from the other flags, because it is used to change the formatting of all log messages
that are printed, as opposed to specifying what kinds of messages should be printed. If D_PID is set,
HTCondor will always print out the process identifier (PID) of the process writing each line to the log
file. This is especially helpful for HTCondor daemons that can fork multiple helper-processes (such as
the condor_schedd or condor_collector) so the log file will clearly show which thread of execution is
generating each log message.
D_FDS This flag is different from the other flags, because it is used to change the formatting of all log messages that are printed, as opposed to specifying what kinds of messages should be printed. If D_FDS is
set, HTCondor will always print out the file descriptor that the open of the log file was allocated by the
operating system. This can be helpful in debugging HTCondor’s use of system file descriptors as it will
generally track the number of file descriptors that HTCondor has open.
D_CATEGORY This flag is different from the other flags, because it is used to change the formatting of all
log messages that are printed, as opposed to specifying what kinds of messages should be printed. If
HTCondor Version 8.6.4 Manual
240
3.5.3. Daemon Logging Configuration File Entries
D_CATEGORY is set, Condor will include the debugging level flags that were in effect for each line of
output. This may be used to filter log output by the level or tag it, for example, identifying all logging
output at level D_SECURITY, or D_ACCOUNTANT.
D_TIMESTAMP This flag is different from the other flags, because it is used to change the formatting of all
log messages that are printed, as opposed to specifying what kinds of messages should be printed. If
D_TIMESTAMP is set, the time at the beginning of each line in the log file with be a number of seconds
since the start of the Unix era. This form of timestamp can be more convenient for tools to process.
D_SUB_SECOND This flag is different from the other flags, because it is used to change the formatting of all
log messages that are printed, as opposed to specifying what kinds of messages should be printed. If
D_SUB_SECOND is set, the time at the beginning of each line in the log file will contain a fractional part
to the seconds field that is accurate to the millisecond.
ALL_DEBUG Used to make all subsystems share a debug flag. Set the parameter ALL_DEBUG instead of
changing all of the individual parameters. For example, to turn on all debugging in all subsystems, set
ALL_DEBUG = D_ALL.
TOOL_DEBUG Uses the same values (debugging levels) as <SUBSYS>_DEBUG to describe the amount of debugging
information sent to stderr for HTCondor tools.
Log files may optionally be specified per debug level as follows:
<SUBSYS>_<LEVEL>_LOG The name of a log file for messages at a specific debug level for a specific subsystem.
<LEVEL> is defined by any debug level, but without the D_ prefix. See section 3.5.3 for the list of debug levels.
If the debug level is included in $(<SUBSYS>_DEBUG), then all messages of this debug level will be written
both to the log file defined by <SUBSYS>_LOG and the the log file defined by <SUBSYS>_<LEVEL>_LOG.
As examples, SHADOW_SYSCALLS_LOG specifies a log file for all remote system call debug messages, and
NEGOTIATOR_MATCH_LOG specifies a log file that only captures condor_negotiator debug events occurring
with matches.
MAX_<SUBSYS>_<LEVEL>_LOG See section 3.5.3, the definition of MAX_<SUBSYS>_LOG.
TRUNC_<SUBSYS>_<LEVEL>_LOG_ON_OPEN Similar to TRUNC_<SUBSYS>_LOG_ON_OPEN.
The following macros control where and what is written to the event log, a file that receives job events, but across
all users and user’s jobs.
EVENT_LOG The full path and file name of the event log. There is no default value for this variable, so no event log
will be written, if not defined.
EVENT_LOG_MAX_SIZE Controls the maximum length in bytes to which the event log will be allowed to grow.
The log file will grow to the specified length, then be saved to a file with the suffix .old. The .old files are
overwritten each time the log is saved. A value of 0 specifies that the file may grow without bounds (and
disables rotation). The default is 1 MiB. For backwards compatibility, MAX_EVENT_LOG will be used if
EVENT_LOG_MAX_SIZE is not defined. If EVENT_LOG is not defined, this parameter has no effect.
MAX_EVENT_LOG See EVENT_LOG_MAX_SIZE.
HTCondor Version 8.6.4 Manual
241
3.5.4. DaemonCore Configuration File Entries
EVENT_LOG_MAX_ROTATIONS Controls the maximum number of rotations of the event log that will be stored. If
this value is 1 (the default), the event log will be rotated to a “.old” file as described above. However, if this is
greater than 1, then multiple rotation files will be stores, up to EVENT_LOG_MAX_ROTATIONS of them. These
files will be named, instead of the “.old” suffix, “.1”, “.2”, with the “.1” being the most recent rotation. This is
an integer parameter with a default value of 1. If EVENT_LOG is not defined, or if EVENT_LOG_MAX_SIZE
has a value of 0 (which disables event log rotation), this parameter has no effect.
EVENT_LOG_ROTATION_LOCK Specifies the lock file that will be used to ensure that, when rotating files, the
rotation is done by a single process. This is a string parameter; its default value is $(LOCK)/EventLogLock.
If an empty value is set, then the file that is used is the file path of the event log itself, with the string .lock
appended. If EVENT_LOG is not defined, or if EVENT_LOG_MAX_SIZE has a value of 0 (which disables event
log rotation), this configuration variable has no effect.
EVENT_LOG_FSYNC A boolean value that controls whether HTCondor will perform an fsync() after writing
each event to the event log. When True, an fsync() operation is performed after each event. This fsync()
operation forces the operating system to synchronize the updates to the event log to the disk, but can negatively
affect the performance of the system. Defaults to False.
EVENT_LOG_LOCKING A boolean value that defaults to False on Unix platforms and True on Windows platforms. When True, the event log (as specified by EVENT_LOG) will be locked before being written to. When
False, HTCondor does not lock the file before writing.
EVENT_LOG_USE_XML A boolean value that defaults to False. When True, events are logged in XML format.
If EVENT_LOG is not defined, this parameter has no effect.
EVENT_LOG_JOB_AD_INFORMATION_ATTRS A comma separated list of job ClassAd attributes, whose evaluated values form a new event, the JobAdInformationEvent, given Event Number 028. This new
event is placed in the event log in addition to each logged event. If EVENT_LOG is not defined, this
configuration variable has no effect. This configuration variable is the same as the job ClassAd attribute
JobAdInformationAttrs (see page 1010), but it applies to the system Event Log rather than the user
job log.
3.5.4 DaemonCore Configuration File Entries
Please read section 3.11 for details on DaemonCore. There are certain configuration file settings that DaemonCore uses
which affect all HTCondor daemons (except the checkpoint server, standard universe shadow, and standard universe
starter, none of which use DaemonCore).
HOSTALLOW. . . All macros that begin with either HOSTALLOW or HOSTDENY are settings for HTCondor’s hostbased security. See section 3.8.9 on Setting up IP/host-based security in HTCondor for details on these macros
and how to configure them.
ENABLE_RUNTIME_CONFIG The condor_config_val tool has an option -rset for dynamically setting run time configuration values, and which only affect the in-memory configuration variables. Because of the potential security
implications of this feature, by default, HTCondor daemons will not honor these requests. To use this functionality, HTCondor administrators must specifically enable it by setting ENABLE_RUNTIME_CONFIG to True,
HTCondor Version 8.6.4 Manual
242
3.5.4. DaemonCore Configuration File Entries
and specify what configuration variables can be changed using the SETTABLE_ATTRS. . . family of configuration options. Defaults to False.
ENABLE_PERSISTENT_CONFIG The condor_config_val tool has a -set option for dynamically setting persistent configuration values. These values override options in the normal HTCondor configuration files. Because of the potential security implications of this feature, by default, HTCondor daemons will not honor
these requests. To use this functionality, HTCondor administrators must specifically enable it by setting
ENABLE_PERSISTENT_CONFIG to True, creating a directory where the HTCondor daemons will hold these
dynamically-generated persistent configuration files (declared using PERSISTENT_CONFIG_DIR, described
below) and specify what configuration variables can be changed using the SETTABLE_ATTRS. . . family of
configuration options. Defaults to False.
PERSISTENT_CONFIG_DIR Directory where daemons should store dynamically-generated persistent configuration files (used to support condor_config_val -set) This directory should only be writable by root, or the user
the HTCondor daemons are running as (if non-root). There is no default, administrators that wish to use this
functionality must create this directory and define this setting. This directory must not be shared by multiple
HTCondor installations, though it can be shared by all HTCondor daemons on the same host. Keep in mind
that this directory should not be placed on an NFS mount where “root-squashing” is in effect, or else HTCondor
daemons running as root will not be able to write to them. A directory (only writable by root) on the local file
system is usually the best location for this directory.
SETTABLE_ATTRS_<PERMISSION-LEVEL> All macros that begin with SETTABLE_ATTRS or
<SUBSYS>.SETTABLE_ATTRS are settings used to restrict the configuration values that can be changed
using the condor_config_val command. Section 3.8.9 on Setting up IP/Host-Based Security in HTCondor for
details on these macros and how to configure them. In particular, section 3.8.9 on page 439 contains details
specific to these macros.
SHUTDOWN_GRACEFUL_TIMEOUT Determines how long HTCondor will allow daemons try their graceful shutdown methods before they do a hard shutdown. It is defined in terms of seconds. The default is 1800 (30
minutes).
<SUBSYS>_ADDRESS_FILE A complete path to a file that is to contain an IP address and port number for a daemon. Every HTCondor daemon that uses DaemonCore has a command port where commands are sent. The
IP/port of the daemon is put in that daemon’s ClassAd, so that other machines in the pool can query the condor_collector (which listens on a well-known port) to find the address of a given daemon on a given machine.
When tools and daemons are all executing on the same single machine, communications do not require a query
of the condor_collector daemon. Instead, they look in a file on the local disk to find the IP/port. This macro
causes daemons to write the IP/port of their command socket to a specified file. In this way, local tools will
continue to operate, even if the machine running the condor_collector crashes. Using this file will also generate slightly less network traffic in the pool, since tools including condor_q and condor_rm do not need to
send any messages over the network to locate the condor_schedd daemon. This macro is not necessary for the
condor_collector daemon, since its command socket is at a well-known port.
The macro is named by substituting <SUBSYS> with the appropriate subsystem string as defined in section 3.5.1.
<SUBSYS>_SUPER_ADDRESS_FILE A complete path to a file that is to contain an IP address and port number for a command port that is serviced with priority for a daemon. Every HTCondor daemon that uses
DaemonCore may have a higher priority command port where commands are sent. Any command that
HTCondor Version 8.6.4 Manual
243
3.5.4. DaemonCore Configuration File Entries
goes through condor_sos, and any command issued by the super user (root or local system) for a daemon on the local machine will have the command sent to this port. Default values are provided for the
condor_schedd daemon at $(SPOOL)/.schedd_address.super and the condor_collector daemon at
$(LOG)/.collector_address.super. When not defined for other DaemonCore daemons, there will
be no higher priority command port.
<SUBSYS>_DAEMON_AD_FILE A complete path to a file that is to contain the ClassAd for a daemon. When the
daemon sends a ClassAd describing itself to the condor_collector, it will also place a copy of the ClassAd in
this file. Currently, this setting only works for the condor_schedd.
<SUBSYS>_ATTRS or <SUBSYS>_EXPRS Allows any DaemonCore daemon to advertise arbitrary expressions
from the configuration file in its ClassAd. Give the comma-separated list of entries from the configuration
file you want in the given daemon’s ClassAd. Frequently used to add attributes to machines so that the machines
can discriminate between other machines in a job’s rank and requirements.
The macro is named by substituting <SUBSYS> with the appropriate subsystem string as defined in section 3.5.1.
<SUBSYS>_EXPRS is a historic setting that functions identically to <SUBSYS>_ATTRS. It may be removed
in the future, so use <SUBSYS>_ATTRS.
NOTE: The condor_kbdd does not send ClassAds now, so this entry does not affect it. The condor_startd,
condor_schedd, condor_master, and condor_collector do send ClassAds, so those would be valid subsystems
to set this entry for.
SUBMIT_ATTRS not part of the <SUBSYS>_ATTRS, it is documented in section 3.5.13
Because of the different syntax of the configuration file and ClassAds, a little extra work is required to get a
given entry into a ClassAd. In particular, ClassAds require quote marks (") around strings. Numeric values
and boolean expressions can go in directly. For example, if the condor_startd is to advertise a string macro, a
numeric macro, and a boolean expression, do something similar to:
STRING = This is a string
NUMBER = 666
BOOL1 = True
BOOL2 = time() >= $(NUMBER) || $(BOOL1)
MY_STRING = "$(STRING)"
STARTD_ATTRS = MY_STRING, NUMBER, BOOL1, BOOL2
DAEMON_SHUTDOWN Starting with HTCondor version 6.9.3, whenever a daemon is about to publish a ClassAd
update to the condor_collector, it will evaluate this expression. If it evaluates to True, the daemon will
gracefully shut itself down, exit with the exit code 99, and will not be restarted by the condor_master
(as if it sent itself a condor_off command). The expression is evaluated in the context of the ClassAd
that is being sent to the condor_collector, so it can reference any attributes that can be seen with
condor_status -long [-daemon_type] (for example, condor_status -long [-master]
for the condor_master). Since each daemon’s ClassAd will contain different attributes, administrators should
define these shutdown expressions specific to each daemon, for example:
HTCondor Version 8.6.4 Manual
244
3.5.4. DaemonCore Configuration File Entries
STARTD.DAEMON_SHUTDOWN = when to shutdown the startd
MASTER.DAEMON_SHUTDOWN = when to shutdown the master
Normally, these expressions would not be necessary, so if not defined, they default to FALSE.
NOTE: This functionality does not work in conjunction with HTCondor’s high-availability support (see section 3.13 on page 475 for more information). If you enable high-availability for a particular daemon, you should
not define this expression.
DAEMON_SHUTDOWN_FAST Identical to DAEMON_SHUTDOWN (defined above), except the daemon will use the fast
shutdown mode (as if it sent itself a condor_off command using the -fast option).
USE_CLONE_TO_CREATE_PROCESSES A boolean value that controls how an HTCondor daemon creates a new
process on Linux platforms. If set to the default value of True, the clone system call is used. Otherwise,
the fork system call is used. clone provides scalability improvements for daemons using a large amount of
memory, for example, a condor_schedd with a lot of jobs in the queue. Currently, the use of clone is available
on Linux systems. If HTCondor detects that it is running under the valgrind analysis tools, this setting is ignored
and treated as False, to work around incompatibilities.
MAX_TIME_SKIP When an HTCondor daemon notices the system clock skip forwards or backwards more than the
number of seconds specified by this parameter, it may take special action. For instance, the condor_master will
restart HTCondor in the event of a clock skip. Defaults to a value of 1200, which in effect means that HTCondor
will restart if the system clock jumps by more than 20 minutes.
NOT_RESPONDING_TIMEOUT When an HTCondor daemon’s parent process is another HTCondor daemon, the
child daemon will periodically send a short message to its parent stating that it is alive and well. If the parent
does not hear from the child for a while, the parent assumes that the child is hung, kills the child, and restarts
the child. This parameter controls how long the parent waits before killing the child. It is defined in terms of
seconds and defaults to 3600 (1 hour). The child sends its alive and well messages at an interval of one third of
this value.
<SUBSYS>_NOT_RESPONDING_TIMEOUT Identical to NOT_RESPONDING_TIMEOUT, but controls the timeout
for a specific type of daemon. For example, SCHEDD_NOT_RESPONDING_TIMEOUT controls how long the
condor_schedd’s parent daemon will wait without receiving an alive and well message from the condor_schedd
before killing it.
NOT_RESPONDING_WANT_CORE A boolean value with a default value of False. This parameter is for debugging
purposes on Unix systems, and it controls the behavior of the parent process when the parent process determines that a child process is not responding. If NOT_RESPONDING_WANT_CORE is True, the parent will
send a SIGABRT instead of SIGKILL to the child process. If the child process is configured with the configuration variable CREATE_CORE_FILES enabled, the child process will then generate a core dump. See
NOT_RESPONDING_TIMEOUT on page 245, and CREATE_CORE_FILES on page 231 for related details.
LOCK_FILE_UPDATE_INTERVAL An integer value representing seconds, controlling how often valid lock files
should have their on disk timestamps updated. Updating the timestamps prevents administrative programs, such
as tmpwatch, from deleting long lived lock files. If set to a value less than 60, the update time will be 60 seconds.
The default value is 28800, which is 8 hours. This variable only takes effect at the start or restart of a daemon.
HTCondor Version 8.6.4 Manual
245
3.5.5. Network-Related Configuration File Entries
SOCKET_LISTEN_BACKLOG An integer value that defaults to 500, which defines the backlog value for the
listen() network call when a daemon creates a socket for incoming connections. It limits the number of
new incoming network connections the operating system will accept for a daemon that the daemon has not yet
serviced.
MAX_ACCEPTS_PER_CYCLE An integer value that defaults to 8. It is a rarely changed performance tuning parameter to limit the number of accepts of new, incoming, socket connect requests per DaemonCore event cycle. A
value of zero or less means no limit. It has the most noticeable effect on the condor_schedd, and would be given
a higher integer value for tuning purposes when there is a high number of jobs starting and exiting per second.
MAX_REAPS_PER_CYCLE An integer value that defaults to 0. It is a rarely changed performance tuning parameter
that places a limit on the number of child process exits to process per DaemonCore event cycle. A value of zero
or less means no limit.
CORE_FILE_NAME Defines the name of the core file created. Defaults to core.$(SUBSYSTEM) on Unix platforms, and core.$(SUBSYSTEM).WIN32 on Windows platforms.
PIPE_BUFFER_MAX The maximum number of bytes read from a stdout or stdout pipe. The default value is
10240. A rare example in which the value would need to increase from its default value is when a hook must
output an entire ClassAd, and the ClassAd may be larger than the default.
3.5.5 Network-Related Configuration File Entries
More information about networking in HTCondor can be found in section 3.9 on page 450.
BIND_ALL_INTERFACES For systems with multiple network interfaces, if this configuration setting is False,
HTCondor will only bind network sockets to the IP address specified with NETWORK_INTERFACE (described
below). If set to True, the default value, HTCondor will listen on all interfaces. However, currently HTCondor
is still only able to advertise a single IP address, even if it is listening on multiple interfaces. By default, it will
advertise the IP address of the network interface used to contact the collector, since this is the most likely to be
accessible to other processes which query information from the same collector. More information about using
this setting can be found in section 3.9.3 on page 456.
CCB_ADDRESS This is the address of a condor_collector that will serve as this daemon’s HTCondor Connection
Broker (CCB). Multiple addresses may be listed (separated by commas and/or spaces) for redundancy. The CCB
server must authorize this daemon at DAEMON level for this configuration to succeed. It is highly recommended
to also configure PRIVATE_NETWORK_NAME if you configure CCB_ADDRESS so communications originating
within the same private network do not need to go through CCB. For more information about CCB, see page 459.
CCB_HEARTBEAT_INTERVAL This is the maximum number of seconds of silence on a daemon’s connection to
the CCB server after which it will ping the server to verify that the connection still works. The default is
5 minutes. This feature serves to both speed up detection of dead connections and to generate a guaranteed
minimum frequency of activity to attempt to prevent the connection from being dropped. The special value 0
disables the heartbeat. The heartbeat is automatically disabled if the CCB server is older than HTCondor version
7.5.0. Having the heartbeat interval greater than the job ClassAd attribute JobLeaseDuration may cause
unnecessary job disconnects in pools with network issues.
HTCondor Version 8.6.4 Manual
246
3.5.5. Network-Related Configuration File Entries
CCB_POLLING_INTERVAL In seconds, the smallest amount of time that could go by before CCB would begin another round of polling to check on already connected clients. While the value of this variable does not change,
the actual interval used may be exceeded if the measured amount of time previously taken to poll to check on already connected clients exceeded the amount of time desired, as expressed with CCB_POLLING_TIMESLICE.
The default value is 20 seconds.
CCB_POLLING_MAX_INTERVAL In seconds, the interval of time after which polling to check on already connected
clients must occur, independent of any other factors. The default value is 600 seconds.
CCB_POLLING_TIMESLICE A floating point fraction representing the fractional amount of the total run time of
CCB to set as a target for the maximum amount of CCB running time used on polling to check on already
connected clients. The default value is 0.05.
CCB_READ_BUFFER The size of the kernel TCP read buffer in bytes for all sockets used by CCB. The default value
is 2 KiB.
CCB_WRITE_BUFFER The size of the kernel TCP write buffer in bytes for all sockets used by CCB. The default
value is 2 KiB.
CCB_SWEEP_INTERVAL The interval, in seconds, between times when the CCB server writes its information about
open TCP connections to a file. Crash recovery is accomplished using the information. The default value is
1200 seconds (20 minutes).
CCB_RECONNECT_FILE The full path and file name of the file that the CCB server writes its information about
open TCP connections to a file. Crash recovery is accomplished using the information. The default value is
$(SPOOL)/.ccb_reconnect.
COLLECTOR_USES_SHARED_PORT A boolean value that specifies whether the condor_collector uses the condor_shared_port daemon. When true, the condor_shared_port will transparently proxy queries to the condor_collector so users do not need to be aware of the presence of the condor_shared_port when querying the
collector and configuring other daemons. The default is True
SHARED_PORT_DEFAULT_ID When COLLECTOR_USES_SHARED_PORT is set to True, this is the shared port
ID used by the condor_collector. This defaults to collector and will not need to be changed by most sites.
AUTO_INCLUDE_SHARED_PORT_IN_DAEMON_LIST A boolean value that specifies whether SHARED_PORT
should be automatically inserted into condor_master’s DAEMON_LIST when USE_SHARED_PORT is True.
The default for this setting is True.
<SUBSYS>_MAX_FILE_DESCRIPTORS This setting is identical to MAX_FILE_DESCRIPTORS, but it only applies to a specific subsystem. If the subsystem-specific setting is unspecified, MAX_FILE_DESCRIPTORS is
used. For the condor_collector daemon, the value defaults to 10240, and for the condor_schedd daemon, the
value defaults to 4096. If the condor_shared_port daemon is in use, its value for this parameter should match
the largest value set for the other daemons.
MAX_FILE_DESCRIPTORS Under Unix, this specifies the maximum number of file descriptors to allow the HTCondor daemon to use. File descriptors are a system resource used for open files and for network connections.
HTCondor daemons that make many simultaneous network connections may require an increased number of file
descriptors. For example, see page 459 for information on file descriptor requirements of CCB. Changes to this
configuration variable require a restart of HTCondor in order to take effect. Also note that only if HTCondor is
running as root will it be able to increase the limit above the hard limit (on maximum open files) that it inherits.
HTCondor Version 8.6.4 Manual
247
3.5.5. Network-Related Configuration File Entries
NETWORK_HOSTNAME The host name to use, overriding the value returned by gethostname(), which will be
invoked by default to query the operating system to obtain the host name of the local machine. Among other
things, the host name is used to identify daemons in an HTCondor pool, via the Machine and Name attributes
of daemon ClassAds. This variable can be used when a machine has multiple network interfaces with different
host names, to use a host name that is not the primary one.
NETWORK_INTERFACE An IP address of the form 123.123.123.123 or the name of a network device, as in
the example eth0. The wild card character (*) may be used within either. For example, 123.123.* would
match a network interface with an IP address of 123.123.123.123 or 123.123.100.100. The default
value is *, which matches all network interfaces.
The effect of this variable depends on the value of BIND_ALL_INTERFACES. There are two cases:
If BIND_ALL_INTERFACES is True (the default), NETWORK_INTERFACE controls what IP address will
be advertised as the public address of the daemon. If multiple network interfaces match the value and
ENABLE_ADDRESS_REWRITING is True (the default), the IP address that is chosen to be advertised will
be the one that is used to communicate with the condor_collector. If ENABLE_ADDRESS_REWRITING is
False, the IP address that is chosen to be advertised will be the one associated with the first device (in systemdefined order) that is in a public address space, or a private address space, or a loopback address, in that order
of preference. If it is desired to advertise an IP address that is not associated with any local network interface,
for example, when TCP forwarding is being used, then TCP_FORWARDING_HOST should be used instead of
NETWORK_INTERFACE.
If BIND_ALL_INTERFACES is False, then NETWORK_INTERFACE specifies which IP address HTCondor
should use for all incoming and outgoing communication. If more than one IP address matches the value, then
the IP address that is chosen will be the one associated with the first device (in system-defined order) that is in
a public address space, or a private address space, or a loopback address, in that order of preference.
More information about configuring HTCondor on machines with multiple network interfaces can be found in
section 3.9.3 on page 455.
PRIVATE_NETWORK_NAME If two HTCondor daemons are trying to communicate with each other, and they both
belong to the same private network, this setting will allow them to communicate directly using the private network interface, instead of having to use CCB or to go through a public IP address. Each private network should
be assigned a unique network name. This string can have any form, but it must be unique for a particular private
network. If another HTCondor daemon or tool is configured with the same PRIVATE_NETWORK_NAME, it will
attempt to contact this daemon using its private network address. Even for sites using CCB, this is an important
optimization, since it means that two daemons on the same network can communicate directly, without having to
go through the broker. If CCB is enabled, and the PRIVATE_NETWORK_NAME is defined, the daemon’s private
address will be defined automatically. Otherwise, you can specify a particular private IP address to use by defining the PRIVATE_NETWORK_INTERFACE setting (described below). The default is $(FULL_HOSTNAME).
After changing this setting and running condor_reconfig, it may take up to one condor_collector update interval
before the change becomes visible.
PRIVATE_NETWORK_INTERFACE For systems with multiple network interfaces, if this configuration setting and
PRIVATE_NETWORK_NAME are both defined, HTCondor daemons will advertise some additional attributes in
their ClassAds to help other HTCondor daemons and tools in the same private network to communicate directly.
PRIVATE_NETWORK_INTERFACE defines what IP address of the form 123.123.123.123 or name of a
network device (as in the example eth0) a given multi-homed machine should use for the private network.
HTCondor Version 8.6.4 Manual
248
3.5.5. Network-Related Configuration File Entries
The asterisk (*) may be used as a wild card character within either the IP address or the device name. If another HTCondor daemon or tool is configured with the same PRIVATE_NETWORK_NAME, it will attempt to
contact this daemon using the IP address specified here. The syntax for specifying an IP address is identical
to NETWORK_INTERFACE. Sites using CCB only need to define the PRIVATE_NETWORK_NAME, and the
PRIVATE_NETWORK_INTERFACE will be defined automatically. Unless CCB is enabled, there is no default value for this variable. After changing this variable and running condor_reconfig, it may take up to one
condor_collector update interval before the change becomes visible.
TCP_FORWARDING_HOST This specifies the host or IP address that should be used as the public address of this
daemon. If a host name is specified, be aware that it will be resolved to an IP address by this daemon, not by the
clients wishing to connect to it. It is the IP address that is advertised, not the host name. This setting is useful if
HTCondor on this host may be reached through a NAT or firewall by connecting to an IP address that forwards
connections to this host. It is assumed that the port number on the TCP_FORWARDING_HOST that forwards to
this host is the same port number assigned to HTCondor on this host. This option could also be used when ssh
port forwarding is being used. In this case, the incoming addresses of connections to this daemon will appear
as though they are coming from the forwarding host rather than from the real remote host, so any authorization
settings that rely on host addresses should be considered accordingly.
ENABLE_ADDRESS_REWRITING A boolean value that defaults to True. When NETWORK_INTERFACE matches
only one IP address or TCP_FORWARDING_HOST is defined or NET_REMAP_ENABLE is True, this setting
has no effect and the behavior is as though it had been set to False. When True, IP addresses published
by HTCondor daemons are automatically rewritten to match the IP address of the network interface used to
make the publication. For example, if the condor_schedd advertises itself to two pools via flocking, and the
condor_collector for one pool is reached by the condor_schedd through a private network interface, while the
condor_collector for the other pool is reached through a different network interface, the IP address published
by the condor_schedd daemon will match the address of the respective network interfaces used in the two cases.
The intention is to make it easier for HTCondor daemons to operate in a multi-homed environment.
HIGHPORT Specifies an upper limit of given port numbers for HTCondor to use, such that HTCondor is restricted to a
range of port numbers. If this macro is not explicitly specified, then HTCondor will not restrict the port numbers
that it uses. HTCondor will use system-assigned port numbers. For this macro to work, both HIGHPORT and
LOWPORT (given below) must be defined.
LOWPORT Specifies a lower limit of given port numbers for HTCondor to use, such that HTCondor is restricted to a
range of port numbers. If this macro is not explicitly specified, then HTCondor will not restrict the port numbers
that it uses. HTCondor will use system-assigned port numbers. For this macro to work, both HIGHPORT (given
above) and LOWPORT must be defined.
IN_LOWPORT An integer value that specifies a lower limit of given port numbers for HTCondor to use on incoming
connections (ports for listening), such that HTCondor is restricted to a range of port numbers. This range implies
the use of both IN_LOWPORT and IN_HIGHPORT. A range of port numbers less than 1024 may be used for
daemons running as root. Do not specify IN_LOWPORT in combination with IN_HIGHPORT such that the
range crosses the port 1024 boundary. Applies only to Unix machine configuration. Use of IN_LOWPORT and
IN_HIGHPORT overrides any definition of LOWPORT and HIGHPORT.
IN_HIGHPORT An integer value that specifies an upper limit of given port numbers for HTCondor to use on incoming
connections (ports for listening), such that HTCondor is restricted to a range of port numbers. This range implies
the use of both IN_LOWPORT and IN_HIGHPORT. A range of port numbers less than 1024 may be used for
HTCondor Version 8.6.4 Manual
249
3.5.5. Network-Related Configuration File Entries
daemons running as root. Do not specify IN_LOWPORT in combination with IN_HIGHPORT such that the
range crosses the port 1024 boundary. Applies only to Unix machine configuration. Use of IN_LOWPORT and
IN_HIGHPORT overrides any definition of LOWPORT and HIGHPORT.
OUT_LOWPORT An integer value that specifies a lower limit of given port numbers for HTCondor to use on outgoing
connections, such that HTCondor is restricted to a range of port numbers. This range implies the use of both
OUT_LOWPORT and OUT_HIGHPORT. A range of port numbers less than 1024 is inappropriate, as not all
daemons and tools will be run as root. Applies only to Unix machine configuration. Use of OUT_LOWPORT
and OUT_HIGHPORT overrides any definition of LOWPORT and HIGHPORT.
OUT_HIGHPORT An integer value that specifies an upper limit of given port numbers for HTCondor to use on outgoing connections, such that HTCondor is restricted to a range of port numbers. This range implies the use of
both OUT_LOWPORT and OUT_HIGHPORT. A range of port numbers less than 1024 is inappropriate, as not all
daemons and tools will be run as root. Applies only to Unix machine configuration. Use of OUT_LOWPORT
and OUT_HIGHPORT overrides any definition of LOWPORT and HIGHPORT.
UPDATE_COLLECTOR_WITH_TCP This boolean value controls whether TCP or UDP is used by daemons to send
ClassAd updates to the condor_collector. Please read section 3.9.5 for more details and a discussion of when this
functionality is needed. When using TCP in large pools, it is also necessary to ensure that the condor_collector
has a large enough file descriptor limit using COLLECTOR_MAX_FILE_DESCRIPTORS. The default value is
True.
UPDATE_VIEW_COLLECTOR_WITH_TCP This boolean value controls whether TCP or UDP is used by
the condor_collector to forward ClassAd updates to the condor_collector daemons specified by
CONDOR_VIEW_HOST. Please read section 3.9.5 for more details and a discussion of when this functionality is needed. The default value is False.
TCP_UPDATE_COLLECTORS The list of condor_collector daemons which will be updated with TCP instead of
UDP when UPDATE_COLLECTOR_WITH_TCP or UPDATE_VIEW_COLLECTOR_WITH_TCP is False.
Please read section 3.9.5 for more details and a discussion of when a site needs this functionality.
<SUBSYS>_TIMEOUT_MULTIPLIER An integer value that defaults to 1. This value multiplies configured timeout
values for all targeted subsystem communications, thereby increasing the time until a timeout occurs. This
configuration variable is intended for use by developers for debugging purposes, where communication timeouts
interfere.
NONBLOCKING_COLLECTOR_UPDATE A boolean value that defaults to True. When True, the establishment of
TCP connections to the condor_collector daemon for a security-enabled pool are done in a nonblocking manner.
NEGOTIATOR_USE_NONBLOCKING_STARTD_CONTACT A boolean value that defaults to True. When True,
the establishment of TCP connections from the condor_negotiator daemon to the condor_startd daemon for a
security-enabled pool are done in a nonblocking manner.
UDP_NETWORK_FRAGMENT_SIZE An integer value that defaults to 1000 and represents the maximum size in bytes of an outgoing UDP packet.
If the outgoing message is larger than
$(UDP_NETWORK_FRAGMENT_SIZE), then the message will be split (fragmented) into multiple packets
no larger than $(UDP_NETWORK_FRAGMENT_SIZE). If the destination of the message is the loopback network interface, see UDP_LOOPBACK_FRAGMENT_SIZE below. For instance, the maximum payload size of a
UDP packet over Ethernet is typically 1472 bytes, and thus if a UDP payload exceeds 1472 bytes the IP network
HTCondor Version 8.6.4 Manual
250
3.5.6. Shared File System Configuration File Macros
stack on either hosts or forwarding devices (such as network routers) will have to perform message fragmentation on transmission and reassembly on receipt. Experimentation has shown that such devices are more likely to
simply drop a UDP message under high-traffic scenarios if the message requires reassembly. HTCondor avoids
this situation via the capability to perform UDP fragmentation and reassembly on its own.
UDP_LOOPBACK_FRAGMENT_SIZE An integer value that defaults to 60000 and represents the maximum size in
bytes of an outgoing UDP packet that is being sent to the loopback network interface (e.g. 127.0.0.1). If
the outgoing message is larger than $(UDP_LOOPBACK_FRAGMENT_SIZE), then the message will be split
(fragmented) into multiple packets no larger than $(UDP_LOOPBACK_FRAGMENT_SIZE). If the destination
of the message is not the loopback interface, see UDP_NETWORK_FRAGMENT_SIZE above.
ALWAYS_REUSEADDR A boolean value that, when True, tells HTCondor to set SO_REUSEADDR socket option, so
that the schedd can run large numbers of very short jobs without exhausting the number of local ports needed
for shadows. The default value is True. (Note that this represents a change in behavior compared to versions
of HTCondor older than 8.6.0, which did not include this configuration macro. To restore the previous behavior,
set this value to False.)
3.5.6 Shared File System Configuration File Macros
These macros control how HTCondor interacts with various shared and network file systems. If you are using AFS as
your shared file system, be sure to read section 3.14.1 on Using HTCondor with AFS. For information on submitting
jobs under shared file systems, see section 2.5.8.
UID_DOMAIN The UID_DOMAIN macro is used to decide under which user to run jobs. If the $(UID_DOMAIN)
on the submitting machine is different than the $(UID_DOMAIN) on the machine that runs a job, then HTCondor runs the job as the user nobody. For example, if the submit machine has a $(UID_DOMAIN) of
flippy.cs.wisc.edu, and the machine where the job will execute has a $(UID_DOMAIN) of cs.wisc.edu, the job
will run as user nobody, because the two $(UID_DOMAIN)s are not the same. If the $(UID_DOMAIN) is
the same on both the submit and execute machines, then HTCondor will run the job as the user that submitted
the job.
A further check attempts to assure that the submitting machine can not lie about its UID_DOMAIN. HTCondor
compares the submit machine’s claimed value for UID_DOMAIN to its fully qualified name. If the two do not
end the same, then the submit machine is presumed to be lying about its UID_DOMAIN. In this case, HTCondor
will run the job as user nobody. For example, a job submission to the HTCondor pool at the UW Madison
from flippy.example.com, claiming a UID_DOMAIN of of cs.wisc.edu, will run the job as the user nobody.
Because of this verification, $(UID_DOMAIN) must be a real domain name. At the Computer Sciences department at the UW Madison, we set the $(UID_DOMAIN) to be cs.wisc.edu to indicate that whenever someone
submits from a department machine, we will run the job as the user who submits it.
Also see SOFT_UID_DOMAIN below for information about one more check that HTCondor performs before
running a job as a given user.
A few details:
An administrator could set UID_DOMAIN to *. This will match all domains, but it is a gaping security hole. It
is not recommended.
HTCondor Version 8.6.4 Manual
251
3.5.6. Shared File System Configuration File Macros
An administrator can also leave UID_DOMAIN undefined. This will force HTCondor to always run jobs as user
nobody. Running standard universe jobs as user nobody enhances security and should cause no problems,
because the jobs use remote I/O to access all of their files. However, if vanilla jobs are run as user nobody, then
files that need to be accessed by the job will need to be marked as world readable/writable so the user nobody
can access them.
When HTCondor sends e-mail about a job, HTCondor sends the e-mail to user@$(UID_DOMAIN). If
UID_DOMAIN is undefined, the e-mail is sent to user@submitmachinename.
TRUST_UID_DOMAIN As an added security precaution when HTCondor is about to spawn a job, it ensures that the
UID_DOMAIN of a given submit machine is a substring of that machine’s fully-qualified host name. However,
at some sites, there may be multiple UID spaces that do not clearly correspond to Internet domain names.
In these cases, administrators may wish to use names to describe the UID domains which are not substrings
of the host names of the machines. For this to work, HTCondor must not do this regular security check. If
the TRUST_UID_DOMAIN setting is defined to True, HTCondor will not perform this test, and will trust
whatever UID_DOMAIN is presented by the submit machine when trying to spawn a job, instead of making sure
the submit machine’s host name matches the UID_DOMAIN. When not defined, the default is False, since it
is more secure to perform this test.
SOFT_UID_DOMAIN A boolean variable that defaults to False when not defined. When HTCondor is about to
run a job as a particular user (instead of as user nobody), it verifies that the UID given for the user is in the
password file and actually matches the given user name. However, under installations that do not have every
user in every machine’s password file, this check will fail and the execution attempt will be aborted. To cause
HTCondor not to do this check, set this configuration variable to True. HTCondor will then run the job under
the user’s UID.
SLOT<N>_USER The name of a user for HTCondor to use instead of user nobody, as part of a solution that plugs a
security hole whereby a lurker process can prey on a subsequent job run as user name nobody. <N> is an integer
associated with slots. On Windows, SLOT<N>_USER will only work if the credential of the specified user is
stored on the execute machine using condor_store_cred. See Section 3.8.13 for more information.
STARTER_ALLOW_RUNAS_OWNER A boolean expression evaluated with the job ad as the target, that determines
whether the job may run under the job owner’s account (True) or whether it will run as SLOT<N>_USER
or nobody (False). On Unix, this defaults to True. On Windows, it defaults to False. The job ClassAd
may also contain the attribute RunAsOwner which is logically ANDed with the condor_starter daemon’s
boolean value. Under Unix, if the job does not specify it, this attribute defaults to True. Under Windows, the
attribute defaults to False. In Unix, if the UidDomain of the machine and job do not match, then there is no
possibility to run the job as the owner anyway, so, in that case, this setting has no effect. See Section 3.8.13 for
more information.
DEDICATED_EXECUTE_ACCOUNT_REGEXP This is a regular expression (i.e. a string matching pattern) that
matches the account name(s) that are dedicated to running condor jobs on the execute machine and which
will never be used for more than one job at a time. The default matches no account name. If you have configured SLOT<N>_USER to be a different account for each HTCondor slot, and no non-condor processes will ever
be run by these accounts, then this pattern should match the names of all SLOT<N>_USER accounts. Jobs run
under a dedicated execute account are reliably tracked by HTCondor, whereas other jobs, may spawn processes
that HTCondor fails to detect. Therefore, a dedicated execution account provides more reliable tracking of CPU
usage by the job and it also guarantees that when the job exits, no “lurker” processes are left behind. When the
job exits, condor will attempt to kill all processes owned by the dedicated execution account. Example:
HTCondor Version 8.6.4 Manual
252
3.5.6. Shared File System Configuration File Macros
253
SLOT1_USER = cndrusr1
SLOT2_USER = cndrusr2
STARTER_ALLOW_RUNAS_OWNER = False
DEDICATED_EXECUTE_ACCOUNT_REGEXP = cndrusr[0-9]+
You can tell if the starter is in fact treating the account as a dedicated account, because it will print a line such
as the following in its log file:
Tracking process family by login "cndrusr1"
EXECUTE_LOGIN_IS_DEDICATED This
configuration setting
is
deprecated because
it
not handle the case where some jobs run as dedicated accounts and some do not.
DEDICATED_EXECUTE_ACCOUNT_REGEXP instead.
canUse
A boolean value that defaults to False. When True, HTCondor knows that all jobs are being run by dedicated
execution accounts (whether they are running as the job owner or as nobody or as SLOT<N>_USER). Therefore,
when the job exits, all processes running under the same account will be killed.
FILESYSTEM_DOMAIN An arbitrary string that is used to decide if the two machines, a submit machine and an
execute machine, share a file system. Although this configuration variable name contains the word “DOMAIN”,
its value is not required to be a domain name. It often is a domain name.
Note that this implementation is not ideal: machines may share some file systems but not others. HTCondor
currently has no way to express this automatically. A job can express the need to use a particular file system
where machines advertise an additional ClassAd attribute and the job requires machines with the attribute, as
described on the question within the https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToAdminRecipes
page for how to run jobs on a subset of machines that have required software installed.
Note that if you do not set $(FILESYSTEM_DOMAIN), the value defaults to the fully qualified host name
of the local machine. Since each machine will have a different $(FILESYSTEM_DOMAIN), they will not be
considered to have shared file systems.
RESERVE_AFS_CACHE If your machine is running AFS and the AFS cache lives on the same partition as the other
HTCondor directories, and you want HTCondor to reserve the space that your AFS cache is configured to use,
set this macro to True. It defaults to False.
USE_NFS This macro influences how HTCondor jobs running in the standard universe access their files. By default,
HTCondor will redirect the file I/O requests of standard universe jobs from the executing machine to the submitting machine. So, as an HTCondor job migrates around the network, the file system always appears to be
identical to the file system where the job was submitted. However, consider the case where a user’s data files
are sitting on an NFS server. The machine running the user’s program will send all I/O over the network to the
submitting machine, which in turn sends all the I/O back over the network to the NFS file server. Thus, all of
the program’s I/O is being sent over the network twice.
If this configuration variable is True, then HTCondor will attempt to read/write files directly on the executing machine without redirecting I/O back to the submitting machine, if both the submitting machine
and the machine running the job are both accessing the same NFS servers (if they are both in the same
$(FILESYSTEM_DOMAIN) and in the same $(UID_DOMAIN), as described above). The result is I/O performed by HTCondor standard universe jobs is only sent over the network once. While sending all file operations
HTCondor Version 8.6.4 Manual
3.5.6. Shared File System Configuration File Macros
over the network twice might sound really bad, unless you are operating over networks where bandwidth as at a
very high premium, practical experience reveals that this scheme offers very little real performance gain. There
are also some (fairly rare) situations where this scheme can break down.
Setting $(USE_NFS) to False is always safe. It may result in slightly more network traffic, but HTCondor
jobs are most often heavy on CPU and light on I/O. It also ensures that a remote standard universe HTCondor
job will always use HTCondor’s remote system calls mechanism to reroute I/O and therefore see the exact same
file system that the user sees on the machine where she/he submitted the job.
Some gritty details for folks who want to know: If the you set $(USE_NFS) to True, and the
$(FILESYSTEM_DOMAIN) of both the submitting machine and the remote machine about to execute the
job match, and the $(FILESYSTEM_DOMAIN) claimed by the submit machine is indeed found to be a subset
of what an inverse look up to a DNS (domain name server) reports as the fully qualified domain name for the
submit machine’s IP address (this security measure safeguards against the submit machine from lying), then the
job will access files using a local system call, without redirecting them to the submitting machine (with NFS).
Otherwise, the system call will get routed back to the submitting machine using HTCondor’s remote system call
mechanism. NOTE: When submitting a vanilla job, condor_submit will, by default, append requirements to the
Job ClassAd that specify the machine to run the job must be in the same $(FILESYSTEM_DOMAIN) and the
same $(UID_DOMAIN).
This configuration variable similarly changes the semantics of Chirp file I/O when running in the vanilla, java
or parallel universe. If this variable is set in those universes, Chirp will not send I/O requests over the network
as requested, but perform them directly to the locally mounted file system. Other than Chirp file access, this
variable is unused outside of the standard universe.
IGNORE_NFS_LOCK_ERRORS When set to True, all errors related to file locking errors from NFS are ignored.
Defaults to False, not ignoring errors.
USE_AFS If your machines have AFS, this macro determines whether HTCondor will use remote system calls for
standard universe jobs to send I/O requests to the submit machine, or if it should use local file access on the execute machine (which will then use AFS to get to the submitter’s files). Read the setting above on $(USE_NFS)
for a discussion of why you might want to use AFS access instead of remote system calls.
One important difference between $(USE_NFS) and $(USE_AFS) is the AFS cache. With $(USE_AFS)
set to True, the remote HTCondor job executing on some machine will start modifying the AFS cache, possibly
evicting the machine owner’s files from the cache to make room for its own. Generally speaking, since we try
to minimize the impact of having an HTCondor job run on a given machine, we do not recommend using this
setting.
While sending all file operations over the network twice might sound really bad, unless you are operating over
networks where bandwidth as at a very high premium, practical experience reveals that this scheme offers very
little real performance gain. There are also some (fairly rare) situations where this scheme can break down.
Setting $(USE_AFS) to False is always safe. It may result in slightly more network traffic, but HTCondor
jobs are usually heavy on CPU and light on I/O. False ensures that a remote standard universe HTCondor job
will always see the exact same file system that the user on sees on the machine where he/she submitted the job.
Plus, it will ensure that the machine where the job executes does not have its AFS cache modified as a result of
the HTCondor job being there.
However, things may be different at your site, which is why the setting is there.
HTCondor Version 8.6.4 Manual
254
3.5.7. Checkpoint Server Configuration File Macros
3.5.7 Checkpoint Server Configuration File Macros
These macros control whether or not HTCondor uses a checkpoint server. This section describes the settings that the
checkpoint server itself needs defined. See section 3.10 on Installing a Checkpoint Server for details on installing and
running a checkpoint server.
CKPT_SERVER_HOST The host name of a checkpoint server.
STARTER_CHOOSES_CKPT_SERVER If this parameter is True or undefined on the submit machine, the checkpoint server specified by $(CKPT_SERVER_HOST) on the execute machine is used. If it is False on the
submit machine, the checkpoint server specified by $(CKPT_SERVER_HOST) on the submit machine is used.
CKPT_SERVER_DIR The full path of the directory the checkpoint server should use to store checkpoint files. Depending on the size of the pool and the size of the jobs submitted, this directory and its subdirectories might
need to store many MiB of data.
USE_CKPT_SERVER A boolean which determines if a given submit machine is to use a checkpoint server if one
is available. If a checkpoint server is not available or the variable USE_CKPT_SERVER is set to False,
checkpoints will be written to the local $(SPOOL) directory on the submission machine.
MAX_DISCARDED_RUN_TIME If the condor_shadow daemon is unable to read a checkpoint file from the checkpoint server, it keeps trying only if the job has accumulated more than this many seconds of CPU usage. Otherwise, the job is started from scratch. Defaults to 3600 (1 hour). This variable is only used if
$(USE_CKPT_SERVER) is True.
CKPT_SERVER_CHECK_PARENT_INTERVAL This is the number of seconds between checks to see whether the
parent of the checkpoint server (usually the condor_master) has died. If the parent has died, the checkpoint
server shuts itself down. The default is 120 seconds. A setting of 0 disables this check.
CKPT_SERVER_INTERVAL The maximum number of seconds the checkpoint server waits for activity on network
sockets before performing other tasks. The default value is 300 seconds.
CKPT_SERVER_CLASSAD_FILE A string that represents a file in the file system to which ClassAds will be written.
The ClassAds denote information about stored checkpoint files, such as owner, shadow IP address, name of the
file, and size of the file. This information is also independently recorded in the TransferLog. The default
setting is undefined, which means a checkpoint server ClassAd file will not be kept.
CKPT_SERVER_CLEAN_INTERVAL The number of seconds that must pass until the ClassAd log file as described
by the CKPT_SERVER_CLASSAD_FILE variable gets truncated. The default is 86400 seconds, which is one
day.
CKPT_SERVER_REMOVE_STALE_CKPT_INTERVAL The number of seconds between attempts to discover and
remove stale checkpoint files. It defaults to 86400 seconds, which is one day.
CKPT_SERVER_SOCKET_BUFSIZE The number of bytes representing the size of the TCP send/recv buffer on the
socket file descriptor related to moving the checkpoint file to and from the checkpoint server. The default value
is 0, which allows the operating system to decide the size.
CKPT_SERVER_MAX_PROCESSES The maximum number of child processes that could be working on behalf of
the checkpoint server. This includes store processes and restore processes. The default value is 50.
HTCondor Version 8.6.4 Manual
255
3.5.8. condor_master Configuration File Macros
CKPT_SERVER_MAX_STORE_PROCESSES The maximum number of child process strictly devoted to the storage
of checkpoints. The default is the value of CKPT_SERVER_MAX_PROCESSES.
CKPT_SERVER_MAX_RESTORE_PROCESSES The maximum number of child process strictly devoted to the
restoring of checkpoints. The default is the value of CKPT_SERVER_MAX_PROCESSES.
CKPT_SERVER_STALE_CKPT_AGE_CUTOFF The number of seconds after which if a checkpoint file has not been
accessed, it is considered stale. The default value is 5184000 seconds, which is sixty days.
ALWAYS_USE_LOCAL_CKPT_SERVER A boolean value that defaults to False. When True, it forces all checkpoints to be read from a checkpoint server running on the same machine where the job is running. This is
intended to be used when all checkpoint servers access a shared file system.
3.5.8 condor_master Configuration File Macros
These macros control the condor_master.
DAEMON_LIST This macro determines what daemons the condor_master will start and keep its watchful eyes on.
The list is a comma or space separated list of subsystem names (listed in section 3.5.1). For example,
DAEMON_LIST = MASTER, STARTD, SCHEDD
NOTE: This configuration variable cannot be changed by using condor_reconfig or by sending a SIGHUP. To
change this configuration variable, restart the condor_master daemon by using condor_restart. Only then will
the change take effect.
NOTE: On your central manager, your $(DAEMON_LIST) will be different from your regular pool, since it
will include entries for the condor_collector and condor_negotiator.
DC_DAEMON_LIST A list delimited by commas and/or spaces that lists the daemons in DAEMON_LIST which use
the HTCondor DaemonCore library. The condor_master must differentiate between daemons that use DaemonCore and those that do not, so it uses the appropriate inter-process communication mechanisms. This list
currently includes all HTCondor daemons except the checkpoint server by default.
As of HTCondor version 7.2.1, a daemon may be appended to the default DC_DAEMON_LIST value by placing
the plus character (+) before the first entry in the DC_DAEMON_LIST definition. For example:
DC_DAEMON_LIST = +NEW_DAEMON
<SUBSYS> Once you have defined which subsystems you want the condor_master to start, you must provide it with
the full path to each of these binaries. For example:
MASTER
STARTD
SCHEDD
= $(SBIN)/condor_master
= $(SBIN)/condor_startd
= $(SBIN)/condor_schedd
HTCondor Version 8.6.4 Manual
256
3.5.8. condor_master Configuration File Macros
These are most often defined relative to the $(SBIN) macro.
The macro is named by substituting <SUBSYS> with the appropriate subsystem string as defined in section 3.5.1.
<DaemonName>_ENVIRONMENT <DaemonName> is the name of a daemon listed in DAEMON_LIST. Defines
changes to the environment that the daemon is invoked with. It should use the same syntax for specifying the
environment as the environment specification in a submit description file. For example, to redefine the TMP and
CONDOR_CONFIG environment variables seen by the condor_schedd, place the following in the configuration:
SCHEDD_ENVIRONMENT = "TMP=/new/value CONDOR_CONFIG=/special/config"
When the condor_schedd daemon is started by the condor_master, it would see the specified values of TMP and
CONDOR_CONFIG.
<SUBSYS>_ARGS This macro allows the specification of additional command line arguments for any process
spawned by the condor_master. List the desired arguments using the same syntax as the arguments specification in a condor_submit submit file (see page 914), with one exception: do not escape double-quotes when
using the old-style syntax (this is for backward compatibility). Set the arguments for a specific daemon with
this macro, and the macro will affect only that daemon. Define one of these for each daemon the condor_master
is controlling. For example, set $(STARTD_ARGS) to specify any extra command line arguments to the condor_startd.
The macro is named by substituting <SUBSYS> with the appropriate subsystem string as defined in section 3.5.1.
<SUBSYS>_USERID The account name that should be used to run the SUBSYS process spawned by the condor_master. When not defined, the process is spawned as the same user that is running condor_master. When
defined, the real user id of the spawned process will be set to the specified account, so if this account is not
root, the process will not have root privileges. The condor_master must be running as root in order to start
processes as other users. Example configuration:
COLLECTOR_USERID = condor
NEGOTIATOR_USERID = condor
The above example runs the condor_collector and condor_negotiator as the condor user with no root privileges. If we specified some account other than the condor user, as set by the (CONDOR_IDS) configuration
variable, then we would need to configure the log files for these daemons to be in a directory that they can write
to. When using GSI security or any other security method in which the daemon credential is owned by root, it
is also necessary to make a copy of the credential, make it be owned by the account the daemons are using, and
configure the daemons to use that copy.
PREEN In addition to the daemons defined in $(DAEMON_LIST), the condor_master also starts up a special process,
condor_preen to clean out junk files that have been left laying around by HTCondor. This macro determines
where the condor_master finds the condor_preen binary. If this macro is set to nothing, condor_preen will not
run.
PREEN_ARGS Controls how condor_preen behaves by allowing the specification of command-line arguments. This
macro works as $(<SUBSYS>_ARGS) does. The difference is that you must specify this macro for condor_preen if you want it to do anything. condor_preen takes action only because of command line arguments.
-m means you want e-mail about files condor_preen finds that it thinks it should remove. -r means you want
condor_preen to actually remove these files.
HTCondor Version 8.6.4 Manual
257
3.5.8. condor_master Configuration File Macros
PREEN_INTERVAL This macro determines how often condor_preen should be started. It is defined in terms of
seconds and defaults to 86400 (once a day).
PUBLISH_OBITUARIES When a daemon crashes, the condor_master can send e-mail to the address specified by
$(CONDOR_ADMIN) with an obituary letting the administrator know that the daemon died, the cause of death
(which signal or exit status it exited with), and (optionally) the last few entries from that daemon’s log file. If
you want obituaries, set this macro to True.
OBITUARY_LOG_LENGTH This macro controls how many lines of the log file are part of obituaries. This macro has
a default value of 20 lines.
START_MASTER If this setting is defined and set to False the condor_master will immediately exit upon startup.
This appears strange, but perhaps you do not want HTCondor to run on certain machines in your pool, yet the
boot scripts for your entire pool are handled by a centralized set of files – setting START_MASTER to False
for those machines would allow this. Note that START_MASTER is an entry you would most likely find in a
local configuration file, not a global configuration file. If not defined, START_MASTER defaults to True.
START_DAEMONS This macro is similar to the $(START_MASTER) macro described above. However, the condor_master does not exit; it does not start any of the daemons listed in the $(DAEMON_LIST). The daemons
may be started at a later time with a condor_on command.
MASTER_UPDATE_INTERVAL This macro determines how often the condor_master sends a ClassAd update to the
condor_collector. It is defined in seconds and defaults to 300 (every 5 minutes).
MASTER_CHECK_NEW_EXEC_INTERVAL This macro controls how often the condor_master checks the timestamps of the running daemons. If any daemons have been modified, the master restarts them. It is defined in
seconds and defaults to 300 (every 5 minutes).
MASTER_NEW_BINARY_RESTART Defines a mode of operation for the restart of the condor_master, when it notices that the condor_master binary has changed. Valid values are GRACEFUL, PEACEFUL, and NEVER, with
a default value of GRACEFUL. On a GRACEFUL restart of the master, child processes are told to exit, but if they
do not before a timer expires, then they are killed. On a PEACEFUL restart, child processes are told to exit, after
which the condor_master waits until they do so.
MASTER_NEW_BINARY_DELAY Once the condor_master has discovered a new binary, this macro controls how
long it waits before attempting to execute the new binary. This delay exists because the condor_master might
notice a new binary while it is in the process of being copied, in which case trying to execute it yields unpredictable results. The entry is defined in seconds and defaults to 120 (2 minutes).
SHUTDOWN_FAST_TIMEOUT This macro determines the maximum amount of time daemons are given to perform
their fast shutdown procedure before the condor_master kills them outright. It is defined in seconds and defaults
to 300 (5 minutes).
DEFAULT_MASTER_SHUTDOWN_SCRIPT A full path and file name of a program that the condor_master is to execute via the Unix execl() call, or the similar Win32 _execl() call, instead of the normal call to exit().
This allows the admin to specify a program to execute as root when the condor_master exits. Note that a successful call to the condor_set_shutdown program will override this setting; see the documentation for config
knob MASTER_SHUTDOWN_<Name> below.
HTCondor Version 8.6.4 Manual
258
3.5.8. condor_master Configuration File Macros
MASTER_SHUTDOWN_<Name> A full path and file name of a program that the condor_master is to execute via
the Unix execl() call, or the similar Win32 _execl() call, instead of the normal call to exit(). Multiple programs to execute may be defined with multiple entries, each with a unique Name. These macros have
no effect on a condor_master unless condor_set_shutdown is run. The Name specified as an argument to the
condor_set_shutdown program must match the Name portion of one of these MASTER_SHUTDOWN_<Name>
macros; if not, the condor_master will log an error and ignore the command. If a match is found, the condor_master will attempt to verify the program, and it will store the path and program name. When the condor_master shuts down (that is, just before it exits), the program is then executed as described above. The
manual page for condor_set_shutdown on page 890 contains details on the use of this program.
NOTE: This program will be run with root privileges under Unix or administrator privileges under Windows.
The administrator must ensure that this cannot be used in such a way as to violate system integrity.
MASTER_BACKOFF_CONSTANT and MASTER_<name>_BACKOFF_CONSTANT When a daemon crashes, condor_master uses an exponential back off delay before restarting it; see the discussion at the end of this section
for a detailed discussion on how these parameters work together. These settings define the constant value of the
expression used to determine how long to wait before starting the daemon again (and, effectively becomes the
initial backoff time). It is an integer in units of seconds, and defaults to 9 seconds.
$(MASTER_<name>_BACKOFF_CONSTANT)
is
the
daemon-specific
form
of
MASTER_BACKOFF_CONSTANT; if this daemon-specific macro is not defined for a specific daemon,
the non-daemon-specific value will used.
MASTER_BACKOFF_FACTOR and MASTER_<name>_BACKOFF_FACTOR When a daemon crashes, condor_master uses an exponential back off delay before restarting it; see the discussion at the end of this section
for a detailed discussion on how these parameters work together. This setting is the base of the exponent used
to determine how long to wait before starting the daemon again. It defaults to 2 seconds.
$(MASTER_<name>_BACKOFF_FACTOR) is the daemon-specific form of MASTER_BACKOFF_FACTOR;
if this daemon-specific macro is not defined for a specific daemon, the non-daemon-specific value will used.
MASTER_BACKOFF_CEILING and MASTER_<name>_BACKOFF_CEILING When a daemon crashes, condor_master uses an exponential back off delay before restarting it; see the discussion at the end of this section for a detailed discussion on how these parameters work together. This entry determines the maximum
amount of time you want the master to wait between attempts to start a given daemon. (With 2.0 as the
$(MASTER_BACKOFF_FACTOR), 1 hour is obtained in 12 restarts). It is defined in terms of seconds and
defaults to 3600 (1 hour).
$(MASTER_<name>_BACKOFF_CEILING)
is
the
daemon-specific
form
of
MASTER_BACKOFF_CEILING; if this daemon-specific macro is not defined for a specific daemon, the
non-daemon-specific value will used.
MASTER_RECOVER_FACTOR and MASTER_<name>_RECOVER_FACTOR A macro to set how long a daemon
needs to run without crashing before it is considered recovered. Once a daemon has recovered, the number of
restarts is reset, so the exponential back off returns to its initial state. The macro is defined in terms of seconds
and defaults to 300 (5 minutes).
$(MASTER_<name>_RECOVER_FACTOR) is the daemon-specific form of MASTER_RECOVER_FACTOR;
if this daemon-specific macro is not defined for a specific daemon, the non-daemon-specific value will used.
HTCondor Version 8.6.4 Manual
259
3.5.8. condor_master Configuration File Macros
When a daemon crashes, condor_master will restart the daemon after a delay (a back off). The length of this delay
is based on how many times it has been restarted, and gets larger after each crashes. The equation for calculating this
backoff time is given by:
t = c + kn
where t is the calculated time, c is the constant defined by $(MASTER_BACKOFF_CONSTANT), k is the “factor”
defined by $(MASTER_BACKOFF_FACTOR), and n is the number of restarts already attempted (0 for the first restart,
1 for the next, etc.).
With default values, after the first crash, the delay would be t = 9 + 2.00 , giving 10 seconds (remember, n = 0).
If the daemon keeps crashing, the delay increases.
For example, take the $(MASTER_BACKOFF_FACTOR) (which defaults to 2.0) to the power the number of times
the daemon has restarted, and add $(MASTER_BACKOFF_CONSTANT) (which defaults to 9). Thus:
1st crash: n = 0, so: t = 9 + 20 = 9 + 1 = 10 seconds
2nd crash: n = 1, so: t = 9 + 21 = 9 + 2 = 11 seconds
3rd crash: n = 2, so: t = 9 + 22 = 9 + 4 = 13 seconds
...
6th crash: n = 5, so: t = 9 + 25 = 9 + 32 = 41 seconds
...
9th crash: n = 8, so: t = 9 + 28 = 9 + 256 = 265 seconds
And, after the 13 crashes, it would be:
13th crash: n = 12, so: t = 9 + 212 = 9 + 4096 = 4105 seconds
This is bigger than the $(MASTER_BACKOFF_CEILING), which defaults to 3600, so the daemon would really
be restarted after only 3600 seconds, not 4105. The condor_master tries again every hour (since the numbers would
get larger and would always be capped by the ceiling). Eventually, imagine that daemon finally started and did not
crash. This might happen if, for example, an administrator reinstalled an accidentally deleted binary after receiving
e-mail about the daemon crashing. If it stayed alive for $(MASTER_RECOVER_FACTOR) seconds (defaults to 5
minutes), the count of how many restarts this daemon has performed is reset to 0.
The moral of the example is that the defaults work quite well, and you probably will not want to change them for
any reason.
MASTER_NAME Defines a unique name given for a condor_master daemon on a machine. For a condor_master
running as root, it defaults to the fully qualified host name. When not running as root, it defaults to the user
that instantiates the condor_master, concatenated with an at symbol (@), concatenated with the fully qualified
host name. If more than one condor_master is running on the same host, then the MASTER_NAME for each
condor_master must be defined to uniquely identify the separate daemons.
A defined MASTER_NAME is presumed to be of the form identifying-string@full.host.name. If
the string does not include an @ sign, HTCondor appends one, followed by the fully qualified host name of
HTCondor Version 8.6.4 Manual
260
3.5.8. condor_master Configuration File Macros
the local machine. The identifying-string portion may contain any alphanumeric ASCII characters or
punctuation marks, except the @ sign. We recommend that the string does not contain the : (colon) character,
since that might cause problems with certain tools. Previous to HTCondor 7.1.1, when the string included an @
sign, HTCondor replaced whatever followed the @ sign with the fully qualified host name of the local machine.
HTCondor does not modify any portion of the string, if it contains an @ sign. This is useful for remote job
submissions under the high availability of the job queue.
If the MASTER_NAME setting is used, and the condor_master is configured to spawn a condor_schedd, the
name defined with MASTER_NAME takes precedence over the SCHEDD_NAME setting (see section 3.5.10 on
page 287). Since HTCondor makes the assumption that there is only one instance of the condor_startd running
on a machine, the MASTER_NAME is not automatically propagated to the condor_startd. However, in situations
where multiple condor_startd daemons are running on the same host, the STARTD_NAME should be set to
uniquely identify the condor_startd daemons.
If an HTCondor daemon (master, schedd or startd) has been given a unique name, all HTCondor tools that need
to contact that daemon can be told what name to use via the -name command-line option.
MASTER_ATTRS This macro is described in section 3.5.4 as <SUBSYS>_ATTRS.
MASTER_DEBUG This macro is described in section 3.5.3 as <SUBSYS>_DEBUG.
MASTER_ADDRESS_FILE This macro is described in section 3.5.4 as <SUBSYS>_ADDRESS_FILE.
ALLOW_ADMIN_COMMANDS If set to NO for a given host, this macro disables administrative commands, such as
condor_restart, condor_on, and condor_off, to that host.
MASTER_INSTANCE_LOCK Defines the name of a file for the condor_master daemon to lock in order to prevent
multiple condor_masters from starting. This is useful when using shared file systems like NFS which do not
technically support locking in the case where the lock files reside on a local disk. If this macro is not defined, the
default file name will be $(LOCK)/InstanceLock. $(LOCK) can instead be defined to specify the location
of all lock files, not just the condor_master’s InstanceLock. If $(LOCK) is undefined, then the master log
itself is locked.
ADD_WINDOWS_FIREWALL_EXCEPTION When set to False, the condor_master will not automatically add HTCondor to the Windows Firewall list of trusted applications. Such trusted applications can accept incoming
connections without interference from the firewall. This only affects machines running Windows XP SP2 or
higher. The default is True.
WINDOWS_FIREWALL_FAILURE_RETRY An integer value (default value is 2) that represents the number of times
the condor_master will retry to add firewall exceptions. When a Windows machine boots up, HTCondor starts
up by default as well. Under certain conditions, the condor_master may have difficulty adding exceptions to the
Windows Firewall because of a delay in other services starting up. Examples of services that may possibly be
slow are the SharedAccess service, the Netman service, or the Workstation service. This configuration variable
allows administrators to set the number of times (once every 5 seconds) that the condor_master will retry to add
firewall exceptions. A value of 0 means that HTCondor will retry indefinitely.
USE_PROCESS_GROUPS A boolean value that defaults to True. When False, HTCondor daemons on Unix
machines will not create new sessions or process groups. HTCondor uses processes groups to help it track the
descendants of processes it creates. This can cause problems when HTCondor is run under another job execution
system.
HTCondor Version 8.6.4 Manual
261
3.5.9. condor_startd Configuration File Macros
DISCARD_SESSION_KEYRING_ON_STARTUP A boolean value that defaults to True. When True, the condor_master daemon will replace the kernel session keyring it was invoked with with a new keyring named
htcondor. Various Linux system services, such as OpenAFS and eCryptFS, use the kernel session keyring
to hold passwords and authentication tokens. By replacing the keyring on start up, the condor_master ensures
these keys cannot be unintentionally obtained by user jobs.
ENABLE_KERNEL_TUNING Relevant only to Linux platforms, a boolean value that defaults to True. When
True, the condor_master daemon invokes the kernel tuning script specified by configuration variable
LINUX_KERNEL_TUNING_SCRIPT once as root when the condor_master daemon starts up.
KERNEL_TUNING_LOG A string value that defaults to $(LOG)/KernelTuningLog. If the kernel tuning script
runs, its output will be logged to this file.
LINUX_KERNEL_TUNING_SCRIPT A string value that defaults to $(LIBEXEC)/linux_kernel_tuning.
This is the script that the condor_master runs to tune the kernel when ENABLE_KERNEL_TUNING is True.
3.5.9 condor_startd Configuration File Macros
NOTE: If you are running HTCondor on a multi-CPU machine, be sure to also read section 3.7.1 on page 396 which
describes how to set up and configure HTCondor on multi-core machines.
These settings control general operation of the condor_startd. Examples using these configuration macros, as well
as further explanation is found in section 3.7 on Configuring The Startd Policy.
START A boolean expression that, when True, indicates that the machine is willing to start running an HTCondor
job. START is considered when the condor_negotiator daemon is considering evicting the job to replace it with
one that will generate a better rank for the condor_startd daemon, or a user with a higher priority.
SUSPEND A boolean expression that, when True, causes HTCondor to suspend running an HTCondor job. The
machine may still be claimed, but the job makes no further progress, and HTCondor does not generate a load on
the machine.
PREEMPT A boolean expression that, when True, causes HTCondor to stop a currently running job once
MAXJOBRETIREMENTTIME has expired. This expression is not evaluated if WANT_SUSPEND is True. The
default value is False, such that preemption is disabled.
WANT_HOLD A boolean expression that defaults to False. When True and the value of PREEMPT becomes True
and WANT_SUSPEND is False and MAXJOBRETIREMENTTIME has expired, the job is put on hold for the
reason (optionally) specified by the variables WANT_HOLD_REASON and WANT_HOLD_SUBCODE. As usual,
the job owner may specify periodic_release and/or periodic_remove expressions to react to specific hold states
automatically. The attribute HoldReasonCode in the job ClassAd is set to the value 21 when WANT_HOLD
is responsible for putting the job on hold.
Here is an example policy that puts jobs on hold that use too much virtual memory:
VIRTUAL_MEMORY_AVAILABLE_MB = (VirtualMemory*0.9)
MEMORY_EXCEEDED = ImageSize/1024 > $(VIRTUAL_MEMORY_AVAILABLE_MB)
PREEMPT = ($(PREEMPT)) || ($(MEMORY_EXCEEDED))
HTCondor Version 8.6.4 Manual
262
3.5.9. condor_startd Configuration File Macros
WANT_SUSPEND = ($(WANT_SUSPEND)) && ($(MEMORY_EXCEEDED)) =!= TRUE
WANT_HOLD = ($(MEMORY_EXCEEDED))
WANT_HOLD_REASON = \
ifThenElse( $(MEMORY_EXCEEDED), \
"Your job used too much virtual memory.", \
undefined )
WANT_HOLD_REASON An expression that defines a string utilized to set the job ClassAd attribute HoldReason
when a job is put on hold due to WANT_HOLD. If not defined or if the expression evaluates to Undefined, a
default hold reason is provided.
WANT_HOLD_SUBCODE An expression that defines an integer value utilized to set the job ClassAd attribute
HoldReasonSubCode when a job is put on hold due to WANT_HOLD. If not defined or if the expression
evaluates to Undefined, the value is set to 0. Note that HoldReasonCode is always set to 21.
CONTINUE A boolean expression that, when True, causes HTCondor to continue the execution of a suspended job.
KILL A boolean expression that, when True, causes HTCondor to immediately stop the execution of a vacating job,
without delay. The job is hard-killed, so any attempt by the job to checkpoint or clean up will be aborted. This
expression should normally be False. When desired, it may be used to abort the graceful shutdown of a job
earlier than the limit imposed by MachineMaxVacateTime.
PERIODIC_CHECKPOINT A boolean expression that, when True, causes HTCondor to initiate a checkpoint of the
currently running job. This setting applies to all standard universe jobs and to vm universe jobs that have set
vm_checkpoint to True in the submit description file.
RANK A floating point value that HTCondor uses to compare potential jobs. A larger value for a specific job ranks
that job above others with lower values for RANK.
ADVERTISE_PSLOT_ROLLUP_INFORMATION A boolean value that defaults to True, causing the condor_startd
to advertise ClassAd attributes that may be used in partitionable slot preemption. The attributes are
• ChildAccountingGroup
• ChildActivity
• ChildCPUs
• ChildCurrentRank
• ChildEnteredCurrentState
• ChildMemory
• ChildName
• ChildRemoteOwner
• ChildRemoteUser
• ChildRetirementTimeRemaining
• ChildState
• PslotRollupInformation
HTCondor Version 8.6.4 Manual
263
3.5.9. condor_startd Configuration File Macros
STARTD_PARTITIONABLE_SLOT_ATTRS A list of additional from the above default attributes from dynamic
slots that will be rolled up into a list attribute in their parent partitionable slot, prefixed with the name Child.
IS_VALID_CHECKPOINT_PLATFORM A boolean expression that is logically ANDed with the with the START
expression to limit which machines a standard universe job may continue execution on once they have produced
a checkpoint. The default expression is
IS_VALID_CHECKPOINT_PLATFORM =
(
( (TARGET.JobUniverse == 1) == FALSE) ||
(
(MY.CheckpointPlatform =!= UNDEFINED) &&
(
(TARGET.LastCheckpointPlatform =?= MY.CheckpointPlatform) ||
(TARGET.NumCkpts == 0)
)
)
)
CHECKPOINT_PLATFORM A string used to override the automatically-generated machine ClassAd attribute
CheckpointPlatform (see section 12), which is used to identify the platform upon which a job previously
generated a checkpoint under the standard universe. This restricts the machine matches that may be considered
for a job and where the job may resume. Overriding the value may be necessary for architectures that are the
same in name, but actually have differences in instruction sets, such as the AVX extensions to the Intel processor.
WANT_SUSPEND A boolean expression that, when True, tells HTCondor to evaluate the SUSPEND expression to
decide whether to suspend a running job. When True, the PREEMPT expression is not evaluated. When not
explicitly set, the condor_startd exits with an error. When explicitly set, but the evaluated value is anything
other than True, the value is utilized as if it were False.
WANT_VACATE A boolean expression that, when True, defines that a preempted HTCondor job is to be vacated,
instead of killed. This means the job will be soft-killed and given time to checkpoint or clean up. The amount
of time given depends on MachineMaxVacateTime and KILL. The default value is True.
ENABLE_VERSIONED_OPSYS A boolean expression that determines whether pre-7.7.2 strings used for the machine ClassAd attribute OpSys are used or not. Defaults to False on Windows platforms, meaning
that the newer behavior of setting OpSys = "WINDOWS" and OpSysVer = 601 (for example), while
OpSysAndVer = "WINNT61". On platforms other than Windows, the default value is True, meaning
that the values for OpSys and OpSysAndVer are the same, implementing the pre-7.7.2 behavior.
IS_OWNER A boolean expression that defaults to being defined as
IS_OWNER = (START =?= FALSE)
Used to describe the state of the machine with respect to its use by its owner. Job ClassAd attributes are not
used in defining IS_OWNER, as they would be Undefined.
HTCondor Version 8.6.4 Manual
264
3.5.9. condor_startd Configuration File Macros
STARTD_HISTORY A file name where the condor_startd daemon will maintain a job history file in an analogous
way to that of the history file defined by the configuration variable HISTORY. It will be rotated in the same way,
and the same parameters that apply to the HISTORY file rotation apply to the condor_startd daemon history as
well. This can be read with the condor_history command by passing the name of the file to the -file option of
condor_history.
condor_history -file `condor_config_val LOG`/startd_history
STARTER This macro holds the full path to the condor_starter binary that the condor_startd should spawn. It is
normally defined relative to $(SBIN).
KILLING_TIMEOUT The amount of time in seconds that the condor_startd should wait after sending a fast shutdown
request to condor_starter before forcibly killing the job and condor_starter. The default value is 30 seconds.
POLLING_INTERVAL When a condor_startd enters the claimed state, this macro determines how often the state
of the machine is polled to check the need to suspend, resume, vacate or kill the job. It is defined in terms of
seconds and defaults to 5.
UPDATE_INTERVAL Determines how often the condor_startd should send a ClassAd update to the condor_collector. The condor_startd also sends update on any state or activity change, or if the value of its START
expression changes. See section 3.7.1 on condor_startd states, section 3.7.1 on condor_startd Activities, and
section 3.7.1 on condor_startd START expression for details on states, activities, and the START expression.
This macro is defined in terms of seconds and defaults to 300 (5 minutes).
UPDATE_OFFSET An integer value representing the number of seconds of delay that the condor_startd should
wait before sending its initial update, and the first update after a condor_reconfig command is sent
to the condor_collector. The time of all other updates sent after this initial update is determined by
$(UPDATE_INTERVAL). Thus, the first update will be sent after $(UPDATE_OFFSET) seconds, and the
second update will be sent after $(UPDATE_OFFSET) + $(UPDATE_INTERVAL). This is useful when used
in conjunction with the $RANDOM_INTEGER() macro for large pools, to spread out the updates sent by a large
number of condor_startd daemons. Defaults to zero. The example configuration
startd.UPDATE_INTERVAL = 300
startd.UPDATE_OFFSET
= $RANDOM_INTEGER(0,300)
causes the initial update to occur at a random number of seconds falling between 0 and 300, with all further
updates occurring at fixed 300 second intervals following the initial update.
MachineMaxVacateTime An integer expression representing the number of seconds the machine is willing to
wait for a job that has been soft-killed to gracefully shut down. The default value is 600 seconds (10 minutes). This expression is evaluated when the job starts running. The job may adjust the wait time by setting
JobMaxVacateTime. If the job’s setting is less than the machine’s, the job’s specification is used. If the job’s
setting is larger than the machine’s, the result depends on whether the job has any excess retirement time. If the
job has more retirement time left than the machine’s maximum vacate time setting, then retirement time will be
converted into vacating time, up to the amount of JobMaxVacateTime. The KILL expression may be used to
abort the graceful shutdown of the job at any time. At the time when the job is preempted, the WANT_VACATE
expression may be used to skip the graceful shutdown of the job.
HTCondor Version 8.6.4 Manual
265
3.5.9. condor_startd Configuration File Macros
MAXJOBRETIREMENTTIME When the condor_startd wants to evict a job, a job which has run for less than the
number of seconds specified by this expression will not be hard-killed. The condor_startd will wait for the job
to finish or to exceed this amount of time, whichever comes sooner. Time spent in suspension does not count
against the job. The default value of 0 (when the configuration variable is not present) means that the job gets
no retirement time. If the job vacating policy grants the job X seconds of vacating time, a preempted job will be
soft-killed X seconds before the end of its retirement time, so that hard-killing of the job will not happen until the
end of the retirement time if the job does not finish shutting down before then. Note that in peaceful shutdown
mode of the condor_startd, retirement time is treated as though infinite. In graceful shutdown mode, the job will
not be preempted until the configured retirement time expires or SHUTDOWN_GRACEFUL_TIMEOUT expires.
In fast shutdown mode, retirement time is ignored. See MAXJOBRETIREMENTTIME in section 3.7.1 for further
explanation.
By default the condor_negotiator will not match jobs to a slot with retirement time remaining. This behavior is
controlled by NEGOTIATOR_CONSIDER_EARLY_PREEMPTION.
There is no default value for this configuration variable.
CLAIM_WORKLIFE This expression specifies the number of seconds after which a claim will stop accepting additional jobs. The default is 1200, which is 20 minutes. Once the condor_negotiator gives a condor_schedd a
claim to a slot, the condor_schedd will keep running jobs on that slot as long as it has more jobs with matching
requirements, and CLAIM_WORKLIFE has not expired, and it is not preempted. Once CLAIM_WORKLIFE
expires, any existing job may continue to run as usual, but once it finishes or is preempted, the claim is closed.
When CLAIM_WORKLIFE is -1, this is treated as an infinite claim worklife, so claims may be held indefinitely
(as long as they are not preempted and the user does not run out of jobs, of course). A value of 0 has the effect
of not allowing more than one job to run per claim, since it immediately expires after the first job starts running.
MAX_CLAIM_ALIVES_MISSED The condor_schedd sends periodic updates to each condor_startd as a keep alive
(see the description of ALIVE_INTERVAL on page 285). If the condor_startd does not receive any keep alive
messages, it assumes that something has gone wrong with the condor_schedd and that the resource is not being
effectively used. Once this happens, the condor_startd considers the claim to have timed out, it releases the
claim, and starts advertising itself as available for other jobs. Because these keep alive messages are sent via
UDP, they are sometimes dropped by the network. Therefore, the condor_startd has some tolerance for missed
keep alive messages, so that in case a few keep alives are lost, the condor_startd will not immediately release the
claim. This setting controls how many keep alive messages can be missed before the condor_startd considers
the claim no longer valid. The default is 6.
STARTD_HAS_BAD_UTMP When the condor_startd is computing the idle time of all the users of the machine (both
local and remote), it checks the utmp file to find all the currently active ttys, and only checks access time
of the devices associated with active logins. Unfortunately, on some systems, utmp is unreliable, and the
condor_startd might miss keyboard activity by doing this. So, if your utmp is unreliable, set this macro to
True and the condor_startd will check the access time on all tty and pty devices.
CONSOLE_DEVICES This macro allows the condor_startd to monitor console (keyboard and mouse) activity by
checking the access times on special files in /dev. Activity on these files shows up as ConsoleIdle time
in the condor_startd’s ClassAd. Give a comma-separated list of the names of devices considered the console,
without the /dev/ portion of the path name. The defaults vary from platform to platform, and are usually
correct.
One possible exception to this is on Linux, where we use “mouse” as one of the entries. Most Linux installations
put in a soft link from /dev/mouse that points to the appropriate device (for example, /dev/psaux for a
HTCondor Version 8.6.4 Manual
266
3.5.9. condor_startd Configuration File Macros
PS/2 bus mouse, or /dev/tty00 for a serial mouse connected to com1). However, if your installation does
not have this soft link, you will either need to put it in (you will be glad you did), or change this macro to point
to the right device.
Unfortunately, modern versions of Linux do not update the access time of device files for USB devices. Thus,
these files cannot be be used to determine when the console is in use. Instead, use the condor_kbdd daemon,
which gets this information by connecting to the X server.
KBDD_BUMP_CHECK_SIZE The number of pixels that the mouse can move in the X and/or Y direction, while still
being considered a bump, and not keyboard activity. If the movement is greater than this bump size then the
move is not a transient one, and it will register as activity. The default is 16, and units are pixels. Setting the
value to 0 effectively disables bump testing.
KBDD_BUMP_CHECK_AFTER_IDLE_TIME The number of seconds of keyboard idle time that will pass before
bump testing begins. The default is 15 minutes.
STARTD_JOB_ATTRS When the machine is claimed by a remote user, the condor_startd can also advertise arbitrary
attributes from the job ClassAd in the machine ClassAd. List the attribute names to be advertised. NOTE: Since
these are already ClassAd expressions, do not do anything unusual with strings. By default, the job ClassAd
attributes JobUniverse, NiceUser, ExecutableSize and ImageSize are advertised into the machine ClassAd. This
setting was formerly called STARTD_JOB_EXPRS. The older name is still supported, but support for the older
name may be removed in a future version of HTCondor.
STARTD_ATTRS This macro is described in section 3.5.4 as <SUBSYS>_ATTRS.
STARTD_DEBUG This macro (and other settings related to debug logging in the condor_startd) is described in section 3.5.3 as <SUBSYS>_DEBUG.
STARTD_ADDRESS_FILE This macro is described in section 3.5.4 as <SUBSYS>_ADDRESS_FILE
STARTD_SHOULD_WRITE_CLAIM_ID_FILE The condor_startd can be configured to write out the ClaimId for
the next available claim on all slots to separate files. This boolean attribute controls whether the condor_startd
should write these files. The default value is True.
STARTD_CLAIM_ID_FILE This macro controls what file names are used if the above
STARTD_SHOULD_WRITE_CLAIM_ID_FILE is true. By default, HTCondor will write the ClaimId
into a file in the $(LOG) directory called .startd_claim_id.slotX, where X is the value of SlotID,
the integer that identifies a given slot on the system, or 1 on a single-slot machine. If you define your own value
for this setting, you should provide a full path, and HTCondor will automatically append the .slotX portion
of the file name.
SlotWeight This may be used to give a slot greater weight when calculating usage, computing fair shares, and
enforcing group quotas. For example, claiming a slot with SlotWeight = 2 is equivalent to claiming two
SlotWeight = 1 slots. The default value is Cpus, the number of CPUs associated with the slot, which is
1 unless specially configured. Any expression referring to attributes of the slot ClassAd and evaluating to a
positive floating point number is valid.
NUM_CPUS An integer value, which can be used to lie to the condor_startd daemon about how many CPUs a machine
has. When set, it overrides the value determined with HTCondor’s automatic computation of the number of
CPUs in the machine. Lying in this way can allow multiple HTCondor jobs to run on a single-CPU machine,
HTCondor Version 8.6.4 Manual
267
3.5.9. condor_startd Configuration File Macros
by having that machine treated like a multi-core machine with multiple CPUs, which could have different
HTCondor jobs running on each one. Or, a multi-core machine may advertise more slots than it has CPUs.
However, lying in this manner will hurt the performance of the jobs, since now multiple jobs will run on the
same CPU, and the jobs will compete with each other. The option is only meant for people who specifically
want this behavior and know what they are doing. It is disabled by default.
The default value is $(DETECTED_CPUS).
The condor_startd only takes note of the value of this configuration variable on start up, therefore it cannot be
changed with a simple reconfigure. To change this, restart the condor_startd daemon for the change to take
effect. The command will be
condor_restart -startd
MAX_NUM_CPUS An integer value used as a ceiling for the number of CPUs detected by HTCondor on a machine.
This value is ignored if NUM_CPUS is set. If set to zero, there is no ceiling. If not defined, the default value is
zero, and thus there is no ceiling.
Note that this setting cannot be changed with a simple reconfigure, either by sending a SIGHUP or by using the
condor_reconfig command. To change this, restart the condor_startd daemon for the change to take effect. The
command will be
condor_restart -startd
COUNT_HYPERTHREAD_CPUS This configuration variable controls how HTCondor sees hyper-threaded processors.
When set to the default value of True, it includes virtual CPUs in the default value of DETECTED_CPUS.
On dedicated cluster nodes, counting virtual CPUs can sometimes improve total throughput at the expense
of individual job speed. However, counting them on desktop workstations can interfere with interactive job
performance.
MEMORY Normally, HTCondor will automatically detect the amount of physical memory available on your machine.
Define MEMORY to tell HTCondor how much physical memory (in MB) your machine has, overriding the value
HTCondor computes automatically. The actual amount of memory detected by HTCondor is always available
in the pre-defined configuration macro DETECTED_MEMORY.
RESERVED_MEMORY How much memory would you like reserved from HTCondor? By default, HTCondor considers all the physical memory of your machine as available to be used by HTCondor jobs. If RESERVED_MEMORY
is defined, HTCondor subtracts it from the amount of memory it advertises as available.
STARTD_NAME Used to give an alternative value to the Name attribute in the condor_startd’s ClassAd. This esoteric
configuration macro might be used in the situation where there are two condor_startd daemons running on one
machine, and each reports to the same condor_collector. Different names will distinguish the two daemons. See
the description of MASTER_NAME in section 3.5.8 on page 260 for defaults and composition of valid HTCondor
daemon names.
RUNBENCHMARKS A boolean expression that specifies whether to run benchmarks. When the machine is in the
Unclaimed state and this expression evaluates to True, benchmarks will be run. If RUNBENCHMARKS is
specified and set to anything other than False, additional benchmarks will be run once, when the condor_startd
starts. To disable start up benchmarks, set RunBenchmarks to False.
HTCondor Version 8.6.4 Manual
268
3.5.9. condor_startd Configuration File Macros
269
DedicatedScheduler A string that identifies the dedicated scheduler this machine is managed by. Section 3.14.8
on page 495 details the use of a dedicated scheduler.
STARTD_NOCLAIM_SHUTDOWN The number of seconds to run without receiving a claim before shutting HTCondor
down on this machine. Defaults to unset, which means to never shut down. This is primarily intended to facilitate
glidein; use in other situations is not recommended.
STARTD_PUBLISH_WINREG A string containing a semicolon-separated list of Windows registry key names. For
each registry key, the contents of the registry key are published in the machine ClassAd. All attribute names
are prefixed with WINREG_. The remainder of the attribute name is formed in one of two ways. The first way
explicitly specifies the name within the list with the syntax
STARTD_PUBLISH_WINREG = AttrName1 = KeyName1; AttrName2 = KeyName2
The second way of forming the attribute name derives the attribute names from the key names in the list. The
derivation uses the last three path elements in the key name and changes each illegal character to an underscore
character. Illegal characters are essentially any non-alphanumeric character. In addition, the percent character
(%) is replaced by the string Percent, and the string /sec is replaced by the string _Per_Sec.
HTCondor expects that the hive identifier, which is the first element in the full path given by a key name, will
be the valid abbreviation. Here is a list of abbreviations:
HKLM is the abbreviation for HKEY_LOCAL_MACHINE
HKCR is the abbreviation for HKEY_CLASSES_ROOT
HKCU is the abbreviation for HKEY_CURRENT_USER
HKPD is the abbreviation for HKEY_PERFORMANCE_DATA
HKCC is the abbreviation for HKEY_CURRENT_CONFIG
HKU is the abbreviation for HKEY_USERS
The HKPD key names are unusual, as they are not shown in regedit. Their values are periodically updated at the
interval defined by UPDATE_INTERVAL. The others are not updated until condor_reconfig is issued.
Here is a complete example of the configuration variable definition,
STARTD_PUBLISH_WINREG = HKLM\Software\Perl\BinDir; \
BATFile_RunAs_Command = HKCR\batFile\shell\RunAs\command; \
HKPD\Memory\Available MBytes; \
BytesAvail = HKPD\Memory\Available Bytes; \
HKPD\Terminal Services\Total Sessions; \
HKPD\Processor\% Idle Time; \
HKPD\System\Processes
which generates the following portion of a machine ClassAd:
WINREG_Software_Perl_BinDir = "C:\Perl\bin\perl.exe"
WINREG_BATFile_RunAs_Command = "%SystemRoot%\System32\cmd.exe /C \"%1\" %*"
WINREG_Memory_Available_MBytes = 5331
HTCondor Version 8.6.4 Manual
3.5.9. condor_startd Configuration File Macros
270
WINREG_BytesAvail = 5590536192.000000
WINREG_Terminal_Services_Total_Sessions = 2
WINREG_Processor_Percent_Idle_Time = 72.350384
WINREG_System_Processes = 166
MOUNT_UNDER_SCRATCH A ClassAd expression, which when evaluated in the context of the job ClassAd evaluates
to a comma separated list of directories. For each directory in the list, HTCondor creates a directory in the job’s
temporary scratch directory with that name, and makes it available at the given name using bind mounts. This
is available on Linux systems which provide bind mounts and per-process tree mount tables, such as Red Hat
Enterprise Linux 5. A bind mount is like a symbolic link, but is not globally visible to all processes. It is only
visible to the job and the job’s child processes. As an example:
MOUNT_UNDER_SCRATCH = ifThenElse(TARGET.UtsnameSysname ? "Linux", "/tmp,/var/tmp",
If the job is running on a Linux system, it will see the usual /tmp and /var/tmp directories, but when
accessing files via these paths, the system will redirect the access. The resultant files will actually end up in
directories named tmp or var/tmp under the the job’s temporary scratch directory. This is useful, because the
job’s scratch directory will be cleaned up after the job completes, two concurrent jobs will not interfere with
each other, and because jobs will not be able to fill up the real /tmp directory. Another use case might be for
home directories, which some jobs might want to write to, but that should be cleaned up after each job run. The
default value if not defined will be that no directories are mounted in the job’s temporary scratch directory.
If the job’s execute directory is encrypted, /tmp and /var/tmp are automatically added to
MOUNT_UNDER_SCRATCH when the job is run (they will not show up if MOUNT_UNDER_SCRATCH is examined with condor_config_val).
Note that the MOUNT_UNDER_SCRATCH mounts do not take place until the PreCmd of the job, if any,
completes. (See 11 for information on PreCmd.)
Also note that, if MOUNT_UNDER_SCRATCH is defined, it must either be a string or an expression that evaluates
to a string.
For Docker Universe jobs, any directories that are mounted under scratch are also volume mounted on the same
paths inside the container. That is, any reads or writes to files in those directories goes to the host filesytem
under the scratch directory. This is useful if a container has limited space to grow a filesytem.
The following macros control if the condor_startd daemon should perform backfill computations whenever resources would otherwise be idle. See section 3.14.9 on page 498 on Configuring HTCondor for Running Backfill Jobs
for details.
ENABLE_BACKFILL A boolean value that, when True, indicates that the machine is willing to perform backfill
computations when it would otherwise be idle. This is not a policy expression that is evaluated, it is a simple
True or False. This setting controls if any of the other backfill-related expressions should be evaluated. The
default is False.
BACKFILL_SYSTEM A string that defines what backfill system to use for spawning and managing backfill computations. Currently, the only supported value for this is "BOINC", which stands for the Berkeley Open Infrastructure for Network Computing. See http://boinc.berkeley.edu for more information about BOINC. There is no
default value, administrators must define this.
HTCondor Version 8.6.4 Manual
3.5.9. condor_startd Configuration File Macros
START_BACKFILL A boolean expression that is evaluated whenever an HTCondor resource is in the Unclaimed/Idle
state and the ENABLE_BACKFILL expression is True. If START_BACKFILL evaluates to True, the machine will enter the Backfill state and attempt to spawn a backfill computation. This expression is analogous
to the START expression that controls when an HTCondor resource is available to run normal HTCondor
jobs. The default value is False (which means do not spawn a backfill job even if the machine is idle and
ENABLE_BACKFILL expression is True). For more information about policy expressions and the Backfill
state, see section 3.7 beginning on page 370, especially sections 3.7.1, 3.7.1, and 3.7.1.
EVICT_BACKFILL A boolean expression that is evaluated whenever an HTCondor resource is in the Backfill state
which, when True, indicates the machine should immediately kill the currently running backfill computation
and return to the Owner state. This expression is a way for administrators to define a policy where interactive
users on a machine will cause backfill jobs to be removed. The default value is False. For more information
about policy expressions and the Backfill state, see section 3.7 beginning on page 370, especially sections 3.7.1,
3.7.1, and 3.7.1.
The following macros only apply to the condor_startd daemon when it is running on a multi-core machine. See
section 3.7.1 on page 396 for details.
STARTD_RESOURCE_PREFIX A string which specifies what prefix to give the unique HTCondor resources that
are advertised on multi-core machines. Previously, HTCondor used the term virtual machine to describe these
resources, so the default value for this setting was vm. However, to avoid confusion with other kinds of virtual
machines, such as the ones created using tools like VMware or Xen, the old virtual machine terminology has
been changed, and has become the term slot. Therefore, the default value of this prefix is now slot. If sites
want to continue using vm, or prefer something other slot, this setting enables sites to define what string the
condor_startd will use to name the individual resources on a multi-core machine.
SLOTS_CONNECTED_TO_CONSOLE An integer which indicates how many of the machine slots the condor_startd
is representing should be "connected" to the console. This allows the condor_startd to notice console activity.
Defaults to the number of slots in the machine, which is $(NUM_CPUS).
SLOTS_CONNECTED_TO_KEYBOARD An integer which indicates how many of the machine slots the condor_startd
is representing should be "connected" to the keyboard (for remote tty activity, as well as console activity). This
defaults to all slots (N in a machine with N CPUs).
DISCONNECTED_KEYBOARD_IDLE_BOOST If there are slots not connected to either the keyboard or the console,
the corresponding idle time reported will be the time since the condor_startd was spawned, plus the value of this
macro. It defaults to 1200 seconds (20 minutes). We do this because if the slot is configured not to care about
keyboard activity, we want it to be available to HTCondor jobs as soon as the condor_startd starts up, instead of
having to wait for 15 minutes or more (which is the default time a machine must be idle before HTCondor will
start a job). If you do not want this boost, set the value to 0. If you change your START expression to require
more than 15 minutes before a job starts, but you still want jobs to start right away on some of your multi-core
nodes, increase this macro’s value.
STARTD_SLOT_ATTRS The list of ClassAd attribute names that should be shared across all slots on the same
machine. This setting was formerly know as STARTD_VM_ATTRS or STARTD_VM_EXPRS (before version
6.9.3). For each attribute in the list, the attribute’s value is taken from each slot’s machine ClassAd and placed
into the machine ClassAd of all the other slots within the machine. For example, if the configuration file for a
2-slot machine contains
HTCondor Version 8.6.4 Manual
271
3.5.9. condor_startd Configuration File Macros
STARTD_SLOT_ATTRS = State, Activity, EnteredCurrentActivity
then the machine ClassAd for both slots will contain attributes that will be of the form:
slot1_State = "Claimed"
slot1_Activity = "Busy"
slot1_EnteredCurrentActivity = 1075249233
slot2_State = "Unclaimed"
slot2_Activity = "Idle"
slot2_EnteredCurrentActivity = 1075240035
The following settings control the number of slots reported for a given multi-core host, and what attributes each
one has. They are only needed if you do not want to have a multi-core machine report to HTCondor with a separate
slot for each CPU, with all shared system resources evenly divided among them. Please read section 3.7.1 on page 397
for details on how to properly configure these settings to suit your needs.
NOTE: You can only change the number of each type of slot the condor_startd is reporting with a simple reconfig
(such as sending a SIGHUP signal, or using the condor_reconfig command). You cannot change the definition of the
different slot types with a reconfig. If you change them, you must restart the condor_startd for the change to take
effect (for example, using condor_restart -startd).
NOTE: Prior to version 6.9.3, any settings that included the term slot used to use virtual machine or vm. If
searching for information about one of these older settings, search for the corresponding attribute names using slot,
instead.
MAX_SLOT_TYPES The maximum number of different slot types. Note: this is the maximum number of different
types, not of actual slots. Defaults to 10. (You should only need to change this setting if you define more than
10 separate slot types, which would be pretty rare.)
SLOT_TYPE_<N> This setting defines a given slot type, by specifying what part of each shared system resource (like RAM, swap space, etc) this kind of slot gets. This setting has no effect unless you also define
NUM_SLOTS_TYPE_<N>. N can be any integer from 1 to the value of $(MAX_SLOT_TYPES), such as
SLOT_TYPE_1. The format of this entry can be somewhat complex, so please refer to section 3.7.1 on page 397
for details on the different possibilities.
SLOT_TYPE_<N>_PARTITIONABLE A boolean variable that defaults to False. When True, this slot permits
dynamic provisioning, as specified in section 3.7.1.
CLAIM_PARTITIONABLE_LEFTOVERS A boolean variable that defaults to True. When True within the configuration for both the condor_schedd and the condor_startd, and the condor_schedd claims a partitionable slot, the
condor_startd returns the slot’s ClassAd and a claim id for leftover resources. In doing so, the condor_schedd
can claim multiple dynamic slots without waiting for a negotiation cycle.
MACHINE_RESOURCE_NAMES A comma and/or space separated list of resource names that represent custom resources specific to a machine. These resources are further intended to be statically divided or partitioned, and
these resource names identify the configuration variables that define the partitioning. If used, custom resources
without names in the list are ignored.
HTCondor Version 8.6.4 Manual
272
3.5.9. condor_startd Configuration File Macros
MACHINE_RESOURCE_<name> An integer that specifies the quantity of or list of identifiers for the customized local machine resource available for an SMP machine. The portion of this configuration variable’s name identified
with <name> will be used to label quantities of the resource allocated to a slot. If a quantity is specified, the
resource is presumed to be fungible and slots will be allocated a quantity of the resource but specific instances
will not be identified. If a list of identifiers is specified the quantity is the number of identifiers and slots will be
allocated both a quantity of the resource and assigned specific resource identifiers.
OFFLINE_MACHINE_RESOURCE_<name> A comma and/or space separated list of resource identifiers for
any customized local machine resources that are currently offline, and therefore should not be allocated
to a slot. The identifiers specified here must match those specified by value of configuration variables
MACHINE_RESOURCE_<name> or MACHINE_RESOURCE_INVENTORY_<name>, or the identifiers will
be ignored. The <name> identifies the type of resource, as specified by the value of configuration variable
MACHINE_RESOURCE_NAMES. This configuration variable is used to have resources that are detected and
reported to exist by HTCondor, but not assigned to slots. A restart of the condor_startd is required for changes
to this configuration variable to take effect.
MACHINE_RESOURCE_INVENTORY_<name> Specifies a command line that is executed upon start up of the condor_startd daemon. The script is expected to output an attribute definition of the form
Detected<xxx>=y
or of the form
Detected<xxx>="y, z, a, ..."
where <xxx> is the name of a resource that exists on the machine, and y is the quantity of the resource or
"y, z, a, ..." is a comma and/or space separated list of identifiers of the resource that exist on the
machine. This attribute is added to the machine ClassAd, such that these resources may be statically divided or partitioned. A script may be a convenient way to specify a calculated or detected quantity of the
resource, instead of specifying a fixed quantity or list of the resource in the the configuration when set by
MACHINE_RESOURCE_<name>.
ENVIRONMENT_FOR_Assigned<name> A space separated list of environment variables to set for the job.
Each environment variable will be set to the list of assigned resources defined by the slot ClassAd attribute
Assigned<name>. Each environment variable name may be followed by an equals sign and a Perl style
regular expression that defines how to modify each resource ID before using it as the value of the environment
variable. As a special case for CUDA GPUs, if the environment variable name is CUDA_VISIBLE_DEVICES,
then the correct Perl style regular expression is applied automatically.
For example, with the configuration
ENVIRONMENT_FOR_AssignedGPUs = VISIBLE_GPUS=/^/gpuid:/
and with the machine ClassAd attribute AssignedGPUs = "CUDA1, CUDA2", the job’s environment will
contain
VISIBLE_GPUS = gpuid:CUDA1, gpuid:CUDA2
HTCondor Version 8.6.4 Manual
273
3.5.9. condor_startd Configuration File Macros
ENVIRONMENT_VALUE_FOR_UnAssigned<name> Defines the value to set for environment variables specified
in by configuration variable ENVIRONMENT_FOR_Assigned<name> when there is no machine ClassAd
attribute Assigned<name> for the slot. This configuration variable exists to deal with the situation where
jobs will use a resource that they have not been assigned because there is no explicit assignment. The CUDA
runtime library (for GPUs) has this problem.
For example, where configuration is
ENVIRONMENT_FOR_AssignedGPUs = VISIBLE_GPUS
ENVIRONMENT_VALUE_FOR_UnAssignedGPUs = none
and there is no machine ClassAd attribute AssignedGPUs, the job’s environment will contain
VISIBLE_GPUS = none
MUST_MODIFY_REQUEST_EXPRS A boolean value that defaults to False. When False, configuration variables
whose names begin with MODIFY_REQUEST_EXPR are only applied if the job claim still matches the partitionable slot after modification. If True, the modifications always take place, and if the modifications cause the
claim to no longer match, then the condor_startd will simply refuse the claim.
MODIFY_REQUEST_EXPR_REQUESTMEMORY An integer expression used by the condor_startd daemon to modify
the evaluated value of the RequestMemory job ClassAd attribute, before it used to provision a dynamic slot.
The default value is given by
quantize(RequestMemory,{128})
MODIFY_REQUEST_EXPR_REQUESTDISK An integer expression used by the condor_startd daemon to modify
the evaluated value of the RequestDisk job ClassAd attribute, before it used to provision a dynamic slot.
The default value is given by
quantize(RequestDisk,{1024})
MODIFY_REQUEST_EXPR_REQUESTCPUS An integer expression used by the condor_startd daemon to modify
the evaluated value of the RequestCpus job ClassAd attribute, before it used to provision a dynamic slot.
The default value is given by
quantize(RequestCpus,{1})
NUM_SLOTS_TYPE_<N> This macro controls how many of a given slot type are actually reported to HTCondor.
There is no default.
NUM_SLOTS An integer value representing the number of slots reported when the multi-core machine is being evenly
divided, and the slot type settings described above are not being used. The default is one slot for each CPU.
This setting can be used to reserve some CPUs on a multi-core machine, which would not be reported to the
HTCondor pool. This value cannot be used to make HTCondor advertise more slots than there are CPUs on the
machine. To do that, use NUM_CPUS.
HTCondor Version 8.6.4 Manual
274
3.5.9. condor_startd Configuration File Macros
ALLOW_VM_CRUFT A boolean value that HTCondor sets and uses internally, currently defaulting to True. When
True, HTCondor looks for configuration variables named with the previously used string VM after searching
unsuccessfully for variables named with the currently used string SLOT. When False, HTCondor does not
look for variables named with the previously used string VM after searching unsuccessfully for the string SLOT.
The following variables set consumption policies for partitionable slots. Section 3.7.1 details consumption policies.
CONSUMPTION_POLICY A boolean value that defaults to False.
When True, consumption policies
are enabled for partitionable slots within the condor_startd daemon.
Any definition of the form
SLOT_TYPE_<N>_CONSUMPTION_POLICY overrides this global definition for the given slot type.
CONSUMPTION_<Resource> An expression that specifies a consumption policy for a particular resource within
a partitionable slot. To support a consumption policy, each resource advertised by the slot must have such a
policy configured. Custom resources may be specified, substituting the resource name for <Resource>. Any
definition of the form SLOT_TYPE_<N>_CONSUMPTION_<Resource> overrides this global definition for
the given slot type. CPUs, memory, and disk resources are always advertised by condor_startd, and have the
default values:
CONSUMPTION_CPUS = quantize(target.RequestCpus,{1})
CONSUMPTION_MEMORY = quantize(target.RequestMemory,{128})
CONSUMPTION_DISK = quantize(target.RequestDisk,{1024})
Custom resources have no default consumption policy.
SLOT_WEIGHT An expression that specifies a slot’s weight, used as a multiplier the condor_negotiator daemon
during matchmaking to assess user usage of a slot, which affects user priority. Defaults to Cpus. In the case
of slots with consumption policies, the cost of each match is is assessed as the difference in the slot weight
expression before and after the resources consumed by the match are deducted from the slot. Only Memory,
Cpus and Disk are valid attributes for this parameter.
NUM_CLAIMS Specifies the number of claims a partitionable slot will advertise for use by the condor_negotiator
daemon. In the case of slots with a defined consumption policy, the condor_negotiator may match more than
one job to the slot in a single negotiation cycle. For partitionable slots with a consumption policy, NUM_CLAIMS
defaults to the number of CPUs owned by the slot. Otherwise, it defaults to 1.
The following configuration variables support java universe jobs.
JAVA The full path to the Java interpreter (the Java Virtual Machine).
JAVA_CLASSPATH_ARGUMENT The command line argument to the Java interpreter (the Java Virtual Machine) that
specifies the Java Classpath. Classpath is a Java-specific term that denotes the list of locations (.jar files and/or
directories) where the Java interpreter can look for the Java class files that a Java program requires.
JAVA_CLASSPATH_SEPARATOR The single character used to delimit constructed entries in the Classpath for the
given operating system and Java Virtual Machine. If not defined, the operating system is queried for its default
Classpath separator.
HTCondor Version 8.6.4 Manual
275
3.5.9. condor_startd Configuration File Macros
JAVA_CLASSPATH_DEFAULT A list of path names to .jar files to be added to the Java Classpath by default. The
comma and/or space character delimits list entries.
JAVA_EXTRA_ARGUMENTS A list of additional arguments to be passed to the Java executable.
The following configuration variables control .NET version advertisement.
STARTD_PUBLISH_DOTNET A boolean value that controls the advertising of the .NET framework on Windows
platforms. When True, the condor_startd will advertise all installed versions of the .NET framework within
the DotNetVersions attribute in the condor_startd machine ClassAd. The default value is True. Set the
value to false to turn off .NET version advertising.
DOT_NET_VERSIONS A string expression that administrators can use to override the way that .NET versions are
advertised. If the administrator wishes to advertise .NET installations, but wishes to do so in a format different
than what the condor_startd publishes in its ClassAds, setting a string in this expression will result in the
condor_startd publishing the string when STARTD_PUBLISH_DOTNET is True. No value is set by default.
These macros control the power management capabilities of the condor_startd to optionally put the machine in to
a low power state and wake it up later. See section 3.18 on page 519 on Power Management for more details.
HIBERNATE_CHECK_INTERVAL An integer number of seconds that determines how often the condor_startd
checks to see if the machine is ready to enter a low power state. The default value is 0, which disables the
check. If not 0, the HIBERNATE expression is evaluated within the context of each slot at the given interval. If
used, a value 300 (5 minutes) is recommended.
As a special case, the interval is ignored when the machine has just returned from a low power state, excluding
"SHUTDOWN". In order to avoid machines from volleying between a running state and a low power state, an
hour of uptime is enforced after a machine has been woken. After the hour has passed, regular checks resume.
HIBERNATE A string expression that represents lower power state. When this state name evaluates to a valid state
other than "NONE", causes HTCondor to put the machine into the specified low power state. The following
names are supported (and are not case sensitive):
"NONE", "0": No-op; do not enter a low power state
"S1", "1", "STANDBY", "SLEEP": On Windows, this is Sleep (standby)
"S2", "2": On Windows, this is Sleep (standby)
"S3", "3", "RAM", "MEM", "SUSPEND": On Windows, this is Sleep (standby)
"S4", "4", "DISK", "HIBERNATE": Hibernate
"S5", "5", "SHUTDOWN", "OFF": Shutdown (soft-off)
The HIBERNATE expression is written in terms of the S-states as defined in the Advanced Configuration and
Power Interface (ACPI) specification. The S-states take the form S<n>, where <n> is an integer in the range 0
to 5, inclusive. The number that results from evaluating the expression determines which S-state to enter. The
notation was adopted because it appears to be the standard naming scheme for power states on several popular
HTCondor Version 8.6.4 Manual
276
3.5.9. condor_startd Configuration File Macros
operating systems, including various flavors of Windows and Linux distributions. The other strings, such as
"RAM" and "DISK", are provided for ease of configuration.
Since this expression is evaluated in the context of each slot on the machine, any one slot has veto power over
the other slots. If the evaluation of HIBERNATE in one slot evaluates to "NONE" or "0", then the machine will
not be placed into a low power state. On the other hand, if all slots evaluate to a non-zero value, but differ in
value, then the largest value is used as the representative power state.
Strings that do not match any in the table above are treated as "NONE".
UNHIBERNATE A boolean expression that specifies when an offline machine should be woken up. The default value
is MachineLastMatchTime =!= UNDEFINED. This expression does not do anything, unless there is an
instance of condor_rooster running, or another program that evaluates the Unhibernate expression of offline
machine ClassAds. In addition, the collecting of offline machine ClassAds must be enabled for this expression
to work. The variable COLLECTOR_PERSISTENT_AD_LOG on page ?? detailed on page 278 explains this.
The special attribute MachineLastMatchTime is updated in the ClassAds of offline machines when a job
would have been matched to the machine if it had been online. For multi-slot machines, the offline ClassAd
for slot1 will also contain the attributes slot<X>_MachineLastMatchTime, where X is replaced by the
slot id of the other slots that would have been matched while offline. This allows the slot1 UNHIBERNATE
expression to refer to all of the slots on the machine, in case that is necessary. By default, condor_rooster will
wake up a machine if any slot on the machine has its UNHIBERNATE expression evaluate to True.
HIBERNATION_PLUGIN A string which specifies the path and executable name of the hibernation plug-in that the
condor_startd should use in the detection of low power states and switching to the low power states. The default
value is $(LIBEXEC)/power_state. A default executable in that location which meets these specifications
is shipped with HTCondor.
The condor_startd initially invokes this plug-in with both the value defined for
HIBERNATION_PLUGIN_ARGS and the argument ad, and expects the plug-in to output a ClassAd to
its standard output stream. The condor_startd will use this ClassAd to determine what low power setting to use on further invocations of the plug-in. To that end, the ClassAd must contain the attribute
HibernationSupportedStates, a comma separated list of low power modes that are available. The
recognized mode strings are the same as those in the table for the configuration variable HIBERNATE. The
optional attribute HibernationMethod specifies a string which describes the mechanism used by the
plug-in. The default Linux plug-in shipped with HTCondor will produce one of the strings NONE, /sys,
/proc, or pm-utils. The optional attribute HibernationRawMask is an integer which represents the bit
mask of the modes detected.
Subsequent condor_startd invocations of the plug-in have command line arguments defined by
HIBERNATION_PLUGIN_ARGS plus the argument set <power-mode>, where <power-mode> is one of the
supported states as given in the attribute HibernationSupportedStates.
HIBERNATION_PLUGIN_ARGS Command line arguments appended to the command that invokes the plug-in. The
additional argument ad is appended when the condor_startd initially invokes the plug-in.
HIBERNATION_OVERRIDE_WOL A boolean value that defaults to False. When True, it causes the condor_startd daemon’s detection of the whether or not the network interface handles WOL packets to be ignored.
When False, hibernation is disabled if the network interface does not use WOL packets to wake from hibernation. Therefore, when True hibernation can be enabled despite the fact that WOL packets are not used to wake
machines.
HTCondor Version 8.6.4 Manual
277
3.5.9. condor_startd Configuration File Macros
278
LINUX_HIBERNATION_METHOD A string that can be used to override the default search used by HTCondor on
Linux platforms to detect the hibernation method to use. This is used by the default hibernation plug-in executable that is shipped with HTCondor. The default behavior orders its search with:
1. Detect and use the pm-utils command line tools. The corresponding string is defined with "pm-utils".
2. Detect and use the directory in the virtual file system /sys/power. The corresponding string is defined
with "/sys".
3. Detect and use the directory in the virtual file system /proc/ACPI. The corresponding string is defined
with "/proc".
To override this ordered search behavior, and force the use of one particular method,
LINUX_HIBERNATION_METHOD to one of the defined strings.
OFFLINE_LOG This configuration variable
COLLECTOR_PERSISTENT_AD_LOG.
is
no
longer
used.
It
has
been
replaced
set
by
OFFLINE_EXPIRE_ADS_AFTER An integer number of seconds specifying the lifetime of the persistent machine
ClassAd representing a hibernating machine. Defaults to the largest 32-bit integer.
The following macros control the optional computation of resource availability statistics in the condor_startd.
STARTD_COMPUTE_AVAIL_STATS A boolean value that determines if the condor_startd computes resource availability statistics. The default is False.
If STARTD_COMPUTE_AVAIL_STATS is True, the condor_startd will define the following ClassAd attributes for resources:
AvailTime The proportion of the time (between 0.0 and 1.0) that this resource has been in a state other than
Owner.
LastAvailInterval The duration in seconds of the last period between Owner states.
The following attributes will also be included if the resource is not in the Owner state:
AvailSince The time at which the resource last left the Owner state. Measured in the number of seconds
since the epoch (00:00:00 UTC, Jan 1, 1970).
AvailTimeEstimate Based on past history, an estimate of how long the current period between Owner
states will last.
STARTD_AVAIL_CONFIDENCE A floating point number representing the confidence level of the condor_startd
daemon’s AvailTime estimate. By default, the estimate is based on the 80th percentile of past values, so the
value is initially set to 0.8.
STARTD_MAX_AVAIL_PERIOD_SAMPLES An integer that limits the number of samples of past available intervals
stored by the condor_startd to limit memory and disk consumption. Each sample requires 4 bytes of memory
and approximately 10 bytes of disk space.
HTCondor Version 8.6.4 Manual
3.5.10. condor_schedd Configuration File Entries
279
DOCKER Defines the path and executable name of the Docker CLI. The default value is /usr/bin/docker. Remember
that the condor user must also be in the docker group for Docker Universe to work.
An example of the configuration for running the Docker CLI:
DOCKER = /usr/bin/docker
DOCKER_IMAGE_CACHE_SIZE The number of most recently used Docker images that will be kept on the local
machine. The default value is 20.
OPENMPI_INSTALL_PATH The location of the Open MPI installation on the local machine. Referenced by
examples/openmpiscript, which is used for running Open MPI jobs in the parallel universe. The Open
MPI bin and lib directories should exist under this path. The default value is /usr/lib64/openmpi.
OPENMPI_EXCLUDE_NETWORK_INTERFACES A comma-delimited list of network interfaces that Open MPI
should not use for MPI communications. Referenced by examples/openmpiscript, which is used for
running Open MPI jobs in the parallel universe.
The list should contain any interfaces that your job could potentially see from any execute machine. The list
may contain undefined interfaces without generating errors. Open MPI should exclusively use low latency/high
speed networks it finds (e.g. InfiniBand) regardless of this setting. The default value is docker0,virbr0.
3.5.10 condor_schedd Configuration File Entries
These macros control the condor_schedd.
SHADOW This macro determines the full path of the condor_shadow binary that the condor_schedd spawns. It is
normally defined in terms of $(SBIN).
START_LOCAL_UNIVERSE A boolean value that defaults to TotalLocalJobsRunning < 200. The condor_schedd uses this macro to determine whether to start a local universe job. At intervals determined by
SCHEDD_INTERVAL, the condor_schedd daemon evaluates this macro for each idle local universe job that it
has. For each job, if the START_LOCAL_UNIVERSE macro is True, then the job’s Requirements expression is evaluated. If both conditions are met, then the job is allowed to begin execution.
The following example only allows 10 local universe jobs to execute concurrently.
TotalLocalJobsRunning is supplied by condor_schedd’s ClassAd:
The attribute
START_LOCAL_UNIVERSE = TotalLocalJobsRunning < 10
STARTER_LOCAL The complete path and executable name of the condor_starter to run for local universe jobs. This
variable’s value is defined in the initial configuration provided with HTCondor as
STARTER_LOCAL = $(SBIN)/condor_starter
HTCondor Version 8.6.4 Manual
3.5.10. condor_schedd Configuration File Entries
280
This variable would only be modified or hand added into the configuration for a pool to be upgraded from one
running a version of HTCondor that existed before the local universe to one that includes the local universe, but
without utilizing the newer, provided configuration files.
LOCAL_UNIV_EXECUTE A string value specifying the execute location for local universe jobs. Each running local
universe job will receive a uniquely named subdirectory within this directory. If not specified, it defaults to
$(SPOOL)/local_univ_execute.
START_SCHEDULER_UNIVERSE A boolean value that defaults to TotalSchedulerJobsRunning < 200.
The condor_schedd uses this macro to determine whether to start a scheduler universe job. At intervals determined by SCHEDD_INTERVAL, the condor_schedd daemon evaluates this macro for each idle scheduler
universe job that it has. For each job, if the START_SCHEDULER_UNIVERSE macro is True, then the job’s
Requirements expression is evaluated. If both conditions are met, then the job is allowed to begin execution.
The following example only allows 10 scheduler universe jobs to execute concurrently.
TotalSchedulerJobsRunning is supplied by condor_schedd’s ClassAd:
The attribute
START_SCHEDULER_UNIVERSE = TotalSchedulerJobsRunning < 10
SCHEDD_USES_STARTD_FOR_LOCAL_UNIVERSE A boolean value that defaults to false. When true, the condor_schedd will spawn a special startd process to run local universe jobs. This allows local universe jobs to run
with both a condor_shadow and a condor_starter, which means that file transfer will work with local universe
jobs.
MAX_JOBS_RUNNING An integer representing a limit on the number of condor_shadow processes spawned by a
given condor_schedd daemon, for all job universes except grid, scheduler, and local universe. Limiting the
number of running scheduler and local universe jobs can be done using START_LOCAL_UNIVERSE and
START_SCHEDULER_UNIVERSE. The actual number of allowed condor_shadow daemons may be reduced,
if the amount of memory defined by RESERVED_SWAP limits the number of condor_shadow daemons. A
value for MAX_JOBS_RUNNING that is less than or equal to 0 prevents any new job from starting. Changing
this setting to be below the current number of jobs that are running will cause running jobs to be aborted until
the number running is within the limit.
Like all integer configuration variables, MAX_JOBS_RUNNING may be a ClassAd expression that evaluates
to an integer, and which refers to constants either directly or via macro substitution. The default value is an
expression that depends on the total amount of memory and the operating system. The default expression
requires 1MByte of RAM per running job on the submit machine. In some environments and configurations,
this is overly generous and can be cut by as much as 50%. On Windows platforms, the number of running jobs
is capped at 2000. A 64-bit version of Windows is recommended in order to raise the value above the default.
Under Unix, the maximum default is now 10,000. To scale higher, we recommend that the system ephemeral
port range is extended such that there are at least 2.1 ports per running job.
Here are example configurations:
## Example 1:
MAX_JOBS_RUNNING = 10000
## Example 2:
## This is more complicated, but it produces the same limit as the default.
HTCondor Version 8.6.4 Manual
3.5.10. condor_schedd Configuration File Entries
## First define some expressions to use in our calculation.
## Assume we can use up to 80% of memory and estimate shadow private data
## size of 800k.
MAX_SHADOWS_MEM = ceiling($(DETECTED_MEMORY)*0.8*1024/800)
## Assume we can use ~21,000 ephemeral ports (avg ~2.1 per shadow).
## Under Linux, the range is set in /proc/sys/net/ipv4/ip_local_port_range.
MAX_SHADOWS_PORTS = 10000
## Under windows, things are much less scalable, currently.
## Note that this can probably be safely increased a bit under 64-bit windows.
MAX_SHADOWS_OPSYS = ifThenElse(regexp("WIN.*","$(OPSYS)"),2000,100000)
## Now build up the expression for MAX_JOBS_RUNNING. This is complicated
## due to lack of a min() function.
MAX_JOBS_RUNNING = $(MAX_SHADOWS_MEM)
MAX_JOBS_RUNNING = \
ifThenElse( $(MAX_SHADOWS_PORTS) < $(MAX_JOBS_RUNNING), \
$(MAX_SHADOWS_PORTS), \
$(MAX_JOBS_RUNNING) )
MAX_JOBS_RUNNING = \
ifThenElse( $(MAX_SHADOWS_OPSYS) < $(MAX_JOBS_RUNNING), \
$(MAX_SHADOWS_OPSYS), \
$(MAX_JOBS_RUNNING) )
MAX_JOBS_SUBMITTED This integer value limits the number of jobs permitted in a condor_schedd daemon’s
queue. Submission of a new cluster of jobs fails, if the total number of jobs would exceed this limit. The
default value for this variable is the largest positive integer value.
MAX_JOBS_PER_OWNER This integer value limits the number of jobs any given owner (user) is permitted to have
within a condor_schedd daemon’s queue. A job submission fails if it would cause this limit on the number of
jobs to be exceeded. The default value is 100000.
This configuration variable may be most useful in conjunction with MAX_JOBS_SUBMITTED, to ensure that
no one user can dominate the queue.
MAX_RUNNING_SCHEDULER_JOBS_PER_OWNER This integer value limits the number of scheduler universe jobs
that any given owner (user) can have running at one time. This limit will affect the number of running Dagman
jobs, but not the number of nodes within a DAG. The default is no limit
MAX_JOBS_PER_SUBMISSION This integer value limits the number of jobs any single submission is permitted to
add to a condor_schedd daemon’s queue. The whole submission fails if the number of jobs would exceed this
limit. The default value is 20000.
This configuration variable may be useful for catching user error, and for protecting a busy condor_schedd
daemon from the excessively lengthy interruption required to accept a very large number of jobs at one time.
MAX_SHADOW_EXCEPTIONS This macro controls the maximum number of times that condor_shadow processes
can have a fatal error (exception) before the condor_schedd will relinquish the match associated with the dying
shadow. Defaults to 5.
MAX_PENDING_STARTD_CONTACTS An integer value that limits the number of simultaneous connection attempts
by the condor_schedd when it is requesting claims from one or more condor_startd daemons. The intention is
to protect the condor_schedd from being overloaded by authentication operations. The default value is 0. The
special value 0 indicates no limit.
HTCondor Version 8.6.4 Manual
281
3.5.10. condor_schedd Configuration File Entries
CURB_MATCHMAKING A ClassAd expression evaluated by the condor_schedd in the context of the condor_schedd
daemon’s own ClassAd. While this expression evaluates to True, the condor_schedd will refrain from requesting more resources from a condor_negotiator. Defaults to False.
MAX_CONCURRENT_DOWNLOADS This specifies the maximum number of simultaneous transfers of output files
from execute machines to the submit machine. The limit applies to all jobs submitted from the same condor_schedd. The default is 10. A setting of 0 means unlimited transfers. This limit currently does not apply to
grid universe jobs or standard universe jobs, and it also does not apply to streaming output files. When the limit
is reached, additional transfers will queue up and wait before proceeding.
MAX_CONCURRENT_UPLOADS This specifies the maximum number of simultaneous transfers of input files from
the submit machine to execute machines. The limit applies to all jobs submitted from the same condor_schedd.
The default is 10. A setting of 0 means unlimited transfers. This limit currently does not apply to grid universe
jobs or standard universe jobs. When the limit is reached, additional transfers will queue up and wait before
proceeding.
FILE_TRANSFER_DISK_LOAD_THROTTLE This configures throttling of file transfers based on the disk
load generated by file transfers. The maximum number of concurrent file transfers is specified by
MAX_CONCURRENT_UPLOADS and MAX_CONCURRENT_DOWNLOADS. Throttling will dynamically reduce
the level of concurrency further to attempt to prevent disk load from exceeding the specified level. Disk load
is computed as the average number of file transfer processes conducting read/write operations at the same time.
The throttle may be specified as a single floating point number or as a range. Syntax for the range is the smaller
number followed by 1 or more spaces or tabs, the string "to", 1 or more spaces or tabs, and then the larger
number. Example:
FILE_TRANSFER_DISK_LOAD_THROTTLE = 5 to 6.5
If only a single number is provided, this serves as the upper limit, and the lower limit is set to 90% of the upper
limit. When the disk load is above the upper limit, no new transfers will be started. When between the lower
and upper limits, new transfers will only be started to replace ones that finish. There is no default value if this
variable is not defined.
FILE_TRANSFER_DISK_LOAD_THROTTLE_WAIT_BETWEEN_INCREMENTS This
rarely
configured variable sets the waiting period between increments to the concurrency level set by
FILE_TRANSFER_DISK_LOAD_THROTTLE. The default is 1 minute. A value that is too short risks
starting too many transfers before their effect on the disk load becomes apparent.
FILE_TRANSFER_DISK_LOAD_THROTTLE_SHORT_HORIZON This rarely configured variable specifies the
string name of the short monitoring time span to use for throttling. The named time span must exist in
TRANSFER_IO_REPORT_TIMESPANS. The default is 1m, which is 1 minute.
FILE_TRANSFER_DISK_LOAD_THROTTLE_LONG_HORIZON This rarely configured variable specifies the
string name of the long monitoring time span to use for throttling. The named time span must exist in
TRANSFER_IO_REPORT_TIMESPANS. The default is 5m, which is 5 minutes.
TRANSFER_QUEUE_USER_EXPR This rarely configured expression specifies the user name to be used for scheduling purposes in the file transfer queue. The scheduler attempts to give equal weight to each user when there
are multiple jobs waiting to transfer files within the limits set by MAX_CONCURRENT_UPLOADS and/or
MAX_CONCURRENT_DOWNLOADS. When choosing a new job to allow to transfer, the first job belonging
HTCondor Version 8.6.4 Manual
282
3.5.10. condor_schedd Configuration File Entries
to the transfer queue user who has least number of active transfers will be selected. In case of a tie, the
user who has least recently been given an opportunity to start a transfer will be selected. By default, a
transfer queue user is identified as the job owner. A different user name may be specified by configuring
TRANSFER_QUEUE_USER_EXPR to a string expression that is evaluated in the context of the job ad. For
example, if this expression were set to a name that is the same for all jobs, file transfers would be scheduled in
first-in-first-out order rather than equal share order. Note that the string produced by this expression is used as
a prefix in the ClassAd attributes for per-user file transfer I/O statistics that are published in the condor_schedd
ClassAd.
MAX_TRANSFER_INPUT_MB This integer expression specifies the maximum allowed total size in MiB of the input
files that are transferred for a job. This expression does not apply to grid universe, standard universe, or files
transferred via file transfer plug-ins. The expression may refer to attributes of the job. The special value -1
indicates no limit. The default value is -1. The job may override the system setting by specifying its own limit
using the MaxTransferInputMB attribute. If the observed size of all input files at submit time is larger than
the limit, the job will be immediately placed on hold with a HoldReasonCode value of 32. If the job passes
this initial test, but the size of the input files increases or the limit decreases so that the limit is violated, the job
will be placed on hold at the time when the file transfer is attempted.
MAX_TRANSFER_OUTPUT_MB This integer expression specifies the maximum allowed total size in MiB of the
output files that are transferred for a job. This expression does not apply to grid universe, standard universe, or
files transferred via file transfer plug-ins. The expression may refer to attributes of the job. The special value
-1 indicates no limit. The default value is -1. The job may override the system setting by specifying its own
limit using the MaxTransferOutputMB attribute. If the total size of the job’s output files to be transferred is
larger than the limit, the job will be placed on hold with a HoldReasonCode value of 33. The output will be
transferred up to the point when the limit is hit, so some files may be fully transferred, some partially, and some
not at all.
MAX_TRANSFER_QUEUE_AGE The number of seconds after which an aged and queued transfer may be dequeued
from the transfer queue, as it is presumably hung. Defaults to 7200 seconds, which is 120 minutes.
TRANSFER_IO_REPORT_INTERVAL The sampling interval in seconds for collecting I/O statistics for file transfer.
The default is 10 seconds. To provide sufficient resolution, the sampling interval should be small compared to
the smallest time span that is configured in TRANSFER_IO_REPORT_TIMESPANS. The shorter the sampling
interval, the more overhead of data collection, which may slow down the condor_schedd. See page 1046 for a
description of the published attributes.
TRANSFER_IO_REPORT_TIMESPANS A string that specifies a list of time spans over which I/O statistics are
reported, using exponential moving averages (like the 1m, 5m, and 15m load averages in Unix). Each entry in
the list consists of a label followed by a colon followed by the number of seconds over which the named time
span should extend. The default is 1m:60 5m:300 1h:3600 1d:86400. To provide sufficient resolution,
the smallest reported time span should be large compared to the sampling interval, which is configured by
TRANSFER_IO_REPORT_INTERVAL. See page 1046 for a description of the published attributes.
SCHEDD_QUERY_WORKERS This specifies the maximum number of concurrent sub-processes that the condor_schedd will spawn to handle queries. The setting is ignored in Windows. In Unix, the default is 8. If
the limit is reached, the next query will be handled in the condor_schedd’s main process.
HTCondor Version 8.6.4 Manual
283
3.5.10. condor_schedd Configuration File Entries
CONDOR_Q_USE_V3_PROTOCOL A boolean value that, when True, causes the condor_schedd to use an algorithm
that responds to condor_q requests by not forking itself to handle each request. It instead handles the requests
in a non-blocking way. The default value is True.
CONDOR_Q_DASH_BATCH_IS_DEFAULT A boolean value that, when True, causes condor_q to print the -batch
output unless the -nobatch option is used or the other arguments to condor_q are incompatible with batch mode.
For instance -long is incompatible with -batch. The default value is True.
CONDOR_Q_ONLY_MY_JOBS A boolean value that, when True, causes condor_q to request that only the current
user’s jobs be queried unless the current user is a queue superuser. It also causes the condor_schedd to honor
that request. The default value is True. A value of False in either condor_q or the condor_schedd will result
in the old behavior of querying all jobs.
SCHEDD_INTERVAL This macro determines the maximum interval for both how often the condor_schedd sends a
ClassAd update to the condor_collector and how often the condor_schedd daemon evaluates jobs. It is defined
in terms of seconds and defaults to 300 (every 5 minutes).
ABSENT_SUBMITTER_LIFETIME This macro determines the maximum time that the condor_schedd will remember a submitter after the last job for that submitter leaves the queue. It is defined in terms of seconds and defaults
to 1 week.
ABSENT_SUBMITTER_UPDATE_RATE This macro can be used to set the maximum rate at which the condor_schedd sends updates to the condor_collector for submitters that have no jobs in the queue. It is defined in
terms of seconds and defaults to 300 (every 5 minutes).
WINDOWED_STAT_WIDTH The number of seconds that forms a time window within which performance statistics
of the condor_schedd daemon are calculated. Defaults to 300 seconds.
SCHEDD_INTERVAL_TIMESLICE The bookkeeping done by the condor_schedd takes more time when there are
large numbers of jobs in the job queue. However, when it is not too expensive to do this bookkeeping, it is best
to keep the collector up to date with the latest state of the job queue. Therefore, this macro is used to adjust
the bookkeeping interval so that it is done more frequently when the cost of doing so is relatively small, and
less frequently when the cost is high. The default is 0.05, which means the schedd will adapt its bookkeeping
interval to consume no more than 5% of the total time available to the schedd. The lower bound is configured by
SCHEDD_MIN_INTERVAL (default 5 seconds), and the upper bound is configured by SCHEDD_INTERVAL
(default 300 seconds).
JOB_START_COUNT This macro works together with the JOB_START_DELAY macro to throttle job starts. The
default and minimum values for this integer configuration variable are both 1.
JOB_START_DELAY This integer-valued macro works together with the JOB_START_COUNT macro to throttle job starts. The condor_schedd daemon starts $(JOB_START_COUNT) jobs at a time, then delays for
$(JOB_START_DELAY) seconds before starting the next set of jobs. This delay prevents a sudden, large
load on resources required by the jobs during their start up phase. The resulting job start rate averages as fast
as ($(JOB_START_COUNT)/$(JOB_START_DELAY)) jobs/second. This setting is defined in terms of seconds and defaults to 0, which means jobs will be started as fast as possible. If you wish to throttle the rate of
specific types of jobs, you can use the job attribute NextJobStartDelay.
MAX_NEXT_JOB_START_DELAY An integer number of seconds representing the maximum allowed value of the
job ClassAd attribute NextJobStartDelay. It defaults to 600, which is 10 minutes.
HTCondor Version 8.6.4 Manual
284
3.5.10. condor_schedd Configuration File Entries
JOB_STOP_COUNT An integer value representing the number of jobs operated on at one time by the condor_schedd
daemon, when throttling the rate at which jobs are stopped via condor_rm, condor_hold, or condor_vacate_job.
The default and minimum values are both 1. This variable is ignored for grid and scheduler universe jobs.
JOB_STOP_DELAY An integer value representing the number of seconds delay utilized by the condor_schedd
daemon, when throttling the rate at which jobs are stopped via condor_rm, condor_hold, or condor_vacate_job. The condor_schedd daemon stops $(JOB_STOP_COUNT) jobs at a time, then delays for
$(JOB_STOP_DELAY) seconds before stopping the next set of jobs. This delay prevents a sudden, large load
on resources required by the jobs when they are terminating. The resulting job stop rate averages as fast as
JOB_STOP_COUNT/JOB_STOP_DELAY jobs per second. This configuration variable is also used during the
graceful shutdown of the condor_schedd daemon. During graceful shutdown, this macro determines the wait
time in between requesting each condor_shadow daemon to gracefully shut down. The default value is 0, which
means jobs will be stopped as fast as possible. This variable is ignored for grid and scheduler universe jobs.
JOB_IS_FINISHED_COUNT An integer value representing the number of jobs that the condor_schedd will let
permanently leave the job queue each time that it examines the jobs that are ready to do so. The default value is
1.
JOB_IS_FINISHED_INTERVAL The condor_schedd maintains a list of jobs that are ready to permanently leave
the job queue, for example, when they have completed or been removed. This integer-valued macro specifies a
delay in seconds between instances of taking jobs permanently out of the queue. The default value is 0, which
tells the condor_schedd to not impose any delay.
ALIVE_INTERVAL An initial value for an integer number of seconds defining how often the condor_schedd sends a
UDP keep alive message to any condor_startd it has claimed. When the condor_schedd claims a condor_startd,
the condor_schedd tells the condor_startd how often it is going to send these messages. The utilized interval for sending keep alive messages is the smallest of the two values ALIVE_INTERVAL and the expression
JobLeaseDuration/3, formed with the job ClassAd attribute JobLeaseDuration. The value of the interval is further constrained by the floor value of 10 seconds. If the condor_startd does not receive any of these
keep alive messages during a certain period of time (defined via MAX_CLAIM_ALIVES_MISSED, described
on page 266) the condor_startd releases the claim, and the condor_schedd no longer pays for the resource (in
terms of user priority in the system). The macro is defined in terms of seconds and defaults to 300, which is 5
minutes.
STARTD_SENDS_ALIVES Note: This setting is deprecated, and may go away in a future version of HTCondor.
This setting is mainly useful when running mixing very old condor_schedd daemons with newer pools. A
boolean value that defaults to True, causing keep alive messages to be sent from the condor_startd to the
condor_schedd by TCP during a claim. When False, the condor_schedd daemon sends keep alive signals to
the condor_startd, reversing the direction. If both condor_startd and condor_schedd daemons are HTCondor
version 7.5.4 or more recent, this variable is only used by the condor_schedd daemon. For earlier HTCondor
versions, the variable must be set to the same value, and it must be set for both daemons.
REQUEST_CLAIM_TIMEOUT This macro sets the time (in seconds) that the condor_schedd will wait for a
claim to be granted by the condor_startd. The default is 30 minutes. This is only likely to matter if
NEGOTIATOR_CONSIDER_EARLY_PREEMPTION is True, and the condor_startd has an existing claim,
and it takes a long time for the existing claim to be preempted due to MaxJobRetirementTime. Once a
request times out, the condor_schedd will simply begin the process of finding a machine for the job all over
again.
HTCondor Version 8.6.4 Manual
285
3.5.10. condor_schedd Configuration File Entries
Normally, it is not a good idea to set this to be very small, where a small value is a few minutes. Doing so can
lead to failure to preempt, because the preempting job will spend a significant fraction of its time waiting to be
re-matched. During that time, it would miss out on any opportunity to run if the job it is trying to preempt gets
out of the way.
SHADOW_SIZE_ESTIMATE The estimated private virtual memory size of each condor_shadow process in KiB.
This value is only used if RESERVED_SWAP is non-zero. The default value is 800.
SHADOW_RENICE_INCREMENT When the condor_schedd spawns a new condor_shadow, it can do so with a nicelevel. A nice-level is a Unix mechanism that allows users to assign their own processes a lower priority so that
the processes run with less priority than other tasks on the machine. The value can be any integer between 0 and
19, with a value of 19 being the lowest priority. It defaults to 0.
SCHED_UNIV_RENICE_INCREMENT Analogous
to
JOB_RENICE_INCREMENT
and
SHADOW_RENICE_INCREMENT, scheduler universe jobs can be given a nice-level. The value can be
any integer between 0 and 19, with a value of 19 being the lowest priority. It defaults to 0.
QUEUE_CLEAN_INTERVAL The condor_schedd maintains the job queue on a given machine. It does so in a persistent way such that if the condor_schedd crashes, it can recover a valid state of the job queue. The mechanism
it uses is a transaction-based log file (the job_queue.log file, not the SchedLog file). This file contains
an initial state of the job queue, and a series of transactions that were performed on the queue (such as new
jobs submitted, jobs completing, and checkpointing). Periodically, the condor_schedd will go through this log,
truncate all the transactions and create a new file with containing only the new initial state of the log. This is
a somewhat expensive operation, but it speeds up when the condor_schedd restarts since there are fewer transactions it has to play to figure out what state the job queue is really in. This macro determines how often the
condor_schedd should rework this queue to cleaning it up. It is defined in terms of seconds and defaults to
86400 (once a day).
WALL_CLOCK_CKPT_INTERVAL The job queue contains a counter for each job’s “wall clock” run time, i.e., how
long each job has executed so far. This counter is displayed by condor_q. The counter is updated when the
job is evicted or when the job completes. When the condor_schedd crashes, the run time for jobs that are
currently running will not be added to the counter (and so, the run time counter may become smaller than the
CPU time counter). The condor_schedd saves run time “checkpoints” periodically for running jobs so if the
condor_schedd crashes, only run time since the last checkpoint is lost. This macro controls how often the
condor_schedd saves run time checkpoints. It is defined in terms of seconds and defaults to 3600 (one hour). A
value of 0 will disable wall clock checkpoints.
QUEUE_ALL_USERS_TRUSTED Defaults to False. If set to True, then unauthenticated users are allowed to write to
the queue, and also we always trust whatever the Owner value is set to be by the client in the job ad. This was
added so users can continue to use the SOAP web-services interface over HTTP (w/o authenticating) to submit
jobs in a secure, controlled environment – for instance, in a portal setting.
QUEUE_SUPER_USERS A comma and/or space separated list of user names on a given machine that are given superuser access to the job queue, meaning that they can modify or delete the job ClassAds of other users. When
not on this list, users can only modify or delete their own ClassAds from the job queue. Whatever user name
corresponds with the UID that HTCondor is running as – usually user condor – will automatically be included
in this list, because that is needed for HTCondor’s proper functioning. See section 3.8.13 on UIDs in HTCondor
for more details on this. By default, the Unix user root and the Windows user administrator are given the
HTCondor Version 8.6.4 Manual
286
3.5.10. condor_schedd Configuration File Entries
287
ability to remove other user’s jobs, in addition to user condor. In addition to a single user, Unix user groups
may be specified by using a special syntax defined for this configuration variable; the syntax is the percent
character (%) followed by the user group name. All members of the user group are given super-user access.
QUEUE_SUPER_USER_MAY_IMPERSONATE A regular expression that matches the user names (that is, job owner
names) that the queue super user may impersonate when managing jobs. When not set, the default behavior is to
allow impersonation of any user who has had a job in the queue during the life of the condor_schedd. For proper
functioning of the condor_shadow, the condor_gridmanager, and the condor_job_router, this expression, if set,
must match the owner names of all jobs that these daemons will manage. Note that a regular expression that
matches only part of the user name is still considered a match. If acceptance of partial matches is not desired,
the regular expression should begin with ^ and end with $.
SYSTEM_JOB_MACHINE_ATTRS This macro specifies a space and/or comma separated list of machine attributes that should be recorded in the job ClassAd. The default attributes are Cpus and SlotWeight.
When there are multiple run attempts, history of machine attributes from previous run attempts
may be kept.
The number of run attempts to store is specified by the configuration variable
SYSTEM_JOB_MACHINE_ATTRS_HISTORY_LENGTH. A machine attribute named X will be inserted into
the job ClassAd as an attribute named MachineAttrX0. The previous value of this attribute will be named
MachineAttrX1, the previous to that will be named MachineAttrX2, and so on, up to the specified history length. A history of length 1 means that only MachineAttrX0 will be recorded. Additional attributes
to record may be specified on a per-job basis by using the job_machine_attrs submit file command. The value
recorded in the job ClassAd is the evaluation of the machine attribute in the context of the job ClassAd when the
condor_schedd daemon initiates the start up of the job. If the evaluation results in an Undefined or Error
result, the value recorded in the job ClassAd will be Undefined or Error respectively.
SYSTEM_JOB_MACHINE_ATTRS_HISTORY_LENGTH The integer number of run attempts to store in the job
ClassAd when recording the values of machine attributes listed in SYSTEM_JOB_MACHINE_ATTRS. The
default is 1. The history length may also be extended on a per-job basis by using the submit file command
job_machine_attrs_history_length. The larger of the system and per-job history lengths will be used. A
history length of 0 disables recording of machine attributes.
SCHEDD_LOCK This macro specifies what lock file should be used for access to the SchedLog file. It must be
a separate file from the SchedLog, since the SchedLog may be rotated and synchronization across log file
rotations is desired. This macro is defined relative to the $(LOCK) macro.
SCHEDD_NAME Used to give an alternative value to the Name attribute in the condor_schedd’s ClassAd.
See the description of MASTER_NAME in section 3.5.8 on page 260 for defaults and composition of valid HTCondor daemon names. Also, note that if the MASTER_NAME setting is defined for the condor_master that
spawned a given condor_schedd, that name will take precedence over whatever is defined in SCHEDD_NAME.
SCHEDD_ATTRS This macro is described in section 3.5.4 as <SUBSYS>_ATTRS.
SCHEDD_DEBUG This macro (and other settings related to debug logging in the condor_schedd) is described in
section 3.5.3 as <SUBSYS>_DEBUG.
SCHEDD_ADDRESS_FILE This macro is described in section 3.5.4 as <SUBSYS>_ADDRESS_FILE.
SCHEDD_EXECUTE A directory to use as a temporary sandbox for local universe jobs.
$(SPOOL)/execute.
HTCondor Version 8.6.4 Manual
Defaults to
3.5.10. condor_schedd Configuration File Entries
FLOCK_NEGOTIATOR_HOSTS Defines a comma and/or space separated list of condor_negotiator host names for
pools in which the condor_schedd should attempt to run jobs. If not set, the condor_schedd will query the condor_collector daemons for the addresses of the condor_negotiator daemons. If set, then the condor_negotiator
daemons must be specified in order, corresponding to the list set by FLOCK_COLLECTOR_HOSTS. In the
typical case, where each pool has the condor_collector and condor_negotiator running on the same machine,
$(FLOCK_NEGOTIATOR_HOSTS) should have the same definition as $(FLOCK_COLLECTOR_HOSTS).
This configuration value is also typically used as a macro for adding the condor_negotiator to the relevant
authorization lists.
FLOCK_COLLECTOR_HOSTS This macro defines a list of collector host names (not including the local
$(COLLECTOR_HOST) machine) for pools in which the condor_schedd should attempt to run jobs. Hosts
in the list should be in order of preference. The condor_schedd will only send a request to a central
manager in the list if the local pool and pools earlier in the list are not satisfying all the job requests.
$(HOSTALLOW_NEGOTIATOR_SCHEDD) (see section 3.5.4) must also be configured to allow negotiators
from all of the pools to contact the condor_schedd at the NEGOTIATOR authorization level. Similarly, the central managers of the remote pools must be configured to allow this condor_schedd to join the pool (this requires
ADVERTISE_SCHEDD authorization level, which defaults to WRITE).
FLOCK_INCREMENT This integer value controls how quickly flocking to various pools will occur. It defaults to
1, meaning that pools will be considered for flocking slowly. The first condor_collector daemon listed in
FLOCK_COLLECTOR_HOSTS will be considered for flocking, and then the second, and so on. A larger value
increases the number of condor_collector daemons to be considered for flocking. For example, a value of 2
will partition the FLOCK_COLLECTOR_HOSTS into sets of 2 condor_collector daemons, and each set will be
considered for flocking.
NEGOTIATE_ALL_JOBS_IN_CLUSTER If this macro is set to False (the default), when the condor_schedd fails
to start an idle job, it will not try to start any other idle jobs in the same cluster during that negotiation cycle.
This makes negotiation much more efficient for large job clusters. However, in some cases other jobs in the
cluster can be started even though an earlier job can’t. For example, the jobs’ requirements may differ, because
of different disk space, memory, or operating system requirements. Or, machines may be willing to run only
some jobs in the cluster, because their requirements reference the jobs’ virtual memory size or other attribute.
Setting this macro to True will force the condor_schedd to try to start all idle jobs in each negotiation cycle.
This will make negotiation cycles last longer, but it will ensure that all jobs that can be started will be started.
PERIODIC_EXPR_INTERVAL This macro determines the minimum period, in seconds, between evaluation of periodic job control expressions, such as periodic_hold, periodic_release, and periodic_remove, given by the user
in an HTCondor submit file. By default, this value is 60 seconds. A value of 0 prevents the condor_schedd from
performing the periodic evaluations.
MAX_PERIODIC_EXPR_INTERVAL This macro determines the maximum period, in seconds, between evaluation
of periodic job control expressions, such as periodic_hold, periodic_release, and periodic_remove, given by the
user in an HTCondor submit file. By default, this value is 1200 seconds. If HTCondor is behind on processing
events, the actual period between evaluations may be higher than specified.
PERIODIC_EXPR_TIMESLICE This macro is used to adapt the frequency with which the condor_schedd evaluates periodic job control expressions. When the job queue is very large, the cost of evaluating all of the
ClassAds is high, so in order for the condor_schedd to continue to perform well, it makes sense to evaluate
these expressions less frequently. The default time slice is 0.01, so the condor_schedd will set the interval
HTCondor Version 8.6.4 Manual
288
3.5.10. condor_schedd Configuration File Entries
289
between evaluations so that it spends only 1% of its time in this activity. The lower bound for the interval is
configured by PERIODIC_EXPR_INTERVAL (default 60 seconds) and the upper bound is configured with
MAX_PERIODIC_EXPR_INTERVAL (default 1200 seconds).
SYSTEM_PERIODIC_HOLD This expression behaves identically to the job expression periodic_hold, but it is
evaluated for every job in the queue. It defaults to False. When True, it causes the job to stop running
and go on hold. Here is an example that puts jobs on hold if they have been restarted too many times, have
an unreasonably large virtual memory ImageSize, or have unreasonably large disk usage for an invented
environment.
SYSTEM_PERIODIC_HOLD = \
(JobStatus == 1 || JobStatus == 2) && \
(JobRunCount > 10 || ImageSize > 3000000 || DiskUsage > 10000000)
SYSTEM_PERIODIC_HOLD_REASON This string expression is evaluated when the job is placed on hold due to
SYSTEM_PERIODIC_HOLD evaluating to True. If it evaluates to a non-empty string, this value is used to set
the job attribute HoldReason. Otherwise, a default description is used.
SYSTEM_PERIODIC_HOLD_SUBCODE This integer expression is evaluated when the job is placed on hold due to
SYSTEM_PERIODIC_HOLD evaluating to True. If it evaluates to a valid integer, this value is used to set the
job attribute HoldReasonSubCode. Otherwise, a default of 0 is used. The attribute HoldReasonCode is
set to 26, which indicates that the job went on hold due to a system job policy expression.
SYSTEM_PERIODIC_RELEASE This expression behaves identically to a job’s definition of a periodic_release expression in a submit description file, but it is evaluated for every job in the queue. It defaults to False. When
True, it causes a Held job to return to the Idle state. Here is an example that releases jobs from hold if they
have tried to run less than 20 times, have most recently been on hold for over 20 minutes, and have gone on
hold due to Connection timed out when trying to execute the job, because the file system containing the
job’s executable is temporarily unavailable.
SYSTEM_PERIODIC_RELEASE = \
(JobRunCount < 20 && (time() - EnteredCurrentStatus) > 1200 ) &&
(HoldReasonCode == 6 && HoldReasonSubCode == 110)
\
SYSTEM_PERIODIC_REMOVE This expression behaves identically to the job expression periodic_remove,
but it is evaluated for every job in the queue. As it is in the configuration file, it is easy for an administrator to
set a remove policy that applies to all jobs. It defaults to False. When True, it causes the job to be removed
from the queue. Here is an example that removes jobs which have been on hold for 30 days:
SYSTEM_PERIODIC_REMOVE = \
(JobStatus == 5 && time() - EnteredCurrentStatus > 3600*24*30)
SCHEDD_ASSUME_NEGOTIATOR_GONE This macro determines the period, in seconds, that the condor_schedd
will wait for the condor_negotiator to initiate a negotiation cycle before the schedd will simply try to claim
any local condor_startd. This allows for a machine that is acting as both a submit and execute node to run
jobs locally if it cannot communicate with the central manager. The default value, if not specified, is 1200 (20
minutes).
HTCondor Version 8.6.4 Manual
3.5.10. condor_schedd Configuration File Entries
GRACEFULLY_REMOVE_JOBS A boolean value that causes jobs to be gracefully removed when the default value
of True. A submit description file command want_graceful_removal overrides the value set for this configuration variable.
SCHEDD_ROUND_ATTR_<xxxx> This is used to round off attributes in the job ClassAd so that similar jobs may
be grouped together for negotiation purposes. There are two cases. One is that a percentage such as 25% is
specified. In this case, the value of the attribute named <xxxx>\ in the job ClassAd will be rounded up to the
next multiple of the specified percentage of the values order of magnitude. For example, a setting of 25% will
cause a value near 100 to be rounded up to the next multiple of 25 and a value near 1000 will be rounded up to
the next multiple of 250. The other case is that an integer, such as 4, is specified instead of a percentage. In this
case, the job attribute is rounded up to the specified number of decimal places. Replace <xxxx> with the name
of the attribute to round, and set this macro equal to the number of decimal places to round up. For example, to
round the value of job ClassAd attribute foo up to the nearest 100, set
SCHEDD_ROUND_ATTR_foo = 2
When the schedd rounds up an attribute value, it will save the raw (un-rounded) actual value in an attribute
with the same name appended with “_RAW". So in the above example, the raw value will be stored in attribute
foo_RAW in the job ClassAd. The following are set by default:
SCHEDD_ROUND_ATTR_ResidentSetSize = 25%
SCHEDD_ROUND_ATTR_ProportionalSetSizeKb = 25%
SCHEDD_ROUND_ATTR_ImageSize = 25%
SCHEDD_ROUND_ATTR_ExecutableSize = 25%
SCHEDD_ROUND_ATTR_DiskUsage = 25%
SCHEDD_ROUND_ATTR_NumCkpts = 4
Thus, an ImageSize near 100MB will be rounded up to the next multiple of 25MB. If your batch slots have less
memory or disk than the rounded values, it may be necessary to reduce the amount of rounding, because the job
requirements will not be met.
SCHEDD_BACKUP_SPOOL A boolean value that, when True, causes the condor_schedd to make a backup of the
job queue as it starts. When True, the condor_schedd creates a host-specific backup of the current spool file to
the spool directory. This backup file will be overwritten each time the condor_schedd starts. Defaults to False.
SCHEDD_PREEMPTION_REQUIREMENTS This boolean expression is utilized only for machines allocated by a
dedicated scheduler. When True, a machine becomes a candidate for job preemption. This configuration
variable has no default; when not defined, preemption will never be considered.
SCHEDD_PREEMPTION_RANK This floating point value is utilized only for machines allocated by a dedicated
scheduler. It is evaluated in context of a job ClassAd, and it represents a machine’s preference for running
a job. This configuration variable has no default; when not defined, preemption will never be considered.
ParallelSchedulingGroup For parallel jobs which must be assigned within a group of machines (and not
cross group boundaries), this configuration variable is a string which identifies a group of which this machine is
a member. Each machine within a group sets this configuration variable with a string that identifies the group.
HTCondor Version 8.6.4 Manual
290
3.5.10. condor_schedd Configuration File Entries
PER_JOB_HISTORY_DIR If set to a directory writable by the HTCondor user, when a job leaves the condor_schedd’s queue, a copy of the job’s ClassAd will be written in that directory. The files are named
history, with the job’s cluster and process number appended. For example, job 35.2 will result in a file
named history.35.2. HTCondor does not rotate or delete the files, so without an external entity to clean
the directory, it can grow very large. This option defaults to being unset. When not set, no files are written.
DEDICATED_SCHEDULER_USE_FIFO When this parameter is set to true (the default), parallel universe jobs will
be scheduled in a first-in, first-out manner. When set to false, parallel jobs are scheduled using a best-fit algorithm. Using the best-fit algorithm is not recommended, as it can cause starvation.
DEDICATED_SCHEDULER_WAIT_FOR_SPOOLER A boolean value that when True, causes the dedicated scheduler to schedule parallel universe jobs in a very strict first-in, first-out manner. When the default value of False,
parallel jobs that are being remotely submitted to a scheduler and are on hold, waiting for spooled input files to
arrive at the scheduler, will not block jobs that arrived later, but whose input files have finished spooling. When
True, jobs with larger cluster IDs, but that are in the Idle state will not be scheduled to run until all earlier jobs
have finished spooling in their input files and have been scheduled.
DEDICATED_SCHEDULER_DELAY_FACTOR Limits the cpu usage of the dedicated scheduler within the condor_schedd. The default value of 5 is the ratio of time spent not in the dedicated scheduler to the time scheduling
parallel jobs. Therefore, the default caps the time spent in the dedicated scheduler to 20%.
SCHEDD_SEND_VACATE_VIA_TCP A boolean value that defaults to True. When True, the condor_schedd daemon sends vacate signals via TCP, instead of the default UDP.
SCHEDD_CLUSTER_INITIAL_VALUE An integer that specifies the initial cluster number value to use
within a job id when a job is first submitted. If the job cluster number reaches the value set by
SCHEDD_CLUSTER_MAXIMUM_VALUE and wraps, it will be re-set to the value given by this variable. The
default value is 1.
SCHEDD_CLUSTER_INCREMENT_VALUE A positive integer that defaults to 1, representing a stride used for the
assignment of cluster numbers within a job id. When a job is submitted, the job will be assigned a job id. The
cluster number of the job id will be equal to the previous cluster number used plus the value of this variable.
SCHEDD_CLUSTER_MAXIMUM_VALUE An integer that specifies an upper bound on assigned job cluster id values.
For value M , the maximum job cluster id assigned to any job will be M − 1. When the maximum id is reached,
cluster ids will continue assignment using SCHEDD_CLUSTER_INITIAL_VALUE. The default value of this
variable is zero, which represents the behavior of having no maximum cluster id value.
Note that HTCondor does not check for nor take responsibility for duplicate cluster ids for queued jobs. If
SCHEDD_CLUSTER_MAXIMUM_VALUE is set to a non-zero value, the system administrator is responsible for
ensuring that older jobs do not stay in the queue long enough for cluster ids of new jobs to wrap around and
reuse the same id. With a low enough value, it is possible for jobs to be erroneously assigned duplicate cluster
ids, which will result in a corrupt job queue.
GRIDMANAGER_SELECTION_EXPR By default, the condor_schedd daemon will start a new condor_gridmanager
process for each discrete user that submits a grid universe job, that is, for each discrete value of job attribute
Owner across all grid universe job ClassAds. For additional isolation and/or scalability of grid job management,
additional condor_gridmanager processes can be spawned to share the load; to do so, set this variable to be a
ClassAd expression. The result of the evaluation of this expression in the context of a grid universe job ClassAd
HTCondor Version 8.6.4 Manual
291
3.5.10. condor_schedd Configuration File Entries
will be treated as a hash value. All jobs that hash to the same value via this expression will go to the same
condor_gridmanager. For instance, to spawn a separate condor_gridmanager process to manage each unique
remote site, the following expression works:
GRIDMANAGER_SELECTION_EXPR = GridResource
CKPT_SERVER_CLIENT_TIMEOUT An integer which specifies how long in seconds the condor_schedd is willing
to wait for a response from a checkpoint server before declaring the checkpoint server down. The value of 0
makes the schedd block for the operating system configured time (which could be a very long time) before the
connect() returns on its own with a connection timeout. The default value is 20.
CKPT_SERVER_CLIENT_TIMEOUT_RETRY An integer which specifies how long in seconds the condor_schedd
will ignore a checkpoint server that is deemed to be down. After this time elapses, the condor_schedd will try
again in talking to the checkpoint server. The default is 1200.
SCHEDD_JOB_QUEUE_LOG_FLUSH_DELAY An integer which specifies an upper bound in seconds on how long
it takes for changes to the job ClassAd to be visible to the HTCondor Job Router. The default is 5 seconds.
ROTATE_HISTORY_DAILY A boolean value that defaults to False. When True, the history file will be rotated
daily, in addition to the rotations that occur due to the definition of MAX_HISTORY_LOG that rotate due to size.
ROTATE_HISTORY_MONTHLY A boolean value that defaults to False. When True, the history file will be rotated
monthly, in addition to the rotations that occur due to the definition of MAX_HISTORY_LOG that rotate due to
size.
SCHEDD_COLLECT_STATS_FOR_<Name> A boolean expression that when True creates a set of condor_schedd
ClassAd attributes of statistics collected for a particular set. These attributes are named using the prefix of
<Name>. The set includes each entity for which this expression is True. As an example, assume that condor_schedd statistics attributes are to be created for only user Einstein’s jobs. Defining
SCHEDD_COLLECT_STATS_FOR_Einstein = (Owner=="einstein")
causes the creation of the set of statistics attributes with names such as EinsteinJobsCompleted and
EinsteinJobsCoredumped.
SCHEDD_COLLECT_STATS_BY_<Name> Defines a string expression. The evaluated string is used in the naming of a set of condor_schedd statistics ClassAd attributes. The naming begins with <Name>, an underscore
character, and the evaluated string. Each character not permitted in an attribute name will be converted to the
underscore character. For example,
SCHEDD_COLLECT_STATS_BY_Host = splitSlotName(RemoteHost)[1]
a set of statistics attributes will be created and kept.
If the string expression
were
to
evaluate
to
"storm.04.cs.wisc.edu",
the
names
of
two
of
these
attributes
will
be
Host_storm_04_cs_wisc_edu_JobsCompleted
and
Host_storm_04_cs_wisc_edu_JobsCoredumped.
SCHEDD_EXPIRE_STATS_BY_<Name> The number of seconds after which the condor_schedd daemon will stop
collecting and discard the statistics for a subset identified by <Name>, if no event has occurred to cause any
counter or statistic for the subset to be updated. If this variable is not defined for a particular <Name>, then the
default value will be 60*60*24*7, which is one week’s time.
HTCondor Version 8.6.4 Manual
292
3.5.10. condor_schedd Configuration File Entries
SIGNIFICANT_ATTRIBUTES A comma and/or space separated list of job ClassAd attributes that are to be added
to the list of attributes for determining the sets of jobs considered as a unit (an auto cluster) in negotiation, when
auto clustering is enabled. When defined, this list replaces the list that the condor_negotiator would define
based upon machine ClassAds.
ADD_SIGNIFICANT_ATTRIBUTES A comma and/or space separated list of job ClassAd attributes that will always
be added to the list of attributes that the condor_negotiator defines based upon machine ClassAds, for determining the sets of jobs considered as a unit (an auto cluster) in negotiation, when auto clustering is enabled.
REMOVE_SIGNIFICANT_ATTRIBUTES A comma and/or space separated list of job ClassAd attributes that are
removed from the list of attributes that the condor_negotiator defines based upon machine ClassAds, for determining the sets of jobs considered as a unit (an auto cluster) in negotiation, when auto clustering is enabled.
SCHEDD_AUDIT_LOG The path and file name of the condor_schedd log that records user-initiated commands that
modify the job queue. If not defined, there will be no condor_schedd audit log.
MAX_SCHEDD_AUDIT_LOG Controls the maximum amount of time that a log will be allowed to grow. When it is
time to rotate a log file, it will be saved to a file with an ISO timestamp suffix. The oldest rotated file receives
the file name suffix .old. The .old files are overwritten each time the maximum number of rotated files
(determined by the value of MAX_NUM_SCHEDD_AUDIT_LOG) is exceeded. A value of 0 specifies that the file
may grow without bounds. The following suffixes may be used to qualify the integer:
Sec for seconds
Min for minutes
Hr for hours
Day for days
Wk for weeks
MAX_NUM_SCHEDD_AUDIT_LOG The integer that controls the maximum number of rotations that the condor_schedd audit log is allowed to perform, before the oldest one will be rotated away. The default value is
1.
SCHEDD_USE_SLOT_WEIGHT A boolean that defaults to False. When True, the condor_schedd does use configuration variable SLOT_WEIGHT to weight running and idle job counts in the submitter ClassAd.
JOB_TRANSFORM_NAMES A comma and/or space separated list of unique names, where each is used in the
formation of a configuration variable name that will contain a set of rules governing the transformation
of jobs during submission. Each name in the list will be used in the name of configuration variable
JOB_TRANSFORM_<Name>. Transforms are applied in the order in which names appear in this list. Names
are not case-sensitive. There is no default value.
JOB_TRANSFORM_<Name> A single job transform specified as a set of transform rules in new classad syntax. The
transform rules are applied to jobs that match the transform’s Requirements expression as they are submitted.
<Name> corresponds to a name listed in JOB_TRANSFORM_NAMES. Names are not case-sensitive. There is
no default value.
HTCondor Version 8.6.4 Manual
293
3.5.11. condor_shadow Configuration File Entries
SUBMIT_REQUIREMENT_NAMES A comma and/or space separated list of unique names, where each is used in
the formation of a configuration variable name that will represent an expression evaluated to decide whether
or not to reject a job submission. Each name in the list will be used in the name of configuration variable
SUBMIT_REQUIREMENT_<Name>. There is no default value.
SUBMIT_REQUIREMENT_<Name> A boolean expression evaluated in the context of the condor_schedd daemon
ClassAd, which is the MY. name space and the job ClassAd, which is the TARGET. name space. When
False, it causes the condor_schedd to reject the submission of the job or cluster of jobs. <Name> corresponds
to a name listed in SUBMIT_REQUIREMENT_NAMES. There is no default value.
SUBMIT_REQUIREMENT_<Name>_REASON An expression that evaluates to a string, to be printed for the job
submitter when SUBMIT_REQUIREMENT_<Name> evaluates to False and the condor_schedd rejects the
job. There is no default value.
SCHEDD_RESTART_REPORT The complete path to a file that will be written with report information. The report
is written when the condor_schedd starts. It contains statistics about its attempts to reconnect to the condor_startd daemons for all jobs that were previously running. The file is updated periodically as reconnect
attempts succeed or fail. Once all attempts have completed, a copy of the report is emailed to address specified
by CONDOR_ADMIN. The default value is $(LOG)/ScheddRestartReport. If a blank value is set, then
no report is written or emailed.
JOB_SPOOL_PERMISSIONS Control the permissions on the job’s spool directory. Defaults to user which sets
permissions to 0700. Possible values are user, group, and world. If set to group, then the directory is
group-accessible, with permissions set to 0750. If set to world, then the directory is created with permissions
set to 0755.
CHOWN_JOB_SPOOL_FILES Prior to HTCondor 8.5.0 on unix, the condor_schedd would chown job files in the
SPOOL directory between the condor account and the account of the job submitter. Now, these job files are
always owned by the job submitter by default. To restore the older behavior, set this parameter to True. The
default value is False.
IMMUTABLE_JOB_ATTRS A comma and/or space separated list of attributes provided by the administrator that
cannot be changed, once they have committed values. No attributes are in this list by default.
SYSTEM_IMMUTABLE_JOB_ATTRS A predefined comma and/or space separated list of attributes that cannot be
changed, once they have committed values. The hard-coded value is: Owner ClusterId ProcId MyType
TargetType.
PROTECTED_JOB_ATTRS A comma and/or space separated list of attributes provided by the administrator that can
only be altered by the queue super-user, once they have committed values. No attributes are in this list by default.
SYSTEM_PROTECTED_JOB_ATTRS A predefined comma and/or space separated list of attributes that can only be
altered by the queue super-user, once they have committed values. The hard-code value is empty.
3.5.11 condor_shadow Configuration File Entries
These settings affect the condor_shadow.
HTCondor Version 8.6.4 Manual
294
3.5.11. condor_shadow Configuration File Entries
SHADOW_LOCK This macro specifies the lock file to be used for access to the ShadowLog file. It must be a separate
file from the ShadowLog, since the ShadowLog may be rotated and you want to synchronize access across
log file rotations. This macro is defined relative to the $(LOCK) macro.
SHADOW_DEBUG This macro (and other settings related to debug logging in the shadow) is described in section 3.5.3
as <SUBSYS>_DEBUG.
SHADOW_QUEUE_UPDATE_INTERVAL The amount of time (in seconds) between ClassAd updates that the condor_shadow daemon sends to the condor_schedd daemon. Defaults to 900 (15 minutes).
SHADOW_LAZY_QUEUE_UPDATE This boolean macro specifies if the condor_shadow should immediately update
the job queue for certain attributes (at this time, it only effects the NumJobStarts and NumJobReconnects
counters) or if it should wait and only update the job queue on the next periodic update. There is a tradeoff between performance and the semantics of these attributes, which is why the behavior is controlled by a
configuration macro. If the condor_shadow do not use a lazy update, and immediately ensures the changes to
the job attributes are written to the job queue on disk, the semantics for the attributes are very solid (there’s
only a tiny chance that the counters will be out of sync with reality), but this introduces a potentially large
performance and scalability problem for a busy condor_schedd. If the condor_shadow uses a lazy update, there
is no additional cost to the condor_schedd, but it means that condor_q will not immediately see the changes to
the job attributes, and if the condor_shadow happens to crash or be killed during that time, the attributes are
never incremented. Given that the most obvious usage of these counter attributes is for the periodic user policy
expressions (which are evaluated directly by the condor_shadow using its own copy of the job’s ClassAd,
which is immediately updated in either case), and since the additional cost for aggressive updates to a busy
condor_schedd could potentially cause major problems, the default is True to do lazy, periodic updates.
SHADOW_WORKLIFE The integer number of seconds after which the condor_shadow will exit when the current job
finishes, instead of fetching a new job to manage. Having the condor_shadow continue managing jobs helps
reduce overhead and can allow the condor_schedd to achieve higher job completion rates. The default is 3600,
one hour. The value 0 causes condor_shadow to exit after running a single job.
COMPRESS_PERIODIC_CKPT A boolean value that when True, directs the condor_shadow to instruct applications to compress periodic checkpoints when possible. The default is False.
COMPRESS_VACATE_CKPT A boolean value that when True, directs the condor_shadow to instruct applications
to compress vacate checkpoints when possible. The default is False.
PERIODIC_MEMORY_SYNC This boolean value specifies whether the condor_shadow should instruct applications
to commit dirty memory pages to swap space during a periodic checkpoint. The default is False. This
potentially reduces the number of dirty memory pages at vacate time, thereby reducing swapping activity on the
remote machine.
SLOW_CKPT_SPEED This macro specifies the speed at which vacate checkpoints should be written, in kilobytes per
second. If zero (the default), vacate checkpoints are written as fast as possible. Writing vacate checkpoints
slowly can avoid overwhelming the remote machine with swapping activity.
SHADOW_JOB_CLEANUP_RETRY_DELAY This integer specifies the number of seconds to wait between tries to
commit the final update to the job ClassAd in the condor_schedd’s job queue. The default is 30.
SHADOW_MAX_JOB_CLEANUP_RETRIES This integer specifies the number of times to try committing the final
update to the job ClassAd in the condor_schedd’s job queue. The default is 5.
HTCondor Version 8.6.4 Manual
295
3.5.12. condor_starter Configuration File Entries
SHADOW_CHECKPROXY_INTERVAL The number of seconds between tests to see if the job proxy
has been updated or should be refreshed.
The default is 600 seconds (10 minutes).
This
variable’s value should be small in comparison to the refresh interval required to keep delegated credentials from expiring (configured via DELEGATE_JOB_GSI_CREDENTIALS_REFRESH and
DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME). If this variable’s value is too small, proxy updates
could happen very frequently, potentially creating a lot of load on the submit machine.
SHADOW_RUN_UNKNOWN_USER_JOBS A boolean that defaults to False. When True, it allows the condor_shadow daemon to run jobs as user nobody when remotely submitted and from users not in the local
password file.
SHADOW_STATS_LOG The full path and file name of a file that stores TCP statistics for shadow file transfers.
(Note that the shadow logs TCP statistics to this file by default. Adding D_STATS to the SHADOW_DEBUG
value will cause TCP statistics to be logged to the normal shadow log file ($(SHADOW_LOG)).) If not
defined, SHADOW_STATS_LOG defaults to $(LOG)/XferStatsLog. Setting SHADOW_STATS_LOG to
/dev/null disables logging of shadow TCP file transfer statistics.
MAX_SHADOW_STATS_LOG Controls the maximum size in bytes or amount of time that the shadow TCP statistics
log will be allowed to grow. If not defined, MAX_SHADOW_STATS_LOG defaults to $(MAX_DEFAULT_LOG),
which currently defaults to 10 MiB in size. Values are specified with the same syntax as MAX_DEFAULT_LOG.
3.5.12 condor_starter Configuration File Entries
These settings affect the condor_starter.
EXEC_TRANSFER_ATTEMPTS Sometimes due to a router misconfiguration, kernel bug, or other network problem,
the transfer of the initial checkpoint from the submit machine to the execute machine will fail midway through.
This parameter allows a retry of the transfer a certain number of times that must be equal to or greater than 1. If
this parameter is not specified, or specified incorrectly, then it will default to three. If the transfer of the initial
executable fails every attempt, then the job goes back into the idle state until the next renegotiation cycle.
NOTE: : This parameter does not exist in the NT starter.
JOB_RENICE_INCREMENT When the condor_starter spawns an HTCondor job, it can do so with a nice-level. A
nice-level is a Unix mechanism that allows users to assign their own processes a lower priority, such that these
processes do not interfere with interactive use of the machine. For machines with lots of real memory and swap
space, such that the only scarce resource is CPU time, use this macro in conjunction with a policy that allows
HTCondor to always start jobs on the machines. HTCondor jobs would always run, but interactive response on
the machines would never suffer. A user most likely will not notice HTCondor is running jobs. See section 3.7
on Startd Policy Configuration for more details on setting up a policy for starting and stopping jobs on a given
machine.
The ClassAd expression is evaluated in the context of the job ad to an integer value, which is set by the condor_starter daemon for each job just before the job runs. The range of allowable values are integers in the range
of 0 to 19 (inclusive), with a value of 19 being the lowest priority. If the integer value is outside this range, then
on a Unix machine, a value greater than 19 is auto-decreased to 19; a value less than 0 is treated as 0. For values
outside this range, a Windows machine ignores the value and uses the default instead. The default value is 0, on
Unix, and the idle priority class on a Windows machine.
HTCondor Version 8.6.4 Manual
296
3.5.12. condor_starter Configuration File Entries
STARTER_LOCAL_LOGGING This macro determines whether the starter should do local logging to its own log file,
or send debug information back to the condor_shadow where it will end up in the ShadowLog. It defaults to
True.
STARTER_LOG_NAME_APPEND A fixed value that sets the file name extension of the local log file used by the
condor_starter daemon. Permitted values are true, false, slot, cluster and jobid. A value of false
will suppress the use of a file extension. A value of true gives the default behavior of using the slot name,
unless there is only a single slot. A value of slot uses the slot name. A value of cluster uses the job’s
ClusterId ClassAd attribute. A value of jobid uses the job’s ProcId ClassAd attribute. If cluster or
jobid are specified, the resulting log files will persist until deleted by the user, so these two options should
only be used to assist in debugging, not as permanent options.
STARTER_DEBUG This setting (and other settings related to debug logging in the starter) is described above in
section 3.5.3 as $(<SUBSYS>_DEBUG).
STARTER_UPDATE_INTERVAL An integer value representing the number of seconds between ClassAd updates
that the condor_starter daemon sends to the condor_shadow and condor_startd daemons. Defaults to 300 (5
minutes).
STARTER_UPDATE_INTERVAL_TIMESLICE A floating point value, specifying the highest fraction of time that
the condor_starter daemon should spend collecting monitoring information about the job, such as disk usage.
The default value is 0.1. If monitoring, such as checking disk usage takes a long time, the condor_starter will
monitor less frequently than specified by STARTER_UPDATE_INTERVAL.
USER_JOB_WRAPPER The full path and file name of an executable or script. If specified, HTCondor never directly
executes a job, but instead invokes this executable, allowing an administrator to specify the executable (wrapper
script) that will handle the execution of all user jobs. The command-line arguments passed to this program
will include the full path to the actual user job which should be executed, followed by all the command-line
parameters to pass to the user job. This wrapper script must ultimately replace its image with the user job; thus,
it must exec() the user job, not fork() it.
For Bourne type shells (sh, bash, ksh), the last line should be:
exec "$@"
For the C type shells (csh, tcsh), the last line should be:
exec $*:q
On Windows, the end should look like:
REM set some environment variables
set LICENSE_SERVER=192.168.1.202:5012
set MY_PARAMS=2
REM Run the actual job now
%*
HTCondor Version 8.6.4 Manual
297
3.5.12. condor_starter Configuration File Entries
This syntax is precise, to correctly handle program arguments which contain white space characters.
For Windows machines, the wrapper will either be a batch script with a file extension of .bat or .cmd, or an
executable with a file extension of .exe or .com.
If the wrapper script encounters an error as it runs, and it is unable to run the user job, it is important that the
wrapper script indicate this to the HTCondor system so that HTCondor does not assign the exit code of the
wrapper script to the job. To do this, the wrapper script should write a useful error message to the file named in
the environment variable _CONDOR_WRAPPER_ERROR_FILE, and then the wrapper script should exit with
a non-zero value. If this file is created by the wrapper script, HTCondor assumes that the wrapper script has
failed, and HTCondor will place the job back in the queue marking it as Idle, such that the job will again be run.
The condor_starter will also copy the contents of this error file to the condor_starter log, so the administrator
can debug the problem.
When a wrapper script is in use, the executable of a job submission may be specified by a relative path, as long
as the submit description file also contains:
+PreserveRelativeExecutable = True
For example,
# Let this executable be resolved by user's path in the wrapper
cmd = sleep
+PreserveRelativeExecutable = True
Without this extra attribute:
# A typical fully-qualified executable path
cmd = /bin/sleep
CGROUP_MEMORY_LIMIT_POLICY A string with possible values of hard, soft and none. The default value
is none. If set to hard, the cgroup-based limit on the total amount of physical memory used by the sum of
all processes in the job will not be allowed to exceed the limit given by the cgroup memory controller attribute
memory.limit_in_bytes. If the processes try to allocate more memory, the allocation will succeed, and virtual
memory will be allocated, but no additional physical memory will be allocated. If set to the default value soft,
the cgroup-based limit on the total amount of physical memory used by the sum of all processes in the job will
be allowed to go over the limit, if there is free memory available on the system. If set to none, no limit will be
enforced, but the memory usage of the job will be accurately measured by a cgroup.
USE_VISIBLE_DESKTOP This boolean variable is only meaningful on Windows machines. If True, HTCondor
will allow the job to create windows on the desktop of the execute machine and interact with the job. This is
particularly useful for debugging why an application will not run under HTCondor. If False, HTCondor uses
the default behavior of creating a new, non-visible desktop to run the job on. See section 7.2 for details on how
HTCondor interacts with the desktop.
STARTER_JOB_ENVIRONMENT This macro sets the default environment inherited by jobs. The syntax is the same
as the syntax for environment settings in the job submit file (see page 916). If the same environment variable is
assigned by this macro and by the user in the submit file, the user’s setting takes precedence.
HTCondor Version 8.6.4 Manual
298
3.5.12. condor_starter Configuration File Entries
JOB_INHERITS_STARTER_ENVIRONMENT A boolean value that defaults to False.
When True, it
causes jobs to inherit all environment variables from the condor_starter. When the user job and/or
STARTER_JOB_ENVIRONMENT define an environment variable that is in the condor_starter’s environment,
the setting from the condor_starter’s environment is overridden. This variable does not apply to standard universe jobs.
NAMED_CHROOT A comma and/or space separated list of full paths to one or more directories, under which the
condor_starter may run a chroot-ed job. This allows HTCondor to invoke chroot() before launching a job, if
the job requests such by defining the job ClassAd attribute RequestedChroot with a directory that matches
one in this list. There is no default value for this variable.
STARTER_UPLOAD_TIMEOUT An integer value that specifies the network communication timeout to use when
transferring files back to the submit machine. The default value is set by the condor_shadow daemon to 300.
Increase this value if the disk on the submit machine cannot keep up with large bursts of activity, such as many
jobs all completing at the same time.
ASSIGN_CPU_AFFINITY A boolean expression that defaults to False. When it evaluates to True, each job
under this condor_startd is confined to using only as many cores as the configured number of slots. When using
partitionable slots, each job will be bound to as many cores as requested by specifying request_cpus. When
True, this configuration variable overrides any specification of ENFORCE_CPU_AFFINITY. The expression
is evaluated in the context of the Job ClassAd.
ENFORCE_CPU_AFFINITY This configuration variable is replaced by ASSIGN_CPU_AFFINITY. Do not enable
this configuration variable unless using glidein or another unusual setup.
A boolean value that defaults to False. When False, the CPU affinity of processes in a job is not enforced.
When True, the processes in an HTCondor job maintain their affinity to a CPU. This means that this job will
only run on that particular CPU, even if other CPU cores are idle.
If True and SLOT<N>_CPU_AFFINITY is not set, the CPU that the job is locked to is the same as
SlotID - 1. Note that slots are numbered beginning with the value 1, while CPU cores are numbered
beginning with the value 0.
When True, more fine grained affinities may be specified with SLOT<N>_CPU_AFFINITY.
SLOT<N>_CPU_AFFINITY This configuration variable is replaced by ASSIGN_CPU_AFFINITY. Do not enable
this configuration variable unless using glidein or another unusual setup.
A comma separated list of cores to which an HTCondor job running on a specific slot given by the value of
<N> show affinity. Note that slots are numbered beginning with the value 1, while CPU cores are numbered
beginning with the value 0. This affinity list only takes effect when ENFORCE_CPU_AFFINITY = True.
ENABLE_URL_TRANSFERS A boolean value that when True causes the condor_starter for a job to invoke all
plug-ins defined by FILETRANSFER_PLUGINS to determine their capabilities for handling protocols to be
used in file transfer specified with a URL. When False, a URL transfer specified in a job’s submit description
file will cause an error issued by condor_submit. The default value is True.
FILETRANSFER_PLUGINS A comma separated list of full and absolute path and executable names for plug-ins
that will accomplish the task of doing file transfer when a job requests the transfer of an input file by specifying
a URL. See section 3.14.2 for a description of the functionality required of a plug-in.
HTCondor Version 8.6.4 Manual
299
3.5.12. condor_starter Configuration File Entries
RUN_FILETRANSFER_PLUGINS_WITH_ROOT A boolean value that affects only Unix platforms and defaults to
False, causing file transfer plug-ins invoked for a job to run with both the real and the effective UID set to user
that the job runs as. The user that the job runs as may be the job owner, nobody, or the slot user. The group
is set to primary group of the user that the job runs as, and all supplemental groups are dropped. The default
gives the behavior exhibited prior to the existence of this configuration variable. When set to True, file transfer
plug-ins are invoked with a real UID of 0 (root), provided the HTCondor daemons also run as root. The
effective UID is set to the user that the job runs as.
This configuration variable can permit plug-ins to do privileged operations, such as access a credential protected
by file system permissions. The default value is recommended unless privileged operations are required.
ENABLE_CHIRP A boolean value that defaults to True. An administrator would set the value to False to disable
Chirp remote file access from execute machines.
ENABLE_CHIRP_UPDATES A boolean value that defaults to True. If ENABLE_CHIRP is True, and
ENABLE_CHIRP_UPDATES is False, then the user job can only read job attributes from the submit side;
it cannot change them or write to the job event log. If ENABLE_CHIRP is False, the setting of this variable
does not matter, as no Chirp updates are allowed in that case.
ENABLE_CHIRP_IO A boolean value that defaults to True. If False, the file I/O condor_chirp commands are
prohibited.
ENABLE_CHIRP_DELAYED A boolean value that defaults to True. If False, the condor_chirp commands
get_job_attr_delayed and set_job_attr_delayed are prohibited.
CHIRP_DELAYED_UPDATE_PREFIX This string-valued parameter, which defaults to "Chirp*", defines
the allowed prefixes for attribute names which can be used with the condor_chirp commands
set_job_attribute_delayed and get_job_attribute_delayed. Because it must be set to the same value on both
the submit and execute nodes, it is advised that this parameter not be changed from its built-in default.
CHIRP_DELAYED_UPDATE_MAX_ATTRS This integer-valued parameter, which defaults to 100, represents the
maximum number of pending delayed chirp updates buffered by the condor_starter. If the number of unique
attributes updated by the condor_chirp command set_job_attr_delayed exceeds this parameter, it is possible
for these updates to be ignored.
USE_PSS A boolean value, that when True causes the condor_starter to measure the PSS (Proportional Set Size)
of each HTCondor job. The default value is True. When running many short lived jobs, performance problems
in the condor_procd have been observed, and a setting of False may relieve these problems.
MEMORY_USAGE_METRIC A ClassAd expression that produces an initial value for the job ClassAd attribute
MemoryUsage in jobs that are not standard universe and not vm universe.
MEMORY_USAGE_METRIC_VM A ClassAd expression that produces an initial value for the job ClassAd attribute
MemoryUsage in vm universe jobs.
STARTER_RLIMIT_AS An integer ClassAd expression, expressed in MiB, evaluated by the condor_starter to set
the RLIMIT_AS parameter of the setrlimit() system call. This limits the virtual memory size of each
process in the user job. The expression is evaluated in the context of both the machine and job ClassAds, where
the machine ClassAd is the MY. ClassAd, and the job ClassAd is the TARGET. ClassAd. There is no default
value for this variable. Since values larger than 2047 have no real meaning on 32-bit platforms, values larger
than 2047 result in no limit set on 32-bit platforms.
HTCondor Version 8.6.4 Manual
300
3.5.13. condor_submit Configuration File Entries
301
USE_PID_NAMESPACES A boolean value that, when True, enables the use of per job PID namespaces for HTCondor jobs run on Linux kernels. Defaults to False.
PER_JOB_NAMESPACES A boolean value that defaults to False. Relevant only for Linux platforms using file
system namespaces. The default value of False ensures that there will be no private mount points, because
auto mounts done by autofs would use the wrong name for private file system mounts. A True value is useful
when private file system mounts are permitted and autofs (for NFS) is not used.
DYNAMIC_RUN_ACCOUNT_LOCAL_GROUP For Windows platforms, a value that sets the local group to a group
other than the default Users for the condor-slot<X> run account. Do not place the local group name
within quotation marks.
JOB_EXECDIR_PERMISSIONS Control the permissions on the job’s scratch directory. Defaults to user which
sets permissions to 0700. Possible values are user, group, and world. If set to group, then the directory is
group-accessible, with permissions set to 0750. If set to world, then the directory is created with permissions
set to 0755.
STARTER_STATS_LOG The full path and file name of a file that stores TCP statistics for starter file transfers.
(Note that the starter logs TCP statistics to this file by default. Adding D_STATS to the STARTER_DEBUG
value will cause TCP statistics to be logged to the normal starter log file ($(STARTER_LOG)).) If not defined, STARTER_STATS_LOG defaults to $(LOG)/XferStatsLog. Setting STARTER_STATS_LOG to
/dev/null disables logging of starter TCP file transfer statistics.
MAX_STARTER_STATS_LOG Controls the maximum size in bytes or amount of time that the starter TCP
statistics log will be allowed to grow.
If not defined, MAX_STARTER_STATS_LOG defaults to
$(MAX_DEFAULT_LOG), which currently defaults to 10 MiB in size. Values are specified with the same
syntax as MAX_DEFAULT_LOG.
SINGULARITY The path to the Singularity binary. The default value is /usr/bin/singularity.
SINGULARITY_JOB A boolean value specifying whether this startd should run jobs under Singularity. The default
value is False.
SINGULARITY_IMAGE_EXPR The path to the Singularity container image file.
"SingularityImage".
The default value is
SINGULARITY_TARGET_DIR A directory within the Singularity image to which $_CONDOR_SCRATCH_DIR on
the host should be mapped. The default value is "".
SINGULARITY_BIND_EXPR A string value containing a list of bind mount specifications to be passed to Singularity. The default value is "SingularityBind".
3.5.13 condor_submit Configuration File Entries
DEFAULT_UNIVERSE The universe under which a job is executed may be specified in the submit description file.
If it is not specified in the submit description file, then this variable specifies the universe (when defined). If
the universe is not specified in the submit description file, and if this variable is not defined, then the default
universe for a job will be the vanilla universe.
HTCondor Version 8.6.4 Manual
3.5.13. condor_submit Configuration File Entries
302
JOB_DEFAULT_NOTIFICATION The default that sets email notification for jobs. This variable defaults to NEVER,
such that HTCondor will not send email about events for jobs. Possible values are NEVER, ERROR, ALWAYS,
or COMPLETE. If ALWAYS, the owner will be notified whenever the job produces a checkpoint, as well as when
the job completes. If COMPLETE, the owner will be notified when the job terminates. If ERROR, the owner will
only be notified if the job terminates abnormally, or if the job is placed on hold because of a failure, and not by
user request. If NEVER, the owner will not receive email.
JOB_DEFAULT_REQUESTMEMORY The amount of memory in MiB to acquire for a job, if the job does not specify
how much it needs using the request_memory submit command. If this variable is not defined, then the default
is defined by the expression
ifThenElse(MemoryUsage =!= UNDEFINED,MemoryUsage,(ImageSize+1023)/1024)
JOB_DEFAULT_REQUESTDISK The amount of disk in KiB to acquire for a job, if the job does not specify how
much it needs using the request_disk submit command. If the job defines the value, then that value takes
precedence. If not set, then then the default is defined as DiskUsage.
JOB_DEFAULT_REQUESTCPUS The number of CPUs to acquire for a job, if the job does not specify how many it
needs using the request_cpus submit command. If the job defines the value, then that value takes precedence.
If not set, then then the default is 1.
DEFAULT_JOB_MAX_RETRIES The default value for the maximum number of job retries, if the condor_submit
retry feature is used. (Note that this value is only relevant if either retry_until or success_exit_code is defined
in the submit file, and max_retries is not.) (See section 11 for more information.) The default value if not
defined is 10.
If you want condor_submit to automatically append an expression to the Requirements expression or Rank
expression of jobs at your site use the following macros:
APPEND_REQ_VANILLA Expression to be appended to vanilla job requirements.
APPEND_REQ_STANDARD Expression to be appended to standard job requirements.
APPEND_REQUIREMENTS Expression to be appended to any type of universe jobs.
However, if APPEND_REQ_VANILLA or APPEND_REQ_STANDARD is defined, then ignore the
APPEND_REQUIREMENTS for those universes.
APPEND_RANK Expression to be appended to job rank.
APPEND_RANK_VANILLA will override this setting if defined.
APPEND_RANK_STANDARD
or
APPEND_RANK_STANDARD Expression to be appended to standard job rank.
APPEND_RANK_VANILLA Expression to append to vanilla job rank.
The APPEND_RANK_STANDARD and APPEND_RANK_VANILLA macros
NOTE:
APPEND_PREF_STANDARD and APPEND_PREF_VANILLA in previous versions of HTCondor.
In addition, you may provide default Rank expressions if your users do not specify their own with:
HTCondor Version 8.6.4 Manual
were
called
3.5.13. condor_submit Configuration File Entries
DEFAULT_RANK Default rank expression for any job that does not specify its own rank expression in the submit
description file. There is no default value, such that when undefined, the value used will be 0.0.
DEFAULT_RANK_VANILLA Default rank for vanilla universe jobs. There is no default value, such that when undefined, the value used will be 0.0. When both DEFAULT_RANK and DEFAULT_RANK_VANILLA are defined,
the value for DEFAULT_RANK_VANILLA is used for vanilla universe jobs.
DEFAULT_RANK_STANDARD Default rank for standard universe jobs. There is no default value, such that when
undefined, the value used will be 0.0. When both DEFAULT_RANK and DEFAULT_RANK_STANDARD are
defined, the value for DEFAULT_RANK_STANDARD is used for standard universe jobs.
DEFAULT_IO_BUFFER_SIZE HTCondor keeps a buffer of recently-used data for each file an application opens.
This macro specifies the default maximum number of bytes to be buffered for each open file at the executing
machine. The condor_status buffer_size command will override this default. If this macro is undefined, a
default size of 512 KB will be used.
DEFAULT_IO_BUFFER_BLOCK_SIZE When buffering is enabled, HTCondor will attempt to consolidate small
read and write operations into large blocks. This macro specifies the default block size HTCondor will use.
The condor_status buffer_block_size command will override this default. If this macro is undefined, a
default size of 32 KB will be used.
SUBMIT_SKIP_FILECHECKS If True, condor_submit behaves as if the -disable command-line option is used.
This tells condor_submit to disable file permission checks when submitting a job for read permissions on all
input files, such as those defined by commands input and transfer_input_files, as well as write permission to
output files, such as a log file defined by log and output files defined with output or transfer_output_files.
This can significantly decrease the amount of time required to submit a large group of jobs. The default value is
False.
WARN_ON_UNUSED_SUBMIT_FILE_MACROS A boolean variable that defaults to True. When True, condor_submit performs checks on the job’s submit description file contents for commands that define a macro,
but do not use the macro within the file. A warning is issued, but job submission continues. A definition of
a new macro occurs when the lhs of a command is not a known submit command. This check may help spot
spelling errors of known submit commands.
SUBMIT_SEND_RESCHEDULE A boolean expression that when False, prevents condor_submit from automatically
sending a condor_reschedule command as it completes. The condor_reschedule command causes the condor_schedd daemon to start searching for machines with which to match the submitted jobs. When True, this
step always occurs. In the case that the machine where the job(s) are submitted is managing a huge number of
jobs (thousands or tens of thousands), this step would hurt performance in such a way that it became an obstacle
to scalability. The default value is True.
SUBMIT_ATTRS A comma-separated and/or space-separated list of ClassAd attribute names for which the attribute
and value will be inserted into all the job ClassAds that condor_submit creates. In this way, it is like the "+" syntax in a submit description file. Attributes defined in the submit description file with "+" will override attributes
defined in the configuration file with SUBMIT_ATTRS. Note that adding an attribute to a job’s ClassAd will
not function as a method for specifying default values of submit description file commands forgotten in a job’s
submit description file. The command in the submit description file results in actions by condor_submit, while
the use of SUBMIT_ATTRS adds a job ClassAd attribute at a later point in time. SUBMIT_EXPRS is a historic
setting that functions identically to SUBMIT_ATTRS. It may be removed in the future, so use SUBMIT_ATTRS.
HTCondor Version 8.6.4 Manual
303
3.5.14. condor_preen Configuration File Entries
LOG_ON_NFS_IS_ERROR A boolean value that controls whether condor_submit prohibits job submit description files with job event log files on NFS. If LOG_ON_NFS_IS_ERROR is set to True, such submit files
will be rejected. If LOG_ON_NFS_IS_ERROR is set to False, the job will be submitted. If not defined,
LOG_ON_NFS_IS_ERROR defaults to False.
SUBMIT_MAX_PROCS_IN_CLUSTER An integer value that limits the maximum number of jobs that would be
assigned within a single cluster. Job submissions that would exceed the defined value fail, issuing an error
message, and with no jobs submitted. The default value is 0, which does not limit the number of jobs assigned
a single cluster number.
ENABLE_DEPRECATION_WARNINGS A boolean value that defaults to False. When True, condor_submit issues
warnings when a job requests features that are no longer supported.
INTERACTIVE_SUBMIT_FILE The path and file name of a submit description file that condor_submit will use in
the specification of an interactive job. The default is $(RELEASE_DIR)/libexec/interactive.sub
when not defined.
3.5.14 condor_preen Configuration File Entries
These macros affect condor_preen.
PREEN_ADMIN This macro sets the e-mail address where condor_preen will send e-mail (if it is configured to send
email at all; see the entry for PREEN). Defaults to $(CONDOR_ADMIN).
VALID_SPOOL_FILES A comma or space separated list of files that condor_preen considers valid files to find
in the $(SPOOL) directory, such that condor_preen will not remove these files. There is no default value.
condor_preen will add to the list files and directories that are normally present in the $(SPOOL) directory. A
single asterisk (*) wild card character is permitted in each file item within the list.
SYSTEM_VALID_SPOOL_FILES A comma or space separated list of files that condor_preen considers valid files
to find in the $(SPOOL) directory. The default value is all files known by HTCondor to be valid. This variable
exists such that it can be queried; it should not be changed. condor_preen use it to initialize the the list files and
directories that are normally present in the $(SPOOL) directory. A single asterisk (*) wild card character is
permitted in each file item within the list.
INVALID_LOG_FILES This macro contains a (comma or space separated) list of files that condor_preen considers
invalid files to find in the $(LOG) directory. There is no default value.
3.5.15 condor_collector Configuration File Entries
These macros affect the condor_collector.
CLASSAD_LIFETIME The default maximum age in seconds for ClassAds collected by the condor_collector. ClassAds older than the maximum age are discarded by the condor_collector as stale.
HTCondor Version 8.6.4 Manual
304
3.5.15. condor_collector Configuration File Entries
If present, the ClassAd attribute ClassAdLifetime specifies the ClassAd’s lifetime in seconds.
If ClassAdLifetime is not present in the ClassAd, the condor_collector will use the value of
$(CLASSAD_LIFETIME). This variable is defined in terms of seconds, and it defaults to 900 seconds (15
minutes).
To ensure that the condor_collector does not miss any ClassAds, the frequency at which all other subsystems
that report using an update interval must be tuned. The configuration variables that set these subsystems are
•
•
•
•
•
•
•
UPDATE_INTERVAL (for the condor_startd daemon)
NEGOTIATOR_UPDATE_INTERVAL
SCHEDD_INTERVAL
MASTER_UPDATE_INTERVAL
CKPT_SERVER_INTERVAL
DEFRAG_UPDATE_INTERVAL
HAD_UPDATE_INTERVAL
MASTER_CHECK_INTERVAL This macro defines how often the collector should check for machines that have ClassAds from some daemons, but not from the condor_master (orphaned daemons) and send e-mail about it. It is
defined in seconds and defaults to 10800 (3 hours).
COLLECTOR_REQUIREMENTS A boolean expression that filters out unwanted ClassAd updates. The expression
is evaluated for ClassAd updates that have passed through enabled security authorization checks. The default
behavior when this expression is not defined is to allow all ClassAd updates to take place. If False, a ClassAd
update will be rejected.
Stronger security mechanisms are the better way to authorize or deny updates to the condor_collector. This
configuration variable exists to help those that use host-based security, and do not trust all processes that run on
the hosts in the pool. This configuration variable may be used to throw out ClassAds that should not be allowed.
For example, for condor_startd daemons that run on a fixed port, configure this expression to ensure that only
machine ClassAds advertising the expected fixed port are accepted. As a convenience, before evaluating the
expression, some basic sanity checks are performed on the ClassAd to ensure that all of the ClassAd attributes
used by HTCondor to contain IP:port information are consistent. To validate this information, the attribute to
check is TARGET.MyAddress.
CLIENT_TIMEOUT Network timeout that the condor_collector uses when talking to any daemons or tools that are
sending it a ClassAd update. It is defined in seconds and defaults to 30.
QUERY_TIMEOUT Network timeout when talking to anyone doing a query. It is defined in seconds and defaults to
60.
CONDOR_DEVELOPERS By default, HTCondor will send e-mail once per week to this address with the output of the
condor_status command, which lists how many machines are in the pool and how many are running jobs. The
default value of condor-admin@cs.wisc.edu will send this report to the Center for High Throughput Computing
at the University of Wisconsin-Madison. The Center for High Throughput Computing uses these weekly status
messages in order to have some idea as to how many HTCondor pools exist in the world. We appreciate getting
the reports, as this is one way we can convince funding agencies that HTCondor is being used in the real world.
If you do not wish this information to be sent to the Center for High Throughput Computing, explicitly set the
value to NONE to disable this feature, or replace the address with a desired location. If undefined (commented
out) in the configuration file, HTCondor follows its default behavior.
HTCondor Version 8.6.4 Manual
305
3.5.15. condor_collector Configuration File Entries
COLLECTOR_NAME This macro is used to specify a short description of your pool. It should be about 20 characters
long. For example, the name of the UW-Madison Computer Science HTCondor Pool is "UW-Madison CS".
While this macro might seem similar to MASTER_NAME or SCHEDD_NAME, it is unrelated. Those settings are
used to uniquely identify (and locate) a specific set of HTCondor daemons, if there are more than one running
on the same machine. The COLLECTOR_NAME setting is just used as a human-readable string to describe the
pool, which is included in the updates sent to the CONDOR_DEVELOPERS_COLLECTOR.
CONDOR_DEVELOPERS_COLLECTOR By default, every pool sends periodic updates to a central condor_collector
at UW-Madison with basic information about the status of the pool. Updates include only the number of total
machines, the number of jobs submitted, the number of machines running jobs, the host name of the central
manager, and the $(COLLECTOR_NAME). These updates help the Center for High Throughput Computing see
how HTCondor is being used around the world. By default, they will be sent to condor.cs.wisc.edu.
To discontinue sending updates, explicitly set this macro to NONE. If undefined or commented out in the
configuration file, HTCondor follows its default behavior.
COLLECTOR_UPDATE_INTERVAL This variable is defined in seconds and defaults to 900 (every 15 minutes). It
controls the frequency of the periodic updates sent to a central condor_collector at UW-Madison as defined by
CONDOR_DEVELOPERS_COLLECTOR.
COLLECTOR_SOCKET_BUFSIZE This specifies the buffer size, in bytes, reserved for condor_collector network
UDP sockets. The default is 10240000, or a ten megabyte buffer. This is a healthy size, even for a large pool.
The larger this value, the less likely the condor_collector will have stale information about the pool due to
dropping update packets. If your pool is small or your central manager has very little RAM, considering setting
this parameter to a lower value (perhaps 256000 or 128000).
NOTE: For some Linux distributions, it may be necessary to raise the OS’s system-wide limit for network buffer
sizes. The parameter that controls this limit is /proc/sys/net/core/rmem_max. You can see the values that the
condor_collector actually uses by enabling D_FULLDEBUG for the collector and looking at the log line that
looks like this:
Reset OS socket buffer size to 2048k (UDP), 255k (TCP).
For changes to this parameter to take effect, condor_collector must be restarted.
COLLECTOR_TCP_SOCKET_BUFSIZE This specifies the TCP buffer size, in bytes, reserved for condor_collector
network sockets. The default is 131072, or a 128 kilobyte buffer. This is a healthy size, even for a large pool.
The larger this value, the less likely the condor_collector will have stale information about the pool due to
dropping update packets. If your pool is small or your central manager has very little RAM, considering setting
this parameter to a lower value (perhaps 65536 or 32768).
NOTE: See the note for COLLECTOR_SOCKET_BUFSIZE.
KEEP_POOL_HISTORY This boolean macro is used to decide if the collector will write out statistical information
about the pool to history files. The default is False. The location, size, and frequency of history logging is
controlled by the other macros.
POOL_HISTORY_DIR This macro sets the name of the directory where the history files reside (if history logging is
enabled). The default is the SPOOL directory.
POOL_HISTORY_MAX_STORAGE This macro sets the maximum combined size of the history files. When the size
of the history files is close to this limit, the oldest information will be discarded. Thus, the larger this parameter’s
value is, the larger the time range for which history will be available. The default value is 10000000 (10 MB).
HTCondor Version 8.6.4 Manual
306
3.5.15. condor_collector Configuration File Entries
POOL_HISTORY_SAMPLING_INTERVAL This macro sets the interval, in seconds, between samples for history
logging purposes. When a sample is taken, the collector goes through the information it holds, and summarizes
it. The information is written to the history file once for each 4 samples. The default (and recommended) value
is 60 seconds. Setting this macro’s value too low will increase the load on the collector, while setting it to high
will produce less precise statistical information.
COLLECTOR_DAEMON_STATS A boolean value that controls whether or not the condor_collector daemon keeps
update statistics on incoming updates. The default value is True. If enabled, the condor_collector will insert
several attributes into the ClassAds that it stores and sends. ClassAds without the UpdateSequenceNumber
and DaemonStartTime attributes will not be counted, and will not have attributes inserted (all modern HTCondor daemons which publish ClassAds publish these attributes).
The attributes inserted are UpdatesTotal, UpdatesSequenced, and UpdatesLost. UpdatesTotal
is the total number of updates (of this ClassAd type) the condor_collector has received from this host.
UpdatesSequenced is the number of updates that the condor_collector could have as lost. In particular, for
the first update from a daemon, it is impossible to tell if any previous ones have been lost or not. UpdatesLost
is the number of updates that the condor_collector has detected as being lost. See page 1056 for more information on the added attributes.
COLLECTOR_STATS_SWEEP This value specifies the number of seconds between sweeps of the condor_collector’s
per-daemon update statistics. Records for daemons which have not reported in this amount of time are purged
in order to save memory. The default is two days. It is unlikely that you would ever need to adjust this.
COLLECTOR_DAEMON_HISTORY_SIZE This variable controls the size of the published update history that
the condor_collector inserts into the ClassAds it stores and sends. The default value is 128, which
means that history is stored and published for the latest 128 updates. This variable’s value is ignored, if
COLLECTOR_DAEMON_STATS is not enabled.
If the value is a non-zero one, the condor_collector will insert attribute UpdatesHistory into the ClassAd
(similar to UpdatesTotal). AttrUpdatesHistory is a hexadecimal string which represents a bitmap of the last
COLLECTOR_DAEMON_HISTORY_SIZE updates. The most significant bit (MSB) of the bitmap represents
the most recent update, and the least significant bit (LSB) represents the least recent. A value of zero means that
the update was not lost, and a value of 1 indicates that the update was detected as lost.
For example, if the last update was not lost, the previous was lost, and the previous two not, the bitmap would
be 0100, and the matching hex digit would be "4". Note that the MSB can never be marked as lost because its loss can only be detected by a non-lost update (a gap is found in the sequence numbers). Thus,
UpdatesHistory = "0x40" would be the history for the last 8 updates. If the next updates are all successful, the values published, after each update, would be: 0x20, 0x10, 0x08, 0x04, 0x02, 0x01, 0x00.
See page 1056 for more information on the added attribute.
COLLECTOR_CLASS_HISTORY_SIZE This variable controls the size of the published update history that the condor_collector inserts into the condor_collector ClassAds it produces. The default value is zero.
If this variable has a non-zero value, the condor_collector will insert UpdatesClassHistory into the condor_collector ClassAd (similar to UpdatesHistory). These are added per class of ClassAd, however. The
classes refer to the type of ClassAds. Additionally, there is a Total class created, which represents the history of
all ClassAds that this condor_collector receives.
Note that the condor_collector always publishes Lost, Total and Sequenced counts for all ClassAd classes. This
is similar to the statistics gathered if COLLECTOR_DAEMON_STATS is enabled.
HTCondor Version 8.6.4 Manual
307
3.5.15. condor_collector Configuration File Entries
COLLECTOR_QUERY_WORKERS This variable sets the maximum number of worker processes that the condor_collector can have. When receiving a query request, the Unix condor_collector will fork() a new process
to handle the query, freeing the main process to handle other requests. When the number of outstanding worker
processes reaches this maximum, the request is handled by the main process. This variable is ignored on Windows, and its default value is zero. The default configuration, however, has a value of 2.
COLLECTOR_DEBUG This macro (and other macros related to debug logging in the condor_collector is described in
section 3.5.3 as <SUBSYS>_DEBUG.
CONDOR_VIEW_CLASSAD_TYPES Provides the ClassAd types that will be forwarded to the
CONDOR_VIEW_HOST. The ClassAd types can be found with condor_status -any. The default forwarding
behavior of the condor_collector is equivalent to
CONDOR_VIEW_CLASSAD_TYPES=Machine,Submitter
There is no default value for this variable.
COLLECTOR_FORWARD_FILTERING When this boolean variable is set to True, Machine and Submitter ad updates are not forwarded to the CONDOR_VIEW_HOST if certain attributes are unchanged from the previous
update of the ad. The default is False, meaning all updates are forwarded.
COLLECTOR_FORWARD_WATCH_LIST When COLLECTOR_FORWARD_FILTERING is set to True, this variable provides the list of attributes that controls whether a Machine or Submitter ad update is forwarded to the
CONDOR_VIEW_HOST. If all attributes in this list are unchanged from the previous update, then the new update
is not forwarded. The default value is State,Cpus,Memory,IdleJobs.
COLLECTOR_FORWARD_INTERVAL When COLLECTOR_FORWARD_FILTERING is set to True, this variable
limits how long forwarding of updates for a given ad can be filtered before an update must be forwarded. The
default is one third of CLASSAD_LIFETIME.
The following macros control where, when, and for how long HTCondor persistently stores absent ClassAds. See
section 3.12.2 on page 474 for more details.
ABSENT_REQUIREMENTS A boolean expression evaluated by the condor_collector when a machine ClassAd would
otherwise expire. If True, the ClassAd instead becomes absent. If not defined, the implementation will behave
as if False, and no absent ClassAds will be stored.
ABSENT_EXPIRE_ADS_AFTER The integer number of seconds after which the condor_collector forgets about an
absent ClassAd. If 0, the ClassAds persist forever. Defaults to 30 days.
COLLECTOR_PERSISTENT_AD_LOG The full path and file name of a file that stores machine ClassAds for every
hibernating or absent machine. This forms a persistent storage of these ClassAds, in case the condor_collector
daemon crashes.
To avoid condor_preen removing this log, place it in a directory other than the directory defined by $(SPOOL).
Alternatively, if this log file is to go in the directory defined by $(SPOOL), add the file to the list given by
VALID_SPOOL_FILES.
This configuration variable replaces OFFLINE_LOG, which is no longer used.
HTCondor Version 8.6.4 Manual
308
3.5.16. condor_negotiator Configuration File Entries
EXPIRE_INVALIDATED_ADS A boolean value that defaults to False. When True, causes all invalidated ClassAds to be treated as if they expired. This permits invalidated ClassAds to be marked absent, as defined in
section 3.12.2.
3.5.16 condor_negotiator Configuration File Entries
These macros affect the condor_negotiator.
NEGOTIATOR_INTERVAL Sets how often the condor_negotiator starts a negotiation cycle. It is defined in seconds
and defaults to 60 (1 minute).
NEGOTIATOR_UPDATE_INTERVAL This macro determines how often the condor_negotiator daemon sends a
ClassAd update to the condor_collector. It is defined in seconds and defaults to 300 (every 5 minutes).
NEGOTIATOR_CYCLE_DELAY An integer value that represents the minimum number of seconds that must pass
before a new negotiation cycle may start. The default value is 20. NEGOTIATOR_CYCLE_DELAY is intended
only for use by HTCondor experts.
NEGOTIATOR_TIMEOUT Sets the timeout that the negotiator uses on its network connections to the condor_schedd
and condor_startds. It is defined in seconds and defaults to 30.
NEGOTIATION_CYCLE_STATS_LENGTH Specifies how many recent negotiation cycles should be included in the
history that is published in the condor_negotiator’s ad. The default is 3 and the maximum allowed value is
100. Setting this value to 0 disables publication of negotiation cycle statistics. The statistics about recent
cycles are stored in several attributes per cycle. Each of these attribute names will have a number appended
to it to indicate how long ago the cycle happened, for example: LastNegotiationCycleDuration0,
LastNegotiationCycleDuration1, LastNegotiationCycleDuration2, . . .. The attribute
numbered 0 applies to the most recent negotiation cycle. The attribute numbered 1 applies to the next most
recent negotiation cycle, and so on. See page 1049 for a list of attributes that are published.
PRIORITY_HALFLIFE This macro defines the half-life of the user priorities. See section 2.7.2 on User Priorities
for details. It is defined in seconds and defaults to 86400 (1 day).
DEFAULT_PRIO_FACTOR Sets the priority factor for local users as they first submit jobs, as described in section 3.6.
Defaults to 1000.
NICE_USER_PRIO_FACTOR Sets the priority factor for nice users, as described in section 3.6. Defaults to
10000000000.
REMOTE_PRIO_FACTOR Defines the priority factor for remote users, which are those users who who do not belong
to the local domain. See section 3.6 for details. Defaults to 10000000.
ACCOUNTANT_LOCAL_DOMAIN Describes the local UID domain. This variable is used to decide if a user is local
or remote. A user is considered to be in the local domain if their UID domain matches the value of this variable.
Usually, this variable is set to the local UID domain. If not defined, all users are considered local.
HTCondor Version 8.6.4 Manual
309
3.5.16. condor_negotiator Configuration File Entries
MAX_ACCOUNTANT_DATABASE_SIZE This macro defines the maximum size (in bytes) that the accountant
database log file can reach before it is truncated (which re-writes the file in a more compact format). If, after truncating, the file is larger than one half the maximum size specified with this macro, the maximum size
will be automatically expanded. The default is 1 megabyte (1000000).
NEGOTIATOR_DISCOUNT_SUSPENDED_RESOURCES This macro tells the negotiator to not count resources that
are suspended when calculating the number of resources a user is using. Defaults to false, that is, a user is still
charged for a resource even when that resource has suspended the job.
NEGOTIATOR_SOCKET_CACHE_SIZE This macro defines the maximum number of sockets that the condor_negotiator keeps in its open socket cache. Caching open sockets makes the negotiation protocol more
efficient by eliminating the need for socket connection establishment for each negotiation cycle. The default is
currently 16. To be effective, this parameter should be set to a value greater than the number of condor_schedds
submitting jobs to the negotiator at any time. If you lower this number, you must run condor_restart and not
just condor_reconfig for the change to take effect.
NEGOTIATOR_INFORM_STARTD Boolean setting that controls if the condor_negotiator should inform the condor_startd when it has been matched with a job. The default is False. When this is set to the default value of
False, the condor_startd will never enter the Matched state, and will go directly from Unclaimed to Claimed.
Because this notification is done via UDP, if a pool is configured so that the execute hosts do not create UDP
command sockets (see the WANT_UDP_COMMAND_SOCKET setting described in section 3.5.2 on page 232 for
details), the condor_negotiator should be configured not to attempt to contact these condor_startd daemons by
using the default value.
NEGOTIATOR_PRE_JOB_RANK Resources that match a request are first sorted by this expression. If there are any
ties in the rank of the top choice, the top resources are sorted by the user-supplied rank in the job ClassAd, then
by NEGOTIATOR_POST_JOB_RANK, then by PREEMPTION_RANK (if the match would cause preemption
and there are still any ties in the top choice). MY refers to attributes of the machine ClassAd and TARGET refers
to the job ClassAd. The purpose of the pre job rank is to allow the pool administrator to override any other
rankings, in order to optimize overall throughput. For example, it is commonly used to minimize preemption,
even if the job rank prefers a machine that is busy. If explicitly set to be undefined, this expression has no effect
on the ranking of matches. The default value prefers to match multi-core jobs to dynamic slots in a best fit
manner:
NEGOTIATOR_PRE_JOB_RANK = (10000000 * My.Rank) + \
(1000000 * (RemoteOwner =?= UNDEFINED)) - (100000 * Cpus) - Memory
NEGOTIATOR_POST_JOB_RANK Resources
that
match
a
request
are
first
sorted
by
NEGOTIATOR_PRE_JOB_RANK. If there are any ties in the rank of the top choice, the top resources
are sorted by the user-supplied rank in the job ClassAd, then by NEGOTIATOR_POST_JOB_RANK, then by
PREEMPTION_RANK (if the match would cause preemption and there are still any ties in the top choice). MY.
refers to attributes of the machine ClassAd and TARGET. refers to the job ClassAd. The purpose of the post
job rank is to allow the pool administrator to choose between machines that the job ranks equally. The default
value is
NEGOTIATOR_POST_JOB_RANK = \
(RemoteOwner =?= UNDEFINED) * \
(ifThenElse(isUndefined(KFlops), 1000, Kflops) - \
SlotID - 1.0e10*(Offline=?=True))
HTCondor Version 8.6.4 Manual
310
3.5.16. condor_negotiator Configuration File Entries
PREEMPTION_REQUIREMENTS When considering user priorities, the negotiator will not preempt a job running on
a given machine unless this expression evaluates to True, and the owner of the idle job has a better priority
than the owner of the running job. The PREEMPTION_REQUIREMENTS expression is evaluated within the
context of the candidate machine ClassAd and the candidate idle job ClassAd; thus the MY scope prefix refers to
the machine ClassAd, and the TARGET scope prefix refers to the ClassAd of the idle (candidate) job. There is
no direct access to the currently running job, but attributes of the currently running job that need to be accessed
in PREEMPTION_REQUIREMENTS can be placed in the machine ClassAd using STARTD_JOB_EXPRS.
If not explicitly set in the HTCondor configuration file, the default value for this expression is False.
PREEMPTION_REQUIREMENTS should include the term (SubmitterGroup =?= RemoteGroup), if a
preemption policy that respects group quotas is desired. Note that this variable does not influence other potential
causes of preemption, such as the RANK of the condor_startd, or PREEMPT expressions. See section 3.7.1 for a
general discussion of limiting preemption.
PREEMPTION_REQUIREMENTS_STABLE A boolean value that defaults to True, implying that all attributes utilized to define the PREEMPTION_REQUIREMENTS variable will not change within a negotiation period time
interval. If utilized attributes will change during the negotiation period time interval, then set this variable to
False.
PREEMPTION_RANK Resources that match a request are first sorted by NEGOTIATOR_PRE_JOB_RANK. If there
are any ties in the rank of the top choice, the top resources are sorted by the user-supplied rank in the job ClassAd,
then by NEGOTIATOR_POST_JOB_RANK, then by PREEMPTION_RANK (if the match would cause preemption and there are still any ties in the top choice). MY refers to attributes of the machine ClassAd and TARGET
refers to the job ClassAd. This expression is used to rank machines that the job and the other negotiation expressions rank the same. For example, if the job has no preference, it is usually preferable to preempt a job with
a small ImageSize instead of a job with a large ImageSize. The default value first considers the user’s
priority and chooses the user with the worst priority. Then, among the running jobs of that user, it chooses the
job with the least accumulated run time:
PREEMPTION_RANK = (RemoteUserPrio * 1000000) - \
ifThenElse(isUndefined(TotalJobRunTime), 0, TotalJobRunTime)
PREEMPTION_RANK_STABLE A boolean value that defaults to True, implying that all attributes utilized to define
the PREEMPTION_RANK variable will not change within a negotiation period time interval. If utilized attributes
will change during the negotiation period time interval, then set this variable to False.
NEGOTIATOR_SLOT_CONSTRAINT An expression which constrains which machine ClassAds are fetched from
the condor_collector by the condor_negotiator during a negotiation cycle.
NEGOTIATOR_TRIM_SHUTDOWN_THRESHOLD This setting is not likely to be customized, except perhaps within
a glidein setting. An integer expression that evaluates to a value within the context of the condor_negotiator
ClassAd, with a default value of 0. When this expression evaluates to an integer X greater than 0, the condor_negotiator will not make matches to machines that contain the ClassAd attribute DaemonShutdown
which evaluates to True, when that shut down time is X seconds into the future. The idea here is a mechanism to prevent matching with machines that are quite close to shutting down, since the match would likely be
a waste of time.
NEGOTIATOR_SLOT_POOLSIZE_CONSTRAINT or GROUP_DYNAMIC_MACH_CONSTRAINT This
optional
expression specifies which machine ClassAds should be counted when computing the size of the pool. It
applies both for group quota allocation and when there are no groups. The default is to count all machine
ClassAds. When extra slots exist for special purposes, as, for example, suspension slots or file transfer slots,
HTCondor Version 8.6.4 Manual
311
3.5.16. condor_negotiator Configuration File Entries
312
this expression can be used to inform the condor_negotiator that only normal slots should be counted when
computing how big each group’s share of the pool should be.
The name NEGOTIATOR_SLOT_POOLSIZE_CONSTRAINT replaces GROUP_DYNAMIC_MACH_CONSTRAINT
as of HTCondor version 7.7.3. Using the older name causes a warning to be logged, although the behavior is
unchanged.
NEGOTIATOR_DEBUG This macro (and other settings related to debug logging in the negotiator) is described in
section 3.5.3 as <SUBSYS>_DEBUG.
NEGOTIATOR_MAX_TIME_PER_SUBMITTER The maximum number of seconds the condor_negotiator will
spend with each individual submitter during one negotiation cycle. Once this time limit has been reached,
the condor_negotiator will skip over requests from this submitter until the next negotiation cycle. It defaults to
the number of seconds in one year.
NEGOTIATOR_MAX_TIME_PER_SCHEDD The maximum number of seconds the condor_negotiator will spend
with each individual condor_schedd during one negotiation cycle. Once this time limit has been reached, the
condor_negotiator will skip over requests from this condor_schedd until the next negotiation cycle. It defaults
to the number of seconds in one year.
NEGOTIATOR_MAX_TIME_PER_CYCLE The maximum number of seconds the condor_negotiator will spend in
total across all submitters during one negotiation cycle. Once this time limit has been reached, the condor_negotiator will skip over requests from all submitters until the next negotiation cycle. It defaults to the
number of seconds in one year.
NEGOTIATOR_MAX_TIME_PER_PIESPIN The maximum number of seconds the condor_negotiator will spend
with a submitter in one pie spin. A negotiation cycle is composed of at least one pie spin, possibly more,
depending on whether there are still machines left over after computing fair shares and negotiating with each
submitter. By limiting the maximum length of a pie spin or the maximum time per submitter per negotiation
cycle, the condor_negotiator is protected against spending a long time talking to one submitter, for example
someone with a very slow condor_schedd daemon. But, this can result in unfair allocation of machines or some
machines not being allocated at all. See section 3.6.6 on page 365 for a description of a pie slice. It defaults to
the number of seconds in one year.
USE_RESOURCE_REQUEST_COUNTS A boolean value that defaults to True. When True, the latency of negotiation will be reduced when there are many jobs next to each other in the queue with the same auto cluster, and
many matches are being made. When True, the condor_schedd tells the condor_negotiator to send X matches
at a time, where X equals number of consecutive jobs in the queue within the same auto cluster.
NEGOTIATOR_RESOURCE_REQUEST_LIST_SIZE An integer tuning parameter used by the condor_negotiator
to control the number of resource requests fetched from a condor_schedd per network round-trip. With higher
values, the latency of negotiation can be significantly be reduced when negotiating with a condor_schedd running HTCondor version 8.3.0 or more recent, especially over a wide-area network. Setting this value too high,
however, could cause the condor_schedd to unnecessarily block on network I/O. The default value is 20. If
USE_RESOURCE_REQUEST_COUNTS is set to False, then this variable will be unconditionally set to a
value of 1.
NEGOTIATOR_MATCH_EXPRS A comma-separated list of macro names that are inserted as ClassAd attributes into
matched job ClassAds. The attribute name in the ClassAd will be given the prefix NegotiatorMatchExpr,
if the macro name does not already begin with that. Example:
HTCondor Version 8.6.4 Manual
3.5.16. condor_negotiator Configuration File Entries
NegotiatorName = "My Negotiator"
NEGOTIATOR_MATCH_EXPRS = NegotiatorName
As a result of the above configuration, jobs that are matched by this condor_negotiator will contain the following
attribute when they are sent to the condor_startd:
NegotiatorMatchExprNegotiatorName = "My Negotiator"
The expressions inserted by the condor_negotiator may be useful in condor_startd policy expressions, when
the condor_startd belongs to multiple HTCondor pools.
NEGOTIATOR_MATCHLIST_CACHING A boolean value that defaults to True. When True, it enables
an optimization in the condor_negotiator that works with auto clustering. In determining the sorted
list of machines that a job might use, the job goes to the first machine off the top of the list. If
NEGOTIATOR_MATCHLIST_CACHING is True, and if the next job is part of the same auto cluster, meaning
that it is a very similar job, the condor_negotiator will reuse the previous list of machines, instead of recreating
the list from scratch.
If matching grid resources, and the desire is for a given resource to potentially match multiple times per
condor_negotiator pass, NEGOTIATOR_MATCHLIST_CACHING should be False. See section 5.3.10 on
page 599 in the subsection on Advertising Grid Resources to HTCondor for an example.
NEGOTIATOR_CONSIDER_PREEMPTION For expert users only. A boolean value that defaults to True. When
False, it can cause the condor_negotiator to run faster and also have better spinning pie accuracy. Only set
this to False if PREEMPTION_REQUIREMENTS is False, and if all condor_startd rank expressions are
False.
NEGOTIATOR_CONSIDER_EARLY_PREEMPTION A boolean value that when False (the default), prevents
the condor_negotiator from matching jobs to claimed slots that cannot immediately be preempted due to
MAXJOBRETIREMENTTIME.
ALLOW_PSLOT_PREEMPTION A boolean value that defaults to False. When set to True for the condor_negotiator, it enables a new matchmaking mode in which one or more dynamic slots can be preempted
in order to make enough resources available in their parent partitionable slot for a job to successfully match to
the partitionable slot.
STARTD_AD_REEVAL_EXPR A boolean value evaluated in the context of each machine ClassAd within a negotiation cycle that determines whether the ClassAd from the condor_collector is to replace the stashed ClassAd
utilized during the previous negotiation cycle. When True, the ClassAd from the condor_collector does replace
the stashed one. When not defined, the default value is to replace the stashed ClassAd if the stashed ClassAd’s
sequence number is older than its potential replacement.
NEGOTIATOR_UPDATE_AFTER_CYCLE A boolean value that defaults to False. When True, it will force the
condor_negotiator daemon to publish an update to the condor_collector at the end of every negotiation cycle.
This is useful if monitoring statistics for the previous negotiation cycle.
NEGOTIATOR_READ_CONFIG_BEFORE_CYCLE A boolean value that defaults to False. When True, the condor_negotiator will re-read the configuration prior to beginning each negotiation cycle. Note that this operation
will update configured behaviors such as concurrency limits, but not data structures constructed during a full
HTCondor Version 8.6.4 Manual
313
3.5.16. condor_negotiator Configuration File Entries
reconfiguration, such as the group quota hierarchy. A full reconfiguration, for example as accomplished with
condor_reconfig, remains the best way to guarantee that all condor_negotiator configuration is completely updated.
<NAME>_LIMIT An integer value that defines the amount of resources available for jobs which declare that they use
some consumable resource as described in section 3.14.15. <Name> is a string invented to uniquely describe
the resource.
CONCURRENCY_LIMIT_DEFAULT An integer value that describes the number of resources available for any resources that are not explicitly named defined with the configuration variable <NAME>_LIMIT. If not defined,
no limits are set for resources not explicitly identified using <NAME>_LIMIT.
CONCURRENCY_LIMIT_DEFAULT_<NAME> If set, this defines a default concurrency limit for all resources that
start with <NAME>.
The following configuration macros affect negotiation for group users.
GROUP_NAMES A comma-separated list of the recognized group names, case insensitive. If undefined (the default),
group support is disabled. Group names must not conflict with any user names. That is, if there is a physics
group, there may not be a physics user. Any group that is defined here must also have a quota, or the group
will be ignored. Example:
GROUP_NAMES = group_physics, group_chemistry
GROUP_QUOTA_<groupname> A floating point value to represent a static quota specifying an integral number of
machines for the hierarchical group identified by <groupname>. It is meaningless to specify a non integer
value, since only integral numbers of machines can be allocated. Example:
GROUP_QUOTA_group_physics = 20
GROUP_QUOTA_group_chemistry = 10
When both static and dynamic quotas are defined for a specific group, the static quota is used and the dynamic
quota is ignored.
GROUP_QUOTA_DYNAMIC_<groupname> A floating point value in the range 0.0 to 1.0, inclusive, representing a fraction of a pool’s machines (slots) set as a dynamic quota for the hierarchical group identified by
<groupname>. For example, the following specifies that a quota of 25% of the total machines are reserved
for members of the group_biology group.
GROUP_QUOTA_DYNAMIC_group_biology = 0.25
The group name must be specified in the GROUP_NAMES list.
This section has not yet been completed
HTCondor Version 8.6.4 Manual
314
3.5.16. condor_negotiator Configuration File Entries
GROUP_PRIO_FACTOR_<groupname> A floating point value greater than or equal to 1.0 to specify the default
user priority factor for <groupname>. The group name must also be specified in the GROUP_NAMES list.
GROUP_PRIO_FACTOR_<groupname> is evaluated when the negotiator first negotiates for the user as a
member of the group. All members of the group inherit the default priority factor when no other value is
present. For example, the following setting specifies that all members of the group named group_physics
inherit a default user priority factor of 2.0:
GROUP_PRIO_FACTOR_group_physics = 2.0
GROUP_AUTOREGROUP A boolean value (defaults to False) that when True, causes users who submitted to a
specific group to also negotiate a second time with the <none> group, to be considered with the independent
job submitters. This allows group submitted jobs to be matched with idle machines even if the group is over its
quota. The user name that is used for accounting and prioritization purposes is still the group user as specified
by AccountingGroup in the job ClassAd.
GROUP_AUTOREGROUP_<groupname> This is the same as GROUP_AUTOREGROUP, but it is settable on
a per-group basis. If no value is specified for a given group, the default behavior is determined by
GROUP_AUTOREGROUP, which in turn defaults to False.
GROUP_ACCEPT_SURPLUS A boolean value that, when True, specifies that groups should be allowed to use more
than their configured quota when there is not enough demand from other groups to use all of the available
machines. The default value is False.
GROUP_ACCEPT_SURPLUS_<groupname> A boolean value applied as a group-specific version of
GROUP_ACCEPT_SURPLUS. When not specified, the value of GROUP_ACCEPT_SURPLUS applies to
the named group.
GROUP_QUOTA_ROUND_ROBIN_RATE The maximum sum of weighted slots that should be handed out to an individual submitter in each iteration within a negotiation cycle. If slot weights are not being used by the condor_negotiator, as specified by NEGOTIATOR_USE_SLOT_WEIGHTS = False, then this value is just the
(unweighted) number of slots. The default value is a very big number, effectively infinite. Setting the value to
a number smaller than the size of the pool can help avoid starvation. An example of the starvation problem is
when there are a subset of machines in a pool with large memory, and there are multiple job submitters who
desire all of these machines. Normally, HTCondor will decide how much of the full pool each person should
get, and then attempt to hand out that number of resources to each person. Since the big memory machines
are only a subset of pool, it may happen that they are all given to the first person contacted, and the remainder
requiring large memory machines get nothing. Setting GROUP_QUOTA_ROUND_ROBIN_RATE to a value that
is small compared to the size of subsets of machines will reduce starvation at the cost of possibly slowing down
the rate at which resources are allocated.
GROUP_QUOTA_MAX_ALLOCATION_ROUNDS An integer that specifies the maximum number of times within one
negotiation cycle the condor_negotiator will calculate how many slots each group deserves and attempt to
allocate them. The default value is 3. The reason it may take more than one round is that some groups may not
have jobs that match some of the available machines, so some of the slots that were withheld for those groups
may not get allocated in any given round.
HTCondor Version 8.6.4 Manual
315
3.5.17. condor_procd Configuration File Macros
NEGOTIATOR_USE_SLOT_WEIGHTS A boolean value with a default of True. When True, the condor_negotiator pays attention to the machine ClassAd attribute SlotWeight. When False, each slot effectively has a weight of 1.
NEGOTIATOR_USE_WEIGHTED_DEMAND A boolean value that defaults to True. When False, the behavior is
the same as for HTCondor versions prior to 7.9.6. If True, when the condor_schedd advertises IdleJobs
in the submitter ClassAd, which represents the number of idle jobs in the queue for that submitter, it will also
advertise the total number of requested cores across all idle jobs from that submitter, WeightedIdleJobs. If
partitionable slots are being used, and if hierarchical group quotas are used, and if any hierarchical group quotas
set GROUP_ACCEPT_SURPLUS to True, and if configuration variable SlotWeight is set to the number of
cores, then setting this configuration variable to True allows the amount of surplus allocated to each group to
be calculated correctly.
GROUP_SORT_EXPR A floating point ClassAd expression that controls the order in which the condor_negotiator
considers groups when allocating resources. The smallest magnitude positive value goes first. The default value
is set such that group <none> always goes last when considering group quotas, and groups are considered in
starvation order (the group using the smallest fraction of its resource quota is considered first).
NEGOTIATOR_ALLOW_QUOTA_OVERSUBSCRIPTION A boolean value that defaults to True. When True, the
behavior of resource allocation when considering groups is more like it was in the 7.4 stable series of HTCondor.
In implementation, when True, the static quotas of subgroups will not be scaled when the sum of these static
quotas of subgroups sums to more than the group’s static quota. This behavior is desirable when using static
quotas, unless the sum of subgroup quotas is considerably less than the group’s quota, as scaling is currently
based on the number of machines available, not assigned quotas (for static quotas).
3.5.17 condor_procd Configuration File Macros
USE_PROCD This boolean variable determines whether the condor_procd will be used for managing process families.
If the condor_procd is not used, each daemon will run the process family tracking logic on its own. Use
of the condor_procd results in improved scalability because only one instance of this logic is required. The
condor_procd is required when using group ID-based process tracking (see Section 3.14.11). In this case, the
USE_PROCD setting will be ignored and a condor_procd will always be used. By default, the condor_master
will start a condor_procd that all other daemons that need process family tracking will use. A daemon that uses
the condor_procd will start a condor_procd for use by itself and all of its child daemons.
PROCD_MAX_SNAPSHOT_INTERVAL This setting determines the maximum time that the condor_procd will wait
between probes of the system for information about the process families it is tracking.
PROCD_LOG Specifies a log file for the condor_procd to use. Note that by design, the condor_procd does not
include most of the other logic that is shared amongst the various HTCondor daemons. This means that the
condor_procd does not include the normal HTCondor logging subsystem, and thus multiple debug levels are
not supported. PROCD_LOG defaults to $(LOG)/ProcLog. Note that enabling D_PROCFAMILY in the
debug level for any other daemon will cause it to log all interactions with the condor_procd.
MAX_PROCD_LOG Controls the maximum length in bytes to which the condor_procd log will be allowed to grow.
The log file will grow to the specified length, then be saved to a file with the suffix .old. The .old file is
HTCondor Version 8.6.4 Manual
316
3.5.18. condor_credd Configuration File Macros
overwritten each time the log is saved, thus the maximum space devoted to logging will be twice the maximum
length of this log file. A value of 0 specifies that the file may grow without bounds. The default is 10 MiB.
PROCD_ADDRESS This specifies the address that the condor_procd will use to receive requests from other HTCondor daemons. On Unix, this should point to a file system location that can be used for a named pipe. On
Windows, named pipes are also used but they do not exist in the file system. The default setting therefore depends on the platform and distribution: $(LOCK)/procd_pipe or $(RUN)/procd_pipe on Unix and
\\.\pipe\procd_pipe on Windows.
USE_GID_PROCESS_TRACKING A boolean value that defaults to False. When True, a job’s initial process is
assigned a dedicated GID which is further used by the condor_procd to reliably track all processes associated
with a job. When True, values for MIN_TRACKING_GID and MAX_TRACKING_GID must also be set, or
HTCondor will abort, logging an error message. See section 3.14.11 on page 504 for a detailed description.
MIN_TRACKING_GID An integer value, that together with MAX_TRACKING_GID specify a range of GIDs to be
assigned on a per slot basis for use by the condor_procd in tracking processes associated with a job. See
section 3.14.11 on page 504 for a detailed description.
MAX_TRACKING_GID An integer value, that together with MIN_TRACKING_GID specify a range of GIDs to be
assigned on a per slot basis for use by the condor_procd in tracking processes associated with a job. See
section 3.14.11 on page 504 for a detailed description.
BASE_CGROUP The path to the directory used as the virtual file system for the implementation of Linux kernel
cgroups. This variable defaults to the string htcondor, and is only used on Linux systems. To disable cgroup
tracking, define this to an empty string. See section 3.14.12 on page 505 for a description of cgroup-based
process tracking.
3.5.18 condor_credd Configuration File Macros
These macros affect the condor_credd.
CREDD_HOST The host name of the machine running the condor_credd daemon.
CREDD_POLLING_TIMEOUT An integer value that determines how long the condor_credd daemon will poll for
credentials in seconds. The default value is 20.
CREDD_CACHE_LOCALLY A boolean value that defaults to False. When True, the first successful password
fetch operation to the condor_credd daemon causes the password to be stashed in a local, secure password
store. Subsequent uses of that password do not require communication with the condor_credd daemon.
SKIP_WINDOWS_LOGON_NETWORK A boolean value that defaults to False. When True, Windows authentication skips trying authentication with the LOGON_NETWORK method first, and attempts authentication with
LOGON_INTERACTIVE method. This can be useful if many authentication failures are noticed, potentially
leading to users getting locked out.
HTCondor Version 8.6.4 Manual
317
3.5.19. condor_gridmanager Configuration File Entries
3.5.19 condor_gridmanager Configuration File Entries
These macros affect the condor_gridmanager.
GRIDMANAGER_LOG Defines the path and file name for the log of the condor_gridmanager. The owner of the file
is the condor user.
GRIDMANAGER_CHECKPROXY_INTERVAL The number of seconds between checks for an updated X509 proxy
credential. The default is 10 minutes (600 seconds).
GRIDMANAGER_PROXY_REFRESH_TIME For GRAM jobs, the condor_gridmanager will not forward a refreshed
proxy until the lifetime left for the proxy on the remote machine falls below this value. The value is in seconds
and the default is 21600 (6 hours).
GRIDMANAGER_MINIMUM_PROXY_TIME The minimum number of seconds before expiration of the X509 proxy
credential for the gridmanager to continue operation. If seconds until expiration is less than this number, the
gridmanager will shutdown and wait for a refreshed proxy credential. The default is 3 minutes (180 seconds).
HOLD_JOB_IF_CREDENTIAL_EXPIRES True or False. Defaults to True. If True, and for grid universe jobs only,
HTCondor-G will place a job on hold GRIDMANAGER_MINIMUM_PROXY_TIME seconds before the proxy
expires. If False, the job will stay in the last known state, and HTCondor-G will periodically check to see if the
job’s proxy has been refreshed, at which point management of the job will resume.
GRIDMANAGER_CONTACT_SCHEDD_DELAY The minimum number of seconds between connections to the condor_schedd. The default is 5 seconds.
GRIDMANAGER_JOB_PROBE_INTERVAL The number of seconds between active probes for the status of a submitted job. The default is 1 minute (60 seconds). Intervals specific to grid types can be set by appending the
name of the grid type to the configuration variable name, as the example
GRIDMANAGER_JOB_PROBE_INTERVAL_GT5 = 300
GRIDMANAGER_JOB_PROBE_RATE The maximum number of job status probes per second that will be issued
to a given remote resource. The time between status probes for individual jobs may be lengthened beyond
GRIDMANAGER_JOB_PROBE_INTERVAL to enforce this rate. The default is 5 probes per second. Rates
specific to grid types can be set by appending the name of the grid type to the configuration variable name, as
the example
GRIDMANAGER_JOB_PROBE_RATE_GT5 = 15
GRIDMANAGER_RESOURCE_PROBE_INTERVAL When a resource appears to be down, how often (in seconds) the
condor_gridmanager should ping it to test if it is up again.
GRIDMANAGER_RESOURCE_PROBE_DELAY The number of seconds between pings of a remote resource that is
currently down. The default is 5 minutes (300 seconds).
HTCondor Version 8.6.4 Manual
318
3.5.19. condor_gridmanager Configuration File Entries
GRIDMANAGER_EMPTY_RESOURCE_DELAY The number of seconds that the condor_gridmanager retains information about a grid resource, once the condor_gridmanager has no active jobs on that resource. An active job
is a grid universe job that is in the queue, for which JobStatus is anything other than Held. Defaults to 300
seconds.
GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE An integer value that limits the number of jobs that
a condor_gridmanager daemon will submit to a resource. A comma-separated list of pairs that follows this
integer limit will specify limits for specific remote resources. Each pair is a host name and the job limit for that
host. Consider the example:
GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE = 200, foo.edu, 50, bar.com, 100
In this example, all resources have a job limit of 200, except foo.edu, which has a limit of 50, and bar.com,
which has a limit of 100.
Limits specific to grid types can be set by appending the name of the grid type to the configuration variable
name, as the example
GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE_CREAM = 300
In this example, the job limit for all CREAM resources is 300. Defaults to 1000.
GRIDMANAGER_MAX_JOBMANAGERS_PER_RESOURCE For grid jobs of type gt2, limits the number of globusjob-manager processes that the condor_gridmanager lets run at a time on the remote head node. Allowing too
many globus-job-managers to run causes severe load on the head note, possibly making it non-functional. This
number may be exceeded if it is reduced through the use of condor_reconfig while the condor_gridmanager is
running, or if some globus-job-managers take a few extra seconds to exit. The value 0 means there is no limit.
The default value is 10.
GAHP The full path to the binary of the GAHP server. This configuration variable is no longer used. Use GT2_GAHP
at section 3.5.19 instead.
GAHP_ARGS Arguments to be passed to the GAHP server. This configuration variable is no longer used.
GAHP_DEBUG_HIDE_SENSITIVE_DATA A boolean value that determines when sensitive data such as security
keys and passwords are hidden, when communication to or from a GAHP server is written to a daemon log. The
default is True, hiding sensitive data.
GRIDMANAGER_GAHP_CALL_TIMEOUT The number of seconds after which a pending GAHP command should
time out. The default is 5 minutes (300 seconds).
GRIDMANAGER_GAHP_RESPONSE_TIMEOUT The condor_gridmanager will assume a GAHP is hung if this many
seconds pass without a response. The default is 20.
GRIDMANAGER_MAX_PENDING_REQUESTS The maximum number of GAHP commands that can be pending at
any time. The default is 50.
GRIDMANAGER_CONNECT_FAILURE_RETRY_COUNT The number of times to retry a command that failed due
to a timeout or a failed connection. The default is 3.
HTCondor Version 8.6.4 Manual
319
3.5.19. condor_gridmanager Configuration File Entries
320
GRIDMANAGER_GLOBUS_COMMIT_TIMEOUT The duration, in seconds, of the two phase commit timeout to
Globus for gt2 jobs only. This maps directly to the two_phase setting in the Globus RSL.
GLOBUS_GATEKEEPER_TIMEOUT The number of seconds after which if a gt2 grid universe job fails to ping the
gatekeeper, the job will be put on hold. Defaults to 5 days (in seconds).
EC2_RESOURCE_TIMEOUT The number of seconds after which if an EC2 grid universe job fails to ping the EC2
service, the job will be put on hold. Defaults to -1, which implements an infinite length, such that a failure to
ping the service will never put the job on hold.
EC2_GAHP_RATE_LIMIT The minimum interval, in whole milliseconds, between requests to the same EC2 service
with the same credentials. Defaults to 100.
GRAM_VERSION_DETECTION A boolean value that defaults to True. When True, the condor_gridmanager
treats grid types gt2 and gt5 identically, and queries each server to determine which protocol it is using.
When False, the condor_gridmanager trusts the grid type provided in job attribute GridResource, and
treats the server accordingly. Beware that identifying a gt2 server as gt5 can result in overloading the server,
if a large number of jobs are submitted.
BATCH_GAHP_CHECK_STATUS_ATTEMPTS The number of times a failed status command issued to the
batch_gahp should be retried. These retries allow the condor_gridmanager to tolerate short-lived failures of
the underlying batch system. The default value is 5.
C_GAHP_LOG The complete path and file name of the HTCondor GAHP server’s log.
/tmp/CGAHPLog.$(USERNAME).
The default value is
MAX_C_GAHP_LOG The maximum size of the C_GAHP_LOG.
C_GAHP_WORKER_THREAD_LOG The complete path and file name of the HTCondor GAHP worker process’ log.
The default value is /temp/CGAHPWorkerLog.$(USERNAME).
C_GAHP_CONTACT_SCHEDD_DELAY The number of seconds that the condor_C-gahp daemon waits between consecutive connections to the remote condor_schedd in order to send batched sets of commands to be executed on
that remote condor_schedd daemon. The default value is 5.
GLITE_LOCATION The complete path to the directory containing the Glite software. The default value is
$(LIBEXEC)/glite. The necessary Glite software is included with HTCondor, and is required for gridtype batch jobs.
CONDOR_GAHP The complete path and file name of the HTCondor GAHP executable. The default value is
$(SBIN)/condor_c-gahp.
EC2_GAHP The complete path and file name of the EC2 GAHP executable.
$(SBIN)/ec2_gahp.
The default value is
GT2_GAHP The complete path and file name of the GT2 GAHP executable.
$(SBIN)/gahp_server.
The default value is
BATCH_GAHP The complete path and file name of the batch GAHP executable, to be used for PBS, LSF, SGE, and
similar batch systems. The default location is $(GLITE_LOCATION)/bin/batch_gahp.
HTCondor Version 8.6.4 Manual
3.5.20. condor_job_router Configuration File Entries
321
PBS_GAHP The complete path and file name of the PBS GAHP executable. The use of the configuration variable
BATCH_GAHP is preferred and encouraged, as this variable may no longer be supported in a future version of
HTCondor. A value given with this configuration variable will override a value specified by BATCH_GAHP, and
the value specified by BATCH_GAHP is the default if this variable is not defined.
LSF_GAHP The complete path and file name of the LSF GAHP executable. The use of the configuration variable
BATCH_GAHP is preferred and encouraged, as this variable may no longer be supported in a future version of
HTCondor. A value given with this configuration variable will override a value specified by BATCH_GAHP, and
the value specified by BATCH_GAHP is the default if this variable is not defined.
UNICORE_GAHP The complete path and file name of the wrapper script that invokes the Unicore GAHP executable.
The default value is $(SBIN)/unicore_gahp.
NORDUGRID_GAHP The complete path and file name of the wrapper script that invokes the NorduGrid GAHP executable. The default value is $(SBIN)/nordugrid_gahp.
CREAM_GAHP The complete path and file name of the CREAM GAHP executable.
$(SBIN)/cream_gahp.
The default value is
SGE_GAHP The complete path and file name of the SGE GAHP executable. The use of the configuration variable
BATCH_GAHP is preferred and encouraged, as this variable may no longer be supported in a future version of
HTCondor. A value given with this configuration variable will override a value specified by BATCH_GAHP, and
the value specified by BATCH_GAHP is the default if this variable is not defined.
GCE_GAHP The complete path and file name of the GCE GAHP executable.
$(SBIN)/gce_gahp.
BOINC_GAHP The complete path and file name of the BOINC GAHP executable.
$(SBIN)/boinc_gahp.
The default value is
The default value is
3.5.20 condor_job_router Configuration File Entries
These macros affect the condor_job_router daemon.
JOB_ROUTER_DEFAULTS Defined by a single ClassAd in New ClassAd syntax, used to provide default values for
all routes in the condor_job_router daemon’s routing table. Where an attribute is set outside of these defaults,
that attribute value takes precedence. The enclosing square brackets are optional.
JOB_ROUTER_ENTRIES Specification of the job routing table. It is a list of ClassAds, in New ClassAd syntax,
where each individual ClassAd is surrounded by square brackets, and the ClassAds are separated from each
other by spaces. Each ClassAd describes one entry in the routing table, and each describes a site that jobs may
be routed to.
A condor_reconfig command causes the condor_job_router daemon to rebuild the routing table. Routes are
distinguished by a routing table entry’s ClassAd attribute Name. Therefore, a Name change in an existing route
has the potential to cause the inaccurate reporting of routes.
Instead of setting job routes using this configuration variable, they may be read from an external source
using the JOB_ROUTER_ENTRIES_FILE or be dynamically generated by an external program via the
JOB_ROUTER_ENTRIES_CMD configuration variable.
HTCondor Version 8.6.4 Manual
3.5.20. condor_job_router Configuration File Entries
JOB_ROUTER_ENTRIES_FILE A path and file name of a file that contains the ClassAds, in New ClassAd syntax,
describing the routing table. The specified file is periodically reread to check for new information. This occurs
every $(JOB_ROUTER_ENTRIES_REFRESH) seconds.
JOB_ROUTER_ENTRIES_CMD Specifies the command line of an external program to run. The output of the
program defines or updates the routing table, and the output must be given in New ClassAd syntax. The
specified command is periodically rerun to regenerate or update the routing table. This occurs every
$(JOB_ROUTER_ENTRIES_REFRESH) seconds. Specify the full path and file name of the executable within
this command line, as no assumptions may be made about the current working directory upon command invocation. To enter spaces in any command-line arguments or in the command name itself, surround the right hand
side of this definition with double quotes, and use single quotes around individual arguments that contain spaces.
This is the same as when dealing with spaces within job arguments in an HTCondor submit description file.
JOB_ROUTER_ENTRIES_REFRESH The number of seconds between updates to the routing table described by
JOB_ROUTER_ENTRIES_FILE or JOB_ROUTER_ENTRIES_CMD. The default value is 0, meaning no periodic updates occur. With the default value of 0, the routing table can be modified when a condor_reconfig
command is invoked or when the condor_job_router daemon restarts.
JOB_ROUTER_LOCK This specifies the name of a lock file that is used to ensure that multiple instances of condor_job_router never run with the same JOB_ROUTER_NAME. Multiple instances running with the same name
could lead to mismanagement of routed jobs. The default value is $(LOCK)/$(JOB_ROUTER_NAME)Lock.
JOB_ROUTER_SOURCE_JOB_CONSTRAINT Specifies a global Requirements expression that must be true for
all newly routed jobs, in addition to any Requirements specified within a routing table entry. In addition to
the configurable constraints, the condor_job_router also has some hard-coded constraints. It avoids recursively
routing jobs by requiring that the job’s attribute RoutedBy does not match JOB_ROUTER_NAME. When not
running as root, it also avoids routing jobs belonging to other users.
JOB_ROUTER_MAX_JOBS An integer value representing the maximum number of jobs that may be routed, summed
over all routes. The default value is -1, which means an unlimited number of jobs may be routed.
MAX_JOB_MIRROR_UPDATE_LAG An integer value that administrators will rarely consider changing, representing
the maximum number of seconds the condor_job_router daemon waits, before it decides that routed copies have
gone awry, due to the failure of events to appear in the condor_schedd’s job queue log file. The default value is
600. As the condor_job_router daemon uses the condor_schedd’s job queue log file entries for synchronization
of routed copies, when an expected log file event fails to appear after this wait period, the condor_job_router
daemon acts presuming the expected event will never occur.
JOB_ROUTER_POLLING_PERIOD An integer value representing the number of seconds between cycles in the
condor_job_router daemon’s task loop. The default is 10 seconds. A small value makes the condor_job_router
daemon quick to see new candidate jobs for routing. A large value makes the condor_job_router daemon
generate less overhead at the cost of being slower to see new candidates for routing. For very large job queues
where a few minutes of routing latency is no problem, increasing this value to a few hundred seconds would be
reasonable.
JOB_ROUTER_NAME A unique identifier utilized to name multiple instances of the condor_job_router daemon on
the same machine. Each instance must have a different name, or all but the first to start up will refuse to run.
The default is "jobrouter".
HTCondor Version 8.6.4 Manual
322
3.5.20. condor_job_router Configuration File Entries
Changing this value when routed jobs already exist is not currently gracefully handled. However, it can be done
if one also uses condor_qedit to change the value of ManagedManager and RoutedBy from the old name
to the new name. The following commands may be helpful:
condor_qedit -constraint 'RoutedToJobId =!= undefined && \
ManagedManager == "insert_old_name"' \
ManagedManager '"insert_new_name"'
condor_qedit -constraint 'RoutedBy == "insert_old_name"' \
RoutedBy '"insert_new_name"'
JOB_ROUTER_RELEASE_ON_HOLD A boolean value that defaults to True. It controls how the condor_job_router
handles the routed copy when it goes on hold. When True, the condor_job_router leaves the original job
ClassAd in the same state as when claimed. When False, the condor_job_router does not attempt to reset the
original job ClassAd to a pre-claimed state upon yielding control of the job.
JOB_ROUTER_SCHEDD1_SPOOL The path to the spool directory for the condor_schedd serving as the source of
jobs for routing. If not specified, this defaults to $(SPOOL). If specified, this parameter must point to the spool
directory of the condor_schedd identified by JOB_ROUTER_SCHEDD1_NAME.
JOB_ROUTER_SCHEDD2_SPOOL The path to the spool directory for the condor_schedd to which the routed copy
of the jobs are submitted. If not specified, this defaults to $(SPOOL). If specified, this parameter must point
to the spool directory of the condor_schedd identified by JOB_ROUTER_SCHEDD2_NAME. Note that when
condor_job_router is running as root and is submitting routed jobs to a different condor_schedd than the
source condor_schedd, it is required that condor_job_router have permission to impersonate the job owners of
the routed jobs. It is therefore usually necessary to configure QUEUE_SUPER_USER_MAY_IMPERSONATE in
the configuration of the target condor_schedd.
JOB_ROUTER_SCHEDD1_NAME The advertised daemon name of the condor_schedd serving as the source of jobs
for routing. If not specified, this defaults to the local condor_schedd. If specified, this parameter must name
the same condor_schedd whose spool is configured in JOB_ROUTER_SCHEDD1_SPOOL. If the named condor_schedd is not advertised in the local pool, JOB_ROUTER_SCHEDD1_POOL will also need to be set.
JOB_ROUTER_SCHEDD2_NAME The advertised daemon name of the condor_schedd to which the routed copy of
the jobs are submitted. If not specified, this defaults to the local condor_schedd. If specified, this parameter must name the same condor_schedd whose spool is configured in JOB_ROUTER_SCHEDD2_SPOOL.
If the named condor_schedd is not advertised in the local pool, JOB_ROUTER_SCHEDD2_POOL will also
need to be set. Note that when condor_job_router is running as root and is submitting routed jobs to
a different condor_schedd than the source condor_schedd, it is required that condor_job_router have permission to impersonate the job owners of the routed jobs. It is therefore usually necessary to configure
QUEUE_SUPER_USER_MAY_IMPERSONATE in the configuration of the target condor_schedd.
JOB_ROUTER_SCHEDD1_POOL The Condor pool (condor_collector address) of the condor_schedd serving as the
source of jobs for routing. If not specified, defaults to the local pool.
JOB_ROUTER_SCHEDD2_POOL The Condor pool (condor_collector address) of the condor_schedd to which the
routed copy of the jobs are submitted. If not specified, defaults to the local pool.
HTCondor Version 8.6.4 Manual
323
3.5.21. condor_lease_manager Configuration File Entries
3.5.21 condor_lease_manager Configuration File Entries
These macros affect the condor_lease_manager.
The condor_lease_manager expects to use the syntax
<subsystem name>.<parameter name>
in configuration. This allows multiple instances of the condor_lease_manager to be easily configured using the syntax
<subsystem name>.<local name>.<parameter name>
LeaseManager.GETADS_INTERVAL An integer value, given in seconds, that controls the frequency with which
the condor_lease_manager pulls relevant resource ClassAds from the condor_collector. The default value is 60
seconds, with a minimum value of 2 seconds.
LeaseManager.UPDATE_INTERVAL An integer value, given in seconds, that controls the frequency with which
the condor_lease_manager sends its ClassAds to the condor_collector. The default value is 60 seconds, with a
minimum value of 5 seconds.
LeaseManager.PRUNE_INTERVAL An integer value, given in seconds, that controls the frequency with which
the condor_lease_manager prunes its leases. This involves checking all leases to see if they have expired. The
default value is 60 seconds, with no minimum value.
LeaseManager.DEBUG_ADS A boolean value that defaults to False. When True, it enables extra debugging
information about the resource ClassAds that it retrieves from the condor_collector and about the search ClassAds that it sends to the condor_collector.
LeaseManager.MAX_LEASE_DURATION An integer value representing seconds which determines the maximum duration of a lease. This can be used to provide a hard limit on lease durations. Normally, the condor_lease_manager honors the MaxLeaseDuration attribute from the resource ClassAd. If this configuration variable is defined, it limits the effective maximum duration for all resources to this value. The default value
is 1800 seconds.
Note that leases can be renewed, and thus can be extended beyond this limit. To provide a limit on the total
duration of a lease, use LeaseManager.MAX_TOTAL_LEASE_DURATION.
LeaseManager.MAX_TOTAL_LEASE_DURATION An integer value representing seconds used to limit the total
duration of leases, over all its renewals. The default value is 3600 seconds.
LeaseManager.DEFAULT_MAX_LEASE_DURATION The
condor_lease_manager
uses
the
MaxLeaseDuration attribute from the resource ClassAd to limit the lease duration. If this attribute
is not present in a resource ClassAd, then this configuration variable is used instead. This integer value is given
in units of seconds, with a default value of 60 seconds.
LeaseManager.CLASSAD_LOG This variable defines a full path and file name to the location where the condor_lease_manager keeps persistent state information. This variable has no default value.
HTCondor Version 8.6.4 Manual
324
3.5.22. Grid Monitor Configuration File Entries
LeaseManager.QUERY_ADTYPE This parameter controls the type of the query in the ClassAd sent to the condor_collector, which will control the types of ClassAds returned by the condor_collector. This parameter must
be a valid ClassAd type name, with a default value of "Any".
LeaseManager.QUERY_CONSTRAINTS A ClassAd expression that controls the constraint in the query sent to
the condor_collector. It is used to further constrain the types of ClassAds from the condor_collector. There is
no default value, resulting in no constraints being placed on query.
3.5.22 Grid Monitor Configuration File Entries
These macros affect the Grid Monitor.
ENABLE_GRID_MONITOR A boolean value that when True enables the Grid Monitor. The Grid Monitor is
used to reduce load on Globus gatekeepers. This parameter only affects grid jobs of type gt2. The variable
GRID_MONITOR must also be correctly configured. Defaults to True. See section 5.3.2 on page 588 for more
information.
GRID_MONITOR The complete path name of the grid_monitor.sh tool used to reduce the load on Globus gatekeepers. This parameter only affects grid jobs of type gt2. This parameter is not referenced unless
ENABLE_GRID_MONITOR is set to True (the default value).
GRID_MONITOR_HEARTBEAT_TIMEOUT The integer number of seconds that may pass without hearing from a
working Grid Monitor before it is assumed to be dead. Defaults to 300 (5 minutes). Increasing this number will
improve the ability of the Grid Monitor to survive in the face of transient problems, but will also increase the
time before HTCondor notices a problem.
GRID_MONITOR_RETRY_DURATION When HTCondor-G attempts to start the Grid Monitor at a particular site, it
will wait this many seconds to start hearing from the Grid Monitor. Defaults to 900 (15 minutes). If this duration
passes without success, the Grid Monitor will be disabled for the site in question for the period of time set by
GRID_MONITOR_DISABLE_TIME.
GRID_MONITOR_NO_STATUS_TIMEOUT Jobs can disappear from the Grid Monitor’s status reports for short periods of time under normal circumstances, but a prolonged absence is often a sign of problems on the remote machine. This variable sets the amount of time (in seconds) that a job can be absent before the condor_gridmanager
reacts by restarting the GRAM jobmanager. The default is 900, which is 15 minutes.
GRID_MONITOR_DISABLE_TIME When an error occurs with a Grid Monitor job, this parameter controls how long
the condor_gridmanager will wait before attempting to start a new Grid Monitor job. The value is in seconds
and the default is 3600 (1 hour).
3.5.23 Configuration File Entries Relating to Grid Usage
These macros affect the HTCondor’s usage of grid resources.
GLEXEC_JOB A boolean value that defaults to False. When True, it enables the use of glexec on the machine.
HTCondor Version 8.6.4 Manual
325
3.5.24. Configuration File Entries for DAGMan
GLEXEC The full path and file name of the glexec executable.
GLEXEC_RETRIES An integer value that specifies the maximum number of times to retry a call to glexec when
glexec exits with status 202 or 203, error codes that indicate a possible transient error condition. The default
number of retries is 3.
GLEXEC_RETRY_DELAY An integer value that specifies the minimum number of seconds to wait between retries
of a failed call to glexec. The default is 5 seconds. The actual delay to be used is determined by a random
exponential backoff algorithm that chooses a delay with a minimum of the value of GLEXEC_RETRY_DELAY
and a maximum of 100 times that value.
GLEXEC_HOLD_ON_INITIAL_FAILURE A boolean value that when False prevents a job from being put on
hold when a failure is encountered during the glexec setup phase of managing a job. The default is True.
glexec is invoked multiple times during each attempt to run a job. This configuration setting only disables
putting the job on hold for the initial invocation. Subsequent failures during that run attempt always put the job
on hold.
3.5.24 Configuration File Entries for DAGMan
These macros affect the operation of DAGMan and DAGMan jobs within HTCondor.
Note: Many, if not all, of these configuration variables will be most appropriately set on a per DAG basis, rather
than in the global HTCondor configuration files. Per DAG configuration is explained in section 2.10.9. Also note that
configuration settings of a running condor_dagman job are not changed by doing a condor_reconfig.
General
DAGMAN_CONFIG_FILE The path and name of the configuration file to be used by condor_dagman. This configuration variable is set automatically by condor_submit_dag, and it should not be explicitly set by the user.
Defaults to the empty string.
DAGMAN_USE_STRICT An integer defining the level of strictness condor_dagman will apply when turning warnings
into fatal errors, as follows:
• 0: no warnings become errors
• 1: severe warnings become errors
• 2: medium-severity warnings become errors
• 3: almost all warnings become errors
Using a strictness value greater than 0 may help find problems with a DAG that may otherwise escape notice.
The default value if not defined is 1.
DAGMAN_STARTUP_CYCLE_DETECT A boolean value that defaults to False. When True, causes condor_dagman to check for cycles in the DAG before submitting DAG node jobs, in addition to its run time
cycle detection. Note that setting this value to True will impose significant startup delays for large DAGs.
HTCondor Version 8.6.4 Manual
326
3.5.24. Configuration File Entries for DAGMan
DAGMAN_ABORT_DUPLICATES A boolean value that controls whether to attempt to abort duplicate instances of
condor_dagman running the same DAG on the same machine. When condor_dagman starts up, if no DAG
lock file exists, condor_dagman creates the lock file and writes its PID into it. If the lock file does exist, and
DAGMAN_ABORT_DUPLICATES is set to True, condor_dagman checks whether a process with the given PID
exists, and if so, it assumes that there is already another instance of condor_dagman running the same DAG.
Note that this test is not foolproof: it is possible that, if condor_dagman crashes, the same PID gets reused
by another process before condor_dagman gets rerun on that DAG. This should be quite rare, however. If not
defined, DAGMAN_ABORT_DUPLICATES defaults to True. Note: users should rarely change this setting.
DAGMAN_USE_OLD_DAG_READER As of HTCondor version 8.3.3, this variable is no longer supported. Its value
will always be False. A setting of True will result in a warning, and the setting will have no effect on how
a DAG input file is read. The variable was previously used to change the reading of DAG input files to that of
HTCondor versions prior to 8.0.6. Note: users should never change this setting.
DAGMAN_USE_SHARED_PORT A boolean value that controls whether condor_dagman will attempt to connect to the
shared port daemon. If not defined, DAGMAN_USE_SHARED_PORT defaults to False. There is no reason to
ever change this value; it was introduced to prevent spurious shared port-related error messages from appearing
in dagman.out files. (Introduced in version 8.6.1.)
Throttling
DAGMAN_MAX_JOBS_IDLE An integer value that controls the maximum number of idle procs allowed within the
DAG before condor_dagman temporarily stops submitting jobs. condor_dagman will resume submitting jobs
once the number of idle procs falls below the specified limit. DAGMAN_MAX_JOBS_IDLE currently counts
each individual proc within a cluster as a job, which is inconsistent with DAGMAN_MAX_JOBS_SUBMITTED.
Note that submit description files that queue multiple procs can cause the DAGMAN_MAX_JOBS_IDLE limit to
be exceeded. If a submit description file contains queue 5000 and DAGMAN_MAX_JOBS_IDLE is set to 250,
this will result in 5000 procs being submitted to the condor_schedd, not 250; in this case, no further jobs will
then be submitted by condor_dagman until the number of idle procs falls below 250. The default value is 1000.
To disable this limit, set the value to 0. This configuration option can be overridden by the condor_submit_dag
-maxidle command-line argument (see 11).
DAGMAN_MAX_JOBS_SUBMITTED An integer value that controls the maximum number of node jobs (clusters)
within the DAG that will be submitted to HTCondor at one time. A single invocation of condor_submit by
condor_dagman counts as one job, even if the submit file produces a multi-proc cluster. The default value is 0
(unlimited). This configuration option can be overridden by the condor_submit_dag -maxjobs command-line
argument (see 11).
DAGMAN_MAX_PRE_SCRIPTS An integer defining the maximum number of PRE scripts that any given condor_dagman will run at the same time. The value 0 allows any number of PRE scripts to run. The default
value if not defined is 20. Note that the DAGMAN_MAX_PRE_SCRIPTS value can be overridden by the condor_submit_dag -maxpre command line option.
DAGMAN_MAX_POST_SCRIPTS An integer defining the maximum number of POST scripts that any given condor_dagman will run at the same time. The value 0 allows any number of POST scripts to run. The default
value if not defined is 20. Note that the DAGMAN_MAX_POST_SCRIPTS value can be overridden by the
condor_submit_dag -maxpost command line option.
HTCondor Version 8.6.4 Manual
327
3.5.24. Configuration File Entries for DAGMan
328
Priority, node semantics
DAGMAN_DEFAULT_PRIORITY An integer value defining the minimum priority of node jobs running under this
condor_dagman job. Defaults to 0.
DAGMAN_SUBMIT_DEPTH_FIRST A boolean value that controls whether to submit ready DAG node jobs in (moreor-less) depth first order, as opposed to breadth-first order. Setting DAGMAN_SUBMIT_DEPTH_FIRST to
True does not override dependencies defined in the DAG. Rather, it causes newly ready nodes to be added to
the head, rather than the tail, of the ready node list. If there are no PRE scripts in the DAG, this will cause the
ready nodes to be submitted depth-first. If there are PRE scripts, the order will not be strictly depth-first, but it
will tend to favor depth rather than breadth in executing the DAG. If DAGMAN_SUBMIT_DEPTH_FIRST is set
to True, consider also setting DAGMAN_RETRY_SUBMIT_FIRST and DAGMAN_RETRY_NODE_FIRST to
True. If not defined, DAGMAN_SUBMIT_DEPTH_FIRST defaults to False.
DAGMAN_ALWAYS_RUN_POST A boolean value defining whether condor_dagman will ignore the return value of a
PRE script when deciding whether to run a POST script. The default is False, which means that the failure
of a PRE script causes the POST script to not be executed. Changing this to True will restore the previous
behavior of condor_dagman, which is that a POST script is always executed, even if the PRE script fails. (The
default for this value had originally been False, was changed to True in version 7.7.2, and then was changed
back to False in version 8.5.4.)
Node job submission/removal
DAGMAN_USER_LOG_SCAN_INTERVAL An integer value representing the number of seconds that condor_dagman waits between checking the workflow log file for status updates. Setting this value lower than the
default increases the CPU time condor_dagman spends checking files, perhaps fruitlessly, but increases responsiveness to nodes completing or failing. The legal range of values is 1 to INT_MAX. If not defined, it defaults
to 5 seconds. (As of version 8.4.2, the default may be automatically decreased if DAGMAN_MAX_JOBS_IDLE
is set to a small value. If so, this will be noted in the dagman.out file.)
DAGMAN_MAX_SUBMITS_PER_INTERVAL An integer that controls how many individual jobs condor_dagman
will submit in a row before servicing other requests (such as a condor_rm). The legal range of values is 1 to
1000. If defined with a value less than 1, the value 1 will be used. If defined with a value greater than 1000, the
value 1000 will be used. If not defined, it defaults to 5. (As of version 8.4.2, the default may be automatically
decreased if DAGMAN_MAX_JOBS_IDLE is set to a small value. If so, this will be noted in the dagman.out
file.)
Note:
The
maximum
rate
at
which
DAGMan
can
submit
DAGMAN_MAX_SUBMITS_PER_INTERVAL / DAGMAN_USER_LOG_SCAN_INTERVAL.
jobs
is
DAGMAN_MAX_SUBMIT_ATTEMPTS An integer that controls how many times in a row condor_dagman will attempt to execute condor_submit for a given job before giving up. Note that consecutive attempts use an exponential backoff, starting with 1 second. The legal range of values is 1 to 16. If defined with a value less than 1,
the value 1 will be used. If defined with a value greater than 16, the value 16 will be used. Note that a value
of 16 would result in condor_dagman trying for approximately 36 hours before giving up. If not defined, it
defaults to 6 (approximately two minutes before giving up).
HTCondor Version 8.6.4 Manual
3.5.24. Configuration File Entries for DAGMan
DAGMAN_MAX_JOB_HOLDS An integer value defining the maximum number of times a node job is allowed to go on
hold. As a job goes on hold this number of times, it is removed from the queue. For example, if the value is 2,
as the job goes on hold for the second time, it will be removed. At this time, this feature is not fully compatible
with node jobs that have more than one ProcID. The number of holds of each process in the cluster count
towards the total, rather than counting individually. So, this setting should take that possibility into account,
possibly using a larger value. A value of 0 allows a job to go on hold any number of times. The default value if
not defined is 100.
DAGMAN_HOLD_CLAIM_TIME An integer defining the number of seconds that condor_dagman will cause a hold
on a claim after a job is finished, using the job ClassAd attribute KeepClaimIdle. The default value is 20. A
value of 0 causes condor_dagman not to set the job ClassAd attribute.
DAGMAN_SUBMIT_DELAY An integer that controls the number of seconds that condor_dagman will sleep before
submitting consecutive jobs. It can be increased to help reduce the load on the condor_schedd daemon. The
legal range of values is any non negative integer. If defined with a value less than 0, the value 0 will be used.
DAGMAN_PROHIBIT_MULTI_JOBS A boolean value that controls whether condor_dagman prohibits node
job submit description files that queue multiple job procs other than parallel universe. If a DAG
references such a submit file, the DAG will abort during the initialization process. If not defined,
DAGMAN_PROHIBIT_MULTI_JOBS defaults to False.
DAGMAN_GENERATE_SUBDAG_SUBMITS A boolean value specifying whether condor_dagman itself should
create the .condor.sub files for nested DAGs. If set to False, nested DAGs will fail unless the
.condor.sub files are generated manually by running condor_submit_dag -no_submit on each nested DAG,
or the -do_recurse flag is passed to condor_submit_dag for the top-level DAG. DAG nodes specified with the
SUBDAG EXTERNAL keyword or with submit description file names ending in .condor.sub are considered
nested DAGs. The default value if not defined is True.
DAGMAN_REMOVE_NODE_JOBS A boolean value that controls whether condor_dagman removes its node
jobs itself when it is removed (in addition to the condor_schedd removing them). Note that setting
DAGMAN_REMOVE_NODE_JOBS to True is the safer option (setting it to False means that there is some
chance of endig up with "orphan" node jobs). Setting DAGMAN_REMOVE_NODE_JOBS to False is a performance optimization (decreasing the load on the condor_schedd when a condor_dagman job is removed). Note
that even if DAGMAN_REMOVE_NODE_JOBS is set to False, condor_dagman will remove its node jobs in
some cases, such as a DAG abort triggered by an ABORT-DAG-ON command. Defaults to True.
DAGMAN_MUNGE_NODE_NAMES A boolean value that controls whether condor_dagman automatically renames
nodes when running multiple DAGs. The renaming is done to avoid possible name conflicts. If this value
is set to True, all node names have the DAG number followed by the period character (.) prepended to them.
For example, the first DAG specified on the condor_submit_dag command line is considered DAG number 0,
the second is DAG number 1, etc. So if DAG number 2 has a node named B, that node will internally be renamed
to 2.B. If not defined, DAGMAN_MUNGE_NODE_NAMES defaults to True. Note: users should rarely change
this setting.
DAGMAN_SUPPRESS_JOB_LOGS A boolean value specifying whether events should be written to a log file specified in a node job’s submit description file. The default value is False, such that events are written to a log file
specified by a node job.
HTCondor Version 8.6.4 Manual
329
3.5.24. Configuration File Entries for DAGMan
DAGMAN_SUPPRESS_NOTIFICATION A boolean value defining whether jobs submitted by condor_dagman will
use email notification when certain events occur. If True, all jobs submitted by condor_dagman will have the
equivalent of the submit command notification = never set. This does not affect the notification for
events relating to the condor_dagman job itself. Defaults to True.
DAGMAN_CONDOR_SUBMIT_EXE The executable that condor_dagman will use to submit HTCondor jobs. If not
defined, condor_dagman looks for condor_submit in the path. Note: users should rarely change this setting.
DAGMAN_CONDOR_RM_EXE The executable that condor_dagman will use to remove HTCondor jobs. If not defined,
condor_dagman looks for condor_rm in the path. Note: users should rarely change this setting.
DAGMAN_ABORT_ON_SCARY_SUBMIT A boolean value that controls whether to abort a DAG upon detection of a
scary submit event. An example of a scary submit event is one in which the HTCondor ID does not match the
expected value. Note that in all HTCondor versions prior to 6.9.3, condor_dagman did not abort a DAG upon detection of a scary submit event. This behavior is what now happens if DAGMAN_ABORT_ON_SCARY_SUBMIT
is set to False. If not defined, DAGMAN_ABORT_ON_SCARY_SUBMIT defaults to True. Note: users
should rarely change this setting.
DAGMAN_STORK_SUBMIT_EXE This configuration variable is no longer used; as of HTCondor version 8.3.4, condor_dagman no longer supports Stork jobs. Setting this configuration variable will result in a warning from
condor_dagman (which will be turned into a fatal error if DAGMAN_USE_STRICT is set to 1 or above). Note:
users should never change this setting.
For completeness, here is the definition for historical purposes: The executable that condor_dagman will use to
submit Stork jobs. If not defined, condor_dagman looks for stork_submit in the path.
DAGMAN_STORK_RM_EXE This configuration variable is no longer used; as of HTCondor version 8.3.4, condor_dagman no longer supports Stork jobs. Setting this configuration variable will result in a warning from
condor_dagman (which will be turned into a fatal error if DAGMAN_USE_STRICT is set to 1 or above). Note:
users should never change this setting.
For completeness, here is the definition for historical purposes: The executable that condor_dagman will use to
remove Stork jobs. If not defined, condor_dagman looks for stork_rm in the path.
Rescue/retry
DAGMAN_AUTO_RESCUE A boolean value that controls whether condor_dagman automatically runs Rescue DAGs.
If DAGMAN_AUTO_RESCUE is True and the DAG input file my.dag is submitted, and if a Rescue DAG such
as the examples my.dag.rescue001 or my.dag.rescue002 exists, then the largest magnitude Rescue
DAG will be run. If not defined, DAGMAN_AUTO_RESCUE defaults to True.
DAGMAN_MAX_RESCUE_NUM An integer value that controls the maximum Rescue DAG number that will be written, in the case that DAGMAN_OLD_RESCUE is False, or run if DAGMAN_AUTO_RESCUE is True. The
maximum legal value is 999; the minimum value is 0, which prevents a Rescue DAG from being written at all,
or automatically run. If not defined, DAGMAN_MAX_RESCUE_NUM defaults to 100.
DAGMAN_RESET_RETRIES_UPON_RESCUE A boolean value that controls whether node retries are reset in a Rescue DAG. If this value is False, the number of node retries written in a Rescue DAG is decreased, if any retries
HTCondor Version 8.6.4 Manual
330
3.5.24. Configuration File Entries for DAGMan
were used in the original run of the DAG; otherwise, the original number of retries is allowed when running the
Rescue DAG. If not defined, DAGMAN_RESET_RETRIES_UPON_RESCUE defaults to True.
DAGMAN_WRITE_PARTIAL_RESCUE A boolean value that controls whether condor_dagman writes a partial or
a full DAG file as a Rescue DAG. As of HTCondor version 7.2.2, writing a partial DAG is preferred. If not
defined, DAGMAN_WRITE_PARTIAL_RESCUE defaults to True. Note: users should rarely change this
setting.
DAGMAN_RETRY_SUBMIT_FIRST A boolean value that controls whether a failed submit is retried first (before any
other submits) or last (after all other ready jobs are submitted). If this value is set to True, when a job submit
fails, the job is placed at the head of the queue of ready jobs, so that it will be submitted again before any other
jobs are submitted. This had been the behavior of condor_dagman. If this value is set to False, when a job
submit fails, the job is placed at the tail of the queue of ready jobs. If not defined, it defaults to True.
DAGMAN_RETRY_NODE_FIRST A boolean value that controls whether a failed node with retries is retried first
(before any other ready nodes) or last (after all other ready nodes). If this value is set to True, when a node
with retries fails after the submit succeeded, the node is placed at the head of the queue of ready nodes, so that
it will be tried again before any other jobs are submitted. If this value is set to False, when a node with retries
fails, the node is placed at the tail of the queue of ready nodes. This had been the behavior of condor_dagman.
If not defined, it defaults to False.
DAGMAN_OLD_RESCUE This configuration variable is no longer used. Note: users should never change this
setting.
Log files
DAGMAN_DEFAULT_NODE_LOG The default name of a file to be used as a job event log by all node jobs of a DAG.
This configuration variable uses a special syntax in which @ instead of $ indicates an evaluation of special
variables. Normal HTCondor configuration macros may be used with the normal $ syntax.
Special variables to be used only in defining this configuration variable:
• @(DAG_DIR): The directory in which the primary DAG input file resides. If more than one DAG input
file is specified to condor_submit_dag, the primary DAG input file is the leftmost one on the command
line.
• @(DAG_FILE): The name of the primary DAG input file. It does not include the path.
• @(CLUSTER): The ClusterId attribute of the condor_dagman job.
• @(OWNER): The user name of the user who submitted the DAG.
• @(NODE_NAME): For SUBDAGs, this is the node name of the SUBDAG in the upper level DAG; for a
top-level DAG, it is the string "undef".
If not defined, @(DAG_DIR)/@(DAG_FILE).nodes.log is the default value.
Notes:
• Using $(LOG) in defining a value for DAGMAN_DEFAULT_NODE_LOG will not have the expected effect,
because $(LOG) is defined as "." for condor_dagman. To place the default log file into the log directory,
write the expression relative to a known directory, such as $(LOCAL_DIR)/log (see examples below).
HTCondor Version 8.6.4 Manual
331
3.5.24. Configuration File Entries for DAGMan
• A default log file placed in the spool directory will need extra configuration to prevent condor_preen from
removing it; modify VALID_SPOOL_FILES. Removal of the default log file during a run will cause
severe problems.
• The value defined for DAGMAN_DEFAULT_NODE_LOG must ensure that the file is unique for each
DAG. Therefore, the value should always include @(DAG_FILE). For example,
DAGMAN_DEFAULT_NODE_LOG = $(LOCAL_DIR)/log/@(DAG_FILE).nodes.log
is okay, but
DAGMAN_DEFAULT_NODE_LOG = $(LOCAL_DIR)/log/dag.nodes.log
will cause failure when more than one DAG is run at the same time on a given submit machine.
DAGMAN_LOG_ON_NFS_IS_ERROR A boolean value that controls whether condor_dagman prohibits a DAG
workflow log from being on an NFS file system. This value is ignored if CREATE_LOCKS_ON_LOCAL_DISK
and ENABLE_USERLOG_LOCKING are both True. If a DAG uses such a workflow log file file and
DAGMAN_LOG_ON_NFS_IS_ERROR is True (and not ignored), the DAG will abort during the initialization
process. If not defined, DAGMAN_LOG_ON_NFS_IS_ERROR defaults to False.
DAGMAN_ALLOW_LOG_ERROR This configuration variable is no longer used; as of HTCondor version 8.3.4, condor_dagman no longer supports Stork jobs. Note: users should never change this setting.
For completeness, here is the definition for historical purposes: A boolean value defining whether condor_dagman will still attempt to run a node job, even if errors are detected in the job event log specification.
This setting has an effect only on nodes that are Stork jobs (not HTCondor jobs). The default value if not defined
is False.
DAGMAN_ALLOW_EVENTS An integer that controls which bad events are considered fatal errors by condor_dagman.
This macro replaces and expands upon the functionality of the
DAGMAN_IGNORE_DUPLICATE_JOB_EXECUTION macro. If DAGMAN_ALLOW_EVENTS is set, it
overrides the setting of DAGMAN_IGNORE_DUPLICATE_JOB_EXECUTION. Note: users should rarely
change this setting.
The DAGMAN_ALLOW_EVENTS value is a logical bitwise OR of the following values:
0 = allow no bad events
1 = allow all bad events, except the event "job re-run after terminated event"
2 = allow terminated/aborted event combination
4 = allow a "job re-run after terminated event" bug
8 = allow garbage or orphan events
16 = allow an execute or terminate event before job’s submit event
32 = allow two terminated events per job, as sometimes seen with grid jobs
64 = allow duplicated events in general
HTCondor Version 8.6.4 Manual
332
3.5.24. Configuration File Entries for DAGMan
The default value is 114, which allows terminated/aborted event combination, allows an execute and/or terminated event before job’s submit event, allows double terminated events, and allows general duplicate events.
As examples, a value of 6 instructs condor_dagman to allow both the terminated/aborted event combination
and the "job re-run after terminated event" bug. A value of 0 means that any bad event will be
considered a fatal error.
A value of 5 will never abort the DAG because of a bad event. But this value should almost never be used,
because the "job re-run after terminated event" bug breaks the semantics of the DAG.
DAGMAN_IGNORE_DUPLICATE_JOB_EXECUTION This configuration variable is no longer used. The improved
functionality of the DAGMAN_ALLOW_EVENTS macro eliminates the need for this variable. Note: users should
never change this setting.
For completeness, here is the definition for historical purposes: A boolean value that controls whether
condor_dagman aborts or continues with a DAG in the rare case that HTCondor erroneously executes
the job within a DAG node more than once. A bug in HTCondor very occasionally causes a job
to run twice. Running a job twice is contrary to the semantics of a DAG. The configuration macro
DAGMAN_IGNORE_DUPLICATE_JOB_EXECUTION determines whether condor_dagman considers this a fatal error or not. The default value is False; condor_dagman considers running the job more than once a fatal
error, logs this fact, and aborts the DAG. When set to True, condor_dagman still logs this fact, but continues
with the DAG.
This configuration macro is to remain at its default value except in the case where a site encounters the HTCondor bug in which DAG job nodes are executed twice, and where it is certain that having a DAG job node run
twice will not corrupt the DAG. The logged messages within *.dagman.out files in the case of that a node
job runs twice contain the string "EVENT ERROR."
DAGMAN_ALWAYS_USE_NODE_LOG As of HTCondor version 8.3.1, the value must always be the default value of
True. Attempting to set it to False results in an error. This causes incompatibility with using a condor_submit
executable that is older than HTCondor version 7.9.0. Note: users should never change this setting.
For completeness, here is the definition for historical purposes: A boolean value that when True causes condor_dagman to read events from its default node log file, as defined by DAGMAN_DEFAULT_NODE_LOG,
instead of from the log file(s) defined in the node job submit description files. When True, condor_dagman
will read events only from the default log file, and POST script terminated events will be written only to the
default log file, and not to the log file(s) defined in the node job submit description files. The default value is
True.
Debug output
DAGMAN_DEBUG This variable is described in section 3.5.3 as <SUBSYS>_DEBUG.
DAGMAN_VERBOSITY An integer value defining the verbosity of output to the dagman.out file, as follows (each
level includes all output from lower debug levels):
• level = 0; never produce output, except for usage info
• level = 1; very quiet, output severe errors
• level = 2; output errors and warnings
HTCondor Version 8.6.4 Manual
333
3.5.24. Configuration File Entries for DAGMan
• level = 3; normal output
• level = 4; internal debugging output
• level = 5; internal debugging output; outer loop debugging
• level = 6; internal debugging output; inner loop debugging
• level = 7; internal debugging output; rarely used
The default value if not defined is 3.
DAGMAN_DEBUG_CACHE_ENABLE A boolean value that determines if log line caching for the dagman.out file
should be enabled in the condor_dagman process to increase performance (potentially by orders of magnitude)
when writing the dagman.out file to an NFS server. Currently, this cache is only utilized in Recovery Mode.
If not defined, it defaults to False.
DAGMAN_DEBUG_CACHE_SIZE An integer value representing the number of bytes of log lines to be stored in the
log line cache. When the cache surpasses this number, the entries are written out in one call to the logging
subsystem. A value of zero is not recommended since each log line would surpass the cache size and be emitted
in addition to bracketing log lines explaining that the flushing was happening. The legal range of values is 0 to
INT_MAX. If defined with a value less than 0, the value 0 will be used. If not defined, it defaults to 5 Megabytes.
DAGMAN_PENDING_REPORT_INTERVAL An integer value representing the number of seconds that controls how
often condor_dagman will print a report of pending nodes to the dagman.out file. The report will only
be printed if condor_dagman has been waiting at least DAGMAN_PENDING_REPORT_INTERVAL seconds
without seeing any node job events, in order to avoid cluttering the dagman.out file. This feature is mainly
intended to help diagnose condor_dagman processes that are stuck waiting indefinitely for a job to finish. If not
defined, DAGMAN_PENDING_REPORT_INTERVAL defaults to 600 seconds (10 minutes).
MAX_DAGMAN_LOG This variable is described in section 3.5.3 as MAX_<SUBSYS>_LOG. If not defined,
MAX_DAGMAN_LOG defaults to 0 (unlimited size).
HTCondor attributes
DAGMAN_COPY_TO_SPOOL A boolean value that when True copies the condor_dagman binary to the spool directory when a DAG is submitted. Setting this variable to True allows long-running DAGs to survive a DAGMan
version upgrade. For running large numbers of small DAGs, leave this variable unset or set it to False. The
default value if not defined is False. Note: users should rarely change this setting.
DAGMAN_INSERT_SUB_FILE A file name of a file containing submit description file commands to be inserted into
the .condor.sub file created by condor_submit_dag. The specified file is inserted into the .condor.sub
file before the queue command and before any commands specified with the -append condor_submit_dag
command line option. Note that the DAGMAN_INSERT_SUB_FILE value can be overridden by the condor_submit_dag -insert_sub_file command line option.
DAGMAN_ON_EXIT_REMOVE Defines the OnExitRemove ClassAd expression placed into the condor_dagman
submit description file by condor_submit_dag. The default expression is designed to ensure that condor_dagman is automatically re-queued by the condor_schedd daemon if it exits abnormally or is killed (for
example, during a reboot). If this results in condor_dagman staying in the queue when it should exit, consider
changing to a less restrictive expression, as in the example
HTCondor Version 8.6.4 Manual
334
3.5.25. Configuration File Entries Relating to Security
(ExitBySignal == false || ExitSignal =!= 9)
If not defined, DAGMAN_ON_EXIT_REMOVE defaults to the expression
( ExitSignal =?= 11 || (ExitCode =!= UNDEFINED && ExitCode >=0 && ExitCode <= 2))
Metrics
DAGMAN_PEGASUS_REPORT_METRICS The path to the condor_dagman_metrics_reporter executable, which
is optionally used to anonymously report workflow metrics for Pegasus workflows.
Defaults to
$(LIBEXEC)/condor_dagman_metrics_reporter. Note: users should rarely change this setting.
DAGMAN_PEGASUS_REPORT_TIMEOUT An integer value specifying the maximum number of seconds that the
condor_dagman_metrics_reporter will spend attempting to report metrics to the Pegasus metrics server. Defaults to 100.
3.5.25 Configuration File Entries Relating to Security
These macros affect the secure operation of HTCondor. Many of these macros are described in section 3.8 on Security.
SEC_*_AUTHENTICATION This section has not yet been written
SEC_*_ENCRYPTION This section has not yet been written
SEC_*_INTEGRITY This section has not yet been written
SEC_*_NEGOTIATION This section has not yet been written
SEC_*_AUTHENTICATION_METHODS This section has not yet been written
SEC_*_CRYPTO_METHODS This section has not yet been written
GSI_DAEMON_NAME This configuration variable is retired. Instead use ALLOW_CLIENT or DENY_CLIENT as appropriate. When used, this variable defined a comma separated list of the subject name(s) of the certificate(s)
used by Condor daemons to which this configuration of Condor will connect. The * character may be used as a
wild card character. When GSI_DAEMON_NAME is defined, only certificates matching GSI_DAEMON_NAME
pass the authentication step, and no check is performed to require that the host name of the daemon matches
the host name in the daemon’s certificate. When GSI_DAEMON_NAME is not defined, the host name of
the daemon and certificate must match unless exempted by the use of GSI_SKIP_HOST_CHECK and/or
GSI_SKIP_HOST_CHECK_CERT_REGEX.
GSI_SKIP_HOST_CHECK A boolean variable that controls whether a check is performed during GSI authentication
of a Condor daemon. When the default value of False, the check is not skipped, so the daemon host name must
match the host name in the daemon’s certificate, unless otherwise exempted by the use of GSI_DAEMON_NAME
or GSI_SKIP_HOST_CHECK_CERT_REGEX. When True, this check is skipped, and hosts will not be rejected due to a mismatch of certificate and host name.
HTCondor Version 8.6.4 Manual
335
3.5.25. Configuration File Entries Relating to Security
GSI_SKIP_HOST_CHECK_CERT_REGEX This may be set to a regular expression. GSI certificates of Condor
daemons with a subject name that are matched in full by this regular expression are not required to have a
matching daemon host name and certificate host name. The default is an empty regular expression, which will
not match any certificates, even if they have an empty subject name.
HOST_ALIAS Specifies the fully qualified host name that clients authenticating this daemon with GSI should expect
the daemon’s certificate to match. The alias is advertised to the condor_collector as part of the address of the
daemon. When this is not set, clients validate the daemon’s certificate host name by matching it against DNS A
records for the host they are connected to. See GSI_SKIP_HOST_CHECK for ways to disable this validation
step.
GSI_DAEMON_DIRECTORY A directory name used in the construction of complete paths for the configuration variables GSI_DAEMON_CERT, GSI_DAEMON_KEY, and GSI_DAEMON_TRUSTED_CA_DIR, for any of these
configuration variables are not explicitly set. The value is unset by default.
GSI_DAEMON_CERT A complete path and file name to the X.509 certificate to be used in GSI authentication. If
this configuration variable is not defined, and GSI_DAEMON_DIRECTORY is defined, then HTCondor uses
GSI_DAEMON_DIRECTORY to construct the path and file name as
GSI_DAEMON_CERT
= $(GSI_DAEMON_DIRECTORY)/hostcert.pem
GSI_DAEMON_KEY A complete path and file name to the X.509 private key to be used in GSI authentication. If
this configuration variable is not defined, and GSI_DAEMON_DIRECTORY is defined, then HTCondor uses
GSI_DAEMON_DIRECTORY to construct the path and file name as
GSI_DAEMON_KEY
= $(GSI_DAEMON_DIRECTORY)/hostkey.pem
GSI_DAEMON_TRUSTED_CA_DIR The directory that contains the list of trusted certification authorities to be used
in GSI authentication. The files in this directory are the public keys and signing policies of the trusted certification authorities. If this configuration variable is not defined, and GSI_DAEMON_DIRECTORY is defined, then
HTCondor uses GSI_DAEMON_DIRECTORY to construct the directory path as
GSI_DAEMON_TRUSTED_CA_DIR
= $(GSI_DAEMON_DIRECTORY)/certificates
The EC2 GAHP may use this directory in the specification a trusted CA.
GSI_DAEMON_PROXY A complete path and file name to the X.509 proxy to be used in GSI authentication. When
this configuration variable is defined, use of this proxy takes precedence over use of a certificate and key.
GSI_AUTHZ_CONF A complete path and file name of the Globus mapping library that looks for the mapping call out
configuration. There is no default value; as such, HTCondor uses the environment variable GSI_AUTHZ_CONF
when this variable is not defined. Setting this variable to /dev/null disables callouts.
HTCondor Version 8.6.4 Manual
336
3.5.25. Configuration File Entries Relating to Security
GSS_ASSIST_GRIDMAP_CACHE_EXPIRATION The length of time, in seconds, to cache the result of the Globus
mapping lookup result when using Globus to map certificates to HTCondor user names. The lookup only occurs
when the canonical name GSS_ASSIST_GRIDMAP is present in the HTCondor map file. The default value is
0 seconds, which is a special value that disables caching. The cache uses the DN and VOMS FQAN as a key;
very rare Globus configurations that utilize other certificate attributes for the mapping may cause the cache to
return a different user than Globus.
DELEGATE_JOB_GSI_CREDENTIALS A boolean value that defaults to True for HTCondor version 6.7.19 and
more recent versions. When True, a job’s GSI X.509 credentials are delegated, instead of being copied. This
results in a more secure communication when not encrypted.
DELEGATE_FULL_JOB_GSI_CREDENTIALS A boolean value that controls whether HTCondor will delegate a
full or limited GSI X.509 proxy. The default value of False indicates the limited GSI X.509 proxy.
DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME An integer value that specifies the maximum number of seconds for which delegated proxies should be valid. The default value is one day. A value of 0 indicates that the
delegated proxy should be valid for as long as allowed by the credential used to create the proxy. The job may
override this configuration setting by using the delegate_job_GSI_credentials_lifetime submit file command.
This configuration variable currently only applies to proxies delegated for non-grid jobs and HTCondor-C jobs.
It does not currently apply to globus grid jobs, which always behave as though the value is 0. This variable has
no effect if DELEGATE_JOB_GSI_CREDENTIALS is False.
DELEGATE_JOB_GSI_CREDENTIALS_REFRESH A floating point number between 0 and 1 that indicates the
fraction of a proxy’s lifetime at which point delegated credentials with a limited lifetime should be renewed.
The renewal is attempted periodically at or near the specified fraction of the lifetime of the delegated credential.
The default value is 0.25. This setting has no effect if DELEGATE_JOB_GSI_CREDENTIALS is False or if
DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME is 0. For non-grid jobs, the precise timing of the proxy
refresh depends on SHADOW_CHECKPROXY_INTERVAL. To ensure that the delegated proxy remains valid,
the interval for checking the proxy should be, at most, half of the interval for refreshing it.
GSI_DELEGATION_KEYBITS The integer number of bits in the GSI key. If set to 0, the number of bits will be
that preferred by the GSI library. If set to less than 1024, the value will be ignored, and the key size will be the
default size of 1024 bits. Setting the value greater than 4096 is likely to cause long compute times.
GSI_DELEGATION_CLOCK_SKEW_ALLOWABLE The number of seconds of clock skew permitted for delegated
proxies. The default value is 300 (5 minutes). This default value is also used if this variable is set to 0.
GRIDMAP The complete path and file name of the Globus Gridmap file. The Gridmap file is used to map X.509
distinguished names to HTCondor user ids.
SEC_<access-level>_SESSION_DURATION The amount of time in seconds before a communication session
expires. A session is a record of necessary information to do communication between a client and daemon, and
is protected by a shared secret key. The session expires to reduce the window of opportunity where the key
may be compromised by attack. A short session duration increases the frequency with which daemons have to
reauthenticate with each other, which may impact performance.
If the client and server are configured with different durations, the shorter of the two will be used. The default
for daemons is 86400 seconds (1 day) and the default for command-line tools is 60 seconds. The shorter default
for command-line tools is intended to prevent daemons from accumulating a large number of communication
HTCondor Version 8.6.4 Manual
337
3.5.25. Configuration File Entries Relating to Security
sessions from the short-lived tools that contact them over time. A large number of security sessions consumes
a large amount of memory. It is therefore important when changing this configuration setting to preserve the
small session duration for command-line tools.
One example of how to safely change the session duration is to explicitly set a short duration for tools and
condor_submit and a longer duration for everything else:
SEC_DEFAULT_SESSION_DURATION = 50000
TOOL.SEC_DEFAULT_SESSION_DURATION = 60
SUBMIT.SEC_DEFAULT_SESSION_DURATION = 60
Another example of how to safely change the session duration is to explicitly set the session duration for a
specific daemon:
COLLECTOR.SEC_DEFAULT_SESSION_DURATION = 50000
SEC_<access-level>_SESSION_LEASE The maximum number of seconds an unused security session will be
kept in a daemon’s session cache before being removed to save memory. The default is 3600. If the server and
client have different configurations, the smaller one will be used.
SEC_INVALIDATE_SESSIONS_VIA_TCP Use TCP (if True) or UDP (if False) for responding to attempts to use
an invalid security session. This happens, for example, if a daemon restarts and receives incoming commands
from other daemons that are still using a previously established security session. The default is True.
FS_REMOTE_DIR The location of a file visible to both server and client in Remote File System authentication. The
default when not defined is the directory /shared/scratch/tmp.
ENCRYPT_EXECUTE_DIRECTORY A boolean value that, when True, causes the execute directory for jobs on
Linux or Windows platforms to be encrypted. Defaults to False. Note that even if False, the user can
require encryption of the execute directory on a per-job basis by setting encrypt_execute_directory to True in
the job submit description file. Enabling this functionality requires that the HTCondor service is run as user root
on Linux platforms, or as a system service on Windows platforms. On Linux platforms, the encryption method
is ecryptfs, and therefore requires an installation of the ecryptfs-utils package. On Windows platforms,
the encryption method is the EFS (Encrypted File System) feature of NTFS.
ENCRYPT_EXECUTE_DIRECTORY_FILENAMES A boolean value relevant on Linux platforms only. Defaults to
False. On Windows platforms, file names are not encrypted, so this variable has no effect. When using an
encrypted execute directory, the contents of the files will always be encrypted. On Linux platforms, file names
may or may not be encrypted. There is some overhead and there are restrictions on encrypting file names (see
the ecryptfs documentation). As a result, the default does not encrypt file names on Linux platforms, and the
administrator may choose to enable encryption behavior by setting this configuration variable to True.
ECRYPTFS_ADD_PASSPHRASE The path to the ecryptfs-add-passphrase command-line utility. If the path is not
fully-qualified, then safe system path subdirectories such as /bin and /usr/bin will be searched. The default
value is ecryptfs-add-passphrase, causing the search to be within the safe system path subdirectories.
This configuration variable is used on Linux platforms when a job sets encrypt_execute_directory to True in
the submit description file.
HTCondor Version 8.6.4 Manual
338
3.5.25. Configuration File Entries Relating to Security
SEC_TCP_SESSION_TIMEOUT The length of time in seconds until the timeout on individual network operations
when establishing a UDP security session via TCP. The default value is 20 seconds. Scalability issues with a
large pool would be the only basis for a change from the default value.
SEC_TCP_SESSION_DEADLINE An integer representing the total length of time in seconds until giving up when
establishing a security session. Whereas SEC_TCP_SESSION_TIMEOUT specifies the timeout for individual
blocking operations (connect, read, write), this setting specifies the total time across all operations, including
non-blocking operations that have little cost other than holding open the socket. The default value is 120
seconds. The intention of this setting is to avoid waiting for hours for a response in the rare event that the other
side freezes up and the socket remains in a connected state. This problem has been observed in some types of
operating system crashes.
SEC_DEFAULT_AUTHENTICATION_TIMEOUT The length of time in seconds that HTCondor should attempt authenticating network connections before giving up. The default imposes no time limit, so the attempt never
gives up. Like other security settings, the portion of the configuration variable name, DEFAULT, may be replaced by a different access level to specify the timeout to use for different types of commands, for example
SEC_CLIENT_AUTHENTICATION_TIMEOUT.
SEC_PASSWORD_FILE For Unix machines, the path and file name of the file containing the pool password for
password authentication.
AUTH_SSL_SERVER_CAFILE The path and file name of a file containing one or more trusted CA’s certificates for
the server side of a communication authenticating with SSL.
AUTH_SSL_CLIENT_CAFILE The path and file name of a file containing one or more trusted CA’s certificates for
the client side of a communication authenticating with SSL.
AUTH_SSL_SERVER_CADIR The path to a directory that may contain the certificates (each in its own file) for
multiple trusted CAs for the server side of a communication authenticating with SSL. When defined, the authenticating entity’s certificate is utilized to identify the trusted CA’s certificate within the directory.
AUTH_SSL_CLIENT_CADIR The path to a directory that may contain the certificates (each in its own file) for multiple trusted CAs for the client side of a communication authenticating with SSL. When defined, the authenticating
entity’s certificate is utilized to identify the trusted CA’s certificate within the directory.
AUTH_SSL_SERVER_CERTFILE The path and file name of the file containing the public certificate for the server
side of a communication authenticating with SSL.
AUTH_SSL_CLIENT_CERTFILE The path and file name of the file containing the public certificate for the client
side of a communication authenticating with SSL.
AUTH_SSL_SERVER_KEYFILE The path and file name of the file containing the private key for the server side of
a communication authenticating with SSL.
AUTH_SSL_CLIENT_KEYFILE The path and file name of the file containing the private key for the client side of a
communication authenticating with SSL.
CERTIFICATE_MAPFILE A path and file name of the unified map file.
HTCondor Version 8.6.4 Manual
339
3.5.25. Configuration File Entries Relating to Security
CERTIFICATE_MAPFILE_ASSUME_HASH_KEYS For HTCondor version 8.5.8 and later. When this is true, the
second field of the CERTIFICATE_MAPFILE is not interpreted as a regular expression unless it begins and
ends with the slash / character.
SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION This is a special authentication mechanism designed to
minimize overhead in the condor_schedd when communicating with the execute machine. Essentially, matchmaking results in a secret being shared between the condor_schedd and condor_startd, and this is used to
establish a strong security session between the execute and submit daemons without going through the usual
security negotiation protocol. This is especially important when operating at large scale over high latency networks (for example, on a pool with one condor_schedd daemon and thousands of condor_startd daemons on a
network with a 0.1 second round trip time).
The default value is True. To have any effect, it must be True in the configuration of both the execute
side (condor_startd) as well as the submit side (condor_schedd). When True, all other security negotiation between the submit and execute daemons is bypassed. All inter-daemon communication between the
submit and execute side will use the condor_startd daemon’s settings for SEC_DAEMON_ENCRYPTION and
SEC_DAEMON_INTEGRITY; the configuration of these values in the condor_schedd, condor_shadow, and
condor_starter are ignored.
Important: For strong security, at least one of the two, integrity or encryption, should be enabled in the startd
configuration. Also, some form of strong mutual authentication (e.g. GSI) should be enabled between all
daemons and the central manager or the shared secret which is exchanged in matchmaking cannot be safely
encrypted when transmitted over the network.
The condor_schedd and condor_shadow will be authenticated as submit-side@matchsession when they
talk to the condor_startd and condor_starter. The condor_startd and condor_starter will be authenticated as
execute-side@matchsession when they talk to the condor_schedd and condor_shadow. These identities is automatically added to the DAEMON, READ, and CLIENT authorization levels in these daemons when
needed.
KERBEROS_SERVER_KEYTAB The path and file name of the keytab file that holds the necessary Kerberos principals. If not defined, this variable’s value is set by the installed Kerberos; it is /etc/v5srvtab on most
systems.
KERBEROS_SERVER_PRINCIPAL An exact Kerberos principal to use.
The default value
is host/<hostname>@<realm>, as set by the installed Kerberos.
Where both
KERBEROS_SERVER_PRINCIPAL and KERBEROS_SERVER_SERVICE are defined, this value takes
precedence.
KERBEROS_SERVER_USER The user name that the Kerberos server principal will map to after authentication. The
default value is condor.
KERBEROS_SERVER_SERVICE A string representing the Kerberos service name. This string is prepended with
a slash character (/) and the host name in order to form the Kerberos server principal. This value defaults to host, resulting in the same default value as specified by using KERBEROS_SERVER_PRINCIPAL.
Where both KERBEROS_SERVER_PRINCIPAL and KERBEROS_SERVER_SERVICE are defined, the value
of KERBEROS_SERVER_PRINCIPAL takes precedence.
KERBEROS_CLIENT_KEYTAB The path and file name of the keytab file for the client in Kerberos authentication.
This variable has no default value.
HTCondor Version 8.6.4 Manual
340
3.5.26. Configuration File Entries Relating to Virtual Machines
341
3.5.26 Configuration File Entries Relating to Virtual Machines
These macros affect how HTCondor runs vm universe jobs on a matched machine within the pool. They specify items
related to the condor_vm-gahp.
VM_GAHP_SERVER The complete path and file name of the condor_vm-gahp.
$(SBIN)/condor_vm-gahp.
The default value is
VM_GAHP_LOG The complete path and file name of the condor_vm-gahp log. If not specified on a Unix platform,
the condor_starter log will be used for condor_vm-gahp log items. There is no default value for this required
configuration variable on Windows platforms.
MAX_VM_GAHP_LOG Controls the maximum length (in bytes) to which the condor_vm-gahp log will be allowed to
grow.
VM_TYPE Specifies the type of supported virtual machine software. It will be the value kvm, xen or vmware. There
is no default value for this required configuration variable.
VM_MEMORY An integer specifying the maximum amount of memory in MiB to be shared among the VM universe
jobs run on this machine.
VM_MAX_NUMBER An integer limit on the number of executing virtual machines. When not defined, the default value
is the same NUM_CPUS. When it evaluates to Undefined, as is the case when not defined with a numeric value,
no meaningful limit is imposed.
VM_STATUS_INTERVAL An integer number of seconds that defaults to 60, representing the interval between job
status checks by the condor_starter to see if the job has finished. A minimum value of 30 seconds is enforced.
VM_GAHP_REQ_TIMEOUT An integer number of seconds that defaults to 300 (five minutes), representing the
amount of time HTCondor will wait for a command issued from the condor_starter to the condor_vm-gahp
to be completed. When a command times out, an error is reported to the condor_startd.
VM_RECHECK_INTERVAL An integer number of seconds that defaults to 600 (ten minutes), representing the amount
of time the condor_startd waits after a virtual machine error as reported by the condor_starter, and before
checking a final time on the status of the virtual machine. If the check fails, HTCondor disables starting any
new vm universe jobs by removing the VM_Type attribute from the machine ClassAd.
VM_SOFT_SUSPEND A boolean value that defaults to False, causing HTCondor to free the memory of a vm
universe job when the job is suspended. When True, the memory is not freed.
VM_UNIV_NOBODY_USER Identifies a login name of a user with a home directory that may be used for job owner
of a vm universe job. The nobody user normally utilized when the job arrives from a different UID domain
will not be allowed to invoke a VMware virtual machine.
ALWAYS_VM_UNIV_USE_NOBODY A boolean value that defaults to False. When True, all vm universe jobs
(independent of their UID domain) will run as the user defined in VM_UNIV_NOBODY_USER.
VM_NETWORKING A boolean variable describing if networking is supported. When not defined, the default value is
False.
HTCondor Version 8.6.4 Manual
3.5.26. Configuration File Entries Relating to Virtual Machines
VM_NETWORKING_TYPE A string describing the type of networking, required and relevant only when
VM_NETWORKING is True. Defined strings are
bridge
nat
nat, bridge
VM_NETWORKING_DEFAULT_TYPE Where multiple networking types are given in VM_NETWORKING_TYPE, this
optional configuration variable identifies which to use. Therefore, for
VM_NETWORKING_TYPE = nat, bridge
this variable may be defined as either nat or bridge. Where multiple networking types are given in
VM_NETWORKING_TYPE, and this variable is not defined, a default of nat is used.
VM_NETWORKING_BRIDGE_INTERFACE For Xen and KVM only, a required string if bridge networking is to be
enabled. It specifies the networking interface that vm universe jobs will use.
LIBVIRT_XML_SCRIPT For Xen and KVM only, a path and executable specifying a program. When the
condor_vm-gahp is ready to start a Xen or KVM vm universe job, it will invoke this program to generate the
XML description of the virtual machine, which it then provides to the virtualization software. The job ClassAd
will be provided to this program via standard input. This program should print the XML to standard output.
If this configuration variable is not set, the condor_vm-gahp will generate the XML itself. The provided script
in $(LIBEXEC)/libvirt_simple_script.awk will generate the same XML that the condor_vm-gahp
would.
LIBVIRT_XML_SCRIPT_ARGS For Xen and KVM only, the command-line arguments to be given to the program
specified by LIBVIRT_XML_SCRIPT.
The following configuration variables are specific to the VMware virtual machine software.
VMWARE_PERL The complete path and file name to Perl. There is no default value for this required variable.
VMWARE_SCRIPT The complete path and file name of the script that controls VMware. There is no default value for
this required variable.
VMWARE_NETWORKING_TYPE An optional string used in networking that the condor_vm-gahp inserts into the
VMware configuration file to define a networking type. Defined types are nat or bridged. If a default value
is needed, the inserted string will be nat.
VMWARE_NAT_NETWORKING_TYPE An optional string used in networking that the condor_vm-gahp inserts into
the VMware configuration file to define a networking type. If nat networking is used, this variable’s definition
takes precedence over one defined by VMWARE_NETWORKING_TYPE.
VMWARE_BRIDGE_NETWORKING_TYPE An optional string used in networking that the condor_vm-gahp inserts
into the VMware configuration file to define a networking type. If bridge networking is used, this variable’s
definition takes precedence over one defined by VMWARE_NETWORKING_TYPE.
HTCondor Version 8.6.4 Manual
342
3.5.27. Configuration File Entries Relating to High Availability
VMWARE_LOCAL_SETTINGS_FILE The complete path and file name to a file, whose contents will be inserted into
the VMware description file (i.e., the .vmx file) before HTCondor starts the virtual machine. This parameter is
optional.
The following configuration variables are specific to the Xen virtual machine software.
XEN_BOOTLOADER A required full path and executable for the Xen bootloader, if the kernel image includes a disk
image.
The following two macros affect the configuration of HTCondor where HTCondor is running on a host machine,
the host machine is running an inner virtual machine, and HTCondor is also running on that inner virtual machine.
These two variables have nothing to do with the vm universe.
VMP_HOST_MACHINE A configuration variable for the inner virtual machine, which specifies the host name.
VMP_VM_LIST For the host, a comma separated list of the host names or IP addresses for machines running inner
virtual machines on a host.
3.5.27 Configuration File Entries Relating to High Availability
These macros affect the high availability operation of HTCondor.
MASTER_HA_LIST Similar to DAEMON_LIST, this macro defines a list of daemons that the condor_master starts
and keeps its watchful eyes on. However, the MASTER_HA_LIST daemons are run in a High Availability mode.
The list is a comma or space separated list of subsystem names (as listed in section 3.5.1). For example,
MASTER_HA_LIST = SCHEDD
The High Availability feature allows for several condor_master daemons (most likely on separate machines) to
work together to insure that a particular service stays available. These condor_master daemons ensure that one
and only one of them will have the listed daemons running.
To use this feature, the lock URL must be set with HA_LOCK_URL.
Currently, only file URLs are supported (those with file:. . .). The default value for MASTER_HA_LIST is
the empty string, which disables the feature.
HA_LOCK_URL This macro specifies the URL that the condor_master processes use to synchronize for the High
Availability service. Currently, only file URLs are supported; for example, file:/share/spool. Note that
this URL must be identical for all condor_master processes sharing this resource. For condor_schedd sharing,
we recommend setting up SPOOL on an NFS share and having all High Availability condor_schedd processes
sharing it, and setting the HA_LOCK_URL to point at this directory as well. For example:
HTCondor Version 8.6.4 Manual
343
3.5.27. Configuration File Entries Relating to High Availability
MASTER_HA_LIST = SCHEDD
SPOOL = /share/spool
HA_LOCK_URL = file:/share/spool
VALID_SPOOL_FILES = SCHEDD.lock
A separate lock is created for each High Availability daemon.
There is no default value for HA_LOCK_URL.
Lock files are in the form <SUBSYS>.lock. condor_preen is not currently aware of the lock files and will delete
them if they are placed in the SPOOL directory, so be sure to add <SUBSYS>.lock to VALID_SPOOL_FILES
for each High Availability daemon.
HA_<SUBSYS>_LOCK_URL This macro controls the High Availability lock URL for a specific subsystem as specified in the configuration variable name, and it overrides the system-wide lock URL specified by HA_LOCK_URL.
If not defined for each subsystem, HA_<SUBSYS>_LOCK_URL is ignored, and the value of HA_LOCK_URL is
used.
HA_LOCK_HOLD_TIME This macro specifies the number of seconds that the condor_master will hold the lock
for each High Availability daemon. Upon gaining the shared lock, the condor_master will hold the lock for
this number of seconds. Additionally, the condor_master will periodically renew each lock as long as the
condor_master and the daemon are running. When the daemon dies, or the condor_master exists, the condor_master will immediately release the lock(s) it holds.
HA_LOCK_HOLD_TIME defaults to 3600 seconds (one hour).
HA_<SUBSYS>_LOCK_HOLD_TIME This macro controls the High Availability lock hold time for a specific subsystem as specified in the configuration variable name, and it overrides the system wide poll period specified by
HA_LOCK_HOLD_TIME. If not defined for each subsystem, HA_<SUBSYS>_LOCK_HOLD_TIME is ignored,
and the value of HA_LOCK_HOLD_TIME is used.
HA_POLL_PERIOD This macro specifies how often the condor_master polls the High Availability locks to see if any
locks are either stale (meaning not updated for HA_LOCK_HOLD_TIME seconds), or have been released by the
owning condor_master. Additionally, the condor_master renews any locks that it holds during these polls.
HA_POLL_PERIOD defaults to 300 seconds (five minutes).
HA_<SUBSYS>_POLL_PERIOD This macro controls the High Availability poll period for a specific subsystem
as specified in the configuration variable name, and it overrides the system wide poll period specified by
HA_POLL_PERIOD. If not defined for each subsystem, HA_<SUBSYS>_POLL_PERIOD is ignored, and the
value of HA_POLL_PERIOD is used.
MASTER_<SUBSYS>_CONTROLLER Used only in HA configurations involving the condor_had.
The condor_master has the concept of a controlling and controlled daemon, typically with the condor_had
daemon serving as the controlling process. In this case, all condor_on and condor_off commands directed at
controlled daemons are given to the controlling daemon, which then handles the command, and, when required,
sends appropriate commands to the condor_master to do the actual work. This allows the controlling daemon
to know the state of the controlled daemon.
As of 6.7.14, this configuration variable must be specified for all configurations using condor_had. To configure
the condor_negotiator controlled by condor_had:
HTCondor Version 8.6.4 Manual
344
3.5.27. Configuration File Entries Relating to High Availability
MASTER_NEGOTIATOR_CONTROLLER = HAD
The macro is named by substituting <SUBSYS> with the appropriate subsystem string as defined in section 3.5.1.
HAD_LIST A comma-separated list of all condor_had daemons in the form IP:port or hostname:port. Each
central manager machine that runs the condor_had daemon should appear in this list. If HAD_USE_PRIMARY
is set to True, then the first machine in this list is the primary central manager, and all others in the list are
backups.
All central manager machines must be configured with an identical HAD_LIST. The machine addresses are
identical to the addresses defined in COLLECTOR_HOST.
HAD_USE_PRIMARY Boolean value to determine if the first machine in the HAD_LIST configuration variable is a
primary central manager. Defaults to False.
HAD_CONTROLLEE This variable is used to specify the name of the daemon which the condor_had daemon controls.
This name should match the daemon name in the condor_master daemon’s DAEMON_LIST definition. The
default value is NEGOTIATOR.
HAD_CONNECTION_TIMEOUT The time (in seconds) that the condor_had daemon waits before giving up on the
establishment of a TCP connection. The failure of the communication connection is the detection mechanism
for the failure of a central manager machine. For a LAN, a recommended value is 2 seconds. The use of
authentication (by HTCondor) increases the connection time. The default value is 5 seconds. If this value is set
too low, condor_had daemons will incorrectly assume the failure of other machines.
HAD_ARGS Command line arguments passed by the condor_master daemon as it invokes the condor_had daemon.
To make high availability work, the condor_had daemon requires the port number it is to use. This argument is
of the form
-p $(HAD_PORT_NUMBER)
where HAD_PORT_NUMBER is a helper configuration variable defined with the desired port number. Note that
this port number must be the same value here as used in HAD_LIST. There is no default value.
HAD The path to the condor_had executable. Normally it is defined relative to $(SBIN). This configuration variable
has no default value.
MAX_HAD_LOG Controls the maximum length in bytes to which the condor_had daemon log will be allowed to grow.
It will grow to the specified length, then be saved to a file with the suffix .old. The .old file is overwritten
each time the log is saved, thus the maximum space devoted to logging is twice the maximum length of this log
file. A value of 0 specifies that this file may grow without bounds. The default is 1 MiB.
HAD_DEBUG Logging level for the condor_had daemon. See <SUBSYS>_DEBUG for values.
HAD_LOG Full path and file name of the log file. The default value is $(LOG)/HADLog.
REPLICATION_LIST A comma-separated list of all condor_replication daemons in the form IP:port or
hostname:port. Each central manager machine that runs the condor_had daemon should appear in this
list. All potential central manager machines must be configured with an identical REPLICATION_LIST.
HTCondor Version 8.6.4 Manual
345
3.5.27. Configuration File Entries Relating to High Availability
346
STATE_FILE A full path and file name of the file protected by the replication mechanism. When not defined, the
default path and file used is
$(SPOOL)/Accountantnew.log
REPLICATION_INTERVAL Sets how often the condor_replication daemon initiates its tasks of replicating the
$(STATE_FILE). It is defined in seconds and defaults to 300 (5 minutes).
MAX_TRANSFER_LIFETIME A timeout period within which the process that transfers the state file must complete
its transfer. The recommended value is 2 * average size of state file / network rate. It
is defined in seconds and defaults to 300 (5 minutes).
HAD_UPDATE_INTERVAL Like UPDATE_INTERVAL, determines how often the condor_had is to send a ClassAd
update to the condor_collector. Updates are also sent at each and every change in state. It is defined in seconds
and defaults to 300 (5 minutes).
HAD_USE_REPLICATION A boolean value that defaults to False. When True, the use of condor_replication
daemons is enabled.
REPLICATION_ARGS Command line arguments passed by the condor_master daemon as it invokes the condor_replication daemon. To make high availability work, the condor_replication daemon requires the port
number it is to use. This argument is of the form
-p $(REPLICATION_PORT_NUMBER)
where REPLICATION_PORT_NUMBER is a helper configuration variable defined with the desired port number.
Note that this port number must be the same value as used in REPLICATION_LIST. There is no default value.
REPLICATION The full path and file name of the condor_replication executable. It is normally defined relative to
$(SBIN). There is no default value.
MAX_REPLICATION_LOG Controls the maximum length in bytes to which the condor_replication daemon log will
be allowed to grow. It will grow to the specified length, then be saved to a file with the suffix .old. The .old
file is overwritten each time the log is saved, thus the maximum space devoted to logging is twice the maximum
length of this log file. A value of 0 specifies that this file may grow without bounds. The default is 1 MiB.
REPLICATION_DEBUG Logging level for the condor_replication daemon. See <SUBSYS>_DEBUG for values.
REPLICATION_LOG Full path and file name to the log file. The default value is $(LOG)/ReplicationLog.
TRANSFERER The full path and file name of the condor_transferer executable.
$(LIBEXEC)/condor_transferer.
The default value is
TRANSFERER_LOG Full path and file name to the log file. The default value is $(LOG)/TransfererLog.
TRANSFERER_DEBUG Logging level for the condor_transferer daemon. See <SUBSYS>_DEBUG for values.
MAX_TRANSFERER_LOG Controls the maximum length in bytes to which the condor_transferer daemon log will
be allowed to grow. A value of 0 specifies that this file may grow without bounds. The default is 1 MiB.
HTCondor Version 8.6.4 Manual
3.5.28. MyProxy Configuration File Macros
347
3.5.28 MyProxy Configuration File Macros
In some cases, HTCondor can autonomously refresh GSI certificate proxies via MyProxy, available from
http://myproxy.ncsa.uiuc.edu/.
MYPROXY_GET_DELEGATION The full path name to the myproxy-get-delegation executable, installed as part of the
MyProxy software. Often, it is necessary to wrap the actual executable with a script that sets the environment,
such as the LD_LIBRARY_PATH, correctly. If this macro is defined, HTCondor-G and condor_credd will have
the capability to autonomously refresh proxy certificates. By default, this macro is undefined.
3.5.29
Configuration File Macros Affecting APIs
ENABLE_SOAP A boolean value that defaults to False. When True, HTCondor daemons will respond to HTTP
PUT commands as if they were SOAP calls. When False, all HTTP PUT commands are denied.
ENABLE_WEB_SERVER A boolean value that defaults to False. When True, HTCondor daemons will respond to
HTTP GET commands, and send the static files sitting in the subdirectory defined by the configuration variable
WEB_ROOT_DIR. In addition, web commands are considered a READ command, so the client will be checked
by host-based security.
SOAP_LEAVE_IN_QUEUE A boolean expression that when True, causes a job in the completed state to remain in
the queue, instead of being removed based on the completion of file transfer. If provided, this expression will be
logically ANDed with the default behavior of leaving the job in the queue until FilesRetrieved becomes
True.
WEB_ROOT_DIR A complete path to the directory containing all the files served by the web server.
<SUBSYS>_ENABLE_SOAP_SSL A boolean value that defaults to False. When True, enables SOAP over SSL
for the specified <SUBSYS>. Any specific <SUBSYS>_ENABLE_SOAP_SSL setting overrides the value of
ENABLE_SOAP_SSL.
ENABLE_SOAP_SSL A boolean value that defaults to False. When True, enables SOAP over SSL for all daemons.
<SUBSYS>_SOAP_SSL_PORT The port number on which SOAP over SSL messages are accepted, when SOAP
over SSL is enabled. The <SUBSYS> must be specified, because multiple daemons running on a single machine
may not share a port. This parameter is required when SOAP over SSL is enabled. There is no default value.
The macro is named by substituting <SUBSYS> with the appropriate subsystem string as defined in section 3.5.1.
SOAP_SSL_SERVER_KEYFILE The complete path and file name to specify the daemon’s identity, as used in authentication when SOAP over SSL is enabled. The file is to be an OpenSSL PEM file containing a certificate
and private key. This parameter is required when SOAP over SSL is enabled. There is no default value.
SOAP_SSL_SERVER_KEYFILE_PASSWORD An optional complete path and file name to specify a password for
unlocking the daemon’s private key. There is no default value.
HTCondor Version 8.6.4 Manual
3.5.30. Configuration File Entries Relating to condor_ssh_to_job
SOAP_SSL_CA_FILE The complete path and file name to specify a file containing certificates of trusted Certificate
Authorities (CAs). Only clients who present a certificate signed by a trusted CA will be authenticated. When
SOAP over SSL is enabled, this parameter or SOAP_SSL_CA_DIR must be set. There is no default value. The
EC2 GAHP may use this file to specify a trusted CA.
SOAP_SSL_CA_DIR The complete path to a directory containing certificates of trusted Certificate Authorit