TDA version 5.7, version 6.02, version 6.03, version 6.1, version 6.2 Data Analysis Software User's Manual

TDA User’s Manual
February 14, 2005
Götz Rohwer
Ulrich Pötter
Goetz.Rohwer @ ruhr-uni-bochum.de
Ulrich.Poetter @ ruhr-uni-bochum.de
Ruhr-Universität Bochum. Fakultät für Sozialwissenschaften, GB 1.
44780 Bochum, Germany.
contents
Contents
1
Introduction
1.1 Preface
1.2 Documentation
1.3 Installation
1.4 How to Use the Program
1.5 The Example Archive
1.6 Errors and Error Messages
1.7 Some General Commands
1.8 Command Flow Control
1.9 Programming Repetitions
1.10 User-defined Macros
2
Data Handling
2.1 Variables
2.2 Creating New Variables
2.3 Removing Variables
2.4 Recoding Variables
2.5 Dummy Variables
2.6 String Variables
2.7 Namelists
2.8 Temporary Case Selection
2.9 Writing Data Files
2.10 Data Archives
2.10.1 Creating Data Archives
2.10.2 Variable Description Files
2.10.3 Archive Description Files
2.10.4 Data Archive Commands
2.11 Special Format Data Files
2.11.1 TDA System Files
2.11.2 SPSS Files
2.11.2.1 SPSS Portable Files
2.11.2.2 SPSS Sav Files
2.11.3 Stata Files
2.12 Data File Utilities
2.12.1 Characters in a File
2.12.2 Records in a File
2.12.3 Binary Contents of File
2.12.4 Splitting Files into Parts
2.12.5 External Sorting
1
contents
2.12.6 External Merging
2.12.7 Selection of Records
2.12.8 Dropping Selected Columns
3
Data Structures
3.1 Cross-sectional Data
3.2 Panel Data
3.3 Episode Data
3.3.1 Episode Data Concepts
3.3.2 Defining Episode Data
3.3.3 An Example Data Set
3.3.4 Writing Episode Data
3.3.5 Merging Episode Data
3.4 Sequence Data
3.4.1 Sequence Data Concepts
3.4.2 Defining Sequence Data
3.4.3 Writing Sequence Data
3.4.4 Deriving Sequences from Episode Data
3.4.5 State Indicator Matrices
3.4.6 Random Sequences
3.6 Graphs and Relations
3.6.1 Terminology
3.6.2 Graph Data Structures
3.6.3 Creating Relational Data
3.6.3.1 Simple Test Data
3.6.4 Relational Data Files
3.6.4.1 Writing Relational Data Files
3.6.4.2 Creating Adjacency Matrices
3.6.5 Drawing Graphs
3.6.6 Trees
4
PostScript Plots
4.1 Preparing the Plot Environment
4.2 Plotting Coordinate Systems
4.3 Combining Plot Files
4.4 Plot Objects
4.4.1 Polygons and Scatterplots
4.4.2 Line Types and Marker Symbols
4.4.3 Plotting Text
4.4.4 Rectangles
4.4.5 Step Functions
2
contents
4.5
4.6
5
4.4.6 General Functions
4.4.7 Arrows
4.4.8 Circles, Arcs, Ellipses
4.4.9 Convex Hull
4.4.10 Smoothing Polygons
4.4.11 Contour Plots
Built-in Previewer
Ready-made Plot Commands
Mathematical Procedures
5.1 Working with Matrices
5.1.1 Introduction and Overview
5.1.2 Matrices
5.1.3 Matrix Expressions
5.1.4 Matrix Commands
5.1.4.1 Elementary Matrix Commands
5.1.4.2 Sorting and Ranking
5.1.4.3 Standardization and Scaling
5.1.4.4 Matrix Norms
5.1.4.5 Inverse Matrices
5.1.4.6 Cholesky Decomposition
5.1.4.7 Least Squares Regression
5.1.4.8 Linear Equations
5.1.4.9 Linear Inequalities
5.1.4.10 Regression with Constraints
5.1.4.11 Eigenvalues and Eigenvectors
5.1.4.12 Singular Value Decomposition
5.1.4.13 Linear Programming
5.1.4.14 Some Further Commands
5.2 Expressions
5.2.1 Introduction
5.2.2 Evaluating Expressions
5.2.3 Numerical Constants
5.2.4 Random Numbers
5.2.5 Type 1 Operators
5.2.5.1 General Type 1 Operators
5.2.5.2 Arithmetical Operators
5.2.5.3 Mathematical Operators
5.2.5.4 Logical Operators
5.2.5.5 Dummy Variables
3
contents
5.3
5.4
5.5
5.6
6
5.2.5.6 Density Functions
5.2.5.7 Distribution Functions
5.2.5.8 Julian Calendar
5.2.5.9 String Variables
5.2.6 Type 2 Operators
5.2.6.1 General Type 2 Operators
5.2.6.2 Recode Operators
5.2.6.3 Leads and Lags
5.2.6.4 Sorting and Ranking
5.2.6.5 Counting Subsets
5.2.6.6 Counting Changes
5.2.6.7 Aggregates
5.2.6.8 String Variables
5.2.7 Block Mode Operators
5.2.8 Operators for Episode Data
5.2.9 Operators for Sequence Data
Functions
5.3.1 Syntax for Functions
5.3.2 Automatic Differentiation
5.3.3 Evaluating Functions
Numerical Integration
5.4.1 Commands for Numerical Integration
5.4.2 Operators for Numerical Integration
Smoothing Procedures
5.5.1 Moving Averages
5.5.2 Running Median Smoothers
5.5.3 Smoothing Splines
Function Minimization
5.6.1 The fmin Command
5.6.2 Minimization Algorithms
5.6.3 Protocol File
5.6.4 Starting Values
5.6.5 Covariance Matrix
5.6.6 Constraints
5.6.7 Possible Difficulties
Statistical Procedures
6.1 Introduction
6.1.1 Missing Values
6.1.2 Case Weights
4
contents
6.2
6.4
6.5
6.6
6.7
6.9
Elementary Descriptive Procedures
6.2.1 Mean and Range of Variables
6.2.2 Empirical Distribution Functions
6.2.3 Quantiles
6.2.4 Frequency Tables
6.2.5 Aggregated Tables
6.2.6 Covariance and Correlation
6.2.7 Contingency Measures
6.2.8 Scatterplots
6.2.9 Histogram Plots
6.2.10 Density Plots
Concentration and Inequality
6.4.2 Inequality Measures
6.4.3 Segregation Indices
Describing Episode Data
6.5.1 Life Tables
6.5.2 Kaplan-Meier Procedure
6.5.3 Survivor Function Quantiles
6.5.4 Comparing Survivor Functions
6.5.5 State Distributions
Describing Sequence Data
6.6.1 Sequence Length and Gaps
6.6.2 General Characteristics
6.6.3 State Distributions
6.6.4 Pattern Matching
Investigating Proximities
6.7.2 Sequence Proximity Measures
6.7.2.1 Optimal Matching
6.7.2.2 The seqm Command
6.7.2.3 Examples
6.7.2.4 Consecutive Indel Operations
6.7.2.5 Data-based Substitution Costs
6.7.2.6 Working with Large Samples
6.7.2.7 Comparing Parallel Sequences
Least Squares Regression
6.9.1 The lsreg Command
6.9.1.1 Residuals
6.9.1.2 Parameter Constraints
6.9.1.3 Dummy Variables
6.9.2 Regression with Censored Data
5
contents
6.10 Alternative Regression Methods
6.10.1 L1-Norm Regression
6.10.2 Nonparametric Regression
6.11 Maximum Likelihood Estimation
6.11.1 Introduction
6.11.2 The fml Command
6.12 Quantal Response Models
6.12.1 The qreg Command
6.12.2 Binary Logit and Probit
6.12.2.1 Grouped Data
6.12.3 Ordinal Logit and Probit
6.12.4 Multinomial Logit
6.12.5 Multivariate Probit
6.14 Models for Count Data
6.14.1 Poisson Regression
6.14.2 Compound Poisson Models
6.16 Nonlinear Regression
6.16.1 The freg Command
6.17 Transition Rate Models
6.17.1 Introduction
6.17.1.1 Parametric Rate Models
6.17.1.2 Maximum Likelihood Estimation
6.17.1.3 Time-varying Covariates
6.17.1.4 The rate Command
6.17.1.5 Relative Risks
6.17.1.6 Generalized Residuals
6.17.2 Exponential Models
6.17.2.1 The Basic Exponential Model
6.17.2.2 Models with Time Periods
6.17.2.3 Period-specific Effects
6.17.3 Parametric Duration Dependence
6.17.3.1 Polynomial Rates
6.17.3.2 Gompertz-Makeham Models
6.17.3.3 Weibull Models
6.17.3.4 Sickle Models
6.17.3.5 Log-logistic Models
6.17.3.6 Log-normal Models
6.17.3.7 Generalized Gamma Models
6.17.3.8 Inverse Gaussian Models
6.17.4 Mixture Models
6
contents
6.17.4.1 Gamma Mixture Models
6.17.5 User-defined Rate Models
6.17.5.1 The frml Command
6.17.5.2 The Hernes Model
6.17.5.3 The Coale-McNeil Model
6.17.5.4 Models with Several Domains
6.17.6 Discrete Time Rate Models
6.17.6.1 Introduction
6.17.6.2 Logistic Regression Models
6.17.6.3 Complementary Log-Log Models
6.17.7 Semi-parametric Rate Models
6.17.7.1 Partial Likelihood Estimation
6.17.7.2 Time-dependent Covariates
6.17.7.3 Episode Splitting
6.17.7.4 The Proportionality Assumption
6.17.7.5 Stratification
6.17.7.6 The Baseline Rate
6.18 Regression Models for Events
6.18.1 A Simple Modeling Approach
6.18.2 Investigating Events
6.18.3 Data Generation
6.19 Loglinear Models
6.19.1 Contingency Tables
6.19.2 Loglinear Models
7
Relational Data
7.1 Introduction and Overview
7.2 General Graph Algorithms
7.2.1 Direct Links
7.2.1.1 Degree of Nodes
7.2.1.2 Direct Links
7.2.2 Partial Orders
7.2.2.1 Topological Sorting
7.2.3 Connectivity
7.2.3.1 Connected Components
7.2.3.2 Reachable Nodes in Digraphs
7.2.3.3 Cut Nodes and Blocks
7.2.4 Paths
7.2.4.1 Enumeration of Paths
7.2.4.2 All Shortest Paths
7
contents
7.2.4.3 Transitive Closure
Spanning Trees
7.2.5.1 Depth-first Spanning Trees
7.2.5.2 Minimum Spanning Trees
7.2.5.3 Enumeration of Spanning Trees
7.2.6 Cycles
7.2.6.1 Fundamental Set of Cycles
7.2.6.2 Enumeration of Cycles
7.2.7 Eigenvalues and Eigenvectors
7.2.8 Flows
7.2.8.1 Maximal Flows
7.2.9 Subgroups
7.2.9.1 Cliques
7.2.9.2 Compact Sets
7.2.9.3 Independent Sets
Combinatorial Optimization
7.3.1 Assignment Problems
7.3.1.1 Column Permutations
7.3.1.2 Quadratic Assignment
Representation of Proximities
7.4.2 Spectrum-based Projections
7.4.2.1 Direct Projection of Proximities
7.4.2.2 Spectrum-based Configurations
7.4.4 Comparing Configurations
7.4.4.1 Procrustes Rotation
7.4.5 Tree Representations
7.4.5.1 Hierarchical Trees
Clustering Procedures
7.5.1 Hierarchical Agglomerative Clustering
7.5.1.1 SAHN Algorithms
7.5.1.2 Nearest-Neighbor Clustering
7.5.2 Hierarchical Divisive Clustering
7.5.2.1 Maximally Different Cluster Centers
7.5.2.2 Partition with Minimal Diameter
7.5.3 Non-Hierarchical Clustering
7.5.3.1 The Bond Energy Approach
Social Network Analysis
7.6.1 Characterizing Nodes
7.6.1.1 Direct and Indirect Control
7.6.1.2 Integrated Ownership
7.2.5
7.3
7.4
7.5
7.6
8
contents
7.6.2
8
7.6.1.3 Measures of Flow Control
Structural Similarity
Set-valued Data
8.1 Introduction and Overview
8.2 Interval Expressions
8.2.1 Definition of Interval Expressions
8.2.2 Evaluation of Interval Expressions
8.3 Functions with Interval Arguments
8.3.1 Definition of Interval Functions
8.3.2 Evaluation of Inclusion Functions
8.4 Range of Interval Functions
8.4.1 The gmin Command
8.4.2 The range Command
8.5 Distribution Functions
8.5.1 Set-valued Discrete Variables
8.5.2 Interval-valued Variables
8.6 Descriptive Statistics
8.6.1 Mean Values
8.6.2 Variances
References
Index
9
1
1
introduction
1. Introduction
This introductory part of the TDA User’s Manual contains the following
sections.
1.1
Preface
1.2
Documentation explains the structure of this manual and our
usage of cross-references.
1.3
Installation explains how to install the program on different platforms.
1.4
How to Use the Program provides basic information on invoking
the program and using its commands.
1.5
The Example Archive explains how to find the examples that are
part of the program’s distribution.
1.6
Errors and Error Messages discusses errors that may occur when
using the program.
1.7
Some General Commands describes a few commands that may
be helpful in different contexts.
1.8
Command Flow Control describes commands that can be used
to control command flow, e.g., while and repeat.
1.9
Programming Repetitions describes some commands that can be
used to program repetitions, e.g., for bootstrap applications.
1.10
Using Macros explains how to define and use macros.
d01.tex
December 20, 1999
1.1
preface
1
1.1 Preface
TDA, an abbreviation of Transition Data Analysis, is a computer pro-
gram for statistical computations. Developing the program is a work in
progress. First versions have been developed in 1989. Since then, several
new features have been added. With version 5.7 (June 1994) the program has reached some state of maturity. This is reflected in the book
by Blossfeld and Rohwer [1995] providing an introduction to techniques
of event history modeling based on TDA. In fact, this book is supplemented by a disk containing, together with a lot of examples, version 5.7
of the program.
We then experimented with several different approaches to develop
a more flexible and easier to use version of the program. In version 5.7
there was no clear distinction between commands, in the sense of basic
procedures, and additional parameters to control the behavior of these
commands. In the preliminary new versions, 6.02 and 6.03, we tried
to cope with this drawback by introducing the concept of sections (of
a command file). However, this idea turned out to be very unwieldy
and, beginning with version 6.1, we therefore follow a different approach:
There is now a clear distinction between commands and parameters.
Each command now represents a single procedure.
Another drawback of the older version has been that the user had
only limited possibilities to influence the order of command execution.
Based on the distinction between commands and parameters there is now
a simple solution. Each command is executed as a separate procedure,
and commands are executed in the same order as given by the user.
As will be explained in the remainder of this documentation, many
more new features have been added to the program. There is no longer a
limited focus on transition data, but we try to make the program useful
for a broad variety of applications. We also mention that, beginning with
version 6.2, the program and its documentation have two authors: Götz
Rohwer and Ulrich Pötter.
Copyright and Distribution. TDA is a non-commercial project. The
program is distributed as freeware under the terms of the GNU General
Public License (GPL) and can be distributed freely according to this
license. Each TDA distribution should contain a file, named copying,
d0101.tex
April 19, 1998
1.1
preface
2
providing a full statement of the GNU General Public License.
TDA contains some functions, or parts of functions, that have been
taken from published sources. Whenever this is the case, this has been
explicitly noted in TDA’s source code and the documentation.1 The copyright for these parts is, of course, with their authors.
Finally, we have to add the usual disclaimer. We hope that the program will prove to be efficient and reliable. However, as with all complex
programs it is impossible to test its behavior for all possible run-time
conditions and environments.2 Most probably, there are still some errors
in the current version of the program. Therefore, we have to add the
following disclaimer: TDA is distributed without any warranty; without
even the implied warranty of merchantability or fitness for a particular
purpose.
Acknowledgments. Many people and institutions have given support
to the TDA project. We thank the Hamburger Institut für Sozialforschung
(Hamburg) for support of the first steps in the development of TDA. The
program was further developed as part of the research project Household Dynamics and Social Inequality, directed by Hans-Peter Blossfeld,
and supported by grants from the European University Institute (Florence) and the European Commission (Brussels), and then continued
at the University of Bremen. We also thank a number of people who
have supported the development of TDA, or have given useful comments: Marco Becht, Stefan Bender, Francesco Billari, Hans-Peter Blossfeld, Hannah Brückner, Josef Brüderl, Tak Wing Chan, Juri Ciborra,
Sonja Drobnič, Greg Duncan, Christian Dustmann, Beate Ernicke, John
Haisken-DeNew, Jan Hoem, Steffen Kühnel, Anders Holm Larsen, Søren
Leth-Sørensen, Wolfgang Ludwig-Mayerhofer, John Micklewright, Simon
Peters, Rainer Pischner, Ulrich Rendtel, Hilmar Schneider, Gilles Teyssiere,
Andreas Timm, Wolfgang Voges, Achim Wackerow. Karl Ulrich Mayer
(Max Planck Institut für Bildungsforschung, Berlin) and Hans-Peter
1 A complex statistical program consists of several different parts. With respect to
numerical problems, one can distinguish three stages. First the framework providing
the command language to control the program, means to handle input and output of
data, and so on. Second, some algorithms that are, in general, also not numerically
critical, for instance calculation of log-likelihood functions and their derivatives. And
finally some numerically (or for other aspects) critical algorithms, for instance matrix
inversion and function minimization. A basic decision has been to take all critical
algorithms from published sources in the literature and to document this explicitly
in the program’s manual.
2 The current version of the program has been developed and tested on a SUN SparcStation 10/30. Its behavior on other machines has not been tested.
1.1
preface
3
Blossfeld (University of Bremen) have kindly provided some example
data (from the Lebensverlaufstudie) for many examples in the documentation.
Finally, we should look forward. To become a useful program, the
development of TDA needs some response from its users. We would be
glad to receive information about problems and bugs and, of course,
suggestions for improving the program.
Götz Rohwer
Ulrich Pötter
1.2
documentation
1
1.2 Documentation
Development of TDA will be documented in a history.txt file and in
a User’s Manual. The history.txt file keeps track of all changes in a
chronological order. Whenever we change some part of the program or
the User’s Manual, and when we add some new features, there should
be a corresponding entry in the history.txt. This file should be part of
each distribution of the program to allow users to update their version
of the program and its documentation.
Since TDA is a work in progress, we make its User’s Manual something like a loose-leaf book that consists of a set of (more or less) related
single documents (URLs in the WWW’s HTML jargon). An overview of
all available documents can be found in the current table of contents.
Identification and ordering of the single documents follow a hierarchical
scheme with up to four levels, roughly corresponding to parts, chapters, sections, and subsections. The first page of each document shows
its name and the date of its last update. Names of the documents are
created as
dl1 .tex
dl1 l2 .tex
dl1 l2 l3 .tex
dl1 l2 l3 l4 .tex
first level
second level
third level
fourth level
For example, the current document is d0102.tex. The extension, .tex,
shows that we use LaTeX to create the documents. Actually, the distributed files are PostScript files and have the extension .ps. We hope
that this strategy allows for an easy update of the User’s Manual. It
should only be necessary to make print outs of new documents, or documents that have been changed since the last release. Information about
what is new and what has been changed can be found in the history.txt
file.
Page numbering is done separately for each document. Headers show
the document and page numbers. Since most of the single documents are
very short, cross-references are given only by document numbers. Also
the subject index uses only document numbers, no page references.
d0102.tex
December 18, 1999
1.3
installation
1
1.3 Installation
TDA is written in the programming language C. Current development is
on a SparcStation 10/30 in a UNIX environment. However, since the pro-
gram is command-driven and only marginally relies on platform-specific
features, it is highly portable. The distribution package contains the
source code as well as some precompiled versions. Installation is particularly easy if an executable version of the program is already available. With the exception of DOS where the program needs an additional
DOS-extender, it will then consist of a single executable file. In order
to install the program one simply needs to copy the executable file into
a suitable directory of the target machine. Of course, the distribution
package might contain the program in a compressed file format and it
will then be necessary first to uncompress the file.
DOS. A precompiled version for DOS is created with Watcom’s C compiler and called tda.exe. In order to be executable it needs an additional
DOS-extender. The distribution package contains dos4gw.exe, a DOSextender made available by Watcom as part of its C compiler package.
Installation requires to copy both files, tda.exe and dos4gw.exe, into a
directory of the target machine.
The DOS-extender allows TDA to assess extended memory and work
as a 32-bit program. This sometimes conflicts with other programs when
they try to use the same resources. In any case will it be a good idea to
have a look into the file dos4gw.doc (a plain ASCII file) that describes
several options to re-configure the DOS-extender.
Windows-NT. An executable version of TDA, again compiled with
Watcom’s C compiler, is available. An additional DOS-extender is not
needed. Of course, since TDA is a command-driven program, it must be
executed in a suitable shell.
LINUX. The distribution package of TDA contains an executable version for LINUX. The read.me file will give information about the version
of LINUX and the compiler used to compile the program. If this might
not fit with the target LINUX system, it will be be easy to re-compile
the program on the target machine (see below).
UNIX Workstations. For most UNIX workstations there will be no
d0103.tex
December 18, 1999
1.3
installation
2
precompiled versions of the program. However, assuming that a C-compiler is available, it will be very easy to create an executable version for
the target machine. As part of the TDA distribution package comes an
archive that contains the complete source and also a makefile. Simply
extract all files from this archive, then edit the makefile and also have
a look into the main include file, tda.h, in order to check for some
adjustments that might be necessary for the target machine. A final
make tda will then create an executable version of the program.
1.4
1
how to use the program
1.4 How to Use the Program
TDA is a command-driven program. The normal way to invoke the pro-
gram is to write, in a suitable shell, the command line
tda command1 command2 ...
TDA then executes the commands given in the command line and finally
terminates. Each command is executed separately, in the order given
in the command line (from left to right). If invoked without any commands, the program only shows a short identification message and then
terminates.
Some output is always written into the program’s standard output.
Default is the terminal’s screen. This standard output can be redirected
into an output file by using the syntax
tda command1 command2 ... > name of an output file
To append the standard output to an already existing file one can use
‘>>’ instead of ‘>’. Note that some commands create separate output files
to be used for additional output.
Each command given in TDA’s command line can be a standard
executable command or a special command providing the name of a
command file. This special command is
cf=name of a command file
A command file is a simple text file containing commands to be executed
by the program. A command must begin with a valid keyword identifying the command and must be terminated with a semicolon (;). All text,
beginning with the command’s keyword until the final semicolon, is interpreted as belonging to the same command. Any text following a #
character, until the end of the current line, is interpreted as a comment
and not executed.
Both ways of providing commands can be combined. For instance,
invoking the program as
tda command1 cf=file1 command2 cf=file2 command3 ...
d0104.tex
December 18, 1999
1.4
how to use the program
2
would first execute command1, then read the command file file1 and
execute the commands found in that file, then execute command2, then
execute the commands in file2, and so on. cf commands can also be
used inside of a command file. Nesting of cf-commands inside of command files is possible up to 10 levels.
Each command begins with a keyword identifying the command that
must be given in lower-case letters. Many commands also need or allow
for additional parameters. There are basically four syntactical forms.
1. Commands without any parameters:
keyword;
where keyword identifies the command.1 For example, time is a command without parameters.
2. A second syntactical form is
keyword = rpar1,rpar2,... ;
where keyword identifies the command and rpar1,rpar2,... are additional parameters that must be separated by commas. As an example,
consider the command
pdata = fname;
In this example, pdata is the keyword identifying the command (print
data), and the parameter fname provides the name of an output file to
be used for writing the data.
3. A third syntactical form is
keyword (par1,par2,...);
where again keyword identifies the command and par1,par2,... are
additional parameters. The brackets indicate that the parameters belong
to the same command. They must be separated by commas.
4. Finally, combining the second and third forms leads to the general
command structure
keyword (par1,par2,...) = rpar1,rpar2 ;
1 When explaining commands we always use a terminating semicolon as required in
command files. A semicolon must not be used when the command is given directly
in the program’s command line.
1.4
how to use the program
3
Box 1 Commands for changing some limitations
maxnv=...
maxmat=...
maxstd=...
cblen=...
maximum number of variables, def. 2000
maximum number of matrices, def. 200
stack size for derivatives, def. 20
command buffer length, def. 20000
Most TDA commands have this general structure. Note that there are two
types of parameters. If parameters have default values they are optional,
and when not explicitly given, TDA uses their default values instead. On
the other hand, some parameters do not have default values and must be
provided by the user. For example, the fname parameter for the pdata
command has no default value; the user must explicitly provide the name
of an output file.
If all parameters on the right-hand side are optional, the command
can be used as shown in (3); if all parameters inside the left-hand side
brackets are optional, the command can be given as shown in (2); and
if all parameters are optional, one can simply use the keyword without
any parameters.
Upper and Lower-case Letters. TDA is case-sensitive and distinguishes between upper and lower-case letters. Commands and operators
must always be given in lower-case letters. There is basically only one
exception: variable names must begin with an upper case letter, or with
a special character, as will be explained in 2.1.
Termination. Normally, the program executes one command after the
other until execution of the final command is completed. However, each
command has a return status indicating whether the command executed
successfully. As a default, when a command returns with an error, the
program terminates and does not execute any further commands. This
default behavior can be changed with the ierr command. For instance,
given
tda ierr command1 command2 ...
the program will try to execute all commands regardless of whether some
commands return with an error.
Online Use of TDA. There is the possibility to use TDA in such a
way that the program remains loaded. This is achieved by invoking the
1.4
how to use the program
4
program as
tda i
The program then shows up with a prompt (:). Following the prompt,
one can input commands. A command can be continued on several lines
and must finally be terminated with a semicolon. The command will then
be executed and, having given the standard output for that command,
the program will show up again with its prompt to await new commands.
Termination is with “quit” or “exit”.
Limitations. Most commands have some limitations due to the fact
that the program stores all data in internal memory and must then
know in advance the maximum amount of data to be expected. Some of
these limitations can be controlled by optional parameters. For instance,
when defining new variables, one can use the noc parameter to provide
an upper limit for the maximum number of cases (default is 1000).
However, some limitations already occur when the program initializes
its basic data structures before executing any commands. In particular,
the maximum number of variables, string variables, and matrices cannot
be changed after initialization of the basic data structures. Some of these
limitations can be modified by using the commands shown in Box 1.
These commands must be given in the command line when invoking the
program.
1.5
the example archive
1
1.5 The Example Archive
Command files and data files for all examples used in the User’s Manual
to illustrate some featurs of TDA are contained in the program’s example
archive, called exam.zoo in the distribution package. One can use the
ZOO archive program to list the contents and to extract all or part of the
files contained in the archive. With the exception of a few files that illustrate TDA’s options for using compressed data archives, all files in the
example archive are ASCII text files and can be edited. Most probably,
the files will be UNIX text files, meaning that the end-of-line character
follows the UNIX convention (that is, a single linefeed character). However, TDA should be able to read data and command files regardless of
the end-of-line convention used to create the files.
The example archive contains additional data and command files not
explicitly referred to in the User’s Manual. In particular, there are several
command files that have been used to create the PostScript plots in the
manual. They might be interesting for people who want to create similar
plots.
The example archive also contains a command file, test.cf, that
simply consists of a sequence of cf commands to call most of the other
command files. Running TDA with test.cf will then sequentially execute most of the command files contained in the example archive. This
might be helpful in order to check whether the program performs correctly.
There is a second example archive, ehhnew.zoo, that contains the command files discussed in Blossfeld and Rohwer [1995], updated for the new
syntax of TDA.
d0105.tex
April 19, 1998
1.6
errors and error messages
1
1.6 Errors and Error Messages
There are two types of errors. First, the user can make errors, meaning
that he or she is not using the program correctly. Second, the program
can make errors, meaning that the program does not perform as it is
expected to do, or does not give proper messages when something goes
wrong. In principle, the program should provide useful error messages
whenever something goes wrong. However, this is not always the case,
and when users find errors or insufficient error messages, they should
send us a note.
Since most error messages should be understandable without further
explanation, it seems not necessary to list them here. However, we give
a few remarks about the main types of errors. (In addition to what is
described below, there are some “fatal” error messages indicating serious
problems in the program’s source code. If such a message occurs, one
should always contact the authors.)
Syntax Errors. The most common type of errors is syntax errors meaning that there is some mistake in providing commands to the program.
When these errors occur, the user should check the command’s definition. As a general rule, the key words for all commands must be given in
lower case letters. On the other hand, variable names must begin with
an upper case letter in order to be distinguishable from commands. It is
also important that each command in a command file is terminated by
a semicolon.
Errors in expressions. Expressions are used to define variables, functions, or case select statements. Two types of errors can occur. First,
syntax errors, meaning that the definition of an expression cannot be
parsed correctly by the program. We have tried to make the syntax for
expressions to some degree flexible, but there are some restrictions in
using and combining operators. In particular, whenever using variables
as part of expressions, they must already be defined.
A second type of error can occur when it is not possible to evaluate
an expression, for instance, to calculate the logarithm of a negative or
zero argument. These errors will only be detected when actually using
the expression. As a general rule, the program will then give an error
message (hopefully indicating the type of error) and stop the current
d0106.tex
April 19, 1998
1.6
errors and error messages
2
procedure. It is intended that the program’s internal status is then the
same as at the beginning of the procedure. But most probably this will
not always be so.
Errors in reading and writing files. If the program is given a command to read a file (command file or data file), it first tries to open the
file. If the file name is not preceded by a path, the file is searched in
the current directory. If not found there, the error message will be can’t
open ... Errors can also occur while reading a file; in particular when the
file does not contain proper records, or when the record length exceeds
the maximum (20000 characters), or when the file contains unexpected
characters.
Only two types of errors may occur when writing files. First, when
the name given for the output file does not conform to the rules for
file names as expected by the operating system where the program is
running; and second, when there is not enough space for the file on the
disk.
Errors when exceeding internal limits. Although TDA has been
developed for use with large models and large data sets there are some
limits. Some of these limits can be changed when invoking the program
(see 1.4), others can only be changed when compiling the program (almost all definitions can be found in the include file tda.h). In any case,
the program should give an error message when one of these internal
limits is reached.
Insufficient memory. Another type of error occurs when there is insufficient memory. TDA is based on dynamic memory allocation, meaning that, whenever the program requires some memory, this amount of
memory is requested from the operating system, and when the memory
is no longer needed, it is given back to the system. As a consequence
of dynamic memory allocation, memory may be exhausted any time
during the execution of the program, not just during its initialization
phase. When this happens, the program will give an insufficient memory message and stop with the currently performed procedure. Again,
it is intended that the program’s internal status is then the same as at
the beginning of the procedure. But there are chances that this will not
always be so.
Errors in numerical algorithms. Finally, errors can occur when executing numerical algorithms. There are three types of problems. First,
the program may not be able to execute and finish an algorithm in the
1.6
errors and error messages
3
expected way. For instance, this can always happen when maximizing
a likelihood with one of TDA’s iterative algorithms. The program may
not be able to find a (local) maximum, or to determine the maximum
with the required accuracy. The program should give, then, an appropriate error message. A second type of problem regards the numerical
behavior of an algorithm. This includes rounding errors, numerical overflow and underflow, and numerical instabilities. These problems will be
discussed separately. Finally, there are simply unknown errors due to insufficient practical experience with the algorithms as they are currently
implemented in the program.
1.7
some general commands
1
1.7 Some General Commands
There are a few general commands which are sometimes useful. Box 1
provides a summary.
1. TDA is based on dynamic memory allocation, meaning that the program only requests memory from the operating system when this
is actually needed; and when this memory is no longer needed, it
is given back to the system. Most commands need some memory
during execution but it should be no longer allocated when command execution has finished. There are, of course, some exceptions.
For example, when defining new variables, memory used for the corresponding data remains allocated until the variables are explicitly
removed with the clear command (see 2.3).
The mem command can be used to get information about the currently
used memory. Messages about currently used memory do not include
the memory used by the operating system to load the program. Only
the amount of memory explicitly allocated by the program is shown.
This is done first immediately after the program has initialized is basic data structures. The amount of allocated memory shown when the
program terminates should be equal. As an additional information
when terminating, the program also shows the maximum amount of
memory used during execution of commands.
2. The time command prints the current date and time into the program’s standard output and might be useful to get information about
the execution time of commands.
3. If the ierr command is given in the command line, the program tries
to execute all following commands regardless of whether they return
with an error.
4. The print command can be used to print some text into the program’s standard output. The syntax is
print (text) ;
Note that TDA always removes blank characters from strings. To
keep these characters, text should be enclosed in single or double
quotation marks.
d0107.tex
December 19, 1999
1.7
some general commands
2
Box 1 General commands
mem;
time;
ierr;
print(text);
data;
data1;
mach;
silent;
$string;
help;
shows current memory usage
shows current date and time
ignore return status of commands
print text into standard output
info about current data matrix
info about currently available variables
info about machine parameters
controls amount of standard output
invokes a shell to execute string
online help
5. The data and data1 commands provide information about the currently defined data matrix. data only shows the number of cases and
variables; data1 additionally shows names and definition of variables.
6. The mach command can be used to get information about some
parameters of the computer where TDA is running. This includes
information about largest and smallest floating point numbers and
the machine’s epsilon (precision). Calculation is based on algorithms
adapted from Malcolm [1972] and Gentleman and Marovich [1974].
7. The silent command controls the amount of information written
into the program’s standard and error output. By default, silent=0.
If silent=1 nothing is written into the standard output, and if
silent=2 nothing is written into the error output. Finally, if silent=3,
neither the standard nor the error output is used.
A further option is provided by silent=-1. By default, matrix commands, and also while, repeat, if, and break commands, do not
echo in the standard output (except when errors occur). While testing command files, it might be useful that also these commands give
some echo and this can be requested with silent=-1.
8. The command
$ string ;
invokes a shell in order to execute string. For example, $dir would
show the current directory. This command is most useful when working in the online mode (tda i, see 1.4). Note that string must be
enclosed in quotation marks in order to preserve blank characters.
1.7
some general commands
3
9. The help command is for TDA’s online help. This requires that the
program has access to the tda.hlp file that contains the help information. If given as
help;
the command shows the entry for the help command that explains
how to use the command. Otherwise, the command is
help [=] string;
The command then shows the entry that has a keyword matched by
string. If there is more than one keyword matched by string, the
command only shows the available keywords. Note that string may
contain the wild cards (.,?,*).
1.8
1
command flow control
1.8 Command Flow Control
This section describes a few commands that may be helpful to control
the flow of commands in a command file.
Repeat – endrepeat
There are two possibilities to use a repeat–endrepeat construction. The
first is:
repeat (n = expression);
...
endrepeat;
In this construction, expression must be a scalar expression that evaluates to a positive integer, say n. The commands in between repeat and
endrepeat are then repeated unconditionally n times.
A second possibility is to use the repeat command in the form
repeat (n = expression, I );
where I is an arbitrary matrix name. The command then creates a (1, 1)Matrix with this name and while repeating the loop sets I(1,1) to the
value of the iteration counter.
While – endwhile
For loops that might depend on a condition one can use the following
construction:
while (expression);
...
endwhile;
The commands in between while and endwhile are repeated as long as
expression is true, i.e., evaluates to a non-zero value. expression can
be any standard or matrix expression that results in a scalar value.
d0108.tex
December 20, 1999
1.8
command flow control
2
If – endif
To check for conditions one can use the construction
if (expression);
...
endif;
The commands in between if and endif are executed if expression is
true, i.e., evaluates to a non-zero value. expression can be any standard
or matrix expression that results in a scalar value.
Break
Inside repeat and while loops one can use the command
break;
All commands following a break command are skipped until the next
endrepeat or endwhile command (whatever applies) is reached.
Nesting
The repeat, while, and if commands can be nested up to a level of
100. (This number is defined by MaxRep in the header file tda.h and can
be changed when compiling the program.)
1.9
1
programming repetitions
1.9 Programming Repetitions
This section describes a few commands that might be helpful for command repetitions as used, for example, in bootstrap applications or other
methods of variance estimation.
Case Selection for Repetitions
There are two related commands. The first one is
dblock (mdef=...) = varlist;
All parameters are optional. If provided, varlist must be a list of
comma-separated variable names and the mdef parameter must specify a valid matrix name. The command first turns of any possibly active
tsel (temporary case) selection command. It then creates a data structure that keeps record of blocks in TDA’s internal data matrix, defined
by the variables in varlist. Blocks are defined as contiguous blocks of
data matrix rows where the variables in varlist have identical values;
or, if varlist is empty, each data matrix row is treated as a separate
block.
The dblock command sets the global variable BNOC to the number of
blocks found in the data matrix. This number is available with the type
1 operators bnoc (see 5.2.5.1).
If the optional parameter
mdef = M,
is used, where M is a valid matrix name, the dblock command creates a
(NOC,1)-matrix with the specified name, say M, and sets
M(i,1) = block number of case (data matrix row) i
NOC is the global variable that keeps record of the number of currently
selected cases (rows) in TDA’s internal data matrix.
The second command is
repsel (id = V) = S;
d0109.tex
December 22, 1999
1.9
programming repetitions
2
where S is a matrix expression with dimension (BNOC,1). The command
requires that BNOC has previously be defined with the dblock command.
Like the dblock command, the repsel command turns off any possibly
active tsel (temporary case) selection.
The repsel command works in two different ways depending on whether
the optional id parameter is used.
1. If the id parameter is not used, the command creates a new data
matrix (actually it creates a filter for the current data matrix) where
the cases (rows) in block i (i = 1, . . . , BNOC) are repeated S(i, 1) times,
or omitted if S(i, 1) ≤ 0.
2. If the id parameter is used, it must specify a valid variable name, say
V. The repsel command then creates a new data matrix where block
i consists of those cases (rows) where the value of V equals S(i, 1).
Finally note that the repsel command remains active until: (1) a new
repsel command is given, or (2) the data matrix is removed, or (3) a
tsel command is given, or (4) a dblock command is given, or (5) the
command is explicitly turned off with the command
repsel = off;
Saving Data Matrix Blocks in a Matrix
The comamnd
mdefb(B,expression);
can be used to save a block of data matrix rows in a matrix. The command requires that the dblock command has been used previously to define data matrix blocks. B must be a valid matrix name, and expression
must be a scalar expression that evaluates to a valid block number, say
n. The mdefb command then creates a matrix named B and copies block
number n from the current data matrix into this matrix.
1.10
1
user-defined macros
1.10 User-defined Macros
Defining Macros
A macro can be defined with the command
macrodef(MNAME) = { string1; string2; ... };
MNAME is an arbitrary string to be used as the name of the macro.
string1, string2, ... must be valid TDA commands, or already defined
macro names. MNAME then becomes a new command, and
MNAME;
is executed as
string1;
string2;
...
string1, string2, ... may contain variable arguments having the predefined names:
$1, $2, ..., $100
The maximal number of arguments equals the maximal number of macros,
defined by the constant MaxMacro in the header file tda.h. (The value is
currently set to 100.) When defining a macro, variable arguments must
be contiguous, that is, $1, $2, $3, ... Apart from this requirement,
they can be used in any way. When invoking the macro name with arguments, TDA uses the following ordering:
MNAME(arg n+1, arg n+2, ...) = arg 1, arg 2,...,arg n;
arg i is then substituted for $i in the macro definition. Arguments can
be skipped by using consecutive commas. The corresponding argument
in the macro definition will then be empty. (This can be checked with
the exists operator, see 5.2.5.1.)
d0110.tex
December 23, 1999
1.10
user-defined macros
2
Local Matrix Names
Inside a macro definition one can use the command
local(A,B,C,...);
where A,B,C,... are matrix names. The command declares the specified
matrix names as being only locally defined. Then, using these names for
matrix operations inside the macro will not conflict with identical names
that are already defined outside of the macro. The command is ignored
if used outside of a macro definition.
Additional Commands
1. In order to check the expansion of macros one can use the command
expand = MNAME (arguments) = arguments;
This will show the expanded macro based on the provided arguments
but will not execute the commands.
2. The command
macrolist;
gives a list of currently defined macros.
3. The command
macroclear;
deletes all currently defined macros.
2
data handling
1
2. Data Handling
Like many other statistical programs, TDA is build around the concept of
a rectangular data matrix with rows corresponding to cases and columns
corresponding to variables. This data matrix is used for quite different
data structures, like cross-sectional data, panel data, event-history data,
and relational data. In fact, the concept of a data matrix is quite independent from the interpretation of its contents providing information about
a specific type of data. This interpretation is only done by commands
when using part of the data matrix as data input.
Part 3 will give more information on different kinds of data structure.
The sections in the current part explain how to create and modify TDA’s
internal data matrix, quite independent of its interpretation as a specific
data structure.
2.1
Variables explains TDA’s concept of variables as it is used when
referring to the internal data matrix.
2.2
Creating New Variables explains the nvar command to be used
to create and modify a data matrix.
2.3
Removing Variables explains how to remove variables from the
current data matrix and how to delete the whole matrix.
2.4
Recoding Variables explains how to recode already existing variables.
2.5
Dummy Variables explains the ndvar command that can be used
to create dummy variables and interaction effects.
2.6
String Variables discusses a few possibilities to use string variables.
2.7
Namelists explains the concept of namelists that can be used to
abbreviate a list of variable names.
2.8
Temporary Case Selection explains the tsel command for temporary case selection.
d02.tex
April 19, 1998
2
data handling
2
2.9
Writing Data Files explains the pdata command that can be
used to write the internal data matrix, or selected variables, into
an output file.
2.10
Data Archives explains how to create and use compressed data
archives.
2.11
Special Format Data Files explains TDA system files and commands that provide an interface to SPSS portable files and to
Stata files.
2.12
Data File Utilities describes some commands that can be used
to investigate data files.
2.1
variables
1
2.1 Variables
TDA’s data matrix is a rectangular array with rows corresponding to
cases and columns corresponding to variables. In this context, a variable
is defined as a column in the data matrix. Creating a data matrix means
to define one or more variables. The data matrix can then be modified
by adding and removing variables.
Each variable has a unique name. Variable names are strings of up
to 16 characters which may consist of the following letters:
A,. . . ,Z, a,. . . ,z, 0,1,. . . ,9, , @, $
The first character of a variable name must be an upper case letter, or
one of the characters “ ”, “@”, “$”. The other characters can be upper
or lower case. Variable names are case sensitive, that is, lower and upper
case letters are distinguished. The syntax to define variables is:
VName <s> [pfmt] (label) = definition,
The terminating comma indicates that these expressions are not commands, but parameters which can only be used as part of the nvar
command that will be explained in 2.2.
VName is required and defines the variable’s name.
<s> is optional and can be used to define a storage size (see below) for
the variable’s values; the default is <4>, that is, values are stored
as single precision floating point numbers.
[pfmt ] is optional and can be given to define a format statement to be
used when writing the variable into an output file. The syntax for
pfmt is width.prec, where width is the field width and prec the
precision. It will be an F format (for example, 0.100) if width is
positive, or an E format (for example, 1.0E-1) if width is negative.
The default print format depends on the variable’s type and storage
size and will be explained in 2.2.
(label) is optional and can be used to define a variable label. If defined,
the label is (sometimes) used in TDA’s standard output. The maximum length is 120 characters. There are no default labels.
d0201.tex
December 18, 1999
2.1
variables
2
definition is required and specifies how the variable is to be generated.
There are three different possibilities: standard numerical variables,
archive variables, and string variables.
1. The syntax for standard numerical variables is
VName <s> [pfmt] (label) = expression,
where expression may consist of numerical constants, random numbers,
references to an external data file or to previously defined variables, and
operators. How to define expressions will be explained in 5.2. Here we
only mention how to refer to entries in an external data file. This is done
by using the special keywords c1,c2,c3,... referring, respectively, to
the first, second, third, and so on numerical entries in the records of an
external data file. For example, Y=c2, would define a variable Y taking
values corresponding to the second numerical entry in the records of an
external data file. It is possible directly to combine such references. For
example, S = c1 + c2, would define a variable S with values created as
the sum of the first and second numerical entries in the records of an
external data file.
When referring to an external data file, it is possible to create a
sequence of variables simultaneously. The syntax is
VName{n, m} <s> [pfmt] (label) = ck,
with n and m (0 ≤ n ≤ m) and k (k ≥ 1) integers. This defines a
sequence of m − n + 1 variables with names created by appending an
integer i = n, . . . , m to VName. The first of these variables refers to the
kth entry in the data file, the second refers to the k + 1.th entry, and so
on. For example, Y{1,3} = c2, would be expanded into Y1=c2, Y2=c3,
and Y3=c4.
2. Variables can also be created by referring to a compressed data
archive. Such variables are called archive variables, the syntax is
VName <s> [pfmt] (label) = A:AVNAME,
where AVNAME is the name of a variable in the data archive. Note that
the prefix, A:, is required to indicate that the following expression is to
be interpreted as the name of a variable in the archive. The concept of
data archives will be explained in 2.10.
2.1
variables
3
Box 1 Types of variables
1
2
3
4
5
if the variable is a string variable.
if the variable is a numerical constant.
numerical variables which do not (directly or indirectly) involve
type 2 operators.
numerical variables which (directly or indirectly) involve type 2
operators.
if the variable is defined inside of an edef command for episode
data.
3. TDA’s data matrix may only contain data for numerical and for string
variables. variables having How to define and use string variables will be
explained in 2.6.
Types of Variables. Each variable has a specific type providing information about how the values of the variable can be generated and,
consequently, how the variable can be combined with other variables by
operators. Box 1 explains the different types.
There is an important distinction between type 1 and type 2 operators. Type 1 operators can be evaluated by using only information from
a single row in the data matrix, or a single record in a data file. On
the other hand, type 2 operators need information from other parts of a
data matrix. For instance, the lag operator is a typical type 2 operator
referring to lagged values of a variable. If the definition of a variable
contains type 2 operators, its values cannot be generated sequentially
but only when all required information is available.
The definition of variable types applies recursively. For example, consider the definition of a variable that does not contain type 2 operators
but refers to type 4 variables. It will then, nonetheless, become a type 4
variable.
Storage Size Options. Each data matrix variable is stored according
to a specific storage size. Whenever TDA displays the currently defined
data matrix variables, the storage size is shown in a column labeled S.
The possible value are shown in Box 2.
The storage size can be specified individually for each variable in its
definition. The default storage size is 4, meaning that values are stored
as single precision floating point numbers. The required memory for the
data matrix is then 4 · NV · NOCMAX, where NV is the number of variables
2.1
variables
4
Box 2 Storage size options
0
1
2
4
5
8
<0
values
range:
values
range:
values
range:
values
range:
values
range:
values
range:
are stored as single bits,
0 – 1.
are stored as single bytes,
+/ − 127.
are stored as two-byte integers,
+/ − 32000.
are stored as single precision floating point numbers,
about 7 significant digits.
are stored as four-byte integers,
9 significant digits.
are stored as double precision floating point numbers,
about 15 significant digits.
negative values are used for string variables. The absolute
value of the storage size equals then the length of the string
variable.
and NOCMAX the maximum number of cases. However, if a variable is
defined as a single numerical constant, the storage size is always 8 and
only one double precision value, regardless of the number of cases in the
data matrix, is allocated in internal memory. One should note that TDA
does not check whether values of a variable fit into a specified storage
size. In particular this may cause problems if missing value codes have
to be inserted for variables with a <0> storage size.
2.2
creating new variables
1
2.2 Creating New Variables
There is a single command, nvar, for creating new variables. If a data
matrix does not already exist, the command creates a new data matrix,
otherwise the command adds variables to the existing data matrix. Box 1
shows the syntax. Most parameters are optional. Required is at least one
vdef parameter, that is, an expression of the form
VName <s> [pfmt] (label) = definition,
to define a variable (see 2.1). Variable names must be unique. A variable name can be used only once on the left-hand side of an expression
defining a variable.
Number of cases. If a data matrix does not already exist, one can use
the noc parameter to set an upper limit for the number of cases. Default
is noc=1000. If none of the variables refers to an external data file, or to
a data archive, TDA will create a data matrix with exactly noc cases. If
at least one of the variables refers to an external data file (defined with
the dfile parameter), the number of cases will be min{noc, n}, where n
is the number of records that can be read from the data file. If one of the
variables refers to a data archive file, the default maximum number of
cases will be equal to the number of records in the archive data file. The
noc parameter is only effective if a data matrix does not already exist.
The number of cases in an existing data matrix cannot be changed.
External data files. As explained in 2.1, the definition of variables can
refer to entries in an external data file by using the keywords c1,c2,...
It is then necessary to provide the name of the external data file with
the dfile parameter. The syntax is
dfile = name of data file,
This parameter can be used several times (maximum is 100). TDA reads
all data files sequentially. Of course, the data files should have an identical record structure. Alternatively, variables can refer to a data file in a
data archive, see 2.10. Note that variables defined in an nvar command
must not refer simultaneously to an external data file and to an archive
data file. Also, only a single archive data file can be referenced in each
nvar command.
d0202.tex
December 18, 1999
2.2
creating new variables
2
Box 1 Syntax for nvar command
nvar (
vdef,
noc=...,
isc=...,
dfile=...,
dreclen=...,
nmrec=...,
ffmt=...,
match=...,
isel=...,
vsel=...,
break=...,
dblock=...,
bsel=...,
fmt=...,
arcdic,
mblnk=...,
mstar=...,
mpnt=...,
mmatch=...,
mgen=...,
df=...,
keep=...,
drop=...,
bsize=...,
dtda=...,
dspss=...,
definition of a variable; this parameter can be used
several times to define several variables simultaneously
maximum number of cases, def. 1000
separation character between entries
definition of an external data file
fixed length of data file records
number of multiple records, def. 1
format information for data file records
information about matching variables
case selection while reading data
case selection while creating variables
break on condition
definition of block mode
block mode case selection
new print format
shows value labels for archive variables
new missing value code: blanks, def. -1
new missing value code: stars, def. -1
new missing value code: points, def. -1
new missing value code: mismatches, def. -3
new missing value code: general, no default
creates an output data file
keep variables for df option
drop variables for df option
maximum block size, def. 1000
TDA description file
SPSS description file
);
As default, TDA assumes that data file records are terminated by endof-record characters. This can be a single line-feed character (LF), or a
single carriage-return character (CR), or a LF-CR sequence as used by
DOS. This allows the records in a data file to have varying length. If
all records in a data file have the same length one can use the dreclen
parameter to specify this type of data file. Given the parameter
dreclen = n,
2.2
creating new variables
3
with a positive integer n, TDA assumes that each record consist of exactly n characters, including any end-of-record characters (if present).
Whether a data file has fixed record length can be checked with the
lcnt command, see 2.12.2.
Free and fixed format files. External data files should be plain ASCII
files with a free or fixed record structure. Records in a data file may be
data records or comments. Empty records consisting of only blank characters, and records beginning with a # character, are interpreted as comments; all other records are assumed to contain valid data. As default,
TDA assumes a free format record structure, that is, numerical entries in
the data file records are separated by at least one: blank, comma, semicolon, or tabulator character. This then identifies the numerical entries
referenced by the c1,c2,c3,... keywords.
The isc parameter can be used to change the separation character
in a free-format data file. The syntax is
isc = ’c’,
where c is a specific character.1 Then, only the character c is treated as a
separation character and each single occurrence of this character counts
as a separate separation character. For example, using the parameter
isc=’,’, the record 10,,20 would result in three values: 10, a missing
value, and 20. The missing value would be of type MBlnk.
Alternatively, there can be a fixed format with, or without, separation
characters, meaning that the data in a data file record have fixed locations, beginning at a fixed physical column. It is then necessary to
provide information about the location of variables by using the ffmt
parameter. The syntax is
ffmt = c1(n1-m1),c2(n2-m2),c3(n3-m3),...,
This means that numerical entries referenced by c1 begin in physical
column n1 and end in physical column m1, and analogously for c2,c3,...
If a variable occupies only a single physical column, one can use c1(n1)
instead of c1(n1-n1). Note that it suffices to provide this information
for variables which are actually referenced by c1,c2,... keywords. It
is not necessary to provide additional information about the type of
1 All
characters are used as given, except for the character t which is translated to a
tabulator character. A blank characters can be specified as isc=’ ’.
2.2
creating new variables
4
numerical entries. Numerical entries can be integers or floating point
values. Floating point values can be given in an F format, for example
-0.312, or in an E format, for example -3.12e-1. The number of significant
digits used depends on the variable’s storage size, see 2.1.
Missing values. A data file may contain non-numerical entries to indicate missing values. With free format data files, TDA automatically
recognizes two types of missing values: a single point (.) and a single
star (*). With fixed format data files, TDA also recognizes blank fields,
meaning that the location where to expect a numerical entry consists of
only blank characters. In all these cases, the default missing value code
is -1, but may be changed with the mpnt, mstar and mblnk parameters,
respectively. In all other situations, when TDA cannot correctly read a
numerical entry in a data file record, the program stops with an error
message. This behavior can be changed with the parameter
mgen = value,
Given this parameter, TDA substitutes value for all numerical entries
which cannot be read correctly.
Multiple records. As default, TDA assumes that each physical record
in an external data file corresponds to one logical record. There is then
a one-to-one correspondence between physical data records and rows of
the data matrix. Alternatively, one can define logical records consisting
of two or more physical records. Given the parameter
nmrec = n,
TDA creates logical records by concatenating n physical records (default
is nmrec=1). End-of-record characters are not removed but replaced by
blank characters to provide separation characters between numerical entries, see Example 2.
Matching new variables. When a data matrix already exists, the nvar
command adds the new variables to the existing data matrix. For each
row in the existing data matrix, and each new variable, a numerical value
will be added to the data matrix.
There are then two possibilities for adding the new variables to the
already existing data matrix. As default, TDA assumes trivial matching,
meaning that values for the new variables are created sequentially for
each row in the existing data matrix. If the data matrix contains n rows
2.2
creating new variables
5
and the definition of the new variables refers to an external data file
containing less than n records, the variables get the missing value code
-3 in the remaining data matrix rows. This missing value code can be
changed with the mmatch parameter.
Alternatively, one can control the matching of new variables with the
match parameter; the syntax is
match = A1,B1,A2,B2,...,
where A1,A2,... must be new variables and B1,B2,... must be variables in the already existing data matrix. Values for the new variables
are then created in such a way that values of A1 and B1, A2 and B2, and so
on, are equal. Implied is a one-to-many mapping. Values of the new variables should be unique with respect to A1,A2,...; and identical values
will be created for all data matrix rows having corresponding B1,B2,...
values. An example will be given below. Again, the new variables get the
missing value code -3 (optionally modified by the mmatch parameter) for
all data matrix rows without a match with the B1,B2,... variables. The
variables A1,A2,... used in the match parameter must be of type 2 or
3 and, in particular, must not contain type 2 operators. Also, the match
option is not compatible with block mode (see below), and none of the
variables must be a string variable.
Case selection. As default, TDA uses all data records from an external
data file, or archive data file.
1. The isel parameter can be used to select data file records. The syntax
is
isel = expression,
expression is then evaluated for each data file record, and a record is
only used if the result is not equal to zero. For example, given
isel=c1[10,,20],
TDA will only use records where the first numerical entry (c1) has a
value in the range from 10 to 20. Or, given
isel=le(rd,0.1),
only a 10 % random selection of the input records will be used.
2.2
creating new variables
6
The isel expression is evaluated while reading the records from
an input data file. Therefore, expression can only refer to information that is available in this situation. For external data files specified
with the dfile parameter, expression may only contain numerical constants, random numbers, type 1 operators, already existing variables, and
references to data file entries (c1,c2,c3,...). When using an archive
data file, expression may only contain numerical constants, random
numbers, type 1 operators, already existing variables, and references to
archive variables; these archive variables must be specified in the nvar
command and must belong to the same data file.
2. The isel parameter can only be used to select cases based on information in an external or archive data file; it is not possible to reference
variables specified as new variables in the current nvar command. To
allow for this possibility, one can use the vsel parameter with syntax
vsel = expression,
meaning, again, that only those cases are selected for the data matrix
where expression results in a nonzero value. In this case, expression
may contain numerical constants, random numbers, type 1 operators and
references to variables of type 2 or 3. The parameter works as follows:
For each case in the data matrix, the program first creates values for all
new variables, then the vsel expression is evaluated based on this data
matrix row, and depending on the result, the data matrix row is kept or
removed. Note that vsel expressions may not contain type 2 operators
and, if referring to any of the new variables, these variables should be of
type 1, 2 or 3. — Note that the vsel parameter can only be used when
the definition of at least one of the new variables refers to an archive or
external data file. Note also that the vsel parameter cannot be used in
block mode.
3. The break parameter provides a further option. Given
break = expression,
the creation of new data stops with the first record where expression
becomes a nonzero value. — Note that the break parameter can only be
used when the definition of at least one of the new variables refers to an
archive or external data file. Note also that the break parameter cannot
be used in block mode.
2.2
creating new variables
7
Block mode. As default, TDA treats each data file record, and corresponding data matrix row, separately. Alternatively, one can select block
mode by using the parameter
dblock = VName,
where VName is the name of a variable. TDA then treats each sequence
of records (data matrix rows) where VName has the same value as one
block of records. For instance, if VName refers to an ID variable in a data
file containing a certain (variable) number of records for each ID value,
the records belonging to the same ID number are treated as one block
of records.
The effect is that type 2 operators are evaluated differently. As default, type 2 operators are evaluated for all records in the data matrix.
For example, mean(X) will result in the mean value of variable X calculated for all data matrix rows. However, if operating in block mode,
type 2 operators are evaluated separately for each block of records;
mean(X), for example, will then result in a different mean value for each
block of records. More detailed information can be found in 5.2.6.
The variable used to define block mode must be of type 2 or 3, and
that the block mode option is not compatible with the match parameter
(non-trivial record matching).
If new variables are created in block mode there is an additional
option for case selection. The parameter is
bsel = expression,
At the end of each block of records, TDA evaluates expression for each
record in the current block and, if the result is zero for at least one
record, the whole block is skipped. expression may contain numerical
constants, random numbers, type 1 operators and references to variables
of type 2, 3, and 4. It is thus possible to use type 2 operators for defining
variables and then use these variables for the bsel expression. For example, assume we want only select individuals with at least five records.
One can then first define a variable, say BN = bnrec, counting the number of records in each block, and then use the parameter bsel=ge(BN,5).
Note that the bsel parameter can only be used when the definition of at
least one of the new variables refers to an archive or external data file.
While vsel and break parameters cannot be used in block mode, it
is possible to use the isel parameter. isel expressions are evaluated
while reading new records from the input file.
2.2
creating new variables
8
Box 2 Default print formats
storage size
print format
0
1
2
4
5
8
2.0
4.0
6.0
0.0
11.0
0.0
Print formats. Associated with each variable is a specific print format
to be used when values of the variable are written into an output file.
This print format can be specified in the variable’s definition (see 2.1). If
this option is not used the variable gets a default print format depending
on its storage size. Default print formats are shown in Box 2, where 0.0
indicates a “free format” meaning that only up to seven significant digits
are used.
Since the default storage size is 4, all variables without an explicitly
specified print format will get a free format. This can be changed with
the fmt parameter. All variables having a (default) free format will then
get the new print format specified with the fmt parameter. The syntax
is
fmt = n.m,
where n and m are integers and m ≥ 0. |n| is the field width, and m is
the precision, i.e., the number of digits after the decimal point. If n ≥ 0
it will be an F format, if n < 0 it will be an E format.
Print formats for archive variables are created in the same way as
for other variables: If a format information is provided in the definition
of a variable, this will be used, otherwise the print format found in the
archive’s variable description file will be used.
In connection with archive variables one can also use the arcdic
parameter to request information about value labels. If available in the
archive’s variable description file, this information is then written into
TDA’s standard output.
Directly writing output files. Using the df parameter, it is possible
to write data directly into an output file without creating a data matrix.
2.2
creating new variables
9
Box 3 Example data file d1.dat
1 1.1 2 3
2 2.2 3 4
3 . 4 *
4 4.4 5 6
Box 4 Command file d1.cf
nvar(
dfile = d1.dat,
X1 = c1,
X2 = c2,
X3 = c3,
X4 = c4,
);
pdata = d1;
# specifies data file
# define variables
# print data matrix to output file d1
The syntax is
df = name of an output file,
Given this parameter, TDA generates data for all new variables but does
not store the data in the internal data matrix. Instead, the data are
written directly into the output file. This allows to create subsets of very
large data files which cannot be kept in internal memory. As default,
all variables are written into the output file; optionally, one can select
subsets of variables, alternatively with the keep or drop parameter. Note
that this parameter can only be used if a data matrix does not already
exist and can only handle variables of type 1 – 4. Furthermore, type 4
variables can only be used if the nvar command operates in block mode.
If in block mode, this procedure needs memory for at least the maximum
number of records per block. The default maximum block size is 1000,
but can be changed with the bsize parameter.
In addition, one can use the dtda and dspss parameter to request
additional output files describing the generated data. The description
conforms to the TDA or SPSS syntax, respectively.
Example 1 Box 3 shows a simple data file. There are four records
containing integer and floating point values. Two values are missing. It
is a free format data file since all entries are separated by at least one
blank character, but also a fixed format data file.
2.2
creating new variables
10
Box 5 Part of standard output from d1.cf
Creating new variables.
Idx Variable T S PFmt Definition
----------------------------------1 X1
3 4
0.0 c1
2 X2
3 4
0.0 c2
3 X3
3 4
0.0 c3
4 X4
3 4
0.0 c4
Creating a new data matrix.
Maximum number of cases: 1000
Using data file(s): d1.dat
Free format. Separation character(s): default.
Reading data file: d1.dat
Read records: 4
Created a new data matrix.
Number of cases: 4
Number of variables: 4
Missing values numerical code
-----------------------------Blank
0
-1
Star
1
-1
Point
1
-1
Box 6 Command file d2.cf
nvar(
dfile = d1.dat,
X1 = c1,
X2 = c2,
X3 = c3,
X4 = c4,
ffmt = c1(1),c2(3-5),c3(7),c4(9),
);
pdata = d2;
Command file d1.cf, shown in Box 4, can be used to create a data
matrix containing the data from this data file. The parameters in the
nvar command first specify the data file and then define four variables
corresponding to the four entries in the data file records. The pdata
2.2
11
creating new variables
Box 7 Command file d3.cf
nvar(
dfile = d1.dat,
X1 = c1,
X2 = c2,
X3 = c3,
X4 = c4,
X5 = c5,
X6 = c6,
X7 = c7,
X8 = c8,
nmrec = 2,
fmt = 4.1,
);
pdata = d3;
resulting output file d3
-----------------------1.0 1.1 2.0 3.0 2.0 2.2
3.0 -1.0 4.0 -1.0 4.0 4.4
3.0
5.0
4.0
6.0
command prints the data matrix into an output file.
Box 5 shows TDA’s standard output from the nvar command in d1.cf.
A table replicates the definition of variables. The columns labeled T, S,
and PFmt show the type, the storage size and the print format, respectively. Some information about missing values is given at the end.
The data file can also be referenced as a fixed format data file. This
is illustrated by command file d2.cf, shown in Box 6. The two output
file, d1 and d2, should be identical.
Example 2 Command file d3.cf in Box 7 illustrates the nmrec parameter for concatenating physical records. nmrec=2 means that always two
physical records are combined into one logical record. Each logical record
then has 8 entries which are referenced by the variables X1,...,X8. The
right part of the box shows the resulting output file, d3. Note that the
missing entries in the data file have been substituted by the default
missing value code, -1.
Example 3 Command file d4.cf, shown in Box 8, illustrates how to
add new variables to an existing data matrix. The first nvar command
creates a data matrix in the same way as was done by command file
d1.cf in Box 4. The second nvar command reads the data file again
and defines identical variables, Y1,...,Y4. Without using the match
parameter this would be identical to defining Y1 = X1,... However, the
command file uses the parameter match=Y3,X1, and the resulting data
matrix, shown in the bottom of the box, is therefore created in such a
way that these two variables match. Note that there is no match for the
2.2
creating new variables
12
first data matrix row. In this row, all new variables get the missing value
code for mismatches, -3.
2.2
creating new variables
Box 8 Command file d4.cf
nvar(
dfile = d1.dat,
X1 = c1,
X2 = c2,
X3 = c3,
X4 = c4,
fmt = 4.1,
);
nvar(
dfile = d1.dat,
Y1 = c1,
Y2 = c2,
Y3 = c3,
Y4 = c4,
match = Y3,X1,
fmt = 5.2,
);
pdata = d4;
# creating a new data matrix
# adding variables
# match Y3 with X1
resulting output file d4
-----------------------1.0 1.1 2.0 3.0 -3.00 -3.00 -3.00 -3.00
2.0 2.2 3.0 4.0 1.00 1.10 2.00 3.00
3.0 -1.0 4.0 -1.0 2.00 2.20 3.00 4.00
4.0 4.4 5.0 6.0 3.00 -1.00 4.00 -1.00
13
2.3
removing variables
1
2.3 Removing Variables
To remove variables from the current data matrix one can use the command
clear = varlist;
where varlist is a list of variable names separated by commas. All variables specified in varlist must be available in the current data matrix.
These variables are then removed, that is, all data and data structures
associated with these variables are deallocated. Alternatively, one can
use the command
clear;
without any parameters to delete the whole data matrix. One can then
begin to create a new data matrix by using nvar commands. Note that
it is always possible to get information about the currently defined data
matrix by using the data and data1 commands, see 1.4.
If varlist on the right-hand side of the clear command contains
namelists (see 2.7), these nameslists and also the variables contained
in these namelists are deleted. To remove nameslists, but to keep the
contained variables, one can use the command
clearnl = list of namelists;
Then all namelists given on the right-hand side, but no variables, will
be deleted.
d0203.tex
April 19, 1998
2.4
1
recoding variables
2.4 Recoding Variables
Variable names must be unique, it is not possible to use the name of
an already existing variable inside an nvar command to define a new
variable with the same name. It is possible, however, to change the values
of an existing variable. The command is recode with syntax shown in
the following box.
recode (
dblock=...,
VName[fmt]=expression,
...
);
definition of block mode
recode variable VName
optionally repeated
VName must be the name of an already existing variable. The values
of this variable are changed according to the expression given on the
right-hand side. As an option, a new print format can be given in square
brackets. Storage size and labels cannot be changed. Also optionally, one
can use the dblock parameter with syntax
dblock = name of a variable,
to turn on block mode. If the recode expression contains type 2 operators,
they are then evaluated separately for each block. The recode command
automatically turns off a temporary case selection (see 2.8) that might
be active when the command is executed.
The recode command should be distinguished from the recode operator that allows to use matrices for recoding variable, see 5.2.6.2.
d0204.tex
April 19, 1998
2.5
dummy variables
1
2.5 Dummy Variables
Individual dummy variables can easily be defined by using the [.] operator. For instance, Y = X[3] would define a dummy variable Y taking
the value 1 if X has value 3, and zero otherwise. For a complete explanation of the [.] operator, see 5.2.5.5. If one has to create a long series
of dummy variables and possibly also interaction effects, it is sometimes
more comfortable to use the ndvar command with syntax shown in the
following box.
ndvar (
Y = dum(X),
L = dmul(A,B),
...
);
create dummy variables
create interaction effects
optionally repeated
Two kinds of parameters can be used (repeatedly). First, the parameter
Y = dum(X),
can be used to create a set of dummy variables, based on the values of the
variable X. X must be the name of an existing variable, Y must be a valid
variable name not already used. (The names X and Y are here only used
to illustrate the command; they can be substituted by any other valid
variable names.) Given this parameter, TDA creates a separate dummy
variable for each nonnegative value of X. If X has some negative values,
an additional dummy variable, YM, is created taking value 1 whenever X
has a negative value. For example, if X has values 1, 4, 17, and -1, the
command creates four dummy variables: Y1, Y4, Y17, and YM. In addition,
the command automatically creates a namelist, named Y, containing all
dummy variables corresponding to nonnegative values of X.
Second, one can use the parameter
L = dmul(A,B),
to create interaction effects. A and B can be variables or, more general,
namelists. L must be a valid variable name not already used. This pad0205.tex
April 19, 1998
2.5
dummy variables
2
rameter then creates a set of dummy variables for all possible interaction
effects between the variables (in) A and (in) B. For example, assume that
A = {A1,A2,A3} and B = {B1,B2}. The interaction effects will be
L = {A1&B1, A1&B2, A2&B1, A2&B2, A3&B1, A3&B2 }
In addition, a namelist containing the dummy variables representing the
interaction effects will be created. The names of the dummy variables
will be A1 B1, and so on. The parameter can be used several times to
create higher-order interaction effects. For example, LL = dmul(L,C)
would create all interaction effects between the variables in the nameslists
L and C. Of course, creating higher-order interaction effects soon exhausts
the maximal length of variable names. Automatically created dummy
variables always have storage size 0, that is, only a single bit is used to
store their values.
The ndvar command automatically turns off a temporary case selection that might be active when the command is executed.
2.6
1
string variables
2.6 String Variables
TDA’s data matrix may contain both numerical and string variables.
String variables can become part of the data matrix in one of the following ways:
a) String variables may be imported from SPSS or Stata files.
b) String variables may be present in a TDA data archive.
c) String variables can be defined when reading a standard (ASCII)
data file with the str operator (see below).
As has been explained in 2.1, string variables always have a fixed length.
This is the absolute value of the storage size which, for string variables,
is always a negative integer. There is no specific print format for string
variables. The strings are always written with the fixed length.
The str Operator. String variables can be defined inside the nvar
command in the following way:
VName = str(n,m),
where n and m are integers. This requires that a data file has been specified with the dfile parameter of the nvar command. Values of the
string variables are then the strings from column n up to, and including,
column m of the data file. There is no limitation to the length of the
strings.
One should note that using data files that contain strings as well as
numerical entries may result in confusion when the data file is taken as
having a free format. The reason is that the counting of numerical entries
in the data file records, necessary in order to interpret the c1,c2,...
references for numerical entries, may become indeterminate. In order to
establish determinate references to numerical entries one should use the
ffmt parameter for fixed format. However, to allow also for a free format
interpretation of data files, TDA proceeds as follows. Given the record of
a data file, TDA first extracts all strings that are defined with the str
operator. Then, before references to numerical variables are evaluated,
the strings are replaced by blank columns.
d0206.tex
December 19, 1999
2.6
string variables
2
Box 1 Data file d2.dat
1 abc ef 2 ab
2 xyz
3 ac
2 ijklm 4 bc
Box 2 Command file d6.cf
nvar(
dfile = d2.dat,
X1 = c1,
X2 = c2,
S1 = str(2,8),
S2 = str(12,12),
S3 = str(13,13),
);
pdata = d6;
resulting output file
--------------------1 2 abc ef a b
2 3 xyz
a c
2 4 ijklm b c
Using String Variables. Almost all TDA commands require numerical variables. In fact, the only command that currently can use string
variables is pdata; and this command simply writes the strings into an
output file. In order to avoid a check for string variables in all other
commands, we employ the following convention: whenever a command
expects a numerical variable and is given, instead, a string variable, the
value of the string variable is taken to be zero.
Operators for String Variables. There are, however, a few operators
that can be used to create numerical variables from string variables. The
available operators are described in 5.2.5.9 and 5.2.6.8.
Example 1 To illustrate string variables, command file d6.cf (Box 2)
uses the data file d2.dat shown in Box 1. The data file contains numerical entries and strings. The nvar command defines first two numerical
variables, X1 and X2, by referring to data file entries c1 and c2. (Remember that strings are replaced by blank characters before such references
are evaluated.) Then follow three string variables. The resulting output
file created by the pdata command is shown in the same box.
2.7
namelists
1
2.7 Namelists
The word namelist is used to denote a comma-separated list of variable
names. Namelists can be defined with the nlist command. The syntax
is
nlist (LName = V1,V2,...);
where LName is the name used to refer to the list of variables given
on the right-hand side. Then, in most TDA commands, one can use the
short name of the namelist instead of writing the complete list of variable
names. The syntax for the names of namelists is the same as for variables.
Up to 50 namelists can be defined. However, only a single namelist can
be defined with each nlist command. The data1 command shows the
currently defined nameslists (if any).
Namelists can be deleted in the same way as variables with the
clearnl command, see 2.3. Note also that namelists can be combined
with variable names. For example, if Y and Z are names of variables and
NList is the name of a namelist, it would be possible to request a linear
regression with the command
lsreg = Y,NList,Z;
The dependent variable would be Y and the set of independent variables
would consist of the variables in NList plus Z.
It is possible to define a new namelist using already defined namelists.
For example,
nlist (NList1 = Y,NList,Z);
would define another namelist consisting of Y, followed by the variable
names in NList, and then Z. The ordering of variable names is kept.
d0207.tex
April 19, 1998
2.8
temporary case selection
1
2.8 Temporary Case Selection
TDA’s internal data matrix always has a fixed number of cases (rows)
that is determined when the data matrix is created for the first time.
Therefore, the number of cases actually contained in the data matrix
cannot be changed afterwards. As default, all TDA procedures which
require data use all cases in the currently defined data matrix. However,
one can request a temporary case selection by using the command
tsel = expression;
TDA then creates an indicator variable by evaluating expression for all
cases in the current data matrix, and all subsequent procedures will only
use those cases from the data matrix where expression has a nonzero
value. A request for temporary case selection remains active until one of
the following conditions occur:
• A new tsel command substitutes the previous one.
• The current case selection is explicitly turned off with the command
tsel = off;
• It follows a command that automatically turns off any active temporary case selection (nvar, ndvar, recode).
The tsel command implies deallocation of any data structures which
depend on case selection. In particular, data structures for episode, sequence and relational data will be deallocated if there is a request for
a new temporary case selection. The tsel command also implies a reevaluation of case weights, if defined, see 6.1.2.
Case Selection with a Matrix Expression
The usage of tsel described above requires a standard (scalar) expression on the right-hand side. It is possible, however, to use the tsel
command in the following way:
tsel = matrix expression;
d0208.tex
December 23, 1999
2.8
temporary case selection
2
It is required that the matrix expression (or simply a matrix), say S,
has dimensions NOCDM × 1 where NOCDM is the number of cases in the
data matrix. (This is available from the nocdm operator, see 5.2.5.1.)
The command then selects all those rows i from the current data matrix
where the value of S(i) is not equal to zero.
2.9
1
writing data files
2.9 Writing Data Files
Data contained in the internal data matrix can be written into an output
file. The command is pdata, the syntax is shown in Box 1. All parameters
are optional. Without any parameters, the command is simply
pdata = fname;
where fname is the name of an output file. TDA will then write the data
for all variables contained in the current data matrix into this output file.
If a temporary case selection is active (see 2.8), only the selected cases
will be used. For writing the data, the program uses the print format
associated with each variable. By default, following each numerical entry,
a blank character is inserted allowing to use the data file as a free format
input file.
1. To select a subset of variables one can use the parameter
keep = varlist,
where varlist is a list of variable names separated by commas. Only
the variables specified in varlist will then be written into the output
file. Alternatively, one can use the parameter
drop = varlist,
Then, only variables which are not specified in varlist will be written
into the output file.
2. By default, the ordering of the records in the output file will be equal
to the ordering of the rows in the data matrix. Alternatively, one can use
the sort parameter to sort the data matrix before variables are written
into the output file. The syntax is
sort = S1,S2,...,
where S1,S2,... are names of variables. The data matrix is then sorted
in ascending order first with respect to S1, then with respect to S2, and
so on. Note that string variables cannot be used for sorting. Of course,
one can first create a numerical variable that sorts the strings with the
d0209.tex
December 19, 1999
2.9
writing data files
2
Box 1 Syntax for pdata command
pdata (
keep=...,
drop=...,
sort=...,
noc=...,
nn = n1,n2,
transp,
nc = n,
nq = n,
ap = 1,
l0 = ...,
sepc = ...,
prn =...,
dtda=...,
dspss=...,
) = fname;
keep specified variables
drop specified variables
sort according to specified variables
write only first noc cases
use only rows n1 – n2
transpose data matrix
create a new record after each n variables
concatenate n records, def. 1
append data to output file, def. ap = 0
controls printing of numerical values, def. 0
0 : leading blanks remain empty
1 : fills leading blanks with zeros
separation character, def. blank
sepc=none : no separation character
sepc=c : use c as separation character
option for writing single variables, def. prn = 0
1 : write a variable as triangle matrix
2 : write a variable as square matrix
create a TDA description file
create an SPSS description file
help of the strsp operator, and then use this variable to sort the output
file.
3. By default, all cases contained in the data matrix are written into
the output file. When using the parameter
noc = n,
only the first n data matrix rows are written into the output file. Alternatively, one can use the parameter
nn = n1 , n2 ,
Then, only data matrix rows i = n1 , . . . , n2 are written.
4. As an option one can use the parameter transp to request that the
data matrix is transposed before variables are written into the output
file.
2.9
writing data files
3
Box 2 Command file d5.cf
nvar(
dfile = d1.dat,
X1 = c1,
X2 = c2,
X3 = c3,
X4 = c4,
fmt = 4.1,
);
pdata (
keep = X1,X3,X2,X4,
nc = 2,
) = d5;
output file
----------1.0 2.0
1.1 3.0
2.0 3.0
2.2 4.0
3.0 4.0
-1.0 -1.0
4.0 5.0
4.4 6.0
5. When creating a data file with the pdata command it is possible to
replace each data matrix row by several records in the output data file.
The parameter is
nc = n,
where n is an integer, n ≥ 1. Then a newline character is inserted after
each n values written into the output file, see Example 1.
6. Alternatively, one can use the nq parameter. If nq = n, then each
consecutive set of n rows of the data matrix is written as a single record
into the output file.
7. By default, values of variables are separated by one blank character in
the output file. This may be changed with the sepc parameter. Also, by
default, numerical entries are right justified and leading columns remain
empty. The l0=1 parameter can be used to request that leading columns
are filled with zeros.
8. By default, the pdata commands creates a new output file. An already
existing file with the same name is overwritten without warning. When
using the ap=1 parameter (default is ap=0), the data is appended to an
already existing file.
9. The dtda parameter can be used to request an additional output file
containing a TDA description of the data file. The syntax is
dtda = name of an output file,
The output file contains an nvar command that can be used to read the
data file in a subsequent call of TDA. In a similar way, one can use the
2.9
writing data files
4
parameter
dspss = name of an output file,
to request an additional output file that can be used to read the data
file with SPSS or, after some editing, with SAS. Note that the dtda and
dspss parameters are ignored when using the transp option.
10. The prn parameter can be used when writing a single variable into
an output file. If prn=1, the variable is written as a triangle matrix, if
prn=2, it is written as a square matrix. This option is mainly used to
change the format of relational data which, in TDA, are assumed to be
given as a single column vector (variable) but in other programs are
often required in the format of an adjacency matrix (or part of it).
Example 1 Command file d5.cf, shown in Box 2, illustrates the nc
parameter. The command file uses data file d1.dat (see 2.2). We want
an output file where each data matrix row is replaced by two data file
records. The first record shall contain variables X1 and X3, the second
record X2 and X4. We therefore use the keep command to change the
order of variables, and then use nc=2 to create two records for each data
matrix row.
2.10
data archives
1
2.10 Data Archives
In empirical research, in particular when working with longitudinal data,
one often has to handle very large data sets. Given this situation, it
would be nice if the data could be stored and directly accessed in a compressed way. This chapter describes how this can be done with TDA,
based on Rahul Dhesi’s program ZOO. This is a general purpose archive
program to generate and manage an arbitrary set of files into a compressed archive. Files can be added, deleted, and updated; also some
options to repair damaged archives are available.
ZOO is similar to other archive programs, for instance PKZIP, ARC,
and LHARC. To be used with TDA, we have chosen ZOO because it
is highly portable, already running on many different computer platforms, and is freely available. (Rahul Dhesi, who owns the copyright of
ZOO, has made his program publicly available without any license fee.
So it is possible to distribute his program together with TDA as an
additional data management utility.) Moreover, ZOO incorporates very
efficient compression algorithms achieving 80 – 90 percent compression
rates with typical data sets used in empirical research.
Of course, to be useful one not only needs a way to create compressed
archives but, in addition, have direct and selective access to the data
stored in the archive without the need of prior extraction of all involved
data files. This capability has been build into TDA. So one can use ZOO
to build and manage data archives and then use TDA to access the
compressed data. Details are given in the following sections.
2.10.1
Creating Data Archives explains how to create a compressed
data archive to be used with TDA.
2.10.2
Variable Description Files explains how to create a description
of the variables contained in a data archive.
2.10.3
Archive Description Files explains how to create a description
of data archives.
2.10.4
Data Archive Commands summarizes the steps required to create a data archive and explains how to use such archives with
TDA.
d0210.tex
April 19, 1998
2.10.1
1
creating data archives
2.10.1 Creating Data Archives
Creating a data archive is done with ZOO. Like TDA, ZOO is a commanddriven program. Quite a large number of commands can be used to
control the program; a full description can be found in the program’s
manual that is always distributed along with ZOO. Here we only explain
some basic commands.
Generally, the syntax for using ZOO is
zoo commands [archive name] [file1 file2 ...]
archive name is the name of the file to be processed by ZOO as an
archive. Usually, an archive name has the extension .zoo; this is the
default, assumed by ZOO if a file name without an extension is given.
commands is a string of letters interpreted by ZOO as a command, or a
set of commands. The letters are case sensitive. The following examples
will explain some basic options.
displays a short list of commands to be used with
ZOO on the system’s standard output. Using an
upper case H, instead of h, gives an extended description.
l exam.zoo
lists the contents of the archive exam.zoo. As already noted, the name of the archive can be given
without an extension: zoo l exam.
ah exam fname adds the file fname to the archive exam using a
high compression mode.
x exam fname extracts the file fname from the archive exam.zoo.
If no file name is given (zoo x exam) all files contained in the archive are extracted.
D exam fname deletes the file fname from the archive exam.zoo.
In fact, the file is not deleted but kept as an earlier
version in the archive. To definitively delete the
file, the archive must be re-packed.
P exam
rebuilds the archive exam.zoo with all background
files actually deleted. The original archive is saved
as exam.bak.
zoo h
zoo
zoo
zoo
zoo
zoo
d021001.tex
April 19, 1998
2.10.1
2
creating data archives
Box 1 Example data files.
adat.1
---------1 -1.01 90
6 -1.06 95
2 -1.02 90
7 -1.07 95
3 -1.03 91
8 -1.08 95
4 -1.04 91
9 -1.09 97
5 -1.10 91
adat.2
----------1 1 1.01 -1
2 6 1.06 -2
3 2 1.02 -3
4 7
-4
5 3 1.03 -5
6 8 1.08 *
744 1.04-77
8 9
-8
adat.3
--------90 9001 1
91 9003 3
95 9008 8
97 9009 9
adat.p
------1 90
2 90
3 91
4 91
5 91
6 95
7 95
8 95
9 97
Box 2 Table of contents for archive exam.zoo
Archive examz.zoo:
Length
CF Size Now
-------- --- -------108 48%
56
96 33%
64
40 20%
32
45 27%
33
-------- --- -------289 36%
185
Date
Time
--------- -------3 Nov 55 20:35:44+457279
3 Nov 55 20:35:44+457279
3 Nov 55 20:35:44+457279
3 Nov 55 20:35:44+457279
--------- -------4 files
adat.1
adat.2
adat.3
adat.p
As an example, used also below to explain TDA’s commands to work
with ZOO archives, we will take the four data files shown in Box 1. The
ZOO archive will be called examz.zoo; it is generated with the command
zoo ah examz.zoo adat.*
Looking at the result with
zoo l examz
shows a table of contents (Box 2). Any kind of files can be put into a ZOO
archive. However, our main interest is in data files containing variables
which can be accessed selectively. Consequently, to be useful with TDA,
all data files in an archive should have a record structure.
A record structure can be given in two different ways. First, records
can be defined by a fixed record length. It is assumed, then, that all
records in a file have the same length to be defined in the archive description file (see below). Second, the records in a file may have variable
2.10.1
creating data archives
3
length. In this case, the end of each record must be given explicitly by
an end-of-record character.
End-of-record (newline) characters are not standardized across different platforms. On UNIX machines it is a single line-feed (0a, hexadecimal); on Macintosh computers it is a single carriage return (0d,
hexadecimal); and DOS uses two characters: carriage return and linefeed. All three possibilities may be used for files with variable length
records contained in an archive. However, when the program writes an
end-of-record (newline) character it is always the specific one for the
machine where the program is running.
One should note that TDA only supports archives with data files
containing variables in a fixed-field format. This restriction is reflected
by the requirement that for all variables their offset as well as their field
length must be explicitly specified (see 2.10.2). This requirement also
applies to variable length records. In this case, if a variable is defined
with an offset out of the actual length of a record, the record is filled up
with blank characters resulting in missing values for these variables.
2.10.2
variable description files
1
2.10.2 Variable Description Files
In order to access variables contained in the archive’s data files, these
variables have to be described. Otherwise it would be impossible to recognize that a file actually contains data organized as variables. TDA is
told about variables by means of a variable description file.
A variable description file is a standard ASCII text file. Each line of
this file describes a variable, or is a comment line if its first non-blank
character is a # character. A line describing a variable consists of five
fields, the last one is optional. Fields are taken to be in free format,
separated by at least one blank character. However, the first field must
begin in the first column of a record.
1. The first field is the name of the variable. The number of characters of
a variable’s name is not fixed. It is proposed, however, that a maximum
of 16 characters is used. The characters can be upper and lower case
letters, digits, and the underline character. The first character must be a
letter. Note that variable names are case sensitive. If retrieving variables
from a compressed data archives, the variable names must be exactly
the same as defined in the variable description file.
2. The second field must be a numerical entry and is interpreted as the
ID number of the data file where the variable is to be found. It must be
the same number as assigned to the data file in the archive description
file (see 2.10.3).
3. The third field is again numerical and is interpreted as an offset in the
data file record were the value of the variable begins. Counting begins
with zero as the first column in a data file record.
4. The fourth field is also numerical and, for numerical variables, may
be given as n or n.m with n and m integers. n must be greater than zero
and is interpreted as the field width for the values of the variable. The
syntax n.m should be used to provide a print format for floating point
values; see the description of the arcv command in 2.10.4.
Data archives may contain string variables. These are identified by a
single negative integer in the fourth field of the variable description file.
The absolute value of this entry is then taken as the length of the string
d021002.tex
December 18, 1999
2.10.2
variable description files
2
Box 1 Variable description file for archive examz.zoo
# examz.var variable description file for examz.zoo
A1
1
0
2
first variable of dat.1
This is some additional information
about the first variable.
A2
A3
1
1
3
9
5.2
2
second variable of dat.1
third variable of dat.1
B1
B2
B3
B4
2
2
2
2
0
1
4
8
1
2
4.2
3
first variable of dat.2
second variable of dat.2
third variable of dat.2
fourth variable of dat.2
C1
C2
C3
3
3
3
0
3
7
2
4
2
first variable of dat.3
second variable of dat.3
third variable of dat.3
Additional information may be given as value labels
(1) = label ...
(3) = label ...
(8) = label ...
(9) = missing value
P1
P2
4
4
0
1
1
3
first variable of dat.p
second variable of dat.p
variable.
5. The last field is optional and can be an arbitrary string of characters
that describes the variable as a variable label. Different from the other
items, this field may contain blank characters. If it is present it starts
with the first non-blank character after the fourth field up to the end of
the current line.
TDA has no provision to use value labels in its printouts. However, such
information may be provided in a variable description file simply for
informational purposes. To do so, the following convention is adopted.
Whenever a non-comment line in a variable description file begins in
the first column of a record it is taken as the description of a variable
as described above. All non-comment lines beginning with at least one
blank character are taken as additional information about the preceding
variable. This additional information is not interpreted by TDA but is
2.10.2
variable description files
3
shown when variable descriptions are printed and the arcdic command
is given.
A variable description file, examz.var, for the example data archive
examz.zoo is shown in Box 1. One should note that a single variable description file is used to describe all variables in an archive. It is required,
therefore, that variable names are unique across all data files. On the
other hand, it is not required that all variables, actually contained in an
archive, are explicitly described. Of course, only variables described in
the variable description file can be used by TDA. The ordering of the
variable descriptions is not important.
Having created a variable description file, it must be added to the
archive. For our example archive, we can do this with the command
zoo ah examz examz.var
2.10.3
archive description files
1
2.10.3 Archive Description Files
A data archive contains one or more data files and a single variable
description file. In addition, TDA needs some information about the
contents of the archive. This information is contained in an archive description file. This is again a standard ASCII text file, similar to a TDA
command file. Empty lines, or lines where the first non-blank character
is a # character, are treated as comments and ignored. All other lines
are interpreted by TDA.
The first line that is not a comment is always interpreted as the name
of an archive, possibly headed by a path where to find it. All other noncomment lines are interpreted as descriptions of files contained in the
archive. It is not necessary to describe all files contained in the archive;
however, only files described in the archive description file are recognized
by TDA. The maximum number of files that can be described for a single
archive is set to 200 in the standard compilation procedure for TDA.
For each file, six pieces of information must be present, given in six
fields separated by one or more blank characters:
1. The first field is the ID number of the data file described in this line
of the archive description file. It must be a non-negative integer number,
unique for each of the files. The ordering is not important. This number
is taken as the ID number of the data file, and it must be the same
number as used for attaching variable names to data files.
2. The second field is the name of the file. It must be the same name
as used in the creation of the ZOO archive and should conform to the
operating system’s requirements for file names.
3. The third field is a numerical entry and is interpreted as the type of
the file. Currently, only two types are distinguished: If the value is 1 the
file is assumed to be a data file. If the value is 2 the file is taken to be the
variable description file of the archive (see 2.10.2). Each archive description file must contain exactly one description of a variable description
file.
4. The fourth field is a numerical entry and is interpreted as the record
length of the file. It must be a non-negative integer. If its value is greater
than zero it is assumed that the file consists of records of equal length
d021003.tex
April 19, 1998
2.10.3
archive description files
2
Box 1 Archive description file for archive examz.zoo
# examz.des
# archive description file for examz.zoo
examz.zoo
# N
1
2
3
4
9
Name
adat.1
adat.2
adat.3
adat.p
examz.var
Type
RecLen
1
1
1
1
2
0
12
10
5
0
Records
9
8
4
9
29
NVar
3
4
3
2
0
given by this value. (If the records have one or more end-of-line characters they should be included in the record length.) If the value is zero, it
is assumed that the file has variable length records. In addition, it is then
assumed that each record is terminated by an end-of-line character. Normally, data files will have fixed length records, while variable description
files will have variable length records. However, also data files may have
variable length records, if only the variables that are described in the
variable description file can be found at fixed positions in the records.
5. The fifth field is a numerical entry and is interpreted as the number
of records of the file.
6. The sixth field is a numerical entry and is interpreted as the number of variables contained in the file. Currently, this value is used for
informational purposes only. In the case of a variable description file this
entry should be set to zero.
Box 1 shows as an archive description file, examz.des, for the archive
examz.zoo. The first non-comment line contains the name and optionally the path of the ZOO archive. The following lines describe the files
contained in the archive. The ordering is not important. In this example, there are four data files of type 1, and one variable description file of
type 2. The data files are defined with a fixed record length, the variable
description file has variable length records.
Notice that the definition of record length always includes any end-ofrecord characters. This may result in differences between DOS and UNIX
systems. The above example was created on a UNIX system where only
one character is used for the end of a record in a text file. However, ZOO
2.10.3
archive description files
3
data archives are portable across different computer platforms without
any modifications;1 and also the same archive description files can be
used.
1 Of
course, ZOO archives must be transferred as binary files between different computers.
2.10.4
1
data archive commands
2.10.4 Data Archive Commands
We begin with a short summary of the steps required to create a compressed data archive. It is assume that there is a set of ASCII data files
with a fixed format record structure.
1. Create a ZOO archive containing the data files. This is done directly
with the ZOO program. Note that one should always use ZOO’s ah option
to add files in the high compression mode.
2. Choose ID numbers for all data files in the archive and create a
variable description file describing all variables in the archive. (If the data
files are available as SPSS portable files, this can be done automatically
with the rspss command, see 2.11.2.) Check the file with the arcvc
command and be sure that it does not contain errors. Then add the
variable description file to the archive using again the ah option for high
compression mode.
3. Create an archive description file, check the archive with
tda arcd=... arcc
and check the output. If TDA does not find any errors, the archive can
be used for data retrieval.
Having created a data archive, it can be used with TDA to retrieve data.
The basic command is nvar as described in 2.2. One simply uses the nvar
command and specifies the required archive variable using the syntax:
VName = A:AVNAME,
where AVNAME is a variable name contained in the archive’s variable description file and VName is a corresponding variable name to be used by
TDA. It would also be possible to use the extended syntax
VName <s>[pfmt](label) = A:AVNAME,
but TDA automatically tries to find an optimal storage size and print
format for archive variables, based on information in the variable description file. However, TDA is only able to retrieve archive variables if
d021004.tex
May 29, 2002
2.10.4
data archive commands
2
given the name of an archive description file. The command is1
arcd = name of an archive description file;
Given this command, TDA reads the archive description file and tries to
open the archive for subsequent reading. Having successfully opened an
archive, the connection remains valid until it is explicitly turned off with
the command
arcd = off;
or until another archive is opened with a new arcd command. This allows
to use several different archives in a single command file. It also allows
to assess the same archive with a series of nvar commands. This is, in
fact, necessary because each separate nvar command can only retrieve
variables from a single archive data file. It is required to explicitly match
variables contained in different archive data files.
In addition to the basic arcd command to open a data archive, there are
three commands to support working with archives.
1. The arcc command, without any parameters, can be used to check
whether TDA can successfully read the files in an archive. Given this
command, TDA tries to read all files in an archive and compares the
results with the information in the corresponding archive description file.
A message about any discrepancies is given in the standard output. In
addition, the command checks whether all variable names in the variable
description file are unique. To illustrate, we check our example archive
with
tda arcd=examz.des arcc
The result is shown in Box 1.
2. To check a variable description file one can use the command
arcvc (df=fname) = name of an variable description file;
TDA then reads the variable description file and checks for correct syntax
and unique variable names. The result is shown in the standard output.
For example, the command
tda arcvc=examz.var
1 If
the arcd command is used without any arguments, it provides information about
the currently defined data archive (if any).
2.10.4
data archive commands
3
Box 1 Output from arcc command.
> arcd=examz.des
--------------------------------------------------------Reading archive description file: examz.des
ZOO data archive: examz.zoo
Checking definition of files in archive.
FN Type RLen Records NVar
Size
M Name
-----------------------------------------------1
1
0
9
3
108 2 adat.1
2
1
12
8
4
96 2 adat.2
3
1
10
4
3
40 2 adat.3
4
1
5
9
2
45 2 adat.p
9
2
0
29
0
911 2 examz.var
> arcc
--------------------------------------------------------Archive check. Reading files defined in: examz.des
File:
1 adat.1
File:
2 adat.2
File:
3 adat.3
File:
4 adat.p
File:
9 examz.var
No errors found.
records:
records:
records:
records:
records:
9
8
4
9
29
should show that the variable description file for our example archive is
correct and that all variable names are unique. The optional parameter
df=fname, with fname the name of an output file, can be used to create
a variable description file with unique variable names. If the input file
uses variable names more than once, these names are made unique by
appending the corresponding logical file number. Note that this procedure assumes that variable names corresponding to the same logical file
number in the input file are already unique. This is not checked, but
might be checked by using the arcvc command a second time with the
output file created by the df parameter.
3. The command
arcv (fn=f,arcdic) = fname;
can be used to get information about the variables contained in a data
archive. All parameters are optional. If used without parameters, the
command prints a list of all variables contained in the data archive into
2.10.4
data archive commands
4
Box 2 Output from arcv(fn=adat.2) command
#
#
#
#
#
#
#
arcd = examz.des;
nvar(
B1
<1>[2.0] = A:B1
B2
<1>[2.0] = A:B2
B3
<4>[5.2] = A:B3
B4
<2>[3.0] = A:B4
);
,
,
,
,
#
#
#
#
[adat.2]
[adat.2]
[adat.2]
[adat.2]
first variable of dat.2
second variable of dat.2
third variable of dat.2
fourth variable of dat.2
the standard output. The printout follows the syntax of the nvar command and can be used as a command file for retrieving variables from the
archive. If the name of an output file, fname, is given on the right-hand
side, this file is used instead of the standard output. The fn parameter
can be used to specify data archive files. It can be used more than once.
If used, only variables contained in the specified data file(s) are written.
The arcdic parameter can be used to request that also value label information (if present in the archive’s variable description file) is written
into the output file. As an illustration, the result of
tda arcd=examz.des arcv(fn=adat.2)
is shown in Box 2.
Creating a storage size and a print format is based on the format
information in the variable description file. If the format information is
given by a single integer, n, it is assumed that the variable takes only
integer values. Also if the format is given as n.m and m = 0. Otherwise,
it is assumed that the variable has floating point values.
In both cases, calculation of a storage size is based on n∗ = max{n, 2}.
In the case of integers, the storage size is 2 if 2 ≤ n∗ ≤ 4 and otherwise
5. For floating point numbers the storage size is 4 if n∗ ≤ 7 and otherwise it is 8. The print format is w.m with w = n∗ for integers and
w = max{n∗ , m + 3} for floating point numbers. Note that the nvar
command needs the afmt parameter to enable these conventions. If the
nvar command is used without the afmt parameter, archive variables
are treated like all other variables.
2.11
1
special format data files
2.11 Special Format Data Files
This chapter contains the following sections.
2.11.1
TDA System Files explains how to use system files for TDA’s
internal data matrix.
2.11.2
SPSS Files explains how to read and write SPSS portable and
sav files.
2.11.3
Stata Files explains how to read and write Stata files.
d0211.tex
December 18, 1999
2.11.1
1
tda system files
2.11.1 TDA System Files
TDA system files are used to save the current data matrix in a file, or to
restore the data matrix from such a file. Since a system file stores data
in a machine-dependent binary format, reading the data from a system
file is much faster than creating an internal data matrix from external
data or archive files. The command to save the data matrix in a system
file is
wsys = name of an output file;
The right-hand side is optional. The default system file name is tda.sys.
Given this command, TDA write the contents of the currently defined
data matrix into the output file; any currently active temporary case
selection is ignored. String variables and type 5 variables are not written.
Also information about case weights, if defined, is not written into the
system file.
To restore a data matrix from a system file, the command is
rsys = name of a system file;
Again, the file name is optional and the default name is tda.sys. Given
this command, TDA tries to restore a data matrix from the specified
input file. A list of variables read from the system file is written into
the standard output. Note that this command can only be used if a
data matrix does not already exist. Otherwise, one should first execute
a clear command (see 2.3).
A TDA system file always begins with some plain ASCII records, then
follow the data in a machine-dependent binary format. The structure is
as follows:
1. The first record contains an identification string: TDA System File
(n), with n the version number, followed by the date and time when the
file was created.
2. Then follows the number of cases and variables, and a description of
all variables in plain ASCII.
3. Finally, for each variable there is a single binary record containing
the data.
d021101.tex
April 19, 1998
2.11.2
spss files
1
2.11.2 SPSS Files
This section describes commands that can be used to read and write
SPSS portable and sav files.
2.11.2.1 SPSS Portable Files describes commands for reading and writing SPSS portable files.
2.11.2.2 SPSS Sav Files describes commands for reading and writing
SPSS (for Windows) sav files.
d021102.tex
December 18, 1999
2.11.2.1
spss portable files
1
2.11.2.1 SPSS Portable Files
There are two commands, rspss and wspss, to read and write SPSS
portable files. The SPSS Base System Syntax Reference Guide (1992, p.
234) gives the following explanation: “A portable data file is a data file
created by SPSS and used to transport data between different types of
computers and operating systems [. . . ] or between SPSS, SPSS/PC+, or
other software using the same portable file format. Like an SPSS data file,
a portable file contains all of the data and dictionary information stored
in the working data file from which it was created.” Implementation in
TDA is based on: SPSS for Unix. The PPF Portable File Format, Release
4.0 , published by SPSS Inc. in April 1990.
Reading SPSS portable files. The command is rspss with syntax
shown in Box 1. It is expected that the name of an SPSS portable file
is given on the right-hand side; all other parameters are optional. TDA
then tries to create a data matrix based on the contents of the SPSS file
or, if the df parameter is used to give the name of an output file, the
data are directly written into this output file without creating an internal
data matrix. Note that, in both cases, the command is only executed if
a data matrix does not already exist. Note also that one can use dfa,
instead of df, in order to append the data to an already existing files.
1. The input file is expected to be a plain ASCII file consisting of records
each having 80 characters (bytes) plus end-of-record characters (a single
line-feed (UNIX) or a carriage return – line-feed sequence (DOS). If a
record contains less than 80 characters it is filled with blank characters.
If the input file does not contain end-of-record characters, one can
use the len parameter with syntax
len = n,
where n is one of the integers 80, 81, or 82. TDA then assumes that each
record consists of exactly n characters (bytes).
2. Basic information about variables in an SPSS portable file is given by
the variable’s name and, optionally, by variable and value labels. TDA
uses the same variable names as found in the SPSS file. If present, also
the variable labels are used. Values labels are ignored.
d02110201.tex
December 18, 1999
2.11.2.1
spss portable files
2
Box 1 Syntax of rspss and wspss commands
rspss (
len=...,
noc=...,
msys=...,
fmt=...,
df=...,
dvar()=...,
arcd()=...,
) = file name;
record length, def. 80
maximum number of cases
system missing value code, def. -5
new print format
write data directly to output file
create/update variable description file
create/update archive description file
wspss (
keep=...,
drop=...,
sort=...,
) = file name;
keep variables
drop variables
sort cases
3. Since SPSS portable files do not contain explicit information about
the number of data records and the storage format of the variables, TDA
reads the data in the input file two times. In a first run, it tries to determine the number of data records and the storage requirements for each
variable. Memory allocation for the data is based on this information.
Each variable gets the minimum storage size to save its values as found
in the input file. Floating point values are always stored in double precision, i.e. with storage size 8. Storage size for string variables always
equals the (negative value of the) length of the string variable.
4. TDA ignores any information on print formats that might be present
in the SPSS portable file and tries to calculate appropriate print formats
based on each variable’s range of values. However, TDA’s default behavior in creating print formats will not always give sensible results for
floating point numbers,1 and one can therefore use the fmt parameter
to explicitly specify a print format for floating point values.
5. TDA recognizes so-called system missing values that may be present
in the data portion of an SPSS portable file (indicated by an asterisk).
By default, these missings are substituted by the numerical value -5 (or
1 TDA
uses the maximum length of the integral part of the floating point number
and then provides for 6 digits after the decimal point, based on an F format.
2.11.2.1
spss portable files
3
by blanks if it is a string variable). The missing value code for numerical
variables can be changed with the msys parameter. Note that TDA ignores any definitions of user-defined missing values since these definitions
to not alter the data portion of an SPSS portable file.
6. By default, TDA creates an internal data matrix, or an output file
defined with the df parameter, for all data records found in the input
file. Optionally, one can use the noc parameter to define an upper limit
of cases.
7. The dvar parameter can be used to create a variable description file
for the variables found in the SPSS input file. The syntax is
dvar (argument1,argument2, ...) = output file name,
For each variable in the SPSS input file, the output file will contain one
record describing the variable according to the conventions for variable
description files for TDA data archives.2 The optional arguments are as
follows:
a) The argument
fn = file number,
can be used to specify a file number that is used in the variable
description file; the default file number is 1.
b) The argument
p = n,
where n is a positive integer. Each string variable will then be partitioned into substrings of n characters length, and corresponding
variable definitions will be added to the variable description file.
The additional variables are of type string.
c) The argument
pn = n,
where n is a positive integer. Same as p = n, but the new variables
will be defined as numerical variables.
2 If a file with the same name already exists, it is overwritten without warning. To
append the variable descriptions to the end of an already existing file, one can use
dvara instead of dvar.
2.11.2.1
spss portable files
4
d) The argument
p(VNAME) = n,
where n is a positive integer and VNAME is the name of a variable.
Then, if VNAME is the name of a string variable, this variable will be
partitioned into substrings of n characters length, and corresponding variable definitions will be added to the variable description
file. The additional variables are of type string.
e) The argument
pn(VNAME) = n,
where n is a positive integer. Same as p(VNAME) = n, but the new
variables will be defined as numerical variables.
Note that the p and pn arguments are incompatible. The p(VNAME) and
pn(VNAME) arguments are compatible and can be used several times.
8. The arcd option has the syntax
arcd(argument1,argument2) = file name,
and can be used to create, or update, an archive description file. The
optional arguments are:
zoo = file name,
This file name is then used in the archive description file for the ZOO
archive.
vdf = file name,
This file name is then used in the archive description file for the variable
description file. The file number for this file is always 999.
Note that the arcd option is only recognized if the rspss command
contains both a df and a dvar parameter. In order to get valid information about the number of records in the variable description file, the
name used with the vdf parameter should be the same as the name used
for the dvar parameter. Also note that for the calculation of physical
record lengths that is required for the archive description file, it is assumed that there is a 1-byte EOL character when running under UNIX,
and a 2-byte EOL character when running under DOS and MS Windows.
2.11.2.1
spss portable files
5
Writing SPSS portable files. The command for writing SPSS portable
files is wspss with syntax shown in Box 1. All parameters, except an
output file name on the right-hand side, are optional. Without additional
parameters, TDA tries to create an SPSS portable file containing all
currently defined variables.
1. To select a subset of variables for the output file one can use the
parameter
keep = varlist,
with varlist a list of variables separated by commas. Then only these
variables are written into the output file. Alternatively, one can use the
parameter
drop = varlist,
Then all variables except variables specified in varlist are used.
2. By default, cases are not sorted before writing data into the output
file. As an option, one can use the parameter
sort = V1,V2,...,
to sort the data matrix first w.r.t. V1, then w.r.t. V2, and so on, in
ascending order.
3. Except for string variables, to define print formats for the SPSS
portable file, TDA always uses an F format. A free format is translated
to 10.4, an E format is translated to 12.4.
2.11.2.2
1
spss sav files
2.11.2.2 SPSS Sav Files
There are two commands, rspss1 and wspss1, to read and write SPSS
sav files that have been created by, or can be used with, SPSS for Windows running on an Intel (reversed byte order) platform. The syntax of
these commands is shown in the following box.
rspss1 (
noc=...,
msys=...,
df=...,
dvar=...,
) = file name;
maximum number of cases, def. all
system missing value code, def. -5
write data directly to output file
create file with variable descriptions
wspss1 (
keep=...,
drop=...,
sort=...,
) = file name;
keep variables
drop variables
sort cases
1. Except for the file name on the right-hand side, all parameters are
optional.
2. The rspss1 command can only be used when a data matrix does not
already exist. By default, the command creates a new data matrix.
If the df parameter (or dfa to append data) is used, the data are
written directly to an output file and a data matrix is not created.
3. The dvar parameter can be used to request an additional output
file that contains a description of variables (variable labels and, if
present, also value labels) as found in the input file. Note however,
that this will not be a variable description file in the sense of TDA’s
data archives.
4. The rspss1 command assumes that the SPSS sav file was created by
SPSS for Windows on an Intel platform, meaning that binary values
are stored in reversed byte order. Correspondingly, the binary files
d02110202.tex
December 19, 1999
2.11.2.2
spss sav files
2
created by the wspss1 file are only appropriate for this architecture.
5. The keep, drop, and sort parameters in the wspss1 command expect
comma-separated lists of variables. String variables cannot be used
for sorting.
2.11.3
1
stata files
2.11.3 Stata Files
There are two commands to read and write Stata files (versions 4.0 and
6.0). Implementation is based on the description of the .dta file format
in the documentations of Stata releases 4.0 and 6.0.
Reading Stata files. The command for reading Stata files is rstata
with syntax shown in Box 1. On the right-hand side the name of a Stata
data file must be given. All other parameters are optional. By default,
TDA tries to create a data matrix based on the information in the input
file. Alternatively, the df parameter can be used to define the name of an
output file. The data read from the Stata file are then written directly
into that output file without creating an internal data matrix. Note that
in both cases the command is not executed if a data matrix already
exists.
1. When reading Stata files, TDA should be able to correctly recognize
whether the data are stored according to a HiLo or LoHi byte ordering.
2. If the first character of a Stata variable name does not conform to
the TDA conventions for variable names (see 2.1), it is preceded by an
underline character, otherwise it is always converted to an upper case
letter.
3. Translation of Stata variable types to TDA variables types is as follows:
1. Stata variables of type b (byte) are stored with storage size 1.
2. Stata variables of type i (short integer) are stored with storage
size 2.
3. Stata variables of type l (long integer) are stored with storage
size 5.
4. Stata variables of type f (single precision floating point) are
stored with storage size 4.
5. Stata variables of type d (double precision floating point) are
stored with storage size 8.
d021103.tex
December 23, 1999
2.11.3
stata files
2
Box 1 Syntax of rstata and wstata commands
rstata (
noc=...,
msys=...,
df=...,
dvar()=...,
arcd()=...,
) = fname;
wstata (
keep=...,
drop=...,
sort=...,
ptyp=...,
maximum number of cases, def. all
system missing value code, def. -5
write data directly to output file
create/update variable description file
create/update archive description file
keep variables
drop variables
sort cases
type of output file, def. 6
4 : for Stata release 4
6 : for Stata release 6
) = fname;
6. Stata string variables will result in corresponding TDA string
variables.
4. The print format for numerical variables is based on the information
in the Stata file. Note, however, that g-type formats are translated to
f -type formats.
5. A Stata input file may contain missing value codes. These are substituted by the numerical value -5. Another missing value code can be
specified with the msys parameter.
6. By default, TDA creates an internal data matrix for all data records
found in the input file or. Using the df parameter, all records are written
into the specified output file without creating an internal data matrix.
Optionally, one can use the noc parameter to define an upper limit of
cases.
7. The dvar parameter can be used to create a variable description file
for the variables found in the Stata input file. The syntax is
dvar (optional arguments) = output file name,
For a description of the optional arguments see 2.11.2.1. Also note that
2.11.3
stata files
3
one can use dvara to append information to an already existing file. Note
that the rstata command only tries to find value label information if
the noc parameter is not used. This is due to the fact that, in a Stata
file, value label information is always at the end of the file.
8. The arcd parameter can be used to create an archive description file.
For an explanation of this option see 2.11.2.1.
Writing Stata files. The command for writing Stata files is wstata
with syntax shown in Box 1. All parameters, except an output file name
on the right-hand side, are optional. Without additional parameters,
TDA tries to create a Stata file containing all currently defined variables.
1. By default, the wstata command creates a Stata file for release 6.0.
The ptyp parameter can be used to create a file for release 4.0.
2. To select a subset of variables for the output file one can use the
parameter
keep = varlist,
with varlist a list of variables separated by commas. Then only
these variables are written into the output file. Alternatively, one
can use the parameter
drop = varlist,
Then all variables, except those specified in varlist, are used.
3. By default, cases are not sorted before writing data into the output
file. One can use the parameter
sort = varlist,
to sort the data matrix, based on the variables contained in varlist
(from left to right), in ascending order. Note that string variables
cannot be used for sorting.
2.12
data file utilities
1
2.12 Data File Utilities
This chapter describes commands which are sometimes helpful to investigate and manipulate (data) files. The sections are as follows.
2.12.1
Characters in a File explains the ccnt command that can be
used to get a frequency distribution of the characters in a file.
2.12.2
Records in a File explains the lcnt command that can be used
to get a frequency distribution of the lengths of records in a file.
2.12.3
Binary Contents of a File explains the dump command that can
be used to see into the binary contents of a file.
2.12.4
Splitting Files into Parts explains the dsplit command that
can be used to split a file into parts.
2.12.5
External Sorting explains the esort command that can be
used to sort big data files.
2.12.6
External Merging explains the emerge command that can be
used to merge big data files.
2.12.7
Selection of Records explains the eselect command that can
be used to select records from big data files.
2.12.8
Dropping Selected Columns explains the eskip command that
can be used to drop selected columns from big data files.
d0212.tex
June 14, 1998
2.12.1
characters in a file
1
2.12.1 Characters in a File
The ccnt command can be used to get a frequency distribution of the
characters contained in a file. The syntax is
ccnt = fname;
where fname is the name of a file. The resulting frequency distribution
is written into the standard output. The characters in the file are shown
in their hexadecimal representation and, if possible, as visible ASCII
characters.
d021201.tex
April 19, 1998
2.12.2
records in a file
1
2.12.2 Records in a File
The lcnt command can be used to get a frequency distribution of the
lengths of records in a file. The syntax is
lcnt (noc=...) = fname;
where fname is the name of a data file. It is assumed that this file has
end-of-record characters: single line-feed characters (UNIX) or carriagereturn line-feed sequences (DOS and DOS-like platforms). The end-ofrecord characters are not included in counting the record length.
The noc parameter is optional and can be used to set an upper limit
for the number of cases, default is noc=1000. The maximum record length
is 20000 characters.
d021202.tex
April 19, 1998
2.12.3
binary contents of a file
1
2.12.3 Binary Contents of a File
The dump command can be used to show the binary contents of a file.
Each character is shown in its hexadecimal representation and, if possible, as a visible ASCII character. The syntax is
dump (nc=...,s=...) = fname;
where fname is the name of a file. Parameters are optional. By default,
the command begins with the first character in the input file (s = 0).
Alternatively, it begins with an offset given by the s parameter. The nc
parameter can be used to limit the number of characters; by default, all
characters of the input file are used.
d021203.tex
April 19, 1998
2.12.4
splitting files into parts
1
2.12.4 Splitting Files into Parts
The dsplit command can be used to split a file into parts. The syntax
of this command is
dsplit (len=...) = fname;
where fname is the name of a file. The size of the parts can be defined
with the parameter len, default is len=1000. The file specified by fname
is then split into parts of n bytes (of course, the last part could be less
than n bytes if the size of fname is not a multiple of n). The parts are
named:
fname.a, fname.b, fname.c, and so on.
Note that on DOS-like platforms, fname should not already contain a
period (.) because file names may only contain a single point.
d021204.tex
April 19, 1998
2.12.5
1
external sorting
2.12.5 External Sorting
The esort command can be used to sort big data files that do not fit
into main memory. The syntax is shown in the following box.
esort (
df=...,
sk=...,
noc=...,
len=...,
) = input file;
name of output file (required)
up to five sort keys
number of records in temp. file, def. 1000
maximal record length, def. 1000
The command expects the name of a standard ASCII data file on its
right-hand side. This file must have valid end-of-record characters. The
maximal record length can be specified with the len parameter. Also
required is the name of an output file to be given with the df parameter.
Finally, at least one sort key must be defined with the sk parameter.
The syntax is
sk = i1 , j1 , i2 , j2 , . . . ,
where the right-hand side is a sequence of up to five pairs of column
numbers. Each pair specifies one sort key. The first sort key is taken to
be the field from column i1 up to, and including, column j1 ; and so on.
If there are two or more sort keys they are used hierarchically from left
to right.
The command works as follows.
1. The command reads up to noc records from the input file and
creates a corresponding array of sort keys and pointers to the data
file records. This array is then sorted in ascending order and the
sorted records are written into a temporary output file.
2. Step 1 is repeated until no more records can be read from the input
file.
3. Finally, the temporarary files are simultaneously read and merged
in order to create a new sorted output file.
d021205.tex
June 14, 1998
2.12.5
external sorting
2
One should note that the maximal number of temporary files is 100,
but the actually available number may be less due to limitations of the
operating system. One should also note that names of temporary files
are created by adding “.i” to the name of the input file (i = 0, 1, 2, . . .).
Therefore, when working on a DOS-like platform, the name of the input
file should not already contain a period.
2.12.6
1
external merging
2.12.6 External Merging
The emerge command can be used to merge big data files that do not
fit into main memory. The syntax is shown in the following box.
emerge (
df=...,
mf=...,
len=...,
m=...,
sepc=...,
) = input file;
name of output file (required)
list of merge files
maximal record length, def. 1000
missing value character, def. blank
separation character, def. blank
The command expects the name of a standard ASCII data file on its
right-hand side. This file must have valid end-of-record characters. The
maximal record length can be specified with the len parameter. Also
required is the name of an output file to be given with the df parameter.
Finally, at least one more file to be merged with the input file must be
specified with the mf parameter. The syntax is
mf = fname1 [i1 , j1 , i, j],
= fname2 [i2 , j2 , i, j],
= fname3 [i3 , j3 , i, j], · · · ,
Up to 50 file names can be specified on the right-hand side. (Note, however, that this maximum may be less due to limitations of the operating
system.) Like the main input file, these files must be plain ASCII files
with valid end-of-record characters and record length not exceeding len.
In addition, for each of these files one has to specify two fields. The
first two numbers, ik and jk , must specify, respectively, the first and last
columns of a field in the records of the corresponding file to be merged
(i.e., fnamek). The second two numbers, i and j, must specify a field in
the main input file. While the (ik , jk ) fields can be different for each file,
there can only be a single field in the main input file. Furthermore, all
fields must have the same length.
The command expects that all input files are already sorted with
d021206.tex
June 14, 1998
2.12.6
external merging
2
respect to the specified fields. In fact, the command sequentially reads
records from the main input file and, at the same time, reads records
from all other input files. In case of identical fields records from the
additional input files are added to the main record, otherwise it is filled
with missing value characters.
By default, missing records are substituted by blank characters. Optionally, one can use the m parameter to specify one of the digits (0, 1, . . . , 9).
Also by default, records from the input files are separated by a single
blank character. Optionally, one can specifiy an alternative separation
character with the sepc parameter, or suppress any separation character
with “sepc=none”.
2.12.7
selection of records
1
2.12.7 Selection of Records
The eselect command can be used to select records from a big data
file based on keys provided by another file. The syntax is shown in the
following box.
eselect (
df=...,
if=...,
sk=...,
noc=...,
opt=...,
len=...,
) = input file;
name of output file (required)
additional input file
specification of keys
max number of keys, def. 1000
if 1 (def.) write all records
if 2 supress identical records
maximal record length, def. 1000
The command expects the name of a standard ASCII data file on its
right-hand side. This file must have valid end-of-record characters. The
maximal record length can be specified with the len parameter. Also
required is the name of an output file to be given with the df parameter.
In addition, the command expects another input file to be specified
with the if parameter. Again, this must be a standard ASCII file with
valid end-of-record characters and record length not exceeding len. The
parameter
sk = i1 , i2 , j1 , j2
must be used to specify two fields. i1 and i2 specify, respectively, a field
beginning in column i1 and ending in column i2 in the records of the
main input file. j1 and j2 specify a corresponding field in the records of
the additional input file defined with the if parameter. Both fields must
have the same length. The maximal number of fields (keys) read from
the additional input file can be specified with the noc paramater.
The command works as follows. It reads up to noc keys from the additional input file defined with the if parameter. These keys are then
sorted. Finally, the command reads the records from the main input file
d021207.tex
June 14, 1998
2.12.7
selection of records
2
and whenever a matching key is found, this record is written to the output file. By default, all matching records are written into the output file.
If the opt=2 parameter is given each key is used only once.
2.12.8
dropping selected columns
1
2.12.8 Dropping Selected Columns
The eskip command can be used to drop selected columns from a big
file. The syntax is shown in the following box.
eselect (
df=...,
sk=...,
len=...,
) = input file;
name of output file (required)
up to five fields
maximal record length, def. 1000
The command expects the name of a standard ASCII data file on its
right-hand side. This file must have valid end-of-record characters. The
maximal record length can be specified with the len parameter. Also
required is the name of an output file to be given with the df parameter.
In addition, the command expects a specification of up to five fields
with the parameter
sk = i1 , j1 , i2 , j2 , . . . ,
The kth field begins in column ik and ends in column jk . The command
sequentially reads the records from the input file and write those columns
into the output file that do not fall in any of the column ranges defined
with the sk parameter.
d021208.tex
June 14, 1998
3
data structures
1
3. Data Structures
TDA supports several different data structures, each based on a specific
interpretation of the internal data matrix. Some of these data structures
must be explicitly specified in order to allow subsequent commands to
recognize the data structure. This part of the manual describes which
data structures are available and, if necessary, explains commands that
must be used to specify these data structures.
3.1
Cross-sectional Data
3.2
Panel Data
3.3
Episode Data
3.4
Sequence Data
3.6
Graphs and Relations
d03.tex
April 19, 1998
3.1
cross-sectional data
1
3.1 Cross-sectional Data
We shall use the term cross-sectional data in a broad sense, meaning a
data structure that simply consists of a set of variables without explicitly
recognizing any additional hierarchical or temporal structure. Thus, a
cross-sectional data structure directly corresponds to a rectangular data
matrix,
···
Case
X1
X2
Xm
1
2
..
.
x11
x21
..
.
x12
x22
..
.
x1m
x2m
..
.
n
xn1
xn2
xnm
containing m variables for n cases. Therefore, in order to interpret TDA’s
internal matrix as a set of cross-sectional data, it is not necessary to use
any further specification. One can directly refer to the variables in the
internal data matrix.
d0301.tex
April 19, 1998
3.2
1
panel data
3.2 Panel Data
We speak of panel data if we have a set of variables for several points
in time. A variable may then be denoted by Xjt where j = 1, . . . , m
identifies the variable and t = 1, . . . , T the point in time. (Of course,
instead of thinking of different points in time, the index t can refer to
any other kind of grouping.) The value of Xjt for the ith individual will
be denoted by xijt .
There are two possibilities to interpret a rectangular data matrix as
a set of panel data. The data for each individual can be given in a single
row, or in a sequence of T rows.
Horizontal Data Organization. In this case we assume that there is
a single row for each individual. The data structures looks as follows.
Xm1
···
X1T
x111
x211
..
.
x1m1
x2m1
..
.
···
···
x11T
x21T
..
.
x1mT
x2mT
..
.
xn11
xnm1
···
xn1T
xnmT
Case
X11
1
2
..
.
n
···
···
XmT
This is TDA’s default data structure for panel data. It has the advantage
that variables that do not vary across time need only be stored once.
d0302.tex
April 19, 1998
3.2
panel data
2
Vertical Data Organization. Alternatively, one can assume that the
data for each individual are given in a block of T data matrix rows. The
data structure is then:
···
Case
Wave
X1
X2
Xm
1
1
..
.
1
2
..
.
x111
x112
..
.
x121
x122
..
.
x1m1
x1m2
..
.
1
2
2
..
.
T
1
2
..
.
x11T
x211
x212
..
.
x12T
x221
x222
..
.
x1mT
x2m1
x2m2
..
.
2
..
.
n
n
..
.
T
..
.
1
2
..
.
x21T
..
.
xn11
xn12
..
.
x22T
..
.
xn21
xn22
..
.
x2mT
..
.
xnm1
xnm2
..
.
n
T
xn1T
xn2T
xnmT
If the data structure is balanced, meaning that we have data (possibly
missing values) for all individuals for each point in time, it should be
possible to create a vertically organized data structure from a horizontal
data structure, and vice versa, by using the nc and nq options in TDA’s
pdata command, see 2.9.
3.3
episode data
1
3.3 Episode Data
The basic concepts for defining event-history data are a time axis, T ,
and a discrete state space, Y. One possibility to represent event-history
data is by an ordered set of state variables: Y (t) (t ∈ T ). If the time
axis is discrete, we can simply think of a sequence of state variables,
Y (t), for t = 0, 1, 2, 3, . . . This representation of event-history data is
called sequence data and will be further discussed in 3.4. An alternative
way to represent event-history data uses the concept of episodes, also
called spells, and we then speak of episode or spell data. This chapter
introduces the concept of episode data as used in TDA. The subsections
are as follows.
3.3.1
Episode Data Concepts
3.3.2
Defining Episode Data
3.3.3
An Example Data Set
3.3.4
Writing Episode Data
3.3.5
Merging Episode Data
d0303.tex
December 22, 1999
3.3.1
1
episode data concepts
3.3.1 Episode Data Concepts
An episode, or spell, is the duration an individual stays in a specific
state. The episode begins at the time of entry into that state, and it
ends when a new state is entered. Of course, the definition of episodes
depends on what is regarded as states; and this, in turn, depends on the
substantive issue in question. As implied by the term event-history data,
we always assume a discrete state space.
We speak of episode data if there is a sample of i = 1, . . . , N episodes.
Thinking in terms of individuals, there might be just one episode for
each individual, or each individual might contribute a varying number
of episodes to the data set. Accordingly, one sometimes distinguishes
between single and multi-episode data. Of course, also multi-episode data
where each individual contributes a varying number episodes can be
viewed as just a sample of individual episodes. This view will be taken
until the end of this section where a few more remarks about multiepisode data will be given.
Each of the episodes can be described formally by an expression like
(oi , di , si , ti , xi (t))
i = 1, . . . , N
(1)
si and ti are the starting and ending times, respectively. Since measurement is always discrete one has to adopt a convention about coding these
variables. The basic convention for TDA is that starting and ending times
are coded such that ti − si is the duration of the episode.
oi is the origin state, the state held during the episode until the ending time, and di is the destination state defined as the state reached at
the ending time of the episode. xi (t) is a vector of covariates connected
with the episode and possibly depending on the process time, t. To simplify notation, we generally use the convention that covariates are given
by row vectors and parameters (coefficients) are given by column vectors.
One more piece of information could be added to the description of
episodes given in (1): case weights. It would be straightforward, then,
to use such weights in all formulas based on a sample of episode data.
To simplify the notation we will not write the formulas in this text
with weights. However, if case weights have been defined with the cwt
command, these case weights will be used in all methods based on episode
data.
d030301.tex
April 19, 1998
3.3.1
episode data concepts
2
In all applications the sets of origin and destination states are finite.
It is assumed throughout this text that they are coded by non-negative
integers, including zero. The set of possible origin states is called O.
Then, for each j ∈ O, there is a set of different destination states called
Dj . Accordingly, with this terminology, it is assumed that k 6= j if j ∈ O
and k ∈ Dj . Clearly, if an episode is right censored there is no transition
to a destination state different from the origin state. The set of all possible destination states, for episodes with origin j ∈ O, ending with an
event or are censored, is Dj ∪ {j}.
In addition, we generally use the following notation. Each pair (j, k)
with j ∈ O and k ∈ Dj is called a transition. Nj is the set of all episodes
with origin state j; Ejk is the set of all episodes with a transition from
origin state j to a different destination state k ∈ Dj ; Zj is the set of all
censored episodes with origin state j; and Zjk is the set of all episodes
with origin state j and with a destination state not equal to k.
The Time Axis. As with any longitudinal data it is important how the
time axis is defined and the relevant dates are measured. We distinguish
three different aspects of this problem.
• One needs a time axis that is suitable for the substantive application.
This, of course, depends on the kind of events that one intends to model.
In sociological applications, this will normally be a discrete time axis, for
instance days. The important point is that all events that are normally
investigated in sociological research have some inherent duration, for
instance a birth or becoming married.
• In order to formulate a formal (statistical) model we then need a
mathematical representation of time. Here we have two choices; we can
use a discrete or a continuous time axis. In our view, there are no principal arguments to prefer one over the other. It is often convenient to
use a continuous time axis, and in fact, most transition rate models discussed in 6.17 use a continuous time axis to mathematically represent
the substantive process.
• It remains the question of how dates of events have been measured
and are available in a given set of episode data. This information is, of
course, always given in discrete time units. For instance, in sociological
and demographic research, dates are normally given by the month when
an event happened. The problem then is how to relate these empirical
dates to the mathematical time axis that has been used to define the
model.
3.3.1
episode data concepts
3
No specific problem arises if the mathematical time axis is discrete with
the same time units as used for the empirical dates. Otherwise, there
are two different possibilities. One can simply assume that the empirical
dates can be exactly mapped to the mathematical time axis; or one can
represent the empirical dates by time intervals on the mathematical time
axis. Current applications of event-history methods almost always use
the method of simply identifying empirical dates with some time points
on the mathematical time axis.
If empirical dates are measured very imprecisely, the second approach, based on mapping these dates to time intervals on the mathematical time axis, would be preferable. This would then appropriately
reflect the fact that if dates are measured very imprecisely, we only know
that the event happened in some time interval. This is sometimes called
interval censored dates. Of course, there is no clear-cut demarcation line
between precisely measured and interval censored dates. In any case, statistical methods for interval censored data are currently not supported
by TDA.
Right Censored Data. If event histories could be observed completely,
each individual will eventually reach an absorbing state. However, each
sample of episode data is based on a limited observation period. Most
probably, it will contain some episodes without an observed transition
to a new destination state. These episodes are called right censored .
Since almost all episode data sets contain at least some right censored
episodes, this must be taken into account by all methods appropriate
for such data. In order to do this one needs some information about the
process that generated the censoring. One can distinguish, at least, three
different types of censoring (Lawless [1982, p. 31]).
• Type I Censoring. This is a special type of censoring arising in experiments where the test is for the duration in some state. When there is
a fixed time period during which a set of individuals, or other units of
analysis, is exposed to some risk of leaving the initial state, and if this
fixed time period is the same for all units, then the resulting censoring
is called type I censoring.
• Type II Censoring. This type of censoring again arises in experiments
where the test is for the duration in some state. But now there is not
a fixed time period for observation, but the experiment is stopped after
an event has occurred for a fixed fraction of all units.
• Random Censoring. This third type of censoring is more general and
3.3.1
episode data concepts
4
not restricted to experiments. It is used to describe situations where the
finite observation period for a sample of episode data, which generates
the resulting censoring, is statistically independent from the durations
that are to be studied.
In sociological research, based mainly on survey data and follow-up studies, random censoring can be assumed in most cases. Fortunately, in
practice, it is not necessary to deal separately with all different types
of censoring. There is only one really important assumption: that the
process generating the censoring works independently from the process
that generates the transitions one wishes to investigate. All methods to
analyze episode data, currently implemented in TDA, are based on this
assumption.
Left Censored and Truncated Data. An Episode is called left censored if we do not know its starting time. This is a more complicated situation because there is simply no substitute for the missing information.
Sometimes it will be possible to assume a time interval for the starting
time of a left censored episode. If this would be possible, left censored
episodes could be treated as a special case of interval censored episodes.
But this is currently not supported by TDA and we consequently assume
that there is an approximately exact starting time for all episodes to be
used with TDA.
A quite different situation arises if the observation of episodes is left
truncated , meaning that the episode is not observed from its beginning,
but we are able to find out its starting time retrospectively. Most statistical methods offered by TDA can also be used with left truncated data
as will be explained when describing the methods.
Basic Statistical Concepts. We now introduce some statistical concepts for representing episode data defined on a continuous time axis.
Analogous concepts for a discrete time axis will be defined in 6.17.6.1.
Assuming that the episodes are defined on a process time axis where
each episode begins at time zero, and assuming that there is only a single
destination state, a sample of episodes can be completely represented by
a non-negative stochastic variable T . The distribution of such a variable
can be described by a density function f (t), or by a distribution function
F (t), with the simple relation
Z
Pr (T ≤ t) = F (t) =
t
f (τ ) dτ
0
3.3.1
episode data concepts
5
The use of well-behaved distributions, i.e. F (t) → 1 for t → ∞, implies
that all episodes end at some point in time with an event. But obviously,
this is not always true. For instance, if we are interested in the distribution of durations, starting with birth and ending with first marriage,
then one has to provide for the possibility that some people will never
get married. An appropriate way to model such situations is to allow for
two or more alternative destination states. In this example, one possible
destination state is to become married; and an alternative destination
state is to die before becoming married.1
Two more concepts are often used. First the concept of a survivor
function, generally denoted by G(t). It gives the probability of not to
have an event until the point in time t, so the definition is
G(t) = Pr (T > t) = 1 − F (t)
Another important concept, called the hazard or transition rate, is defined by
r(t) = lim
∆t→0
Pr (t ≤ T < t + ∆t | T ≥ t)
∆t
(2)
The numerator of this expression is the conditional probability of having
an event in a small time interval from t to t + ∆t, conditional on having
no event until t. Then this conditional probability is measured per unit
of time and the limit is taken. Consequently, the transition rate describes
the instantaneous rate, or sometimes called the risk , of having an event
at time t.
As is easily seen, the transition rate can be expressed by the density
and the survivor function of the duration variable T .
r(t) =
f (t)
G(t)
(3)
Therefore, all four concepts are mathematically equivalent. The density
function, the distribution function, the survivor function, and the transition rate can equally well be used to describe the duration of episodes.
1 Another approach is to use so-called mover-stayer models, based on the assumption
that there is an unknown, but estimable proportion of individuals who will never
experience a transition into the destination state. Whether this is a sensible assumption depends on the application. For instance in life course research, the assumption
conflicts with the basic view that life courses are contingent developmental processes.
In our example, this would mean that the risk of becoming married remains until
death.
3.3.1
episode data concepts
Furthermore, from (3) one gets the important equation
Z t
G(t) = exp (−H(t)) = exp −
r(τ ) dτ
6
(4)
0
with
Z
H(t) =
t
r(τ ) dτ
0
called the cumulative hazard or transition rate.
Alternative Destination States. So far we only considered transitions
from a single origin state into a single destination state. Duration was
defined as staying in the origin state until a transition to the destination
state occurred, and if none occurred during the observation period, the
episode was regarded as right censored. However, there may be more
than one transition of interest represented in a sample of transition data.
There may be more than one origin state, and, for each origin state, there
may be more than one destination state.
The possibility of more than one origin state can be dealt with by
conditioning all descriptions and modeling on a given origin state. This
is the standard approach: all analyses are done conditional on being in
a given origin state. For instance, one may be interested in the duration
of poverty spells; descriptions of these durations are then given only for
persons who entered into a state of poverty.
Somewhat more involved is the treatment of situations where a given
origin state can be followed by one of several possible destination states.
We then have to treat the destination state variable as stochastic. Conditional on a given origin state j, we can conceptualize such a situation
by a two-dimensional stochastic variable (Tj , Dj ). The variable Tj gives
the duration in the origin state j until any event occurs, or until the
episode becomes right censored, and Dj ∈ Oj ∪ {j} gives the destination
state reached at Tj , i.e. at the end of the episode, conditional on the
origin state j.
The goal then is to describe this two-dimensional variable.2 We can
first look at the marginal duration variable Tj , i.e. the duration in the
2 Cf. for the following concepts Blossfeld et al. [1986, p. 59], and Lancaster [1990, p. 99].
A discussion of the competing risks problem in the biometrical context is given by
Gail [1975]. However, we should clearly distinguish between the biometrical concept
of competing risks and situations with alternative destination states. In the classical
competing risks case, there is only a single destination state (death of an individual,
3.3.1
7
episode data concepts
origin state j until any event occurs. The distribution of this variable
may be described by overall survivor and density functions, and by an
overall transition rate
Gj (t) = Pr (Tj > t)
fj (t) = −
rj (t) =
d
Gj (t)
dt
fj (t)
Gj (t)
With these concepts we get a description for the duration as if we had
collapsed all different destination states into only one. But, in fact, we are
interested in the different risks associated with the different destination
states. So the concepts must be distinguished accordingly.
This is most easily done using the transition rate.3 In generalizing
(2) the transition rate from the given origin state j to destination state
k ∈ Dj is defined as
Pr (t ≤ Tj < t + ∆t, Dj = k | Tj ≥ t)
(5)
∆t
This gives the risk, or the conditional probability if we look at a small
time interval ∆t, that a transition from the given origin state j to the
destination state k takes place at t, or in the time interval [t, t + ∆t),
given that no event occurred until time t.
Because we always assume that the destination states are mutually
exclusive, the overall transition rate rj (t), from the given origin state j
to any destination state, can be expressed as
X
rj (t) =
rjk (t)
rjk (t) = lim
∆t→0
k∈Dj
Using the general equality (4) it follows that
Z t
Z t
Y
Gj (t) = exp −
rj (τ ) dτ =
exp −
rjk (τ ) dτ
0
k∈Dj
0
or failure of a machine), but two or more risk factors can lead to this event. In typical
sociological applications the situation is quite different: individuals can leave a given
origin state into one of several possible destination states.
3 In the case of two or more destination states, the transition or hazard rate is often
called a transition intensity. However, it should not cause undue confusion to use the
same term transition rate in all cases.
3.3.1
episode data concepts
8
This shows that the overall survivor function Gj (t) may be written as a
product of pseudosurvivor functions defined as
Z t
G̃jk (t) = exp −
rjk (τ ) dτ
0
They are called this way because there is no direct survivor function
interpretation of these functions. They are, however, quite useful when
deriving likelihood expressions for transition rate models.
To get concepts for a direct interpretation of destination-specific transition rates we continue with a definition of transition-specific distribution functions F̃jk (t), giving the probability that a transition to destination state k, from the given origin state j, takes place until t.
F̃jk (t) = Pr (Tj ≤ t, Dj = k)
(6)
Although the meaning of this concept is obvious it is not a proper distribution function in the normal sense, since we have
F̃jk (∞) = πjk
where πjk is the probability of reaching, at any point of time, the destination state k, given the origin state j; and if there is more than one possible
destination state this probability is clearly less then one. Therefore, (6)
is sometimes called a subdistribution function, e.g. by Kalbfleisch and
Prentice [1980, p. 167]. The same is to be said if we look at the density
functions
d
f˜jk (t) = F̃jk (t)
dt
This, again, is not a proper density function, because it does not integrate
to unity, and so it is sometimes called a subdensity function. However,
we have
Pr (t ≤ Tj < t + ∆t, Dj = k)
f˜jk (t) = lim
∆t→0
∆t
(7)
and so it is easy to adjust our definitions to become
Fjk (t)
=
F̃jk (t)
πjk
(8)
fjk (t)
=
f˜jk (t)
πjk
(9)
3.3.1
episode data concepts
9
Now Fjk (t) and fjk (t) may be regarded as concepts describing distributions in the usual sense.4 The range of this distributions may be thought
of as the set of all individuals who leave the given origin state for a
specific destination state. And the overall distribution Fj (t) may be regarded as a mixture of these destination-specific distributions, with the
πjk as weights:
X
X
Fj (t) =
F̃jk (t) =
πjk Fjk (t)
(10)
k∈Dj
k∈Dj
It seems not sensible to define destination-specific survivor functions
since this would imply a “conditioning on the future”. It is quite possible,
however, to express destination-specific transition rates by the overall
survivor function and destination-specific densities:
rjk (t) =
f˜jk (t)
fjk (t)
= πjk
Gj (t)
Gj (t)
(11)
The first part follows from (5) and (7), the second part follows from (9).
The focus of modeling can finally be described as follows: Estimation
and evaluation of models for stochastic variables (Tj , Dj ) that measure
the duration of being in an origin state j until a transition to a destination state Dj occurs. This is done conditionally on a given origin
state.
Multi-episode data. The term multi-episode data is normally used to
denote a situation where each individual contributes a varying number of
episodes to a given episode data set. There is then a set, U, of individuals,
and for each individual u ∈ U, there is a series of Mu episodes (Mu ≥ 1).
This results in a set of N episodes,
X
N=
Mu
u∈U
A formal representation can be given similar to the single episode case.
One only has to add, for each single episode, two more pieces of information: an identification number of the individual to which the episode
belongs, and the serial number of the episode. A complete description of
a sample of multi-episode data is then given by
(ui , mi , oi , di , si , ti , xi )
4 We
i = 1, . . . , N
(12)
assume that the probabilities πjk add to unity. This assumption can be met by
defining a complete destination state space.
3.3.1
episode data concepts
10
where ui is the identification number of the individual, or any other unit
of analysis, the ith episode belongs to; and mi is the serial number of
the episode. All other variables are defined as in the single episode case.
oi is the origin state, di is the destination state, si is the starting time,
ti is the ending time, and xi is a vector of covariates associated with the
episode.
Compared with the single episode case, there are now a few more
consistency requirements. It is not only required that the duration of
each episode is greater than zero. In addition, there is an ordering of the
episodes for each individual, given by the set of serial numbers for the
episodes. In particular, it is required that the starting time of an episode
is not less than the ending time of a previous episode.
As already mentioned, a set of multi-episode data can also be viewed
as a set of single episodes, simply disregarding the connection between
episodes and individuals. Of course, statistical models for episode data
should then take into account that episodes contributed by the same
individual are probably not independent. However, this requirement is
conditional on covariates and it is possible, therefore, to model multiepisode data by adding, for each episode, covariates representing relevant
aspects of its past history.
While TDA is able to recognize the distinction between single and
multi-episode data, transition rate modeling is normally not done with
multi-episode data. Instead, one uses the just mentioned approach. The
data set is treated as a set of single episodes, and covariates are added to
control for serial dependence across time. In fact, the concept of multiepisode data is only necessary when the analytical focus is not on transitions, but on describing individual event histories over a longer period
of time. However, episode data structures are not well suited for this
purpose. The concept of sequence data provides a useful alternative, see
3.4.
3.3.2
defining episode data
1
3.3.2 Defining Episode Data
The basic command for defining an episode data structure is edef with
syntax shown in Box 1. The first four parameters are required.1 The
parameter
org = expression,
must be used to define an origin state for the episodes. expression
can be a numerical constant, for instance, org=0 would define a zero
origin state for all episodes; or can refer to a data matrix variable, for
instance org=ORG would define the origin state by referring to variable
ORG. Also more complicated expression are possible but, in general, it
is more efficient to define variables outside the edef command. In the
same way, one has to use
des = expression,
to define a destination state,
ts = expression,
to define a starting time, and
tf = expression,
to define the ending time of the episodes. Note that origin and destination states must be nonnegative integers; also all starting times must
be nonnegative, and each episode must have a strictly positive duration,
always calculated as tf - ts.
Given these parameters in the edef command, TDA creates an episode
data structure based on the currently selected cases of the data matrix.
This data structure remains active until it is replaced by a new edef command, the tsel command is used for a new temporary case selection, or
the clear command is used to remove a variable required to maintain
the episode data structure. If TDA has successfully created an episode
1 If the edef command is used without any parameters it provides information about
the currently defined episode data (if any).
d030302.tex
May 29, 2002
3.3.2
defining episode data
2
Box 1 Syntax for edef command
edef (
org=...,
des=...,
ts=...,
tf=...,
id=...,
sn=...,
maxtran=...,
VName=...,
...
split=...,
origin state
destination state
starting time
ending time
ID number
spell number
maximum number of transitions, def. 100
definition of type 5 variables
can be repeated
variables for episode splitting
);
data structure, it shows a table with information about all transitions in
the standard output.
Except for org, des, ts and tf, all other parameters in the edef
command are optional.
1. The maxtran parameter can be used to change the maximum number
of transitions, default is maxtran=100.
2. The id and sn parameters can be used to define multi-episode data.
The syntax is
id = expression,
to define an ID number for the episodes, and
sn = expression,
to define spell numbers. TDA then assumes that the episodes in the data
matrix are ordered with respect to the ID variable. Spell numbers must
be nonnegative integers and should be in ascending order.
3. The split parameter can be used to define variables for episode
splitting. If used, it must be the last parameter in the edef command.
The syntax is
split = varlist,
where varlist is a list of variables separated by commas. Type 5 variables (see below) cannot be used for episode splitting. Splitting is done as
3.3.2
defining episode data
3
follows. Let (oi , di , si , ti ) denote the ith episode and Si the corresponding
value of one of the variables specified in varlist. Then, if si < Si < ti ,
the episode is split into two parts: (oi , oi , si , Si ) and (oi , di , Si , ti ). This
is done recursively for all variables in varlist. Note that episode splitting does not change the internal data matrix. Episode splitting is only
performed when (and while) procedures request episode data. In fact,
currently only three commands recognize episode splitting: the epdat
command for writing episode data into an output file and the rate and
frml commands for estimating transition rate models. If the split parameter was used in the edef command, these procedures simply get
splits whenever they request a new episode.
4. The edef command can also be used to create temporary (type 5)
variables based on the episode data structure. The syntax is
VName <s>[pfmt] = expression,
and is identical to the syntax used for defining variables with the nvar
command (see 2.2). VName is the variable name, <s> is an optional storage
size, default is 4, and [pfmt] is an optional print format. In addition to
all standard operators, expression may also contain the episode data
operators, org, des, ts, tf, and sn, described in 5.2.8. These operators
provide the corresponding values for the current episode, or split, and
can be used to create time-varying variables based on episode splitting.
3.3.3
an example data set
1
3.3.3 An Example Data Set
To illustrate episode data we use an example data set taken from the
German Life History Study (GLHS) collected at the Max Planck Institut
für Bildungsforschung in Berlin. The same data set is used in Blossfeld
and Rohwer [1995]. We thank Karl Ulrich Mayer and Hans Peter Blossfeld who kindly provided this data set.
The GLHS provides retrospective information about the life histories
of men and women from the birth cohorts 1929–31, 1939–41, and 1949–
51, collected in the years 1981–1983 (Mayer and Brückner, 1989). Our
example data set contains 600 job episodes from 201 randomly selected
respondents. Each record in this file represents an employment episode,
and the consecutive jobs of a respondent’s career are stored successively
in the file. For some individuals there is only a single job episode, whereas
for others there is a sequence of two or more jobs.
The data file, rrdat.1, contains 12 variables that are described briefly
in Box 2. Column refers to the position of the variable in the data file,
which is free format, meaning that the numerical entries are separated
by a blank character.
ID identifies the individuals in the data set. Because the data file contains
information about 201 individuals, there are 201 different ID numbers.
The numbers are arbitrarily chosen and are not contiguous.
NOJ gives the serial number of the job episode, always beginning with
job number 1. For instance, if an individual in our data set has had
three jobs, the data file contains three records for this individual with
job number 1, 2, and 3, respectively. Note that only job episodes
are included in this data file. If an individual has experienced an
interruption between two consecutive jobs, the difference between the
ending time of a job and the starting time of the next job may be
greater than 1.
TS is the starting time of the job episode, in century months. (A century
month is the number of months from the beginning of the century;
1 = January 1900.) The date given in this variable records the first
month in a new job.
TF is the ending time of the job episode, in century months. The date
given in this variable records the last month in the job.
SEX records the sex of the individual, coded 1 for men and 2 for women.
d030303.tex
April 19, 1998
3.3.3
an example data set
2
Box 2 Variables in data file rrdat.1
Variable Column Description
----------------------------------------------------ID
C1
ID of individual
NOJ
C2
Serial number of the job
TS
C3
Starting time of the job
TF
C4
Ending time of the job
SEX
C5
Sex (1 men, 2 women)
TI
C6
Date of interview
TB
C7
Date of birth
TE
C8
Date of entry into the labor market
TM
C9
Date of marriage (0 if no marriage)
PRES
C10
Prestige score of job i
PRES1
C11
Prestige score of job i + 1
EDU
C12
Highest educational attainment
Box 3 First records of data file rrdat.1
ID NOJ
TS
TF SEX
TI
TB
TE
TM PRES PRES1 EDU
-------------------------------------------------------1
1 555 982
1 982 351 555 679
34
-1
17
2
1 593 638
2 982 357 593 762
22
46
10
2
2 639 672
2 982 357 593 762
46
46
10
2
3 673 892
2 982 357 593 762
46
-1
10
3
1 688 699
2 982 473 688 870
41
41
11
3
2 700 729
2 982 473 688 870
41
44
11
3
3 730 741
2 982 473 688 870
44
44
11
3
4 742 816
2 982 473 688 870
44
44
11
3
5 817 828
2 982 473 688 870
44
-1
11
TI is the date of the interview, in century months. Using this information,
one can decide whether an episode is right censored or not. If the
ending time of an episode (TF) is less than the interview date, the
episode ended with an event, otherwise the episode is right censored.
TB records the birth date of the individual, in century months. Therefore,
TS minus TB is the age, in months, at the beginning of a job episode.
TE records the date of first entry into the labor market, in century months.
TM records whether/when an individual has married. If the value of this
variable is positive, it gives the date of marriage (in century months).
For still unmarried individuals at the time of the interview, the variable is coded 0.
PRES records the prestige score of the current job, that is, the job episode
in the current record of the data file.
3.3.3
an example data set
3
Box 4 Command file ed1.cf (single episode data)
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
# define additional variables
COHO1
COHO2
COHO3
W
=
=
=
=
ge(TB,348) & le(TB,384),
ge(TB,468) & le(TB,504),
ge(TB,588) & le(TB,624),
SEX[2],
#
# birth
# birth
# birth
women =
cohort 1
cohort 2
cohort 3
1
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
);
edef(
ts = 0,
tf = DUR,
org = 0,
des = DES,
# define single episode data
#
#
#
#
starting time
ending time
origin state
destination state
);
PRES1 records the prestige score of the consecutive job, if there is a next job,
otherwise a missing value (-1) is coded.
EDU records the highest educational attainment before entry into the labor
market. Lower secondary school qualification (Hauptschule) without
vocational training is equivalent to 9 years, middle school qualification (Mittlere Reife) is equivalent to 10 years, lower secondary school
qualification with vocational training is equivalent to 11 years, middle
school qualification with vocational training is equivalent to 12 years.
Abitur is equivalent to 13 years, a professional college qualification
is equivalent to 17 years, and a university degree is equivalent to 19
years.
3.3.3
an example data set
4
Box 5 Output from ed1.cf command (single episode data)
> edef(...)
----------------------------------------------------------------Creating new single episode data. Max number of transitions: 100.
Definition: org=0, des=DES, ts=0, tf=DUR
Mean
SN Org Des Episodes
Weighted
Duration
TS Min
TF Max Excl
-------------------------------------------------------------------1
0
0
142
142.00
128.18
0.00
428.00
1
0
1
458
458.00
49.30
0.00
350.00
Sum
600
600.00
Number of episodes: 600
Successfully created new episode data.
Box 3 shows the first nine records of data file rrdat.1. Note that all
dates are coded in century months. Thus, 1 means January 1900, 2 means
February 1900, 13 means January 1901, and so on. In general:
YEAR = floor((DATE − 1) / 12) + 1900
MONTH = (DATE − 1) % 12 + 1
where DATE is given in century months, and MONTH and YEAR refer to
calendar time. The floor operator provides the largest integer less than,
or equal to, its argument, and “%” is the modulus operator .1 For instance,
the first individual (ID = 1) has a single job episode. The starting time
is given as century month 555, corresponding to March 1946, and the
ending time is 982 = October 1981. Because this is equal to the interview
month, the episode is right censored.
Example 1 Command file ed1.cf in Box 4 illustrates how to use the
example data file rrdat.1 to define single episode data. The standard
output from the edef command is shown in Box 5. 458 episodes end
in an event, transition from 0 to 1, the mean duration is 128 months;
142 episodes are right censored. Of course, mean duration for censored
episodes is no reasonable estimate but might provide useful information
about the data. Since we have not used the cwt command to define
case weights, the number of episodes with and without weights are the
same. The column labelled TSMin shows the minimum starting time, the
1 Given
two integer numbers, n and m, n % m is the remainder after dividing n by
m. For instance: 13 % 12 = 1.
3.3.3
an example data set
5
Box 6 Command file ed2.cf (alternative destination states)
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
# define additional variables
COHO1
COHO2
COHO3
W
=
=
=
=
ge(TB,348) & le(TB,384),
ge(TB,468) & le(TB,504),
ge(TB,588) & le(TB,624),
SEX[2],
#
# birth
# birth
# birth
women =
cohort 1
cohort 2
cohort 3
1
# three alternative destination states
DES [1.0] = if eq(TF,TI) then 0
else if gt(PRESN/PRES - 1,0.2) then 1
else if lt(PRESN/PRES - 1,0.0) then 3 else 2,
DUR [3.0] = TF - TS + 1,
);
edef(
# define single episode data
ts = 0,
tf = DUR,
org = 0,
des = DES,
#
#
#
#
starting time
ending time
origin state
destination state
);
column labelled TSMax shows the maximum ending time for the corresponding episodes. The final column, labelled Excl, is only used when
all episodes, for a given origin state, are right censored. There are then
no transitions and these episodes will not be used for model estimation.
Command file ed1.cf will be the starting point for many examples
in the rest of this part of the User’s Manual. We simply add further
3.3.3
an example data set
6
Box 7 Output from ed2.cf command (alternative destination states)
> edef(...)
----------------------------------------------------------------Creating new single episode data. Max number of transitions: 100.
Definition: org=0, des=DES, ts=0, tf=DUR
Mean
SN Org Des Episodes Weighted
Duration
TS Min
TF Max Excl
------------------------------------------------------------------1
0
0
142
142.00
128.18
0.00
428.00
1
0
1
84
84.00
45.05
0.00
326.00
1
0
2
219
219.00
45.91
0.00
350.00
1
0
3
155
155.00
56.40
0.00
332.00
Sum
600
600.00
commands using the episode data defined with the edef command.
Example 2 Command file ed2.cf in Box 6 illustrate the definition of
single episode data with alternative destination states. The command
file is almost identical with ed1.cf, only the definition of destination
states (variable DES) has been changed. We now distinguish three different destination states depending on the kind of occupational move.
Part of the standard output from ed2.cf is shown in Box 7. 84
episodes end in an upward move, 155 in a downward move, and 219
do not change the prestige level. The number of right censored episodes
has, of course, not changed.
This command file file will often be used in later sections to illustrate
statistical methods for episodes with alternative destination states.
Example 3 To illustrate the definition of multi-episode data, we use
command file ed3.cf (not shown). It is almost identical with ed1.cf,
only the parameters id=ID and sn=SN have been added to the edef
command. Part of the standard output is shown in Box 8. The result
is typical for multi-episode data. All 201 individual have a first episode,
somewhat less have also a second episode, only few have more than three
or four episodes. In this example, there are up to 9 episodes, but only a
single individual has, in fact, these 9 episodes. Consequently, this way of
partitioning episode data is not well suited for statistical analysis. While
it would be possible, in principle, to estimate different models for each
spell number, this is practically not possible with most multi-episode
data sets; there are simply not enough data for higher spell numbers.
3.3.3
an example data set
Box 8 Part of standard output from ed3.cf (multi-episode data)
> edef(...)
---------------------------------------------------------------Creating new multi-episode data. Max number of transitions: 100.
Definition: id=ID, sn=SN, org=0, des=DES, ts=0, tf=DUR
Mean
SN Org Des Episodes Weighted
Duration
TS Min
TF Max Excl
------------------------------------------------------------------1
0
0
16
16.00
233.44
0.00
428.00
1
0
1
185
185.00
52.73
0.00
326.00
Sum
201
201.00
------------------------------------------------------------------2
0
0
36
36.00
150.17
0.00
407.00
2
0
1
126
126.00
52.30
0.00
350.00
Sum
162
162.00
------------------------------------------------------------------3
0
0
38
38.00
121.74
0.00
329.00
3
0
1
69
69.00
50.36
0.00
332.00
Sum
107
107.00
------------------------------------------------------------------4
0
0
24
24.00
85.33
0.00
340.00
4
0
1
38
38.00
37.71
0.00
146.00
Sum
62
62.00
------------------------------------------------------------------5
0
0
9
9.00
110.22
0.00
328.00
5
0
1
23
23.00
22.61
0.00
70.00
Sum
32
32.00
------------------------------------------------------------------6
0
0
8
8.00
45.75
0.00
127.00
6
0
1
12
12.00
43.75
0.00
112.00
Sum
20
20.00
------------------------------------------------------------------7
0
0
7
7.00
114.14
0.00
295.00
7
0
1
4
4.00
52.75
0.00
96.00
Sum
11
11.00
------------------------------------------------------------------8
0
0
3
3.00
65.00
0.00
172.00
8
0
1
1
1.00
72.00
0.00
72.00
Sum
4
4.00
------------------------------------------------------------------9
0
0
1
1.00
34.00
0.00
34.00
*
--------------------------------------------------------------------Sum
600
600.00
7
3.3.4
1
writing episode data
3.3.4 Writing Episode Data
Having defined episode data, they can be written into an output file. This
is particularly useful for episode splitting. The command is epdat with
syntax shown in Box 1. All parameters, except the output file name
on the right-hand side, are optional. The epdat command writes the
currently defined episode data into the specified output file; the ap=1
parameter can be used to append the data to the end of an already
existing file. The maximum number of cases can be controlled with the
noc parameter. As default, the following variables are written:
1. The ID number or, for single episode data, simply the case number,
print format: 6.0;
2. The spell number; always 1 for single episode data;
print format 3.0;
3. The number of splits; always 1 if without episode splitting;
print format 3.0;
4. The current split number; always 1 if without episode splitting;
print format 3.0;
5. The origin stae; print format 3.0;
6. The destination state; print format 3.0;
7. The starting time;
8. The ending time. The default print format for the starting and ending
time variables is 6.2 but may be changed with the fmt parameter).
To add further variables to the output file, one can use the parameter
v = varlist,
where varlist is a list of variables separated by commas. These can be
any variables contained in the current data matrix. In particular, one can
use type 5 variables defined inside the currently active edef command.
As a further option, one can use the dtda parameter to request an
additional output file containing a description of the output data file.
This description follows the TDA syntax for reading data files and can
directly be used as a command file.
d030304.tex
April 19, 1998
3.3.4
writing episode data
2
Box 1 Syntax for epdat command
epdat (
noc=...,
fmt=...,
v=...,
ap=1,
dtda=...,
) = fname;
maximum number of cases
print format for ts and tf, def. 6.2
list of additional variables
append new data to end of output file
TDA description of output file
Box 2 Episode data file ed1.dat
ID SN ORG DES TS TF S1 S2
------------------------1 1 1
2
0 10 11 20
1 2 2
3 10 12 11 20
1 3 3
3 12 30 11 20
2 1 2
1
0 20 15 8
2 2 1
1 20 35 15 8
Example 1 To illustrate the epdat command we use the example data
ed1.dat, shown in Box 2. There are two covariates that will be used
for episode splitting. The command file, ed4.cf, is shown in Box 3. It
first reads the data with an nvar command and then defines a single
episode data structure. The episodes are split with variables S1 and
S2. In addition, the edef command defines two time varying dummy
variables, SD1 and SD2. They switch from 0 to 1 as soon as the process
time reaches the dates given in S1 and S2, respectively.
The resulting output file, d, is shown in Box 4. (The header, shown in
this box, is not written into the output file.) How the program performs
episode splitting should be obvious from this example. For instance, the
fourth episode is split two times, first based on S2 = 8, then with S1 =
15. Also the definition of the time-dependent dummy variables should
be easily understandable. For instance, SD1 goes from 0 to 1 as soon as
S1 becomes greater than, or equal, to the starting time of the current
split.
3.3.4
writing episode data
Box 3 Command file ed4.cf
nvar(
dfile = ed1.dat,
# data file
ID
SN
ORG
DES
TS
TF
S1
S2
#
#
#
#
#
#
#
#
[1.0]
[1.0]
[1.0]
[1.0]
[2.0]
[2.0]
[2.0]
[2.0]
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
identification number
spell number
origin state
destination state
starting time
ending time
first covariate
second covariate
#
#
#
#
starting time
ending time
origin state
destination state
);
edef(
ts = TS,
tf = TF,
org = ORG,
des = DES,
SD1
SD2
= le(S1,ts), # time-dependent dummy
= le(S2,ts), # time-dependent dummy
split = S1,S2,
# episode splitting
# must be the last parameter
);
epdat(
v = S1,SD1,S2,SD2,
dtda = t,
) = d;
# write data to d
Box 4 Output file, d, created by command file ed4.cf
ID SN NSP SPN ORG DES
TS
TF
S1 SD1 S2 SD2
-------------------------------------------------1
1
1
1
1
2
0.00 10.00 11 0 20 0
2
1
2
1
2
2 10.00 11.00 11 0 20 0
2
1
2
2
2
3 11.00 12.00 11 1 20 0
3
1
2
1
3
3 12.00 20.00 11 1 20 0
3
1
2
2
3
3 20.00 30.00 11 1 20 1
4
1
3
1
2
2
0.00
8.00 15 0
8 0
4
1
3
2
2
2
8.00 15.00 15 0
8 1
4
1
3
3
2
1 15.00 20.00 15 1
8 1
5
1
1
1
1
1 20.00 35.00 15 1
8 1
3
3.3.5
1
merging episode data
3.3.5 Merging Episode Data
The ejoin command can be used to process one or two episode data
files. In any case, if episodes are overlapping, they are split in such a way
that overlapping parts get appropriate levels. If there are two episode
data files, they are merged into one episode data file. The syntax of the
command is shown in the following box.
ejoin (
if1=...,
if2=...,
max=...,
nw=...,
len=...,
noc=...,
fmt0=...,
fmt1=...,
fmt2=...,
name of first input file
name of second input file
max block size, def. 1000
max number of levels, def. 1
max record length, def. 1000
read maximal noc records, def. all
print format for basic variables
print format for covariates, file 1
print format for covariates, file 2
) = output file;
Input files must be free-format files with at least 6 numerical entries in
each record as follows:
1. Case Id used to identify blocks,
2. Number of records (spells) in current block,
3. Record (spell) number in current block,
4. Starting time of spell,
5. Ending time of spell,
6. State (must be a non-negative integer).
Any additional entries are treated as covariates. The number of entries
is determined from the first data record in each file.
d030305.tex
December 22, 1999
3.3.5
merging episode data
2
Note that it is assumed that the input data files are sorted first with
respect to the Id variable and, inside each block, with respect to the
starting times of the spells.
Each record of the output file will contain the following entries.
1. Case Id,
2. Number of records (spells) in current block,
3. Record (spell) number in current block,
4. Level number (0,1,2,. . . ),
5. Starting time of spell,
6. Ending time of spell,
7. State taken from file 1
8. State taken from file 2 (only if there are two input files).
The print format for these entries can be specified with the fmt0 parameter. Following these first 7 or 8 entries will be written any covariates
that may be present in file 1; the print format for these entries can be
controlled with the fmt2 parameter. Finally follow any covariates from
file 2 if there are any; the print format can be controlled with the fmt2
parameter. If the number of entries in the format parameters is less than
required the last one is repeated.
Example 1 To illustrate the ejoin command we use the two episode
data files shown in Box 1. Both files contain overlapping episodes. The
ejoin command shown in the same box merges these data files into a
single one as shown in Box 2.
3.3.5
merging episode data
Box 1 Two episode data files and a command file (ej1.cf)
ej1.dat
------------------------------ID NS SN TS TF S
X1
X2
------------------------------1
2
1
0
2 1 1.1 11.1
1
2
2
1
5 2 2.2 22.2
2
4
1
3
6 3 3.3 33.3
2
4
2
4
5 4 4.4 44.4
2
4
3
5
8 5 5.5 55.5
2
4
4
9 10 6 6.6 66.6
ej2.dat
-------------------------ID NS SN TS TF S
Y1
-------------------------1
2
1
1
4 3 5.5
1
2
2
3
6 4 6.6
ejoin(
if1 = ej1.dat,
if2 = ej2.dat,
nw = 3,
fmt0 = 4,
fmt1 = 8.4,
fmt2 = 8.4,
) = ej1.d;
Box 2 Output file created by ej1.cf
ID
NS
SN
LEV TS
TS
S1
S2
X1
X2
Y1
---------------------------------------------------------------1
8
1
0
0
1
1
-3
1.1000 11.1000 -3.0000
1
8
2
0
1
2
1
3
1.1000 11.1000
5.5000
1
8
2
1
1
2
2
-3
2.2000 22.2000 -3.0000
1
8
3
0
2
3
2
3
2.2000 22.2000
5.5000
1
8
4
0
3
4
2
3
2.2000 22.2000
5.5000
1
8
4
1
3
4
-3
4 -3.0000 -3.0000
6.6000
1
8
5
0
4
5
2
4
2.2000 22.2000
6.6000
1
8
6
0
5
6
-3
4 -3.0000 -3.0000
6.6000
2
8
1
0
3
4
3
-3
3.3000 33.3000 -3.0000
2
8
2
0
4
5
3
-3
3.3000 33.3000 -3.0000
2
8
2
1
4
5
4
-3
4.4000 44.4000 -3.0000
2
8
3
0
5
6
3
-3
3.3000 33.3000 -3.0000
2
8
3
1
5
6
5
-3
5.5000 55.5000 -3.0000
2
8
4
0
6
8
5
-3
5.5000 55.5000 -3.0000
2
8
5
0
8
9
-3
-3 -3.0000 -3.0000 -3.0000
2
8
6
0
9
10
6
-3
6.6000 66.6000 -3.0000
3
3.4
sequence data
1
3.4 Sequence Data
This chapter introduces the concept of sequence data as used in TDA.
The subsections are as follows.
3.4.1
Sequence Data Concepts
3.4.2
Defining Sequence Data
3.4.3
Writing Sequence Data
3.4.4
Deriving Sequences from Episode Data
3.4.5
State Indicator Matrices
3.4.6
Random Sequences
d0304.tex
April 19, 1998
3.4.1
1
sequence data concepts
3.4.1 Sequence Data Concepts
Sequence data will be defined with respect to a nonnegative discrete time
axis T = {0, 1, 2, . . .} to stress that we are mainly interested in sequences
evolving in time. Of course, in some applications as, for instance, in the
analysis of DNA sequences, such an intrinsic relationship to time need
not be assumed. However, for most applications in the social sciences,
we use sequence data as a representation of event-history data which, by
definition, evolve in time.
We also assume a finite (discrete) state space, say Y, and use the
convention that nonnegative integers represent valid states and negative
integers represent non-valid (undefined or missing) states. Note that valid
states are restricted to the range 0 – 32000. Given a finite length of the
time axis, say T , an individual sequence can then be defined as
y0 , y 1 , y 2 , . . . , yT
where yt is a valid or an undefined state at time t.
In most applications, there is not just one single sequence but a sample of sequences. We refer to the sample members as individuals but, of
course, any other unit of analysis (which can be identified across time)
is possible. Indexing the sample members by i = 1, . . . , N , we then have
a sample of sequences
yi = (yi0 , yi1 , . . . , yiT )
i = 1, . . . , N
In the following, this will be called a sequence data structure. Finally, we
have to consider that, for many applications, there can be two or more
different sequences for each unit of analysis. We may have data showing
how individuals develop in two or more state spaces simultaneously. Or
each unit consists of two or more individuals, and we have one or more
sequences for each of these individuals. We have then to consider two
ore more sequence data structures simultaneously. Using k = 1, . . . , K
to index the sequence data structures, we have
yik = (yik,0 , yik,1 , . . . , yik,T )
i = 1, . . . , N, k = 1, . . . , K
yik denotes the kth sequence for the ith individual. This will be called a
K-dimensional sequence data structure. In addition, we use Yk to denote
the state space for the kth sequence data structure, and Y = Y1 ∪· · ·∪YK .
d030401.tex
April 19, 1998
3.4.1
sequence data concepts
2
As said above, we assume that sequences are defined on a nonnegative
discrete time axis (t = 0, 1, 2, . . .). This can be a calendar time axis or
a process time axis. In both cases, the time axis begins with t = 0. The
difference is that a calendar time axis does not have a natural origin and,
consequently, fixing an origin t = 0 is simply a convention for providing
time points. On the other hand, the origin of a process time axis has a
substantive meaning; t = 0 then represents the date of an event providing
the entry point of the process.
Example 1 To illustrate, assume we have some monthly measured
data on employment status, say 1 = employed, 0 = unemployed, observed for the five-year period 1990 – 1994. We can then define a calendar time axis using the convention 1 = January 1990, 2 = February
1990, and so on. We might have observed two individuals with sequences
1 1
-1 -1
1
1
0
1
0
0
1
0
1
0
1
1
1
0
...
...
In this example, the two sequences are represented on a calendar time
axis. The second individual is only observed beginning in March 1990, so
there are two month with missing states. Of course, there may be more
missing values throughout and at the end of the sequences.
Alternatively, to represent the data on a process time axis, one has
first to decide on an appropriate origin. In this example, this could be
the entry into the first (observed) unemployment spell. The sequences
for the two individuals would then look as follows:
0
0
0
0
1
0
1
1
1
0
1 ...
...
The definition of sequence data for TDA (see 3.4.2) is independent of
the distinction between calendar and process time. The distinction is
important, however, when applying statistical procedures.
3.4.2
defining sequence data
1
3.4.2 Defining Sequence Data
Sequences can be defined as a sequence of states or, alternatively, by a
sequence of events. TDA supports both methods. The basic command
is seqdef with syntax shown in Box 1. The command requires a list of
variables to be given on the right-hand side. All other parameters are
optional.
1. The sn parameter can be used to specify the number of the sequence
data structure: sn = k, where k = 1, . . . , 100. There can be up to 100
simultaneously defined sequence data structures, default is sn=1.
2. The rc parameter can be used for recoding the states of sequences.
This option will be explained below.
3. The m parameter can be used to specify the type of input data. There
are two possibilities. First, if m=1 (default), it is assumed that there is a
separate variable for each time point. Given that the list of variables on
the right-hand side is
seqdef(...) = Y0 , Y1 , . . . , YT ;
it is assumed that variable Yt contains the states for time point t. The
sequence data structure will consist of N sequences, one sequence for
each case (row) in the currently active data matrix, and each sequence
is defined for the time axis t = 0, 1, . . . , T . Alternatively, if m=2, it is
assumed that the sequence data structure is defined by a sequence of
events, specified by an even number of variables in the following way:
seqdef(...) = Y1 , T1 , . . . , Yq , Tq ;
It is assumed, then, that the ith sequence begins at T1i in state Y1i
and ends at Tqi in state Yqi . The time variables T1 , . . . , Tq must contain
integer values, and it is required that
T1i ≥ 0
and Tt,i > Tt−1,i (t = 2, . . . , q)
If the requirement is not met, TDA will print an error message. For
t < T1,i and t > Tq,i , states are undefined and get the missing value code
-1.
d030402.tex
April 19, 1998
3.4.2
defining sequence data
2
Box 1 Syntax for seqdef command
seqdef (
sn=...,
m=...,
rc=...,
number of sequence data structure, def. 1
type of input data, def. 1
option for recoding states
) = varlist;
The time axis. Associated with each sequence data structure is a time
axis. For the kth sequence data structure, the time axis begins at Tkmin
and ends at Tkmax . The definition depends on the sequence data type. If
m=1 and there are state variables Y0 , Y1 , . . . , YT , then
Tkmin = 0
and
Tkmax = T
If m=2 and the variables used to define the sequence data structure are
Y1 , T1 , . . . , Yq , Tq , then
Tkmin = min {T1i | i = 1, . . . , N } , Tkmax = max {Tqi | i = 1, . . . , N }
If there are two or more sequence data structures, the range of their
common time axis is defined by
T min = min {Tkmin }
k
and
T max = max {Tkmin }
k
If an individual sequence is undefined for one of the time points, it will
get the missing value code -1.
Removing sequence data structures. A sequence data structure
defined with the seqdef commands remains defined until one of the
following conditions occurs:
• A new data structure with the same number is defined by using the
seqdef command again;
• A variable required to maintain the data structure is deleted with
the clear command. Note that all currently defined sequences will be
deleted if at least one required variable is removed with clear.
• The sequence data structure is explicitly deleted with the seqdel
command. This command can be used in two different ways. Using the
command
seqdel;
3.4.2
defining sequence data
3
Box 2 Sequence data file seq.d1
ID Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7
--------------------------1
7 3 3 1 3 3 7 7
2
3 3 7 1 1 -1 -1 3
3 -1 3 3 -1 1 1 1 1
4
1 1 3 3 3 3 -1 -1
5
7 3 3 1 3 3 7 7
without any arguments deletes all currently defined sequence data structures. Alternatively,
seqdel = k;
only deletes the sequence data structure k. If this data structure does not
exist, the command is ignored. To get information about the currently
defined sequence data structures, one can use the command
seq;
without any arguments. This displays a short table containing some basic
information about all currently defined sequence data structures.
Example 1 To illustrate the specification of sequence data we use the
data file seq.d1 shown in Box 2. There are fives sequences, the common
sequence length is eight time points. Assuming that the variables are
available in TDA’s internal data matrix, one can use the command
seqdef = Y0,Y1,Y2,Y3,Y4,Y5,Y6,Y7;
or alternatively
seqdef = Y0,,Y7;
to define a sequence data structure with default number 1. Complete
command files are shown in Box 3.
Part of TDA’s standard output when using one of these command files
is shown in Box 4. For each sequence data structure, the output shows
its number, the type (1 = defined by state variables, 2 = defined by a
series of events), the number of (state) variables, the range of the time
axis, the number of different states, and finally a list of state numbers.
3.4.2
defining sequence data
Box 3 Command file seq1.cf using seq.d1
first version
--------------nvar(
dfile = seq.d1,
ID = c1,
Y0 = c2,
Y1 = c3,
Y2 = c4,
Y3 = c5,
Y4 = c6,
Y5 = c7,
Y6 = c8,
Y7 = c9,
);
seqdef = Y0,,Y7;
a shorter version
------------------nvar(
dfile = seq.d1,
ID = c1,
Y{0,7} = c2,
);
seqdef = Y0,,Y7;
Box 4 Part of standard output from command file seq1.cf
Creating a new sequence data structure.
Sequence structure number: 1
Sequence type: 1
Currently defined sequences:
Sequence
State
Time axis
Number
Structure Type Variables Minimum Maximum of States States
-------------------------------------------------------------1
1
8
0
7
3
1 3 7
Range of common time axis: 0 to 7.
Box 5 Sequence data file seq.d2
data file
--------------------ID Y1 T1 Y2 T2 Y3 T3
--------------------1
1 0 1 3 -1 4
2
1 0 2 5 2 6
3
2 0 2 6 1 7
4
1 0 2 6 -1 7
5
1 4 2 6 3 8
corresponding sequences
------------------------0 1 2 3 4 5 6 7 8
------------------------1 1 1 1 -1 -1 -1 -1 -1
1 1 1 1 1 2 2 -1 -1
2 2 2 2 2 2 2 1 -1
1 1 1 1 1 1 2 -1 -1
-1 -1 -1 -1 1 1 2 2 3
4
3.4.2
defining sequence data
5
Box 6 Part of standard output using seq.d2
Creating a new sequence data structure.
Sequence structure number: 1
Sequence type: 2
Currently defined sequences:
Sequence
State
Time axis
Number
Structure Type Variables Minimum Maximum of States States
-------------------------------------------------------------1
2
3
0
8
3
1 2 3
Range of common time axis: 0 to 8.
Box 7 Structure of a sequence data matrix
Sequence data Sequence data
Time-independent
structure 1
structure 2
...
covariates
----------------------------------------------------1 |
|
|
|
|
|
|
|
|
|
N |
|
|
|
|
-----------------------------------------------------
Example 2 To illustrate the definition of sequence data based on events,
we use the data file seq.d2 shown in Box 5. The box also shows the
corresponding sequences. Again, data are for five cases. There are six
variables representing three events. To use this data file for a definition
of sequence data, the command should be
seqdef(m=2) = Y1,T1,Y2,T2,Y3,T3;
(A complete command file is provided as seq2.cf.) Part of TDA’s standard output is shown in Box 6.
The data matrix. Sequence data are defined by using variables which
are available in TDA’s internal data matrix. A sequence data structure
simply consists of pointers to the internal data matrix; there is no separate copy of the data. In general, for each sequence data structure,
the number of sequences equals the number of cases (individuals) in the
data matrix. If one wishes to define two ore more sequences for each
individual, the whole information for each individual must be contained
in a single data matrix row. Box 7 illustrates the data matrix structure.
3.4.2
defining sequence data
6
Box 8 Sequence data file seq.d3
ID Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7
--------------------------1
7 3 3 1 3 3 7 7
2
3 3 7 1 1 -1 -1 3
3 -1 3 3 -1 1 1 1 1
4
1 1 3 3 3 3 -1 -1
5
7 3 3 1 3 3 7 7
S1 T1 S2 T2 S3 T3
----------------1 0 1 3 -1 4
1 0 2 5 2 6
2 0 2 6 1 7
1 0 2 6 -1 7
1 4 2 6 3 8
Box 9 Command file seq3.cf
nvar(
dfile = seq.d3,
ID = c1,
Y{0,7} = c2,
S1 = c10,
T1 = c11,
S2 = c12,
T2 = c13,
S3 = c14,
T3 = c15,
);
seqdef = Y0,,Y7;
# first sequence data structure
seqdef(
# second sequence data structure
sn = 2,
m = 2,
) = S1,T1,S2,T2,S3,T3;
Note that some procedures allow to use time-independent covariates in
addition to sequence data which, by their nature, are time-varying. The
time-independent covariates can simply be specified by referring to some
data matrix variables as indicated in Box 7.
Example 3 To illustrate the definition of two parallel sequence data
structures, we combine seq.d1 and seq.d2 into a single data file, seq.d3,
as shown in Box 8. Box 9 shows the command file, seq3.cf. The nvar
command creates the data matrix corresponding to the data file, seq.d3.
The first seqdef command uses variables Y0,...,Y7 to define the first
sequence data structure. This command uses the default parameter values sn=1 and m=1. Then follows a second seqdef command to define
a second sequence data structure (sn=2) based on a sequence of events
(m=2). TDA’s standard output from the two seqdef commands is shown
in Box 10.
3.4.2
defining sequence data
7
Box 10 Part of standard output from seq3.cf
> seqdef=Y0,,Y7
--------------------------------------Creating a new sequence data structure.
Sequence structure number: 1
Sequence type: 1
Currently defined sequences:
Sequence
State
Time axis
Number
Structure Type Variables Minimum Maximum of States States
-------------------------------------------------------------1
1
8
0
7
3
1 3 7
Range of common time axis: 0 to 7.
> seqdef(sn=2,m=2,)=S1,T1,S2,T2,S3,T3
--------------------------------------Creating a new sequence data structure.
Sequence structure number: 2
Sequence type: 2
Currently defined sequences:
Sequence
State
Time axis
Number
Structure Type Variables Minimum Maximum of States States
-------------------------------------------------------------1
1
8
0
7
3
1 3 7
2
2
3
0
8
3
1 2 3
Range of common time axis: 0 to 8.
Recoding the state space. By default, values of state variables are
used without modification. Nonnegative values represent valid states,
and negative values represent undefined states (missing values). The
range for valid states is 0 – 32000.
When defining a new sequence data structure with the seqdef command, the rc parameter can be used to recode the values of the state
variables. The syntax is
rc = k1 [j11 , j12 , . . .], k2 [j21 , j22 , . . .], . . .,
Then, if one of the state variables has one of the values j11 , j12 , . . ., it
gets the new value k1 , if it has one of the values j21 , j22 , . . ., it gets the
new value k2 , and so on. Some of the integers in square brackets can be
negative to allow a recoding of missing values. It is required, however,
3.4.2
defining sequence data
8
Box 11 Command file seq4.cf
nvar(
dfile = seq.d1,
ID = c1,
Y{0,7} = c2,
);
seqdef(
rc = 9[-1],5[7,3],
)
= Y0,,Y7;
seqpd = d;
Box 12 Part of output from command file seq4.cf
Creating a new sequence data structure.
Sequence structure number: 1
Sequence type: 1
Currently defined sequences:
Sequence
State
Time axis
Number
Structure Type Variables Minimum Maximum of States States
------------------------------------------------------------1
1
8
0
7
3 1 5[3,7] 9[-1]
Range of common time axis: 0 to 7.
Output file d
---------------------------ID Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7
1 5 5 5 1 5 5 5 5
2 5 5 5 1 1 9 9 5
3 9 5 5 9 1 1 1 1
4 1 1 5 5 5 5 9 9
5 5 5 5 1 5 5 5 5
for all values j used inside the square brackets, that
−9 ≤ j ≤ smax
where smax denotes the highest state number in the original sequence
data. It is possible, therefore, only to recode missing values in the range
−1, . . . , −9.
Example 4 To illustrate the recode option, we use data file seq.d1
(Box 2) and command file seq4.cf, shown in Box 11. As defined with
the rc parameter, state -1 is changed into 9, and states 3 and 7 are
3.4.2
defining sequence data
9
changed into 5. Part of the standard output, and the data file d created
with the seqpd command (see section 3.4.3, is shown in Box 12.
Using episode data. Although episode data and sequence data have
a quite different data structure, the seqdef command allows to directly
use single episode data for defining sequence data. Assume there are
variables ORG, DES, TS, and TF, which can be used to define single episode
data with the edef command (see 3.3.2). The same variables can then
be used to define sequence data with the command
seqdef (m=2) = ORG,TS,DES,TF;
Of course, this is no longer possible if the data matrix contains episode
splits or multi-episode data. In general, working with sequence data requires that all information about an individual’s event history is contained in a single row of the data matrix. However, a command to transform multi-episode data into sequence data will be described in 3.4.4.
3.4.3
1
writing sequence data
3.4.3 Writing Sequence Data
Given a specification of sequence data, they can be written into an output
file. The command is seqpd with syntax shown in Box 1. Except for the
output file on the right-hand side, all parameters are optional.
1. The m parameter can be used to select a structure for the output file.
There are four options.
1. If m=1 (default), the output file will contain a single record for each
case in the data matrix, and the sequences will be concatenated horizontally. The first entry in each record is the case number. See the
first example in Box 2.
2. If m=2, the output file contains a separate record for each sequence
data structure. The first entry in each record will contain the case
number, the second entry provides the sequence number. See the
second example in Box 2.
3. If m=3, the output file contains a separate record for each case and
time point. The first entry in each record gives the case number, the
second entry contains the time point. Then follow the states for all
currently defined sequences. This is equivalent to an episode data
file where episodes are split on the basis of time points. See the first
example in Box 3.
4. If m=4, the output file will be an episode data file based on the first
currently defined sequence data structure. See the second example in
Box 3.
2. The sel parameter can be used to select cases. The syntax is
sel = expression,
Then, only data matrix cases where the result of evaluating expression
is not equal to zero will be used for the output data file.
3. The v parameter with syntax v=varlist can be used to write variables into the output file. Values of the variables specified in varlist
will be written at the end of each output file record. The print format
for these variables will be the same as used in their definition. The print
d030403.tex
April 19, 1998
3.4.3
writing sequence data
2
Box 1 Syntax for seqpd command
seqpd (
m=...,
sel=...,
v=...,
dtda=...,
structure of output file, def. 1
expression for case selection
add variables ... to output file
request TDA description file
) = fname;
Box 2 Illustration of seqpd command
Command: seqpd (m=1) = d.1;
first sequence
second sequence
ID 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8
-- -------------------------- ------------------------1 7 3 3 1 3 3 7 7 -1 1 1 1 1 -1 -1 -1 -1 -1
2 3 3 7 1 1 -1 -1 3 -1 1 1 1 1 1 2 2 -1 -1
3 -1 3 3 -1 1 1 1 1 -1 2 2 2 2 2 2 2 1 -1
4 1 1 3 3 3 3 -1 -1 -1 1 1 1 1 1 1 2 -1 -1
5 7 3 3 1 3 3 7 7 -1 -1 -1 -1 -1 1 1 2 2 3
Command: seqpd (m=2) = d.2;
ID SN 0 1 2 3 4 5 6 7 8
-- -- -------------------------1 1 7 3 3 1 3 3 7 7 -1
1 2 1 1 1 1 -1 -1 -1 -1 -1
2 1 3 3 7 1 1 -1 -1 3 -1
2 2 1 1 1 1 1 2 2 -1 -1
3 1 -1 3 3 -1 1 1 1 1 -1
3 2 2 2 2 2 2 2 2 1 -1
4 1 1 1 3 3 3 3 -1 -1 -1
4 2 1 1 1 1 1 1 2 -1 -1
5 1 7 3 3 1 3 3 7 7 -1
5 2 -1 -1 -1 -1 1 1 2 2 3
format for the case number is always 6.0; time points are written in a
4.0 format; and the print format for state variables is 2.0.
4. The dtda parameter with syntax
dtda = name of an output file,
can be used to request an additional output file that will contain a TDA
3.4.3
writing sequence data
Box 3 Illustration of seqpd command
seqpd(m=3) = d.3
---------------ID
T S1 S2
------------1
0 7 1
1
1 3 1
1
2 3 1
1
3 1 1
1
4 3 -1
1
5 3 -1
1
6 7 -1
1
7 7 -1
1
8 -1 -1
2
0 3 1
2
1 3 1
2
2 7 1
2
3 1 1
2
4 1 1
2
5 -1 2
2
6 -1 2
2
7 3 -1
2
8 -1 -1
3
0 -1 2
3
1 3 2
3
2 3 2
3
3 -1 2
3
4 1 2
3
5 1 2
3
6 1 2
3
7 1 1
3
8 -1 -1
4
0 1 1
....
4
5 3 1
4
6 -1 2
4
7 -1 -1
4
8 -1 -1
5
0 7 -1
5
1 3 -1
5
2 3 -1
5
3 1 -1
5
4 3 1
5
5 3 1
5
6 7 2
5
7 7 2
5
8 -1 3
seqpd(m=4) = d.4
--------------------ID SN ORG DES TS
TF
--------------------1 1 7 3
0
1
1 2 3 1
1
3
1 3 1 3
3
4
1 4 3 7
4
6
1 5 7 -1
6
8
2 1 3 7
0
2
2 2 7 1
2
3
2 3 1 -1
3
5
2 4 -1 3
5
7
2 5 3 -1
7
8
3 1 -1 3
0
1
3 2 3 -1
1
3
3 3 -1 1
3
4
3 4 1 -1
4
8
4 1 1 3
0
2
4 2 3 -1
2
6
4 3 -1 -1
6
8
5 1 7 3
0
1
5 2 3 1
1
3
5 3 1 3
3
4
5 4 3 7
4
6
5 5 7 -1
6
8
3
3.4.3
writing sequence data
4
description of the data file created with the seqpd command. This additional output file can then be used as a command file to read the data
file.
Example 1 To illustrate the seqpd command, we use the data file
seq.d3 (see section 3.4.2). The command file seq5.cf (not shown) is
basically identical with seq3.cf (see 3.4.2). We have added four seqpd
commands, corresponding to the options m=1,2,3,4. The output files
created by these print commands are shown in boxes 2 and 3.
3.4.4
deriving sequences from episode data
1
3.4.4 Deriving Sequences from Episode Data
As mentioned in 3.4.2, single episode data can be used directly to define
sequence data. In general, if given an arbitrary multi-episode data file,
this is not possible and one needs a procedure to transform episode into
sequence data. One such procedure is provided by the seqpe command.
The syntax is shown in the following box. All parameters, except v, are
required.
seqpe (
id=...,
org=...,
des=...,
ts=...,
tf=...,
tp=...,
v=...,
ID variable
variable for origin state
variable for destination state
variable for starting time
variable for ending time
definition of time points
add variables ... to output file
) = fname;
1. On the right-hand side of the command, one has to give the name of
an output file.
2. The parameters id=, org=, des=, ts=, and tf= must be used to provide variables containing the ID number, origin state, destination state,
starting time and ending time, respectively. These variables define the
multi-episode data.
3. The tp parameter must be used to provide a sequence of time points.
The syntax is
tp = t1 , t2 , . . . , tn
or
tp = t1 (d) t2
or a mixture of both expressions. Assuming d > 0, the second expression
is expanded into the sequence t1 + id (i = 0, 1, 2, . . .), as long as the
result is less than or equal to t2 .
The output file created by the seqpe command contains one record
for each individual (identified by the ID variable). Each record begins
d030404.tex
April 19, 1998
3.4.4
deriving sequences from episode data
2
Box 1 Command file seq6.cf
nvar(
dfile = d.4, #
noc = 22,
CASE <5>[6.0] =
SN
<2>[2.0] =
ORG <5>[2.0] =
DES <5>[2.0] =
TS
<5>[4.0] =
TF
<5>[4.0] =
);
seqpe(
id = CASE,
org = ORG,
des = DES,
ts = TS,
tf = TF,
tp = 0(1)7,
) = d;
data file created with seq5.cf
c1
c2
c3
c4
c5
c6
,
,
,
,
,
,
#
#
#
#
#
#
case number
spell number
origin state
destination state
starting time
ending time
# create sequence data
Box 2 Output file (d) create by command file seq6.cf
ID 0 1 2 3 4 5 6 7
-- ----------------------1 7 3 3 1 3 3 7 7
2 3 3 7 1 1 -1 -1 3
3 -1 3 3 -1 1 1 1 1
4 1 1 3 3 3 3 -1 -1
5 7 3 3 1 3 3 7 7
with a case number (always written in 6.0 format). Then follows a sequence of states, the length being equal to the number of time points
defined with the tp parameter. For each time point t, the algorithm
checks whether t falls into one of the episodes, that is, TS ≤ t < TF. If
successful, the current state of the individual is taken from the variable
specified with the org parameter, otherwise it is a missing value code
(-1). States are written in a 2.0 print format.
4. The v parameter can be used to specify a list of variables, printed at
the end of each record using their associated print formats. Note that
the values of these variables are taken from the first record for each set
of episodes belonging to the same individual.
Example 1 To illustrate the seqpe command, we use the example
3.4.4
deriving sequences from episode data
3
episode data file d.4, shown in Box 3.4.3-3. Remember that this file was
created from the first part of sequence data file seq.d3. So we can use
the seqpe command to transform this episode data file back into the
original sequence data file (actually seq.d1, see 3.4.2). The command
file is seq6.cf (Box 1). The nvar command reads the variables from the
data file d.4. The seqpe command creates the output file d containing
the sequences. It is shown in Box 2 and identical with the sequence data
file seq.d1 (see 3.4.2).
3.4.5
state indicator matrices
1
3.4.5 State Indicator Matrices
It might happen that one wants to reorganize a sequence data structure
into what has been called a state indicator matrix, for instance, to apply
the method of correspondence analysis or some other projection method
(see the discussion in Heijden [1987]). The idea is to represent the state of
an individual at a specific time point t by a series of q dummy variables,
where q is the number of different states. Then, if the individual is in
state j at this point in time, the jth dummy variable gets the value one.
To create state indicator matrices, TDA provides the command seqsi
with syntax shown in Box 1. The command uses the first of the currently
defined sequence data structures to create a state indicator matrix and
then writes this matrix into the output file specified on the right-hand
side of the command. The tp parameter must be used to provide a
sequence of time points (for a description of the syntax, see 6.5.1); all
other parameters are optional.
1. The m parameter can be used to select sequences. If m=1 (default), the
command uses all sequences. If m=2, the command uses only sequences
having a valid state for all time points defined with the tp parameter.
2. The v parameter can be used to specify a list of variables. These
variables will be added to the records written into the output file.
3. The dtda parameter can be used to request a second output file
containing an nvar command to read the data file created with the
seqsi command.
The structure of the output file records is as follows:
• The first entry is the case number, written in 6.0 format.
• Then follow j = 1, . . . , q entries for the first time point; q is the
number of different states in the sequence data structure. Entry j is
1 if the sequence is in the jth state at this point in time, otherwise
zero. (Note that the states of a sequence are sorted in ascending order
and then mapped to internal state numbers: 0,1,2,. . . ; the q entries
for each point in time correspond to these internal state numbers.)
• For each additional time point specified with the tp parameter there
will be again q entries created in an analogous way.
d030405.tex
April 19, 1998
3.4.5
state indicator matrices
2
Box 1 Syntax for seqsi command
seqsi (
tp=...,
m=...,
v=...,
dtda=...,
definition of time points
option for case selection, def. 1
add variables ... to output file
request TDA description file
) = fname;
Box 2 Illustration of seqsi command, based on seq.d1
Command: seqsi(m=1,t=1(1)5,v=ID) = si.d1;
Output file: si.d1
Case
---1
2
3
4
5
1 3 7
----t=1
----0 1 0
0 1 0
0 1 0
1 0 0
0 1 0
1 3 7
----t=2
----0 1 0
0 0 1
0 1 0
0 1 0
0 1 0
1 3 7
----t=3
----1 0 0
1 0 0
0 0 0
0 1 0
1 0 0
1 3 7
----t=4
----0 1 0
1 0 0
1 0 0
0 1 0
0 1 0
1 3 7
----t=5 ID
----- -0 1 0 1
0 0 0 2
1 0 0 3
0 1 0 4
0 1 0 5
Command: seqsi(m=2,t=1(1)5,v=ID) = si.d2;
Output file: si.d2
Case
---1
4
5
1 3 7
----t=1
----0 1 0
1 0 0
0 1 0
1 3 7
----t=2
----0 1 0
0 1 0
0 1 0
1 3 7
----t=3
----1 0 0
0 1 0
1 0 0
1 3 7
----t=4
----0 1 0
0 1 0
0 1 0
1 3 7
----t=5 ID
----- -0 1 0 1
0 1 0 4
0 1 0 5
• Finally, any variables specified with the v parameter are written at
the end of the current record.
Example 1 To illustrate the seqsi command, we use the data file
seq.d1 (see 3.4.2) with q = 3 states. Box 2 shows the seqsi commands
(contained in command file seq7.cf) and the resulting output files, for
a time axis 1, . . . , 5 and for both options. With m=1, the output file
contains five records; with m=2 there are only three records containing
3.4.5
state indicator matrices
3
those sequences which have valid states for all time points. Note the
mapping of states to the columns of the output file.
3.4.6
random sequences
1
3.4.6 Random Sequences
When experimenting with sequence data, it is sometimes convenient to
use random sequences, that is, sequences filled randomly with state numbers. To create such random sequences, TDA provides the command
seqrd with syntax shown in the following box.
seqrd (
ns=...,
len=...,
noc=...,
number of states, def. 2
length of sequences, def. 1
number of sequences, def. 1000
) = fname;
The right-hand side must provide the name of an output file. The parameters can be used to specify the number of different states (ns), the
number of sequences (noc), and the length of the sequences (len). To
create the random states, TDA uses the formula
floor (r · ns) + 1
where ns is the number of different states, and r is a random number
which is equally distributed in the (0, 1) interval. For example, the command
seqrd (ns=3,noc=100,len=10) = rs.dat;
would generate an output file, rs.dat, containing 100 sequences each
consisting of 10 states randomly selected from {1, 2, 3}.
d030406.tex
April 19, 1998
3.6
graphs and relations
1
3.6 Graphs and Relations
This chapter explains data structures for graphs and relations. It contains the following sections.
3.6.1
Terminology
3.6.2
Graph Data Structures
3.6.3
Creating Relational Data
3.6.4
Relational Data Files
3.6.5
Plotting Graphs
3.6.6
Trees
d0306.tex
December 30, 1998
3.6.1
terminology
1
3.6.1 Terminology
A simple graph consists of a set of nodes, N , and a set of edges, E. It
will be assumed that the node set is a finite set of positive integers. Two
nodes i, j ∈ E are said to be connected if there is an edge (i, j) ∈ E.
The edge set may contain edges (i, i), for i ∈ N ; these edges will be
called loops. (In our terminology, we allow simple graphs to have loops.
The main distinction is then between simple graphs and multigraphs,
see below. If it is required that a simple graph does not contain loops
this will be explicitly mentioned.)
In an unvalued graph we only consider whether there is an edge
connecting two nodes. Alternatively, each edge may have an associated
value which will then be denoted by v(i, j). We shall adopt the convention that only non-negative real numbers are valid values. Consequently,
if v(i, j) ≥ 0, there is an edge connecting nodes i and j having the value
v(i, j). On the other hand, if v(i, j) < 0, this will mean that there is
no edge connecting i and j. (If procedures require valid edges to have
positive values this will be explicitly mentioned when describing these
procedures.) To represent unvalued and valued graphs we use the notation (N , E) and (N , E, v), respectively.
A graph is called undirected if we do not distinguish between the
edges (i, j) and (j, i). In a valued graph this will imply that v(i, j) =
v(j, i). We then simply speak of a connection between nodes i and j.
Alternatively, we may also consider directed graphs where we distinguish
between (i, j), an edge from i to j, and (j, i), an edge from j to i.
Figure 1 illustrates a simple undirected and unvalued graph consisting of four nodes. There are two edges connecting, respectively, nodes 1
and 2, and 1 and 3. Node 4 is not connected to any other node and is
called an isolated node.
In a simple graph there is only one kind of edges represented by a
single edge set E. More generally, we can consider several kind of edges
simultaneously. This will be called a multigraph. A multigraph consisting
of m simple unvalued graphs will be denoted by
(N , E1 , E2 , . . . , Em )
Correspondingly, the notation
(N , E1 , E2 , . . . , Em , v1 , v2 , . . . , vm )
d030601.tex
April 19, 1998
3.6.1
terminology
1
2
3
4
2
Figure 1 Simple undirected and unvalued graph
consisting of four nodes and two edges.
will be used to represent a multigraph consisting of m valued simple
graphs.
A multigraph allows different interpretations. One interpretation assumes that each edge set represents a different kind of connection between the nodes but that all connections hold simultaneously. Another
interpretation views a multigraph as a temporal sequence of simple
graphs.
There is a close connection between graphs and relations. Consider
a binary relation
r : N × N −→ R
defined on a node set N . We can interpret this relation as representing
a graph by using the convention
(i, j) ∈ E ⇐⇒ r(i, j) ≥ 0
This will suffice for an unvalued graph. Of course, we can also think
of r(i, j) as representing the value of the edge (i, j). The graph will be
undirected if the relation is symmetrical.
3.6.2
graph data structures
1
3.6.2 Graph Data Structures
Almost all TDA commands that use relational data (graphs) require a
previous definition of a relational data structure with the gdd command.
The syntax of this command is shown in Box 1.1
1. The command requires a list of variables containing information about
node numbers and edges. These variables must be available in a previously defined TDA data matrix (defined with the nvar command). What
kind of variables are required depends on the option selected with the
opt parameter. Each of these options will be explained below in turn.
2. Node numbers can be any positive integers. When creating a gdd
data structure, TDA maps the external node numbers, as provided by
the user, to internal node numbers being a set of contiguous integer
numbers, 1, 2, . . . , N , where N is the number of nodes of the graph.
3. As will be explained below, the gdd command can be used to define
multigraphs consisting of several single graphs. Most TDA commands
that use relational data provide an optional parameter, gn (graph number ), to select a single graph from the currently defined multigraph. By
default, gn = 1, that is, the command uses the first graph.
4. The user can specify a graph type with the gt parameter. The options
are as follows:
1. gt=1 specifies an undirected unvalued graph: an edge from i to j
is assumed to exist if the input data contain an edge from i to j
or an edge from j to i.
2. gt=2 specifies an undirected valued graph. To make this interpretation unique in cases where the input data contain both, an edge
(i, j) and an edge (j, i), there are three options: (a) If gt(1)=2, the
value of the undirected edge (i, j) is the minimum of v(i, j) and
v(j, i). This is the default interpretation and equal to gt = 2. (b)
If gt(2)=2, the value of the undirected edge (i, j) is the maximum
1 If the gdd command is used without any arguments it provides information about
the currently defined relational data (if any).
d030602.tex
May 29, 2002
3.6.2
graph data structures
2
Box 1 Syntax of gdd command
gdd (
opt=...,
gt=...,
perm=...,
option, def. 1
1 = edge list
2 = pointer to adjacency matrix
3 = lower triangle
4 = upper triangle
5 = lower triangle, including diagonal
6 = upper triangle, including diagonal
7 = full adjacency matrix
graph type
1 = undirected, unvalued
2 = undirected, valued
3 = directed, unvalued
4 = directed, valued
if perm=1 the gdd data structure is
permanently saved, def. perm=0
) = varlist;
of v(i, j) and v(j, i). (c) If gt(3)=2, the value of the undirected
edge (i, j) is the sum of v(i, j) and v(j, i).
3. gt=3 specifies a directed unvalued graph.
4. gt=1 specifies a directed valued graph.
If the gdd data structure consists of a multigraph all of its single graphs
get an identical graph type as specified with the g t parameter.
5. A relational data structure defined with the gdd command remains
available for subsequent commands until one of the following conditions
occurs:
1. The data structure is explicitly removed with
gdd = off;
2. The gdd command is used again to define a new gdd data structure.
3. At least one of the variables used in the currently defined gdd data
structure is no longer available (only if perm=0).
3.6.2
graph data structures
3
4. A new case selection is defined with a tsel command (only if
perm=0).
6. A standard relational data structure defined with the gdd command
uses the data in TDA’s internal data matrix and therefore does not need
to store the data again. This saves memory and suffices for most applications. A problem only occurs if one wants to combine a relational data
structure with additional variables which characterize its nodes. The best
approach would then be to use an additional data file that contains the
information (variables) characterizing the nodes of the graph. However,
one would then need to set up a new data matrix and this in turn would
destroy the previously defined relational data structure.
TDA therefore allows to save a relational data structure permanently,
independent of the current data matrix. This can be requested with
the perm=1 parameter. The gdd command then creates a copy of the
data that are necessary to maintain the relational data structure. Consequently, a relational data structure defined with the perm=1 option
remains defined also if the current data matrix is modified, or substituted by another data matrix. The data structure will only be destroyed
if this is explicitly requested with a gdd=off command, or if the gdd
command is used again to define a new relational data structure.
Note, however, that the perm parameter can only be used with option
1 (edge list). Otherwise, the command will print a warnings message and
the data structure will not be permanently saved.
Option 1: Edge list. For sparse graphs, the most efficient form of
data storage is an edge list. If the input data have this form, one can use
option 1 with syntax
gdd = I, J, V1 , V2 , . . . , Vm ;
where the right-hand side specifies a list of variable names that must be
available in the current data matrix. The variables I and J must contain
positive integers and will be used to construct the node set of the graph.
The node set will consist of all values found in the variables I or J.
Each of the following variables, V1 , . . . , Vm specifies the edges of one
graph. Altogether, the command allows to specify a multigraph consisting of m simple graphs. There is no specific limit for the number of
graphs, m. The interpretation is as follows. If Vk is one of the variables
on the right-hand side and Vki denotes its ith value, then
3.6.2
graph data structures
1
7
10
5
0
4
3
13
4
9
Figure 1 Directed valued graph consisting of five
nodes and four edges, see example 1. The plot was
created with command file gd2.cf.
1. if Vki ≥ 0 there will an edge connecting nodes Ii and Ji having the
(optional) value Vki , and
2. if Vki < 0 there will be no edge connecting nodes Ii and Ji .
Note that this conventions allows to specify isolated nodes by giving an
edge a negative value. By default, when using option 1, the graph type
is 4, i.e., the graph is interpreted as directed and valued.
Example 1 To illustrate, assume we are given the following data:
I
J
V
1
3
1
4
9
7
7
4
7
9
10
0
5
13
−1
The node set will be N = {1, 3, 4, 7, 9}. There are four edges, node 9
will be an isolated node. In this example, all four interpretations that
can be selected with the gt parameter are possible. Figure 1 shows the
resulting graph if interpreted as directed and valued (gt=4).
Example 2 To illustrate a multigraph we use the data in Box 2. The
command
gdd = I,J,V1,V2;
will create edge lists for both simple graphs. Figure 2 (created with
command file gd3.cf) provides a graphical illustration.
3.6.2
5
graph data structures
Box 2 Data file gd1.dat
I J V1 V2
-------------1 5
3 -1
1 7
5 -1
7 1
6 15
11 12 13 -1
9 9 -1 -1
5 1 -1
4
8 7 -1
0
12 11 -1 14
8 8
2 -1
6
1
3
5
5
7
0
8
2
15
4
14
11
13
12
9
Figure 2 Multigraph consisting of two simple graphs,
defined by the edge list shown in Box 2.
Option 2: Pointer to adjacency matrix. If the graph is relatively
dense it can be more efficient, and economical, to use pointers to its
adjacency matrix. The syntax is again
gdd = I, J, V1 , V2 , . . . , Vm ;
and is interpreted in the same way as already explained for option 1. The
difference is only in the internal data interpretation. If opt = 2, TDA
uses a single n × n array of pointers to the edge values of the graphs
that are contained in the multigraph, n being the maximal number of
nodes in the graphs. Of course, there are two drawbacks. If n is large, it
might not be possible to store the matrix of pointers with the available
memory. Also, some algorithms (in particular, those using some version
of a depth-first search) will work less efficient compared with an edge-list
data structure. As with option 1, the default graph type interpretation
for option 2 is gt=4 (directed and valued).
3.6.2
graph data structures
6
Option 3: Lower triangle. Option 3 assumes that the lower triangle
of the adjacency matrix is available for input. The syntax is
gdd = V1 , V2 , . . . , Vm ;
It is assumed, then, that each variable Vk , contains the values of the
lower triangle of the adjacency matrix for the corresponding graph. To
explain the mapping, let Ak = (ak,ij ) denote the adjacency matrix of
the graph corresponding to Vk . Then, for i > j, it is assumed that
ak,ij = Vk,l
where
l=
(i − 1)(i − 2)
+j
2
for l = 1, . . . , NOC, where NOC is the number of cases in the current data
matrix. Note that graphs defined with option 3 are always undirected.
By default, gt=2, meaning that the graph is interpreted as undirected
and valued.
Option 4: Upper triangle. The syntax for option 4 is the same as
explained for option 3. The only difference is that the values of Vk are
interpreted as entries of the upper triangle of the kth adjacency matrix.
Assuming that the graph contains n nodes, and j > i, the following
mapping is used:
ak,ij = Vk,l
where
l = (i − 1) n −
i(i − 1)
+j−i
2
Again, by default, gt=2, that is, the graphs are interpreted as undirected
and valued.
Option 5: Lower triangle, including diagonal. The option assumes
that Vk contains values for the lower triangle of an adjacency matrix, but
includes values of the main diagonal. The syntax is the same as explained
for option 3. Assuming that j ≤ i, the mapping is:
ak,ij = Vk,l
where
l=
i(i − 1)
+j
2
By default, gt=2, that is, the graphs are interpreted as undirected and
valued.
Option 6: Upper triangle, including diagonal. The option assumes
that Vk contains values for the upper triangle of an adjacency matrix, but
3.6.2
graph data structures
7
includes values of the main diagonal. The syntax is the same as explained
for option 3. If the number of nodes is n, and j ≥ i, the mapping is:
ak,ij = Vk,l
where
l = (i − 1)n −
(i − 1)(i − 2)
+j−i+1
2
By default, gt=2, that is, the graphs are interpreted as undirected and
valued.
Option 7: Complete adjacency matrix. Finally, if opt = 7, it is
assumed that Vk contains n2 values for the complete adjacency matrix,
assuming a graph with n nodes. The mapping is
ak,ij = Vk,l
where
l = (i − 1)n + j
Contrary to options 3 – 6, this option can be used to specify directed
graphs. By default, gt=4, that is, the graphs are interpreted as directed
and valued.
Remark 1 Options 3 – 7 are mainly provided for easy input of proximity data which are most often available as (symmetric) adjacency matrices. No additional data structure will be created if one of these options
is selected. The data are directly retrieved from the internal data matrix
using the respective mapping described above.
Remark 2 Relational data given by an edge list can be transformed
into different versions of an adjacency matrix, and vice versa, by using
the gdp command described in Section 3.6.4.1.
3.6.3
creating relational data
1
3.6.3 Creating Relational Data
This section is intended to describe commands to create relational data
having some specified properties. For the moment, there is only a single
subsection describing the gcd command.
3.6.3.1
Simple Test Data describes the gcd command that can be
used to create graphs with specified numbers of nodes and
edges.
d030603.tex
December 12, 1998
3.6.3.1
1
simple test data
3.6.3.1 Simple Test Data
The gcd command can be used to create data files containing unvalued
graphs with a specified number of nodes and edges. The syntax is shown
in the following box.
gcd (
opt=...,
n=...,
m=...,
nfmt=...,
) = fname;
option, def. 1
1 = complete graph, without loops
2 = complete graph, including loops
3 = randomly select m edges
number of nodes, def. 10
number of edges, def. 1
integer print format, def. 4
Except for the name of an output file to be given on the right-hand side,
all parameters are optional. The data are written to the output file in
form of an edge list. Each record contains values of three variables, I,
J, and V . The first two variables contain the node numbers, the third
variable, V , has value 1 or -1. If Vi = 1, there is an edge from Ii to Ji ;
otherwise there is no edge.
Option 1. creates a complete graph having n nodes as specified with
the n parameter. The graph will not contain loops.
Option 2. creates a complete graph having n nodes and includes loops.
Option 3. creates a graph having n nodes and m edges, randomly
selected from the set of all edges.
d03060301.tex
December 12, 1998
3.6.4
relational data files
1
3.6.4 Relational Data Files
This section describes commands intended to support working with relational data. There are two subsections.
3.6.4.1
Writing Relational Data Files describes the gdp command that
can be used to write the data from a relational data structure
into an output file.
3.6.4.2
Creating Adjacency Matrices describes the mdefg command
that can be used to create a TDA matrix that contains the
adjacency matrix of a graph.
d030604.tex
December 30, 1998
3.6.4.1
writing relational data files
1
3.6.4.1 Writing Relational Data Files
Given a relational data structure, all or part of the data can be written into an output file. The command is gdp with syntax shown in the
following box.
gdp (
opt=...,
gn=...,
sc=...,
nfmt=...,
fmt=...,
if=...,
) = fname;
option, def. 1
1 = edge list
2 = lower triangle of adjacency matrix
3 = same as 2 but including the diagonal
4 = complete adjacency matrix
5 = adjacency matrix written as a square
matrix
graph number (for options 5 and 6), def. 1
substitute for missing edges, def. -1
integer print format, def. 4
print format for values, def. 10.4
input file containing node numbers
Except for the name of an output file on the right-hand side, all parameters are optional. One out of six options can be selected to specify
the structure of the output file. By default, for options 1 – 4, the gdp
command writes the edge values for all graphs contained in the currently
defined multigraph. Optionally, one can use the gn parameter to select a
specific graph number. Option 5 writes the adjacency matrix as a square
matrix, plus two leading columns that contain, respectively, the internal
and external node numbers. By default, this option uses the first graph
number if not otherwise specified with the gn parameter.
By default, when writing data for a single graph, missing edges are
omitted. Of course, they need to be included when writing multigraph
data. By default, then, missing edges get the value -1. Alternative values
can be specified with the sc parameter.
As a further option one can specify the name of an input file with the
if parameter. The gdp command tries to interpret the first numerical
d03060401.tex
February 10, 1999
3.6.4.1
writing relational data files
2
entry in each record as a node number. The command then only writes
data for the subgraph induced by the node numbers found in the input
file.
One should note that the contents of the output file will depend on
the graph type specified in the gdd command when defining the relational
data structure.
3.6.4.2
creating adjacency matrices
1
3.6.4.2 Creating Adjacency Matrices
Given a relational data structure, one can use the mdefg command to
create a TDA matrix containing a graph’s adjacency matrix. The syntax
is shown in the following box.
mdefg (
gn=...,
sc=...,
graph number, def. 1
substitute for missing edges, def. -1
) = matrix name;
Except for the name of a matrix to be given on the right-hand side all
parameters are optional. By default, the command uses the first graph
from the current relational data structure. The sc parameter can be used
to substitute specific values for missing edges.
d03060402.tex
February 10, 1999
3.6.5
drawing graphs
1
3.6.5 Drawing Graphs
There is no easy way to plot arbitrary graphs. TDA therefore does not try
to offer a command that automatically plots an arbitrary graph. Instead,
there is a command, plg, that partially supports plots of graphs based
on TDA’s environment for creating PostScript plots.
The command requires that the user has defined a PostScript output
file with the psfile command and set up a coordinate system with
the psetup command (see 4.1). The plg command then allows to place
nodes and edges into this coordinate system. For each node one needs a
separate node parameter, for each edge a separate edge parameter. The
maximum number of nodes can be specified with the nmax parameter,
default is 100 nodes. There is no limit for the number of edges.
1. Nodes must be specified as
node(...) = x,y,
where x and y give the coordinates of the node’s center point. All other
sub-parameters are optional as explained in Box 1. Here are some additional explanations.
1. Without additional parameters, the node parameter plots a circle
at the (x, y) position. The diameter can be specified with the rd
parameter, default is 4 mm.
2. The gt parameter can be used to select a different display of nodes.
gt=2 selects a square with diameter defined by rd. gt=4 selects a
rectangle. Height and widths can be specified with rd=h,w. Default
is h = 4 and w = 8 (mm). gt=3 also displays a rectangle but adds
two semi-circles.
3. Each node can be given a node number with the n parameter. These
node numbers are used when an edge parameter requires to connect
two nodes. Note that there are no default node numbers.
4. Whether a label is written into the node’s display depends on the
fs, str, and n parameters. If the font size, fs, is zero, nothing is
displayed. Otherwise, if some string is specified with the str parameter, this string is written. (Note that a string must be put in double
d030605.tex
December 18, 1999
3.6.5
drawing graphs
2
Box 1 Syntax of plg command
plg (
nmax=...,
node(
n=...,
gt=...,
str=...,
gt=...,
rd=...,
lt=...,
lw=...,
gs=...,
fs=...,
) = x,y,
edge(
ic=...,
fs=...,
lt=...,
lw=...,
a=...,
rd=...,
dir=...,
) = i,j,
maximal number of nodes, def. 100
place node at (x,y)
external node number, must be integer
type of display, def. 1
1 = circle
2 = square
3 = rectangle plus semi-circles
4 = simple rectangle
optional text if not node number
type of display, def. 1
size of display in mm, def. 4 mm
line type, def. 1
line width, def. 0.2 mm
gray scale value, def. 1 (white)
font size
connect nodes i and j
optional edge value
font size for edge value
line type, def. 1
line width, def. 0.2 mm
size of optional arrow head
radius for arcs, in mm
location of loop
);
quotation marks to preserve blank characters.) If the str parameter
is not used, but a node number is specified with the n parameter,
this node number is written as an integer.
5. The node’s display may be grey-scaled with the gs parameter. Possible values are in the range from 0 (black) to 1 (white).
2. Edges must be specified as
edge(...) = i,j,
3.6.5
drawing graphs
3
Box 2 Command file for drawing a graph
psfile = gd3.ps;
psetup(
pxa = 0,9,
pya = 0,4,
pxlen = 80,
pylen = 40,
);
plg(
node(n= 1,gs=0.8)
node(n= 7,gs=0.8)
node(n= 8,gs=0.8)
node(n= 5,gs=0.8)
node(n=11,gs=0.8)
node(n=12,gs=0.8)
node(n= 9,gs=0.8)
PostScript output file
coordinate system
=
=
=
=
=
=
=
3,3,
6,3,
8,3,
1,1,
3,1,
6,1,
8,1,
plot nodes
edge(rd=3,ic=3,a=1.5,1.0) = 1,5,
edge(ic=5,a=1.5,1.0) = 1,7,
edge(rd=3, ic=6,a=1.5,1.0) = 7,1,
edge(ic=2,a=1.5,1.0) = 8,8,
edge(ic=13,a=1.5,1.0) = 11,12,
plot edges
edge(lt=5,ic=4,a=1.5,1.0) = 5,1,
edge(lt=5,rd=-3,ic=15,a=1.5,1.0) = 7,1,
edge(lt=5,ic=0,a=1.5,1.0) = 8,7,
edge(lt=5,rd=3, ic=14,a=1.5,1.0) = 12,11,
);
where i and j give the numbers of nodes to be connected. Note that only
nodes that have a node number specified with the n parameter can be
connected. (Of course, one can use any of TDA’s plot commands to add
plot objects.) All other sub-parameters are optional, see Box 1.
1. Without additional parameters, the edge parameter connects the two
nodes specified by i and j by a straight line. Line type and width
can be specified with the lt and lw parameters, respectively.
2. If i = j, the edge parameter plots a loop. This will be a circle. Its
diameter can be specified with the rd parameter, default is 1.5 times
the size of the corresponding node. As an additional option, one can
use the dir parameter to select a place for the loop. There are four
possible values, dir=0,1,2,3, corresponding to four possible corners.
3. Optionally, two nodes can be connected by a combination of straight
3.6.5
drawing graphs
4
lines and arcs. This can be requested with the rd parameter. A positive value specifies the radius of an arc in clockwise direction, a
negative value does the same in counter-clockwise direction.
4. An edge may end in an arrow. This can be requested with the parameter
a = l,w,
where, respectively, l is the length and w is the width of the arrow
head in mm.
5. Finally, one can use the ic parameter to specify an edge value (integer
or floating point). This value is then written over the edge using the
font size defined by the fs parameter.
Example 1 To illustrate the plg command, Box 2 shows the command
file, gd3.cf, that was used to create the plot shown in Figure 3.6.2-2.
3.6.6
1
trees
3.6.6 Trees
An important special case of graphs is called trees, defined as connected
graphs without cycles. (In this section we always assume undirected
graphs.) Alternatively, a tree can be characterized by one of the following
equivalent conditions (see Barthélemy and Guénoche [1991, p. 4]):
1. The graph is connected and the number of nodes equals the number
of edges plus 1.
2. The graph has no cycles and the number of nodes equals the number of edges plus 1.
3. Any two nodes of the graph are connected by exactly one path.
4. The graph is connected and removing any one of its edges results
in an unconnected graph.
As any other graph, a tree can be valued or unvalued. There are two
kinds of nodes: nodes having degree one are called leaves, and nodes
having degree greater than one are called interior nodes.
A tree is said to be rooted , if one of its nodes is specified as the root
of the tree. Since any node of a tree can be selected as a root, each tree
can be represented by several different rooted trees.
Selecting a root for a tree allows to define a partial order relation
on the set of nodes. Let G = (N , E, r) denote a rooted tree, r ∈ N
being the root. For i, j ∈ N , we can then define: i ≤ j, if j lies on the
path connecting i and r. This defines a partial order relation which is
reflexive, antisymmetric, and transitive (see Barthélemy and Guénoche
[1991, p. 5]). In addition, we can define i < j, if i ≤ j and i 6= j.
Using this order relation, one can introduce the following terminology.
If i < j, j is called the predecessor of i, and i is called the successor of
j. This terminology views the tree as originating in its root. Note that
each node, except the root, has exactly one predecessor, but may have
any number (or zero) of successors.
Example 1 The right part of Box 1 shows data for a valued tree with
13 nodes in the form of an edge list. There are 12 edges connecting nodes
EI and EJ with values given by EV. The right part of the box contains
a tree data structure that will be explained below. Figure 1 shows a
d030606.tex
April 19, 1998
3.6.6
2
trees
Box 1 Edge list (left) and tree data structure (right). (Data file gd4.dat)
EI EJ
EV
----------6
2 1.0
6
4 1.5
7
3 2.0
7
5 2.5
8
1 1.5
8
6 1.0
9
7 3.0
9
8 1.2
1 10 1.6
1 11 2.0
1 12 3.0
7 13 4.0
i
EI EJ
EV
PR
SU
F
------------------------------1
6
2 1.5
8
10
6
2
6
4 1.0
6
0
4
3
7
3 2.0
7
0
5
4
7
5 1.5
6
0
0
5
8
1 2.5
7
0
13
6
8
6 1.0
8
2
0
7
9
7 3.0
9
3
8
8
9
8 1.2
9
1
0
9
10
1 0.0
0
7
0
10
11
1 1.6
1
0
11
11
12
1 2.0
1
0
12
12
13
7 3.0
1
0
0
13
0
0 4.0
7
0
0
12
11
10
13
1
2
6
8
9
7
5
4
3
Figure 1 Axial representation of the tree data shown
in Box 1.
graphical display of this tree. How to create such plots will be explained
below.
Tree Data Structure
Data input for trees is the same as for any other graph, i.e., by an edge list
which is then used by TDA to create a gdd data structure. If a procedure
expects a tree, it is checked whether the currently defined graph is in
fact a tree. If not, the procedure returns with an error message.
3.6.6
trees
3
For internal representation of rooted trees, TDA follows a proposal of
Barthélemy and Guénoche [1991, pp. 5–9] and uses, additionally, a specific data structure that keeps track of the ordering between the nodes,
induced by specifying a root node. To explain this data structure, we
use the example data in Box 1. The left part of the box shows the input
data as an edge list. The right part of the box shows the corresponding
tree data structure based on selecting node 9 as a root.
1. The first column (i) counts the nodes.
2. The next two columns, labeled EI and EJ represent the edges of the
tree. With n nodes, the tree has n − 1 edges and the last row will be
empty.
3. The column labeled EV contains values of the edges; if the tree is
unvalued, all edges will be valued by 1. Note that EV[i] corresponds to
the edge connecting i and PR[i], that is, the predecessor of i.
4. The column labeled PR contains the predecessor of each node; except
the root node where PR contains 0.
5. The column labeled SU contains zero if the corresponding node does
not have a successor, otherwise it contains the number of one arbitrarily
selected successor.
6. The column labeled FR allows to find all successors of a node. Given
node i, the successors of i are given by
SU[i], FR[SU[i]], FR[FR[SU[i]]], ...
until the result becomes zero. For instance, node 8 has predecessor 9.
One of its successors is given by SU[8] = 1, another one by FR[1] = 6.
Since FR[6] = 0 there are no more successors of node 8.
The command ptree, with syntax shown in Box 2, can be used to print
a tree data structure into a file. By default, the command selects an
arbitrary node to become the root of the tree. Alternatively, one can
specify a root number with the rt parameter.
Example 2 To illustrate, we use the ptree command to create the
data shown in Box 1. Having already build a gdd data structure based
on the edge list in the data file gd4.dat, the command is
ptree (rt = 9) = d;
In this example we have selected node 9 to become the root of the tree.
3.6.6
trees
4
Box 2 Syntax of ptree command
ptree (
gn=...,
rt=...,
nfmt=...,
fmt=...,
) = fname;
graph number, def. 1
root of the tree, def. any
integer print format, def. 4
print format for values, def. 10.4
Box 3 Syntax of pltree command
pltree (
gn=...,
gt=...,
rt=...,
pl=...,
lt=...,
lw=...,
fs=...,
nc=...,
ni=1,
);
graph number, def. 1
type of graph (1 or 2), def. 2
number of root node, def. any
type of plot (1,2,3), def. 3
line type, def. 1
line width in mm, def. 0.2
font size in mm, def. 2
label until node ...
label only for leaves, def. 0
Graphical Display of Trees
Barthélemy and Guénoche [1991, pp. 23–31] have developed three different methods for a graphical display of trees. These methods are quite
useful and have been implemented in TDA. The command is pltree with
syntax shown in Box 3.
The command requires a PostScript output file, defined with the
psfile command, and a valid coordinate system, defined with psetup
(see 4.1). Arbitrary coordinates are possible since the trees are automatically scaled to fit into the given coordinate system.1 However, the
pltree command tries to make the physically plotted length of the edges
proportional to their values. One should use, therefore, the pxlen and
pylen parameters in the psetup command to specify a suitable physical
size of the plot.
1A
simple choice would be to set pxa=0,1 and pya=0,1.
3.6.6
trees
13
5
5
3
7
9
8
10
1
6
4
2
11
12
Figure 2 Radial representation of the tree in Box 1.
1. The gn parameter can be used to select a graph number, default is
gn=1.
2. The gt parameter can be used to select an interpretation of the graph
data. Possible choices are gt=1 and gt=2, default is gt=2, that is, the
graph is interpreted as undirected and valued.
3. The lt, lw and fs parameters can be used to modify the line type,
line width, and font size, respectively.
4. The font size is only used to plot labels (node numbers). If fs=0,
labels are not plotted. Also, labels are only plotted for nodes i = 1, . . . , n
where n is an integer given with the nc parameter. (Note that these are
internal node numbers.) By default, nc=0, that is, labels are not plotted.
In addition, one can use the ni parameter. If ni=1, labels are only plotted
for leaves, not for interior nodes.
5. Three different types of tree drawing can be selected with the pl parameter. Radial drawing (pl=1), hierarchical drawing (pl=2), and axial
drawing (pl=3). Default is pl=3.
Radial drawing. The plot begins at the root of the tree, and then
all edges beginning at the root are spread around the root, and then,
recursively, the subtrees are plotted, and so on. Figure 1 illustrates this
3.6.6
6
trees
9
8
6
4
1
2
11
7
10
5
12
3
13
Figure 3 Hierarchical representation of the tree in Box 1.
type of tree plotting with the example data in Box 1.
Hierarchical drawing. The tree is plotted as a hierarchy beginning at
the root. Edges are plotted in the y direction of the coordinate system
with edge lengths reflecting their values. Figure 2 illustrates this type of
tree plotting with the example data in Box 1.
Axial drawing. The root, and as many nodes as possible, are plotted
on a line parallel to the x axis. Any remaining nodes and subtrees are
then plotted across the available space. Figure 3 illustrates this type
of tree plotting with the example data in Box 1. Note that this option
requires that the tree is rooted at a node with degree greater than 1. If
the procedure is not able to plot the tree for the requested root node it
returns with an error message and one should try another root.
Example 3 The example archive contains the command files gd15.cf,
gd15a.cf, and gd15b.cf, that have been used to create the plots shown
in this section.
4
1
postscript plots
4. PostScript Plots
TDA’s plot functions are based on the PostScript language, a trade-
mark of Adobe Systems, Inc. This allows for graphics output which is
independent of a hardware-specific graphics environment. Creating plots
with TDA actually means creating an output file containing a PostScript
description of the plot. It will be possible then to send this output file to
a PostScript printer to get a hard copy of the plot. Using a PostScript
previewer, like GhostScript, it should also be possible to preview plots
on a terminal screen.
Moreover, many text processing packages are able to directly include
PostScript files. For instance, the documents describing TDA have been
written in LATEX, based on Donald Knuth’s TEX. Combined with Tomas
Rokicki’s DVIPS, a TEX to PostScript driver, one can directly include
PostScript files. In fact, almost all figures included in this manual have
been created with TDA. To support this possibility, TDA’s plot output
files follow the conventions for encapsulated PostScript (Version 3.0) as
described in Adobe’s PostScript Language Reference Manual [1990].
4.1
Preparing the Plot Environment explains how to set up the environment for subsequent plot commands.
4.2
Plotting the Coordinate Systems explains how to plot coordinate systems.
4.3
Combining Plot Files shows how to combine separate plot files
into a single plot.
4.4
Plot Objects describes commands for a variety of plot objects.
This manual does not provide an introduction to the PostScript language. Actually, it is not necessary to know very much about this graphics language in order to create PostScript plots with TDA. On the other
hand, the available commands to control TDA’s PostScript output are
limited. To make changes, or to access features of PostScript not supported by TDA, one needs to edit the PostScript output files. This is easy
but requires, of course, some understanding of the PostScript language.
It is also not intended to give an introduction to computer graphics; this might be found, for instance, in Mortensen [1989], Penna and
d04.tex
April 19, 1998
4
postscript plots
2
Patterson [1986], and Watt [1989]. Useful references discussing the possibilities, and limitations, of using graphical displays in applied statistics
are Chambers et al. [1983], and Schnell [1994].
4.1
1
preparing the plot environment
4.1 Preparing the Plot Environment
Creating PostScript plots with TDA actually means to create a file containing a PostScript description of the plot. This is done in three steps.
1. The first step is to create an output file for subsequent plot commands.
The command is
psfile = name of output file;
The file remains opened until the program terminates, the psfile command is used again to create a new PostScript output file, or the file is
explicitly closed with the command
psclose;
2. The second step is to set up a coordinate system and, optionally, to
modify the physical dimensions of the plot. The command is
psetup (parameter);
Parameters are shown in Box 1 and will be explained below. The pxa
and pya parameters are required to define a coordinate system. All other
parameters are options. Note that the psetup command can be used
several times with the same output file allowing several plots on the
same page.
3. In a third step one can add plot objects by using any of the plot
commands that will be described in 4.4.
Parameters for the psetup command. Having defined a PostScript
output file with the psfile command, the psetup command must be
used to specify a logical coordinate system for subsequent plot commands. The required parameters are pxa and pya. The syntax is
pxa = x0 , x1 ,
to specify [x0 , x1 ] as the range of the x axis, and analogously for the y
axis. As default, TDA assumes linear axes. Optionally, one can specify a
logarithmic x axis with the parameter
pxa (log) = x0 , x1 ,
d0401.tex
April 19, 1998
4.1
preparing the plot environment
2
Box 1 Syntax for psetup command
psetup (
pxa=...,
pya=...,
pxlen=...,
pylen=...,
psorg=...,
psscal=...,
psrot=...,
definition of x axis
definition of y axis
length of x axis in mm, def. pxlen=120
length of y axis in mm, def. pylen=80
PostScript coordinates of origin,
def psorg=150,460
scaling factors, def psscal=1,1
rotation, def. psrot=0
);
where x0 > 0, and in the same way for the y axis. All other parameters
of the psetup command are optional.
1. The pxlen and pylen parameters can be used to modify the physical
size of the plot. Default values are pxlen=120 (mm) and pylen=80 (mm).
2. The psorg parameter can be used to modify the place of the plot in
the PostScript coordinate system (see below). Default is psorg=150,460.
3. The psscal parameter can be used to scale a plot. The syntax is
psscal = sx , sy , with sx and sy the scaling factors for the x and y
axes, respectively. Default is psscal=1,1.
4. The psrot parameter can be used to rotate the plot in the PostScript
coordinate system. Syntax is psrot = α, where α is the angle (in degrees) for rotating the plot in counterclockwise direction. Default is
psrot=0.
Example 1 To give a first example, Box 2 shows the command file
plot1.cf that was used to create the plot shown in Figure 1. It begins
with defining the PostScript output file, plot1.ps. Then follows the
psetup command using the pxlen and pylen parameters to specify the
physical size of the plot, and the pxa and pya parameters to specify the
logical coordinate system. Then follow the plxa and plya commands to
plot the axes, and the plframe command to put a frame around the
plot. These commands will be explained in 4.2. Finally, there are four
plot objects (see 4.4). The plotf command plots the sin function and
the plotf1 command its first derivative. The two pltext commands
plot labels.
4.1
3
preparing the plot environment
Box 2 Command file plot1.cf
psfile = plot1.ps;
output file
psetup(
pxlen =
pylen =
# length of x axis in mm
# length of y axis in mm
pxa
pya
90,
50,
= 0,6,
= -1,1,
# user coordinates on x axis
# user coordinates on y axis
);
plxa (sc=1,ic=10);
plya (sc=1,ic=0);
plframe;
plot x axis
plot y axis
plot frame
plotf (rx=0(0.1)6) = sin(x);
plotf1(rx=0(0.1)6) = sin(x);
pltext(xy=2.7,0.6) = sine ;
pltext(xy=4.6,0.6) = cosine ;
1
sine
cosine
0
-1
0
1
2
3
4
5
6
Figure 1 Example of a PostScript plot, created with command
file plot1.cf shown in Box 2.
PostScript Coordinate System. As mentioned, creating PostScript
plots requires the specification of a coordinate system. There are two
kinds of coordinates: logical and physical coordinates. All plot objects
are defined with respect to a system of logical coordinates, specified
with the pxa and pya parameters in the psetup command. A physical
coordinate system is used to map the logical coordinate system onto a
4.1
4
preparing the plot environment
792
720
648
576
504
432
360
288
216
144
72
0
0
72
144
216
288
360
432
504
576
Figure 2 PostScript coordinate system corresponding to a DIN A4
page, created with command file plot2.cf (not shown).
4.1
preparing the plot environment
5
two-dimensional space (a sheet of paper or the screen of a terminal).
The physical coordinate system of the PostScript language is defined
in points (1 inch = 72 points). TDA uses mm (millimeter) and calculates
with 1 mm = 2.835 points. On a standard DIN-A4 page the origin is at
the lower left corner. The x dimension has 595 points, the y dimension
has 842 points. Figure 2 illustrates this coordinate system; the plot was
created with command file plot2.cf which is contained in the TDA
example archive.
PostScript Bounding Box. TDA creates encapsulated PostScript files
as described in Adobe’s PostScript Language Reference Manual [1990].
This requires the whole plot to be encapsulated into a bounding box to
be defined in PostScript coordinates. This then allows the PostScript
output file to become part of other PostScript documents.
TDA tries to create a bounding box being sufficiently large to include all plot objects. There may occur situations, however, when this
cannot be done. For instance, if the whole plot is rotated with the psrot
command, the default bounding box is not adjusted in a proper way.
Another situation where the bounding box is not sufficiently large can
occur when the pltext command has been used to place text outside
the coordinate system. However, it is easy to adjust the bounding box
manually. Two different methods can be used. First, one can use the
plrec command to place rectangles (having zero size) at any desired
position which then becomes part of the bounding box. Alternatively,
one can directly change the PostScript coordinates of the bounding box
in the PostScript output file.
4.2
plotting coordinate systems
1
4.2 Plotting Coordinate Systems
The pxa and pya parameters that are part of the psetup command (see
section 4.1) only define a logical coordinate system; they do not create
any graphical output. To plot axes of a coordinate system, one can use
the plxa and plya commands for the x and the y axis, respectively. Both
commands have the same syntax as shown in the following box.
plxa (
sc=...,
ic=...,
lt=...,
lw=...,
fs=...,
fmt=...,
tl=...,
dir=...,
) = xa,ya,xb,yb;
distance of main tick marks
number of sub-intervals
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
font size for labels, def. 2 (mm)
print format for labels, def. 0.0
length of tick marks, def. 1.8 (mm)
direction of axis, def. 0
1. All parameters are optional. Without any parameters, the command
plots simply the axis, without labels, beginning at the lower left corner of
the coordinate system. The lt and lw parameters can be used to control
the line type and line width, respectively. Default is lt=1 (solid line) and
lw=0.2 (mm).
2. To specify a different place for the axes, one can use the right-hand
side parameters. The axis is then plotted from (xa,ya) to (xb,yb) (logical coordinates).
3. As default, the x and y axes are not labeled. To get numerical labels,
one has to define tick marks. This can be achieved with the optional sc
parameter. Given sc=d, this creates a series of main tick marks placed
at
xa + i d i = 0, 1, 2, . . .
until xb is reached. These main tick marks also get numerical labels. The
font size can be controlled with the fs parameter. The size is in mm;
d0402.tex
April 19, 1998
4.2
plotting coordinate systems
2
Box 1 Command file (plot3.cf)
psfile = plot3.ps;
psetup(
pxlen = 90,
pylen = 50,
pxa
pya
= -2,2,
= 0,1,
);
plxa (sc=1,ic=2);
plya (sc=0.5,ic=0);
plframe;
output file
# length of x axis in mm
# length of y axis in mm
# user coordinates on x axis
# user coordinates on y axis
plot x axis
plot y axis
plot frame
plxgrid = 0.5;
plygrid = -1,0,1;
plabel = "Linear Coordinate System";
pxlabel = "X axis";
pylabel(sc=3) = "Y axis";
shift 3 mm to right
default font size is fs=2 (mm). To suppress labels, one can use fs=0. The
print format for the labels can be controlled with the fmt parameter;
default is a free format. The tl parameter controls the length of the
main tick marks (in mm); default is tl=1.8 (mm).
4. In addition, one can request small tick marks with the ic parameter.
Given the parameter ic=n, each interval between two main tick marks
is subdivided into n subintervals.
5. Finally, one can use the dir parameter to control the direction of
the axes. If dir=0 (default), labels for the x axis are placed below, and
labels for the y axis are placed on the left-hand side of the axis. When
using the dir=1 parameter the tick marks and labels are plotted in the
opposite direction. Note that the plxa and plya commands can be used
more than once, see Example 2.
Additional Options. There are a few additional commands to control
the graphical display of the coordinate system. The Command
plframe (lt=,lw=,gs=);
can be used to plot a frame around the coordinate system. The parameters are optional. The lt and lw parameters can be used to control the
line type and line width, respectively. Default is always lt=1 (solid line)
4.2
3
plotting coordinate systems
Box 2 Command file (plot4.cf)
psfile = plot4.ps;
psetup(
pxlen = 90,
pylen = 50,
output file
pxa(log=1) = 1,1000,
pya(log=1) = 2,500,
);
plxa (sc=10,ic=10);
plya (sc=5,ic=0);
plframe;
plxgrid = 10,50,250;
plygrid = 10,100,1000;
plabel = "Logarithmic Coordinate System";
pxlabel = "X axis";
pylabel(sc=3) = "Y axis";
shift 3 mm to right
Linear Coordinate System
Y axis
1
0.5
0
-2
-1
0
X axis
1
2
Figure 1 Plot of plot3.ps, created by command file
plot3.cf shown in Box 1.
and lw=0.2 (mm). The parameter gs=x can be used to fill the rectangle
with a grey tone. x must be a real value in the range from 0 (black) to
1 (white).
The plxgrid and plygrid commands can be used to put a system
4.2
4
plotting coordinate systems
Logarithmic Coordinate System
Y axis
250
50
10
2
1
10
100
X axis
1000
Figure 2 Plot of plot4.ps, created by command file
plot4.cf shown in Box 2.
Box 3 Command file plot5.cf
psfile = plot5.ps; output file
psetup(
pxlen = 90,
pylen = 50,
pxa
= -2,2,
pya
= 0,1,
);
plxa (sc=1,ic=2);
plot x axis
plya (sc=0.5,ic=0);
plot y axis
plxa(sc=1,ic=5,dir=1) = -1,1,1,1;
plya(sc=0.5,ic=5,dir=1) = 2,0,2,1;
plot another x axis
plot another y axis
of grid lines onto the coordinate system. The syntax for the plxgrid
command is
plxgrid (lt=,lw=) = y1,y2,... ;
This places horizontal grid lines (parallel to the X axes) at the logical
Y coordinates y1,y2,... The default line type is lt=3 (dotted), and
the default line width is lw=0.05 (mm). The plygrid command has an
analogous syntax.
4.2
5
plotting coordinate systems
-1
0
1
1
1
0.5
0.5
0
0
-2
-1
0
1
2
Figure 3 Plot of plot5.ps, created by command file
plot5.cf shown in Box 3.
Three further commands can be used to plot labels. The command
plabel (sc=,fs=) = string ;
places string at the top of the coordinate system to provide a label for
the whole plot. The default font size is fs=3 (mm). The parameter sc
(default sc=0 mm) can be used to change the distance between the top of
the coordinate system and the label. string should be enclosed in (single
or double) quotation marks to maintain blank characters, and should
only contain characters from a single font (see 4.4.3). If one needs labels
containing characters from different fonts, one should use the pltext
command.
Analogously, one can use the commands pxlabel and pylabel to
define labels for the x and y axes, respectively. The syntax is identical
with the plabel command; default font size is fs=2.4 (mm).
Example 1 Command file plot3.cf, shown in Box 1, illustrates how
to plot a linear coordinate system. The resulting PostScript plot, written into the output file plot3.ps, is shown in Figure 1. Command file
plot4.cf (Box 2) illustrates a logarithmic coordinate system; the resulting plot is shown in Figure 2.
Example 2 The plxa and plya commands can be used several times
to place different axes into the same plot. An example, plot5.cf, is
shown in Box 3. The resulting plot is Figure 3.
4.2
6
plotting coordinate systems
Box 4 Command file plot6.cf
psfile = plot6.ps; output file
psetup(
psorg = 100,600,
# set physical origin
pxlen = 50,
pylen = 40,
pxa = 0,6,
pya = -1,1,
);
plxa(sc=1);
plya(sc=1);
plotf(rx=0(0.1)6) = sin(x1);
pltext(xy=4,0.2) = "Plot 1";
psetup(
psorg = 270,600,
pxlen = 50,
pylen = 30,
pxa = 0,6,
pya = -1,1,
);
plxa(sc=1);
plya(sc=1);
add a plot object
give a label
# define a new coordinate system
# set physical origin
plotf(rx=0(0.1)6) = sin(x1);
pltext(xy=4,0.2) = "Plot 2";
add a plot object
give a label
1
1
Plot 1
0
Plot 2
0
-1
-1
0
1
2
3
4
5
6
0
1
2
3
4
5
6
Figure 4 Example of multiple coordinate systems in a single
PostScript file. The plot was created with command file plot6.cf
shown in Box 4.
4.2
plotting coordinate systems
7
Multiple Coordinate Systems. The psetup command can be used
several times with the same PostScript output file to allow multiple coordinate systems in a single plot. For properly placing the coordinate
systems, one should use the psorg parameter; for an orientation see Figure 4.1-2.
Example 3 Command file plot6.cf, shown in Box 4, provides an example of using two coordinate systems for a single PostScript output file.
The first psetup command creates a coordinate system having its lower
left corner at PostScript coordinates (100,600). All subsequent plot commands use this coordinate system, until a new one is created with the
second psetup command. The remaining plot commands use this second
coordinate system. Figure 4 shows the resulting plot. The relative size
and position of the two parts is determined by their physical size and
origins. Note that the origins, that is, the lower left corner of the physical
coordinate systems, must be specified with reference to the PostScript
coordinate system, see section 4.1.
4.3
1
combining plot files
4.3 Combining Plot Files
The dplot command can be used to combine a set of PostScript files into
a single file. This command is independent of all other plot commands
and in particular does not require the psfile and psetup commands.
The syntax is shown in the following box.
dplot (
fn=...,
pxlen=...,
pylen=...,
psorg=...,
) = fname;
list of input files
x-size of output plot in mm, def. 120
y-size of output plot in mm, def. 80
origin of output plot, def. psorg=10,10
The right-hand side must provide the name of an output file, and the
fn parameter must be used to specify at least one set of PostScript files
previously created with TDA. These files are placed in one row of plots
in the output file. If used more than once (up to 50), each set of files is
placed into a separate row of plots.
The other parameters are optional. The pxlen and pylen parameters
can be used to modify the physical size of the resulting output plot. The
psorg parameter can be used to modify the origin of the output plot;
syntax is psorg=x,y where x and y are PostScript coordinates.
d0403.tex
April 19, 1998
4.3
2
combining plot files
Box 1 Command file plot7.cf
dplot(
pxlen = 100,
# length of x axis in mm
pylen = 60,
# length of y axis in mm
fn = plot3.ps, plot3.ps,
fn = plot4.ps, plot4.ps,
) = plot7.ps;
Linear Coordinate System
Linear Coordinate System
1
Y axis
1
Y axis
# name of output file
0.5
0
0.5
0
-2
-1
0
1
X axis
2
-2
-1
1
2
Logarithmic Coordinate System
250
250
50
50
Y axis
Y axis
Logarithmic Coordinate System
0
X axis
10
10
2
2
1
10
X axis
100
1000
1
10
X axis
100
1000
Figure 1 PostScript plot plot7.ps, created with command
file plot7.cf shown in Box 1.
Example 1 To illustrate the dplot command, Box 1 shows the command file plot7.cf. This command files combines four plots and writes
the result into the output file plot7.ps. The resulting plot is shown in
Figure 1.
While this example works fine, problems can occur when the relation
between the physical origin and the bounding box is not the same in all
input files. There is, in general, no safe automatic way to arrange the
individual plots such that the individual coordinate axes are always in
line. If such problems occur, one has to edit the final output file and
adjust the xorg and yorg commands.
4.4
plot objects
1
4.4 Plot Objects
This chapter describes commands for a variety of standard plot objects.
Further plot commands may also be found in other parts of the manual.
4.4.1
Polygons and Scatterplots
4.4.2
Line Types and Marker Symbols
4.4.3
Plotting Text
4.4.4
Rectangles
4.4.5
Step Functions
4.4.6
General Functions
4.4.7
Arrows
4.4.8
Circles, Arcs, Ellipses
4.4.9
Convex Hull
4.4.10
Smoothing Polygons
4.4.11
Contour Plots
d0404.tex
April 19, 1998
4.4.1
polygons and scatterplots
1
4.4.1 Polygons and Scatterplots
There are two commands to plot an arbitrary sequence of data points,
plot and plotp. The only difference is in the way the data points are
supplied. With the former one, the data points are taken from the program’s internal data matrix, with the latter one, the data points can be
explicitly supplied as part of the command.
One important application of these commands is to create scatterplots. However, the same command can be used to plot a line, a rectangle,
or an arbitrary function, y = f (x), given by a sequence of points, (xi , yi ).
The syntax of the first command, plot, is shown in Box 1. The righthand side of the command must contain exactly two variable names,
to be used for the x and y coordinates of the data points. All other
parameters are optional.
1. By default, the command uses all cases (rows) of the currently defined
data matrix. To select a subset of cases one can use the sel parameter
with syntax
sel = expression;
Given any valid expression which may contain names of variables, constants and operators, the command uses only those cases of the data
matrix where the evaluation of expression results in a value not equal
to zero. Note that there must be at least two data points.
2. The parameters lt and lw can be used to change the line type and
line width, respectively. Defaults are lt=1 (solid line) and lw=0.2 (mm);
see the description in 4.4.2.
3. The s parameter can be used to request marker symbols to be plotted
at the location of the data points. The syntax is s=n where n is the
number of one of the available marker symbols, see 4.4.2. The size of
the markers symbols can be controlled with the fs parameter; default is
fs=2 (mm).
4. By default, the data points are connected by straight lines. To suppress these lines, one can use lt=0 or lw=0. However, the line width
defined by the lw parameter is also used for the marker symbols. Thered040401.tex
May 29, 2002
4.4.1
polygons and scatterplots
2
Box 1 Syntax for plot command
plot (
lt=...,
lw=...,
s=...,
fs=...,
gs=...,
r=...,
dir=...,
ns=...,
a=...,
nc=...,
sel=...,
) = VX,VY;
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
marker symbol
size of marker symbol, def. 2 (mm)
grey scale value, def. 1 (white)
creates step function
type of step function
if ns=1 plot only horizontal lines, def. 0
creates an arrow
no clipping option, def. 0
selection of data points
Box 2 Syntax for plotp command
plotp (
lt=...,
lw=...,
s=...,
fs=...,
gs=...,
r=...,
dir=...,
ns=...,
a=...,
nc=...,
) = x1 , y1 , . . . , xn , yn ;
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
marker symbol
size of marker symbol, def. 2 (mm)
grey scale value, def. 1 (white)
creates step function
type of step function
if ns=1 plot only horizontal lines, def. 0
creates an arrow
no clipping option, def. 0
fore, to get only the marker symbols without connecting lines, one should
use lt=0.
5. The r, dir and ns parameters can be used to request step functions
and histograms as will be explained in 4.4.5.
6. The a parameter can be used to request that the polygon ends in an
arrow, see 4.4.7.
7. The gs parameter can be used to fill the area between the x axis and
4.4.1
3
polygons and scatterplots
Box 3 Command file plot8.cf
nvar(
noc = 20,
# create data
RD = rd(-2,2),
RDS = sort(RD),
ND = nd(RDS),
);
psfile = plot8.ps; output file
psetup(
pxlen = 90,
pylen = 50,
pxa
= -2,2,
pya
= 0,1,
);
plxa (sc=0.5,fmt=4.1); plot x axis
plya (sc=1,ic=10);
plot y axis
plframe;
plot frame
plot (s=4,fs=1.5) = RDS,ND;
polygon with marker symbols
1
0
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Figure 1 PostScript plot plot8.ps, created with command
file plot8.cf shown in Box 3.
the polygon with a grey tone. The syntax is gs=x with x a real value
between 0 (black) and 1 (white).
8. By default, the plot is restricted to the area defined by the currently
active coordinate system. Any part of the plot object that falls outside
this area is clipped. To allow for plotting outside this area one can use
the command nc=1. The program then also tries to adjust the bounding
4.4.1
polygons and scatterplots
4
Box 4 Syntax for plotm command
plotm (
lt=...,
lw=...,
s=...,
fs=...,
gs=...,
r=...,
dir=...,
a=...,
nc=...,
sel=...,
) = X1,Y1,X2,Y2,...;
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
marker symbol
size of marker symbol, def. 2 (mm)
grey scale value, def. 1 (white)
creates step function
type of step function
creates an arrow
no clipping option, def. 0
selection of data points
box of the plot properly.
The syntax of the second command, plotp, is shown in Box 2. The righthand side is a sequence of n data points to be given in user coordinates.
There must be at least two data points. All other parameters are optional
and have the same meaning as explained above.
Example 1 To illustrate, command file plot8.cf, shown in Box 3,
generates 20 equally distributed random numbers in the range (−2, +2).
These random numbers are sorted and taken as arguments of the standard normal distribution function. As shown by the plot command, the
data points with coordinates given by variables RDS and ND are plotted
using a marker symbol of type 4 and size 1.5 mm; in addition, the points
are connected by a sequence of straight line segments. The resulting plot
is shown in Figure 1.
Separate Polygons. The plotm command can be used to plot a polygon (or scatter plot) separately for each row in the data matrix. The
syntax is shown in Box 4. The right-hand side must provide at least
two pairs of variables: X1,Y1,X2,Y2,.... The command uses the values
of these variables as coordinates, separately for each row in the data
matrix, meaning that the ith plot is based on
(X1(i),Y1(i)), (X2(i),Y2(i)), ...
where i refers to the ith row in the data matrix. All parameters are
optional and have the same meaning as explained for the plot command.
4.4.2
1
line types and marker symbols
4.4.2 Line Types and Marker Symbols
Many plot commands have an lt parameter to select a line type and an
s parameter to select a marker symbol. The available options are shown
in the following boxes.
line type 9
line type 8
line type 7
line type 6
line type 5
line type 4
line type 3
line type 2
line type 1
marker 1
marker 10
marker 2
marker 11
marker 3
marker 12
marker 4
marker 13
marker 5
marker 14
marker 6
marker 15
marker 7
marker 16
marker 8
marker 17
marker 9
Note that there is an additional line type: zero, meaning that no line is
drawn to connect two data points. This is a useful option if, for instance,
data points shall be plotted by marker symbols without connecting lines.
d040402.tex
April 19, 1998
4.4.2
line types and marker symbols
2
A zero line type can also be used to create grey-scaled plot objects without visible bounding line segments.
4.4.3
plotting text
1
4.4.3 Plotting Text
To plot an arbitrary text, one can use the pltext command with syntax
shown in the following box.
pltext (
xy=...,
fs=...,
sc=...,
r=...,
s=...,
) = string;
coordinates
font size, def. 2 (mm)
sc=1 centers the string, def. 0
rotation, def. 0
number of a marker symbol
The string given on the right-hand side is plotted beginning at the coordinates (x,y) to be given with the xy=x,y parameter. Note that strings
should be enclosed in single or double quotation marks to preserve blank
characters.
All other parameters are optional. The fs parameter can be used
to change the font size; the default is 2 mm. The parameter sc=1 can
be used to center the string around the point (x,y). The r parameter
can be used to rotate the string; r=d rotates the string by d degrees
in counterclockwise direction; default is r=0. Finally, one can use the s
parameter to provide the number of a marker symbol. This symbol is
then plotted in front of the string. Note that this option implies that the
string cannot be centered and/or rotated.
Fonts. The PostScript language provides a variety of different fonts. In
TDA, two standard fonts are used for strings defined with the pltext
and/or plabel commands: Times-Roman and Symbol. It is assumed
that both fonts are available on almost all PostScript devices. Figures 1
and 2 show the available characters and symbols for these two fonts. (We
are grateful to Wolfgang Voges (Bremen) who has provided a convenient
recoding of these fonts containing a lot of useful characters and symbols
and has shown us how to include these fonts into TDA’s PostScript
features.) To reference these characters and symbols, one should use
\nnn
d040403.tex
and
@nnn
April 19, 1998
4.4.3
plotting text
2
for Times-Roman and Symbol fonts, respectively. In both cases, nnn
is the character’s octal code as shown in the top left corner of each
character’s box.
Example. Command file plots.cf, shown in Box 1, illustrates possible
usage of special characters and symbols. The resulting plot is shown in
Figure 3.
4.4.3
3
plotting text
0
1
6
7
8
9
A
B
C
D
E
F
5
6
7
8
9
A
B
C
D
E
F
3
4
5
6
7
10
11
12
13
14
15
16
17
20
21
22
23
24
25
26
27
30
31
32
33
34
35
36
37
40
41
42
43
44
45
46
47
50
51
52
53
54
55
56
57
60
61
62
63
64
65
66
67
70
71
72
73
74
75
76
77
100
101
102
103
104
105
106
107
110
111
112
113
114
115
116
117
120
121
122
123
124
125
126
127
130
131
132
133
134
135
136
137
140
141
142
143
144
145
146
147
150
151
152
153
154
155
156
157
160
161
162
163
164
165
166
167
170
171
172
173
174
175
176
177
200
201
202
203
204
205
206
207
210
211
212
213
214
215
216
217
220
221
222
223
224
225
226
227
230
231
232
233
234
235
236
237
240
241
242
243
244
245
246
247
250
251
252
253
254
255
256
257
260
261
262
263
264
265
266
267
270
271
272
273
274
275
276
277
300
301
302
303
304
305
306
307
310
311
312
313
314
315
316
317
320
321
322
323
324
325
326
327
330
331
332
333
334
335
336
337
340
341
342
343
344
345
346
347
350
351
352
353
354
355
356
357
360
361
362
363
364
365
366
367
370
371
372
373
374
375
376
377
Ð ð Ł ł Š š Ý ý Þ þ Ž ž
½¼ ¹ ¾ ³ ² ¦ − × 2
5
4
2
1
4
3
1
0
3
2
0
! " # $ %& ’ ( ) * + , - . /
0 1 2 3 4 5 6 7 8 9 : ; < = > ?
@A B C D E F G H I J K LMN O
P Q R S T U VWX Y Z [ \ ] ^ _
‘ a b c d e f g h i j k l m n o
p q r s t u v w x y z { | } ~ 
ÄÅÇ ÉÑÖÜ á à â ä ã å ç é è
ê ë í ì î ï ñ ó ò ô ö õ ú ù û ü
† ° ¢ £ § • ¶ ß ® ©™ ´ ¨ ≠ Æ Ø
∞ ± ≤ ≥ ¥ µ Γ∑ ∏ π ∫ ª º Ω æ ø
¿ ¡ ¬ √ ƒ ≈ ∆ « »… À à Ռœ
– — “ ” ‘ ’ ÷ ◊ ÿ Ÿ ⁄ ¤ ‹ › fi fl
‡ · ‚ „ ‰Â Ê Á Ë È Í Î Ï Ì Ó Ô
 Ò
ı
Figure 1 Times-Roman fonts in TDA encoding.
4.4.3
4
plotting text
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
0
1
2
3
4
5
6
7
10
11
12
13
14
15
16
17
20
21
22
23
24
25
26
27
30
31
32
33
34
35
36
37
40
41
42
43
44
45
46
47
50
51
52
53
54
55
56
57
60
61
62
63
64
65
66
67
70
71
72
73
74
75
76
77
100
101
102
103
104
105
106
107
110
111
112
113
114
115
116
117
120
121
122
123
124
125
126
127
130
131
132
133
134
135
136
137
140
141
142
143
144
145
146
147
150
151
152
153
154
155
156
157
160
161
162
163
164
165
166
167
170
171
172
173
174
175
176
177
200
201
202
203
204
205
206
207
210
211
212
213
214
215
216
217
220
221
222
223
224
225
226
227
230
231
232
233
234
235
236
237
240
241
242
243
244
245
246
247
250
251
252
253
254
255
256
257
260
261
262
263
264
265
266
267
270
271
272
273
274
275
276
277
300
301
302
303
304
305
306
307
310
311
312
313
314
315
316
317
320
321
322
323
324
325
326
327
330
331
332
333
334
335
336
337
340
341
342
343
344
345
346
347
350
351
352
353
354
355
356
357
360
361
362
363
364
365
366
367
370
371
372
373
374
375
376
377
0
1
2
3
4
5
6
7
! ∀ # ∃ %& ∋ ( ) ∗ + , − . /
0 1 2 3 4 5 6 7 8 9 : ; < = > ?
≅ Α Β Χ ∆ Ε Φ Γ Η Ι ϑ Κ ΛΜΝ Ο
ΠΘ Ρ Σ Τ Υ ς ΩΞΨΖ [ ∴ ] ⊥ _
 α β χ δ ε φ γ η ι ϕ κ λ µ ν ο
π θ ρ σ τ υϖω ξ ψ ζ { | } ∼
8
9
A
B
C
D
E
F
€ ϒ ′ ≤ ⁄ ∞ ƒ ♣ ♦ ♥ ♠ ↔← ↑ → ↓
° ± ″ ≥ × ∝ ∂ • ÷ ≠ ≡ ≈…↵
ℵ ℑ ℜ℘⊗ ⊕ ∅ ∩ ∪ ⊃ ⊇ ⊄ ⊂ ⊆ ∈ ∉
∠ ∇    ∏ √ ⋅ ¬ ∧ ∨ ⇔⇐ ⇑ ⇒ ⇓
◊ ⟨ ∑          
⟩ ∫ ⌠⌡         
Figure 2 Symbols in TDA encoding.
4.4.3
5
plotting text
Box 1 Command file plots.cf
psfile = plots.ps;
psetup(
pxlen = 120,
pylen = 60,
pxa = 0,10,
pya = 0,5 ,
);
plxa (sc=1);
plya (sc=1,ic=2);
plframe;
output file
pltext(xy=1,4) = "@141 = 2.5 \245 @142";
pltext(xy=4,4) = "pltext (xy=1,4) =
\324\100141 = 2.5 \\245 \100142\324";
pltext(xy=1,2) = "@123 = @173 @161 @174 @161 @316 @121 @175";
pltext(xy=2.2,0.9) = "pltext (xy=1,2) = \324\100123 =
\100173 \100161 \100174 \100161 \100316 \100121 \100175\324";
pltext(xy=1,3) = "Sm\277rebr\277d";
pltext(xy=4,3) = "pltext (xy=1,3) = \324Sm\\277rebr\\277d\324";
plotp (a=1,1) = 3.5,4.05,2.7,4.05;
plotp (a=1,1) = 3.5,3.05,2.7,3.05;
plotk (sc=-90) = 1.5,1.5,3.2,1.5,3.2,2.1;
plotp (a=1,1) = 3.2,2.1,2.9,2.1;
plotk (sc=-90) = 1.5,1.5,1.5,0.95;
plotp = 1.5,0.95,2,0.95;
5
4
α = 2.5 • β
pltext (xy=1,4) = ‘@141 = 2.5 \245 @142‘
3
Smørebrød
pltext (xy=1,3) = ‘Sm\277rebr\277d‘
2
Σ={θ|θ∈Θ}
1
pltext (xy=1,2) = ‘@123 = @173 @161 @174 @161 @316 @121 @175‘
0
0
1
2
3
4
5
6
7
8
9
10
Figure 3 PostScript plot plots.ps, created with command
file plots.cf shown in Box 1.
4.4.4
rectangles
1
4.4.4 Rectangles
To plot a rectangle one can use the plrec command with syntax shown
in the following box.
plrec (
lt=...,
lw=...,
gs=...,
r=...,
) = x,y,xlen,ylen;
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
grey scale value, def. 1 (white)
rotation, def. 0
The command plots a rectangle with the lower left corner at the point
(x,y); xlen is the width and ylen the height of the rectangle (in logical
coordinates). All other parameters are optional. Default line type is lt=1,
default line width is lw=0.2. The gs parameter can be used to fill the
rectangle with a grey tone. The r parameter can be used to rotate the
rectangle. Note that xlen and ylen can be zero. Of course, the rectangle
is not plotted then; but the bounding box is updated to include the point
(x,y).
d040404.tex
April 19, 1998
4.4.5
1
step functions
4.4.5 Step Functions
The plot and plotp commands can be used to plot step functions and
histograms. The parameter r=1 draws straight lines to connects the
points of the polygon with the x axis. The parameter dir=1 has the effect that the data points are connected by “kinked” line segments where
the pen moves first in the x axis direction, then in the y axis direction.
This creates a step function. Alternatively, with dir=2, the pen moves
first in the y axis direction, then in the x axis direction. By default,
the command draws horizontal and vertical lines. Adding the parameter
ns=1, only horizontal lines will be drawn.
Example 1 To illustrate these options, we modify plot8.cf, the command file that was used in 4.4.1 to illustrate the plot command. In the
new command file, plot11.cf (not shown), the plot command is
plot (dir=1,gs=0.9) = RDS,ND;
The resulting grey-scaled step function is shown in the following figure.
1
0
-2.0
d040405.tex
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
May 29, 2002
4.4.6
general functions
1
4.4.6 General Functions
The plotf command can be used to plot a function y = f (x). The
plotf1 and plotf2 commands will plot, respectively, the first and second
derivative of the given function. All three commands use the same syntax
as shown in the following box.
plotf (
rx=...,
lt=...,
lw=...,
gs=...,
nc=...,
) = function;
range of function argument
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
grey scale value, def. 1 (white)
no clipping option, def. 0
The right-hand side must provide the definition of a function, depending
on a single argument, as explained in 5.3.1. If the definition refers to
at least one data matrix variable, the function is summed over all data
matrix cases.
The function is evaluated in the range [a,b] on the x axis, with increments of size d; these values must be provided with the rx=a(d)b
parameter. For each evaluation, the argument is used as the x axis coordinate and the value of the function as the y axis coordinate. The data
points are connected by straight line segments. The line type and line
width can be controlled with the lt and lw parameters, respectively.
Defaults are lt=1 (solid line) and lw=0.2 (mm).
Optionally, one can use the gs parameter to fill the area below the
function with a grey tone. The nc=1 parameter can be used to allow a
plot of the function outside the coordinate system.
d040406.tex
April 19, 1998
4.4.6
2
general functions
Box 1 Command file plot12.cf
psfile = plot12.ps; output file
psetup(
pxlen = 90,
pylen = 50,
pxa
= -3,3,
pya
= 0,1,
);
plxa (sc=0.5,fmt=4.1); plot x axis
plya (sc=1,ic=10);
plot y axis
plframe;
plot frame
plotf (rx=0(0.05)3,) = ndf((x - 1.5) / 0.5) / 0.5;
plotp = 1.5,0,1.5,0.798;
plotf(rx=-3(0.1)3,gs=0.8) = ndf(x);
plotp = 0,0,0,0.399;
1
0
-3.0 -2.5 -2.0 -1.5 -1.0 -0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Figure 1 Example of a PostScript plot, created with command
file plot12.cf shown in Box 1.
Example 1 To illustrate the plotf command, plot12.cf (Box 1) creates plots of two normal density functions, one grey-scaled, and adds two
straight lines at the means. The result is shown in Figure 1. Note that
when using grey scales, the effect depends on the ordering of commands
since a grey-scaled object can mask previous objects.
4.4.7
arrows
1
4.4.7 Arrows
The plot and plotp commands (see 4.4.1) can be modified such that
the resulting polygon ends in an arrow. The parameter is
a = xs,ys
with xs defining the length and ys the width of the arrow head (in mm),
respectively.
Command file plot13.cf, shown in Box 1, uses the plotp command
to illustrate several arrows with different line types and head sizes. The
plot is shown in Figure 1.
d040407.tex
April 19, 1998
4.4.7
2
arrows
Box 1 Command file plot13.cf
psfile = plot13.ps;
psetup(
pxlen = 90,
pylen = 50,
pxa
= 0,10,
pya
= 0,6,
);
plxa (sc=1);
plot
plya (sc=1);
plot
plframe;
plot
output file
x axis
y axis
frame
plotp(a=1,1)
plotp(a=1.5,1.5)
plotp(a=2,2)
plotp(a=2.5,2.5)
plotp(a=1,1,lt=5)
plotp(a=1.5,1.5,lt=5)
plotp(a=2,2,lt=5)
plotp(a=2.5,2.5,lt=5)
=
=
=
=
=
=
=
=
pltext
pltext
pltext
pltext
[1,1];
[1.5,1.5];
[2,2];
[2.5,2.5];
(xy=8.3,3)
(xy=7.3,5)
(xy=5.3,5.6)
(xy=1.6,5)
=
=
=
=
5,3,8,3;
5,3,7,5;
5,3,5,5.5;
5,3,3,5;
5,3,2,3;
5,3,3,1;
5,3,5,0.5;
5,3,7,1;
6
use
use
use
use
line
line
line
line
type
type
type
type
5
5
5
5
some labels
[2,2]
[2.5,2.5]
5
[1.5,1.5]
4
[1,1]
3
2
1
0
0
1
2
3
4
5
6
7
8
9
10
Figure 1 PostScript plot plot13.ps, created with command
file plot13.cf shown in Box 1.
4.4.8
circles, arcs, ellipses
1
4.4.8 Circles, Arcs, Ellipses
The ploto command can be used to plot circles and arcs; for an illustration, see Figure 1. The syntax is shown in the following box.
ploto (
xy=...,
lt=...,
lw=...,
gs=...,
nc=...,
) = r [,alpha,beta];
coordinates of center
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
grey scale value, def. 1 (white)
no clipping option, def. 0
The xy=x,y parameter must be used to define the center of the circle.
The radius must be given on the right-hand side. The angles, α and β,
to be given in degrees, are optional. If these angles are not given, the
result is a full circle, otherwise it is an arc, beginning at the angle α
and ending at the angle β, in counterclockwise direction; default values
are α = 0 and β = 360 to provide a full circle. Other parameters are
optional. lt and lw can be used to change the default line type and line
width. gs can be used to fill the circle with a grey tone. And nc=1 can
be used to plot outside the coordinate system.
d040408.tex
April 19, 1998
4.4.8
2
circles, arcs, ellipses
2
ploto (xy=2,1,gs=0.9) = 1;
plote (xy=7,1,gs=0.9) = 2,0.5;
0.5
1
1
2
0
0
1
2
3
4
5
6
7
8
9
10
Figure 1 PostScript plot plot15.ps, created with command
file plot15.cf (not shown), to illustrate the ploto and plote
commands.
4.4.8
3
circles, arcs, ellipses
2
plote(xy=4,1) = 3,0.3,45;
plote(xy=4,1,gs=0.9) = 3,0.3;
1
plote(xy=4,1) = 3,0.3,-45;
0
0
1
2
3
4
5
6
7
8
9
10
Figure 2 PostScript plot plot16.ps, created with command
file plot16.cf (not shown), to illustrate the ploto and plote
commands.
The plote command can be used to plot ellipses; for an illustration see
Figure 2. The syntax is shown in the following box.
plote (
xy=...,
lt=...,
lw=...,
gs=...,
nc=...,
) = a,b [,r];
coordinates of center
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
grey scale value, def. 1 (white)
no clipping option, def. 0
The xy=x,y parameter must be used to define the center of the ellipse. a
is the half length of the main axis (interpreted in user coordinates of the
x axis), b is the half length of the second axis of the ellipse (interpreted
in user coordinates of the y axis). r is optional; if provided the ellipse is
rotated by r degrees.
4.4.8
4
circles, arcs, ellipses
6
(c,d)
5
4
3
α
2
(a,b)
plotk(sc=45) = 6,2,6,4,7,5;
plotk(sc=-45) = 9,2,9,4,8,5;
plotk(a=1,1,sc=45) = a,b,c,d;
1
0
0
1
2
3
4
5
6
7
8
9
10
Figure 3 PostScript plot plot14.ps, created with command
file plot14.cf (not shown), to illustrate the ploto and plotk
commands.
To plot arcs which optionally end in an arrow, one can use the plotk
command; for an illustration see Figure 3. The syntax is shown in the
following box.
plotk (
sc=...,
lt=...,
lw=...,
a=...,
nc=...,
) = x1 , y1 , . . . , xn , yn ;
degree of curvature, def. 0
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
size of arrow head
no clipping option, def. 0
Given this command, the points (x1 , y1 ), . . . , (xn−1 , yn−1 ) are connected
by a polygon, and the points (xn−1 , yn−1 ) and (xn , yn ) are connected
by an arc. The arc is constructed so that its radius is defined implicitly
by the angle α to be given in degrees with the parameter sc=alpha; see
the example given in Figure 3. Note that α is restricted to the range
−180 < α < 180.
As an additional option one can use the a=xs,ys parameter. The
polygon then ends in an arrow with xs defining the length and ys the
width of the arrow head (in mm), respectively.
4.4.9
convex hull
1
4.4.9 Convex Hull
To plot the convex hull of a set of points, TDA uses an algorithm published by Eddy [1977]. The command is plotch with syntax shown in
the following box.
plotch (
lt=...,
lw=...,
s=...,
fs=...,
gs=...,
nc=...,
ns=...,
ic=...,
sel=...,
) = VX,VY;
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
marker symbol
size of marker symbol, def. 2 (mm)
grey scale value, def. 1 (white)
no clipping option, def. 0
number of intervals for smoothing
added to smoothed hull, def. 0
selection of data points
The right-hand side of the command must contain exactly two variable
names, to be used for the x and y coordinates of the data points. All
other parameters are optional.
1. By default, the command uses all cases (rows) of the currently defined
data matrix. To select a subset of cases one can use the sel parameter,
see the description of the plot command in 4.4.1.
2. The lt and lw parameters can be used to modify the default line
type and line width.
3. By default, only the convex hull polygon is plotted. To plot the data
points, one can provide the number of a marker symbol with the s parameter. The fs parameter can then be used to change the default size
of the symbols.
4. The gs parameter can be used to fill the convex hull with a grey tone.
The nc=1 (no clipping) parameter can be used to allow plotting outside
of the coordinate system.
5. Finally, one can request a smoothing of the convex hull with the ns
d040409.tex
April 19, 1998
4.4.9
convex hull
2
Box 1 Command file plot17.cf
nvar(
noc = 100,
X = rd,
Y = rd,
);
psfile = plot17.ps; output file
psetup(
pxlen = 90,
pylen = 50,
pxa = -0.5,1.5,
pya = -0.5,1.5,
);
plxa (sc=0.5);
plya (sc=0.5);
plframe;
plotch(gs=0.95,s=5,fs=0.7) = X,Y;
parameter. TDA uses then the Akima algorithm, described in 4.4.10, to
create a smoothed version of the convex hull polygon. (Note that the
resulting graph is no longer a convex hull in its strict definition.) The
parameter must be given as ns=n with n an integer greater than, or equal
to, 2. This specifies the number of subintervals used for smoothing. As
an additional option, one can use the ic=x parameter with some value
x > 0. Depending on the value of x, this results in some blow up of
the smoothed convex hull in such a way that all data points are strictly
inside its area.
Example 1 Command file plot17.cf, shown in Box 1, illustrates the
plotch command. In a first step, the command file creates 100 random
data points (using the rd operator). The plotch command is then used
to plot the convex hull of these data points. The area is grey-scaled
with value gs=0.9. A scatterplot of the data points is added with the s
parameter. The resulting plot is shown in Figure 1.
4.4.9
3
convex hull
1.5
1
0.5
0
-0.5
-0.5
0
0.5
1
1.5
Figure 1 PostScript plot plot17.ps, created with command
file plot17.cf shown in Box 1.
1.5
1
0.5
0
-0.5
-0.5
0
0.5
1
1.5
Figure 2 PostScript plot plot18.ps, created with command
file plot18.cf (not shown), to illustrate a smoothed convex
hull plot.
Example 2 To illustrate the smoothing option, we have changed the
plotch command into
plotch(ns=10,ic=2,gs=0.95,s=5,fs=0.7) = X,Y;
The new command file, plot18.cf, is not shown here, but the resulting
plot is shown in Figure 2. The blow up parameter is ic=2 (mm).
4.4.10
1
smoothing polygons
4.4.10 Smoothing Polygons
For smoothing polygons TDA uses an algorithm published by Akima
[1972]. There are two commands. The command plots, with syntax
shown in the following box,
plots (
ns=...,
lt=...,
lw=...,
s=...,
fs=...,
gs=...,
nc=...,
sc=...,
df=...,
fmt=...,
sel=...,
) = VX,VY;
degree of smoothing, def. 2
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
marker symbol
size of marker symbol, def. 2 (mm)
grey scale value, def. 1 (white)
no clipping option, def. 0
request closed curve (sc=1), def. sc=0
write data to output file
print format
selection of data points
can be used analogously to the plot command, see 4.4.1. The right-hand
side must provide the names of two variables containing the coordinates
of the data points.
1. The sel parameter can be used to select a subset of data points.
2. The degree of smoothing can be controlled with the ns parameter.
The syntax is ns=n with n an integer greater than, or equal to, 2; default
is ns=2.
3. The sc=1 parameter can be used to treat the polygon as a closed
curve. The first and last point of the polygon are then connected before
the smoothing algorithm is applied. These points are be marked by symbols if the s parameter has been used to specify the number of a marker
symbol; the fs parameter can then be used to change the default symbol
size.
4. The gs parameter can be used to fill the smoothed curve with a
grey tone. The effect depends on the sc parameter. By default, the area
d040410.tex
April 19, 1998
4.4.10
2
smoothing polygons
3
3
2
2
1
1
0
0
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
Figure 1 PostScript plots created with command files plot19.cf
and plot20.cf to illustrate smoothed polygons. In the right plot, the
parameter sc=1 has been used to request a closed curve.
between the smoothed curve and the x axis is grey-scaled. With the sc=1
parameter, the area inside the closed curve is grey-scaled.
5. The lt and lw parameters can be used to change the default line type
and line width. The nc=1 (no clipping) parameter allows a plot outside
the area of the coordinate system.
5. Finally, one can provide the name of an output file with the df parameter. Coordinates of the smoothed data points will then be written
into this file. The print format can be controlled with the fmt parameter.
Alternatively, one can use the command
plotsp (ns=,...) = x1,y1,x2,y2,...,xn,yn;
that allows to provide a list of data points directly on the right-hand
side of the command. All other parameters have the same meaning as
explained above for the plots command.
Example 1 An example, based on some arbitrarily chosen data points,
is shown in Figure 1. The command for the plot on the right side is
plotsp(ns=10,gs=0.9) = 1,1,2,2.5,3,2,4,2.5,5,2,6,1;
For the plot on the right side the parameter sc=1 was added to request
a closed curve.
4.4.11
contour plots
1
4.4.11 Contour Plots
To get information about a single-valued function with two arguments,
f (x, y), it is sometimes useful to plot its projection onto a two-dimensional plane. This is called a contour plot. To provide this option, TDA
uses an algorithm published by Snyder [1978]. The command to request
a contour plot is plotc with syntax shown in the following box.
plotc (
x=...,
nn=...,
lt=...,
lw=...,
gs=...,
) = function;
specification of levels
grid specification, def. nn=10,10
line type, def. 1 (solid line)
line width, def. 0.2 (mm)
grey scale value, def. 1 (white)
The right-hand side must contain the definition of a function depending
on (up to) two parameters, specified by the key words x1 (for the x axis)
and x2 (for the y axis). Depending on these arguments, the function
can be defined as any valid expression, which may contain numerical
constants, random numbers, and any of TDA’s type 1 operators. It must
be possible to evaluate the function over the whole range of the given
coordinate system.
The function is evaluated for the node points of a grid on the coordinate system. The grid is defined by integers nx and ny, the number
of intervals on the x and y axis, respectively. The parameter to provide
these numbers is nn=nx,ny; default is nn=10,10.
In addition, the command requires the specification of a sequence
of levels with the x=l1,l2,... parameter. For each of these levels the
algorithm tries to plot a contour line defined by the curve f (x, y) = li .
The other parameters are optional. lt and lw can be used to modify
the default line type and line width. The gs parameter can be used to
request a grey-scaled contour plot. The levels should then be ordered
from lowest to highest. Given the parameter gs=g, each level gets a
d040411.tex
April 19, 1998
4.4.11
contour plots
2
Box 1 Command file plot21.cf
psfile = plot21.ps;
psetup(
pxlen = 70,
pylen = 70,
pxa = -2,2,
pya = -2,2,
);
plxa (sc=1);
plya (sc=1);
plframe;
output file
plotc (x=0.01,0.05,0.1,0.125,0.15,nn=50,50,gs=0.9) =
exp(-((x1 + 1)^2 + (x2 + 1)^2) / 2) / (2 * pi) +
exp(-((x1 - 1)^2 + (x2 - 1)^2) / 2) / (2 * pi);
grey-scale value according to
1−i
greyscale value = g 1 +
n
i = 1, . . . , n
One should note that, to save programming overhead, only closed contour lines are filled with grey-scaled shades. However, a contour plot is
just another plot object, so it is possible to have many contour plots and
other plot objects in the same picture. In particular, the bounding box
of the coordinate system can be given a greyscale before adding contour
plots.
Example 1 To illustrate the contour plot command, our function is a
mixture of two two-dimensional normal densities with zero correlation,
defined symmetrically around zero:
2
X
1
1
2
2
f (x1 , x2 ) =
exp − (x1 − µi ) + (x2 − µi )
2π
2
i=1
The means are selected to be µ1 = −1 and µ2 = 1. Box 1 shows the
command file, plot21.cf, to create a contour plot for this function,
using five levels in ascending order. Figure 1 shows the resulting plot.
4.4.11
3
contour plots
2
1
0
-1
-2
-2
-1
0
1
2
Figure 1 PostScript plot plot21.ps, created
with command file plot21.cf shown in Box 1.
4.5
built-in previewer
1
4.5 Built-in Previewer
In order to preview PostScript plots, one most often will use a general
purpose preview utility, e.g., GhostScript or GhostView. For very simple
plots one can also use TDA’s built-in previewer. This option is available
when TDA runs in a Windows environment. It relies on a set of graphic
primitives for an X-Windows environment, and the previewer will therefore work properly only when TDA runs in an X-Windows environment of
a UNIX system. There are, however, some substitutes when TDA runs in
an MS-Windows environment. In any case, the previewer is not available
for a DOS version of TDA.
This section describes four commands that can be used to work “interactively” with TDA PostScript files. Some additional ready-made plot
commands will be described in 4.6.
1. The command
xopen = psfile;
requires that psfile is the name of a PostScript file that was created
by TDA. The command then makes this file TDA’s current PostScript
file and one can add and delete plot objects.
2. The command
xlog [= psfile];
shows the plot objects in TDA’s current PostScript file or, alternatively, in the file specified on the right-hand side. Each plot object
is shown by a number and a short description in TDA’s standard
output.
3. The command
xdelete = n1,n2,... ;
deletes the plot objects with numbers n1,n2,... from the current
PostScript file.
4. The command
xshow [= psfile];
d0405.tex
December 19, 1999
4.5
built-in previewer
2
invokes the built-in previewer to show a picture of the current PostScript file or, alternatively, the PostScript file specified on the righthand side, on the screen. If possible, the previewer opens a plot window and shows the plot in this window. Note that TDA remains
inactive as long as the plot window is open. Note also that, in an MS
Windows environment, one has to “click” on the window in order to
get the picture.
As an option, the command can also be used as
xshow (dscale=...) [= psfile];
where the dscale parameter provides a scaling factor for the plot.
The default scaling factor is 1.
Limitations. One should note that the built-in previewer has a number
of limitations. Each single plot object may only contain up to 1000 data
points; and the previewer can not show text and grey-scaled areas.
4.6
1
ready-made plot commands
4.6 Ready-made Plot Commands
There are a few ready-made plot commands that have the following
features:
• If there is no current open PostScript file, they create a new PostScript file and automatically create a coordinate system in such a
way that the new plot object roughly fits. A new PostScript plot file
is always named xplot.ps.
• If a current PostScript file exists, these commands (like all other plot
commands) add a plot object to this file.
The main usage of these commands is to allow for “quick and dirty” plots
that can be viewed with the built-in previewer (see 4.5). The following
pages describe the currently available commands.
d0406.tex
December 19, 1999
4.6
ready-made plot commands
2
Box 1 Syntax for xplot command
xplot (
opt=...,
s=...,
fs=...,
lt=...,
lw=...,
pxlen=...,
pylen=...,
) = VX,VY [,VG];
plot option, def. 1
1 : scatterplot
2 : line plot
symbol type, def. 1
symbol size, def. 1.3 mm
line type, def. 1
line width, def. 0.2 mm
length of x axis, def 120 mm
length of y axis, def 80 mm
Standard Line and Scatter Plots
By default, the xplot command creates a scatterplot based on the data
in the first two variables given on the right-hand side. As an option
(opt=2) the data points are connected by straight lines. As a further
option, one can add a third variable on the right-hand side. If added,
each set of data points having the same value of this variable is treated
as a separate group and is shown by a separate plot symbol. Note that
the s parameter can only be used when there is only one group.
4.6
ready-made plot commands
3
Box 2 Syntax for xplotf command
xplotf (
cn=...,
gn=...,
opt=...,
s=...,
fs=...,
lt=...,
lw=...,
pxlen=...,
pylen=...,
) = name of data file;
column numbers of variables, def. cn=1,2
column number of group variable
plot option, def. 1
1 : scatterplot
2 : line plot
symbol type, def. 1
symbol size, def. 1.3 mm
line type, def. 1
line width, def. 0.2 mm
length of x axis, def 120 mm
length of y axis, def 80 mm
Standard Line and Scatter Plots Based on a Data File
This command works similar to the xplot command but gets its data
directly from a free format data file that must be specified on the righthand side.
a) By default, the command uses the first two data columns in the input
file to get values for the X and Y variables.
b) Optionally, one can use the cn parameter to specify data column
numbers: cn=ci,cj,ck,... This specifies the data point groups:
(Xi,Xj), (Xi,Xk), ...
and a scatterplot, or line plot, is created separately for each group.
c) As a further option, one can specify
cn=ci,cj, gn=ck,
in order to define corresponding variables VX, VY, and VG. VX and VY
provide the data points, and VG is a grouping variable, meaning that
each set of (VX,VY) values which have the same value in VG is treated
as a separate group.
4.6
ready-made plot commands
4
Box 3 Syntax for xf command
xf (
rx=...,
lt=...,
lw=...,
pxlen=...,
pylen=...,
) = function(x);
definition of x axis, def. rx=1(1)10
line type, def. 1
line width, def. 0.2 mm
length of x axis, def 120 mm
length of y axis, def 80 mm
Functions
This command plots function(x). If there is a currently open PostScript
file, the function is plotted for the already defined x axis. Otherwise, the
command creates a new PostScript file. By default, the function is then
plotted for an x axis from 1 to 10. Alternatively, one can define the x
axis with the parameter
rx = a (d) b,
The range is then from a to b with increments d.
4.6
ready-made plot commands
5
Box 4 Syntax for xconh command
xconh (
lt=...,
line type, def. 1
lw=...,
line width, def. 0.2 mm
) = function(x);
Convex Hulls
This command adds a convex hull to the most recent scatterplot. If the
scatterplot consists of more than one group, the convex hulls are created
separately for each group.
4.6
ready-made plot commands
6
Box 5 Syntax for xreg command
xreg (
sig=...,
lt=...,
lw=...,
) [= n];
sigma for lowess, def. 0.5
line type, def. 1
line width, def. 0.2 mm
Regression Curves
This command adds a regression curve to the most recent scatterplot.
In case of more than one group of points this is done separately for each
group. One of there options can be selected on the right-hand side:
1 Lowess regression curve (default)
2 Least squares regression line
3 L1 norm regression line
4.7
3d coordinate systems
1
4.7 3D Coordinate Systems
This chapter describes how to define and use three-dimensional coordinate systems for PostScript plots.
4.7.1
Coordinates and Projections
4.7.2
The plot3 and plotp3 Commands
4.7.3
Plotting Text
4.7.4
Parametric Curves
4.7.5
Parametric Surfaces
4.7.6
Arbitrarily Distributed Points
d0407.tex
May 29, 2002
4.7.1
coordinates and projections
1
4.7.1 Coordinates and Projections
In order to create 3-dimensional PostScript plots it is necessary to define
a PostScript output file with the psfile command and to set up a
coordinate system and projection with the psetup3 command:
d040701.tex
May 29, 2002
4.7.2
1
the plot3 and plotp3 commands
4.7.2 The plot3 and plotp3 Commands
There are two commands to plot an arbitrary sequence of data points,
plot and plotp. The only difference is in the way the data points are
supplied. With the former one, the data points are taken from the program’s internal data matrix, with the latter one, the data points can be
explicitly supplied as part of the command.
d040702.tex
May 29, 2002
4.7.3
plotting text
1
4.7.3 Plotting Text
There are two commands to plot an arbitrary sequence of data points,
plot and plotp. The only difference is in the way the data points are
supplied. With the former one, the data points are taken from the program’s internal data matrix, with the latter one, the data points can be
explicitly supplied as part of the command.
d040703.tex
May 29, 2002
4.7.4
parametric curves
1
4.7.4 Parametric Curves
There are two commands to plot an arbitrary sequence of data points,
plot and plotp. The only difference is in the way the data points are
supplied. With the former one, the data points are taken from the program’s internal data matrix, with the latter one, the data points can be
explicitly supplied as part of the command.
d040704.tex
May 29, 2002
4.7.5
parametric surfaces
1
4.7.5 Parametric Surfaces
There are two commands to plot an arbitrary sequence of data points,
plot and plotp. The only difference is in the way the data points are
supplied. With the former one, the data points are taken from the program’s internal data matrix, with the latter one, the data points can be
explicitly supplied as part of the command.
d040705.tex
May 29, 2002
4.7.6
1
arbitrarily distributed points
4.7.6 Arbitrarily Distributed Points
There are two commands to plot an arbitrary sequence of data points,
plot and plotp. The only difference is in the way the data points are
supplied. With the former one, the data points are taken from the program’s internal data matrix, with the latter one, the data points can be
explicitly supplied as part of the command.
d040706.tex
May 29, 2002
5
mathematical procedures
1
5. Mathematical Procedures
This part contains the following sections.
5.1
Working with Matrices
5.2
Expressions
5.3
Functions
5.4
Numerical Integration
5.5
Smoothing Procedures
5.6
Function Minimization
d05.tex
April 20, 1998
5.1
working with matrices
1
5.1 Working with Matrices
This chapter explains how to use TDA for matrix calculations. It contains
the following sections.
5.1.1
Introduction and Overview
5.1.2
Matrices explains the basic concepts and how to create and
print matrices.
5.1.3
Matrix Expressions explains TDA’s concept of matrix expressions.
5.1.4
Matrix Commands explains all currently available matrix commands.
d0501.tex
December 20, 1999
5.1.1
1
introduction and overview
5.1.1 Introduction and Overview
Command
Section
mdef
mdefb
mdeff
mdefi
mdefc
mdefg
mfmt
mfree
mnvar
mpr
creates a matrix
creates a matrix from a block
reads a matrix from a file
creates an identity matrix
creates a matrix with constant values
creates a graph’s adjacency matrix
changes the print format
deletes matrices
creates internal data matrix
prints a matrix/expression
5.1.2
5.1.2
5.1.2
5.1.2
5.1.2
5.1.2
5.1.2
5.1.2
5.1.2
5.1.2
mpra
appends a matrix/expression
5.1.2
mexpr
mexpr1
msetv
evaluates a matrix expression
evaluates a matrix expression
changes elements of a matrix
5.1.3
5.1.3
5.1.3
mag
mcath
mcathv
mcatv
mcel
mcent
mchol
mcross
mcsum
mcvec
mdcent
mdcol
mdiag
mdiagd
mdrow
mev
mevs
mginv
minvd
minvs
matrix aggregation
horizontal concatenation
direct sum
vertical concatenation
edge list from adjacency matrix
centering
Cholesky decomposition
cross product
column sums
stacks columns
double centering
creates diagonal matrix
creates diagonal matrix
extracts diagonal
creates diagonal matrix
eigenvalues and eigenvectors
eigenvalues of symmetric matrix
generalized inverse
inverse of diagonal matrix
inverse of symmetric matrix
5.1.4.1
5.1.4.1
5.1.4.1
5.1.4.1
5.1.4.14
5.1.4.3
5.1.4.6
5.1.4.1
5.1.4.1
5.1.4.1
5.1.4.3
5.1.4.1
5.1.4.1
5.1.4.1
5.1.4.1
5.1.4.11
5.1.4.11
5.1.4.5
5.1.4.5
5.1.4.5
d050101.tex
December 23, 2002
5.1.1
Command
mkp
mlp
mlp1
mls
mlse
mlsei
mlsi
mmul
mncol
mnls
mnorm
mnorm1
mnorm2
mnrow
mpbl
mpbu
mpcol
mpfit
mpinv
mpit
mple
mproc
mprow
mpsym
mpz
mqap
mrank
mrsum
mrvec
mscal1
mscol
msort
msort1
msqrtd
msqrti
msrow
mstand
msvd
msvd1
mtrace
mtransp
mwvec
mwvec1
2
introduction and overview
Section
Kronecker product
linear programming
linear programming with constraints
least squares regression
linear equations
regression with constraints
linear inequalities
multiplication
extracts number of columns
regression with constraints
maximal norm
L1 norm
Euclidean norm
extracts number of rows
lower block diagonal matrix
upper block diagonal matrix
column permutation
iterative proportional fitting
inverse permutation
Leslie matrix iteration
Kaplan-Meier estimates
Procrustes rotation
row permutation
symmetrical permutation
zero-free main diagonal
quadratic assignment
ranking
row sums
stacking rows
scaling
selection of columns
sorting
sorting
square root of diagonal matrix
inverse square root of diagonal matrix
selection of rows
standardization
singular value decomposition
singular value decomposition
trace
transposition
construction of weights
construction of weights
5.1.4.1
5.1.4.13
5.1.4.13
5.1.4.7
5.1.4.8
5.1.4.10
5.1.4.9
5.1.4.1
5.1.3
5.1.4.10
5.1.4.4
5.1.4.4
5.1.4.4
5.1.3
5.1.4.14
5.1.4.14
5.1.4.1
5.1.4.14
5.1.4.1
5.1.4.14
5.1.4.14
5.1.4.14
5.1.4.1
5.1.4.1
5.1.4.14
5.1.4.14
5.1.4.2
5.1.4.1
5.1.4.1
5.1.4.3
5.1.4.1
5.1.4.2
5.1.4.2
5.1.4.1
5.1.4.1
5.1.4.1
5.1.4.3
5.1.4.12
5.1.4.12
5.1.4.1
5.1.4.1
5.1.4.14
5.1.4.14
5.1.2
matrices
1
5.1.2 Matrices
Matrix Names
Matrices are rectangular arrays of numerical entries. A general (m, n)
matrix X, that is, a matrix with m rows and n columns, is of the form


x11 x12 · · · x1n
 x21 x22 · · · x2n 


X =  .
..
.. 
 ..
.
. 
xm1
xm2
···
xmn
X is the name of the matrix. TDA uses the same conventions for matrix
and variable names (see 2.1). A matrix name is an arbitrary string of up
to 32 characters which may consist of the following letters:
A,. . . ,Z,a,. . . ,z,0,1,. . . ,9, ,@,$
The first character of a matrix name must be an upper case letter, or
one of the characters “ ”, “@”, “$”. Note that matrix names are case
sensitive, that is, lower and upper case letters are distinguished. There
is no special notation for vectors. Column vectors are treated as (m, 1)
matrices, row vectors as (1, n) matrices.
Storage of Matrices
Matrices are always stored in double precision (8 byte) arrays. To store
an (m, n) matrix needs (m · n + 1) · 8 bytes. There is an upper limit of
MaxMat matrices. When invoking the program, the default (MaxMatDef
= 200) can be changed with the maxmat command (see 1.4). Memory
to store matrices is only allocated when a new matrix becomes defined,
and if a matrix is deleted, the previously allocated memory will be given
back to the operating system and can be used then for other purposes.
Creating Matrices
There are several different options to create matrices.
d050102.tex
December 15, 2002
5.1.2
matrices
2
1. The command
mdef(X);
creates a (m, n)-matrix X with m equal to the number of rows in the
current internal data matrix and n equal to the current number of
variables. The command then copies the internal data matrix into X.
2. The command
mdef(X,m,n);
creates a (m, n)-matrix X and copies a maximum of m rows and n
columns (variables) from the internal data matrix into X.
3. The command
mdef(X) = varlist;
creates a (m, n)-matrix X with m equal to the number of rows in the
current internal data matrix and n equal to the number of variables
in varlist. The command then copies the values of the specified
variables into X.
4. The command
mdef(X,m,n) = varlist;
creates a (m, n)-matrix X and copies the values for maximal n variables from varlist and for maximal m rows in the current data
matrix into X. If m is greater then the number of cases (noc) or if n
is greater than the number of variables in varlist, the corresponding
rows or columns are filled with zeros.
5. The command
mdef(X,m,n) = x11 , . . . , x1,n , . . . , xm1 , . . . , xm,n ;
creates a (m, n)-matrix X and uses for its elements the values given
on the right-hand side. See Box 1 for an example.
6. The command
mdef(X,m,n) = file name;
5.1.2
matrices
3
creates a (m, n)-matrix X. The command then tries to open a data
file with the name given on the right-hand side and reads a maximum
of m data records from the file. From each record, the command uses
the first n numerical entries for the elements of X. It is assumed that
the file is organized as a free-format file.
7. The command
mdeff(X) = file name;
tries to open a free-format data file with the name specified on the
right-hand side. Given that the file contains m data records and n
numerical entries in each record, the command then creates a (m, n)matrix X and copies the numerical entries from the file into the elements of X.
8. Given a relational data structure, the command
mdefg(
gn=...,
sc=...,
) = matrix name;
creates a matrix containing the graph’s adjacency matrix. Except for
the name of a matrix to be given on the right-hand side all parameters
are optional. The gn parameter can be used to select a graph from
the current relational data structure. By default, the command uses
the first graph. The sc parameter can be used to substitute specific
values for missing edges.
Special Matrices
1. The command
mdefc(m,n,x,A);
creates a (m, n)-matrix A and sets all of its elements to the value x.
x must be provided as a scalar (matrix) expression.
2. The command
mdefi(m,n,A);
creates a (m, n)-matrix A and sets A(i, i) = 1, for i = 1, . . . , min(m, n).
5.1.2
matrices
4
Box 1 Illustration of mdef command
# command file: mat1.cf
mdef(I,3,3) = 1,0,0,
0,1,0,
0,0,1;
mpr(I);
Standard output
--------------mdef(I,3,3)=...
mpr(I)
1.0000
0.0000
0.0000
1.0000
0.0000
0.0000
0.0000
0.0000
1.0000
3. The command
mnum(x,d,m,A);
assumes that x, d, and m are scalar (matrix) expressions. The command creates a (m, 1)-matrix A and sets
A(i) = x + (i − 1)d
(i = 1, . . . , m)
x, d, and m can be (1,1) matrices.
Exchange with Internal Data Matrix
As has been described above, part or the whole of TDA’s internal data
matrix can be used to create matrices. On the other hand, one can also
use a matrix to create a new internal data matrix. The command is
mnvar(A);
where A is a (m, n) matrix. The command creates a new internal data
matrix with m rows and n variables. Variable names are created by using
the matrix name and adding column numbers. If a data matrix already
exists the command exits with an error message.
Saving Data Matrix Blocks in a Matrix
The comamnd
mdefb(B,expression);
5.1.2
matrices
5
can be used to save a block of data matrix rows in a matrix. The command requires that the dblock command has been used previously to
define data matrix blocks (see 1.9). B must be a valid matrix name, and
expression must be a scalar expression that evaluates to a valid block
number, say n. The mdefb command then creates a matrix named B and
copies block number n from the current data matrix into this matrix.
Block numbers can be written into a matrix using the dblock command,
see Section 1.9.
Information about Matrices
The command mdef, without parameters, can be used to get information
about the currently defined matrices. A table will be displayed containing, for each matrix, its name, its dimensions, and the command that
was used to define or create the matrix.
Printing Matrices
The command to print matrices has syntax
mpr(X [,string]);
where X is the name of a matrix. This matrix is then written into the
program’s standard output using the currently defined print format for
matrices. The print format can be changed with the command
mfmt = formatstring;
The default print format is mfmt=10.4. Alternatively, using the command
mpr(X [,string]) = name of an output file;
the matrix is written into the specified output file. Using mpra instead
of mpr appends to the output file.
Deleting Matrices
The command
mfree(X);
5.1.2
matrices
6
can be used to delete the matrix X from the list of currently defined matrices. The same command without an argument, simply mfree, deletes
all currently defined matrices. The memory used to hold the matrix data
is then deallocated and can be used for new matrices or other purposes.
5.1.3
1
matrix expressions
5.1.3 Matrix Expressions
Matrix expressions are a generalization of standard (scalar) expressions
that are described in 5.2. The extension is that at all places where a
standard expression may contain numerical constants, a matrix expression may contain the name of a matrix, a variable, or a namelist. The
basic idea can best be explained by referring to the mexpr command that
is used to evaluate matrix expression. The general syntax is
mexpr(op(A,...,V,...,L,...), R);
where A,... refers to matrix names, V refers to variable names, L refers
to namelist names, and op is used to refer to some (combination of) type
1 operators. Data matrix variables are treated as (m, 1) matrices with m
the number of rows in the current data matrix. Namelists are treated as
(m, n) matrices with m the number of rows in the current data matrix
and n the number of variables in the namelist. Consequently, each of
the referred objects, A,..., V,..., and L,..., can be considered as a
matrix with dimensions, say, row(i) and col(i), for the ith object in
the matrix expression. The resulting matrix R then gets the dimensions
max {row(i)} × max {col(i)}
If one of the objects (matrices, variables, namelists) has smaller dimensions, its missing rows and columns are created by cyclically using the
available ones. For example, if a matrix A has n columns, its (n + 1)th
column will equal its first column, and so on; analogously for rows.
We now give some additional hints on the usage of matrix expressions.
1. Names of matrices, variables and namelists must be different.
2. There are two type 1 operators which may help when using matrix
expressions. The operator
row(expression)
returns the row dimension of expression, and the operator
col(expression)
d050103.tex
December 15, 2002
5.1.3
matrix expressions
2
returns the column dimension of expression.1
3. The evaluation of matrix expressions is performed element-wise. For
example, given matrices
 
1
A =  2  , B = (1, 2)
3
the result of the matrix expression
mexpr(A * B, R);
would be

1
R= 2
3

2
4 
6
4. Matrix expressions may contain type 2 operators. Not allowed are
the pre and suc operators (but the lag operator will work), and
the operators for episode and sequence data. When a matrix expression contains type 2 operators, the resulting matrix is calculated
as a sequence of column vectors, meaning that the operators apply
separately for each of these column vectors. For example,
mexpr(sort(X),R);
would separately sort each column of X in ascending order.
5. In order to allow the block mode operators for matrix expressions,
there is an additional command,
mexpr1(B, expression, R);
where B is a (m, n) matrix and expression is some (r,c) matrix
expression. It is required that m ≥ r. The resulting matrix, R, has
dimension max(m, r) × c and is evaluated from expression. While
evaluating the expression, B is used for the definition of blocks, that
is, each consecutive number of identical rows in B is treated as a
separate block.
1 These
are operators; the corresponding commands are mnrow and mncol.
5.1.3
matrix expressions
3
6. A matrix expression may contain arguments
X(row,col) where X is a matrix, or
V(row,1)
where V is a data matrix variable, or
L(row,col) where L is a namelist.
The result is the corresponding element of X, V, or L. For example,
mexpr(A(row,col),R);
would create a scalar matrix, R, containing the value of A(row,col).
row and col can be any scalar expression.
7. Values resulting from row and col expressions must be positive integers. If they exceed the corresponding matrix dimensions, they are
cyclically turned around. For example, if the matrix is
A = (1, 2, 3),
then A(1,5) = 2.
8. In addition, one can use the following constructs (where X is a matrix,
a variable, or a namelist):
X(row,.)
X(.,col)
X(.,.)
sum of the elements in the row
sum of the elements in the column
sum of all elements
9. All constructs mentioned so far are considered as type 1 operators
and get the dimension (1, 1). There is a further type 2 operator,
X(i)
which returns the ith column of the matrix (or variable, or namelist)
X. The returned object has dimension row×1 where row is the number
of rows of object X.
10. Whenever the argument of a matrix command refers to an existing
matrix, it is possible to use an arbitrary matrix expression instead of
a single matrix name. So it is possible to combine scalars, matrices,
variables and namelists in all matrix commands. For example,
mpr(sin(V));
where V is a data matrix variable would print the resulting vector.
5.1.3
matrix expressions
4
11. As a further extension, one can directly refer to vectors with the
syntax
< e1,e2,... >
< e1,e2,... >0
for column vectors
for row vectors
where e1, e2, ... are any scalar expressions. For example,
mpr(<1,2,3>0 );
would be printed as the row vector (1, 2, 3) and
mpr(<1,2,3> + < 4,5,6 >0 );
would be printed as the matrix


5 6 7
 6 7 8 
7 8 9
If a vector has fewer elements as required by the dimension of its
context, it is cyclically updated. For example,
mpr(< 1,2 >0 + < 3,4,5 >0 );
would result in the vector (4, 6, 6).
12. Note that the prime character (’) is different from the single quotation
mark character (‘) which can be used to enclose strings. The prime
character can only be used with vectors, not with general matrices
or matrix expressions. Note also that due to stack size limitations, a
directly specified vector can have at most 9 elements. This may be
less when the vector is used as part of a more complex expression.
13. In addition to its standard form, one can use the mexpr command in
the form
mexpr(<rsel,csel,a>, expression, Matrix-name)
The optional string <rsel,csel,a> controls how the matrix expression is evaluated. rsel selects row indices, csel selects column
indices of the matrix expression. Only elements selected both by rsel
and csel are evaluated via expression, all other elements get a value
defined by a.
a can be a scalar constant or the name of an already existing matrix.
rsel and csel can be specified as follows:
5.1.3
matrix expressions
*
i
-i
(i1,i2,...)
-(i1,i2,...)
i(d)j
-(i(d)j)
selects
selects
selects
selects
selects
selects
selects
i, i+d,
5
all row (column) indices
row (column) index i
all rows (columns) except i
rows (columns) i1,i2,...
all rows (columns) expept i1,i2,...
rows (columns) i, i+d, ...
all rows (columns) except
i+2d,...
Changing the Elements of a Matrix
In order to change specific elements of a matrix one can use the command
msetv(expression, X(row,col));
where expression is a scalar expression, X is the name of an existing
matrix, and row and col are scalar expressions referring to an existing
element of X. This element is then changed into the value of expression.
5.1.4
1
matrix commands
5.1.4 Matrix Commands
This section describes the currently available matrix commands.
5.1.4.1
Elementary Matrix Commands
5.1.4.2
Sorting and Ranking
5.1.4.3
Standardization and Scaling
5.1.4.4
Matrix Norms
5.1.4.5
Inverse Matrices
5.1.4.6
Cholesky Decomposition
5.1.4.7
Least Squares Regression
5.1.4.8
Linear Equations
5.1.4.9
Linear Inequalities
5.1.4.10 Regression with Constraints
5.1.4.11 Eigenvalues and Eigenvectors
5.1.4.12 Singular Value Decomposition
5.1.4.13 Linear Programming
5.1.4.14 Some Further Commands
d050104.tex
December 20, 1999
5.1.4.1
elementary matrix commands
1
5.1.4.1 Elementary Matrix Commands
This section describes some elementary matrix commands. All the commands create new matrices and will result in an error if there is not
enough memory for the new matrices or if the maximum number of matrices is reached. The name of an existing matrix can be used to name a
result. In this case the previously existing matrix is overwritten. Matrix
commands do not echo unless silent = -1.
Number of Rows and Columns
1. Given a (m, n) matrix A, the command
mnrow(A,R);
creates a (1, 1) matrix R that contains m.
2. Given a (m, n) matrix A, the command
mncol(A,R);
creates a (1, 1) matrix R that contains n.
Multiplication, Transposition, Trace
1. The command
mmul(A1,...,Ak,R);
creates a matrix R that contains the product of the matrices A1,...,Ak.
2. The command
mtransp(A,R);
creates a matrix R that contains the transpose of the matrix A.
3. The command
mtrace(A,R);
creates a (1, 1)-matrix R that contains the trace of the matrix A.
d05010401.tex
December 23, 2002
5.1.4.1
elementary matrix commands
2
4. The command
mcross(A,R);
creates a matrix R that contains the cross product A’A.
Diagonal Matrices
1. The command
mdiag(X,D);
assumes that X is a (m, 1)-matrix (column vector). The commands
creates an (m, m)-matrix D and copies X into its main diagonal.
2. The command
mdrow(A,D);
assumes that A is a (m, n)-matrix. The commands creates an (m, m)
diagonal matrix D and sets
dii =
n
X
aij
j=1
3. The command
mdcol(A,D);
assumes that A is a (m, n)-matrix. The commands creates an (n, n)
diagonal matrix D and sets
dii =
m
X
aij
i=1
4. To extract the main diagonal from a matrix one can use the command
mdiagd(A,X);
It is assumed that A is a (m, n)-matrix. The commands creates a
(k, 1) column vector X, k = min(m, n), containing the main diagonal
elements from A.
5.1.4.1
elementary matrix commands
3
5. The command
msqrtd(A,R);
assumes that A is a (m, m) square matrix with non-negative entries
in its main diagonal. The command creates a (m, m) diagonal matrix
R and sets
√
rii = aii
6. The command
msqrti(A,R);
assumes that A is a (m, m) square matrix with non-negative entries
in its main diagonal. The command creates a (m, m) diagonal matrix
R and sets
√
rii = 1/ aii
Concatenation
1. The command
mcath(A1,...,Ak,R);
creates a matrix
R = [A1, . . . , Ak]
meaning that the matrices are horizontally concatenated. The number of rows of R will equal the number of rows of A1.
2. The command
mcatv(A1,...,Ak,R);
stacks the matrices A1,...Ak vertically into the new matrix R. The
number of columns of R will equal the number of columns of A1.
3. The command
mcathv(A1,...,Ak,R);
creates the direct sum of the matrices A1,...Ak:


A1


..
R = 

.
Ak
5.1.4.1
elementary matrix commands
4
Sums of Rows and Columns
1. Given a (m, n)-matrix A, the command
mrsum(A,R);
creates a (m, 1)-matrix R that contains row sums of A.
2. Given a (m, n)-matrix A, the command
mcsum(A,R);
creates a (1, n)-matrix R that contains the column sums of A.
Note that row sums can also be computed with the command mmul(A,1,R),
and column sums with mmul(1,A,R). Also, mmul(1,A,1,R) provides the
sum of the elements of A.
Selection of Rows and Columns
1. The command
msrow(A,D,R);
assumes that A is a (m, n) matrix and D is a row or column vector
providing indices for rows in A. Assuming that D contains k indices,
the command creates (k, n)-matrix R that contains the selected rows
from A.
2. The command
mscol(A,D,R);
assumes that A is a (m, n) matrix and D is a row or column vector providing indices for columns in A. Assuming that D contains
k indices, the command creates (m, k)-matrix R that contains the
selected columns from A.
Rows can also be specified by giving an index list in the form
msrow(A,< i1 , i2 , . . . >,R)
Any sequence of integers between 1 and m can be specified. If elements
in the list are separated by two consecutive commas the corresponding
range is used. The same applies to mscol.
5.1.4.1
elementary matrix commands
5
Trimming Matrices
The command
mtrim(A,ca,ra,cb,rb,R)
expects a (n, m) matrix A and scalar expressions ca, ra, cb, and rb. It
creates a new matrix, R, by deleting the first ca columns of A (if ca ≥ 0)
or adding ca zero columns (if ca < 0), and correspondingly treats the
first ra rows of A, the last cb columns of A, and the last rb rows of A.
Stacking Rows or Columns of a Matrix
Given an (n, m) matrix A, the command
mcvec(A,V)
creates a (n ∗ m, 1) vector V containing the stacked columns of A. Correspondingly, the command
mrvec(A,V)
creates a (n ∗ m, 1) vector V containing the stacked rows of A.
Kronecker Product
Given matrices A and B, the command
mkp(A,B,R);
creates R = A ⊗ B.
Permutations
The command
mprow(A,P,B)
expects an (n, m) matrix A = (aij ) and an (n, 1) or (1, n) permutation
vector P = (pi ) and creates an (n, m) matrix B = (bij ) with bi,j = api ,j .
Analogously, the command
mpcol(A,P,B)
5.1.4.1
elementary matrix commands
6
creates bi,j = ai,pj . The command
mpsym(A,P,B)
expects a quadratic matrix A and creates the symmetrical permutation
bi,j = api ,pj . Finally, given an (n, 1) or (1, n) permutation vector P =
(pi ), the command
mpinv(P,Q)
creates an output vector Q = (qi ) with qpi = i, that is, the inverse
permutation.
Matrix Aggregation
The command
mag(A,R,C,B)
expects an (n, m) matrix A, a (n, 1) vector R and a (1, m) vector C.
Elements of R correspond to the rows of A, elements of C correspond to
columns of A. If these elements are zero the corresponding rows and/or
columns of A are dropped. If two or more elements of R and C have the
same value, the corresponding elements of A are aggregated. The rows
and columns of the resulting matrix B are ordered according to the order
of the elements of R and C, respectively. The following examples illustrate
the command:
1 2 3
A = 5 6 7
1 2 1
1
R = 0
1
C = 1 2 3
B = 2 4 4
1 2 3
A = 5 6 7
1 2 1
1
R = 0
1
C = 1 4 4
B = 2 8
1 2 3
A = 5 6 7
1 2 1
1
R = 0
1
C = 5 4 4
B = 8 2
5.1.4.2
sorting and ranking
1
5.1.4.2 Sorting and Ranking
The following commands are available for sorting and ranking the rows
of a matrix.
1. The command
msort(A,D,R);
assumes that A is a (m, n)-matrix and D is a column vector providing
indices of columns of A. The command then sorts the rows of A in
ascending order, based on the values in the columns selected by D,
and returns the result in a new (m, n)-matrix R.
2. The command
msort1(A,D,R);
is almost identical with the msort command, but this command drops
rows of A which occur more than once. Therefore, R will have the same
number of columns but may have a smaller number of rows than A.
3. The command
mrank(A,D,R);
assumes that A is a (m, n)-matrix and D is a column vector providing
indices of columns of A. The command then sorts the rows of A in
ascending order, based on the values in the columns selected by D.
The command finally returns a new (m, 1)-vector R that contains
rank numbers for the sorted rows of A.
d05010402.tex
December 20, 1999
5.1.4.3
standardization and scaling
1
5.1.4.3 Standardization and Scaling
The following commands are available for centering, standardizing, and
scaling matrices.
1. Given a (m, n)-matrix A, the command
mcent(A,R);
creates a (m, n)-matrix R and sets its columns to the mean-centered
columns of A.
2. Given a (m, n)-matrix A, the command
mstand(A,R);
creates a (m, n)-matrix R and sets its columns to the standardized
columns of A, that is, columns with mean 0 and variance 1. If a
column has variance zero it is filled with zeros.
3. Given a symmetric (n, n)-matrix A, the command
mdcent(A,R);
creates a (n, n)-matrix R that contains the “double-centered” version
of A, that is:
1
rij = − (a2ij − a2i. − a2.j + a2.. )
2
4. Given a (m, n)-matrix A, the command
mscal1(A,R);
creates a (m, n)-matrix R with values
rij = aij /Σkl akl
d05010403.tex
December 15, 2002
5.1.4.4
matrix norms
1
5.1.4.4 Matrix Norms
1. Given a (m, n)-matrix A, the command
mnorm(A,R);
creates a (1, 1)-matrix R that contains the absolute value of that
element of A that has largest absolute value.
2. Given a (m, n)-matrix A, the command
mnorm1(A,R);
creates a (1, 1)-matrix R that contains the sum of the absolute values
of the elements of A.
3. Given a (m, n)-matrix A, the command
mnorm2(A,R);
creates a (1, 1)-matrix R that contains
q
Σij a2ij
d05010404.tex
December 20, 1999
5.1.4.5
1
inverse matrices
5.1.4.5 Inverse Matrices
Inversion of a Diagonal Matrix
Given a diagonal (m, n)-matrix A, the command
minvd(A,R);
returns a (m, n)-matrix R with non-zero elements
rii = 1/aii
(i = 1, . . . , min(m, n))
Inversion of a Symmetric Positive Definite Matrix
Given a symmetric positive definite (n, n)-matrix A, the command
minvs(A,R);
returns a (n, n)-matrix R that contains the inverse of A. Values are taken
only from the lower triangle of A, including its main diagonal.
Generalized Inverse Matrices
If A is a (m, n) matrix, there is a unique (n, m) matrix B with properties
ABA = A
BAB = B
(AB)0 = AB
(BA)0 = BA
B is then called the generalized inverse of A, or sometimes the pseudoinverse or Moore-Penrose inverse of A. See, e.g., Pringle and Rayner
[1971]. For more general definitions see Rao [1973, pp. 24–27]. If A is a
square regular matrix, then its generalized inverse is identical with its
standard inverse.
As shown by Golub and Reinsch [1971], calculation of the generalized inverse of A can be based on its singular value decomposition (see
5.1.4.12). If the singular value decomposition of A is given by
A = U QV 0
d05010405.tex
December 20, 1999
5.1.4.5
inverse matrices
2
Box 1 Example: generalized inverse
Command file: mat8.cf
mfmt = 13.10;
mdef(A,4,3) = 1,5,0, 2,6,0, 3,7,0, 4,8,0;
mpr(A);
mginv(A,B);
mpr(B);
mmul(A,B,C);
mmul(C,A,D);
mpr(D);
Output:
mdef(A,4,3)=...
mpr(A)
1.0000000000 5.0000000000
2.0000000000 6.0000000000
3.0000000000 7.0000000000
4.0000000000 8.0000000000
mginv(A,B)
Pseudorank of A: 2
mpr(B)
-0.5500000000 -0.2250000000
0.2500000000 0.1250000000
0.0000000000 0.0000000000
mmul(A,B,C)
mmul(C,A,D)
mpr(D)
1.0000000000 5.0000000000
2.0000000000 6.0000000000
3.0000000000 7.0000000000
4.0000000000 8.0000000000
0.0000000000
0.0000000000
0.0000000000
0.0000000000
0.1000000000 0.4250000000
0.0000000000 -0.1250000000
0.0000000000 0.0000000000
0.0000000000
0.0000000000
0.0000000000
0.0000000000
one can first define the generalized inverse of the diagonal matrix Q by
1/Qij if Qij 6= 0
Pij =
0 otherwise
In a second step, the generalized inverse of A can be calculated as
B = V PU0
TDA uses this method to calculate generalized inverse matrices. The singular value decomposition is done as described in 5.1.4.12. The command
is
mginv(A,B);
5.1.4.5
inverse matrices
3
A is assumed to be an already defined (m, n) matrix with m ≥ n. The
procedure creates a (n, m) matrix B and the generalized inverse of A is
returned in B. In addition, the rank of A is written in the standard output.
Note that the numerical determination of the rank of a matrix always
involves some inaccuracies due to limitations in representing floating
point numbers on a digital computer. So in general, one only gets the
pseudo-rank of a matrix.
Example 1 An illustration, using a (3, 3) matrix, is given in Box 1.
The result is shown in the same box.
5.1.4.6
cholesky decomposition
1
5.1.4.6 Cholesky Decomposition
An (n, n) square matrix A is called positive definite if
x0 Ax > 0
for all x 6= 0
If a matrix is symmetric and positive definite there is a unique decomposition
A = RR0
(1)
where R is a lower triangular matrix with positive diagonal elements.
This is called a Cholesky decomposition of the matrix A, see, e.g., Mohamed and Wait [1986, p. 10], Martin et al. [1971].
TDA provides the command
mchol(A,R);
to calculate the Cholesky decomposition of A. This must be a (n, n)
square matrix. TDA does not check whether the matrix is symmetric, it
simply uses its lower triangular part to perform the decomposition. The
result is then copied into the lower triangular part of the (n, n) matrix
R. If A is not positive definite, there will be an error message and R = 0.
d05010406.tex
December 20, 1999
5.1.4.7
1
least squares regression
5.1.4.7 Least Squares Regression
In matrix notation, a linear regression model can be written as
Y = Xβ + (1)
with an (m, 1) vector Y (dependent variable), an (m, n) matrix X (independent variables), an (n, 1) vector β (regression coefficients to be
estimated), and an (m, 1) vector (residuals). A least squares solution
of (1) is a vector β̂ that minimizes the Euclidean norm of the residuals:
k k = k Y − Xβ k
(2)
Here we use k x k to denote the Euclidean norm of a vector x, that is,
the square root of Σx2i . In the context of linear regression, it is normally
assumed that m > n and X has full column rank (rank(X) = n). There
is, then, a unique least squares solution
β̂ = (X 0 X)−1 X 0 Y
(3)
If rank(X) < n, there are (in general) many possible solutions. To calculate a least squares solution for (1), TDA uses an algorithm developed
by Hanson and Haskell [1982]. (This algorithm extends earlier work of
Lawson and Hanson [1974].) One should note that this algorithm does
not require that X has full column rank. If rank(X) < n, the algorithm
tries to find one of the possible solutions. The command is
mls(S,B);
with S an (m, n + 1) matrix containing [X, Y ]. If the input data are
not already available in such a matrix, S can easily be created by using
the mcath command; see 5.1.4.1. The contents of S is destroyed by the
command. The command creates an (n, 1) vector B for the least squares
solution, β, of (1). In addition, the rank of X and the Euclidean norm
of the residuals (2) are shown in the standard output.
d05010407.tex
December 20, 1999
5.1.4.7
least squares regression
2
Box 1 Example of least squares regression
Command file: mat2.cf
mfmt = -16.12;
mdef(A,4,3) = 1,2,3, 4,7,11, -1,1,0, 5,6,11;
mpr(A);
mls(A,B);
mpr(B);
Output:
mdef(A,4,3)=...
mpr(A)
1.000000000000e+00
2.000000000000e+00
4.000000000000e+00
7.000000000000e+00
-1.000000000000e+00
1.000000000000e+00
5.000000000000e+00
6.000000000000e+00
mls(A,B)
Rank of left-hand side: 2
Norm of residuals:
4.440892098501e-16
mpr(B)
1.000000000000e+00
1.000000000000e+00
3.000000000000e+00
1.100000000000e+01
0.000000000000e+00
1.100000000000e+01
Example 1 Box 1 illustrates the mls command with a simple example.
The command file is mat2.cf.
5.1.4.8
linear equations
1
5.1.4.8 Linear Equations
A general system of linear equations can be written as
Ax = b
(1)
where A is an (m, n) left-hand side matrix, b is an (m, 1) right-hand side
matrix (vector), and x is an (n, 1) matrix (vector) to be determined as
the solution of the system. The command
mlse(S,X);
can be used to find a solution of the system (1). S should be a (m, n + 1)
matrix containing [A, b]. The command creates an (n, 1) vector X and,
if a solution is found, it is copied into this matrix. In addition, one gets
information about the rank of the left-hand side matrix, A, and the
Euclidean norm of the residuals:
k Ax − b k
To solve system (1), TDA uses an algorithm developed by Hanson and
Haskell [1982]. The algorithm is quite general and can cope also with
under-determined and over-determined systems. In general, there are
three possible situations.
1. A is a square matrix (m = n) and has full rank. Then there is a
unique solution x that solves the system (1). An example (taken
from Lawson and Hanson [1974, p. 14]) is shown in Box 1.
2. If m ≤ n and rank(A) < n, there may be a unique solution, or a set
of different solutions, or the system may be contradictory without a
solution. If there is more than one solution, TDA will provide just
one of the possible solutions. If the equations are contradictory, TDA
calculates a least squares solution.
3. If m > n and rank(A) ≥ n, the algorithm tries to find a least squares
solution of the system, that is, a vector x that minimizes the Euclidean norm of the residuals.
d05010408.tex
December 15, 2002
5.1.4.8
linear equations
Box 1 Example: full rank system of linear equations
Command file: mat3.cf (Lawson, Hanson 1974, p. 14)
mfmt = 12.8;
mdef(A,3,4) = -0.4744,-0.4993,-0.7250,-0.8615,
0.5840,-0.7947, 0.1652, 0.0,
-0.6587,-0.3450, 0.6687, 0.0;
mpr(A);
mlse(A,B);
mpr(B);
Output:
mdef(A,3,4)=...
mpr(A)
-0.47440000 -0.49930000 -0.72500000
0.58400000 -0.79470000
0.16520000
-0.65870000 -0.34500000
0.66870000
mlse(A,B)
Rank of left-hand side: 3
Norm of residuals:
0.00000000
mpr(B)
0.40872542
0.43019126
0.62456022
-0.86150000
0.00000000
0.00000000
2
5.1.4.9
1
linear inequalities
5.1.4.9 Linear Inequalities
A general system of linear inequalities can be written as
Ax ≥ b
(1)
where A is an (m, n) left-hand side matrix, b is an (m, 1) right-hand side
matrix (vector), and x is an (n, 1) matrix (vector) to be determined as
the solution of the system. In general, this system of inequalities may
have a unique solution, many solutions, or no solution if the inequalities
are incompatible. To solve the system, TDA uses again an algorithm
developed by Hanson and Haskell [1982]. This algorithm always tries
to find (at least) one solution with a minimal Euclidean norm of the
residuals. The command is
mlsi(S,X);
with S an (m, n + 1) matrix containing [A, b]. The command creates an
(n, 1) matrix X and, if a solution is found, it is copied into this matrix. The
contents of the matrix S is destroyed by the procedure. Box 1 illustrates
the mlsi command with a simple example.
d05010409.tex
December 20, 1999
5.1.4.9
linear inequalities
Box 1 Example: linear inequalities
Command file: mat4.cf
mfmt = 12.8;
mdef(A,4,3) = 1,2,3,5,6,11,-1,-2,-3,0,1,1;
mpr(A);
mlsi(A,B);
mpr(B);
Output:
mdef(A,4,3)=...
mpr(A)
1.00000000
2.00000000
5.00000000
6.00000000
-1.00000000 -2.00000000
0.00000000
1.00000000
mlsi(A,B)
mpr(B)
1.00000000
1.00000000
3.00000000
11.00000000
-3.00000000
1.00000000
2
5.1.4.10
regression with constraints
1
5.1.4.10 Regression with Constraints
As described in 6.9.1, the lsreg command can solve least squares problems with additional equality and inequality constraints. To allow for
these calculations with the help of matrix commands, we added the following commands.
1. The command
mlsei(W,me,mi,X);
expects a (m, n + 1) matrix W and two scalar expressions, me and mi,
in the following form:


E F
W= A B 
H G
where
E is a (me, n) matrix
F is a (me, 1) vector
A is a (ma, n) matrix
B is a (ma, 1) vector
H is a (mi, n) matrix
G is a (mi, 1) vector
and ma = m − me − mi. Note that m is defined by the number of
rows in W, and me and mi are given by the scalar expressions in the
command. It is possible to have me ≥ 0, ma ≥ 0, and mi ≥ 0. The
command tries to find (not always successfully) an (n, 1) vector X in
such a way that
EX = F
(equality constraints)
HX ≥ G
(inequality constraints)
and the Euclidean norm of (AX − B) is minimal in the least squares
sense. The algorithm is the same as used for TDA’s lsreg command.
d05010410.tex
December 22, 1999
5.1.4.10
regression with constraints
2
2. The command
mnls(W,me,k,X);
expects a (m, n + 1) matrix W and two scalar expressions, me and k,
in the following form:
E F
W=
A B
where
E is a (me, n) matrix
F is a (me, 1) vector
A is a (ma, n) matrix
B is a (ma, 1) vector
and ma = m − me. Note that m is defined by the number of rows in W,
and me ≥ 0 is given by the scalar expressions in the command. The
command tries to find (not always successfully) an (n, 1) vector X in
such a way that
EX = F
(equality constraints)
and the Euclidean norm of (AX − B) is minimal in the least squares
sense. In addition, k is a scalar expression that satisfies 0 ≤ k ≤ n and
is used to specify non-negativity constraints for the solution vector,
X, in such a way that
X(j) ≥ 0
for j = k, . . . , n
The algorithm is the same as used for TDA’s lsreg command.
5.1.4.11
1
eigenvalues and eigenvectors
5.1.4.11 Eigenvalues and Eigenvectors
Let A be a square (n, n) matrix. An (n, 1) vector x 6= 0 is called an
eigenvector of A, if there is a scalar λ so that
Ax = λx
(1)
λ is then called an eigenvalue (or latent root) of A, and x is the corresponding eigenvector (or latent vector ). These concepts play a fundamental role in linear algebra and also have statistical applications. Most
linear algebra textbooks, and many statistical textbooks, introduce the
basic concepts and theorems. Here we only note a few points.
1. Eigenvectors defined by (5.1.4.11) are right eigenvectors. Correspondingly, one can define left eigenvectors as non-trivial solutions of
yA = λy
Left eigenvectors have dimension (1, m) (row vectors). By transposing (1) one sees that the transposed right eigenvectors of A are the
left eigenvectors of A0 .
2. For every (n, n) matrix A one can define a characteristic equation
| A − λIn | = 0
(2)
where | | denotes the determinant of a matrix. The eigenvalues of A
are the roots of this characteristic equation.
3. Every nonzero (n, n) matrix possesses at least one nonzero eigenvector and a corresponding eigenvalue. Or put otherwise, one can say
that every (n, n) matrix has n eigenvalues; but these eigenvalues are
not necessarily distinct, there can be multiple eigenvalues (roots of
the characteristic equation). For example, let A be the (2, 2) matrix
1 1
A =
(3)
0 1
The characteristic equation is
1−λ
1
| A − λI2 | = 0
1−λ
= (1 − λ)2
Obviously, there is only a single root, 1.
d05010411.tex
December 22, 2002
5.1.4.11
eigenvalues and eigenvectors
2
4. A general (n, n) matrix A can have real or complex eigenvalues and
eigenvectors. Complex eigenvalues and eigenvectors always occur in
conjugate complex pairs.1 If λ = λ(r) + iλ(i) is a complex eigenvalue
of A, and x = x(r) + ix(i) the associated complex eigenvector, then
A(x(r) + ix(i) )
= (λ(r) + iλ(i) )(x(r) + ix(i) )
= (λ(r) x(r) − λ(i) x(i) ) + i(λ(r) x(i) + λ(i) x(r) )
It follows that
λ∗ x∗
= (λ(r) x(r) − λ(i) x(i) ) − i(λ(r) x(i) + λ(i) x(r) )
= Ax(r) − iAx(i)
= Ax∗
That is, if λ is a complex eigenvalue of A with complex eigenvector
x, then also λ∗ is an eigenvalue of A with associated eigenvector x∗ .
5. Eigenvectors are not unique. As seen from (1), if x is an eigenvector of A, then also αx is an eigenvector where α is any nonzero
scalar. Therefore, when calculating eigenvalues, one has to adopt a
convention for scaling. The TDA algorithm (described below) uses
the following convention: If an eigenvector is real, it is scaled to have
the Euclidean length 1, that is
kxk=
n
X
!1/2
x2i
= 1
(4)
i=1
If an eigenvector is complex, say x = x(r) ± ix(i) , it is scaled so that
o
n
(r)
max xi
= 1
1≤i≤n
and the imaginary component, corresponding to the maximum real
part component, gets the value zero.
6. If the (n, n) matrix A is symmetric (A = A0 ), then all of its eigenvalues and eigenvectors are real. If, in addition, the matrix is positive
definite (semi-definite), all eigenvalues are positive (not negative).
z = a + ib is a complex number, then the complex conjugate of z, denoted z ∗ , is
defined as z ∗ = a − ib.
1 If
5.1.4.11
eigenvalues and eigenvectors
3
Box 1 Example: eigenvectors and eigenvalues
Command file: mat5.cf
mfmt = 13.10;
mdef(A,4,4) = 6,-3,4,1, 4,2,4,0, 4,-2,3,1, 4,2,3,1;
mpr(A);
mev(A,ER,EI,EVR,EVI);
mpr(ER);
mpr(EI);
mpr(EVR);
mpr(EVI);
Output:
mdef(A,4,4)=6,-3,4,1,4,2,4,0,4,-2,3,1,4,2,3,1
mpr(A)
6.0000000000 -3.0000000000 4.0000000000 1.0000000000
4.0000000000 2.0000000000 4.0000000000 0.0000000000
4.0000000000 -2.0000000000 3.0000000000 1.0000000000
4.0000000000 2.0000000000 3.0000000000 1.0000000000
mev(A,ER,EI,EVR,EVI)
mpr(ER)
5.2360679292
5.2360680258
0.7639320225
0.7639320225
mpr(EI)
0.0000000000
0.0000000000
0.0000000207
-0.0000000207
mpr(EVR)
-0.2627643879 -0.2627643970 -0.3726779962 -0.3726779962
-0.6152998190 -0.6152998181 0.1273220038 0.1273220038
-0.2350236149 -0.2350236199 0.3333333333 0.3333333333
-0.7050708544 -0.7050708501 1.0000000000 1.0000000000
mpr(EVI)
0.0000000000 0.0000000000 0.0000000122 -0.0000000122
0.0000000000 0.0000000000 0.0000000060 -0.0000000060
0.0000000000 0.0000000000 -0.0000000134 0.0000000134
0.0000000000 0.0000000000 -0.0000000000 0.0000000000
General Matrices
TDA offers two algorithms to calculate eigenvalues and eigenvectors. One
algorithm is restricted to symmetric matrices and will be described be-
5.1.4.11
eigenvalues and eigenvectors
4
Box 2 Example: eigenvalues of symmetric matrix
Command file: mat6.cf
silent=-1;
mfmt = 13.10;
mdef(A,3,3) = 1,1,1, 1,2,0, 1,0,10;
mpr(A);
mevs(A,E,V);
mpr(E);
mpr(V);
calculate eigenvalues and vectors
print eigenvalues
print eigenvectors
mmul(A,V,B);
mpr(B);
mscol(V,1,V1);
mexpr(E(1,1)*V1,B1);
mpr(B1);
mscol(V,2,V2);
mexpr(E(2,1)*V2,B2);
mpr(B2);
mscol(V,3,V3);
mexpr(E(3,1)*V3,B3);
mpr(B3);
check results
low. The other algorithm is adopted from Grad and Brebner [1963] and
can be applied to arbitrary real quadratic matrices. The command is
mev(A,ER,EI,EVR,EVI);
The command expects a (n, n) matrix A and creates four matrices ER,
EI, EVR, and EVI. ER and EI are both (n, 1) vectors containing the real
and imaginary parts of the eigenvalues, respectively. EVR and EVI are
(n, n) matrices and will contain the real and imaginary parts of the
right eigenvectors, respectively. Note that the eigenvectors are scaled as
described above.
Example 1 To illustrate the mev command, we use an example given
in Eberlein and Boothroyd [1971, p. 334]. The data and results are shown
in Box
√ 1. The matrix A is defective and has two pairs of multiple roots:
3 ± 5. The right eigenvectors are:
√
√
(± 5, 3 ± 5, 2, 6 )0
The output of the mev command in Box 1 shows the scaled eigenvectors.
As a result of rounding errors, the imaginary parts of the second pair
5.1.4.11
eigenvalues and eigenvectors
5
of eigenvalues and eigenvectors are not exactly zero. These eigenvectors
are therefore scaled according to the complex case mentioned above.
Symmetric Matrices
A quadratic (n, n) matrix A is called symmetric if A = A0 , that is, if
A is equal to its transpose. As already mentioned, symmetric matrices
only have real eigenvalues and eigenvectors and, therefore, the calculation is easier than for general matrices. To calculate eigenvalues and
eigenvectors of a symmetric matrix, TDA offers the command
mevs(A,E,V);
It is expected that A is an (n, n) symmetric matrix. The procedure creates
an (n, 1) matrix (vector) E and an (n, n) matrix V. The eigenvalues of A
are stored in E (in descending order). The eigenvectors of A are stored
in the columns of V. The eigenvector corresponding to the ith eigenvalue
(ith component of E) is stored in the i column of V. Eigenvectors are
scaled to have the Euclidean length 1.
The algorithm to calculate eigenvalues and eigenvectors is adopted
from Sparks and Todd [1974]. Note that the algorithm does not check
whether the input matrix, A, is symmetric; only its lower triangular part
is used.
Example 2 To illustrate the mevs command, Box 2 shows a simple
example, supplemented with commands to check the result. The output
is shown in Box 3.
5.1.4.11
eigenvalues and eigenvectors
Box 3 Example (continued): eigenvalues of symmetric matrix
Output of mat6.cf
mdef(A,3,3)=1,1,1,1,2,0,1,0,10
mpr(A)
1.0000000000 1.0000000000 1.0000000000
1.0000000000 2.0000000000 0.0000000000
1.0000000000 0.0000000000 10.0000000000
mevs(A,E,V)
mpr(E)
10.1112597744
2.5823542359
0.3063859897
mpr(V)
0.1105672008 -0.5020854553 -0.8577208693
0.0136313229 -0.8621650266 0.5064441272
0.9937751663 0.0676879796 0.0884830847
mmul(A,V,B)
mpr(B)
1.1179736900 -1.2965625022 -0.2627936574
0.1378298465 -2.2264155084 0.1551673851
10.0483188637 0.1747943409 0.0271099775
mscol(V,1,V1)
mexpr(E(1,1)*V1,B1)
mpr(B1)
1.1179736900
0.1378298465
10.0483188637
mscol(V,2,V2)
mexpr(E(2,1)*V2,B2)
mpr(B2)
-1.2965625022
-2.2264155084
0.1747943409
mscol(V,3,V3)
mexpr(E(3,1)*V3,B3)
mpr(B3)
-0.2627936574
0.1551673851
0.0271099775
6
5.1.4.12
singular value decomposition
1
5.1.4.12 Singular Value Decomposition
If A is an (m, n) matrix with m ≥ n, there is a decomposition
A = U QV 0
(1)
with the following matrices (In is used to denote the (n, n) unit matrix):
an orthogonal (m, n) matrix U , i.e.,
U 0 U = In
An orthogonal (n, n) matrix V , i.e.,
V 0 V = V V 0 = In
And a (n, n) diagonal matrix
Q = diag {q1 , . . . , qn }
(1) is called the singular value decomposition of the matrix A; and the
entries q1 , . . . , qn are called the singular values of A. See, e.g., Golub and
Reinsch [1971]. Basic properties are:
1. The number of nonzero singular values equals the rank of A.
2. q12 , . . . , qn2 are the eigenvalues of the matrix A0 A.
3. The columns of V are the eigenvectors of A0 A, that is:
(A0 A)V = (QQ)V
The singular value decomposition is a basic tool for linear algebra problems but also has many statistical applications.
d05010412.tex
December 20, 1999
5.1.4.12
singular value decomposition
2
Box 1 Example: singular value decomposition
Command file: mat7.cf
mfmt = 13.10;
mdef(A,3,3) = 1,0,0, 5,7,0, 1,0,1;
mpr(A);
mcross(A,AA);
mpr(AA);
create cross product
msvd1(AA,Q,U,V);
mpr(Q);
mpr(U);
mpr(V);
singular value decomposition
mevs(AA,E,V);
mpr(E);
mpr(V);
calculate eigenvalues and eigenvectors
TDA provides two commands for singular value decomposition. Both
assume a (m, n) matrix, A, with m ≥ n. The command
msvd(A,Q);
performs a singular value decomposition of A and saves the singular values in the (n, 1) matrix Q. The command
msvd1(A,Q,U,V);
does the same, but additionally creates the (m, n) matrix U and the (n, n)
matrix V as described above. Calculation is based on an algorithm by
Golub and Reinsch [1971].
Example 1 To illustrate the msvd1 command, Box 1 shows a simple
example. To check the result, the mevs command is used to calculate
eigenvalues and eigenvectors directly. The output is shown in Box 2.
5.1.4.12
singular value decomposition
Box 2 Example (continued): singular value decomposition
Output of mat7.cf
mdef(A,3,3)=...
mpr(A)
1.0000000000 0.0000000000
5.0000000000 7.0000000000
1.0000000000 0.0000000000
mcross(A,AA)
mpr(AA)
27.0000000000 35.0000000000
35.0000000000 49.0000000000
1.0000000000 0.0000000000
msvd1(AA,Q,U,V)
mpr(Q)
74.6926238609
1.9752562176
0.3321199215
mpr(U)
-0.5917332901 0.6195362372
-0.8060938137 -0.4611140127
-0.0080297492 0.6352548449
mpr(V)
-0.5917332901 0.6195362372
-0.8060938137 -0.4611140127
-0.0080297492 0.6352548449
mevs(AA,E,V)
mpr(E)
0.3321199215
1.9752562176
74.6926238609
mpr(V)
-0.5157776305 -0.6195362372
0.3709267188 0.4611140127
0.7722608401 -0.6352548449
0.0000000000
0.0000000000
1.0000000000
1.0000000000
0.0000000000
1.0000000000
0.5157776305
-0.3709267188
-0.7722608401
0.5157776305
-0.3709267188
-0.7722608401
-0.5917332901
-0.8060938137
-0.0080297492
3
5.1.4.13
1
linear programming
5.1.4.13 Linear Programming
A linear programming problem can be described as follows:
c0 x = z → max
Ax ≤ b
x ≥ 0
(1)
The first line defines the objective function to be maximized; c is an
(n, 1) vector with given values, x is an (n, 1) vector containing variables.
Explicitly written, the objective function is
z(x1 , . . . , xn ) =
n
X
ci xi
i=1
The second and third lines in (1) define constraints. The second line
assumes a given (m, n) matrix A and an (m, 1) vector b resulting in m
inequalities to be satisfied by x. The third line requires that feasible
solutions for x are restricted to be nonnegative.
(1) is called the primal problem. Associated with the primal problem
is a dual problem defined as
b0 y
A0 y
y
= z → min
≥ c
≥ 0
(2)
The variables are now given in an (m, 1) vector y. The objective function,
b0 y, is given the same value, z, because it can be shown that, if the primal
problem has a (finite) solution, then also the dual problem has a solution,
and the values of the primal and dual objective functions are the same.
The formulation in (1) assumes that there is only a specific type
of inequality constraints: Ax ≤ b. However, if in an original problem
formulation, some constraints are formulated with an ≥ relation, that
equation can simply be multiplied by −1 to fit into the formulation
given in (1). It would also be possible to use two inequality constraints
to define an equality to hold exactly, but equality constraints can also
defined directly.
d05010413.tex
December 20, 1999
5.1.4.13
linear programming
2
The algorithm is adopted from Salazar and Sen [1968]. The TDA command can be given in two different forms. A first command is
mlp(T,X,Y);
T is expected to be a (m + 1, n + 1) matrix containing the data for the
linear programming problem in the following form:
0
c 0
T =
A b
with A, b, and c as defined above. If possible, an optimal solution of the
primal and dual problems is calculated. Alternatively, the command can
be given as
mlp1(T,p,X,Y);
The last p rows of Ax ≤ b are then interpreted as equality constraints.
5.1.4.13
linear programming
3
Box 1 Example: linear programming
Command file: lp1.cf
mdef(T,3,3) = 1, 0, 0,
1, 0, 1,
0, 1, 1;
mpr(T);
mlp(T,X,Y,1);
mpr(X);
mpr(Y);
Output:
mdef(T,3,3)=...
mpr(T)
1.0000
1.0000
0.0000
mlp(T,X,Y,1)
Value is:
0.0000
0.0000
1.0000
0.0000
1.0000
1.0000
1.0000
mpr(X)
1.0000
1.0000
mpr(Y)
1.0000
0.0000
Example 1 A trivial example is shown in Box 1. The problem is to
maximize x1 , s.t. x1 , x2 ≥ 0, x1 ≤ 1, and x2 = 1.
5.1.4.14
1
some further commands
5.1.4.14 Some Further Commands
1. Given (m, 1)-vectors T and C, the command
mple(T,C,F,D);
performs a product-limit estimation of the cumulative distribution
function of the values in T. The elements in C are interpreted as censoring information: if C(i) = 0, T(i) is interpreted as not censored;
if C(i) has a non-zero value, the corresponding T(i) is interpreted
as censored. The largest value of T is always interpreted as being not
censored.
The command creates two (m, 1)-vectors, F and D. If the command
terminated successfully, F will contain the cumulative distribution
function, and D will contain the jumps of the function for uncensored
values.
2. Given (m, 1)-vectors A and W, the command
mwvec(A,W,R);
creates a (m, 1)-vector R with elements
P
j∈I(i) aj wj
where I(i) = {k | i < k ≤ m}
ri = P
j∈I(i) wj
If I(i) is empty, or the sum of weights is zero, then ri = ai .
3. Given (m, 1)-vectors A, W, and T, the command
mwvec1(A,W,T,R);
creates a (m, 1)-vector R with elements
P
j∈I(i) aj wj
ri = P
where I(i) = {k | T (k) > T (i)}
j∈I(i) wj
If I(i) is empty, or the sum of weights is zero, then ri = ai .
d05010414.tex
December 22, 2002
5.1.4.14
some further commands
2
4. Given a (n, n) matrix A, the command
mpz(A,B,P);
tries to find a permutation of the rows of A that puts as many nonzero elements in the main diagonal as possible. The permuted matrix
is returned in B. Also returned is an (n, 1) vector P that contains the
permutation. (The algorithm is adapted from ACM algorithm 575
(Permutations for a zero-free diagonal), developed by I.S. Duff.)
5. Given a (n, n) matrix A, the commands
mpbl(A,B,P,N,U)
mpbu(A,B,P,N,U)
try to find a simultaneous permutation of rows and columns to make
A a lower (mpbl) or upper (mpbu) block diagonal matrix. The permuted matrix is returned in B. Also returned is an (n, 1) vector P
that contains the permutation, an (1, 1) scalar N that contains the
number of blocks, and a (k, 1) vector U with k the number of blocks.
ui,1 is the number of the row where the ith block begins. (The algorithm is adapted from ACM algorithm 529 (Permutations to block
triangular form), developed by I.S. Duff and J.K. Reid.)
6. The command
mpit(F,N,k,R)
can be used to iterate a Leslie matrix. It is assumed that F is a (n, 2)
matrix, N is a (n, 1) vector, and k is an integer. The first column
of F contains the first row of the (n, n) Leslie matrix L, and the
second column of F contains the subdiagonal of L (the last element is
ignored). The vector N contains the starting values for the iteration.
The command creates the (k + 1, n) matrix R. The ith row of R
contains Li N.
7. The command mproc (Procrustes rotation) is described in Section
7.4.4.1.
8. The parameters mplog, mppar, mpcov, and mpgrad (giving results for
function maximization) are described in Section 5.6.1.
5.1.4.14
some further commands
3
9. The following commands are only documented in the help file tda.hlp:
a) mbrr (results of balanced replications)
b) mcel (edge list from adjacency matrix)
c) mpfit (iterated proportional fitting)
d) mqap (quadratic assignment)
5.2
expressions
1
5.2 Expressions
This chapter explains TDA’s concept of expressions. It contains the following sections.
5.2.1
Introduction
5.2.2
Evaluating Expressions
5.2.3
Numerical Constants
5.2.4
Random Numbers
5.2.5
Type 1 Operators
5.2.6
Type 2 Operators
5.2.7
Block Mode Operators
5.2.8
Operators for Episode Data
5.2.9
Operators for Sequence Data
d0502.tex
December 18, 1999
5.2.1
1
introduction
5.2.1 Introduction
Many TDA commands require expressions, that is, strings which can be
evaluated numerically; for example:
1.2
a single floating point number
max (1,7)
maximum of two numbers
exp (expression) exponential function of another expression
In TDA, expressions serve mainly three purposes:
1. To define variables. In general, a variable is defined by
VName = expression;
VName is the name of the variable (to be defined), and expression describes how to assign numerical values to the variable (for additional
options see 2.1). Remember that a variable is actually a column vector with a number of components equal to the number of rows in the
data matrix. Consequently, expression is evaluated for each row in
the data matrix and the resulting numerical value is the value of the
variable for that row.
2. To define functions. This will be explained in 5.3.
3. To specify case select statements. For instance, the syntax of the
isel (input select) command is
isel = expression;
meaning that expression is evaluated for each record of the input
data file. The record is then selected (for the data matrix) if the
result is not equal to zero, and otherwise the record is not selected.
In general, expressions are build by using numerical constants and random numbers, references to data file entries or already defined variables,
and by using operators. Several operators will be introduced in subsequent sections. There are operators without any arguments and operators
that have one or more arguments. For instance, rd is an operator without
arguments and provides the next random number, equally distributed in
d050201.tex
April 20, 1998
5.2.1
introduction
2
the 0–1 interval, from the connected random number generator. An example of an operator with two arguments is rd (a,b); the arguments
a and b are real numbers with a < b. This operator provides the next
random number that is equally distributed in the interval [a,b].
In general, most operators are identified by a key word, for instance
rd, that must be given in lower case letters. If an operator has one
or more arguments, these arguments follow in brackets, separated by
commas:
op (arg1,arg2,...)
Some operators have a somewhat different syntax. For instance, the simple arithmetical operators (+, -, . . . ) are operators with two arguments.
They are, however, not written as +(x,y), but more conveniently as x
+ y. Another exception are the operators to create dummy variables.
Most operators have none or a fixed number of arguments. Some
operators have a variable number of arguments, in particular:
min(a1 , a2 , . . . ) minimum of a1 , a2 , . . .
max(a1 , a2 , . . . ) maximum of a1 , a2 , . . .
An important distinction is based on the amount of information required
for the evaluation of operators. In TDA, the basic distinction is between
type 1 and type 2 operators.
1. Type 1 operators can be evaluated using only information from the
current data file record or data matrix row. Therefore, these operators
can be evaluated while sequentially reading an input data file and creating the rows of the data matrix.
2. Type 2 operators are operators that require information about the values of their arguments for all cases of the data matrix (if in record mode)
or for all records in a block (if in block mode). For instance, mean (V), the
mean of a variable V, requires all values of V in the current data matrix
(or data block) for its evaluation. There are, therefore, some restrictions
in using these operators, see 5.2.6.
5.2.2
1
evaluating expressions
5.2.2 Evaluating Expressions
Expressions are evaluated in two steps. In a first parsing step, TDA
creates an algorithm to evaluate the expression. This is actually a stack
that symbolically references all operators and arguments contained in
the expression. In a second step the algorithmic representation of the
expression can be evaluated for any number of different arguments.
1. Understanding details of the first step is actually not necessary when
using expressions. However, TDA offers the command
parse = expression;
to get some information about the result of the parsing procedure.
2. A more interesting command is
mpr(expression);
This command evaluated expression and prints the result into the standard output. Optionally, the command can also be used as
mpr(expression,string);
where string is an arbitrary string. The command then first prints the
string, in a new line, prints the value of expression. As a further option,
the command can be used as
mpr(expression [,string]) = file name;
The result is then written, not to the standard output, but to the file
specified on the right-hand side. By default, a new file is created by the
mpr command. Using the command as mpra, data can be appended to
an already existing file.
d050202.tex
December 19, 1999
5.2.3
1
numerical constants
5.2.3 Numerical Constants
Numerical constants can be used to create expressions. They can be
given in the following formats:
Integer
Floating point F Format
Floating point E Format
for example: 0, 367, -133
for example: 11.87, -1.1, -.12
for example: 1.3E12, -1E-123, 1.e+2
Numerical constants must not begin with a + sign. For instance the
expression +3 will result in a syntax error because + is interpreted as an
operator. One should also note that, if numerical constants are used to
define variables, the possible accuracy depends on the storage size of the
variable (see 2.1). However, if a variable is defined as a single numerical
constant, the storage size is always 8 for a double precision value.
Some numerical constants have predefined names.
1. The operator pi (without arguments) provides the numerical constant
π = 3.14159 . . .
d050203.tex
April 20, 1998
5.2.4
random numbers
1
5.2.4 Random Numbers
The following operators can be used to generate random numbers.
1. The operator rd provides the next random number, equally distributed in the 0–1 interval. The algorithm is adopted from Pike
and Hill [1965], modified for 32 bit machines.
2. The operator rd(a,b) provides the next random number, equally
distributed in the [a, b] interval. The algorithm is the same as used
for the rd operator.
3. The operator grd(V) provides a random number that is equally distributed in the 0–1 interval; a new random number is only generated
in the first row of the data matrix or if the variable V changes its
value. The algorithm is the same as used for the rd operator. Note
that this operator is classified as a type 1 operator but has some
restrictions. In particular, this operator should not directly be used
as part of an if-then-else construction.
4. The operator rdn provides standard normally distributed random
numbers. The algorithm is adopted from Bell [1968].
5. The operator rdn1 provides an alternative method for standard normally distributed random numbers, based on an algorithm described
by Brent [1974].
6. The operator rdmn(i, Σ) creates random numbers that follow a multivariate normal distribution with correlation matrix Σ. The operator
requires that a complete symmetric and positive definite correlation
matrix is defined with the mdef command, see 5.1.2. Given that an
(n, n) correlation matrix Σ has been defined, rdmn(i, Σ) returns the
ith component of
(1 , . . . , n ) ∼ N (0, Σ)
It is required that n ≥ 2. A new random vector is only created when
i = 1, otherwise the operators returns components of a previously
generated random vector.
The algorithm follows Tong [1990, p. 185]. Its input consists of standard normally distributed random numbers that are created with the
d050204.tex
December 19, 1999
5.2.4
random numbers
2
algorithm of Brent [1974].
7. The operator rdp1(z) returns random numbers according to a Poisson distribution with mean z. The range is 0 < z < 350. The algorithm is adapted from Schaffer [1970].
5.2.5
type i operators
1
5.2.5 Type 1 Operators
Type 1 operators can be evaluated by using only information from the
current data file record or data matrix row. The available type 1 operators are described in the following subsections.
5.2.5.1
General Operators
5.2.5.2
Arithmetical Operators
5.2.5.3
Mathematical Operators
5.2.5.4
Logical Operators
5.2.5.5
Dummy Variables
5.2.5.6
Density Functions
5.2.5.7
Distribution Functions
5.2.5.8
Julian Calendar
5.2.5.9
String Variables
d050205.tex
December 19, 1999
5.2.5.1
1
general type i operators
5.2.5.1 General Type 1 Operators
1. The operator case returns the current case number, that is, the
sequence number of the data matrix row. For example,
CASE [6.0] = case;
defines a variable, CASE, with print format [6.0], that will contain
the current case number.
2. The operator nocdm returns the number of cases in the data matrix,
or 0 if there is no data matrix.
3. The operator noc returns the number of currently selected cases in
the data matrix, or 0 if there is no data matrix.
4. The operator nvar returns the number of currently defined variables.
5. The operator bnoc returns the number of blocks that have been found
by the latest use of the dblock command.
6. The operator
num(x,d)
where x and d are (floating point) expressions provides the sequence
of numbers:
x + id for i = 0, 1, 2, . . .
7. The operator
exists(Name)
requires that Name is a syntactically valid matrix, variable, or namelist
name. The operator returns 1 if the object denoted by Name exists,
and otherwise returns 0. In particular,
exists()
returns 0. This is useful in order to check whether $n expressions in
macro definitions are actually available when the macro is executed.
d05020501.tex
December 19, 1999
5.2.5.1
general type i operators
2
For example, exists($1) when used inside a macro, will be expanded
to exists(Name) when Name is given for $1 when invoking the macro,
and is expanded into exists() when the macro argument $1 is not
used.
Remember that $n expressions are substituted by their corresponding
names, when given by the user, or substituted by Null-strings. This
rule also applies when $n expressions are used inside double quotation
marks. The only exception is when $n expressions are inside single
quotation marks. For example,
print("$1 ...")
will substitute $1 by a valid name, or by the Null-string; but
print("... ‘$1’ ...")
will not substitute $1 by a name or the Null-string.
8. The operator
row(expression)
returns the row dimension of expression, and the operator
col(expression)
returns the column dimension of expression (see
5.2.5.2
1
arithmetical operators
5.2.5.2 Arithmetical Operators
The following arithmetical operators are available, all classified as type 1
operators. In describing these operators, x and y are numerical (integer
or floating point) expressions.
x+ y
x− y
x∗ y
x / y
x ∧ y
x%y
x&y
x|y
addition
subtraction
multiplication
division (y must not be zero)
power of (y must not be negative)
modulus operator (x and y must be integer)
logical and; result is 1 if both x and y have nonzero value,
otherwise the result is zero.
logical or; result is 1 if x or y has a nonzero value, otherwise
the result is zero.
When an expression contains more than one of these operators, evaluation follows the usual priority rules: (+, −, |), (∗, /, &), (∧ , %) in ascending order. Brackets may be used to control the order of evaluation.
d05020502.tex
April 20, 1998
5.2.5.3
1
mathematical operators
5.2.5.3 Mathematical Operators
The following mathematical operators are available. All arguments are
numerical (floating point) expressions.
1. rnd (x) provides the integer value of x; rounded to the nearest integer. This is identical with
sign(x) * floor(abs(x) + 0.5)
2. abs(x) evaluates the absolute value of x.
3. sign(x) provides the sign of x, i.e., +1 if x is greater than zero, -1
if x is less than zero; and otherwise 0.
4. floor(x) provides the largest integer not greater than x.
5. ceil(x) provides the smallest integer not less than x.
6. tr(x, a, b) truncates x to the interval [a, b], meaning that the result
is equal to a if x < a, equal to b if x > b, and other is equal to x.
7. min(x1 , x2 , . . .) gives the minimum of its arguments. The operator
has a variable number of arguments.
8. max(x1 , x2 , . . .) gives the maximum of its arguments. The operator
has a variable number of arguments.
9. sqrt(x) gives the square root of x. It is required that x ≥ 0.
10. exp(x) evaluates the exponential function (antilogarithm) of x.
11. eexp(x) evaluates exp(x)/(1 + exp(x)).
12. log(x) evaluates the natural logarithm of x. It is required that x > 0.
13. sin(x) evaluates the sine of x.
14. cos(x) evaluates the cosine of x.
15. lgam(x) provides the natural logarithm of the gamma function, evaluated at x. It is required that x > 0. The algorithm is adopted from
Pike and Hill [1966].
d05020503.tex
April 20, 1998
5.2.5.3
mathematical operators
2
16. digam(x) provides the first derivative of the logarithm of the gamma
function, evaluated at x. It is reqired that x > 0. The algorithm is
adopted from Bernado [1976].
17. trigam(x) provides second derivative of the logarithm of the gamma
function, evaluated at x. It is required that x > 0. The algorithm is
adopted from Schneider [1978].
18. icg(x, q) evaluates the incomplete gamma integral, that is,
Z x
1
tq−1 e−t dt, x, q > 0
icg(x, q) =
Γ(q) 0
The algorithm is adopted from Moore [1982]. Note that there is another operator, icg1, that also evaluates the incomplete gamma integral, based on an algorithm by Lau [1980]. However, while icg can
be used with automatic differentiation, this is not possible with icg1.
19. icb(x, a, b) provides the incomplete beta integral, evaluated for x
with coefficients given by a and b. It is required that 0 < x < 1. The
algorithm is adopted from Baker ([1992], p. 580).
20. bc(n, m) returns the binomial coefficient, n over m. n and m must
be integers. It is required that 0 ≤ m ≤ n. The operator can easily
lead to overflows. One should then consider to use the lgam operator
for the logarithm of the gamma function.
5.2.5.4
logical operators
1
5.2.5.4 Logical Operators
Logical operators to build expressions may have one, two, or three arguments. The arguments are any expressions, the result is a logical expression, that is, only two values are distinguished: a logical expression
is true if it evaluates to a nonzero value, otherwise the logical expression
is false. The following logical operators are available:
1. not (E) returns the logical negation of expression E, that is, if E is
true, not (E) returns 0 (false); if E is false, not (E) returns 1 (true).
2. and (E1,E2) returns the logical AND of expressions E1 and E2, that
is, if both E1 and E2 are true, and (E1,E2) returns 1 (true), otherwise
it returns 0 (false). Note that it is often more convenient to write E1
& E2 instead of and (E1,E2).
3. or (E1,E2) returns the logical OR of expressions E1 and E2, that is,
if E1 or E2 is true, or (E1,E2) returns 1 (true), otherwise it returns 0
(false). Note that it is often more convenient to write E1 | E2 instead
of or (E1,E2).
4. eq (E1,E2) returns 1 (true) if expressions E1 and E2 have equal value,
otherwise it returns 0 (false).
5. ne (E1,E2) returns 1 (true) if expressions E1 and E2 do not have
equal value, otherwise it returns 0 (false).
6. lt (E1,E2) returns 1 (true) if the value of expressions E1 is less than
the value of expression E2, otherwise it returns 0 (false).
7. le (E1,E2) returns 1 (true) if the value of expressions E1 is less than,
or equal to, the value of expression E2, otherwise it returns 0 (false).
8. gt (E1,E2) returns 1 (true) if the value of expressions E1 is greater
than the value of expression E2, otherwise it returns 0 (false).
9. ge (E1,E2) returns 1 (true) if the value of expressions E1 is greater
than, or equal to, the value of expression E2, otherwise it returns 0
(false).
10. if (E,E1,E2) returns E1 if the logical expression E is true (not equal
to zero), and returns E2 otherwise. This operator, like all others, can
d05020504.tex
April 20, 1998
5.2.5.4
logical operators
2
be used recursively, for example,
if (E, E1, if (E2, E3, E4))
When using the if operator for the definition of variables, one can
also use another, sometimes more convenient, syntax:
if E then E1 else E2
is then equivalent to
if(E,E1,E2)
or more general:
if E then E1 else if EE then EE1 else if ... else E2
5.2.5.5
dummy variables
1
5.2.5.5 Dummy Variables
There are two operators to ease creation of dummy variables with cornered or with centered effects. In describing these operators, E is an
expression and J1,J2,..., and K1,K2,..., are integer expressions.
1. The result of
E[J1,J2,...]
is a cornered effect dummy variable taking the value 1 if E has any one
of the integer values J1,J2,...; otherwise the result is 0. A comma can
be substituted by two commas, meaning then the range between two
integer values. For instance, E[J1,,J2] would result in the value 1 if the
expression E takes any one of the values in the range from J1 to J2.
2. The result of
E[J1,J2,...:K1,K2,...]
is a centered dummy variable, taking the value 1 if E is equal to one of
the integer expressions J1,J2,..., taking the value -1 if E is equal to
one of the integer expressions K1,K2,...; and is otherwise 0.
The following box provides some examples.
X X[1] X[1,7] X[1,,3] X[1,3,,7] X[1:2,3]
----------------------------------------------1
1
1
1
1
1
7
0
1
0
1
0
2
0
0
1
0
-1
3
0
0
1
1
-1
1
1
1
1
1
1
In order to give sensible results, E should be an integer expression. In
fact, the expression E is always truncated (not rounded) to an integer
expression before making comparisons with the arguments J1,J2,...,
and K1,K2,... If E is not a simple expression, it should be enclosed in
brackets, for instance:
(E1 * (E2 + E3))[...]
d05020505.tex
April 20, 1998
5.2.5.5
dummy variables
2
Dummy variables can also be created with the ndvar command, see 2.5.
5.2.5.6
1
density functions
5.2.5.6 Density Functions
The following operators are available to calculate density functions. In
describing these operators, x represents a numerical (floating point) expression.
1. ndf(x) returns the value of φ(x), the standard normal density function. The operator can be used for automatic differentiation.
2. mr(x) returns the reciprocal value of Mill’s ratio, that is,
mr(x) = φ(x)/(1 − Φ(x))
where φ is the density and Φ the distribution function of the standard
normal distribution. The algorithm is adopted from Swan [1969].
3. poisson(θ, k) returns the logarithm of the Poisson distribution, i.e.,
θk
log
= k log(θ) − log(Γ(k + 1)) − θ
k! exp(θ)
Automatic differentiation is possible only wrt θ.
4. negbin(α, γ, k) returns the logarithm of the negative binomial distribution, i.e.,
k α !
(Γ(α + k)
γ
α
log
Γ(k + 1) Γ(α) α + γ
α+γ
The operator allows for automatic differentiation with respect to its
first two arguments, α and γ, see 6.14.2.
d05020506.tex
April 20, 1998
5.2.5.7
1
distribution functions
5.2.5.7 Distribution Functions
The following operators evaluate distribution functions. In describing
these operators, x represents a floating-point numerical expression.
1. nd(x) return the value of Φ(x), i.e., the standard normal distribution
function. The algorithm is adopted from Hill [1973]. The operator can
be used for automatic differentiation.
2. ndi(x) returns the value of Φ−1 (x), i.e., the inverse standard normal distribution function. The algorithm is adopted from Hill and
Davis [1971]. (This is a replacement for the algorithm from Beasley
and Springer [1977] that was used in earlier versions of TDA.) The
operator supports automatic differentiation for its first derivative.
3. td(x, n) returns the value of the t-distribution, evaluated at x, with
degress of freedom given by the integer expression n. The algorithm
is adopted from Levine [1969].
4. chd(x, n) returns the value of the χ2 -distribution, evaluated at x,
with degrees of freedom given by the integer expression n. The algorithm is adopted from Hill and Pike [1967].
5. fd(x, n, m) returns the value of the F-distribution, evaluated at x,
with degrees of freedom given by the integer expressions n and m.
The algorithm is adopted from Dorrer [1968].
6. bivn(x, y, ρ) returns the value of the bivariate normal distribution
with correlation ρ, that is,
Z x Z y
bivn(x, y, ρ) = Φ2 (x, y, ρ) =
φ2 (u, v, ρ) dvdu
−∞
−∞
where
1 x2 + y 2 − 2ρxy
p
exp −
φ2 (x, y, ρ) =
2
1 − ρ2
2π 1 − ρ2
1
The algorithm is adopted from Donnelly [1971]. (Values can be compared with tables in Tong [1990].)
d05020507.tex
April 20, 1998
5.2.5.7
2
distribution functions
The operator allows for automatic differentiation. The derivatives are
calculated as follows (see Tong [1980, p 9] for derivatives wrt ρ).
y − ρx ∂Φ2 (x, y, ρ)
= φ(x) Φ p
∂x
1 − ρ2
∂Φ2 (x, y, ρ)
∂ρ
∂ 2 Φ2 (x, y, ρ)
∂x∂x
= φ2 (x, y, ρ)
= −φ(x)
(
y − ρx y − ρx p
φ p
+ xΦ p
1 − ρ2
1 − ρ2
1 − ρ2
x − ρy 1
p
φ(y) φ p
1 − ρ2
1 − ρ2
∂ 2 Φ2 (x, y, ρ)
∂x∂y
=
∂ 2 Φ2 (x, y, ρ)
∂x∂ρ
= −
∂ 2 Φ2 (x, y, ρ)
∂ρ∂ρ
ρ
)
x − ρy
φ2 (x, y, ρ)
1 − ρ2
(
!)
1
x − ρy y − ρx
p
p
= φ2 (x, y, ρ)
+ρ
1 − ρ2
1 − ρ2 1 − ρ2
φ and Φ denote, respectively, the density and distribution function
of the standard normal distribution.
7. mvn(Σ, k, , x1 , . . . , xn ) returns the value of an n-dimensional normal
distribution with correlation matrix Σ, that is
Z xn
Z x1
1
1
0 −1
x
Σ
x
dx1 · · · dxn
·
·
·
exp
−
2
(2π)n/2 |Σ|1/2 −∞
−∞
where x = (x1 , . . . , xn )0 . The number of dimensions is restricted to
2 ≤ n ≤ 20. The operator expects as its first argument the name
of an (n, n) correlation matrix that must be defined with the mdef
command, see 5.1.2. Its upper triangle is used for the correlation
matrix.
The algorithm is adopted from Drezner [1992] and uses an adaptive Gaussian quadrature method for numerical integration. There
are two options, depending on k (integer) and (floating point). If
2 ≤ k ≤ 10, is ignored and the algorithm performs one Gaussian quadrature with k points for each dimension of the integral. If
12 ≤ k ≤ 20, the algorithm performs progressive quadratures with
5.2.5.7
distribution functions
3
number of points m = 2, . . . , k − 10, until the difference between two
successive evaluations of the integral does not exceed . To check the
operator one can use TDA’s calc command. For tables with values
of the multivariate normal integral see Tong [1990].
The operator cannot be used for automatic differentiation. If derivatives are required they must be approximated numerically. For applications, see 6.12.5.
5.2.5.8
julian calendar
1
5.2.5.8 Julian Calendar
Given variables Y (year, four digits), M (month), and D (day), the
operator
jul (Y, M, D)
returns the corresponding day in the Julian calendar. On the other hand,
given a day J in the Julian calendar, then
july(J)
julm(J)
juld(J)
returns the corresponding year
returns the corresponding month
returns the corresponding day
Consequently,
july (jul(Y, M, D)) = Y
julm (jul(Y, M, D)) = M
juld (jul(Y, M, D)) = D
d05020508.tex
April 20, 1998
5.2.5.9
1
string variables
5.2.5.9 String Variables
The following type 1 operators can be used for string variables.
1. Given a string variable S the operator
strlen(S)
returns the length of the string variable, i.e., the number of columns
that are used for each of the strings.
2. Given a string variable S the operator
strv(S)
tries to convert the strings in S into numerical values. If this is not
possible because a string does not consist only of digits, the operator
returns -1.
3. Given a string variable S the operator
strvp(S,n,m)
tries to convert the sub-strings from column n until, and including,
column m into numerical values. If this is not possible because a substring does not consist only of digits, the operator returns -1.
d05020508.tex
December 19, 1999
5.2.6
1
type ii operators
5.2.6 Type 2 Operators
Type 2 operators are specific in that they require information not only
about the current data matrix row. The actual amount of information
depends on the mode of data generation, see 2.2. In record mode (default), type 2 operators are evaluated for all rows in the current data
matrix; in block mode, they are evaluated for all rows in a block. This
implies some restrictions in using these operators to define expressions.
Note that a variable is assigned type 4 (see 2.1) if its definition contains type 2 operators, or if its definition refers to at least one other
variable that, in turn, involves type 2 operators. Here are some examples of type 2 operators.
X vmin(X) vmax(X) sum(X) cum(X) cd(X) cdv(X) mean(X) std(X)
------------------------------------------------------------1
1
3
7
1
0.50 0.2857 1.75
0.9574
1
1
3
7
2
0.50 0.2857 1.75
0.9574
2
1
3
7
4
0.75 0.5714 1.75
0.9574
3
1
3
7
7
1.00 1.0000 1.75
0.9574
Currently available type 2 operators will be described in the following
subsections.
5.2.6.1
General Type 2 Operators
5.2.6.2
Recode Operators
5.2.6.3
Leads and Lags
5.2.6.4
Sorting and Ranking
5.2.6.5
Counting Subsets
5.2.6.6
Counting Changes
5.2.6.7
Aggregates
5.2.6.8
String Variables
Note that type 2 operators cannot be used for automatic differentiation.
d050206.tex
December 19, 1999
5.2.6.1
1
general type ii operators
5.2.6.1 General Type 2 Operators
In describing the following general type 2 operators, X denotes the name
of a variable that must be available in the currently defined data matrix.
1. vmin (X) provides the minimum of the values of the variable X.
2. vmax (X) provides the maximum of the values of the variable X.
3. sum (X) provides the sum of the values of the variable X.
4. cum (X) provides the cumulated values of the variable X.
5. cd (X) provides the empirical distribution function of X.
6. cdv (X) provides the incomplete first moment of the distribution
of variable X, calculated as
P
x ≤x xi
cdv(x) = Pni
i=1 xi
7. mean (X) provides the mean value of the variable X.
8. std (X) provides the standard deviation of the variable X.
9. ndv (X) calculates the number of different values occurring in X.
10. ndv1 (X, A) calculates the number of different values occurring in
variable X that are not less than A (any expression).
11. ndv2 (X, A, B) calculates the number of different values occurring
in variable X that are not less than A and not greater than B.
12. mav (X, n) calculates a moving average for variable X:
mav(X, n) =
n
X
1
Xi+j
2n + 1 j=−n
n must be a positive integer.
d05020601.tex
May 29, 2002
5.2.6.1
general type ii operators
2
13. The operator bfa (X[. . .]) returns the number of times that the
dummy variable X[. . .] (i.e., the argument of the operator) takes a
non-zero value in the current block (or the data matrix, if not in
block mode).
14. The operator bfr (X[. . .]) returns the relative frequency of the
occurrence of non-zero values of the dummy variable X[. . .] in the
current block (or the data matrix, if not in block mode).
15. The operator quant (X, p) returns the p-quantile of the distribution of the variable X. Calculation is done with the same method
als described for the quant command.
16. The operator quant1 (X, Z, p, m) returns the p-quantile of the distribution of X, but X might contain censored values. This is indicated by values of variable Z. If Zi = 0, Xi is assumed to be a
censored value; otherwise Xi is treated as uncensored. If the requested quantile cannot be calculated, the operator returns the
value m.
17. The operator vdif (X, Y ) returns an indicator of different values.
Given values Xi and Yi for i = 1, . . . , n, the result of vdif (X, Y )
for the ith record is 1 if Xi is not equal to any of the values Yj for
j = 1, . . . , n; otherwise the result will be 0. Example:
X
Y
vdif (X, Y )
1
2
7
9
1
3
2
8
1
8
0
0
1
1
0
5.2.6.2
recode operators
1
5.2.6.2 Recode Operators
Assume a variable, X, and a list, L, having two columns. The operator
Y = recode (X, L)
can be used to create a new variable, say Y , such that whenever X has a
value occurring in the first column of L, Y gets the corresponding value
in the second column of L. L must be the name of a matrix having at
least two columns, defined with an mdef command, see 5.1.2.
Using the recode operator, if X has a value not occurring in the
first column of L, Y gets the value of X. Alternatively, one can use the
operator
Y = recode (X, L, v)
Then, if a value of X does not occur in L, the new value in variable Y
will be v.
d05020602.tex
April 20, 1998
5.2.6.3
leads and lags
1
5.2.6.3 Leads and Lags
The following operators are available to refer to leads and lags of the
values of a variable.
1. Given a variable X, pre (X) provides its values lagged by one data
matrix row. X must be the name of a data matrix variable. Note that
the pre operator is specific in that it is the only operator that can
be used in a self-referencing way. In all other cases, when defining a
variable, one can only use references to previously defined variables.
In contrast, the pre operator allows for constructions as
Y = if ne(X,pre(X)) then Y else Y + pre(X)
These expressions must not contain any other type 2 operators; otherwise one will get a wrong result.
2. Given a variable X, suc(X) provides its values leaded by one data
matrix row. X must be the name of a data matrix variable.
3. Given a variable X, lag(X, n) provides its values lagged by n data
matrix rows if n < 0, or leaded by n data matrix rows if n > 0. It is
possible to use a general expressions instead of n.
The result of these operators is zero if they refer to nonexistent data
matrix rows. In block mode, the result is zero if they refer to data matrix
rows outside the current block.
d05020603.tex
April 20, 1998
5.2.6.4
1
sorting and ranking
5.2.6.4 Sorting and Ranking
The following type 2 operators can be used to sort, or rank, the values
of a variable.
1. Given a variable X, sort(X) creates a new variables that contains
the values of X sorted in ascending order.
2. Given a variable X, snum(X) provides serial numbers based on sorting the values of X in ascending order.
3. Given a variable X, rank(X) provides rank numbers for the values
of sort(X).
If the values of the variable X are not unique, the sorting is not stable.
In any case, evaluation depends on being in record mode or in block
mode. Some examples are shown in the following box.
X sort(X) sqrt(abs(sort(X))) -sort(-X) snum(X) rank(X)
------------------------------------------------------------1
-3
1.7321
9
3
3.5
7
-3
1.7321
9
8
8.0
2
1
1.0000
7
6
6.0
9
1
1.0000
2
10
9.0
-3
2
1.4142
2
2
1.5
1
2
1.4142
2
4
3.5
2
2
1.4142
1
7
6.0
-3
7
2.6458
1
1
1.5
9
9
3.0000
-3
9
9.0
2
9
3.0000
-3
5
6.0
d05020604.tex
April 20, 1998
5.2.6.5
1
counting subsets
5.2.6.5 Counting Subsets
The following type 2 operators can be used to count subsets (of cases)
of a variable.
1. cnteq (X, Y ) returns the number of cases where X is equal to Y .
2. cntne (X, Y ) returns the number of cases where X is not equal to
Y.
3. cntlt (X, Y ) returns the number of cases where X is less than Y .
4. cntgt (X, Y ) returns the number of cases where X is greater than
Y.
5. cntle (X, Y ) returns the number of cases where X is less than, or
equal to, Y .
6. cntge (X, Y ) returns the number of cases where X is greater than,
or equal to, Y .
X and Y are variables (or more general, expressions). Calculating the
counts depends on whether the operators are used in record mode or
in block mode. In record mode, the counts are based on the number of
cases in the currently defined data matrix, in block mode the counts are
calculated for each block separately. Some examples, assuming record
mode, are shown in the following box.
V1 V2 cnteq cntne cntlt cntgt cntle cntge
------------------------------------------------1 1
2
8
6
2
8
4
7 2
2
8
6
2
8
4
2 3
2
8
6
2
8
4
9 4
2
8
6
2
8
4
-3 5
2
8
6
2
8
4
1 6
2
8
6
2
8
4
2 7
2
8
6
2
8
4
-3 8
2
8
6
2
8
4
9 9
2
8
6
2
8
4
2 10
2
8
6
2
8
4
d05020605.tex
April 20, 1998
5.2.6.6
counting changes
1
5.2.6.6 Counting Changes
The following type 2 operators can be used to provide information about
changes in the values of a variable.
1. Given a variable X, change(X) provides a dummy variable taking
the value 1 if the current value of X is not equal to its previous value.
X is assumed to be not equal to its predecessor if in the first row of
the data matrix or of a block.
2. Given a variable X, cntch(X) provides the number of changes in
the values of X, that is, the number of rows where X is not equal to
its previous value. X is assumed to be not equal to its predecessor if
in the first row of the data matrix or a block.
3. Given a variable X, ccntch(X) provides the number of changes in
the values of X up to, and including, the current row. X is assumed
to be not equal to its predecessor if in the first row of the data matrix
or a block.
4. Given a variable X, lagch(X) provides the number of rows, calculated backwards from the current row, where the last change of X
occurred. X is assumed to be not equal to its predecessor if in the
first row of the data matrix or a block.
Some examples are shown in Box 1.
d05020606.tex
April 20, 1998
5.2.6.6
counting changes
Box 1 Examples of operators for changes
Record Mode
ID S
T change(S) cntch(S) ccntch(S) lagch(S) lag(T,-lagch(S))
--------------------------------------------------------------------1 1 10
1
7
1
0
10
1 3 15
1
7
2
0
15
1 2 20
1
7
3
0
20
2 2 12
0
7
3
1
20
2 3 18
1
7
4
0
18
3 3
8
0
7
4
1
18
3 1 12
1
7
5
0
12
3 2 15
1
7
6
0
15
3 2 17
0
7
6
1
15
3 2 18
0
7
6
2
15
3 1 19
1
7
7
0
19
Block Mode
ID S
T change(S) cntch(S) ccntch(S) lagch(S) lag(T,-lagch(S))
--------------------------------------------------------------------1 1 10
1
3
1
0
10
1 3 15
1
3
2
0
15
1 2 20
1
3
3
0
20
2 2 12
1
2
1
0
12
2 3 18
1
2
2
0
18
3 3
8
1
4
1
0
8
3 1 12
1
4
2
0
12
3 2 15
1
4
3
0
15
3 2 17
0
4
3
1
15
3 2 18
0
4
3
2
15
3 1 19
1
4
4
0
19
2
5.2.6.7
1
aggregates
5.2.6.7 Aggregates
The term aggregate is used to denote a contiguous sequence of data
matrix rows, identified by having the same value of an identification
variable. The following operators to evaluate characteristics of aggregates
are available.
1. gcnt (A) counts the number of cases in each of the aggregates defined
by the variable A.
2. grec (A) provides a serial number for the records in an aggregate
defined by variable A.
3. gsn (A) provides a serial number for the aggregates defined by A.
4. gfirst (A) provides a dummy variable with value 1 for the first
record in each aggregate defined by A.
5. glast (A) provides a dummy variable with value 1 for the last record
in each aggregate defined by A.
6. gsum (X, A) returns the sum of the values of variable X, calculated
separately for each aggregate defined by A.
7. gmean (X, A) returns the mean value of variable X, calculated separately for each aggregate defined by A.
8. gstd (X, A) returns the standard deviation of variable X, calculated
separately for each aggregate defined by A.
9. gmin (X, A) returns the minimum of variable X, calculated separately for each aggregate defined by A.
10. gmax (X, A) returns the maximum of variable X, calculated separately for each aggregate defined by A.
11. gsort (X, A) sorts the values of variable X, in ascending order, separately for each aggregate defined by A.
12. gndv (X, A) counts the number of different values in variable X,
separately for each aggregate defined by A.
d05020607.tex
April 20, 1998
5.2.6.7
aggregates
2
Box 1 Examples of operators for aggregates
V A gcnt(A) grec(A) gsn(A) gfirst(A) glast(A)
----------------------------------------------------1 1
3
1
1
1
0
4 1
3
2
1
0
0
5 1
3
3
1
0
1
3 2
1
1
2
1
1
6 7
2
1
3
1
0
2 7
2
2
3
0
1
V A gmean(V,A) gstd(V,A) gmin(V,A) gmax(V,A)
--------------------------------------------------1 1
3.333
2.082
1.000
5.000
4 1
3.333
2.082
1.000
5.000
5 1
3.333
2.082
1.000
5.000
3 2
3.000
0.000
3.000
3.000
6 7
4.000
2.828
2.000
6.000
2 7
4.000
2.828
2.000
6.000
13. gndv1 (X, Y, A) counts the number of different values in variable X
that are not less than Y , separately for each aggregate defined by A.
The aggregate consists of all contiguous data matrix rows that have an
identical value of the aggregating variable. Evaluation of the operator
depends on being in record mode or block mode. In block mode, there
are two levels of aggregates: the first level is defined as blocks, the second level is defined by an aggregating variable inside the blocks. Some
examples based on record mode are shown in Box 1.
5.2.6.8
1
string variables
5.2.6.8 String Variables
The following type 2 operators can be used for string variables.
1. Given a string variable S the operator
strsp(S)
returns a numerical variable that sorts the strings in S in ascending
order. For strings which are identical the resulting variable gets the
same value.
d05020608.tex
December 19, 1999
5.2.7
1
block mode operators
5.2.7 Block Mode Operators
The following type 2 operators assume block mode, see 2.2.
1. brec provides the current case number for the rows in a block; brec
is 1 for the first row in block, 2 for the second row, and so on.
2. bnrec provides the number of cases contained in the current block.
3. bfirst provides a dummy variable with value 1 for the first record
in the current block.
4. blast provides a dummy variable with value 1 for the last record in
the current block.
5. bmin(X) provides the minimum of the variable X in the current
block.
6. bmax(X) provides the maximum of the variable X in the current
block.
7. brd provides a random number that is equally distributed in the 0–1
interval. For each block a single new random number is drawn. The
algorithm is the same as used for the rd operator.
8. bnum provides the current block number. Block numbers are: 1, . . . , m
where m is the number of blocks in the data matrix.
The following box shows some examples.
X Y brec bnrec bfirst blast bmin(X) bmax(X)
brd
---------------------------------------------------------1 1
1
3
1
0
1
5
0.0291
4 1
2
3
0
0
1
5
0.0291
5 1
3
3
0
1
1
5
0.0291
3 2
1
1
1
1
3
3
0.9495
6 7
1
2
1
0
2
6
0.0943
2 7
2
2
0
1
2
6
0.0943
d050207.tex
December 19, 1999
5.2.8
operators for episode data
1
5.2.8 Operators for Episode Data
There are a few operators that can be used to get information about
currently defined episode data. These operators are also called type 3
operators and can only be used in expressions defining type 5 variables
in the edef command, see 3.3.2.
1. sn provides the current spell number.
2. org provides the current origin state.
3. des provides the current destination state.
4. ts provides the current starting time.
5. tf provides the current ending time.
6. There is one further operator, time, that can only used when estimating Cox models.
d050208.tex
April 20, 1998
5.2.9
operators for sequence data
1
5.2.9 Operators for Sequence Data
Sequence data are defined by sequences of variables, e.g., Y1 , . . . , Yn ,
where only nonnegative values are valid and negative values indicate
missings, see 3.4.2.
1. slen (Y1 , , Yn ) calculates the sequence length, that is, the number of
variables beginning with the first nonnegative value and ending with
the last nonnegative value.
2. glen (Y1 , , Yn ) calculates the length of internal gaps, that is, the
number of variables inside the valid sequence length having nonnegative values. For more information see 3.4.2.
d050209.tex
April 20, 1998
5.3
functions
1
5.3 Functions
This chapter explains TDA’s concept of functions. It contains the following sections.
5.3.1
Syntax for Functions
5.3.2
Automatic Differentiation
5.3.3
Evaluating Functions
d0503.tex
April 20, 1998
5.3.1
syntax for functions
1
5.3.1 Syntax for Functions
We consider real-valued functions
f (x1 , . . . , xn ) : Rn −→ R
(1)
The function f is an expression depending on the arguments x1 , . . . , xn .
Arguments can be any strings consisting of lower-case letters and digits.
They must begin with a lower-case letter and must be different from
predefined keywords and operators. Given this convention, one can define
functions in the same way as variables, by using a set of arguments,
numerical constants and operators. For example,
f (x, y) ≡ sin(x) + cos(y)
would be a valid function definition depending on two variables. In defining functions one can use most of TDA’s type 1 operators as described
in 5.2.5. Type 2 operators cannot be used. Also, for if-then-else constructions one has to use the if operator. For instance, to define the
function
1 : x ≤ 0 or x ≥ 1
f (x) =
0 : otherwise
one should use
f (x) ≡ if(le(x,0) + ge(x,1),1,0)
When defining functions one can also refer to data matrix variables. In
general, if the definition of a function contains at least one reference to
a data matrix variable, the function values are obtained by summing its
expression over all cases in the currently active data matrix. For example,
assume a data matrix containing the variables A and B for m cases. It
would then be possible to define a function
f (x, a, b) ≡ x + A * a + B * b
TDA will evaluate this expression as
f (x, a, b) =
m
X
x + Ai a + Bi b
i=1
d050301.tex
April 20, 1998
5.3.1
syntax for functions
2
Using this feature, it is quite easy to formulate function expressions for
nonlinear regression and ML estimation. However, such expression can
easily become quite complex. Therefore, to allow for clearly arranged
expressions, functions can also be defined recursively. The basic idea
is first to define parts of a complex expression and then using names
of these parts for defining more complex expressions. The syntax is as
follows:
f (x1 , x2 , x3 , . . .) ≡
t1 = expression depending on (x1,x2,...),
t2 = expression depending on (t1,x1,x2,...),
···
fn = expression depending on (t1,t2,...,x1,x2,...)
fn is a reserved keyword and, although it is optional, should be used
to define the final function expression. t1,t2,... and x1,x2,... can
be any parameter strings consisting of lower case letters and digits. All
parameter string which do occur on the left side of an equal sign will be
interpreted as intermediate expressions (the maximum number is 50);
all remaining strings will become arguments of the function. These final
function arguments are sorted alphabetically to provide an ordering: first
argument, second argument, and so on.
Here is a simple example.
f (x1 , x2 ) ≡ sx1 = sin(x1),
sx2 = sin(x2),
fn = sx1 + sx2
x1 is the first and x2 is the second argument of the function. As another example consider the log-likelihood of a simple logit model with a
dependent 0-1-variable, Y, and two independent variables, A and B. The
log-likelihood can be defined as
f (b0 , b1 , b2 ) ≡
v = b0 + A * b1 + B * b2,
e = exp(v),
fn = Y * log(e / (1 + e)) + (1 - Y) * log(1 / (1 + e))
or in a simpler way as
f (b0 , b1 , b2 ) ≡ e = b0 + A * b1 + B * b2,
fn = Y * e - log(1 + exp(e))
5.3.1
syntax for functions
3
The log-likelihood of a corresponding probit model could be written as
f (b0 , b1 , b2 ) ≡
e = b0 + A * b1 + B * b2,
fn = Y * log(nd(e)) + (1 - Y) * log(1 - nd(e))
5.3.2
1
automatic differentiation
5.3.2 Automatic Differentiation
Most algorithms for function minimization need first, some also need
second derivatives. In principle, there are four different approaches:
1. The user must also provide expressions for derivatives. Of course,
this would be very inconvenient and error prone.
2. Derivatives can be approximated numerically. This is not only very
time-consuming but also introduces numerical errors.
3. One can try to automatically create expressions for derivatives based
on the function expression provided by the user. However, it would be
very difficult to implement this approach in the required generality.
4. Finally, a very simple and efficient method is to calculate derivatives,
if required, simultaneously with evaluating the expression defining
the function. This approach was originally proposed by Wengert
[1964], see also Wexler [1987]. For the purpose of function minimization, this approach seems to be the most efficient and is used in
TDA.
In general, the implementation of this approach depends on how a program evaluates expressions. In TDA, given an expression, the program
first creates some intermediate code for representing the numerical steps
required to evaluate the expression. (To see this intermediate code, one
can use the parse command.) Based on this intermediate code, the expression can be evaluated efficiently for arbitrary arguments. If required,
first and second derivatives are calculated while evaluating this intermediate code for the expression.
There are some limitations for defining the function if derivatives are
required. The function may contain:
1. Numerical constants and references to data matrix variables.
2. The elementary operators +, −, ∗, /, and ∧ .
3. The operators exp, log, abs, sqrt, sin, cos, lgam, icg, eexp, ndf,
and nd. With restrictions, one can also use ndi, poisson, and negbin.
4. The logical operator if. Most other logical operators result in zero
d050302.tex
April 20, 1998
5.3.2
automatic differentiation
2
derivatives. Of course, when using the if operator, the result is not
necessarily a continuously differentiable function.
5. The int operator for numerical integration, see 5.4.
In order to check results one can use the evalf command, see 5.3.3.
5.3.3
1
evaluating functions
5.3.3 Evaluating Functions
The command for function evaluation is evalf with syntax shown in the
following box.
evalf (
fmt=...,
argument1=...,
argument2=...,
...,
) = function;
print format
value of first argument, def. 0
value of second argument, def. 0
and so on
The right-hand side must provide the definition of the function. All other
arguments are optional. As default, all arguments have zero value.
1. The evalf command only calculates the function value. If using the
command evalf1, instead of evalf, one also gets first derivatives, based
on automatic differentiation as explained in 5.3.2.
2. As a further option, one can use evalf2, instead of evalf, and will
then get the function value and first and second derivatives.
d050303.tex
April 20, 1998
5.3.3
evaluating functions
2
Box 1 Command file deriv1.cf and standard output
evalf2(
fmt = 12.6,
x = 2,
y = 5,
# print format
# set value of x argument
# and y argument
)
# define function
=
x * y^2;
Output
-----Function definition:
fn
= x*y^2
Arguments: x=2 y=5
50.000000
25.000000
20.000000
0.000000
10.000000
4.000000
function
gradient
gradient
hessian
hessian
hessian
value
x
y
x x
y x
y y
Example 1 To illustrate, we use the simple function f (x, y) = xy 2 .
Box 1 shows the command file deriv1.cf used to evaluate this function
and its derivatives. The command to request first and second derivatives
is evalf2. Optional parameters are the print format and values for the
arguments of the functions. The output shows the function value and
values of the derivatives, evaluated for the given arguments.
5.4
1
numerical integration
5.4 Numerical Integration
Several different methods have been proposed in the literature for numerical integration. In general, there is no best method, and the behavior
depends strongly on the type of function and the required accuracy. For
this reason, TDA offers a choice between five different methods.
1. Method 1 is the QNG algorithm, adapted from Piessens et al. [1983].
2. Method 2 is the QSUB algorithm adapted from Patterson [1973].
3. Method 3 is the SNIFF algorithm adapted from Garribba et al. [1978].
4. Method 4 is the QTRAP algorithm adapted from Press et al. [1988,
p. 121].
5. Method 5 is the QSIMP algorithm adapted from Press et al. [1988,
p. 123].
All these algorithms are adaptive, meaning that they try to reach a given
error tolerance by a succession of applying some basic integration rules.
Let f (x) denote the integrand, a real-valued function depending on a
single argument. It is tried to approximate the integral
Z
I=
b
f (x) dx
a
for the finite interval [a, b]. Given a relative error tolerance, r , an algorithm terminates successfully if
| Ik − Ik−1 | ≤ r | Ik |
(1)
where Ik denotes the kth approximation to I. All algorithms use this
relative convergence criterion. Only algorithm 1 (QNG) uses additionally
an absolute error tolerance, a , with the criterion
| Ik − Ik−1 | ≤ a
(2)
The algorithm terminates when (1) or (2) is satisfied.
d0504.tex
April 20, 1998
5.4
numerical integration
2
The different methods for numerical integration have been implemented
in TDA both on the level of commands and on the level of operators, see
5.4.1
Commands for Numerical Integration
5.4.2
Operators for Numerical Integration
5.4.1
commands for numerical integration
1
5.4.1 Commands for Numerical Integration
For available algorithms see 5.4. Default is method 1 (the QNG algorithm)
with relative error = 0.0001. To change this default, one can use the
niset command with syntax shown in the following box. The selection
remains valid until changed with another niset command.
niset (
rerr=,
aerr=,
) = n;
relative error, def. rerr=1.e-4
absolute error, def. aerr=1.e-4
method (1 – 5), def. 1.
While the main application of these algorithms in TDA is their use for
evaluating log-likelihood functions, they can also be used directly with
the int command. The following box shows the syntax.
int (
ab=a,b,
fmt=,
) = function;
integration interval, no default
print format, def. fmt=0.0
The command requires the definition of a function that depends on a
single argument on its right-hand side. Also required is the specification
of an integration interval with the ab parameter.
d050401.tex
April 20, 1998
5.4.1
commands for numerical integration
2
Box 1 Illustration of int command
Command:
int (ab=0,1) = sqrt(x);
Output:
> int(...)=...
---------------------------------------------------Numerical integration. Current memory: 132624 bytes.
Function definition:
fn
= sqrt(x)
Function argument: x
Integration interval: 0 -- 1
Method 1 (QNG).
Relative error: 1.00000e-04
Approximation: 0.666667 (successful)
Number of function calls: 43
Example 1 As an example, we try to find the value of
Z 1
I=
x1/2 dx
0
which should be equal to 2/3. When using the default method 1, the
command is simply
int(ab=0,1) = sqrt(x),
Box 1 shows this command and its standard output. The command file
in the example archive is int1.cf.
5.4.2
operators for numerical integration
1
5.4.2 Operators for Numerical Integration
In order to allow for numerical integration as part of expressions, there
are some operators. For integration over a finite interval one can use the
operators
int (a,b,f(t))
where a, b and f(t) are expressions, possibly depending on further arguments. This operator evaluates the integral
Z b
I=
f (t) dt
a
using the currently selected integration method (see the niset command
in 5.4.1). t is a reserved keyword and used to denote the integration
variable. If the expression defining f(t) does not contain t, TDA assumes
a constant function and simply returns its value multiplied with (b − a).
Example 1 To replicate the example given in 5.4.1 with the int operator, one can use the command
evalf = int (0,1,sqrt(t));
The command file in the example archive is int2.cf.
While it is not possible to use the int operator recursively, it can be used
as part of more complex expression, in particular as part of function
definitions. In general, a, b, and f(t) can be functions depending on
arguments. For example, assuming that
a = a(x1 , . . . , xn ),
b = b(x1 , . . . , xn ),
f (t) = f (x1 , . . . , xn , t),
the resulting integral would be the function
Z
b(x1 ,...,xn )
I(x1 , . . . , xn ) =
f (x1 , . . . , xn ; t) dt
a(x1 ,...,xn )
If required, first and second derivatives are calculated using automatic
differentiation, see 5.3.2. This allows, for example, to define log-likelihood
d050402.tex
April 20, 1998
5.4.2
operators for numerical integration
2
expression for transition rate models by only using analytical expressions
for the transition rate. For calculating first derivatives, TDA uses the formula:
∂I(x1 , . . . , xn )
=
∂xi
∂b(x1 , . . . , xn )
f (x1 , . . . , xn ; b) −
∂xi
∂a(x1 , . . . , xn )
f (x1 , . . . , xn ; a) +
∂xi
Z b(x1 ,...,xn )
∂f (x1 , . . . , xn ; t)
dt
∂xi
a(x1 ,...,xn )
For calculating second derivatives, TDA uses the formula:
∂ 2 I(x1 , . . . , xn )
=
∂xi ∂xj
∂ 2 b(x1 , . . . , xn )
f (x1 , . . . , xn ; b) +
∂xi ∂xj
∂b(x1 , . . . , xn ) ∂f (x1 , . . . , xn ; b)
+
∂xi
∂xj
∂b(x1 , . . . , xn ) ∂f (x1 , . . . , xn ; b)
−
∂xj
∂xi
∂ 2 a(x1 , . . . , xn )
f (x1 , . . . , xn ; a) −
∂xi ∂xj
∂a(x1 , . . . , xn ) ∂f (x1 , . . . , xn ; a)
−
∂xi
∂xj
∂a(x1 , . . . , xn ) ∂f (x1 , . . . , xn ; a)
+
∂xj
∂xi
Z b(x1 ,...,xn ) 2
∂ f (x1 , . . . , xn ; t)
dt
∂xi ∂xj
a(x1 ,...,xn )
TDA recognizes constant expressions with zero derivatives and skips
the corresponding calculations. However, evaluating first and second
5.4.2
operators for numerical integration
3
Box 1 Example command file int3.cf and standard output
evalf2(
x = 0,
y = 2,
z = 1.5,
) = int (x,y * z,2 * z^2);
Output:
Function definition:
fn
= int(x,y*z,2*z^2)
Arguments: x=0 y=2 z=1.5
13.5000
-4.5000
6.7500
27.0000
0.0000
0.0000
0.0000
-6.0000
13.5000
36.0000
function
gradient
gradient
gradient
hessian
hessian
hessian
hessian
hessian
hessian
value
x
y
z
x x
y x
y y
z x
z y
z z
derivatives of expressions containing the int operator can be very timeconsuming.
Limitations. One should be aware of an important limitation. The
integration variable, t, can only be used inside the int operator.
Example 2 To illustrate, command file int3.cf in Box 1 evaluates
the integral
Z yz
I(x, y, z) =
2 z 2 dt
x
and its derivatives. In this example, since the function does not depend
on the integration parameter t, TDA has actually not used any of the
specific subroutines for numerical integration. It would do, however, if
the function definition would be changed into
int (x,y * z,2 * z∧ 2 + 0 * t);
Of course, the result should be the same. See command file int3a.cf in
the example archive.
5.5
smoothing procedures
1
5.5 Smoothing Procedures
This chapter describes some general-purpose smoothing procedures. For
a useful introduction see Goodall [1990]. Further approaches will be discussed in the part on statistical procedures.
5.5.1
Moving Averages
5.5.2
Running Median Smoothers
5.5.3
Smoothing Splines
d0505.tex
April 20, 1998
5.5.1
moving averages
1
5.5.1 Moving Averages
Assume a variable, Y , with values y1 , . . . , yn . Also assume a sequence
of weights: wi ≥ 0, for i = 0, . . . , s. We can then defined a smoothed
variable, Y ∗ , by
yi∗
:=
s
X
wk yi+k
k=−s
This is called a moving average of the original variable, Y . Of course,
one will normally use weights having the property
s
X
wk = 1
k=−s
One needs to specify how to deal with the two ends of the sequence.
There are two possibilities.
1. One can restrict the calculation of smoothed values to the range
i = s + 1, . . . , n − s. This is sometimes called the copy-on end-value rule.
2. Alternatively, one can use the end values as often as they are required,
meaning that we introduce
y1−i := y1 for i = 1, . . . , s
yn+i := yn for i = 1, . . . , s
This is sometimes called the replicate end-value rule.
TDA’s command for calculating moving averages is called sma, the syntax
is shown in Box 1. The gss parameter must be used to specify weights
in the following order:
gss = w0 , w1 , . . . , ws ,
The command calculates a sequence of smoothed values for each variable
specified in varlist on the right-hand side. The result is written into
the standard output. Alternatively, one can specify an output file with
the df parameter. As an option, one can specify a repeat factor with the
r parameter meaning that the smoothing procedure is repeated r times.
d050501.tex
April 20, 1998
5.5.1
moving averages
2
Box 1 Syntax for sma command
sma (
specification of weights
treatment of end values, def. 1
1 = copy-on end-values
2 = replicate end values
repeat factor, def. 1
print format, def. 10.4
optional output file
optional case selection
gss=...,
opt=...,
r=...,
fmt=...,
df=...,
sel=...,
) = varlist;
Box 2 Command file ds3.cf
nvar(
dfile = ds3.dat,
Y = c1,
);
sma(
gss = 0.5,0.25,
df = df,
) = Y;
sma(
gss = 0.5,0.25,
opt = 2,
dfa = df,
) = Y;
sma(
gss = 0.5,0.25,
opt = 2,
r = 2,
dfa = df,
) = Y;
# data file
# replicate end values
# append
# replicate end values
# repeat
# append
Example 1 To illustrate, we use the data
Y = 1, 3, 7, 6, 6
contained in data file ds3.dat. The command file, ds3.cf, is shown in
Box 2. The first sma command uses the default copy-on end-value rule.
The second command uses the alternative replicate end-value rule. The
third command requests that the values are smoothed two times. The
resulting output file is shown in Box 3.
5.5.1
moving averages
3
Box 3 Output file, df, created by ds3.cf
# Var: Y
0
1
0
2
0
3
0
4
0
5
# Var: Y
0
1
0
2
0
3
0
4
0
5
# Var: Y
0
1
0
2
0
3
0
4
0
5
1.0000
3.0000
7.0000
6.0000
6.0000
1.0000
3.5000
5.7500
6.2500
6.0000
1.0000
3.0000
7.0000
6.0000
6.0000
1.5000
3.5000
5.7500
6.2500
6.0000
1.0000
3.0000
7.0000
6.0000
6.0000
2.0000
3.5625
5.3125
6.0625
6.0625
The sma Operator It is possible to use the sma procedure directly as
a type 2 operator. The syntax is:
VName = sma[gss=...,r=...,opt=...](Y),
where it is assumed that Y is the name of an already existing variable.
The sma operator creates a new variable, VName, containing the smoothed
values. The gss parameter must be used to specify weights. The other
parameters are optional.
5.5.2
1
running median smoothers
5.5.2 Running Median Smoothers
A useful smoothing procedure is based on running medians, see Goodall
[1990]. The TDA command is smd with syntax shown in Box 1. The sm
parameter must be used to specify a sequence of operations. The syntax
is
sm=[c1 c2 c3 ...],
where each ci












ci =











is one of the following characters:
2
3
4
5
H
R
S
t
2-median
3-median
4-median
5-median
hanning
replicate until no more changes
splitting
combine smooth and rough
There are some restrictions in specifying a sequence of these characters.
The character R can only be used if it comes after a 3 or 5. The t character, if used, must be the last character in the sequence and must have
another character as predecessor. Each character represents an operation; the ordering is from left to right. Given a variable, Y , with values
y1 , . . . , yn , the operations performed are
c
c
c
1
2
3
yi −→
yi0 −→
yi00 −→
...
meaning that each operation uses the result from the previous operation
as its input.
1. If ci = 2, the operation consists in applying an alternate 2-median:
if r = 0 : med(yi−1 , yi ) for i = 2, . . . , n
yi0 =
if r = 1 : med(yi , yi+1 ) for i = 1, . . . , n − 1
At the beginning, r = 0. Then, if the 2-median (or a 4-median) operation
is finished, r gets the value 1. Then, if the second 2-median or 4-median
operation is finished, r gets again the value 0, and so on.
d050502.tex
April 20, 1998
5.5.2
running median smoothers
2
Box 1 Syntax for smd command
smd (
sm=[...],
opt=...,
fmt=...,
df=...,
sel=...,
sequence of operations
treatment of end values, def. 1
1 = copy-on end-values
2 = replicate end values
3 = E rule
print format, def. 10.4
optional output file
optional case selection
) = varlist;
2. If ci = 3, the operation consists in applying a 3-median:
yi0 = med(yi−1 , yi , yi+1 )
3. If ci = 4, the operation consists in applying an alternate 4-median:
if r = 0 : med(yi−2 , yi−1 , yi , yi+1 ) for i = 3, . . . , n − 1
0
yi =
if r = 1 : med(yi−1 , yi , yi+1 , yi+2 ) for i = 2, . . . , n − 2
4. If ci = 5, the operation consists in applying a 5-median:
yi0 = med(yi−2 , yi−1 , yi , yi+1 , yi+2 ) for i = 3, . . . , n − 2
y20 = med(y1 , y2 , y3 )
0
yn−1
= med(yn−2 , yn−1 , yn )
5. If ci = H, the operation consists in calculating the moving average:
yi0 = 0.25yi−1 + 0.5yi + 0.25yi+1 for i = 2, . . . , n − 1
6. If ci = S, the operation consists in a splitting procedure applied to yi
and yi+1 . This is only done if yi = yi+1 and 3 ≤ i ≤ n − 3. The splitting
procedure sets:
yi0 = med(yi , yi−1 , 3yi−1 − 2yi−2 )
0
yi+1
= med(yi+1 , yi+2 , 3yi+2 − 2yi+3 )
5.5.2
running median smoothers
3
Box 2 Command file ds4.cf
nvar(
noc = 20,
# create random data
C = case,
A = rd,
M1 = smd[sm=[3R]](A),
M2 = smd[sm=[4253Ht]](A),
M3 = smd[sm=[3RSSHt]](A),
);
psfile = ds4.ps;
psetup(
pxa = 0,21,
pya = 0,1,
pxlen = 90,
pylen = 50,
);
plxa(sc=1);
plya(sc=1,ic=10);
plot(lt=0,s=5,fs=1.0) = C,M1;
plot=C,M1;
plot(lt=5) = C,M2;
plot(lt=8) = C,M3;
7. If ci = R, this is not a specific operation but the previous operation
(which must be a 3-median or a 5-median) is repeated until no more
changes occur in the smoothed values.
8. If the last character in the sequence of operations specified with the
sm parameter is t, smoothed and rough components are combined as
follows:
yi0 = S(yi ) + S(yi − S(yi ))
where yi refers to the original values of Y and S(y) means the result of
applying the whole sequence of operations to y.
By default (opt=1), the smd command uses the copy-on end-value rule. If
opt=2, the replicate end-value rule is used, but only for the H operation.
If opt=3, and if the operation is a 3-median or 5-median, the algorithm
additionally performs:
y10 = med(y1 , y20 , 3y20 − 2y30 )
0
0
0
yn0 = med(yn , yn−1
, 3yn−1
− 2yn−2
)
5.5.2
running median smoothers
4
1
0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21
Figure 1 PostScript plot, created with command file ds4.cf,
that illustrates the smd operator.
The smd Operator It is possible to use the smd procedure directly as
a type 2 operator. The syntax is:
VName = smd[sm=[...],opt=...](Y),
where it is assumed that Y is the name of an already existing variable.
The smd operator creates a new variable, VName, containing the smoothed
values. The sm parameter must be used to specify a sequence of operations in the same way as has been explained above for the smd command.
Example 1 To illustrate, command file ds4.cf, shown in Box 2, first
creates 20 random numbers and then applies the smd operator with
three different sequences of smoothing operations. It finally generates
a PostScript plot as shown in Figure 1.
5.5.3
smoothing splines
1
5.5.3 Smoothing Splines
Dierckx [1975] has published a useful algorithm that calculates smoothing splines (for a general introduction to splines see Boor [1978]). This
algorithm has been implemented in TDA. The command is spl with syntax shown in Box 1. The command requires two or three variables on
the right-hand side. If there are three variables, the values of the third
one are interpreted as weights.
These variables provide the input data (xi , yi ), or if weights are given
by a third variable, then (xi , yi , wi ), for i = 1, . . . , n. By default, wi = 1
if no weight variable is specified. Weights must be strictly positive. The
command then calculates a spline function, s(x), with degree k ≥ 2, the
default is k = 2. Higher degrees can be specified with the deg parameter.
The number of data points, n, must be at least 2k and the xi values must
be strictly increasing. The spl command tries to achieve this by sorting
the input data in ascending order. It will exit with an error message if
there are two, or more, equal values in the X variable.
The resulting spline also depends on the smoothing factor, σ, that
can be specified with the sig parameter. By default, σ = 0 and the
algorithm calculates an interpolating spline. If σ > 0, a smoothing spline
is calculated, meaning the smoothest spline function satisfying
n
X
wi (yi − s(xi ))2 ≤ σ
i+1
If σ is much larger than 0, the result will be an OLS approximation with
a polynomial of degree k.
The algorithm requires storage space for a maximum number of knots
that cannot be determined in advance. In general, the number of knots
actually used, nk , is
3k ≤ nk ≤ n + k + 1
The default maximum number of knots is
max{3k + 1, n/2 + 1}
If this is not sufficient, the user can specify another maximum with
the max parameter. Furthermore, the algorithm requires some iterative
d050503.tex
April 20, 1998
5.5.3
smoothing splines
2
Box 1 Syntax for spl command
spl (
sig=...,
deg=...,
max=...,
mxit=...,
fmt=...,
df=...,
rx=a(d)b,
smoothing factor, def. 0.0
degree of spline, def. 2
max number of knots, def. see text
max number of iterations, def. 10
print format, def. 10.4
output file
range for interpolation
) = X,Y [,W];
procedures and may not be able to reach convergence. The maximum
number of iterations can be specified with the mxit parameter. The
default value is 10 and should be sufficient for most applications.
There are two options of requesting an output. Both require that
an output file is specified with the df parameter. With the first option
(default), the output file will contain n records, an index number followed
by
xi , yi ,
d sj (x)
(j = 0, . . . , k)
dj x
possibly followed by a column containing the weights. So one not only
gets the smoothed values, but also derivatives up to the number of degrees.
Alternatively, one can use the rx parameter to specify a range of
values for interpolation. The syntax is
rx = a (d) b,
where it is required that a < b and d > 0. It is then tried to find values
of s(x) and its derivatives for the values
a + id
for i = 0, 1, 2, . . . until b
It may happen that the algorithm is not able to calculate the function for
the whole interval. The largest interval that can be used will be shown
in the standard output.
5.5.3
smoothing splines
3
Box 2 Command file ds5.cf
nvar(
noc = 100,
X[10.4] = case / 10,
Z[10.4] = sin(X),
Y[10.4] = Z + rd(-0.2,0.2),
);
psfile = ds5.ps;
psetup(
pxa = 0,10,
pya = -1.5,1.5,
pxlen = 90,
pylen = 50,
);
plxa(sc=1);
plya(sc=0.5,ic=5,fmt=4.1);
plot(lt=0,s=2,fs=1.0) = X,Y;
plot(lt=5) = X,Z;
spl(
sig = 1,
df = df,
) = X,Y;
clear;
nvar(
dfile = df,
X[10.4] = c2,
Y[10.4] = c4,
);
plot = X,Y;
Example 1 To illustrate, command file ds5.cf, shown in Box 2, first
creates some random data and then uses the spl operator to calculate
a smoothing spline with smoothing factor σ = 1. It finally generates a
PostScript plot shown in Figure 1. The solid line represents the spline
function.
5.5.3
4
smoothing splines
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
0
1
2
3
4
5
6
7
8
9
10
Figure 1 PostScript plot, created with command file ds5.cf,
that illustrates the spl command.
5.6
function minimization
1
5.6 Function Minimization
This chapter describes a command, and some algorithms, that can be
used to find a local minimum of a function. The same algorithms can be
used for nonlinear regression and maximum likelihood estimation. This
will be discussed in different sections of the manual. Here we have the
following sections.
5.6.1
The fmin Command
5.6.2
Algorithms
5.6.3
Protocol File
5.6.4
Starting Values
5.6.5
Covariance Matrix
5.6.6
Constraints
5.6.7
Possible Difficulties
d0506.tex
April 20, 1998
5.6.1
1
the fmin command
5.6.1 The fmin Command
The command for function minimization is fmin with syntax
fmin (parameter) = function;
where function is a function as explained in 5.3.1. Optional parameters
are shown in Box 1.
1. The mina parameter can be used to select a minimization algorithm.
For available algorithms and their features see 5.6.2. The default algorithm is Newton (I) and requires first and second derivatives. For most
functions, these derivatives will be available via automatic differentiation. Otherwise one should select an algorithm that only needs first
derivatives, or no derivatives at all. As a special option for algorithms 7
and 8, one can use the dopt=1 parameter to request numerical approximation of derivatives.
2. If f (θ) is the function given as input to the fmin command, it actually
minimizes σf (θ) where σ is some scaling factor. This scaling factor can
be specified with the parameter
dscal = σ,
Scaling is helpful to reduce the probability of numerical overflows and
underflows. However, the scaling factor σ only influences the global range
of the function. There is no automatic procedure for also scaling the
individual parameters in the function. The user should therefore take
care that all variables have similar magnitudes.
3. The mxit parameter can be used to specify a maximal number of iterations. For default values, and also for stopping criteria and convergence
tolerances, see 5.6.2.
4. The prot parameter can be used to request a protocol file. For more
information see 5.6.3.
5. The xp and dsv parameters can be used to define starting values.
How to use these parameters will be explained in 5.6.4.
d050601.tex
December 18, 1999
5.6.1
the fmin command
2
Box 1 Syntax for fmin command
fmin (
mina=...,
dopt=...,
dscal=...,
mxit=...,
mxitl=...,
crit=
tolg=...,
tolf=...,
tolp=...,
tolv=...,
tols=...,
tolsg=...,
tolsp=...,
slen=...,
sred=...,
smin=...,
sst=...,
prot=...,
pfmt=...,
xp=...,
dsv=...,
ppar=...,
tfmt=...,
ccov=...,
pcov=...,
mfmt=...,
mplog=...,
mppar=...,
mpcov=...,
mpgrad=...,
pres=...,
mfmt=...,
v=...,
) = function;
minimization algorithm, def. 5
1 for numerical approx of derivatives, def. 0
scaling factor for function, def. 1
max number of iterations
max number of line search iterations, def. 50
convergence criterion
tolerance: norm of gradient, def. 10−6
tolerance: change of function value, def. 10−12
tolerance: change of parameters, def. 10−4
tolerance: variance of function values, def. 10−10
tolerance: reduction factor, def. 10−4
tolerance: scaled gradient, def. 10−5
tolerance: scaled parameter change, def. 10−8
step length, def. 1
reduction factor, def. 0.5
minimum of step size, def. 10−10
option for evaluation of derivatives, def. 0
(or prot1) name of protocol file
print format for protocol file, def. -19.11
starting values
input file with starting values
output file with estimated parameters
print format for parameters, def. 10.4
type of covariance matrix
output file with covariance matrix
print format for covariance matrix, def. 12.4
write function minimum into matrix
write minimizing arguments into matrix
write covariance matrix into matrix
write gradients into matrix
additional output file
print format for pres option, def. 12.4
varlist for pres option
6. The ppar parameter can be used to request an additional output file
that will contain the estimated parameters (and standard errors, if any).
The print format can be controlled with the tfmt parameter.
5.6.1
the fmin command
3
Box 2 Command file fmin1.cf and output
fmin = (x - 5)^2 + (y + 6)^2;
Output:
Function definition:
fn
= (x-5)^2+(y+6)^2
Function minimization.
Algorithm 5: Newton (I)
Number of model parameters: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: 1
Idx
1
2
Parameter
x
y
Starting value
0.0000
0.0000
Convergence reached in 2 iterations.
Number of function evaluations: 2 (2,2)
Minimum of function: 0
Last absolute change of function value: 1
Last relative change in parameters: 1
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 x
5.0000
2 y
-6.0000
Function (starting values):
Function (final estimates):
61.0000
0.0000
7. A covariance matrix is only calculated if the function refers to at least
one data matrix variable and the number of cases exceeds the number of
function arguments to provide at least one degree of freedom. The ccov
parameter can be used to select a calculation method for the covariance
matrix, and the pcov parameter can be used to write the covariance
matrix into an output file. For more information about covariance matrix
calculation see 5.6.5.
5.6.1
the fmin command
4
8. The pres parameter can be used to request an output file containing
individual function values for all data matrix cases. If the function definition refers to data matrix variables, and if the minimization procedure
finished successfully, the pres parameter creates the specified output
file containing: a case number for the data matrix rows and the function
value evaluated separately for each data matrix row. The print format
for the function values can be controlled with the fmt parameter, default
is fmt=12.4. In addition, one can use the parameter
v = varlist,
to add the variables specified in varlist to the output file.
Example 1 To illustrate the fmin command, we begin with a simple
example without reference to a data matrix. The function is
f (x, y) = (x − 5)2 + (y + 6)2
Box 2 shows the command file, fmin1.cf, and the standard output from
its fmin command. Since the function definition does not refer to data
matrix variables, a covariance matrix is not calculated and there are no
standard errors.
Example 2 To illustrate reference to data matrix variables we calculate the mean value of a variable Ai = 1, . . . , 101 by minimizing
f (x) =
101
X
(x − Ai )2
i=1
Box 3 shows the command file, fmin2.cf, and the standard output from
its fmin command. The covariance matrix is written to cov; the protocol file will give information about intermediate calculations. The pres
parameter in the command file creates an additional output file, res,
containing three variables. First, the case number. Second, the function
value for each case; in this example, it is simply (51 − i)2 . The third
variable is A, specified with the v parameter.
5.6.1
the fmin command
Box 3 Command file fmin2.cf and part of output
nvar(
A = case,
noc = 101,
);
fmin (
prot1 = p,
pcov = cov,
pres = res,
v = A,
# define data: A = 1,...,101
#
#
#
#
protocol file
covariance matrix
print individual function values to res
add variable A
) = fn = (x - A)^2; # function definition
Part of standard output
----------------------Function definition:
fn
= (x-A)^2
Function minimization.
Algorithm 5: Newton (I)
Number of model parameters: 1
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: 1
Idx
1
Parameter
x
Starting value
0.0000
Convergence reached in 2 iterations.
Number of function evaluations: 3 (3,3)
Minimum of function: 85850
Norm of final gradient vector: 4.9738e-13
Last absolute change of function value: 0.753695
Last relative change in parameters: 1
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 x
51.0000
0.0707
721.2489 1.0000
Function (starting values): 348551.0000
Function (final estimates): 85850.0000
5
5.6.2
minimization algorithms
1
5.6.2 Minimization Algorithms
Many different algorithms have been proposed in the literature for function minimization. Our basic references are McCormick [1983] and Fletcher [1987]. For a short overview, see also Harley [1986]. TDA supports
the methods shown in the following box.
1
2
3
4
5
6
7
8
Direct search
Simplex algorithm
Method of conjugate gradients
BFGS algorithm
Newton algorithm (I)
Newton algorithm (II)
CES (quadratic model)
CES (tensor model)
Direct search. The most simple, but least efficient method is direct
search. TDA’s version of this method is based on an algorithm given
by Kaupe [1963] taking into account some modifications proposed by
Bell and Pike [1966], Tomlin and Smith [1969], and Smith [1969]. The
algorithm does not need any derivatives, selection is with mina=1.
The algorithm is controlled by two parameters, a step length and a
reduction factor. They can be changed with the slen and sred parameters, respectively. The step length, multiplied with the absolute values
of the parameter components, determines how the parameter space is
searched; the default step length is 1. The reduction factor is used for a
proportional reduction of the step length in each iteration, the default
value is 0.5. The parameters can be changed with the options
slen = new step length,
sred = new reduction factor,
The algorithm terminates when the maximum number of iterations (default 100) is reached, or when the reduction factor is less than a given
tolerance that can be controlled with the option
tols = minimum of reduction factor,
d050602.tex
April 20, 1998
5.6.2
minimization algorithms
2
The default is tols = 10−4 .
Simplex method. Another algorithm that does not need derivatives
but is more powerful than direct search is based on a simplex procedure,
see Nelder and Mead [1965]. TDA’s version of this algorithm is adopted
from O’Neill [1974] and can be selected with mina=2.
The size of the initial simplex can be controlled with the slen option,
default value is 1. The algorithm terminates if the maximum number of
iterations is reached, default is 100, or if the variance of the function
values evaluated at the simplex vertices is less than a given tolerance
that can be controlled with the parameter
tolv = variance of function values,
The default value is tolv = 10−10 .
Both algorithms, direct search algorithm and the simplex procedure,
are not recommended in most situations but may sometimes be helpful
to find starting values for other methods.
Conjugate gradients and BFGS. There are two variants of QuasiNewton methods, a method of conjugate gradients and the BFGS algorithm. The implementation of these algorithms in TDA is based on
Shanno and Phua [1986]. Both algorithms need first derivatives; the
method of conjugate gradients can be selected with mina=3, the BFGS
method with mina=4.
For both algorithms, the convergence check is based on the Euclidean
norm of the gradient of the function. Iterations end if this norm becomes
less than a given tolerance. The default value of this tolerance is 10−6 ,
but can be changed with the parameter
tolg = norm of gradient,
The default maximum number of iterations for these algorithms is 100,
but can be changed with the mxit option.
Two more parameters can be used to control the line search. The
default maximum number, 50, of sub-iterations during line search can
be changed with the parameter
mxitl = maximum number of line search iterations,
and the minimum step length, default 10−10 , can be changed with the
parameter
smin = minimum step length,
5.6.2
minimization algorithms
3
Both algorithms are quite reliable and can be used with all of TDA’s
models. Normally one would prefer the Newton method discussed below. However, this method may fail due to badly chosen starting values
or because calculation of second derivatives is inexact. The method of
conjugate gradients or the BFGS algorithm is then a good alternative.
With most models implemented in TDA, the BFGS seems to be more
efficient than the method of conjugate gradients.
CES algorithms. Recently, Chow, Eskow and Schnabel [1994] have
published an algorithm based on tensor methods. Instead of approximating the function by a quadratic model, their algorithm uses a fourth-order
model. At least in certain applications, this seems to be more efficient.
The TDA implementation of this method is adapted from the code
published by Chow, Eskow and Schnabel [1994]. There are actually two
methods, one based on a quadratic model, the other one based on the
tensor model. The first method, called CES (quadratic model), can be
selected with mina=7, the second method, called CES (tensor model),
can be selected with mina=8.
Convergence check is based on two criteria. First, the algorithm
checks the size of a scaled gradient and assumes convergence if
|gf (θi )| maxj {θj }
max
≤ TOLSG
i
max{f (θ), 1}
θ = (θ1 , . . . , θn ) is the parameter vector, f (θ) the function value, and
gf (θ) the gradient. TOLSG is the tolerance with default value 1.e-5; this
value can be changed with the tolsg parameter. A second check is based
on changes of the parameter vector. Convergence is then assumed if
|θi − θi∗ |
≤ TOLSP
max
i
max{θi , 1}
where θ∗ is the parameter vector from the previous iteration. TOLSP is
the tolerance with default value 1.e-8; this value can be changed with the
tolsg parameter. The maximum number of iterations can be controlled
with the mxit parameter, default is mxit=20.
Newton methods. If first and second derivatives are available, a very
efficient minimization method for well-behaved functions is the Newton
algorithm. The algorithm requires that the function is well-behaved in
at least a neighborhood of its minimum. Furthermore, one needs appropriate starting values already near to the minimum. In particular, the
5.6.2
minimization algorithms
4
Hessian matrix must be positive definite for all parameter values during
the iteration process.
Consequently, for practical implementation, the Newton method must
be modified to cope with situations where the Hessian is not positive definite. Many different proposals exist. For implementation in TDA, we have
chosen two options. The first (default) option uses simply the gradient
if the Hessian is not positive definite.1 The second, more sophisticated
modification follows a proposal of McCormick [1983, p. 166].
To explain the algorithm we shall use the following terminology. f (θ)
is the function to be minimized. gf (θ) is the gradient, i.e. the (column)
vector of first derivatives. hf (θ) is the Hessian matrix, i.e. the matrix of
second derivatives. The index k is used to denote iterations.
Then, having some start vector, θk , for the kth iteration step, the
algorithm tries to find a better estimate, θk+1 . To find this new parameter
estimate, the algorithm calculates a search direction and a step size. The
calculation depends on whether the Hessian matrix, evaluated at θk , is
positive definite or not.
If the Hessian matrix, hf (θk ), is positive definite, one can use the
Newton search direction defined by
sk = −hf (θk )−1 gf (θk )
(1)
The new parameter vector is then calculated as
θk+1 = θk + λk sk
(2)
where λk is a scalar, called the step size. This step size is chosen as
the largest value of the series 2−i (i = 0, 1, 2, . . .) so that the first-order
Armijo condition (cf. McCormick [1983, p. 134]) is fulfilled:
f (θk+1 ) − f (θk ) ≤ µ λk s0k gf (θk )
0<µ<1
(3)
The scalar constant µ is given a default value of 0.2. In most cases it is
not necessary to try other values. This can be done, however, by using
the amue option.
Two more parameters can be used to control the line search. The
default maximum number, 50, of sub-iterations during line search can
be changed with the mxitl option; and the minimum step length, default
10−10 , can be changed with the smin option.
1 This
has been proposed, for instance, by Polak [1971, p. 304].
5.6.2
minimization algorithms
5
If the Hessian matrix at θk is not positive definite the Newton search
direction is not available. This is possible in intermediate iteration steps,
in particular, if the starting values are not close to the optimizing point.
Of course, it should not be the case in the final iteration steps because
a local minimum requires that the Hessian matrix is positive definite.
If the algorithm finds that the Hessian matrix at θk is not positive
definite, its further behavior depends on the type of modification. With
type I (mina=5), the algorithm reduces then to a simple gradient method,
that is the search vector is determined as the negative of the gradient,
sk = −gf (xk ). The algorithm then proceeds in the same way as explained
above, i.e. the line search is based on the first-order Armijo condition.
With type II of the algorithm (mina=6), the search vector is build
from two parts. The first part is, again, the negative of the gradient.
A second part is added to reach, hopefully, a region of negative (nonpositive) curvature of the function. This is tried by using the eigenvector,
ek , of the Hessian matrix that is associated with its minimum eigenvalue.2 Because the minimum eigenvalue is less than, or equal to zero,
ek is a vector of non-positive curvature,
e0k hf (θk ) ek ≤ 0
(4)
The sign of ek is chosen so that it can be used as part of the search
direction, as a vector with non-ascent direction
e0k gf (θk ) ≤ 0
The new parameter estimate is calculated as
p
θk+1 = θk + λk sk + λk ek
(5)
(6)
λk is again a scalar step size, now the largest of the series 2−i that fulfills
the second-order Armijo condition (cf. McCormick [1983, p. 135])
1 0
0
f (θk+1 ) − f (θk ) ≤ µ λk sk gf (xk ) + ek hf (xk ) ek
(7)
2
where µ is defined the same way as with the first-order Armijo condition
(3). With both modifications of the standard Newton algorithm it should
be possible, not in all but in many cases, to reach within a few iteration
2 The
algorithm to calculate the minimum eigenvector of the Hessian matrix is
adopted from Sparks and Todd [1973].
5.6.2
minimization algorithms
6
steps a region of negative curvature where the standard method applies.
One can expect that the type II algorithm is more efficient than type I.
However, the calculation of ek is expensive, and the algorithm is slowed
down in this case; therefore, the default algorithm in TDA is type I
(mina=5).
For both versions of the Newton methods one has a choice of three
convergence criteria, selectable with the crit parameter.
1. If crit=1, the algorithm uses the Euclidean norm of the gradient. If
this is less than, or equal to a given tolerance value, the algorithm assumes to have reached a local minimum and terminates. The default
value is 10−6 , but can be changed with the tolg option.
2. If crit=2, the algorithm uses the change in function values during the
iterative process. The algorithm terminates if the absolute difference
of the function values in two successive iterations is less than, or
equal to a given tolerance. The default value, 10−12 , can be changed
with the tolf option.
3. If crit=3, the algorithm uses changes in the function arguments to
decide about convergence. First, for each of these arguments, the
relative change across two successive iterations is calculated. Then,
the algorithm terminates if the maximum of these changes is less
than, or equal to a given tolerance. The default value is 10−4 and
can be changed with the tolp parameter.
The default convergence criterion is a combination of the first two criteria. Convergence is assumed whenever the Euclidean norm of the gradient
is less than tolg or the decrease of the function is less than tolf. If a
specific criterion is selected with the crit option, then only this criterion
is used to check convergence. All convergence checks are based on the
scaled function.
In addition to these convergence checks, the algorithm terminates
if the maximum number of iterations is reached. The default value for
both Newton algorithms is 20 iterations. This can be changed with the
mxit option. But, as a general rule, if these algorithms do not reach
convergence in less than 20 iterations there is a high probability that the
model is badly specified.
Another point of consideration is how to evaluate the function during
step size search. This could be done without calculation of derivatives
because they are not needed for the step size search. However, in many
cases the Armijo conditions can be fulfilled with a unit step size, that
5.6.2
minimization algorithms
7
is with only a single sub-iteration. It is more efficient, then, to have
the derivatives already calculated since they can be used for the next
iteration step. Therefore, as a default, all function evaluations during the
step size search include the calculation of derivatives. If this default is
not optimal, it can be changed with the sst=1 option (default is sst=0).
5.6.3
protocol file
1
5.6.3 Protocol File
When TDA runs a minimization algorithm, basic results are written into
the program’s standard output and standard error output. The standard
output will indicate whether the algorithm has converged and found a
local optimum and also provides information about the number of iterations, function (log-likelihood) values, and parameter estimates. While
the algorithm is running, some information about the iterations is also
written into the standard error output. These messages will appear on
the screen even if the standard output has been redirected into an output
file.
In addition, one can request a more detailed protocol of the calculations performed by the selected minimization algorithm. The option to
request such a protocol file is
prot = name of protocol file,
TDA then creates an output file with the specified name and writes ad-
ditional information into this file. By default, only an iteration protocol
is written into the output file. More information about parameter values
and derivatives (if any) can be requested by using prot1 instead of prot.
(An additional option, prot2, is basically only for test purposes.) The
print format for the protocol file can be controlled with the
pfmt = print format for protocol file,
parameter. The default is pfmt=-19.11. If the prot parameter specifies
the name of an already existing file, it is overwritten without warning.
One can use prota, instead of prot, to request that the new information
is appended to the the end of an already existing file.
d050603.tex
April 20, 1998
5.6.4
1
starting values
5.6.4 Starting Values
Iterative procedures for function minimization require starting values for
the parameters to be estimated. Although TDA provides default starting
values, they are sometimes not good enough to find a solution. The user
has then the option to provide his own starting values instead of TDA’s
default values. There are two options.
1. One can use the parameter
xp = list of starting values,
The right-hand side must provide a sequence of numerical values separated by commas. They will be used, in the same order, as starting
values for the model parameters.
2. Alternatively, one can use the parameter
dsv = name of file containing starting values,
The file must be a free-format data file containing the starting values as
first entries in each record. Given that a model has n parameters, TDA
sequentially reads the first n data records and takes the first numerical
entry in the ith record as starting value for the ith model parameter. If
the file contains more than n data records, or if there is more than one
numerical entry in each record, this is ignored. If there are less than n
values, the remaining model parameters get the initial value zero.
In many applications model estimation is done sequentially, starting with
simple models containing only a few parameters and then adding more
parameters. It is often a good idea, to use parameter estimates from a
previously estimated model as starting values for a new one. This can
easily be achieved by using the option
ppar = name of an output file,
This option writes the estimated parameters (and its standard errors)
into the specified output file. And the resulting output file can then be
used as an input file providing the starting values for a new model estimation. Of course, if the new model contains additional parameters,
d050604.tex
April 20, 1998
5.6.4
starting values
2
starting values for these parameters should be added to the file. The
print format for the output file containing parameter estimates can be
controlled with the tfmt option. The default is tfmt=10.4 and is identical to the print format used for parameter estimates in TDA’s standard
output.
5.6.5
covariance matrix
1
5.6.5 Covariance Matrix
Let f (θ) denote the function depending on a parameter vector, θ, that
has q components. To exhibit alternative estimates of a covariance matrix, we use the following definitions.
" n
#
X ∂f (yi , θ) ∂f (yi , θ)
J(θ) =
∂θj
∂θk
i=1
j,k=1,...,q
#
" n
X ∂ 2 f (yi , θ)
H(θ) =
∂θj ∂θk
i=1
j,k=1,...,q
These definitions assume that the function is summed over n data matrix
cases and yi denotes the data for the ith case. For example, f can be a
log-likelihood function.
To select a method of calculation one can use the ccov parameter.
There are three possibilities (additional background information will be
given in the section on maximum likelihood estimation).
J(θ̂)−1
if ccov = 1
−H(θ̂)−1
if ccov = 2
H(θ̂)−1 J(θ̂)H(θ̂)−1
if ccov = 3
Note that the ccov parameter works independent of the selected minimization algorithm. Having found a solution, θ̂, for a local minimum,
TDA uses this parameter vector for an additional function evaluation
that performs the desired covariance matrix calculation.
d050605.tex
April 20, 1998
5.6.6
constraints
1
5.6.6 Constraints
As an option, one can define linear equality constraints for the parameters of a function. If θ is the parameter vector with q components, linear
equality constraints can be written as
Rθ = d
(1)
R is a (r, q) matrix with r the number of constraints and d is a vector
of dimension r. It is required that r < q since otherwise there are no
degrees of freedom for a change of the function values. Also, to simplify
calculations, it is required that all rows of R are linear independent, that
is, R should have rank r.
The problem is to minimize the function f (θ), as defined above, subject to the constraint (1). To find a solution we follow the discussion
given by McCormick [1983, chap. 13]. The basic idea is to define an appropriate subspace of the original parameter space and then to perform
unconstrained function minimization in this subspace. An appropriate
subspace is given as the null space of R. This is easily seen by assuming
that one already has a starting vector θs with Rθs = d. Then all vectors
fulfilling this constraint can be represented as
θs + θ
with
Rθ = 0
(2)
If R has rank r, its null space has dimension q−r. One can define a (q, q−
r) projection matrix Q to map all vectors of the (q−r)-dimensional space
into the null space of R, that is a matrix Q with RQ θ̃ = 0 for all vectors
θ̃ of dimension (q − r). Consequently, the constrained minimization of
f (θ) is equivalent to an unconstrained minimization of the function
F (θ̃) = f (θs + Q θ̃)
(3)
Having found an unconstrained local minimum of F , say θ̃k , a local
minimum of f (θ), subject to the constraint (1), is given by θs + Q θ̃k .
Any of the algorithms discussed above can be used to find an unconstrained local minimum of F (θ). If one needs the gradient and the
Hessian of F , these are given, respectively, by
gF (θ̃)
hF (θ̃)
d050606.tex
= Q0 gf (θs + Q θ̃)
0
= Q hf (θs + Q θ̃) Q
(4)
(5)
April 20, 1998
5.6.6
constraints
2
To find the projection matrix Q, we follow a method described by McCormick [1983, p. 265]. It is tried to find a (r, r) matrix RD and a (r, q−r)
matrix RI , where RD consists of r linearly independent columns of R
and RI consists of the remaining columns of R.1 The projection matrix
Q can then be calculated as
"
−1 I #
− RD
R
Q=
(6)
In−r,n−r
where In−r,n−r denotes an (n−r, n−r) identity matrix. In most cases this
approach to calculating the projection matrix implies an interchange of
the columns of R and, consequently, of the parameter vector. Information
about the construction of Q and the implied column interchange can be
found in the protocol file if requested with the prot1 option.
If convergence has been reached in the reduced parameter space, say
at θ̃k , and if it has been possible to calculate the covariance matrix in this
space, the covariance matrix for the constrained model in the original
space is calculated according to
Cov[θ̂] = Q Cov[θ̃k ] Q0
(7)
This applies to all three options for covariance matrix calculation described in 5.6.5). The covariance matrix is first calculated in the reduced parameter space and then projected (expanded) into the original
parameter space.
Syntax. The syntax to define equality constraints follows formula (1).
The basic key word is con to be used in the following way:
con
= r11 b1 ± · · · ± r1q bq = d1 ,
con
= r21 b1 ± · · · ± r2q bq = d2 ,
..
.
= rr1 b1 ± · · · ± rrq bq = dr ,
con
(8)
The di and rij values must be given as numerical constants (floating
point numbers); if zero, they can be omitted. The b1 , b2 , . . . , bq refer to
the model parameters and must be given as b1,b2,b3,..., independent
of how the model parameters might be called in the model description.
1 All
calculations are done by a function adopted from the routine LHHFTI published
by Lawson and Hanson [1974].
5.6.6
constraints
3
The ordering must conform to the structure of the model’s parameter
vector; b1 refers to the first parameter, b2 refers to the second parameter,
and so on.
Note that using constraints might conflict with user-supplied starting
values. If there is no solution of the constraints with the given starting
values, TDA simply uses one of the feasible solutions as new starting
values. A message is then given in the standard output.
Example 1 To illustrate, we add the constraint x = −y to the function minimization command already used in Example 5.6.1-1. The new
command (see command file fmin3.cf) is
fmin( con = b1 + b2 = 0 ) = (x − 5)∧ 2 + (y + 6)∧ 2;
The solution is x = 5.5, y = −5.5.
5.6.7
possible difficulties
1
5.6.7 Possible Difficulties
There is no guarantee that the algorithms described in 5.6.2 will actually find a local maximum of the function. Several problems may occur
during the iterative process of function minimization, mainly as a consequence of inappropriate starting values, numerical problems like rounding errors and overflows and underflows in function evaluations, or as a
consequence of an ill-conditioned Hessian matrix caused by collinearities
of covariates. Concentrating on the two types of Newton algorithms, we
shortly describe the problems that are recognized by TDA.
1. The Hessian matrix may not be positive definite as is requested for
a local minimum. In many cases this problem may be solved by one of
the modifications of the Newton algorithm described in 5.6.2. However,
it is quite possible that the algorithm does not finally reach a region
of negative curvature with a positive definite Hessian. In this case the
algorithm fails definitely and one should check the model specification
and/or try with some other starting values.
2. It is possible that the algorithm does not find a positive step size
to fulfill the Armijo conditions. Sometimes this problem can be solved
by defining a smaller value for the minimum step size constant, see the
smin parameter. Normally, one has to try with other starting values, or
to try with another algorithm.
3. Another problem is that the algorithm may be unable to decrease
the function value so far as is required by the convergence criterion.
For instance, it may not be possible to find a gradient with a norm
less than some very small tolerance. In most cases this problem is a
consequence of the limited accuracy of function evaluations and/or of
numerical underflows and overflows. Sometimes it may also happen that
the log-likelihood becomes positive. If this problem occurs one should
check the required accuracy for convergence or select another convergence criterion.
4. The algorithms may fail because convergence could not be reached
within the predefined maximum number of iterations. Of course, it is
easily possible to increase this maximum using the mxit parameter. However, in most circumstances this can be taken as a serious hint that there
d050607.tex
April 20, 1998
5.6.7
possible difficulties
2
is a problem with the model specification and/or the data.
5. Finally, the algorithm may give rise to senseless results if there is
high collinearity across some of the covariates. One should note that
such problems are not detected by TDA’s minimization algorithms. In
most cases, collinearity shows up in other problems, caused by an illconditioned Hessian.
To help in diagnostics if the minimization procedure fails, one can request
a protocol of the iterations with the prot option, see 5.6.3. In addition, if
the program encounters numerical problems, it prints some information
about these problems into its standard output. The following types of
numerical problems are most important:
1 Numerical underflows or overflows in the evaluation of elementary
functions like exponential and logarithm.
2 Numerical underflows or overflows in the evaluation of the function
and/or its derivatives.
4 If during the iteration process the Hessian matrix is not always positive definite.
5 If problems occur in evaluating type 5 variables (containing the time
operator) while calculating the likelihood for transition rate models.
6 If problems occur in the calculation of probabilities that should have
values between 0 and 1.
7 Problems in the calculation of the incomplete gamma integral.
8 Problems in the calculation of the incomplete beta integral.
9 If problems occur during calculation of the covariance matrix.
10 If numerical integration cannot achieve the required accuracy.
None of these problems implies that the algorithm has failed. However,
the reliability of the results should be checked, then, by running the
procedure once more with different starting values.
6
statistical procedures
1
6. Statistical Procedures
This part contains the following sections.
6.1
Introduction
6.2
Elementary Descriptive Procedures
6.4
Concentration and Inequality
6.5
Describing Episode Data
6.6
Describing Sequence Data
6.7
Investigating Proximities
6.9
Least Squares Regression
6.10
Alternative Regression Methods
6.11
Maximum Likelihood Estimation
6.12
Quantal Response Models
6.14
Models for Count Data
6.16
Nonlinear Regression
6.17
Transition Rate Models
6.18
Regression Models for Events
6.19
Loglinear Models
d06.tex
December 18, 1999
6.1
1
introduction
6.1 Introduction to Statistical Procedures
This chapter is intended to serve as a collection of introductory remarks
for the discussion of statistical procedures in subsequent sections. Currently, we have just a few remarks about missing values and case weights.
6.1.1
Missing Values
6.1.2
Case Weights
d0601.tex
April 21, 1998
6.1.1
missing values
1
6.1.1 Missing Values
The term missing value is used in two different ways. One meaning
refers to data generation. When the nvar command reads a data file, it
might not be able to find the expected data required to create the values
for a variable. These are then “missing values”, substituted by specific
numerical missing value codes as explained in 2.2. However, when the
data matrix has been created successfully, it only consists of numerical
values, and TDA no longer makes any distinction between valid and
“missing” values.
Consequently, there is no direct support for the second meaning of
the term missing value that refers to the input data given to a statistical
procedure. In principle, each statistical procedure should be given some
information about the character of its input data, whether they are “exact”, have more or less errors, or are totally missing. Currently, TDA’s
statistical procedures do not conform to this requirement. The user is
responsible for the data that are given to the procedures. When there
are missing values, they must be excluded explicitly by appropriate case
selection commands.
We intend to deal with the missing value in a more satisfactory way
in a future version of TDA, based on a notion of set-valued variables. This
should allow to specify, for each variable, a range of possible values. And
we hope that we will be able to develop at least some basic statistical
procedures that can use such kind of input data.
d060101.tex
April 21, 1998
6.1.2
case weights
1
6.1.2 Case Weights
Many procedures in TDA are able to recognize case weights. For historical
reasons, there are two different approaches. Some command, as e.g. the
lsreg command, require a local definition of case weights; some other
commands require a definition of case weights with a separate command,
cwt. The syntax of the cwt command is
cwt = W;
where W is the name of a variable which must be contained in the current
data matrix. The values of W are then interpreted as weights for the corresponding rows of the data matrix. These values must be non-negative,
and the sum of weights for all cases in the data matrix, or cases selected
with the tsel command, must be strictly positive. If one of these conditions is not met the program will terminate with an error message.
Alternatively, one can use the command as
cwt (wnorm=s) = W;
where s is a positive number. The weights are modified in such a way that
the sum of weights for the currently selected data matrix rows becomes
equal to s. When simply using
cwt (wnorm) = W;
the sum of weights becomes equal to the currently selected number of
cases. A specification of case weights remains active until substituted by
another cwt command, or explicitly turned off with the command
cwt = off;
d060102.tex
December 18, 1998
6.2
1
elementary descriptive procedures
6.2 Elementary Descriptive Procedures
6.2.1
Mean and Range of Variables
6.2.2
Empirical Distribution Function
6.2.3
Quantiles
6.2.4
Frequency Tables
6.2.5
Aggregated Tables
6.2.6
Covariance and Correlation
6.2.7
Contingency Measures
6.2.8
Scatterplots
d0602.tex
April 21, 1998
6.2.1
mean and range of variables
1
6.2.1 Mean and Range of Variables
The dstat command is to request basic information about variables.
The syntax is shown in the following box.
dstat (
grp=...,
fmt=...,
df=...,
mppar=...,
group variables
print format, def. 10.4
write to output file
create matrix
) = varlist;
All parameters are optional. The command uses the variables specified
in varlist on its right-hand side or, if varlist is omitted, all currently
defined variables.
1. One can give a list of group variables with the parameter
grp = G1,G2,...,
Each variable on the right-hand side is then taken to define a group
consisting of all cases where the variable has a nonzero value, and all
calculations are done separately for each group.
2. Results are written into the standard output. The print format can
be controlled with the fmt parameter, default is fmt=10.4. If the name
of an output file is specified with the df parameter, the results are also
written into that output file.
3. The mppar parameter can be used to create a matrix that contains, in
each of its rows, the calculated statistics as described below. By default,
the matrix has 6 columns and the first column is zero. If there are groups,
then the first column will contain the corresponding group number.
4. For each variable the command calculates the following quantities:
1. the minimum value
2. the maximum value
d060201.tex
December 19, 1999
6.2.1
mean and range of variables
2
Box 1 Data file ds1.dat
1
2
3
4
1 -1 1.e-1
7 -1 1.e-2
1 -1 1.e-3
7 2 1.e-4
Box 2 Command file ds1.cf
nvar(
dfile = ds1.dat,
X1 = c1,
X2 = c2,
X3 = c3,
X4 = c4,
);
dstat;
# data file
3. the mean value
4. the standard deviation
5. the total (sum of values)
These quantities are calculated for the currently selected cases in the
data matrix, optionally modified with the sel and grp parameters. The
command recognizes case weights if defined with the cwt command. The
mean is calculated as
M(x) = Σi wi xi Σi wi
where wi is the weight for case i; without weights, wi = 1 for all cases.
The standard deviation is calculated as
q
S(x) = Σi wi (xi − M(x))2 ((Σi wi ) − 1)
To calculate standard deviations, TDA uses a two-pass algorithm; first
the mean values are calculated, then the squared deviations from the
mean are accumulated.1
Example 1 To illustrate the dstat command, we use the data file
ds1.dat (Box 1) containing some arbitrary data. The command file,
1 This
is recommended in the literature studying numerical accuracy in algorithms
for simple descriptive statistics; see, e.g., Neely [1966], Chan and Lewis [1979].
6.2.1
mean and range of variables
3
Box 3 Part of standard output from ds1.cf
> dstat
----------------------------------------------------Descriptive statistics. Current memory: 148696 bytes.
Variable
Minimum
Maximum
Mean
Std.Dev.
Sum of values
----------------------------------------------------------------X1
1.0000
4.0000
2.5000
1.2910
10.0000
X2
1.0000
7.0000
4.0000
3.4641
16.0000
X3
-1.0000
2.0000 -0.2500
1.5000
-1.0000
X4
0.0001
0.1000
0.0278
0.0484
0.1111
ds1.cf, is shown in Box 2, and part of the standard output is shown in
Box 3.
6.2.2
empirical distribution functions
1
6.2.2 Empirical Distribution Functions
This section describes the gdf command that can be used to calculate
marginal and joint distribution and survivor functions, based on possibly
incomplete (censored) data. The syntax of the command is shown in Box
1. Most parameters are optional. Required is the name of a variable to
be specified with the yl parameter, and the name of an output file to be
given on the right-hand side.
Input data must be defined for i = 1, . . . , n units. Each unit can
contribute observations for (a subset of) m dimensions. If available, yij
is the observation for unit i in dimension j. In general, each observation
consists of two parts
(yij , δij )
where yij is the observed value, and δij indicates whether the observation is not censored (δij = 1), or is right censored (δij = 0). The
corresponding data matrix variables can be specified with the yl and
cen parameters, respectively. The following combinations are possible.
1. Only observed values are specified with the yl parameter. Then all
observations are assumed to be exact with corresponding values.
2. Observed values are specified with the yl parameter and a censoring
indicator is specified with the cen parameter. An observations is
then interpreted as exact at yij if δij = 1 and is interpreted as right
censored if δij = 0.
By default, the command assumes a single dimension (m = 1) and n =
NOC units, where NOC is the number of cases in the current data matrix.
The grp parameter can be used to specify multivariate data. The syntax
is
grp = ID, L1,
where ID and L1 are names of data matrix variables. Each block of data
matrix rows where ID has identical values is interpreted as data for one
unit. Any values are possible and since the data matrix is always sorted
with respect to ID and L1, it is not required that blocks are contiguous.
The L1 variable must contain positive integers. The number of different
d060202.tex
November 16, 1998
6.2.2
empirical distribution functions
2
Box 1 Syntax for gdf command
gdf (
opt=...,
prn=...,
yl=...,
cen=...,
grp=ID,L1,
sc=...,
n=...,
mxit=...,
tolf=...,
d=...,
fmt=...,
prot=...,
method, def. 1
1 = marginal calculation
2 = joint calculation, method 1
3 = joint calculation, method 2
output option, def. 0
0 = distribution functions
1 = survivor functions
2 = expected values
variable name for observed values
variable name for censoring information
specification of dimensions
offset for domain (method 1 and 2), def. 0
number of boxes in grid (method 1), def. 100
maximal number of iterations (method 1), def. 20
tolerance for convergence (method 1), def. 0.001
delta specification (method 2), def. 0.1
print format for output file, def. 10.4
protocol file with diagnostic information
) = fname;
Box 2 Example data to illustrate grp parameter
ID L1 YL
D
---------------1
1
3
1
1
3
5
1
1
4
4
0
2
3
0
1
5
1
7
0
2
4
1
0
integers found in this variable is interpreted as the number dimensions.
Again, it is not required that these numbers are contiguous. Each data
matrix row provides one observation for the unit given by the ID variable
and dimension given by the corresponding L1 variable.
To illustrate, consider the data in Box 2. In this example there are
three units providing six observations. The number of dimensions is m =
3. Dimensions are mapped to the values of the L1 variable in ascending
6.2.2
empirical distribution functions
3
order. Thus, dimension 1, 2, and 3 correspond, respectively, to L1 = 1, L1
= 3, and L1 = 4. The first unit (ID = 1) has observations for all three
dimensions. The observation is exact in the first and second dimension,
and right censored in the third dimension. Unit 2 contributes an exact
observation to the second dimension and a right censored observation to
the third dimension. The third unit (ID = 5) contributes a right censored
observation to the first dimension.
Marginal Distribution Functions
If opt = 1 (default), the command calculates marginal distributions,
separately for each dimension present in the input data. Assuming that
there are nj observations for the jth dimension, the data are:
(y1j , δ1j ), . . . , (ynj j , δnj j )
Depending on prn, these data are used to calculate a distribution function (prn=0), a survivor function (prn=1), or expected values (prn=2).
Marginal EDF: Exact Observations
If the data for one dimension contain only exact observations, the command calculates a standard empirical distribution function,
Fj (yij ) =
nj
X
I(Yj ≤ yij )
i=1
if prn = 0, or survivor function,
Sj (yij ) = 1 − Fj (yij )
if prn = 1. Here Yj denotes the variable in the jth dimension and refers
to the possible values yij , i = 1, . . . , nj . I() denotes the indicator function. In this case, if prn = 2, the expected values equal the observed
values.
Box 3 provides an illustration. Input data are given by the variable
YL. The command
gdf (yl=YL) = df0;
creates the output file df0, the command
gdf (yl=YL,prn=1) = ds0;
6.2.2
empirical distribution functions
4
Box 3 Illustration of calculations with exact observations
YL
-1
7
-1
5
7
9
df0 (prn = 0)
ds0 (prn = 1)
D
Y
F
----------------------1
-1.0000
0.1667
1
1.0000
0.3333
1
5.0000
0.5000
1
7.0000
0.8333
1
9.0000
1.0000
D
Y
S
----------------------1
-1.0000
0.8333
1
1.0000
0.6667
1
5.0000
0.5000
1
7.0000
0.1667
1
9.0000
0.0000
creates the output file ds0.1 The first column in the output files shows
the dimension, then follow the value of the variable (sorted in ascending
order) and the corresponding value of the distribution or survivor function. If the input data refer to more than one dimension, the output file
will contain the same information separately for each dimension.
Marginal EDF: Right Censored Observations
A second situation occurs if the input data, for one dimension, contain
both, exact and right censored observations. The command then uses the
standard Kaplan-Meier procedure to calculate a marginal distribution,
or survivor, function. Considering the jth dimension, the observations
for Yj are sorted in ascending order. If there are exact and right censored
observations for the same value of the variable, exact observations come
first, followed by right censored observations. The highest value of Yj is
always treated as not censored. Then, in the order from lowest to highest
values, the mass of each right censored observation is distributed over
all observations having higher values.
An example is given in Box 4. Input data are given by the variables
YL and DELTA. The command
gdf (yl=YL,cen=DELTA) = df1;
creates the distribution function. Adding the parameter prn = 1 creates
the corresponding survivor function.
1 The
data file for this and the following examples is gdf1.dat. The command file is
gdf1.cf. Both are contained in the TDA example archive.
6.2.2
empirical distribution functions
5
Box 4 Marginal distributions with right censored observations
df1 (prn = 0)
YL DELTA
-------3
1
3
0
2
0
1
1
4
1
5
1
ds1 (prn = 1)
D
Y
F
----------------------1
1.0000
0.1667
1
3.0000
0.3750
1
4.0000
0.6875
1
5.0000
1.0000
D
Y
S
----------------------1
1.0000
0.8333
1
3.0000
0.6250
1
4.0000
0.3125
1
5.0000
0.0000
Box 5 Expected values based on marginal distribution
de1 (prn = 2)
YL DELTA
-------3
1
3
0
2
0
1
1
4
1
5
1
D
Y
CEN
E
-------------------------1
3.0000 1
3.0000
1
3.0000 0
4.5000
1
2.0000 0
4.1250
1
1.0000 1
1.0000
1
4.0000 1
4.0000
1
5.0000 1
5.0000
Marginal EDF: Expected Values
If prn = 2, the gdf command calculates expected values based on the
marginal distributions. If an observation is exact its expected value
equals its observed value. If the observation is right censored, the command calculates
Z ∞
Z
∞
∗
yij = EFj (Yj | Yj > yij ) =
y dFj
dFj
yij
yij
where Fj is the Kaplan-Meier estimate of the marginal distribution function in the jth dimension.
To illustrate, consider the data in Box 5 (same as in Box 4). The
command is now
gdf (yl=YL,cen=DELTA,prn=2) = de1;
The resulting output file, also shown in Box 5, contains four columns. The
first column refers to the current dimension, the second column contains
6.2.2
empirical distribution functions
6
the observed values, and the third column shows the censoring status of
the observation. The last column contains the expected value. This will
equal the observed value if the observation is not censored, otherwise the
expected value as calculated from the Kaplan-Meier survivor function.
Note that with this option, the input data are not sorted. Note also that
the observation with highest value is always assumed to be uncensored.
Joint Distributions
We now discuss the calculation of joint distribution functions. This is
done separately for each marginal pattern which is found in the input
data. The word “marginal pattern” refers to a combination of dimensions,
(d1 , ..., dmk )
where
1 ≤ d1 < d2 < · · · < dmk ≤ m
m being the maximal number of dimensions as given by the input data.
For ease of notation, the following discussion refers to a full marginal
pattern, meaning that mk = m.
Now, let n denote the number of units for the marginal pattern. We
then have m observations for each unit, as follows:
(yi1 , δi1 ), . . . , (yim , δim )
i = 1, . . . , n
Based on these data, the command calculates marginal distributions (if
opt = 1) or joint distributions (if opt = 2 or opt = 3). Calculations
depend on whether the data contain right censored observations. The
contents of the resulting output file depend on the prn parameter. The
command calculates a distribution function if prn = 0, a survivor function if prn = 1, or expected values if prn = 2.
Joint Distribution: Exact Observations
If all observations are exact, the command calculates a standard mdimensional distribution function, defined as
F (y1 , . . . , ym ) =
n Y
m
X
I(Yj ≤ yi )
i=1 j=1
or the corresponding survivor function
S(y1 , . . . , ym ) =
n Y
m
X
i=1 j=1
I(Yj > yi )
6.2.2
empirical distribution functions
7
Box 6 Joint distribution with exact observations
ID L1
YL
-----------1
1
1.0
1
2
2.0
2
1 -1.0
2
2
1.0
3
1
3.0
3
2
1.5
4
1
2.0
4
2
1.0
MP D1 D2
F
Y1
Y2
-----------------------------------------1
1 2
0.2500
-1.0000
1.0000
1
1 2
0.5000
1.0000
2.0000
1
1 2
0.5000
2.0000
1.0000
1
1 2
0.7500
3.0000
1.5000
MP D1 D2
S
Y1
Y2
-----------------------------------------1
1 2
0.0000
3.0000
1.5000
1
1 2
0.2500
2.0000
1.0000
1
1 2
0.0000
1.0000
2.0000
1
1 2
0.5000
-1.0000
1.0000
The functions are calculated, and tabulated in the output file, for all
data points in the input data.
To illustrate, consider the data shown in Box 6. There are four units,
all having observations for two dimensions. The joint distribution function, shown in the upper half of the right part of the box, was calculated
with the command
gdf (grp=ID,L1,yl=YL,opt=2,prn=0) = df3;
The corresponding survivor function was calculated with the command
gdf (grp=ID,L1,yl=YL,opt=2,prn=1) = ds3;
In both cases, the first column in the output file refers to the current
marginal pattern, followed by the dimensions (values of L1 variable)
that define this pattern. The next m columns contain the observations,
sorted in ascending or descending order. The final column contains the
corresponding value of the distribution or survivor function.
If there are only exact observations, and prn = 2, the output file will
simply contain the observations, a censoring indicator that always has
value 1, and expected values that equal the observed values. In this case,
the input data are not sorted.
Censored Observations, Method 1
We now consider a situation where the multivariate data contain right
censored observations. Unfortunately, there is no simple generalization
6.2.2
empirical distribution functions
8
of the 1-dimensional Kaplan-Meier procedure. Discussion in the literature has proposed several different approaches. The gdf command offers
two methods. The first one (selected with opt = 2) can be used with
observations which might be censored simultaneously in several dimensions. The second method (selected with opt = 3) can only be used with
observations which are censored at most in one dimension. This section
describes the first method (opt = 2).
This method uses an iterative EM-like procedure that tries to find
a self-consistent estimate of the distribution function. In explaining the
procedure we refer, again, to a full marginal pattern. Data are then given
by
(yi1 , δi1 ), . . . , (yim , δim )
i = 1, . . . , n
In a first step, we calculate a domain as an m-dimensional interval
D = D1 × · · · × Dm
where
Dj = ] min{yij } − σ, max{yij } + σ]
i
i
By default, the offset is σ = 0. This implies that observations which have
an exact component on a left side of the domain, or a right censored
component on a right side of the domain, will not be used.2 In order
to include all observations one can specify a positive offset with the sc
parameter, see Box 1.
The iterative procedure is based on a partition of the domain into a
grid of boxes. The jth dimension is partitioned into qj intervals
ij (k) = ] lj (k), uj (k) ]
k = 1, . . . , qj
These intervals, and the corresponding boxes, are treated as open on
the left side and closed on the right side. Since the number of boxes
rapidly increases in higher dimensions, we require the user to specify a
total number of boxes for the whole grid with the n parameter, default is
n = 100. The command then tries to find an integer q such that q m ≈ n
and sets
q1 = . . . = qm = q
2 Corresponding
values of the distribution function, or survivor function, will then be
-1, and expected values will then equal the observed (possibly censored) values.
6.2.2
empirical distribution functions
9
The minimum is q = 1, that is, the grid consists of only a single box.
Let now Bj = {1, . . . , qj }. Then, each
(k1 , . . . , km ) ∈ B1 × · · · × Bm
refers to one box in the grid, namely
B(k1 , . . . , km ) = i1 (k1 ) × · · · × im (km )
We now define for each observation (yij , δij ) a subset
bj (yij , δij ) ⊆ Bj
containing pointers to those boxes (in the jth dimension) where the
observation possibly has values. Explicitly, if the observation is exact,
the definition is
bj (yij , δij ) = k ∈ Bj | yij ∈ ij (k)
and if the observation is right censored, the definition is
bj (yij , δij ) = k ∈ Bj | ∃ δ > 0 : yij + δ ∈ ij (k)
Then, for each unit i,
b̄i = b1 (yi1 , δi1 ) × · · · × bm (yim , δim )
provides pointers to those boxes in the domain where unit i has, possibly,
an m-dimensional value. Of course, b̄i will be empty, if unit i has a
component not covered by the domain.
Using these notations, Box 7 shows the iterative algorithm.3 The
maximal number of iterations can be specified with the mxit parameter,
default is 20. Convergence is assumed if
max { | f (k1 , . . . , km ) − f 0 (k1 , . . . , km ) | } ≤ TOLF
k1 ,...,km
where f 0 () refers to the density from the previous iteration. By default,
TOLF = 0.001; other values can be specified with the tolf parameter.
Information about the number of iterations and the final value of
the convergence criterion is given in the standard output. In any case,
3 The
notation ‘+ =’ means that the expression on the right-hand side is added to
the expression on the left-hand side.
6.2.2
empirical distribution functions
10
Box 7 Iterative algorithm for joint distribution (method 1)
(1) ∀ (k1 , . . . , km ) ∈ B1 × · · · × Bm : f ∗ (k1 , . . . , km ) = 0
(2) ∀ i ∀ (k1 , . . . , km ) ∈ b̄i : f ∗ (k1 , . . . , km ) + =
1
n | b̄i |
(3) ∀ (k1 , . . . , km ) ∈ B1 × · · · × Bm : f (k1 , . . . , km ) = 0
(4) ∀ i ∀ (k1 , . . . , km ) ∈ b̄i :
f ∗ (k1 , . . . , km )
1
f (k1 , . . . , km ) + = P
n (l1 ,...,lm )∈b̄i f ∗ (l1 , . . . , lm )
(5) end if convergence has been achieved, or the maximal
number of iterations has been reached.
(6) ∀ (k1 , . . . , km ) ∈ B1 × · · · × Bm :
f ∗ (k1 , . . . , km ) = f (k1 , . . . , km )
(7) continue with (3)
depending on prn, the gdf command finally calculates a distribution
function, a survivor function, or expected values. In order to explain the
calculation, let
kij = min { kj | kj ∈ bj (yij , δij ) }
meaning that kij refers to the box where, in dimension j, observation yij
begins. The distribution function is then calculated using the formula
X
X
F (yi1 , . . . , yim ) =
···
f (k1 , . . . , km )
k1 =1,...,ki1
km =1,...,kim
Correspondingly, calculation of the survivor function uses the formula
X
X
S(yi1 , . . . , yim ) =
···
f (k1 , . . . , km )
k1 =ki1 ,...,q1
km =kim ,...,qm
If prn = 2, the command calculates expected values. To explain this
option, let (yi1 , . . . , yim ) be one of the m-dimensional observations. The
6.2.2
empirical distribution functions
11
Box 8 Example data set (Pruitt [1993])
ID L1
Y CEN
--------------1
1
1
0
1
2
6
1
2
1
2
0
2
2
4
1
3
1
3
0
3
2
5
0
4
1
4
1
4
2
3
1
5
1
5
1
5
2
2
0
6
1
6
1
6
2
7
1
7
1
7
0
7
2
1
0
8
1
8
1
8
2
8
1
expected value, in the jth dimension, will equal yij if this component is
not censored. Otherwise, it is calculated by
P
(k1 ,...,km )∈b̄i zj (k1 , . . . , km ) f (k1 , . . . , km )
P
(k1 ,...,km )∈b̄i f (k1 , . . . , km )
zj (k1 , . . . , km ) is the jth component of the mean value of all exact observations falling in box (k1 , . . . , km ) or, if the box does not contain exact
observations, equals the mean of ij (kj ).
Example 1. For a first illustration we use some example data from
Pruitt [1993], shown in Box 8.4 In order to estimate a distribution function, we use the command
gdf(opt=2,prn=0,yl=Y,cen=CEN,grp=ID,L1,n=64,sc=0.5) = df5;
In this example, we have used a total number of 64 boxes and added
a small offset to the domain in order to cover all observations.5 Box 9
shows the resulting output file. Estimated expected values, calculated
with the prn=2 option are shown in Box 10.
4 The
5 The
data file is gdf2.dat, contained in the TDA example archive.
command file is provided as gdf2.cf.
6.2.2
12
empirical distribution functions
Box 9 Estimated distribution function
MP D1 D2
Y1
Y2
F
-----------------------------------------1
1 2
1.0000
6.0000
0.0000
1
1 2
2.0000
4.0000
0.0000
1
1 2
3.0000
5.0000
0.0001
1
1 2
4.0000
3.0000
0.1250
1
1 2
5.0000
2.0000
0.0000
1
1 2
6.0000
7.0000
0.6290
1
1 2
7.0000
1.0000
0.0000
1
1 2
8.0000
8.0000
1.0000
Box 10 Estimated expected values
MP D1 D2
Y1 (obs)
Y2 (obs)
CEN
Y1 (est)
Y2 (est)
------------------------------------------------------------1
1 2
1.0000
6.0000
0 1
5.6197
6.0000
1
1 2
2.0000
4.0000
0 1
5.5839
4.0000
1
1 2
3.0000
5.0000
0 0
6.6101
6.9849
1
1 2
4.0000
3.0000
1 1
4.0000
3.0000
1
1 2
5.0000
2.0000
1 0
5.0000
5.1561
1
1 2
6.0000
7.0000
1 1
6.0000
7.0000
1
1 2
7.0000
1.0000
0 0
7.8492
7.1540
1
1 2
8.0000
8.0000
1 1
8.0000
8.0000
110
100
90
80
70
60
50
40
30
20
10
0
-10
0
10
20
30
40
50
60
70
80
90
100
Figure 1 Illustration of observed data points and estimated
expected values. Estimation with marginal Kaplan-Meier procedure (x) and with joint estimation (method 1, n = 100 boxes).
6.2.2
empirical distribution functions
13
Example 2. For a second illustration, we create 100 data points
(yi1 , yi2 )
i = 1, . . . , 100
where
yi1 = i, yi2 = i + ri
and ri are random numbers which are equally distributed in [−10, 10].
We then have randomly censored 13 of these data points in the second
dimension. The resulting data points are shown in Figure 1.
We then estimated expected values, first using a marginal KaplanMeier procedure.6 The resulting estimated values are indicated in Figure 1 by cross (x) symbols. We then used the iterative procedure described above to estimate expected values. The resulting estimated values are indicated in Figure 1 by symbols. They obviously provide
somewhat better estimates.
Censored Observations, Method 2
We now describe an alternative approach (selected with opt = 3) that
can be used when the observations are censored in at most a single dimension. The basic idea is quite simple: we use a local Kaplan-Meier
procedure based on all observations that are available in one of the censored dimensions.
Box 11 explains the algorithm. It is controlled by parameters ∆j that
define the size of the subsets of the domain used for the local KaplanMeier procedure. These parameters are calculated by using the parameter d that can be specified by the user, see Box 1. Then
∆j = d W j
where Wj denotes the width of the domain in dimension j. Default is
d = 0.1.
The algorithm results in densities, f (i), for all data points i that do
not contain censored observations. Depending on prn, they are finally
used to calculate a distribution function
X
F (yi1 , . . . , yim ) =
f (k)
(yk1 ,...,ykm )≤(yi1 ,...,yim )
6 The
command file for data generation and estimation is provided as gdf3.cf in the
TDA example archive.
6.2.2
empirical distribution functions
14
Box 11 Algorithm for joint distribution (method 2)
(1) Sort observations (yi1 , . . . , yim ) in ascending order,
first wrt first component, then wrt second component,
and so on; in case of ties, exact observations precede
censored observations.
(2) for i = 1, . . . , n : f (i) = 1/n
(3) i = 1
(4) If (yi1 , . . . , yim ) is exact in all components,
continue with step (8).
(5) Let (yi1 , . . . , yim ) be censored in dimension j0 .
Calculate an index set B(i, j0 ) containing indices of all
data points (yk1 , . . . , ykm ) for which:
a) ykj > yij
b) ∀ j 6= j0 : ykj is exact
c) ∀ j 6= j0 : | ykj − yij | ≤ ∆j0 /2
f (i)
(6) ∀ k ∈ B(i, j0 ) : f (k) + =
| B(i, j0 ) |
(7) f (i) = 0
(8) i + = 1
(9) if i ≤ n continue with (4).
if prn = 0, or a survivor function
S(yi1 , . . . , yim ) =
X
f (k)
(yk1 ,...,ykm )>(yi1 ,...,yim )
if prn = 1. These values are tabulated in the output file for all data
points. The data points are not sorted.
If prn = 2, the command calculates expected values. For each data
point (yi1 , . . . , yim ), if yij is exact, this will equal the corresponding
expected component. If yij is censored, the expected value is calculated
6.2.2
empirical distribution functions
15
Box 12 Estimated expected values (method 1 and method 2)
method 1 method 2
Y1 (obs) Y2 (obs) CEN
Y1 (est) Y2 (est) Y2 (est)
-------------------------------------------------------34.0000
27.3464
1 0
34.0000
37.6603
37.3177
36.0000
30.3996
1 0
36.0000
37.6603
38.0035
43.0000
15.3593
1 0
43.0000
46.2104
46.2348
45.0000
43.5366
1 0
45.0000
48.9264
48.4735
49.0000
50.6783
1 0
49.0000
54.3749
51.9167
68.0000
50.5112
1 0
68.0000
61.4348
71.7563
69.0000
47.4082
1 0
69.0000
61.4348
71.7563
73.0000
49.9009
1 0
73.0000
74.5214
73.0277
75.0000
65.2591
1 0
75.0000
74.5214
73.7913
79.0000
38.7495
1 0
79.0000
74.5214
79.1623
84.0000
53.5625
1 0
84.0000
82.3746
83.4225
85.0000
62.9593
1 0
85.0000
82.3746
85.3742
93.0000
33.8836
1 0
93.0000
99.9475 103.4777
by
P
ŷij =
k∈B(i,j) ykj f (k)
P
k∈B(i,j) f (k)
For the definition of B(i, j) see Box 11. If this index set is empty, ŷij
will equal the observed value, yij .
Example 3. For an illustration, we use the 2-dimensional data from
example 2. There are 100 data points, 13 are censored in the second
dimension. Since the data points are censored in only a single dimension,
we can use both methods, 1 and 2. Box 12 shows the censored data points
and estimated expected values for their censored component.7 It is seen
that, in this example, both methods give quite similar estimates.
Example 4. If d = 1, the algorithm uses all observations (in the censored dimension) and becomes identical with a standard marginal KaplanMeier procedure. For an illustration see command file gdf5.cf in the
TDA example archive.
7 The
command file is gdf4.cf.
6.2.3
1
quantiles
6.2.3 Quantiles
The syntax of the quant command is
quant (
fmt=...,
df=...,
mppar=...,
print format, def. 7.2
output file
create matrix with quantiles
) = varlist;
The command calculates quantiles corresponding to the orders
0.1
0.2
0.25
0.3
0.4
0.5
0.6
0.7
0.75
0.8
0.9
for the variables given on its right-hand side. The print format can be
controlled with the fmt parameter, default is fmt=7.2. The results are
written into the standard output and, additionally, to an output file if
requested with the df parameter.
The values of each variable, say X, are first sorted in ascending order,
x1 ≤ x2 ≤ . . . , ≤ xn . The pth order quantile is then calculated in the
following way:
a) If p ≤ 1/(n + 1), then: x1
b) If p ≥ n/(n + 1), then: xn
c) Otherwise: (1 − (q − i))xi + (q − i)xi+1
Where q = p (n + 1) and i = b q c denotes the largest integer not
greater than q.
The mppar parameter has syntax
mppar = matrix name,
This creates a matrix with the specified name. The number of rows equals
the number of variables. The number of columns is 11; each row contains
the 11 quantiles for each variable.
d060203.tex
December 12, 2001
6.2.4
1
frequency tables
6.2.4 Frequency Tables
The commands freq, freq1, and freq2, can be used to print frequency
tables. In order to calculate these tables TDA uses an algorithm developed by Leathers [1977]. The syntax is shown in the following box.
freq (
maxcat=...,
fmt=...,
sc=1,
tfmt=...,
df=...,
maximum number of categories, def. 1000
print format, def. 3.0
contingency measures (only freq2)
print format for contingency measures, def. 12.4
output file
) = varlist;
1. All variants of the freq command require a variable list on the righthand side. The freq command creates a joint frequency distribution for
all variables given on the right-hand side. The freq1 command creates
a separate frequency distribution for each of the variables. The freq2
command expects exactly two variable on the right-hand side and creates
a full two-dimensional table.
2. All variants of the freq command expect integer-valued variables. If
the cwt command has been used to define case weights the commands
create weighted frequency distributions.
3. The commands need to know in advance the maximum number of
categories in the (joint) frequency table. This can be controlled with the
maxcat parameter, default is maxcat=1000.
4. When using the freq or freq1 command, the print format for the
values of the variables can be controlled with the fmt parameter, default
is fmt=3.0 (must be an integer format).
5. The sc=1 parameter can be used with the freq2 command to request
the calculation of some contingency measures, see 6.2.7.
6. The frequency tables are written into the standard output. If an
output file is specified with the df parameter, they are also written into
d060204.tex
December 18, 1999
6.2.4
frequency tables
2
Box 1 Command file ds2.cf
nvar(
dfile = ds1.dat,
X1 = c1,
X2 = c2,
X3 = c3,
);
freq (
df = f.out,
) = X1,X2,X3;
Box 2 Part of standard output from ds2.cf
> freq(df=f.out,)=X1,X2,X3
----------------------------------------------Frequency tables. Current memory: 144678 bytes.
Maximum number of categories: 1000
Frequency distribution for variable(s): X1,X2,X3
Number of categories: 4
Index
X1 X2 X3 Frequency
Pct
Cumulated
Pct
-----------------------------------------------------1
1
1 -1
1.00 25.00
1.00 25.00
2
2
7 -1
1.00 25.00
2.00 50.00
3
3
1 -1
1.00 25.00
3.00 75.00
4
4
7
2
1.00 25.00
4.00 100.00
-----------------------------------------------------Sum
4.00 100.00
that output file.
Example 1 To illustrate the freq commands, we use again the data
file ds1.dat (see 6.2.1). The command file is ds2.cf, see Box 1. The
result of the freq command is shown in Box 2. In this example, there
is a three-dimensional frequency distribution. The first column, labelled
Index, counts the rows of the table; its last value is identical to the
number of different table elements. But note that only table elements
with a positive frequency are displayed. Then follow the values of the
variables identifying the table elements, and the corresponding frequencies, given in absolute and percentage values; and finally the table shows
the cumulated frequencies, again in absolute and percentage values. In
this example, the table is also written into the output file f.out. Box 3
6.2.4
frequency tables
3
Box 3 Standard output from command freq2=X2,X1
Frequency distribution for variable(s): X2 X1
Number of categories: 4
Frequency|
Percent |
Row Pct |
Col Pct |
1 |
2 |
3 |
4 |
Total
---------------------------------------------------------1 |
1.00 |
0.00 |
1.00 |
0.00 |
2.00
|
25.00 |
0.00 |
25.00 |
0.00 |
50.00
|
50.00 |
0.00 |
50.00 |
0.00 |
| 100.00 |
0.00 | 100.00 |
0.00 |
---------------------------------------------------------7 |
0.00 |
1.00 |
0.00 |
1.00 |
2.00
|
0.00 |
25.00 |
0.00 |
25.00 |
50.00
|
0.00 |
50.00 |
0.00 |
50.00 |
|
0.00 | 100.00 |
0.00 | 100.00 |
---------------------------------------------------------Total
1.00
1.00
1.00
1.00
4.00
25.00
25.00
25.00
25.00
100.00
illustrates the format of the frequency table if we use the freq2, instead
of the freq command.
6.2.5
1
aggregated tables
6.2.5 Aggregated Tables
For aggregating the values of a variable, and in particular for creating
histograms, one can use the atab command with syntax shown in the
following box.
atab (
x=...,
fmt=...,
s=1,
r=1,
mppar=...,
definition of classes, no default
print format, def. 10.4
classes left open, def. s=0 (right open)
print only nonempty classes
write table into matrix
) = varlist;
varlist on the right-hand side must specify one, say X, or two, say X
and Y , variables. Also required is the x parameter to specify a partition
of the real line into classes for the X variable. The syntax is
x = x1 , x2 , . . . , xn
This defines n + 1 classes: (−∞, x1 ),[x1 , x2 ), . . . , [xn , ∞). Another possible syntax is
x = x1 (d)x2
defining classes (−∞, x1 ),[x1 + id, x1 + (i + 1)d), . . . , for i = 0, 1, . . . until
x2 is reached; an upper open class [x2 , ∞) is added. By default, classes
are closed on the left side and open on the right side. Using s=1 chooses
classes which are open on the left and closed on the right side.
Given such a partition of the real line, the program counts the frequency of X and calculates its mean for each of these classes. If the
command provides a second variables, Y , then also the means of Y with
respect the classes defined by the X variable are calculated. Weighted
means will be used if case weights have been defined with the cwt command. Printing begins with the first non-empty class and ends with the
last non-empty class. By default, the output table contains all classes in
between. To suppress empty classes, one can set the r=1 parameter.
d060205.tex
December 18, 1999
6.2.5
2
aggregated tables
Box 1 Example of an aggregated table
Aggregated table. Current memory: 148696 bytes.
X: X3
Y: X4
Minimum:
Minimum:
-1.0000
0.0001
Maximum:
Maximum:
2.0000
0.1000
Lower
Upper Frequency
Weighted
Mean(X)
Mean(Y)
-----------------------------------------------------------------1.0000
-0.5000
3
3.0000
-1.0000
0.0370
-0.5000
0.0000
0
0.0000
0.0000
0.0000
0.0000
0.5000
0
0.0000
0.0000
0.0000
0.5000
1.0000
0
0.0000
0.0000
0.0000
1.0000
1.5000
0
0.0000
0.0000
0.0000
1.5000
2.0000
0
0.0000
0.0000
0.0000
2.0000
2.5000
1
1.0000
2.0000
0.0001
The mppar parameter has syntax
mppar = matrix name,
This creates a matrix with the specified name which contains the table
of aggregated values.
Example 1 To provide an illustration, we add the command
atab (x=-10(0.5)10) = X3,X4;
to command file ds1.cf (see 6.2.1). The result is shown in Box 1.
6.2.6
1
covariance and correlation
6.2.6 Covariance and Correlation
The cov and corr commands can be used to calculate a covariance or
correlation matrix, respectively. The syntax is shown in the following
box.
cov (
fmt=...,
df=...,
prn=...,
mpcov=...,
print format, def. 7.4
output file
data format in output file, def. 0
create matrix
) = varlist;
corr (
fmt=...,
df=...,
prn=...,
mpcov=...,
print format, def. 7.4
output file
data format in output file, def. 0
create matrix
) = varlist;
prn
prn
prn
prn
prn
=
=
=
=
=
0
1
2
3
4
lower triangle and main diagonal
full square matrix
column vector: lower triangle
column vector: lower triangle and main diagonal
column vector: full matrix
If no list of variables is given on the right-hand side, the commands use
all currently defined variables. The lower triangle, including the main
diagonal, of the resulting covariance or correlation matrix is written into
the standard output. The print format can be controlled with the fmt
parameter. Default is fmt=7.4. If the name of an output file is specified
with the df parameter, the matrix is also written into that file. The data
format depends on the prn parameter as indicated in the box above.
Calculation of the covariance matrix uses the formula
cov(x, y) = Σi wi (xi − x̄)(yi − ȳ)/(Σi wi − 1)
d060206.tex
December 18, 1999
6.2.6
covariance and correlation
2
Box 1 Standard output from corr command
Correlation matrix: corr
X1
X2
X3
X4
-------------------------------------X1
1.0000
X2
0.4472 1.0000
X3
0.7746 0.5774 1.0000
X4
-0.8241 -0.5426 -0.3815 1.0000
with wi optional case weights, if defined with the cwt command. Default
is wi = 1. Calculation of the correlation matrix uses the formula:1
Σi wi (xi − x̄)(yi − ȳ)
corr(x, y) = p
Σi wi (xi − x̄)2 Σi wi (yi − ȳ)2
The mppar parameter has syntax
mppar = matrix name,
This creates a matrix with the specified name which contains the covariance, or correlation, matrix.
Example 1 To provide an illustration, we add the corr command,
without parameters, to command file ds1.cf (see 6.2.1). Box 1 shows
the standard output.
1 The
correlation coefficients are actually calculated in a two-step procedure, according to algorithm P5 in Neely [1966, p. 498].
6.2.7
contingency measures
1
6.2.7 Contingency Measures
The sc=1 parameter in the freq2 command (see 6.2.4) allows to request
a variety of contingency measures. Calculation is done as follows.
Assume a contingency table (fij ) with r rows and c columns (when case
weights are defined with the cwt command these are weighted frequencies) and define ri = Σj fij , cj = Σi fij , n = Σij fij , and eij = ri cj /n.
Calculations are only done with positive degrees of freedom: (r−1)(c−1).
Pearson’s χ2P is calculated as
χ2P = Σij
(fij − eij )2
eij
The likelihood ratio χ2LR is calculated as
fij
χ2LR = 2 Σij fij log
eij
In both cases, prob is the corresponding value of the χ2 distribution
with (r − 1)(c − 1) degrees of freedom. Additional coefficients are calculated as follows (we use the formulas given in SPSS [1991], CROSSTABS
procedure):
r
χ2P
Phi =
n
s
χ2P
Cramer’s V =
n(min{r, c} − 1)
s
χ2P
Contingency coefficient =
2
χP + n
The Lambda coefficients are calculated as
λY |X =
Σi maxj {fij } − maxj {cj }
n − maxj {cj }
λX|Y =
Σj maxi {fij } − maxi {ri }
n − maxi {ri }
d060207.tex
April 21, 1998
6.2.7
contingency measures
λSY M =
2
Σi maxj {fij } + Σj maxi {fij } − maxj {cj } − maxi {ri }
2n − maxj {cj } − maxi {ri }
Goodman and Kruskal’s τ is calculated as
τX|Y =
nΣij fij /cj − Σi ri2
n2 − Σi ri2
τY |X =
nΣij fij /ri − Σj c2j
n2 − Σj c2j
The uncertainty coefficients are
UX|Y =
Ux + Uy − Uxy
Ux
and UY |X =
Ux + Uy − Uxy
Uy
with
r ri
i
log
n
n
c cj
j
Uy = −Σj log
n
n
fij
fij
log
Uxy = −Σij
n
n
Ux = −Σi
Calculation of Kendall’s τ and related measures uses the following quantities:
Dr = n2 − Σi ri2
and Dc = n2 − Σj c2j
P = Σij fij (Σh<i Σk<j fhk + Σh>i Σk>j fhk )
Q = Σij fij (Σh<i Σk>j fhk + Σh>i Σk<j fhk )
Then
P −Q
Kendall’s τb = √
Dr Dc
Kendall’s τc =
Gamma =
min{r, c}(P − Q)
n2 (min{r, c} − 1)
P −Q
P +Q
P −Q
Somers’ DX|Y = √
Dc
P −Q
and DY |X = √
Dr
6.2.7
3
contingency measures
Box 1 Illustration of contingency measures
Contingency measures for table: X (X2) x Y(X1)
Number of cells with less than 5 elements:
Cells with expected frequency less than 5:
Degrees of freedom: 3
8 (100.0 %)
8 (100.0 %)
Chi-square (Pearson)
Chi-square (likelihood ratio)
4.0000
5.5452
Phi
Cramer’s V
Contingency coefficient
Lambda (X dependent)
Lambda (Y dependent)
Lambda (symmetric)
Goodman/Kruskal Tau (X dependent)
Goodman/Kruskal Tau (Y dependent)
Uncertainty coeff (X dependent)
Uncertainty coeff (Y dependent)
Kendall’s tau-b
Kendall’s tau-c
Gamma
Somers’ D (X dependent)
Somers’ D (Y dependent)
1.0000
1.0000
0.7071
1.0000
0.3333
0.6000
1.0000
0.3333
1.0000
0.5000
0.4082
0.5000
0.5000
0.3333
0.5000
prob: 0.2615
prob: 0.1360
Example 1 To illustrate, we use the command
freq2 (sc=1) = X2,X1;
with data file ds1.dat (see 6.2.1), the command file is ds2.cf. Part of
the standard output is shown in Box 1.
6.2.8
1
scatterplots
6.2.8 Scatterplots
This section describes the scplot command that can be used to create
scatterplots, optionally with a regression curve. The syntax is shown in
the following box.
scplot (
opt=...,
s=...,
fs=...,
nn=...,
fss=...,
lt=...,
lw=...,
sig=...,
d=...,
ns=...,
df=...,
fmt=...,
sel=...,
nc=...,
type of plot, def. 1
1 = only symbols
2 = sunflower plot
3 = lowess regression
symbol type for opt=1, def. 0
symbol size for opt=1, def. 2 (mm)
grid definition for opt=2, def. nn=1,1
symbol size for opt=2, def. 2 (mm)
line type for regression lines, def. 1
line width for regression lines, def. 0.2 (mm)
smoothing factor for opt=3, def. 0.5
smoothing factor for opt=3, def. 0.5
smoothing factor for opt=3, def. 0.5
output from lowess regression
print format for df option, def. 10.4
case select option
no clipping if nc=1, def. 0
) = X,Y;
The command requires that a PostScript plot environment is available,
see 4.1. The commands also expects two variable names on the righthand side asumed to contain the x and y coordinates of the data points,
respectively. All other parameters are optional. By default, the command
uses all cases in the currently active data matrix. A subset of cases can
be selected with the
sel = expression,
Then only those data matrix rows are used where expression has a
nonzero value. Also by default, the command will only plot inside the
d060208.tex
April 21, 1998
6.2.8
2
scatterplots
Box 1 Command file scplot1.cf
psfile = scplot1.ps;
psetup(
pxlen = 90,
pylen = 50,
pxa
= 0,3,
pya
= -1,2.5,
);
plxa (sc=1,ic=10);
plya (sc=1,ic=0);
output file
nvar(
noc = 200,
X = rd(0,3),
Y = sin(X) + rd,
);
scplot(
s = 5,
fs = 1,
) = X,Y;
# create some random data
#
#
#
#
length of x axis
length of y axis
user coordinates
user coordinates
in
in
on
on
mm
mm
x axis
y axis
plot x axis
plot y axis
# using default opt = 1
# symbol type
# symbol size
bounding box of the current coordinate system. The nc=1 parameter can
be used to allow for plots outside this box.
1. If opt=1 (default) the command plots a separate symbol for each data
point. This requires that a valid symbol is selected with the s parameter.
For available symbols see 4.4.2. The symbol size can be modified with
the fs parameter.
Command file scplot1.cf (Box 1) illustrates this option with some
random data. The s parameter selects the symbol type and fs=1 sets
the symbol size to 1 mm. Figure shows the resulting plot.
2. If the number of data points is large one sometimes gets a better
view of their density by using sunflower plots, see, e.g., Chambers et
al. [1983, p. 107], Schnell [1994, p. 94]. This option is selected with the
opt=2 parameter. In addition, one can use the nn and fss parameters.
The parameter
nn = nx , ny ,
specifies the grid. The x axis is partitioned into nx intervals, the y axis
into ny intervals. The number of data points contained in each of the
resulting cells is represented by a single sunflower symbol that shows the
number of data points. The default size is 2 mm. The fss parameter can
6.2.8
3
scatterplots
2
1
0
-1
0
1
2
3
Figure 1 PostScript plot scplot1.ps, created with command
file scplot1.cf shown in Box 1.
2
1
0
-1
0
1
2
3
Figure 2 PostScript plot scplot2.ps (sunflower plot), created
with command file scplot2.cf.
be used to specify another size.
To illustrate this option we use command file scplot2.cf, basically
identical with scplot1.cf, but the scplot command is now
scplot(opt=2, nn=15,10, fss=3) = X,Y;
6.2.8
4
scatterplots
2
1
0
-1
0
1
2
3
Figure 3 PostScript plot scplot3.ps (lowess regression
curve) created with command file scplot3.cf.
Figure 2 shows the resulting plot.
3. The opt=3 parameter can be used to add a regression curve to the scatterplot. The regression curve is calculated with the lowess algorithm
proposed by W. S. Cleveland, a procedure for robust locally weighted regression. For an introduction see Chambers et al. [1983, p. 121]. TDA uses
the algorithm developed by W. S. Cleveland (distributed via the Internet
NETLIB). The algorithm can be controlled with three parameters.
1. σ, specified with sig=σ, should be a number between 0 and 1 and
determines the fraction of data points that is used to compute each
fitted value. As σ increases also the amount of smoothing increases,
and computation will take more time. By default, σ = 0.5.
2. ns , specified with ns=ns , determines the number of iterations in the
robust fit. If ns = 0 (default) there will be no iterations and the
algorithm calculates a “non-robust” regression curve.
3. δ, specified with d = δ, can be used to save computations. If δ = 0
(default) the algorithm performs complete computations. If δ > 0 it
only uses data points that are spaced (about) δ apart. Remaining
values are then calculated by linear interpolation.
To illustrate we use command file scplot3.cf. Again, basically identical
6.2.8
scatterplots
5
with scplot1.cf, but the scplot command is now
scplot(opt=3, lw=0.3, s=5, fs=1) = X,Y;
opt=3 selects the lowess procedure with default parameters. lw=0.3
sets the line width to 0.3 mm. We have added the s=5 and fs=1 parameters to include a standard scatterplot. Otherwise, the command
would only plot the regression curve. (Of course, one can use the scplot
command several times, with different options, for the same PostScript
output file; and also add other plot objects.) Figure 3 shows the resulting
plot.
6.2.9
1
histogram plots
6.2.9 Histogram Plots
This section describes the ploth command that can be used to create
and plot histograms. The syntax is shown in the following box.
ploth (
x=...,
w=...,
dscal=...,
s=...,
ns=...,
lt=...,
lw=...,
gs=...,
nc=...,
df=...,
fmt=...,
intervals (required)
variable for weights
scaling factor, def. 1.0
type of intervals, def. 0
0 : intervals are left closed
0 : intervals are right closed
option for vertical lines, def. 0
0 : plot vertical lines
0 : don’t plot vertical lines
line type, def. 1
line width, def. 0.2 (mm)
grey scale value, def. 1 (white)
no clipping if nc=1, def. 0
print histogram data to output file
print format for df option, def. 10.4
) = name of a variable;
The command requires a valid variable name on the right-hand side and
the specification of intervals with the x parameter which has syntax:
x = x1,x2,...,xn, or x = x1(d)xn,
It is required that x1 is not greater than the smallest value of the variable
and xn is not smaller than the greatest value of the variable. All other
parameters are optional. If the df option is used, the command writes
the following values into the output file:
1. Begin of interval
2. End of interval
3. Number of cases in interval
d060209.tex
December 22, 1999
6.2.9
2
histogram plots
Box 1 Command file dh1.cf
nvar(
dfile = dh1.dat,
X = c1,
);
psfile = dh1.ps;
psetup(
pxa = 0,8,
pya = 0,0.4,
);
plxa(sc=1);
plya(sc=0.1);
ploth(
x = 0,2,4,6,8,
df = df,
) = X;
dh1.dat
------1
3
4
5
4
7
1
0.4
0.3
0.2
0.1
0
0
1
2
3
4
5
6
7
8
Figure 1 PostScript plot dh1.ps, created with command file
dh1.cf shown in Box 1.
4. Frequency of these cases
5. Height of histogram box
As an illustration, the command file dh1.cf, shown in Box 1, creates the
plot shown in Figure 1.
6.2.10
1
density plots
6.2.10 Density Plots
This section describes the plotd command that can be used to plot a
density function. The syntax is shown in the following box.
plotd (
x=...,
d=...,
k=...,
dscal=...,
lt=...,
lw=...,
gs=...,
nc=...,
df=...,
fmt=...,
sequence of evaluation points (required)
bandwidth, def. 1.0
selection of kernel, def. 1
1 : uniform
2 : triangle
3 : quartic
4 : Epanechnikov
scaling factor, def. 1.0
line type, def. 1
line width, def. 0.2 (mm)
grey scale value, def. 1 (white)
no clipping if nc=1, def. 0
print histogram data to output file
print format for df option, def. 10.4
) = name of a variable;
The command requires a valid variable name on the right-hand side and
the specification of a sequence of evaluation points with the x parameter
which has syntax:
x = x1,x2,...,xn, or x = x1(d)xn,
For each point x specified with the x parameter the command calculates
m
1 X
Kd (x, yi )
fˆ(x) =
n i=1
Here it is assumed that y1 , . . . , ym are the values of the variable specified
on the right-hand side of the command. A curve connecting the values
of fˆ(x) is plotted in the current coordinate system. If the df option is
used the x and fˆ(x) values are also written into the specified output file.
d060210.tex
December 22, 1999
6.2.10
2
density plots
Box 1 Command file dh2.cf
nvar(
dfile = dh1.dat,
X = c1,
);
psfile = dh2.ps;
psetup(
pxa = 0,8,
pya = 0,0.6,
);
plxa(sc=1);
plya(sc=0.1);
plotd(
k = 2,
x = 0(0.1)8,
df = df,
) = X;
dh1.dat
------1
3
4
5
4
7
1
0.6
0.5
0.4
0.3
0.2
0.1
0
0
1
2
3
4
5
6
7
8
Figure 1 PostScript plot dh2.ps, created with command file
dh2.cf shown in Box 1.
The user has a choice between four kernels. For a description of these
kernels see 6.10.2.
As an illustration, the command file dh2.cf, shown in Box 1, creates the
plot shown in Figure 1.
6.4
concentration and inequality
1
6.4 Concentration and Inequality
This chapter is intended to describe commands that can be used to
calculate concentration and inequality measures. Currently, we have only
two sections.
6.4.2
Inequality Measures
6.4.3
Segregation Indices
d0604.tex
April 21, 1998
6.4.2
1
inequality measures
6.4.2 Inequality Measures
In order to calculate some inequality measures one can use the ineq
command with syntax shown in the following box.
ineq (
fmt=...,
df=...,
mppar=...,
print format, def. 10.4
output file
write parameter into matrix
) = varlist;
The command requires a list of variables on the right-hand side. The
other parameters are optional. The command creates a table, one line
for each variable specified in varlist, containing the following entries:
1. Name of the variable.
2. Number of valid cases. Note that the command uses only nonnegative
values of a variable; negative values are always regarded as missing
values.
3. Minimum value.
4. Maximum value.
5. Mean value:
M(x) = Σi wi xi /w
with w = Σi wi
xi are the valid (nonnegative) values of the variable; wi are case
weights which can be provided with the cwt command; otherwise all
wi = 1.
6. Standard deviation:
q
S(x) = Σi wi (xi − M(x))2 ((Σi wi ) − 1)
7. The variation coefficient, defined as V(x) = S(x)/M(x).
d060402.tex
December 18, 1999
6.4.2
inequality measures
2
Box 1 Example of inequality measures
Inequality measures. Current memory: 148696 bytes.
Variable Cases Minimum
Maximum
Mean
StdDev
VCoeff
Gini
--------------------------------------------------------------------X1
4
1.0000
4.0000
2.5000
1.2910
0.5164
0.3333
X2
4
1.0000
7.0000
4.0000
3.4641
0.8660
0.5000
X3
1
2.0000
2.0000
2.0000
0.0000
0.0000
0.0000
X4
4
0.0001
0.1000
0.0278
0.0484
1.7410
0.9262
8. The Gini coefficient,1 calculated as
G(x) =
1
Σi Σj<i wi wj | xi − xj |
2 M(x)
Σi Σj<i wi wj
The fmt parameter can be used to change the print format for the table
entries; default is fmt=10.4. If the name of an output file is provided
on the right-hand side of the command, a copy of the table (without
header) is written into that file.
The mppar parameter has syntax
mppar = matrix name,
This creates a matrix with the specified name. The number of rows
equals the number of variables. Each row contains the quantities for the
corresponding variable.
Example 1 To provide an illustration, we add the command
ineq = X1,,X4;
to command file ds1.cf (see 6.2.1). The result is shown in Box 1.
1 See,
for instance, Kendall and Stuart [1977, p. 48].
6.4.3
segregation indices
1
6.4.3 Segregation Indices
The segr command can be used to request the calculation of segregation
indices. For a discussion of such indices see, e.g., James and Taeuber
[1985], Deutsch et al. [1994]. The syntax of the command is shown in the
following box.
segr (
g=...,
fmt=...,
df=...,
ptab=...,
name of variable defining groups
print format, def. 10.4
output file
additional output file
) = varlist;
To explain this command, assume a sample of n individuals (cases) in
the current data matrix, and a variable G providing information about
membership of the individuals in one of two groups:

 = 0 if individual i is in group 0
> 0 if individual i is in group 1
Gi =

< 0 if no information (missing values)
This variable G must be provided with the g parameter in the segr
command. Further assume a set of classes (for instance, occupations),
and each individual belonging to one of these classes, indicated by an
integer variable X: individual i belongs to class j if Xi = j ≥ 0 (otherwise
there is a missing value). One or more of these X variables must be
specified on the right-hand side of the segr command.
Given these variables, the segr command creates a table, one line
for each X variable, containing the following entries:
1. Name of the X variable.
2. Number of classes, that is, the number of different nonnegative integer values in variable X.
3. Number of valid cases: N = number of cases with nonnegative values
in G and X.
d060403.tex
April 21, 1998
6.4.3
segregation indices
2
Box 1 Data file ds2.dat
G X1 X2 X3 X4
------------0 10 1 0 1
1 0 2 0 1
0 10 3 1 1
1 0 4 1 2
0 10 5 2 2
1 0 6 -1 2
1 0 7 2 2
4. Number of cases in group 0: F = number of valid cases with Gi = 0.
5. Number of cases in group 1: M = number of valid cases with Gi > 0.
6. The dissimilarity index, calculated as
Fj
Mj Σj Nj | Fj /Nj − F/M |
−
= 0.5
D = 0.5 Σj F
M F M/N
where Fj = number of valid cases belonging to group 0 and class j,
Mj = number of valid cases belonging to group 1 and class j, and
N j = Fj + Mj .
7. The variance ratio (James and Taeuber 1985, p. 6), calculated as
VR =
Σj Nj (Mj /Nj − M/N )2
Σj Nj (Fj /Nj − F/N )2
=
F M/N
F M/N
8. The Gini index (James and Taeuber [1985, p. 5], Deutsch et al. [1994,
p. 134]), calculated as
Fi
0.5
Fj G =
Σi Σj Ni Nj −
FM
Ni
Nj The table containing these entries is always written into the standard
output. If the name of an output file is given with the df parameter,
a copy of the table is written into that file. The fmt parameter can be
used to change the print format, default is fmt=10.4.
In addition, one can specify the name of a second output file with the
ptab parameter. For each X variable, it will contain a separate table
with the following entries:
6.4.3
segregation indices
3
Box 2 Command file ds6.cf
nvar(
dfile = ds2.dat,
G = c1,
X1 = c2,
X2 = c3,
X3 = c4,
X4 = c5,
);
segr(
g=G,
ptab = tab,
) = X1,,X4;
Box 3 Part of output from command file ds6.cf
Segregation measures. Current memory: 152713 bytes.
Grouping variable: G
Variable Classes Cases Group_0 Group_1 D-Index V-Ratio
Gini
--------------------------------------------------------------------X1
2
7.00
3.00
4.00
1.0000
1.0000
1.0000
X2
7
7.00
3.00
4.00
1.0000
1.0000
1.0000
X3
3
6.00
3.00
3.00
0.0000
0.0000
0.0000
X4
2
7.00
3.00
4.00
0.4167
0.1736
0.4167
1. Number of classes, based on the current X variable.
2. Number of valid cases.
3. Number of cases in group 0.
4. Number of cases in group 1.
5. Contribution of the class to the dissimilarity index,
that is, | Fj /F − Mj /M | .
Example 1 To provide an illustration, we use data file ds2.dat, shown
in Box 1, and the command file ds6.cf (Box 2). Part of the standard
output is shown in Box 3.
6.5
describing episode data
1
6.5 Describing Episode Data
This chapter discusses some elementary descriptive procedures for episode
data. They require a valid episode data structure, see 3.3.2.
6.5.1
Life Tables
6.5.2
Kaplan-Meier Procedure
6.5.3
Survivor Function Quantiles
6.5.4
Comparing Survivor Functions
6.5.5
State Distributions
d0605.tex
April 21, 1998
6.5.1
life tables
1
6.5.1 Life Tables
The life table method allows to calculate nonparametric estimates of the
survivor function, the density function, and transition rates, for durations given in a set of episodes. (An extensive discussion of this method
has been given by Namboodiri and Suchindran [1987]; a compilation of
all commonly used formulas for the single transition case may be found in
Smith et al. [1970].) There are two drawbacks because it is necessary to
group the durations according to some intervals on the time axis. First,
results depend more or less on arbitrarily defined time intervals; and second, the method should only be used if there is a relatively large number
of episodes so that estimates conditional for each interval are reliable.
Therefore, in most applications one normally prefers the Kaplan-Meier
procedure discussed in 6.5.2.
Given a valid definition of single episode data as explained in 3.3.2, one
can estimate life tables with the ltb command. The syntax is explained
in Box 1. The command needs the specification of an output file on the
right-hand side. The life tables are then written into this output file.
The print format can be specified with the fmt parameter; default is
fmt=7.5.
Time Periods. Life table estimation, like some other methods for
episode data, requires a specification of time periods (intervals) for the
process time axis that was used to define the episode data. This is done
by defining split points on the time axis
0 ≤ τ1 < τ2 < τ3 < . . . < τq
With the convention that τq+1 = ∞, there are q time intervals each
including the left limit, but not the right one.
Il = {t | τl ≤ t < τl+1 }
l = 1, . . . , q
Given these time intervals, calculation of life tables by TDA is always
done using episode durations. (This implies that the method cannot be
used with split episode data.) In the following discussion we will therefore
assume that all episodes have equal starting times at the origin, zero, of
a common time axis. In addition we will assume that the time intervals
start at zero, i.e. τ1 = 0.
d060501.tex
April 21, 1998
6.5.1
life tables
2
Box 1 Syntax for ltb command
ltb (
time periods, no default
fraction of censored cases that are counted
as part of the risk set, def. 0.5
grp=...,
group indicator variables, def. without groups
fmt=...,
print format for output file, def. fmt=7.5
ap=1,
append data to end of output file
) = name of output file;
tp=...,
cfrac=...,
For the definition of time periods one must use the tp parameter in the
ltb command. The syntax is
tp = t1,t2,t3,...,tq,
where t1,t2,t3,...,tq are non-negative numbers in a strictly ascending order. If the first time point, t1, is greater than zero, TDA inserts
an additional interval beginning at zero and ending at t1. Alternatively,
one can use the syntax
tp = t1 (d) t2,
The expression is then expanded to provide the sequence t1, t1 + d,
t1 + 2d,..., until t2. It is possible to combine both ways of defining
time periods.
Given a valid definition of time periods, the calculation of life tables
depends on the input data.
1. If there are case weights defined with the cwt command, these weights
are used with all calculations. In particular with large data sets where
durations are heavily tied this is an useful option.
2. If the input data are partitioned into groups (see below), life table
estimation is done separately for the episodes in each group.
3. If there is more than one origin state, life table calculation is done
separately for each subset of episodes having the same origin state. Consequently, life table calculation is always conditional on a given origin
state.
6.5.1
life tables
3
4. If for a given origin state there is only a single destination state,
an ordinary life table is calculated; if there are two or more destination
states a so-called multiple-decrement life table is calculated.
It should be stressed that TDA always uses durations, calculated as the
difference between starting and ending times. Consequently, episode data
that have been split with the method of episode splitting cannot be used
as input for life table estimation. If one or more starting times in the
input data are not zero, TDA gives a warning message that the results
will be probably wrong.
Grouping Episodes. It is possible to partition a set of episodes into
groups. This can be done with the optional grp parameter in the ltb
command. The syntax is
grp = G1,G2,...,
where G1,G2,... are names of variables. These variables are then interpreted as indicator variables defining groups. Each of the variables
defines a separate group consisting of all episodes (data matrix rows)
where the variable has a value not equal to zero.
Single Transitions. All formulas used in the calculation of single transition life tables are based on the quantities Il (l = 1, . . . , q).
El = the number of episodes with events in Il
Zl = the number of censored episodes ending in Il
defined for the intervals (time periods) Il (l = 1, . . . , q). The next important point is the definition of a risk set, Rl , for each of the time
intervals, that is the set of units (episodes) that are at risk of having
an event during the lth interval.1 To take into account episodes that are
censored during the interval this is done in two steps. First the number
of episodes, Nl , that enter the lth interval, is defined recursively by
N1 = N,
Nl = Nl−1 − El−1 − Zl−1
In a second step one has to decide how many of the episodes that are
censored during an interval should be contained in the risk set for that
interval. We shall assume a constant ω (0 ≤ ω ≤ 1) for the definition
1 We
generally denote the risk set by the symbol R, the number of units contained
in the risk set by the symbol R.
6.5.1
life tables
4
of the fraction of censored episodes that should be contained in the risk
set. The default value is ω = 0.5, but can be changed with the cfrac
parameter.2
The number of elements in the risk set is defined, then, by
Rl = Nl − ω Zl
Using these basic quantities, it is easy to define all other concepts used
in the life table setup. First the conditional probabilities of having an
event in the lth interval, ql , and for surviving the interval, pl , are
ql =
El
Rl
and pl = 1 − ql
As an implication, one gets the following estimator for the survivor function3
G1 = 1,
Gl = pl−1 Gl−1
Having estimates of the survivor function, the density function is evaluated approximately at the midpoints of the intervals as the first derivative
fl =
Gl − Gl+1
τl+1 − τl
l = 1, . . . , q − 1
Of course, if the last interval is open on the right side, it is not possible
to calculate the survivor function for this interval. Also, estimates of the
transition rate, rl , are calculated at the midpoints of the intervals. They
are defined by
rl =
fl
Ḡl
where
Ḡl =
Gl + Gl+1
2
and this can also be written as
rl =
2 Cf.
1
ql
1
El
=
τl+1 − τl 1 − ql /2
τl+1 − τl Rl − El /2
the discussion given by Namboodiri and Suchindran [1987, p. 58].
that the survivor function is calculated at the begin of each interval. Most
programs use this convention in the printout of life tables. An exception is SPSS; in
the life table output of SPSS the survivor function is given at the end of each interval.
3 Note
6.5.1
life tables
5
Box 2 Command file ltb1.cf for life table estimation
... reading input data file rrdat.1
... set up episode data with edef command
ltb (
tp = 0 (30) 500,
) = ltb.1;
# time periods: 0,30,60,...,300
Finally, it is possible to approximately calculate standard errors for the
estimates of the survivor and density function, and of the transition
rates, by the formulas
" l−1
#1/2
X qi
SE (Gl ) = Gl
p R
i=1 i i
q l Gl
SE (fl ) =
τl+1 − τl
rl
SE (rl ) = √
ql Rl
"
" l−1
#1/2
X qi
pi
+
pR
qi Ri
i=1 i i
rl (τl+1 − τl )
1−
2
2 #1/2
With large samples it may be assumed that the values of the survivor,
density and rate function, divided by their standard errors are approximately standard normally distributed. It is possible, then, to calculate
confidence intervals.
Example 1 To illustrate life table estimation, we use the example data
discussed in 3.3.3. An example command file, ltb1.cf, is shown in Box
2. Its first part, reading the data file and defining single episode data, is
identical with command file ed1.cf (see 3.3.3). We simply add the ltb
command for life table estimation.
TDA’s standard output in response to the ltb command is shown in
Box 3. The table has a separate line for each life table, corresponding to
each set of episodes having the same origin state. In this example there
is only one set of episodes all having origin state 0. The table also shows
the number of episodes and an estimate of the median, calculated by
linear interpolation of the life table.
The life tables are written into the output file specified in the ltb
command. In this example, there is only a single life table. As shown in
6.5.1
life tables
6
Box 3 Part of standard output from command file ltb1.cf
Life table calculation. Current memory: 220963 bytes.
Command: ltb(tp=0(30)500,)=ltb.1
SN Org Group
Median Dest.States Episodes Weighted
-------------------------------------------------------1
0
-47.28
1
600
600.00
-------------------------------------------------------1 table(s) written to: ltb.1
Box 4, the table is in two parts. The first part shows the time intervals
and the basic quantities for each of the time intervals. The last column,
with header Prob, shows the quantities ql , estimates of the conditional
probability of having an event in the corresponding interval.
The second part shows again the time intervals and, in addition,
estimates of the survivor function, the density function and the transition
rate, each with its estimated standard errors. An estimate of the median
is calculated, if possible, by linear interpolation of the survivor function.
Example 2 If one defines two ore more groups of episodes, TDA calculates a separate life table for each group. An example is given in command
file ltb2.cf (not shown) where two groups (men and women) have been
defined.
Multiple-Decrement Life Tables. The concept of a life table can
be extended to the case of two or more transitions starting from the
same given origin state. The resulting life table is sometimes called a
multiple-decrement table.
The given origin state is, say, j ∈ O. The basic quantities are now:
Ejk,l , the number of episodes with a transition to the destination state k
in the lth time interval; Ej,l , the number of episodes with any transition
in the lth interval, and Zj,l , the number of episodes that are censored in
the lth interval.
Having these basic quantities, the number of episodes entering each
interval and the risk set for each interval are defined analogously to the
single transition case. First, the number of episodes entering an interval
may be defined again recursively by
Nj,1 = Nj ,
Nj,l = Nj,l−1 − Ej,l−1 − Zj,l−1
with Nj the set of all episodes with origin state j. The number of elements
6.5.1
7
life tables
Box 4 Output file created by ltb1.cf
# Life table. SN 1. Origin state 0.
# Cases: 600 weighted: 600
# Start of
Number
Number
Exposed
D-State 1
# Interval Midpoint Entering Censored
to Risk Events
Prob
0.00
15.00
600
28
586.0
223 0.38055
30.00
45.00
349
23
337.5
113 0.33481
60.00
75.00
213
15
205.5
51 0.24818
90.00
105.00
147
16
139.0
25 0.17986
120.00
135.00
106
15
98.5
24 0.24365
150.00
165.00
67
5
64.5
9 0.13953
180.00
195.00
53
9
48.5
4 0.08247
210.00
225.00
40
5
37.5
3 0.08000
240.00
255.00
32
5
29.5
0 0.00000
270.00
285.00
27
7
23.5
2 0.08511
300.00
315.00
18
5
15.5
2 0.12903
330.00
345.00
11
1
10.5
2 0.19048
360.00
375.00
8
3
6.5
0 0.00000
390.00
405.00
5
4
3.0
0 0.00000
420.00
435.00
1
1
0.5
0 0.00000
# Start of
Survivor
# Interval Midpoint Function Error
0.00
15.00 1.00000 0.00000
30.00
45.00 0.61945 0.02006
60.00
75.00 0.41205 0.02077
90.00
105.00 0.30979 0.01995
120.00
135.00 0.25407 0.01922
150.00
165.00 0.19217 0.01822
180.00
195.00 0.16535 0.01774
210.00
225.00 0.15172 0.01754
240.00
255.00 0.13958 0.01748
270.00
285.00 0.13958 0.01748
300.00
315.00 0.12770 0.01790
330.00
345.00 0.11122 0.01900
360.00
375.00 0.09004 0.02045
390.00
405.00 0.09004 0.02045
420.00
435.00 0.09004 0.02045
# Median duration: 47.28
D-State 1
Density
Error
0.01268 0.00067
0.00691 0.00058
0.00341 0.00045
0.00186 0.00036
0.00206 0.00040
0.00089 0.00029
0.00045 0.00022
0.00040 0.00023
0.00000
**
0.00040 0.00027
0.00055 0.00037
0.00071 0.00047
0.00000
**
0.00000
**
0.00000
**
D-State 1
Rate
Error
0.01567 0.00102
0.01340 0.00124
0.00944 0.00131
0.00659 0.00131
0.00925 0.00187
0.00500 0.00166
0.00287 0.00143
0.00278 0.00160
0.00000
**
0.00296 0.00209
0.00460 0.00324
0.00702 0.00493
0.00000
**
0.00000
**
0.00000
**
contained in the risk set for the lth episode is defined by
Rj,l = Nj,l − ω Zj,l
ω has the same meaning as in the single transition case, it is the fraction of the censored episodes assumed to be not at risk for an event
6.5.1
life tables
8
in the interval. Next, one can define estimates of the basic conditional
probabilities
qjk,l =
Ejk,l
Rj,l
qj,l =
X
qjk,l
pj,l = 1 − qj,l
k∈Dj
qjk,l is an estimate for the conditional probability of a transition to
destination state k in the lth episode, conditional on having no event
before. qj,l is the conditional probability of having any event in the lth
interval. And pj,l is the conditional probability of having no event in the
lth interval.
Using these conditional probabilities, an estimate of the overall survivor function4 is given by
Gj,1 = 1,
Gj,l = pj,l−1 Gj,l−1
Obviously, it is the same definition as in the single transition case and,
consequently, the overall survivor function could be estimated by collapsing all different destination states into a single one.
Following the remarks given in 3.3.1 about alternative destination
states, the next step should be to get estimates for the destinationspecific subdistribution and subdensity functions and for the transition
rates. To do this, we can first write
qjk,l =
F̃jk,l+1 − F̃jk,l
Gj,l
(1)
with the meaning that qjk,l may be regarded as an estimate of the righthand side. This is obvious if one remembers the definition of the subdistribution functions F̃jk , defined by
F̃jk,l = F̃jk (τl ) = Pr (Tj ≤ τl , Dj = k)
It is the probability of a transition to destination state k up to the
beginning of the lth interval. Then, with the assumption that τ1 = 0,
it follows that F̃jk,1 = 0, and a simple manipulation of (1) gives the
following estimate of the subdistribution functions
F̃jk,l =
l−1
X
qjk,i Gj,i
i=1
4 See
the discussion of this concept in 3.3.1.
6.5.1
life tables
9
Estimates of the subdensity functions f˜jk may be derived, then, by taking
an approximation to the first derivative at the midpoints of the intervals,
i.e. by
F̃jk,l+1 − F̃jk,l
f˜jk,l =
τl+1 − τl
And finally, one gets estimates of the transition rates by
rjk,l =
f˜jk,l
(Gjk,l + Gjk,l+1 )/2
All destination-specific concepts add up to accordingly defined overall
concepts which are the result of an ordinary life table estimation where
all different destination states are collapsed into a single state.
X
F̃jk,l
Fj,l =
k∈Dj
fj,l =
X
f˜jk,l
k∈Dj
rj,l =
X
rjk,l
k∈Dj
TDA commands to request a multiple-decrement life table are identical
to the single transition case. The output file containing the life table(s)
is defined with the ltb command, time periods are defined with the tp
command. In fact, whenever there are two or more different transitions
in the input data, TDA automatically calculates multiple-decrement life
tables.
Example 3 To illustrate the calculation of a multiple-decrement life
table we use our main example data with three alternative destination
states as defined in 3.3.3.5 Our example command file, ltb3.cf (not
shown), is almost identical to the command file ed2.cf that was used
in 3.3.3 to create episode data with three alternative destination states.
We simply add a ltb command to request life table estimation. TDA automatically recognizes that the input data consist of single episodes with
three alternative destination states and calculates a multiple-decrement
life table as described above.
5 Many
more examples of multiple-decrement life tables can be found in Namboodiri
and Suchindran [1987].
6.5.2
kaplan-meier procedure
1
6.5.2 Kaplan-Meier Procedure
Another method for nonparametric estimation of survivor functions and
its derivatives is the Kaplan-Meier [1958], also called product-limit method.
One of the advantages of this method, compared with the life table
method, is that it is not necessary to group the episode durations according to arbitrarily defined time intervals. Instead, the product-limit
method is based on the calculation of a risk set at every point in time
where at least one event occurred. In this way the information contained
in a set of episodes is optimally used. The only drawback of this method
results from the fact that all episodes must be sorted according to their
ending (and starting) times, but with efficient sorting algorithms the
method can be employed with fairly large sets of episodes.
This section describes the product-limit estimation method and its
implementation in TDA. The options, depending on the type of input
data, are essentially the same as with the life table method.
1. If there are sample weights defined with the cwt command, these
weights are used in all calculations.
2. If the input data are split into groups, separate product-limit estimates
are calculated for each of the groups.
3. If there is more than a single origin state, one or more product-limit
estimates are calculated for each subset of episodes having the same
origin state.
4. If there is more than a single destination state, separate product-limit
estimates are calculated for each transition found in the input data.
The basic command to request product-limit estimation is ple. The syntax is shown in Box 1. All parameters, except the name of an output file
on the right-hand side, are optional and will be explained below.
Single Transitions. We assume a sample of N episodes, all having the
same origin state and are censored or have the same destination state.
If groups are defined, it is assumed that all episodes belong to the same
group. For the moment we also assume that all episodes have starting
time zero. (This assumption is actually not necessary for product-limit
calculations with TDA. For instance, it is possible to perform productd060502.tex
April 21, 1998
6.5.2
kaplan-meier procedure
2
Box 1 Syntax for ple command
ple (
grp=,
group indicator variables, def. without groups
csf,
compare survivor functions.
qo=,
request table with quantiles, no default
qt=,
request table with quantiles, no default
fmt=,
print format for output file, def. fmt=7.5
ap=1,
append data to end of output file.
prot=,
protocol for test statistic, no default
pfmt=,
print format for protocol file, def. pfmt=-19.11
) = name of output file;
limit estimations with a set of episode that are split into parts. Results
should be exactly the same before and after splitting a set of episodes.)
The first step is to consider the points in time where at least one of the
episodes ends with an event. There are, say, q such points in time.
τ1 < τ2 < τ3 < . . . < τq
(1)
The second step is to define three basic quantities, all defined for l =
1, . . . , q, with the convention that τ0 = 0.
El = the number of episodes with events at τl
Zl = the number of censored episodes ending in [τl−1 , τl )
Rl = the number of episodes in the risk set at τl , denoted Rl ,
i.e. the number of episodes with starting time
less than τl and ending time ≥ τl
Note that the implied definition of the risk set allows the handling of
episodes with starting times greater than zero. Also note that the risk
set at τl includes episodes which are censored at this point in time. It
is assumed that a censored episode contains the information that there
was no event up to and including the ending time of the episode. As
is sometimes said, censoring takes place an infinitesimal amount to the
right of the observed ending time.
Given these quantities, the product-limit estimator of the survivor func-
6.5.2
kaplan-meier procedure
tion is defined as
Y El
Ĝ(t) =
1−
Rl
3
(2)
l:τl <t
This is a step function with steps at the points in time, τl . The commonly
used formula to calculate estimates of standard errors for the survivor
function is
#1/2
"
X
El
(3)
SE(Ĝ(t)) = Ĝ(t)
Rl (Rl − El )
l:τl <t
In addition to survivor function estimates, the product-limit method
gives a simple estimate of the cumulated transition rate.
(4)
Ĥ(t) = − log Ĝ(t)
This is, again, a step function. It is especially useful for simple graphical checks of distributional assumptions about the underlying durations.
Some examples are given below.
Unfortunately, unlike life table estimation, the product-limit method
does not provide direct estimates of transition rates. Of course, it is
possible to get estimates by numerical differentiation of Ĥ(t), but this is
currently not supported by TDA.
Example 1 To illustrate product-limit estimation we replicate an example given by Lawless [1982, p. 73].1 The data are remission times for
two groups of leukemia patients, one group treated with Drug 6-MP, the
other was given a placebo. The data file, rrdat.1, is shown in Box 2.
The first column gives the remission time in weeks. The second column is
the censoring status (0 if censored). The third column gives the number
of episodes (case weights). The fourth column is a group indicator (0 =
placebo, 1 = Drug 6-MP).
For product-limit estimation of a survivor function we use command
file ple1.cf shown in Box 3. The ple command specifies two groups.
The output file, ple.1 is shown in Box 4. It contains two tables, one
for each of the two groups. The first two columns are to simplify access
to the tables if they are used as data files. The first column assigns a
unique identification number to each table, the second column simply
counts the lines.
1 The
same data have been used by many other authors, for instance Lee [1980, p. 128].
6.5.2
kaplan-meier procedure
Box 2 Example data rrdat.2
C1
6
6
7
9
10
10
11
13
16
17
19
20
22
23
25
32
34
35
C2
1
0
1
0
1
0
0
1
1
0
0
0
1
1
0
0
0
0
C3
3
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
1
1
C4
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
C1
1
2
3
4
5
8
11
12
15
17
22
23
C2
1
1
1
1
1
1
1
1
1
1
1
1
C3
2
2
1
2
2
4
2
2
1
1
1
1
C4
1
1
1
1
1
1
1
1
1
1
1
1
Box 3 Command file ple1.cf for product-limit estimation
nvar(
dfile = rrdat.2,
DUR
CEN
WT
GRP
GRP1
GRP2
[2.0]
[1.0]
[1.0]
[1.0]
[1.0]
[1.0]
=
=
=
=
=
=
c1,
c2,
c3,
c4,
GRP[0],
GRP[1],
);
cwt = WT;
edef(
org
des
ts
tf
);
=
=
=
=
# data file
#
#
#
#
#
#
duration;
censoring status (1 = not censored)
case weight (number of individuals)
group (0 = placebo, 1 = Drug 6-MP)
group 0
group 1
# define case weights
0,
CEN,
0,
DUR,
ple (
grp = GRP1,GRP2,
) = ple.1;
#
#
#
#
origin state
destination state
starting time
ending time
# request product-limit estimation
# define two groups
# output file
4
6.5.2
5
kaplan-meier procedure
Box 4 Output file ple.1 created by command file ple1.cf
# SN 1. Transition: 0,1 - Product-Limit Estimation
# Group: GRP1
#
Number Number
# ID Index
Time Events Censored
0
0
0.00
0
0
0
1
6.00
3
0
0
2
7.00
1
1
0
3 10.00
1
1
0
4 13.00
1
2
0
5 16.00
1
0
0
6 22.00
1
3
0
7 23.00
1
0
# 0
8 35.00
0
5
# Median Duration: 22.42
# Duration times limited to: 23
# Cases: 18 weighted: 21
Exposed
to Risk
21
21
17
15
12
11
7
6
Survivor
Std.
Function Error
1.00000 0.00000
0.85714 0.07636
0.80672 0.08694
0.75294 0.09635
0.69020 0.10681
0.62745 0.11405
0.53782 0.12823
0.44818 0.13459
Cum.
Rate
0.00000
0.15415
0.21478
0.28377
0.37078
0.46609
0.62024
0.80256
# SN 1. Transition: 0,1 - Product-Limit Estimation
# Group: GRP2
#
Number
# ID Index
Time Events
1
0
0.00
0
1
1
1.00
2
1
2
2.00
2
1
3
3.00
1
1
4
4.00
2
1
5
5.00
2
1
6
8.00
4
1
7 11.00
2
1
8 12.00
2
1
9 15.00
1
1
10 17.00
1
1
11 22.00
1
1
12 23.00
1
# Median Duration: 6.12
# Cases: 12 weighted: 21
Number
Censored
0
0
0
0
0
0
0
0
0
0
0
0
0
Exposed
to Risk
21
21
19
17
16
14
12
8
6
4
3
2
1
Survivor
Std.
Function Error
1.00000 0.00000
0.90476 0.06406
0.80952 0.08569
0.76190 0.09294
0.66667 0.10287
0.57143 0.10799
0.38095 0.10597
0.28571 0.09858
0.19048 0.08569
0.14286 0.07636
0.09524 0.06406
0.04762 0.04647
0.00000
**
Cum.
Rate
0.00000
0.10008
0.21131
0.27193
0.40547
0.55962
0.96508
1.25276
1.65823
1.94591
2.35138
3.04452
**
The column labelled Time shows the points in time where at least
one event takes place. The number of events (episodes) is given in the
next column. Then follows the number of censored episodes with ending
times less than the actual value of the Time column and greater or equal
to the preceding value in the Time column. Given this information the
risk set, printed in the fifth column, is easily calculated. The last three
6.5.2
kaplan-meier procedure
6
Box 5 Part of output file ple.2
SN 1. Transition: 0,1 - Product-Limit Estimation
Number Number
Exposed Survivor
Std.
Cum.
ID Index
Time Events Censored to Risk Function Error
Rate
0
0
0.00
0
0
600 1.00000 0.00000 0.00000
0
1
2.00
2
0
600 0.99667 0.00235 0.00334
0
2
3.00
5
1
597 0.98832 0.00439 0.01175
0
3
4.00
9
2
590 0.97324 0.00660 0.02712
0
4
5.00
3
0
581 0.96822 0.00717 0.03230
0
5
6.00
10
1
577 0.95144 0.00880 0.04978
0
6
7.00
9
0
567 0.93634 0.00999 0.06578
...................................................................
0
40
41.00
3
1
277 0.50585 0.02092 0.68152
0
41
42.00
1
1
273 0.50399 0.02093 0.68519
0
42
43.00
2
0
272 0.50029 0.02094 0.69257
0
43
44.00
5
1
269 0.49099 0.02096 0.71133
0
44
45.00
1
1
263 0.48912 0.02096 0.71514
0
45
46.00
2
0
262 0.48539 0.02097 0.72281
...................................................................
0
127 312.00
1
3
16 0.11980 0.01846 2.12190
0
128 326.00
1
1
14 0.11125 0.01902 2.19601
0
129 332.00
1
2
11 0.10113 0.01980 2.29132
0
130 350.00
1
1
9 0.08990 0.02054 2.40910
0
131 428.00
0
8
Median Duration: 43.03
Duration times limited to: 428
Cases: 600 weighted: 600
columns show estimates of the survivor function, its standard errors, and
the cumulated transition rate as defined above. If possible, an estimate of
the median is calculated by linear interpolation of the survivor function.
If the last episode is censored the survivor function does not go to zero;
in this case the last observed time is printed at the end of the table.
Example 2 Command file ple2.cf (not shown) performs productlimit estimation with our main example data (rrdat.1). The command
file is basically identical to ltb1.cf (see 6.5.1), we simply substitute the
ltb command by
ple = ple.2;
Part of the output file, ple.2, is shown in Box 5.
Alternative Destination States. Product-limit estimation can easily
be generalized for the case of single episodes with two or more different
6.5.2
7
kaplan-meier procedure
1
0
0
60
120
180
240
300
Figure 1 Plot of pseudo-survivor functions for three destination
states, estimated with command file ple3.cf, based on our main
example data. Plot created with command file ple3p.cf.
destination states. Again, one beings with Nj episodes having the same
origin state j. Then, for each possible destination state k ∈ Dj , one looks
at the points in time, τjk,l , where at least one transition to destination
state k takes place. There are, say, l = 1, . . . , qjk such points in time.
Let Ejk,l denote the number of events at τjk,l , and let Rj,l denote the
risk set at the same point in time. Note that the risk set depends not on
the destination state, it is defined as in the single transition case as the
set of all episodes with origin state j, with a starting time less than τjk,l ,
and with ending time equal to or greater than τjk,l . The product-limit
estimate of the pseudosurvivor functions may then be formally defined
by
G̃jk (t) =
Y
l:τjk,l <t
1−
Ejk,l
Rj,l
(5)
Obviously, a calculation of this estimate can use the same algorithm as
in the single transition case. In the calculation for a specific destination
state one only has to treat all episodes that do not end in this destination
as if they are censored.
Example 3 Command file ple3.cf (also not shown) illustrates productlimit estimation with multiple destination states. This command file uses
our main example data rrdat.1 and specifies single episodes with three
6.5.2
kaplan-meier procedure
8
alternative destination states. The command
ple = ple.3;
creates the output file ple.3 containing three separate tables, one for
each destination state. The resulting survivor functions are illustrated
in Figure 1. (The command file for the plot is ple3p.cf, contained in
the TDA example archive). When interpreting these curves, one should
remember that they represent pseudo-survivor functions; the overall survivor function results from multiplication (not addition).
6.5.3
survivor function quantiles
1
6.5.3 Survivor Function Quantiles
Output tables from the Kaplan-Meier procedure (see 6.5.2) are often
very long. A shorter way to present most of the information is by tables that only show the estimated survivor function for some predefined
quantiles. This is easily done with the qo option in the ple command.
The syntax is
qo = o1 , o2 , . . . , om ,
where o1 , o2 , . . . , om are the orders of the quantiles. It is required that
1 > o1 > o2 > . . . > om > 0. Alternatively, one can use the syntax
qo = o1 (d)o2 ,
to specify a sequence of orders with an increment d. Given this option
in the ple command, for each transition found in the input data a table is printed summarizing the estimated survivor functions, or pseudosurvivor functions, by the requested quantiles. The quantiles are estimated by linear interpolation of the survivor function as given by the
product-limit method.
Instead of qo, one can use the qt option with syntax
qt = t1 , t2 , . . . , tm ,
The values ti are then interpreted as time points and one gets tables
with the corresponding orders. In this case, it is required that 0 ≤ t1 ≤
t2 ≤ . . . ≤ tm .
When using the qo or qt option, the name of an output file in the ple
command is optional. The tables containing the quantiles are always
written into TDA’s standard output.
d060503.tex
April 21, 1998
6.5.3
survivor function quantiles
2
Box 1 Table of quantiles, estimated with ple3.cf
Survivor
Time
SN Org Des
Function
Quantile
-----------------------------------1
0
1
0.9000
30.0352
0.8000
85.0602
0.7000
233.8785
-----------------------------------1
0
2
0.9000
16.8749
0.8000
27.3090
0.7000
41.6187
0.6000
73.1997
0.5000
122.2409
0.4000
208.5655
-----------------------------------1
0
3
0.9000
22.6799
0.8000
47.1263
0.7000
75.7539
0.6000
131.5378
0.5000
215.9553
0.4000
321.9881
Example 1 To illustrate the qo option we use command file ple3.cf
(see 6.5.2) and add the command
ple (qo = 0.9 (0.1) 0.1) ;
The resulting table is shown in Box 1.
6.5.4
1
comparing survivor functions
6.5.4 Comparing Survivor Functions
Basically two different methods are available to compare survivor functions. The first relies on the calculation of confidence intervals for each of
the survivor functions and then to check if they are overlapping or not.
This is possible both with the life table and the product-limit method.
Both methods provide estimates of standard errors for the survivor function. Another possibility is to calculate specific test statistics to compare
two or more survivor functions.
Defining Groups of Episodes. To make any comparisons there must
be two or more groups of episodes. This can be done with the grp option
in the ple command. The syntax is
grp = G1,G2,...,
with G1,G2,... names of variables contained in the data matrix. The
current set of episodes is then split into m groups with m the number
of indicator variables defined with the grp parameter. The first group,
defined by G1, contains all episodes where the value of this variable is
not zero, the second group is defined by G2 in the same way, and so on.
Confidence Intervals. Using the grp option in the ple command, one
gets a separate survivor function for each group. For each group the
output table also contains the estimated standard errors, SE(Ĝ(t)), of
the survivor function Ĝ(t). This allows the calculation of confidence intervals, based on the assumption that (G(t) − Ĝ(t))/SE(Ĝ(t)) follows
asymptotically a standard normal distribution. For standard 95 % confidence one would use
Ĝ(t) ± 1.96 SE Ĝ(t)
Example 1 To illustrate the comparison of survivor functions by using
confidence intervals, we continue with our main example data. Command
file ple5.cf (contained in the TDA example archive) estimates survivor
function separately for men and women. The resulting tables are written into an output file ple.5. Another command file, ple5p.cf, is then
used to plot the two survivor functions and their confidence intervals.
d060504.tex
April 21, 1998
6.5.4
2
comparing survivor functions
1
0
0
60
120
180
240
300
Figure 1 Survivor functions with confidence intervals for job
durations of men and women, based on main example data rrdat.1.
Plot created with command file ple5p.cf.
The resulting plot is shown in Figure 1. The survivor functions are very
similar in the first two years (the time axis is given in months), but thereafter the job durations of women are significantly shorter. The median
duration for men is 54.43 months, for women it is 35.61 months.
Construction of Test Statistics. Many different test statistics have
been proposed to compare two or more survivor functions. We describes
four which can be calculated with TDA; all are based on product-limit
estimates of survivor functions.
It is assumed that m groups have been defined which are not intersecting. The whole sample is implicitly defined as the set of all episodes
which are contained in one of these groups. Then, in exactly the same
way as explained in connection with the product-limit method, all calculations are done for each transition in the whole sample separately.
Therefore we only consider a sample of episodes which have the same
origin state and are censored or have the same destination state. A sample, defined this way, consists of m groups and the following table can
be calculated.
τ1
τ2
R11
R21
E11
E21
R12
R22
E12
E22
...
...
..
.
R1m
R2m
E1m
E2m
τq
Rn1
Eq1
Rq2
Eq2
...
Rqm
Eqm
(1)
6.5.4
3
comparing survivor functions
These are the basic quantities for product-limit estimation, for the whole
sample, and for each group separately. τ1 < τ2 < . . . < τq are the points
in time where at least one episode contained in the sample has an event.
Elg is the number of episodes contained in group g and having an event
at τl ; Rlg is accordingly defined as the number of elements in the risk set
at τl for the episodes contained in group g, i.e., all episodes belonging
to group g which have starting time less than τl and ending times equal
to, or greater than, τl . Altogether, these quantities are sufficient for a
product-limit estimation in each of the m groups.
Given this, the four test statistics can be defined, they are called Sν
(ν = 1, . . . , 4). Because the calculations only differ in different weights,
(ν)
we give their definition first. The weights are called Wl , and they are
defined, for l = 1, . . . , q, by
(1)
=
(2)
= Rl
p
=
Rl
Wl
Wl
(3)
Wl
(4)
Wl
1
(2)
l
Y
R i − Ei + 1
Ri + 1
i=1
=
The next step is to construct, for each of the four test statistics, one
(m)-vector U (ν) and one (m, m)-matrix V (ν) . The definitions are1
Ug(ν)
q
X
=
(ν)
Wl
(Elg − Rlg
l=1
Vg(ν)
1 g2
n
X
=
(ν) 2
Wl
i=1
El0
)
Rl0
El0 (Rl0 − El0 ) Rlg1
Rl0 − 1
Rl0
(3)
δ g1 g2
Rlg2
−
Rl0
(4)
Finally, the test statistics are defined by
0
Sν = U (ν) V (ν)
−1
U (ν)
(5)
All of them follow, under the null hypothesis of no significant differences
between the group-specific survivor functions, a χ2 -distribution with m−
1 degrees of freedom. Note that, accordingly, the rank of V (ν) is only
1δ
ij
is the Kronecker symbol which is one, if i = j, and otherwise is zero.
6.5.4
comparing survivor functions
4
Box 1 Comparing survivor functions with test statistics
Comparing survivor functions.
SN Org Des
Test Statistic
T-Stat
DF
Signif
-------------------------------------------------------------1
0
1
Log-Rank (Savage)
3.1227
1
0.9228
1
0
1
Wilcoxon (Breslow)
2.6510
1
0.8965
1
0
1
Wilcoxon (Tarone-Ware)
2.9767
1
0.9155
1
0
1
Wilcoxon (Prentice)
3.0014
1
0.9168
m − 1. Therefore, one can use in the calculation of (5) a generalized
inverse or, without loss of generality, one can omit the last dimension.
TDA follows the latter of these two possibilities.2
Unfortunately, there is no uniform convention to name the different
test statistics.3 So we give the names used by TDA with some remarks
about other naming conventions. In the order given by (2) we have:
1. Log-Rank (Savage). Other names are Generalized Savage Test (Andreß 1985, p. 158). The same name is used by BMDP, with Mantel-Cox
added. SAS calculates this test statistics under the name Logrank .
2. Wilcoxon (Breslow). BMDP gives the name Generalized Wilcoxon
(Breslow), SAS uses only the label Wilcoxon.
3. Wilcoxon (Tarone-Ware). This test statistic was proposed by Tarone
and Ware [1977] and is named accordingly. It is also calculated by BMDP,
using the label Tarone-Ware.
4. Wilcoxon (Prentice). Finally, we have a test statistic explained by
Lawless [1982, p. 423] with reference to Prentice [1978]. Since it is some
type of a Wilcoxon test we use this name.
The csf option in the ple command (see 6.5.2) can be used to request
these test statistics. At least two groups must be specified with the grp
parameter. An output file is optional.
Example 2 For an illustration of the csf option we use some biomet2 If there are any more rank deficiencies in the V -matrices, TDA will report this in
the standard output. If the prot option in the ple command is used to request a
protocol file, the U -vectors and V -matrices are written into this protocol file.
3 See Budde and Wargenau [1984] who give an overview with references to some
software packages.
6.5.4
comparing survivor functions
5
rical data published in Kalbfleisch and Prentice [1980, p. 2].4 The data
file is rrdat.3, the command file is ple6.cf (not shown). It uses the
command
ple (csf,grp=Group1,Group2) ;
to request test statistics for a comparison of the survivor functions of
the two groups. Part of the resulting output is shown in Box 1.
4 These
data are also used in some examples in the SAS User’s Guide (Version 5,
1985, lifetest procedure).
6.5.5
1
state distributions
6.5.5 State Distributions
A simple way of describing a set of multi-episode data is to calculate
cross-sectional state distributions for a sequence of time points. This
can be done with the epsdat command.
epsdat (
t=...,
ap=1,
) = fname;
sequence of time points.
append new data to end of output file.
The command requires a valid episode data structure and depends on
whether single or multi-episode data are defined. The t parameter must
be used to provide a sequence of nonnegative time points (for an explanation of the syntax see the tp parameter in 6.5.1.)
1. In the single episode case, the command calculates separately for each
time point a frequency distribution of the origin states of those episodes
which include the time point. (An episode includes a time point t if t
is greater than, or equal to the starting time and less than the ending
time.)
2. In the multi-episode case the command tries to find for each individual (defined by the ID parameter) and each time point an episode
which includes the time point. Using then these episodes, the command
calculates a frequency distribution of its origin states.
The resulting frequency distributions are written into the output file
specified on the right-hand side of the command. If the ap=1 parameter
is used, the data are appended to the end of an existing file.
d060505.tex
April 21, 1998
6.5.5
state distributions
2
Box 1 Command file ed5.cf
nvar(
dfile = ed1.dat,
# data file
ID
SN
ORG
DES
TS
TF
#
#
#
#
#
#
[1.0]
[1.0]
[1.0]
[1.0]
[2.0]
[2.0]
);
edef(
ts = TS,
tf = TF,
org = ORG,
des = DES,
# id = ID,
# sn = SN,
);
epsdat(
t = 0(5)30,
) = d;
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
identification number
spell number
origin state
destination state
starting time
ending time
#
#
#
#
starting time
ending time
origin state
destination state
# write state distributions to d
# time points
Example 1 To illustrate the epsdat command we use the data file
ed1.dat (see 3.3.4). The command file ed5.cf (Box 1) is set up for
single episode data. By removing the comment characters in front of
the id and sn parameter, it specifies multi-episode data. The epsdat
command uses an abbreviation to define the sequence of time points
t = 0, 5, 10, 15, 20, 25, 30
The resulting output files, both for single and multi-episode data, are
shown in Box 2. The first column shows the sequence of time points.
Then follows, separately for each origin state in the input data, a column
showing the number of episodes, or number of individuals, being in the
corresponding state. The final column shows the number of episodes, or
number of individual, without a valid state at the corresponding time
point.
6.5.5
state distributions
Box 2 Output from epsdat command
Singe episode data:
# State distributions.
# Time
1
0.0000
1
5.0000
1
10.0000
0
15.0000
0
20.0000
1
25.0000
1
30.0000
1
2
1
1
2
1
0
0
0
3
0
0
0
1
1
1
0
Total
2
2
2
2
2
2
1
Missing
3
3
3
3
3
3
4
2
1
1
2
1
0
0
0
3
0
0
0
1
1
1
0
Total
2
2
2
2
2
2
1
Missing
0
0
0
0
0
0
1
Multi-episode data:
# State distributions.
# Time
1
0.0000
1
5.0000
1
10.0000
0
15.0000
0
20.0000
1
25.0000
1
30.0000
1
3
6.6
describing sequence data
1
6.6 Describing Sequence Data
This chapter describes some elementary commands that can be used to
get information about a set of sequences. They require the specification
of a sequence data structure as explained in 3.4.2.
6.6.1
Sequence Length and Gaps
6.6.2
General Characteristics
6.6.3
State Distributions
6.6.4
Pattern Matching
d0606.tex
March 23, 2005
6.6.1
1
sequence length and gaps
6.6.1 Sequence Length and Gaps
Given a sequence, let si denote the time point of the first valid state
and ti the time point of the last valid state. si will be called the (valid)
starting time of the sequence, ti its (valid) ending time, and ti − si + 1
will be called the (valid) sequence length. Sequences may contain missing
(negative) values. If they occur before the valid starting time, or after
the valid ending time, they can normally be ignored. More problematic
are missing values between the starting and ending time. They will be
called internal gaps.
To provide basic information about starting and ending times, sequence
length, and internal gaps, TDA provides the command seqlg with syntax
shown in the following box.
seqlg (
sn=...,
sel=...,
v=...,
dtda=...,
number of sequence data structure, def. 1
expression for sequence selection
add variables ... to output file
request TDA description file
) = fname;
The right-hand side must provide the name of an output file, all other
parameters are optional.
1. The sn parameter can be used to select a sequence data structure.
As default, TDA uses the first of the currently defined sequence data
structures.
2. The sel parameter can be used to select sequences. The syntax is
sel = expression,
TDA then only uses sequences (cases) where expression evaluates to a
nonzero value.
3. The v = varlist parameter can be used to add the variables specified in varlist to the output file.
d060601.tex
April 21, 1998
6.6.1
2
sequence length and gaps
Box 1 Illustration of seqlg command with data file seq.d1
Case
starting
time
1
0
2
0
3
1
4
0
5
0
ending
time
7
7
7
5
7
sequence
length
8
8
7
6
8
gap
length
0
2
1
0
0
min gap
length
0
2
1
0
0
max gap
length
0
2
1
0
0
number
of gaps
0
1
1
0
0
4. The dtda parameter can be used to request an additional output
file containing an nvar command to read the data file created with the
seqlg command.
Example 1 To illustrate the information written into the output file,
we use the data file seq.d1 (see 3.4.2). Given that this data file has been
used to define sequence data, the command seqlg=d creates the output
file d shown in Box 1. The column “gap length” shows the total number
of internal time points with a missing state. The command file in the
example archive is seq9.cf.
6.6.2
1
general characteristics
6.6.2 General Characteristics
To get some general information about a sample of sequences one can
use the seqgc command with syntax shown in the following box.
seqgc (
sn=...,
sel=...,
v=...,
dtda=...,
number of sequence data structure, def. 1
expression for sequence selection
add variables ... to output file
request TDA description file
) = fname;
The right-hand side must provide the name of an output file. All other
parameters are optional and have the same meaning as explained in 6.6.1
for the seqgl command.
To illustrate the information written into the output file, we use the data
file seq.d1 (see 3.4.2). Given that this data file has been used to define
sequence data, the command
seqgc (dtda = t) = d;
creates an output file, d, and an additional description file, t, both shown
in Box 1. (The command file in the example archive is seq9.cf.)
1. The first column contains the case number, written with a 6.0 print
format. (All other entries use a 4.0 print format.)
2. The second column records the individual sequence length, defined
as the number of time points from the first valid state until the last
valid state, that is, the valid sequence length. Of course, there might
be missing states in between.
3. It follows an information about the number of different states in the
individual sequences, not counting missing states.
4. Then comes the number of state changes (events), calculated for the
valid sequence length and including changes from and into a missing
state.
d060602.tex
April 21, 1998
6.6.2
2
general characteristics
Box 1 Illustration of seqgc command with data file seq.d1
1
2
3
4
5
8
8
7
6
8
3
3
2
2
3
nvar(
dfile = d,
noc = 5,
CASE <5>[8.0]
SLEN <2>[4.0]
NDS <2>[4.0]
NEV <2>[4.0]
DUR1 <2>[4.0]
DUR3 <2>[4.0]
DUR7 <2>[4.0]
DURM <2>[4.0]
NEP1 <2>[4.0]
NEP3 <2>[4.0]
NEP7 <2>[4.0]
NEPM <2>[4.0]
);
4
4
2
1
4
1
2
4
2
1
=
=
=
=
=
=
=
=
=
=
=
=
c1 ,
c2 ,
c3 ,
c4 ,
c5 ,
c6 ,
c7 ,
c8 ,
c9 ,
c10,
c11,
c12,
4
3
2
4
4
#
#
#
#
#
#
#
#
#
#
#
#
3
1
0
0
3
0
2
1
0
0
1
1
1
1
1
2
2
1
1
2
2
1
0
0
2
0
1
1
0
0
case number
sequence length
number of different states
number state changes
duration in state 1
duration in state 3
duration in state 7
duration in missing state
number of episodes in state 1
number of episodes in state 3
number of episodes in state 7
number of episodes in missing state
5. The next entries show the number of time points, calculated for the
valid sequence length, that the individual sequence stays in the different states.
6. Then follows the total duration of internal gaps.
7. The next entries show the number of episodes, differentiated according to their origin state.
8. Then follows the number of internal gaps.
9. And finally, any variables specified with the v parameter are added
to the individual records.
6.6.3
1
state distributions
6.6.3 State Distributions
To get information about the cross-sectional state distributions in a sample of sequences, one can use the seqsd command with syntax shown in
the following box.
seqsd (
sn=...,
sel=...,
dtda=...,
number of sequence data structure, def. 1
expression for sequence selection
request TDA description file
) = fname;
The right-hand side must provide the name of an output file. All other
parameters are optional and have the same meaning as explained in 6.6.1
for the seqgl command.
Example 1 To illustrate the information written into the output file,
we use the data file seq.d1 (see 3.4.2). Given that this data file has been
used to define sequence data, the command
seqsd (dtda = t) = d;
creates an output file, d, and an additional description file, t, both shown
in Box 1. The command file in the example archive is seq9.cf.
Homogeneity of State Distributions. The homogeneity of state distributions can be assessed by a simple entropy measure (see, e.g., Theil
[1972]). Let Nt denote the number of individuals with a valid state at
time t, and ptj the proportion of Nt being in state j. Then, assuming
states j = 1, . . . , q, the entropy is defined by
Et = −
q
X
ptj log(ptj )
j=1
with the convention that 0 log(0) = 0. It follows that 0 ≤ Et ≤ log(q).
The entropy becomes zero if all individuals are in the same state, it takes
d060603.tex
April 21, 1998
6.6.3
state distributions
2
Box 1 Illustration of seqsd command with data file seq.d1
0
1
2
3
4
5
6
7
1
1
0
3
2
1
1
1
nvar(
dfile
noc =
TIME
NST1
NST3
NST7
VALID
NMISS
TOTAL
);
1
4
4
1
3
3
0
1
= d,
8,
<2>[6.0]
<2>[6.0]
<2>[6.0]
<2>[6.0]
<2>[6.0]
<2>[6.0]
<2>[6.0]
2
0
1
0
0
0
2
2
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
4
5
5
4
5
4
3
4
#
#
#
#
#
#
#
time
cases
cases
cases
cases
cases
total
1
0
0
1
0
1
2
1
5
5
5
5
5
5
5
5
in state 1
in state 3
in state 7
in valid states
in missing state
number of cases
Box 2 Syntax for seqen command
seqen (
sn=...,
sel=...,
fmt=...,
dtda=...,
number of sequence data structure, def. 1
expression for sequence selection
print format, def. 10.4
request TDA description file
) = fname;
its maximum value if the individuals are equally spread over the different
states.
To calculate these entropy measures, TDA provides the command
seqen with syntax shown in Box 2. The right-hand side must provide
the name of an output file. All other parameters are optional and have
the same meaning as explained in 6.6.1 for the seqgl command.
6.6.3
state distributions
3
Box 3 Illustration of seqen command, based on seq.d1
# Time
0
1
2
3
4
5
6
7
N
4
5
5
4
5
4
3
4
Entropy
1.0397
0.5004
0.5004
0.5623
0.6730
0.5623
0.6365
1.0397
Example 2 To illustrate the seqen command, we use again the data
file seq.d1 (see 3.4.2). Given that this data file has been used to define
sequence data, output of the seqen command is shown in Box 3. The
command file in the example archive is seq9.cf.
6.6.4
1
pattern matching
6.6.4 Pattern Matching
Given a definition of sequence data, the seqpm command can be used to
find patterns in the sequences. The syntax is shown in the following box.
seqpm (
sn=...,
ps=...,
df=...,
nfmt=...,
v=...,
dtda=...,
number of sequence data structure, def. 1
definition of patterns
test output file
integer print format, def. 4
additional variables for output file
TDA description for output file
) = output file name;
The right-hand side must provide the name of an output file and the ps
parameter must be used for a definition of patterns. All other parameters
are optional. Patterns must be defined in the following way:
ps = [a1,a2,...],[b1,b2,...],...,
where each pair of square brackets specifies one pattern. The entries,
a1,a2,... and so on, can be one of the following:
1. Non-negative integers for valid states.
2. The character ‘?’ matches any single state.
3. The character ‘*’ matches any sequence of states.
4. The character ‘+’ matches any repeat of the previous state.
5. The character ‘-’ matches any sequence of identical states.
Note that the command only recognizes sequences that do not contain
missing values (negative state numbers).
The seqpm command creates an output file that will contain one record
for each sequence containing the following information:
d060604.tex
December 23, 1999
6.6.4
pattern matching
2
Box 1 Command file seqpm1.cf
nvar(
dfile = seq.d1,
ID = c1,
Y0 = c2,
Y1 = c3,
Y2 = c4,
Y3 = c5,
Y4 = c6,
Y5 = c7,
Y6 = c8,
Y7 = c9,
);
seqdef = Y0,,Y7;
seqpm(
ps = [-],[3,3],[3,*,3],
df = df,
dtda = t,
) = d;
Box 2 Output file from command file seqpm1.cf
ID
LEN P1
P2
P3
---------------------1
8
5
2
1
4
6
2
2
1
5
8
5
2
1
1. The sequence case id number.
2. The length of the sequence.
3. Finally, for each pattern, the number of matches.
In addition, one can add any variables to the output file with the
v = varlist,
parameter. Also, one can request a TDA description of the output file
with the dtda parameter.
Example 1 To illustrate the seqpm command we use data file seq.d1
as described in 3.4.2. The command file is shown in Box 1. The resulting
output file is shown in in Box 2.
6.7
investigating proximities
1
6.7 Investigating Proximities
This chapter is intended to deal with proximity data. There is currently
only a single section discussing the calculation of proximity measures for
sequence data.
6.7.2
d0607.tex
Sequence Proximity Measures
April 21, 1998
6.7.2
1
sequence proximity measures
6.7.2 Sequence Proximity Measures
Regression models for events aim to describe the evolution of sequences
by focusing on transitions. As a complementary strategy, one can try to
describe whole sequences. This can be done in two different ways. One
can search for some interesting patterns in each sequence separately; or
one can compare all sequences in a given sample. In any case, one needs
a proximity measure to assess the similarity of sequences, or of sequences
with some pattern.
This chapter discusses a family of proximity measures for sequences
based on “optimal matching”. Basic information about this approach
can be found in Sankoff and Kruskal [1983]. Bock [1984] provides a short
introduction. A very useful source of mathematical insights and algorithms is Waterman [1995]. For a discussion of sociological applications,
see Abbott [1983], [1995], Abbott and Hrycak [1990].
6.7.2.1
Optimal Matching
6.7.2.2
The seqm Command
6.7.2.3
Examples
6.7.2.4
Consecutive Indel Operations
6.7.2.5
Data-based Substitution Costs
6.7.2.6
Working with Large Samples
6.7.2.7
Comparing Parallel Sequences
d060702.tex
April 21, 1998
6.7.2.1
optimal matching
1
6.7.2.1 Optimal Matching
We consider sequences of states which are elements of a finite state space,
say Y. S denotes the set of all finite sequences over Y, meaning that
if a ∈ S then a = (a1 , . . . , an ) with a1 , . . . , an ∈ Y
n = |a| is the length of the sequence. We want to compare two sequences
a, b ∈ S, possibly with different lengths. The basic idea is to define some
elementary operations which can be used to sequentially transform one
sequence until it becomes equal to the other sequence. Let Ω denote the
set of basic operations and a[ω] the sequence resulting from a by applying
the operation ω ∈ Ω. Most applications consider just three elementary
operations:
• Insertion: a[ι] denotes the sequence resulting from a ∈ S by inserting
one new element (a state from Y) into the sequence a.
• Deletion: a[δ] denotes the sequence resulting from a by deleting one
element from this sequence.
• Substitution: a[σ] denotes the sequence resulting from a by changing
one of its elements into another state.
Of course, we can think of sequentially applying elementary operations to
a given sequence. Let a[ω1 , ω2 , . . . , ωk ] denote the new sequence resulting
from a by applying first the elementary operation ω1 , then ω2 , and so
on until finally ωk . Then, given two sequences a, b ∈ S, we can ask for a
sequence of elementary operations which transforms a into b.
In general, there will be many such sequences of elementary operations which transform a into b. Now, the intuitive idea for developing
a distance measure for sequences is to look for the shortest sequence of
elementary operations which transform a into b. A slightly more general
approach is to evaluate the elementary operations by introducing c(ω)
= cost of applying the elementary operation ω ∈ Ω. We will assume that
0 < c(ω) < ∞. The cost of applying a sequence of elementary operations
will be denoted by
c[ω1 , ω2 , . . . , ωk ] =
k
X
c[ωi ]
i=1
d06070201.tex
April 21, 1998
6.7.2.1
optimal matching
2
Setting c[ ] = 0 for no operation, we can formally define
dΩ (a, b) = min c[ω1 , . . . , ωk ] | b = a[ω1 , . . . , ωk ], ωi ∈ Ω, k ≥ 0
to measure the distance between the sequences a and b. This measure
is by definition nonnegative, and dΩ (a, b) = 0 only if a = b. Symmetry
does not automatically hold, but can be forced by equating insertion
and deletion costs, or by a slightly different definition: minimum cost
of transforming a into b or b into a. Whether the distance measure
will also be transitive, and thus constitutes a metric distance, depends
on the definition of the set of elementary operations, Ω. Transitivity
is automatically guaranteed for the simple case when there are only
insertions, deletions, and substitutions.
“Optimal matching” without further qualification normally means
referring to this simple distance measure based on Ω = {ι, δ, σ} resulting
in a metric distance. Of course, the distance measure also depends on
the definition of the cost functions c[ω]. These cost functions can be
arbitrarily defined with respect to the intended application. As a special
case, one can set
c[ι] = c[δ] = 1 and c[σ] = 2
The distance between two sequences a and b is then simply the number of indel operations (insertions and deletions) which are necessary to
transform one sequence into the other. For the procedure described in
6.7.2.2, this will be the default cost functions if not otherwise specified
by the user.
The set of elementary operations, Ω, can be extended in many different ways. However, the current implementation in TDA offers only one,
but possibly important, extension: so-called indel functions that allow
deleting or inserting a series of consecutive states, see 6.7.2.4.
Basic Algorithm. If the elementary operations consist only of insertions, deletions and substitutions, the distance measure can be calculated
with a simple dynamic programming method. To explain this method,
we follow Kruskal and Sankoff [1983], p. 266).1
Let Y denote the finite state space and φ an “empty state” which is
1 See
also Waterman [1995], pp. 192–194, and for an illustration of the algorithm,
Abbott and Hrycak [1990].
6.7.2.1
optimal matching
3
not contained in Y. The cost functions are
w(x, y) = substitution cost, x, y ∈ Y
w(x, φ) = deletion cost, x ∈ Y
w(φ, y) = insertion cost, y ∈ Y
The expression
x1 · · ·
y1 · · ·
xp
yp
(1)
is called an alignment if xi , yi ∈ Y ∪ {φ} and there is no column with
xi = yi = φ. The length of an alignment (i.e., the distance between the
two sequences) is defined by Σi w(xi , yi ).
Now, let a = (a1 , . . . , am ) and b = (b1 , . . . , bn ) be two sequences
with states in Y. We say that (1) is an alignment between the sequences a and b if by inserting empty states into a it can be made
equal to x = (x1 , . . . , xp ) and in the same way b can be made equal to
y = (y1 , . . . , yp ). Given these definition, we can finally define the (standard) distance between a and b, denoted d(a, b), as the minimum possible
length of any alignment of these two sequences. Additional notation to
describe the algorithm is
ai = (a1 , . . . , ai ) , bj = (b1 , . . . , bj ) , and dij = d(ai , bj )
Using this notation, we need to calculate d(a, b) = dmn . This can be
done by calculating the elements of the (m + 1, n + 1) matrix
D = dij
i = 0, . . . , m; j = 0, . . . , n
(2)
recursively in the following way. The first step is initializing the first row
and first column of this matrix:
d0,0
d0,j
di,0
= 0
= d0,j−1 + w(φ, bj ) j = 1, . . . , n
= di−1,0 + w(ai , φ) i = 1, . . . , m
All other elements of D can then be calculated by using three predecessors. The recurrence relation is:

 di−1,j + w(ai , φ)
di−1,j−1 + w(ai , bj )
dij = min

di,j−1 + w(φ, bj )
6.7.2.1
optimal matching
4
Having calculated all elements of D, one finally finds the required distance in the element dmn .
Computation time for this algorithm, and storage requirements, are
proportional to mn. In the TDA implementation, the matrix D is defined
for single precision floating point numbers, and the required memory is
4L2 bytes, L being 1 plus the maximum sequence length in the input
data.
6.7.2.2
the seqm command
1
6.7.2.2 The seqm Command
The TDA command to request optimal matching of sequences is seqm,
with syntax shown in Box 1. The right-hand side must provide the name
of an output file. All other parameters are optional.
Selecting alignments. There are two basic options which can be selected with the m parameter.
1. If m=1 (default), the seqm command uses a single sequence data structure. If not otherwise specified with the sn parameter, the first of the
currently defined sequence data structures will be used. As default, an
alignment for each pair of sequences in the selected sequence data structure is calculated. If there are n sequences, there will be n(n − 1)/2
alignments, and the output file will contain one record for each alignment.
There are two options to restrict the number of alignments. The r
parameter can be used to request a random selection of alignments (see
6.7.2.6). Another possibility is to use only some specified sequences for
comparison. This can be requested with the parameter
cn = i1 , i2 , . . . , im ,
where i1 , i2 , . . . , im are the case numbers of some sequences in the data
matrix. Then, all sequences of the selected sequence data structure are
only compared with these selected sequences resulting in a total number
of nm alignments. In this case, the output file will contain n records,
each containing m distances.
2. Alternatively, one can use the m=2 option for pairwise optimal matching of sequences taken from two sequence data structures. In this case,
the parameter
sn = k1 , k2 ,
must be used to specify the two sequence data structures, and it is required that the two sequence data structures have the same state space
(see 6.7.2.7).
Preprocessing sequences. Before alignment of any two sequences,
they are shifted to a common process time axis: t = 1, 2, 3, . . . , L, where
d06070202.tex
April 21, 1998
6.7.2.2
the seqm command
2
Box 1 Syntax for seqm command
seqm (
m=...,
sn=...,
icost=...,
scost=...,
rr=1,
sm=...,
r=...,
s=...,
cn=...,
max=...,
tfmt=...,
v=...,
dtda=...,
df=...,
tst=...,
fmt=...,
selection of method, def. 1
selection of sequence data structure(s), def. 1
indel cost specification
substitution cost specification
use common sequence length
option for preprocessing sequences
random selection of sequences
print sequence of distances, or LCS
compare with specified sequences
alignment restriction
print format for distances, def. 5.2
add variables ... to output file
create TDA description file
create test output file
additional test options
print format for test output file
) = fname;
L is the maximum sequence length. As default, only sequences with a
positive length and without internal gaps will be compared. This can be
changed with the sm parameter. There are three options:
• sm=1 Internal gaps will be disregarded, that is, they will simply be
skipped in preparing a sequence for comparison with another one.
• sm=2 Each contiguous subsequence of identical states is substituted
by a single occurrence of that state.
• sm=1,2 This selects sm=1 and sm=2.
The tst parameter (explained below) can be used to check whether the
selected option has the desired effect.1
As mentioned in 6.7.2.1, optimal matching does not require that
sequences have equal length. As an option (rr=1), one can restrict the
1 Of course, any other preprocessing of sequences can be done before inputting the
data to the program, or using TDA’s operators to change variables. For example,
Dijkstra and Taris [1995] have proposed a method which only takes into account
states that appear in both sequences to be compared. Whether this makes sense is
questionable, see the comments by Abbott [1995a].
6.7.2.2
the seqm command
3
alignment to the common sequence length. Then, if li is the length of
sequence i and lj is the length of sequence j, the alignment procedure
uses only the common sequence length min{li , lj }.
Specifying indel cost. The default distance measure is d(a, b), based
on a set of elementary functions, Ω, only consisting of insertions, deletions, and substitutions. Insertion and deletion costs are always identical
and called indel cost. The default indel cost is 1, and the default substitution cost is 2. The alignment, then, amounts to calculating the longest
common subsequence (LCS).
Indel cost can be changed with the icost parameter. There are three
possible ways to specify alternative definitions. Using
icost = α,
with some nonnegative real value α, indel cost will be equal to α. Alternatively, indel costs can be specified as a sequence of values for the
common time axis. This option requires that the user first defines a (1, L)
or (L, 1) matrix containing the indel cost values; L being the maximal
sequence length in the selected sequence data structure.2 Assuming that
such a matrix has been defined and named M, one can use the parameter
icost = M,
to request using the values from this matrix.
As a third possibility, one can extend the set of operations, Ω, to
allow for simultaneously inserting or deleting contiguous subsequences
consisting of one or more states, based on a linear indel cost function
g(l) = α + β (l − 1)
with α, β ≥ 0
l being the length of the subsequence. The extension will be effective if
the user specifies this cost function with the parameter
icost = α, β,
This extension will be explained and illustrated in 6.7.2.4. Note that the
max parameter to restrict the possible alignments is not effective when
using an extended distance measure.
Specifying substitution cost. Given two sequences a = (a1 , . . . , am )
and b = (b1 , . . . , bn ), default substitution cost is w(ai , bj ) = 2, if ai 6= bj ;
2 See
5.1.2 for commands to define matrices.
6.7.2.2
the seqm command
4
substitution cost for identical states is always zero. Some alternative
definitions can be selected with the parameter
scost = n,
If n = 1, substitution cost is calculated as the absolute difference w(ai , bj )
= | ai −bj | . If n = 2, substitution cost will be derived from the frequency
of transitions in the sequences, see 6.7.2.5.
Alternatively, one can define a complete substitution cost matrix
with the mdef command for defining matrices, see 5.1.2. This must be a
(ns , ns ) matrix, ns being the number of states in the selected sequence
data structure. Given that such a matrix M = (mij ) has been defined,
one can use the parameter
scost = M,
to request that this matrix shall be used, meaning that w(ai , bj ) = mij .
Remember that the states occurring in a sequence data structure are
always sorted in ascending order and then mapped to internal state
numbers 0, 1, . . . , ns − 1. The ordering of substitution cost values in the
matrix M must correspond to these internal state numbers. In any case,
one should check the specification of indel and substitution cost with the
test output option explained below.
Restricted alignment. The max parameter can be used to restrict the
alignment procedure. This option will be explained in 6.7.2.6.
Output options. By default, the output file will contain a separate
record for each alignment. If n is the number of sequences, the number
of records will be n(n−1)/2, if m=1, and n, if m=2. This might be different
if the r parameter is used to request a random selection of alignments,
or if the cn parameter is used to request alignments with some fixed
reference sequences.
By default, each record of the output file will contain the case numbers of the sequences, their length and the value of the distance measure.
The print format for the distance measure can be controlled with the
tfmt parameter, default is tfmt=5.2.
The s parameter can be used to modify the printing of distances. If
s=0 (default), only the final distance, d(a, b), between any two sequences
a = (a1 , . . . , am ) and b = (b1 , . . . , bn ) is written into the output file.
Alternatively, with s=1, one gets all sequentially calculated distances
6.7.2.2
the seqm command
5
d(ai , bj ) (see the definition in 6.7.2.1), meaning that each record of the
output file will contain the sequential distances
d(ai , bi ) for i = 1, . . . , min {m, n}
d(am , bj ) for j = m + 1, . . . , n
d(ai , bn ) for i = n + 1, . . . , m
This option is useful for sequentially comparing sequences. The sequence
of distances, d(ai , bi ), can be interpreted as showing how the distance
between a and b evolves over time.
When using default cost functions, one can alternatively request a
printing of the longest common subsequences with the s=2 parameter.
In addition, one can use the v=varlist parameter to add variables
to the output data file. Values of the variables specified in varlist will
be written into the output file for both sequences which take part in
the alignment. To see which variables have been written into the output
file, one can request an additional data description file with the dtda
parameter.
Test output. The df parameter can be used to request an additional
test output file. By default, this file will provide information about the
set up of indel and substitution cost. Additional information can be
requested with the parameter
tst = i, j, . . .,
where i, j, . . . are integers in the range 1, . . . , 3. The meaning is as follows.
1 This option prints the initial D matrix into the test output file.
2 This option prints the sequences into the test output file. This should
be helpful in checking whether the sm parameter has the desired effect.
3 Using this option, for each alignment of two sequences, the matrix D,
defined in (2) above, is written into the test output file. (Additional
matrices will be written into the test output file when working with
an extended distance measure.) Illustration will be given in 6.7.2.3.
6.7.2.3
examples
1
6.7.2.3 Examples
This section illustrates the seqm command with some examples. We use
very small example data files in order to easily check what the command
is doing.
Example 1 We begin with the four sequences shown in Box 1. A command file, seqm1.cf, is shown in Box 2. This command file first reads
the data file and defines a sequence data structure and then requests
optimal matching with the seqm command. Only the dtda parameter
is used to request a description of the output file. All other parameters
have default values.
Standard output from the command file is shown in Box 3. The command has performed pairwise alignment of the four sequences using default indel and substitution cost. The command has created two files, a
data file (seqm1.d) and a description file (seqm1.tda) that describes the
variables in the data file. Both files are shown in Box 4.
To add variables to the output file, one can use the v parameter.
For example, if adding the parameter v=Y1,,Y5 to the seqm command
in seqm1.cf, the output file would contain 10 additional variables (the
variables Y1,...,Y5 for both sequences).
Box 1 shows a second data file, seqm.d1a, which is basically identical
with seqm.d1 but where the sequences are differently placed on the time
axis. Since, as default, TDA uses only sequences without internal gaps
and skips leading missing values, using this second data file will give
identical results when used for optimal matching. (See the command file
seqm1a.cf in the example archive.)
The data file seqm1.d in Box 4 only shows the final distance for each
alignment. The s=1 option can be used to get information about how
the distances evolve during process time. Adding this parameter to the
seqm command would result in the output file shown in Box 5.
d06070203.tex
April 21, 1998
6.7.2.3
examples
Box 1 Example sequence data files
Data file: seqm.d1
0 0 0 0 0
0 0 1 0 0
0 0 0 1 0
0 0 1 1 0
Data file: seqm.d1a
-1 0 0 0 0 0 -1 -1 -1
-1 -1 0 0 1 0 0 -1 -1
-1 -1 -1 0 0 0 1 0 -1
-1 -1 -1 -1 0 0 1 1 0
Box 2 Command file seqm1.cf
nvar(
dfile = seqm.d1,
Y{1,5} = c1,
);
seqdef = Y1,,Y5;
seqm(
dtda = seqm1.tda,
) = seqm1.d;
Box 3 Standard output from seqm1.cf
Sequence proximity measures. Current memory: 260732 bytes.
Optimal matching.
Using sequence data structure 1.
Number of states: 2. Max sequence length: 5
Default indel cost: 1.
Default substitution cost: 2.
Starting alignment procedure.
Number of sequences (cases): 4
Sequences with zero length or internal gaps: 0
Sequences used for alignment: 4
Number of alignments: 6
6 record(s) written to output file: seqm1.d
Maximum distance between sequences 4 and 1: 4
TDA description written to: seqm1.tda
2
6.7.2.3
3
examples
Box 4 Output files created by seqm1.cf
Output file: seqm1.d
----------------------------2
1
5
5 2.00
3
1
5
5 2.00
3
2
5
5 2.00
4
1
5
5 4.00
4
2
5
5 2.00
4
3
5
5 2.00
TDA description file: seqm1.tda
------------------------------nvar(
dfile = seqm1.d,
noc = 6,
SEQ1N <5>[6.0] = c1 , # sequence
SEQ2N <5>[6.0] = c2 , # sequence
SEQ1L <2>[6.0] = c3 , # sequence
SEQ2L <2>[6.0] = c4 , # sequence
DIST <4>[5.2] = c5 , # distance
);
A
B
A
B
case number
case number
length
length
In the first example we have used default indel and substitution cost and
optimal matching is then basically identical to calculating the longest
common subsequences (LCS). This allows to use the s=2 parameter to
request a printing of the LCS. To illustrate, adding the parameter s=2
to command file seqm1.cf will create the output files shown in Box 6.
The additional variables in the output file show the length of the
LCS and its values. If the LCS is shorter than the maximal sequence
length, the remaining columns are filled with missing values (-1). Note
the simple relationship
length of longest common subsequence =
1
(m + n − d(a, b))
2
where d(a, b) the the optimal matching distance when using default cost
functions; m and n denote the length of the first and second sequence,
respectively.
6.7.2.3
examples
Box 5 Illustration of s=1 option (seqm1b.cf)
Data file
----------------------------------------------------2
1
5
5 0.00 0.00 2.00 2.00 2.00
3
1
5
5 0.00 0.00 0.00 2.00 2.00
3
2
5
5 0.00 0.00 2.00 2.00 2.00
4
1
5
5 0.00 0.00 2.00 4.00 4.00
4
2
5
5 0.00 0.00 0.00 2.00 2.00
4
3
5
5 0.00 0.00 2.00 2.00 2.00
Description file
----------------------------------------------------nvar(
dfile = ...,
noc = 6,
SEQ1N <5>[6.0] = c1 , # sequence A case number
SEQ2N <5>[6.0] = c2 , # sequence B case number
SEQ1L <2>[6.0] = c3 , # sequence A length
SEQ2L <2>[6.0] = c4 , # sequence B length
DIST1 <4>[5.2] = c5 , # distance, time 1
DIST2 <4>[5.2] = c6 , # distance, time 2
DIST3 <4>[5.2] = c7 , # distance, time 3
DIST4 <4>[5.2] = c8 , # distance, time 4
DIST5 <4>[5.2] = c9 , # distance, time 5
);
4
6.7.2.3
examples
Box 6 Illustration of s=2 option (seqm1c.cf)
Data file
------------------------------------------------2
1
5
5 2.00
4 0 0 0 0 -1
3
1
5
5 2.00
4 0 0 0 0 -1
3
2
5
5 2.00
4 0 0 1 0 -1
4
1
5
5 4.00
3 0 0 0 -1 -1
4
2
5
5 2.00
4 0 0 1 0 -1
4
3
5
5 2.00
4 0 0 1 0 -1
Description file
------------------------------------------------nvar(
dfile = ...,
noc = 6,
SEQ1N <5>[6.0] = c1 , # sequence A case number
SEQ2N <5>[6.0] = c2 , # sequence B case number
SEQ1L <2>[6.0] = c3 , # sequence A length
SEQ2L <2>[6.0] = c4 , # sequence B length
DIST <4>[5.2] = c5 , # distance
LCSL <2>[4.0] = c6 , # length of LCS
LCS1 <2>[2.0] = c7 , # LCS t=1
LCS2 <2>[2.0] = c8 , # LCS t=2
LCS3 <2>[2.0] = c9 , # LCS t=3
LCS4 <2>[2.0] = c10 , # LCS t=4
LCS5 <2>[2.0] = c11 , # LCS t=5
);
5
6.7.2.3
examples
6
Box 7 Sequence data file seqm.d2
-1
2
1
1 1 -1 2
2 -1 -1 2
1 2 2 -1
Box 8 Command file seqm2.cf
nvar(
dfile = seqm.d2,
Y{1,5} = c1,
);
seqdef = Y1,,Y5;
seqm(
sm = 1,
df = seqm2.tst,
tst = 2,
dtda = seqm2.tda,
) = seqm2.d;
Example 2 To illustrate the preprocessing of sequences, we use the
data file seqm.d2 in Box 7. The first two sequences contain internal
gaps. Without preprocessing these sequences, there would be only one
valid sequence and the seqm command would perform no alignments.
As explained in 6.7.2.2, there are two preprocessing options. The first
one, sm=1, simply skips internal gaps. This has been done with command
file seqm2.cf in Box 8. This command file also requests a test output file,
seqm2.tst, shown in Box 9. This file shows the set up of cost functions
(in this example we have again default indel and substitution cost), and
since we have used the tst=2 option, it also shows the sequences actually
used for alignment. In this example, there are three pairs of sequences
for three alignments.
The lower part of Box 9 illustrates how the sequences would look like
if we had used both preprocessing options, sm=1,2. Each consecutive
subsequence of identical states is then substituted by a single occurrence
of that state.
6.7.2.3
examples
Box 9 Test output file created by seqm2.cf
Optimal matching test output file.
Number of states: 2
Max sequence length: 5
Indel cost
1 1 1 1 1
Substitution cost
0 2
2 0
Sequence A (case number 2)
2 2 2
Sequence B (case number 1)
1 1 2
Sequence A (case number 3)
1 1 2 2
Sequence B (case number 1)
1 1 2
Sequence A (case number 3)
1 1 2 2
Sequence B (case number 2)
2 2 2
Sequences used for alignment if sm=1,2
-------------------------------------Sequence A (case number 2)
2
Sequence B (case number 1)
1 2
Sequence A (case number 3)
1 2
Sequence B (case number 1)
1 2
Sequence A (case number 3)
1 2
Sequence B (case number 2)
2
7
6.7.2.3
8
examples
Box 10 Sequence data file seqm.d3
4 5 6 7 7 7 7 7
1 1 2 3 4 5 6 -1
Box 11 Command file seqm3.cf
nvar(
dfile = seqm.d3,
Y{1,8} = c1,
);
seqdef = Y1,,Y8;
mdef(SCOST,10,10) =
0.0, 0.1, 0.2, 0.3,
0.1, 0.0, 0.1, 0.2,
0.2, 0.1, 0.0, 0.1,
0.3, 0.2, 0.1, 0.0,
0.4, 0.3, 0.2, 0.1,
0.5, 0.4, 0.3, 0.2,
0.6, 0.5, 0.4, 0.3,
0.7, 0.6, 0.5, 0.4,
0.8, 0.7, 0.6, 0.5,
0.9, 0.8, 0.7, 0.6,
0.4,
0.3,
0.2,
0.1,
0.0,
0.1,
0.2,
0.3,
0.4,
0.5,
0.5,
0.4,
0.3,
0.2,
0.1,
0.0,
0.1,
0.2,
0.3,
0.4,
0.6,
0.5,
0.4,
0.3,
0.2,
0.1,
0.0,
0.1,
0.2,
0.3,
0.7,
0.6,
0.5,
0.4,
0.3,
0.2,
0.1,
0.0,
0.1,
0.2,
0.8,
0.7,
0.6,
0.5,
0.4,
0.3,
0.2,
0.1,
0.0,
0.1,
0.9,
0.8,
0.7,
0.6,
0.5,
0.4,
0.3,
0.2,
0.1,
0.0;
seqm(
icost = 1,
scost = SCOST,
df = seqm3.tst,
tst = 2,3,
dtda = seqm3.tda,
) = seqm3.d;
Example 3 To illustrate user-defined cost functions we use an example
given by Abbott and Hrycak ([1990], p. 179). The data file, seqm.d3 (Box
10), consists of two sequences.
Box 11 shows the command file, seqm3.cf. Indel cost is simply 1
(actually the default); substitution cost is defined by a matrix, SCOST.
One should note how this matrix is used. The state space for the two
sequences is {1, 2, 3, 4, 5, 6, 7}. These states are mapped to internal state
number {0, 1, 2, 3, 4, 5, 6}. The matrix element SCOST(i, j) corresponds
to the internal state numbers i − 1 and j − 1.
Command file seqm3.cf also requests a test output file with options
tst=2,3. Part of this test output file is shown in Box 12. The files shows
6.7.2.3
examples
9
Box 12 Part of test output file from seqm3.cf
Sequence
1 1 2
Sequence
4 5 6
A (case number 2)
3 4 5 6
B (case number 1)
7 7 7 7 7
D Matrix
0
1
2
3
4
5
6
7
A
1
1
2
3
4
5
6
0
1
2
3
4
5
6
7
8
-----------------------------------------------------B
4
5
6
7
7
7
7
7
-----------------------------------------------------| 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00
| 1.00 0.30 1.30 2.30 3.30 4.30 5.30 6.30 7.30
| 2.00 1.30 0.70 1.70 2.70 3.70 4.70 5.70 6.70
| 3.00 2.20 1.60 1.10 2.10 3.10 4.10 5.10 6.10
| 4.00 3.10 2.40 1.90 1.50 2.50 3.50 4.50 5.50
| 5.00 4.00 3.20 2.60 2.20 1.80 2.80 3.80 4.80
| 6.00 5.00 4.00 3.30 2.80 2.40 2.00 3.00 4.00
| 7.00 6.00 5.00 4.00 3.40 2.90 2.50 2.10 3.10
the sequences used in the alignment (requested by tst=2) and, in addition, the matrix D used in the dynamic programming algorithm (see
6.7.2.1).
6.7.2.4
1
consecutive indel operations
6.7.2.4 Consecutive Indel Operations
In the default alignment procedure, deleting (or inserting) a subsequence
consisting of l elements (states) would cost l times the elementary deletion (or insertion) cost. It is sometimes desirable that the deletion (or
insertion) cost of such an operation is a more general function of l. Extending the basic algorithm to allow for this modification of the distance
measure is quite possible but requires, in general, much more computational effort. See Kruskal and Sankoff [1983, pp. 296–299], Waterman
[1995, pp. 194–197]. In general, computation time would be proportional
to n3 (n being the (common) sequence length), instead of n2 for the basic algorithm. However, if the cost function for multiple indels is linear
in l, the basic algorithm can easily be extended to remain proportional
in n2 . This section is restricted to this special case.
To explain the extended procedure, we follow Waterman [1995, p. 195].
The terminology follows the introduction in 6.7.2.1. The two sequences
for alignment are a = (a1 , . . . , am ) and b = (b1 , . . . , bn ). The substitution cost is given by w(ai , bj ), and the indel cost function for deleting or
inserting a subsequence of l states is assumed to be linear:
g(l) = α + β(l − 1)
α, β ≥ 0
dij = (ai , bj ) denotes the optimal distance for the subsequences ai =
(a1 , . . . , ai ) and bj = (b1 , . . . , bj ). The basic recurrence relation is then

 di−1,j−1 + w(ai , bj )
min1≤l≤i {di−l,j + g(l)}
dij = min

min1≤k≤j {di,j−k + g(k)}
The first of these three possibilities amounts to substituting ai by bj ;
the second results from deleting the last l elements in ai ; and the third
results from deleting the last k elements in bj .
In addition to the matrix D = (dij ), the algorithm needs two more
matrices: E = (eij ) and F = (fij ); all with dimension (m + 1, n + 1).
d06070204.tex
April 21, 1998
6.7.2.4
consecutive indel operations
2
The algorithm is defined as follows. First there is an initialization:1
d0,0 = e0,0 = f0,0 = 0
di,0 = g(i), ei,0 = g(i) + α for i = 1, . . . , m
d0,j = g(j), f0,j = g(j) + α for j = 1, . . . , n
The recurrence relation is
eij = min {di,j−1 + α, ei,j−1 + β}
fij = min {di−1,j + α, fi−1,j + β}
dij = min {di−1,j−1 + w(ai , bj ), eij , fij }
To see that the algorithm works correctly, one needs to show that
eij =
fij =
min {di,j−k + g(k)}
1≤k≤j
min {di−l,j + g(l)}
1≤l≤i
The proof is by induction over j for the first equation, and over i for the
second equation. To illustrate with the first equation, it certainly holds
for j = 1. Then, for an arbitrary j > 1:
min {di,j−k + g(k)} =
1≤k≤j
min {di,j−1 + g(1), min {di,j−k + g(k)}} =
2≤k≤j
min {di,j−1 + g(1),
min {di,j−1 + g(1),
min
{di,(j−1)−(k−1) + g(k − 1) + β}} =
min
{di,(j−1)−k + g(k) + β}} =
2≤k−1≤j
1≤k≤j−1
min {di,j−1 + α, ei,j−1 + β} = eij
In the same way, one can show the second equation.
Syntax. As explained in 6.7.2.2, this extension of the basic distance
measure can be selected by specifying an indel cost function with the
parameter
icost = α, β,
where α ≥ 0 and β ≥ 0.
1 This
is slightly different from Waterman’s formulation ([1995, p. 195], but necessary
for general α and β.
6.7.2.4
consecutive indel operations
3
Box 1 Sequence data file seqm.d4
Sequence 1
2
3
4
5
0
0
0
0
0
0
1
1
1
1
0
0
1
1
1
0
0
0
1
1
0
0
0
0
1
0
0
0
0
0
Box 2 Command file seqm4.cf
nvar(
dfile = seqm.d4,
Y{1,6} = c1,
);
seqdef = Y1,,Y6;
seqm(
icost = 1,0.5,
df = seqm4.tst,
tst = 2,3,
dtda = seqm4.tda,
) = seqm4.d;
Box 3 Illustration of results for different indel cost functions
default
-------------2 | 2
3 | 4 2
4 | 6 4 2
5 | 8 6 4 2
-----------1 2 3 4
icost = 1,0.5
-------------2 | 2
3 | 3 2
4 | 4 3 2
5 | 5 4 3 2
-----------1 2 3 4
icost = 1,0
-------------2 | 2
3 | 2 2
4 | 2 2 2
5 | 2 2 2 2
-----------1 2 3 4
Example 1 To illustrate linear indel cost functions, we use the example data file seqm.d4 in Box 1. The command file is shown in Box 2. Box
3 shows the calculated distances for three different indel cost functions.
6.7.2.5
1
data-based substitution costs
6.7.2.5 Data-based Substitution Costs
The distance measure for sequences heavily depends on the specification of substitution and indel cost and, in general, there are only few
guidelines. One finally has to find a specification of cost functions that is
appropriate for the specific application. One should therefore investigate
the behavior of different specifications at least for a subset of “typical”
or easily interpretable sequences.
As a starting point, one can use the default cost functions where indel
cost = 1, and substitution cost = 2. The alignment of two sequences
provides then the longest common subsequence and, as mentioned in
6.7.2.3, there is a simple relationship between the distance and the length
of the longest common subsequence. This option is an interesting starting
point because it makes a substitution equivalent to two indel operations
(a deletion followed by an insertion), both cost the same.
This shows that the alignment will depend, in particular, on the
relation between indel and substitution cost. If substitution cost is higher
than the cost of two indel operations, the algorithm will never chose
substitutions. It has been suggested, therefore, to set indel cost equal to,
or slightly higher, than substitution cost, see Abbott [1990, p. 155]. But
as a consequence, the algorithm will then primarily use substitutions and
might not be able to fully exploit the possibilities of indel operations.
Further complications arise if the set of elementary operations is
extended to include consecutive indel operations, see 6.7.2.4. In this
case, the cost of inserting or deleting l consecutive elements is g(l) =
α + β(l − 1) and one has to find reasonable values for α and β. For some
sociological applications, it might be a good idea to set α equal to the
highest substitution cost and 0 < β < α.
For some applications, it might also be worthwhile to derive substitution cost from the frequency of transitions in the given sequence data.
To explain this idea, let us assume a sample of i = 1, . . . , n sequences
yi = (yi1 , . . . , yiT )
where the elements are valid states in the state space Y = {1, . . . , q}, or
are (negative) missing values.
d06070205.tex
April 21, 1998
6.7.2.5
data-based substitution costs
2
We can then define:
Nt (x) = # of sequences being in state x at t
Nt,t0 (x, y) = # of sequences being in state x at t
and in state y at t0
Using these quantities, there are several possibilities to define substitution cost. The intuition is that two states are less different when there
are, in the given data set, less transitions from one state into the other.
Time-independent substitution cost. The most simple way of using
the information about transitions would be to ignore the time structure
and, using
PT −1
p(x, y) =
t=1 Nt,t+1 (x, y)
PT −1
t=1 Nt (x)
define substitution cost by
2 − p(ai , bj ) − p(bj , ai ) if ai 6= bj
w(ai , bj ) =
0 if ai = bj
Then 0 ≤ w(ai , bj ) ≤ 2, and substitution cost directly reflects the cumulated transitions across states. If there are many transitions from ai to
bj (both directions), substitution cost is low, and vice versa.
This definition of substitution cost can be requested with the parameter
scost=2; it is easily calculated and does not require additional storage
space. If requested, a test output file will show the corresponding substitution cost matrix which, in turn, can be used to find an appropriate
indel cost function.
Example 1 To illustrate this option, we use data file seqm.d4 that
was introduced in 6.7.2.3. There are two states, 0 and 1; p(0, 1) = 4/15
and p(1, 0) = 4/10. Therefore, w(0, 1) = w(1, 0) = 4/3. See command
file seqm5.cf in the TDA example archive.
6.7.2.6
1
working with large samples
6.7.2.6 Working with Large Samples
Pairwise comparison of n sequences requires n(n − 1)/2 alignments. If
n is big, say greater than 1000, the computational burden becomes prohibitive, in particular with long sequences, since computing time is approximately proportional to the square of the sequence length.
Consider, for example, n = 1000 sequences with length 100 and 10
different states. On a SparcStation 10/30, using the current implementation of the alignment algorithm, 100 alignments need about 4 seconds.
In this example, n(n − 1)/2 = 499500, and the total time to calculate
these distances would be about 5.5 hours. For n = 2000, computing time
would already be more than 22 hours. However, in many social science
applications, the number of sequences (cases) is several thousand and the
calculation of all pairwise distances is no longer feasible. We therefore
shortly discuss three possible ways to reduce the computational burden
if the number of sequences is large.
Random selection. The easiest way is to select a random subsample
and calculate all pairwise distances for the sequences in the subsample.
Since this is easily done by a random select statement when reading the
input data, a separate option in the seqm command is not necessary.
It could be preferable, however, to randomly select pairs of sequences
from the full sample. To provide this option, the seqm command offers the
r parameter (only with m=1). The syntax is r=m, where m is the number
of desired alignments. The algorithm to randomly select approximately
m pairs of sequences is shown in Box 1.
Predefined sequences for comparison. It sometimes may make sense
to compare all sequences with only a few predefined sequences. This
might be sensible if one already has some ideas about “typical” sequences
to be used for comparison. The seqm command provides for this strategy
by its cn parameter to select a set of sequences for comparison.
Alternatively, one can try a two-step procedure. In a first step, find
two sequences having a large distance based on randomly selected pairs
of sequences. This can be done since as part of its standard output,
TDA prints the case numbers of those two sequences which have the
largest distance. Then, in a second step, compare all sequences with
these two sequences. The resulting distances can then be plotted in a
d06070206.tex
April 21, 1998
6.7.2.6
working with large samples
2
Box 1 Random selection of pairs of sequences
for (i = 2,...,n) {
for (j = 1,...,i - 1) {
r = random number between 0 and 1
if (r < m / (n (n - 1) / 2))
calculate distance between sequences i and j;
else
skip;
}
}
two-dimensional coordinate system.
Restricted alignment. Another strategy to reduce the computational
burden is to restrict the possible alignments of sequences, see Kruskal
and Sankoff [1983, pp. 276–281]. Given two sequences, a = (a1 , . . . , am )
and b = (b1 , . . . , bn ), the algorithm described in 6.7.2.1 requires the
recursive calculation of an (m + 1, n + 1) matrix D = (dij ); dij being the
optimal distance between the subsequences ai and bj .
The idea is to calculate only a part of this D matrix, for instance,
only elements with | i − j | ≤ K. In effect, this excludes some possible
alignments from consideration; but this might be justifiable not only
for computational reasons but also with respect to the application. As
discussed by Kruskal and Sankoff [1983], p. 278], a suitable formula that
covers the case of unequal sequence lengths is
i
j −
m+1 n+1 ≤
m+1
1
n + 1 .p
K√
+
(m + 1)2 + (n + 1)2
m+1
2 n+1
(In the special case that m = n, this reduces to | i − j | ≤ K.) The seqm
command provides this option with the max parameter. The syntax is
max=K with an integer K ≥ 1. Then only that part of the D matrix is
calculated and used to find an optimal alignment where the indices are
in the range given by the above mentioned formula. For a variety of other
approaches to restrict the alignment of sequences, see the discussion in
Kruskal and Sankoff [1983].
6.7.2.7
1
comparing parallel sequences
6.7.2.7 Comparing Parallel Sequences
Sometimes one is interested in comparing pairs of sequences where each
pair of sequences is defined simultaneously for each case in the input
data. A typical application would be a comparison of partners living
together in some relationship. This can be done with the m=2 option.
The option requires to specify two sequence data structures with the
parameter sn=k1 , k2 . There are then two sequences for each case in the
input data to be compared and the result will be written into the output
file. Given n cases, the output file will contain just n records.
A useful additional option is provided by the s=1 parameter providing information about how the distance between the two sequences
evolves over time. This could be useful, for instance, for a dynamical
investigation of homogamy.
d06070207.tex
April 21, 1998
6.9
1
least squares regression
6.9 Least Squares Regression
This chapter deals with least squares regression. Available commands
are described in the following subsections.
6.9.1
The lsreg Command to be used for linear OLS regression.
6.9.2
Regression with Censored Data describes the lsreg1 command
that can be used when observations of the dependent variable
are right censored. Note that this is currently only an experimental command.
d0609.tex
December 23, 1999
6.9.1
1
the lsreg command
6.9.1 The lsreg Command
The discussion of least squares estimation in this section is confined to
cross-section data. There is a single dependent variable, Yi , and a vector
of exogenous (non-stochastic) covariates, Xi , for cases i = 1, . . . , n. For
ease of notation, Xi is taken as a row vector including a constant one to
provide an intercept term in the regression model. As an option one can
suppress the intercept, see below.
An ordinary least squares (OLS) model can be written as
Yi = Xi β + i
i = 1, . . . , n
β is the parameter vector with length equal to the number of covariates
in Xi . i is the residual for individual i. The same model may be written
in matrix notation as
Y = Xβ + (1)
With m covariates, X is an (n, m) matrix, Y and are both vectors of
length n, and β is a vector of length m.
A least squares estimate of the parameter vector β is found by minimizing the Euclidean norm of the residuals as a function of the model
parameters, i.e.
!1/2
n
X
2
kk =
i
→ min
(2)
i=1
The solution, if existent, is found by
β̂ = (X 0 X)−1 X 0 Y
(3)
where X 0 is the transpose of X. A unique solution exists if X has full
column rank; this implies that n ≥ m.
Several different algorithms can be used to solve (2). TDA’s main algorithm to solve least squares problems is based on algorithms published
by Hanson and Haskell [1982]. These algorithms extend earlier work of
Lawson and Hanson [1974], providing, in particular, the possibility to
solve the least squares problem with linear equality and inequality constraints. A variety of other algorithms is discussed in Späth [1992].
d060901.tex
December 18, 1999
6.9.1
the lsreg command
2
Box 1 Syntax for lsreg command
lsreg (
ni=1,
lsecon=...,
lsicon=...,
dgrp=...,
tfmt=...,
s=1,
df=...,
fmt=...,
dtda=...,
ppar=...,
pcov=...,
pres=...,
mfmt=...,
mplog=...,
mppar=...,
mpcov=...,
if without intercept, def. ni=0
equality constraints, see 6.9.1.2
inequality constraints, see 6.9.1.2
estimate dummy variables with constraints
print format for results, def. tfmt=10.4
use robust covariance matrix, def. s=0
print data to an output file
print format for df option, def. 10.4
TDA description file for df option
print estimated coefficients to output file
print covariance matrix to output file
print residuals to output file
print format for pcov and pres option
write norm of residuals into matrix
write estimated parameters into matrix
write estimated covariance matrix
into matrix
) = varlist;
Command Description. The command for least squares regression is
lsreg. Its syntax is
lsreg (parameter) = varlist;
The optional parameters are described below, a summary is given in
Box 1. The right-hand side must provide a list of variables for the regression. It is a comma-separated list of variable names that must be
given in the following form:
Y1 , X1 , X2 , . . . , Xm
First comes the name of the dependent variable, then follow names of
independent variables. There must be at least one variable name.
1. As default, the model specification contains an intercept term. Using
the parameter ni=1 estimates a model without an intercept.
2. The parameters lsecon and lsicon can be used to specify parameter
constraints and will be explained in 6.9.1.2.
6.9.1
the lsreg command
3
3. The dgrp parameter can be used to specify sets of dummy variables
to be estimated as deviations from a weighted mean, see 6.9.1.3.
4. Results of model estimation are written into the standard output.
The print format can be controlled with the tfmt parameter, default is
tfmt=10.4. A description of the output is given below in Example 1.
5. The data used for model estimation can be written into an output
file with the parameter
df = name of an output file,
If there are n cases and m independent variables, the output file will
contain n records. Each record has m + 1 entries, first the independent
variables and finally the dependent variable. The print format can be
controlled with the fmt parameter, default is fmt=10.4. In connection
with the df parameter one can request an additional output file with the
dtda parameter containing a TDA description of the data file.
6. The parameter
ppar = name of an output file,
can be used to write the estimated model coefficients (and standard
errors) into an output file. The print format is determined by tfmt.
7. As default, calculation of standard errors uses the least squares covariance matrix
σ̂ 2 (X 0 X)−1
where σ̂ 2 is the estimated variance of residuals. Optionally, one can use
the parameter s=1 to request a “robust” covariance matrix as proposed
by White [1980]. The covariance matrix is then calculated as
(X 0 X)−1 (X 0 diag{e2i } X) (X 0 X)−1
where diag{e2i } is an (n, n) diagonal matrix containing the squared residuals.1
1 There are also other proposals for defining a robust covariance matrix, see Greene
[1992, p. 250]. The TDA example archive contains a command file, lsreg12.cf, that
can be used to replicate the example given there. The command file lsreg12a.cf
contains a TDA matrix procedure for calculating White’s robust covariance matrix.
6.9.1
the lsreg command
4
Box 2 Command file lsreg1.cf
nvar(
dfile = lsreg1.dat,
Height = c1,
Weight = c2,
);
lsreg(
pcov = cov,
ppar = par,
df = df,
pres = res,
) = Weight,Height;
# data file
#
#
#
#
write
write
write
write
covariance matrix to cov
parameter to par
data to df
residuals to res
# variables
8. The parameter
pcov = name of an output file,
can be used to write the estimated covariance matrix into an output file.
This will depend on the s parameter. The print format can be controlled
with the mfmt parameter, default is mfmt=12.4.
9. The pres parameter can be used to request an additional output file
with regression residuals, see 6.9.1.1.
Example 1 To illustrate the output of the lsreg command, we use a
simple example from SAS/STAT User’s Guide, Vol. 2, p. 1354. The data
file, lsreg1.dat, contains two variables, Weight, and Height. Box 2
shows the command file, lsreg1.cf.
The nvar command reads the data file and defines the two variables,
Weight and Height. Then follows the lsreg command for least squares
regression. Weight is the dependent and Height is the independent variable. The command contains parameters to write for additional output
files. Standard output from the lsreg command is shown in Box 3. It
begins with information about the variables and number of cases. Then
follows some information about the least squares solution.
1. The rank of the least squares data matrix is the rank of the matrix
X in (1). For a full rank situation, it should be equal to the number of
independent variables, including (if not suppressed) the intercept. If the
data matrix is rank deficient, TDA’s least squares algorithm calculates
a solution based on a generalized inverse. This solution is not unique. A
warning message is then written into the standard output.
6.9.1
the lsreg command
5
Box 3 Part of standard output from command file lsreg1.cf
Least squares regression.
Variables (cross-section)
------------------------Y
: Weight
X1 : Height
Equality constraints: 0
Inequality constraints: 0
Reading data. Cases: 19
Data written to: df
Rank of least squares data matrix: 2
Norm of least squares residuals: 46.287
Degrees of freedom: 17
Sum of squared residuals: 2142.49
Variance of residuals: 126.029
Squared multiple correlation: 0.770507
Adjusted: 0.757007
F-statistic: 57.0763
Level of significance: 0.999999
Covariance matrix written to: cov
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept -143.0269
32.2746
-4.4316 0.9996
2
1 Height
3.8990
0.5161
7.5549 1.0000
Parameter estimates written to: par
Residuals written to: res
2. Then follows the norm of the least squares residuals, see (2).
3. The degrees of freedom is the number of rows minus the number of
linear independent columns in the data matrix X.
4. The sum of squared residuals, sr , is calculated as the square of the
norm of the residuals.
5. The variance of the residuals is sr /d, that is, the sum of squared
residuals divided by the number of degrees of freedom.
6.9.1
the lsreg command
6
6. The squared multiple correlation is calculated as
R2 =
sq − sr
sq
where sq = syy −
sy sy
, sy = Σyi , syy = Σyi2
n
yi (i = 1, . . . , n) are the values of the dependent variable. The adjusted
R2 is only calculated if the number of degrees of freedom, d, is positive.
The formula is
2
Radj
= 1 − (n − 1)(1 − R2 ) / d
7. The value of the F -statistic is calculated as
F =
R2 /(m − 1)
(1 − R2 )/d
given that the data matrix has full column rank and there is a positive
number of degrees of freedom. m is the number of independent variables, including the intercept. The significance level is the value of the
F -distribution, at F , with m − 1 and d degrees of freedom.
8. Finally, the lsreg command prints a table with estimated coefficients,
β̂j , and standard errors, σ̂j . If the model is specified without constraints,
standard errors are only calculated if the data matrix has full column
rank and there is a positive number of degrees of freedom. The significance is calculated as 2 t(β̂j /σ̂j , d) − 1 from the t-distribution with
d degrees of freedom. As described above, standard errors depend on
the type of covariance matrix. As default, TDA uses the standard OLS
covariance matrix. If using the s=1 parameter, calculation of standard
errors is based on White’s [1980] robust covariance matrix.
6.9.1.1
residuals
1
6.9.1.1 Residuals
When using the lsreg command for least squares regression, the pres
parameter can be used to request an additional output file containing
regression residuals. The syntax is
pres = name of an output file,
The number of records in the output file equals the number of data matrix cases used for the regression. Each record of the output file contains
the following entries.
1. The case number.
2. Values of the independent variables Xi1 , . . . , Xim used in the regression model, including (if not suppressed) a constant 1 for the intercept.
3. The value of the dependent variable, Yi .
4. The estimate of the dependent variable, Ŷi .
5. The residual, ei = Yi − Ŷi .
6. The leverage value, hii , i.e., the diagonal elements in X(X 0 X)−1 X 0 .
7. An estimate for the standard deviation of the residual, SD(ei ), calculated as
p
SD(ei ) = σ̂ 1 − hii
σ̂ is the estimated standard deviation of residuals.
8. The standardized residuals, ei / SD(ei ).
The output file can be used as a data input file in subsequent calls of TDA
to investigate and plot residuals; some header lines describe the variables.
The print format can be controlled with the mfmt parameter, default is
mfmt=12.4. Leverage values and standard deviations for residuals are
only calculated if there is a valid estimate for the covariance matrix,
otherwise the corresponding columns in the output file will contain the
value -1.
d06090101.tex
April 21, 1998
6.9.1.2
parameter constraints
1
6.9.1.2 Parameter Constraints
The least squares problem described in 6.9.1 can be modified by imposing constraints for the model parameters, often called restricted least
squares. Our discussion will be confined to two kinds of linear constraints.
First, equality constraints which may be written
Rβ = d
(1)
where R is an arbitrary (nr , m) matrix and d is a vector of length nr .
Second, there may be inequality constraints which can be written
Uβ ≥ h
with U an arbitrary (nu , m) matrix and h a vector of length nu . In both
cases, the constraints are assumed to be non-stochastic.
Taking into account the possibility of linear equality and/or inequality constraints, a general least squares problem may be written as




R
d
} nr
. 



(2)
 X  β =  Y  } nx
U
h
} nu
nr is the number of equations to be satisfied exactly, nx (equal to n
as used in the introductory section) is the number of equations to be
satisfied in a least squares sense, and nu is the number of inequality
.
constraints. The symbol = is used to denote an equality, a least squares,
and an inequality relation, respectively. Some of the many possible combinations are shortly discussed.
1. If nx > 0 and nr = nu = 0, the general system (2) reduces to a
standard regression approach without constraints. TDA tries to calculate
the standard OLS solution as defined in 6.9.1. A unique solution exists
if nx ≥ m and X has full column rank, m. If X has less than full column
rank, many solutions exist that equally well minimize the norm of the
residuals. TDA gives one of these solutions. In any case, the program
provides information about the rank of X. Due to limited numerical
accuracy, this is actually a pseudo rank.
d06090102.tex
April 21, 1998
6.9.1.2
parameter constraints
2
2. If nr > 0 and nx = nu = 0, one has a simple system of linear
equations. If nr = m and R has full column rank, m, there is exactly one
solution. If nr < m there may be a unique solution, or a set of different
solutions, or the equations may be contradictory. If the solution is not
unique, TDA provides just one. In any case, one gets information about
the number of independent rows in R and about whether the system is
contradictory or not. If nr > m the system is solved in a least squares
sense.
3. If nu > 0 and nx = nr = 0, one has a pure system of inequalities. Again, there may be a unique solution (unique in the sense of a
solution vector with minimal Euclidean length), there may exist many
(linear independent) solutions, or the inequalities may be contradictory.
If possible, TDA calculates a solution with minimal norm.
4. If nx > 0, nr > 0, and nu = 0, one has a least squares problem
with equality constraints. To find a solution of this problem is tried in
two steps. First, it is tried to solve the equality constraints. If there is
no, or exactly one, solution, the least squares equations are ignored. If
the solution space has a positive dimension, the least squares problem
is reformulated in this solution space and an optimal solution, based on
the least squares equations, is determined. Of course, a unique solution
is not guaranteed if X is rank deficient.
5. If nx > 0, nu > 0, and nr = 0, one has a least squares problem
with inequality constraints. To find an optimal solution the system of
inequalities must not be contradictory. In addition, the rank of the least
squares equations must be at least equal to the dimension of the solution
space of the inequality constraints.
6. Finally, if nx > 0, nr > 0, and nu > 0, one has a least squares
problem with both, equality and inequality constraints. Again, it is tried
to find a solution by first solving the system of equality constraints.
The remaining problem is reformulated, then, in the solution space of
the equality system. If this solution space has positive dimension, the
algorithm proceeds in the same way as explained above.
Specification of constraints. TDA uses all rows of the data matrix to
build the least squares problem, Xβ ∼
= Y ; constraints are only added if
defined explicitly. Following (2), the syntax to define equality constraints
is
lsecon = ri0 b0 ± ri1 b1 ± . . . ± rin bn = di
(3)
6.9.1.2
parameter constraints
3
rij and di are floating point numbers to be used as the coefficients of the
matrix R and the vector d, respectively. The bj are key words and must
be given as b0,b1,... There must be a separate lsecon parameter for
each equality constraint. If rij is zero, the term can be omitted. b0 can
only be used if the model contains an intercept term.
Analogously, the syntax for the specification of inequality constraints is
lsicon = ui0 b0 ± ui1 b1 ± . . . ± uin bn = hi
(4)
with uij and hi , respectively, the coefficients of the U matrix and the h
vector, see (2).
Covariance matrix. Given a regression model with nr equality constraints,1 the least squares estimator is the solution of
(Y − Xβ)0 (Y − Xβ) → min
s.t. Rβ = d
The solution can be found by using an nr vector, λ, of Lagrangian multipliers, i.e. by minimizing
L(β, λ) = (Y − Xβ)0 (Y − Xβ) + 2 λ0 (Rβ − d)
In matrix notation, the first-order conditions can be written
0
0 X X R0
β
XY
=
R
0
λ
d
(5)
We shall assume that there is a unique solution, (β̂, λ̂)0 , i.e. that the
matrix on the left-hand side has full rank, m + nr . This implies that the
equality constraints are linearly independent. As pointed out by Greene
and Seaks [1991], it is not required that also X has full rank. As an
implication of the first-order conditions, one finds that
X 0 X(β̂ − β) = X 0 − R0 λ̂
This shows that, if the unrestricted least squares estimator exists (X has
full rank), then β̂ is an unbiased estimator of β,
β̂ − β = (X 0 X)−1 X 0 − (X 0 X)−1 R0 λ̂
1 A covariance matrix is only calculated if there are no inequality constraints. Derivation of stochastic properties for models with both, equality and inequality constraints
is difficult; see, for instance, Judge et al. [1988, chap. 20]. So we shall currently not
discuss the case of inequality constraints.
6.9.1.2
parameter constraints
4
Box 1 Data file lsreg2.dat
X1 X2
Y
-----------1
0
1
0
1
1
1
1
1
Box 2 Command file lsreg2.cf
nvar(
dfile = lsreg2.dat,
X1 = c1,
X2 = c2,
Y = c3,
);
lsreg (
ni=1,
# without intercept
lsecon = b1 + b2 = 6,
lsicon = b1 = 2,
lsicon = b2 = 2,
) = Y,X1,X2;
# variables
and the covariance matrix of β̂ may be derived2 as
h
−1 i 0 −1
Var[β̂] = σ 2 I − (X 0 X)−1 R0 R(X 0 X)−1 R0
R (X X)
As shown by Greene and Seaks [1992], there is a more general formulation
of the covariance matrix of β̂ which does not require that the unrestricted
least squares estimator exists. Using W to denote the matrix on the lefthand side of (5), it follows that
Var
β̂
λ̂
=W
−1
= W −1
X 0Y
d
σ2 X 0 X
0
0
0
Var
W −1
W −1
It is only required that W has full rank. The formula may be further
simplified as shown by Greene and Seaks [1991].
2 See,
e.g., Johnston [1972, p. 158].
6.9.1.2
parameter constraints
5
Box 3 Part of standard output from command file lsreg2.cf
Least squares regression. Current memory: 144677 bytes.
Variables (cross-section)
------------------------Y
: Y
X1 : X1
X2 : X2
Model without intercept.
LSECon: 1 b1 + 1 b2 = 6
LSICon: 1 b1 = 2
LSICon: 1 b2 = 2
Equality constraints: 1
Inequality constraints: 2
Reading data. Cases: 3
Covariance matrix not calculated.
Rank of reduced least squares data matrix: 1
Norm of least squares residuals: 5.74456
Rank of equality constraints: 1
Norm of residuals of equality constraints: 0
Sum of squared residuals: 33
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
1 X1
3.0000
------2
1 X2
3.0000
-------
Example 1 To illustrate constrained least squares we consider the following example:
β1
β1
β1
β1
+ β2
β2
+ β2
β2
=
∼
=
∼
=
∼
=
≥
≥
6
1
1
1
2
2
The data correspond to example 51 in Späth [1992, p. 322]. In this example, nr = 1, nx = 3, and nu = 2. Box 2 shows the command file
6.9.1.2
parameter constraints
6
lsreg2.cf that can be used to find a solution, Box 3 shows part of the
standard output. Since the rank of the reduced least squares matrix is
less than 2, there are no standard errors.
6.9.1.3
1
dummy variables
6.9.1.3 Dummy Variables
Let D1 , . . . , Dk denote a complete set of dummy variables, meaning that
for each case in the data set exactly one of these variables takes the
value 1. Since the intercept column is linear dependent on this set of
dummy variables, one normally has to drop one of them that creates the
reference category. Using parameter constraints provides an alternative.
One can keep the full set of dummy variables and add the constraint
X
wj Dj βj = 0
j
with wi some set of weights; wi > 0, Σi wi = 1, and βj the regression
parameters corresponding to the dummy variables Dj . This can be specified with the lsecon parameter.
As discussed by Haisken-DeNew and Schmidt [1997], a reasonable
approach for many applications is to define the weights to be the proportion of cases in the respective category. This is supported by the dgrp
parameter in the lsreg command. The syntax is
dgrp = [DL1],[DL2],...,
and DL1, DL2, and so on are namelists containing dummy variables which
are part of the set of independent variables used in the currently specified
regression model. For each of these namelists TDA calculates the proportion of cases in each dummy variable and creates an equality constraint
using these proportions as weights. Up to 20 sets of dummy variables can
be specified on the right-hand side of the dgrp parameter. For example,
assume that the lsreg command specifies the variables
lsreg (parameter) = Y, D1 , D2 , D3 , D4 , U1 , U2 , U3 , X1 , . . .;
where DL1 = {D1 , D2 , D3 , D4 } and DL2 = {U1 , U2 , U3 } are two sets of
dummy variables. Given then the parameter
dgrp = [DL1],[DL2],
TDA creates two equality constraints:
d1 D1 β1 + d2 D2 β2 + d3 D3 β3 + d4 D4 β4 = 0
d06090103.tex
April 21, 1998
6.9.1.3
dummy variables
2
Box 1 Command file lsreg3.cf
nvar(
dfile = lsreg3.dat,
G = c2,
Y = c3,
NE = c4,
NC = c5,
SO = c6,
WE = c7,
Y2 = Y * Y,
);
nlist(
DL = NE,NC,SO,WE,
);
lsreg (
dgrp = [DL],
tfmt = -14.6,
) = G,Y,Y2,DL;
and
u1 U1 γ1 + u2 U2 γ2 + u3 U3 γ3 = 0
with di and ui the proportion of cases in the corresponding category.
The regression model is then estimated by using these constraints (in
addition to any constraints explicitly defined with the lsecon and/or
lsicon parameters).
Information about the dummy variables and the corresponding proportion of cases is written into the standard output. In addition, as
proposed by Haisken-DeNew and Schmidt [1997], TDA calculates, separately for each group of dummy variables, a dispersion measure. If Di
denotes the index set for the ith set of dummy variables, this dispersion
measure is calculated as
sX
wj (β̂j2 − σ̂j2 )
j∈Di
where wj are the weights of the dummy variables and β̂j and σ̂j are the
corresponding parameter and standard error estimates.
Example 1 To illustrate the dgrp parameter, we replicate an example
given in Haisken-DeNew and Schmidt [1997]. The data file, lsreg3.dat,
contains information about gasoline demand in four different regions
6.9.1.3
dummy variables
Box 2 Standard output from lsreg3.cf
Least squares regression. Current memory: 268781 bytes.
Processing dgrp option.
Group selection: [DL]
Variable
cases
weight
------------------------------NE
12
0.2500
NC
12
0.2500
SO
12
0.2500
WE
12
0.2500
Equality constraints: 1
Inequality constraints: 0
Reading data. Cases: 48
Rank
Norm
Rank
Norm
of
of
of
of
reduced least squares data matrix: 6
least squares residuals: 3.50111
equality constraints: 1
residuals of equality constraints: 0
Degrees of freedom: 42
Sum of squared residuals: 12.2578
Variance of residuals: 0.291852
Squared multiple correlation: 0.973878
Adjusted: 0.970768
Idx Wave Variable
Coeff
Error
Coeff/E
----------------------------------------------------------------1
- Intercept
1.045236e+00
1.978744e-01
5.282321e+00
2
1 Y
7.778210e-04
2.982920e-05
2.607583e+01
3
1 Y2
-1.332484e-08
7.953897e-10 -1.675260e+01
4
1 NE
-7.873645e-01
1.351198e-01 -5.827160e+00
5
1 NC
-3.929084e-02
1.351144e-01 -2.907968e-01
6
1 SO
4.296827e-01
1.351670e-01
3.178902e+00
7
1 WE
3.969727e-01
1.350824e-01
2.938745e+00
DGroup weighted adj. stand. dev.
--------------------------------[DL]
4.718767e-01
3
6.9.1.3
dummy variables
4
and is taken from Suits [1984].1 The variables are G (gasoline demand),
Y (income), and four dummy variables for regions (NE,NC,SO,WE).
Box 1 shows the command file lsreg3.cf. The lsreg command
specifies a simple regression model with dependent variable G. The independent variables are Y, Y squared, and the four dummy variables.
As requested by the dgrp option, these dummy variables are treated as
a complete set and a corresponding constraint is added to the model
specification.
Box 2 shows the standard output. In this example, there are 12 cases
in each region, so all weights are equal to 0.25 and the equality constraint
is simply
0.25 NE + 0.25 NC + 0.25 SO + 0.25 WE = 0
1 The
same data were used by Greene and Seaks [1991].
6.9.2
regression with censored data
1
6.9.2 Regression with Censored Data
This section describes the lsreg1 command that can be used for least
squares estimation when dependent variables refer to possibly right censored observations. Note that the current implementation is experimental
and not yet finished.
The command is intended to estimate parameters for several regression
equations simultaneously. The basic model consists of n regression equations
Y1 = x1 β1 + 1
Y2 = x2 β2 + 2
..
.
Yn = xn βn + n
Yj is the dependent (random) variable in the jth regression equation, xj
is a (1, m) vector of given covariates, and j is the corresponding residual.
It is assumed that, for each dimension j = 1, . . . , n, observations are
given in the following way:
yj1
yj2
..
.
δj1
δj2
..
.
xj,11
xj,21
..
.
···
···
xj,1m
xj.2m
..
.
yjnj
δjnj
xj,nj 1
···
xj,nj m
nj is the number of observations for the jth regression equation (dimension). yji is the ith observed value for dependent variable Yj ; δji is a
corresponding censoring indicator: if δji = 1, yji is assumed to be an
exactly observed value, if δji = 0, yji is assumed to be a right censored
observation.
Box 1 shows the syntax for the lsreg1 command. Most parameters
are optional. Required is the name of the dependent variable (yl parameter) and the name of a variable containing the censoring information
(cen parameter).
1. How the command estimates model parameters depends on the selected option (opt parameter). Available options will be explained below.
d060902.tex
December 23, 1999
6.9.2
regression with censored data
2
Box 1 Syntax for lsreg1 command
lsreg1 (
opt=...,
yl=...,
cen=...,
ni=1,
grp=ID,L1,
mxit=...,
tolp=...,
sc=...,
n=...,
mxitl=...,
tolf=...,
d=...,
tfmt=...,
prot=...,
ppar=...,
pres=...,
fmt=...,
dtda=...,
calculation of expected values, def. 1
1 = based on marginal Kaplan-Meier
2 = based on joint distribution, method 1
3 = based on joint distribution, method 2
name of dependent variable
variable name for censoring information
if without intercept, def. ni=0
specification of dimensions
maximal number of iterations, def. 20
tolerance for convergence, def. 0.0001
offset for domain (method 1 and 2), def. 0
number of boxes in grid (method 1), def. 100
maximal number of iterations (method 1), def. 20
tolerance for convergence (method 1), def. 0.001
delta specification (method 2), def. 0.1
print format for results, def. 10.4
protocol file for diagnostic information
print estimated parameters to output file
print data and residuals to output file
print format for pres option, def. 10.4
TDA description file for pres option
) = varlist (indep. variables);
2. Independent variables can be specified on the right-hand side. If no
variables are given, the command estimates a model where each regression equation only contains a single intercept. By default, an intercept is
added to each regression equation. The ni = 1 parameter can be used
to suppress intercepts. Of course, the command then needs at least one
independent variable.
3. By default, the command assumes a single regression equation where
each row in the current data matrix provides one observation for this
regression equation (n = 1, n1 = NOC). The grp parameter can be used
to specify two or more regression equations. The syntax is
grp = ID, L1,
where ID and L1 are names of data matrix variables. Each block of data
matrix rows where ID has identical values is interpreted as data for one
6.9.2
regression with censored data
3
unit of observation. Any values are possible and since the data matrix is
always sorted with respect to ID and L1, it is not required that blocks are
contiguous. The L1 variable must contain positive integers. The number
of different integers found in this variable is interpreted as the number
of regression equations (dimensions). Again, it is not required that these
numbers are contiguous. Each data matrix row provides one observation
for the regression equation specified by the corresponding value of the
L1 variable. (For an illustration, see the description of the gdf command
in 6.2.2.)
4. Resulting parameter estimates are written into the standard output,
the print format can be specified with the tfmt parameter. Parameter
estimates are also written to an output file when an output file name is
given with the ppar parameter.
5. A further output file can be requested with the pres parameter. This
file will then contain a copy of the data, expected values of the dependent
variables, given the data, and residuals. A description of the variables
contained in this file can be requested with the dtda parameter. The
print format can be controlled with the fmt parameter.
6. The remaining parameters are used to control the iterative estimation procedure and depend on which option is selected with the opt
parameter. They will be explained below.
Estimation Procedure
The estimation procedure is a multivariate generalization of an approach
originally proposed by Buckley and James [1979]. Assume the data are
given by






y1,1
δ1,1
x1,11
· · · x1,1m
 ..

 ..
 ..


..
 .

 .

 .

.






 x1,n1 1 · · · x1,n1 m 
 y1,n1 
 δ1,n1 








 .
 .


..
X =  ...
 y =  ..
 δ =  ..

.






 xn,11





·
·
·
x
y
δ
n,1m 

 n,1 
 n,1 

 .

 .

 .
.
..
 ..

 ..
 ..


xn,nn 1 · · · xn,nn m
yn,nn
δn,nn
Ignoring censoring, standard OLS estimation would result in
β̂ = (X 0 X)−1 X 0 y
6.9.2
regression with censored data
4
Box 2 Iterative algorithm for parameter estimation
(1) b = (X 0 X)−1 X 0 y
(2) e = y − Xb
(3) calculate
new values for dependent variable:
yji if not censored
∗
yji =
conditional expectation, based on data and e
(4) b∗ = (X 0 X)−1 X 0 y ∗
∗
| bk − bk |
(5) check convergence: max
≤ TOLP
k
max{b∗k , 1}
(6) end if convergence has been achieved, or the maximal
number of iterations has been reached.
(7) b = b∗ , continue with step (2)
These parameter estimates are then used as starting values for an iterative procedure where, in each iteration, censored observations are substituted by their conditional expectations, given the data and the current
values of the parameter estimates. Box 2 shows the basic steps of the
iterative algorithm.
A problem only occurs in step (3) where we have to calculate expected
values for the censored observations, conditional on data and current
values of model parameters. The proposal made by Buckley and James
[1979] was to use
∗
yji
= δji yji + (1 − δji ) EF̂j (Yj | Yj > yji , xji , bj , ej )
(1)
where bj and ej refer, respectively, to the current parameter estimates
and residuals for the jth regression equation, and F̂j is the Kaplan-Meier
estimate of the distribution function of these residuals. For censored
observations, the conditional expectation is then calculates by
EF̂j (Yj | Yj > yji , xji , bj , ej )
= EF̂j (xji bj + j | yji − xji bj < j ≤ yji − xji bj , xji , bj )
= xji bj + EF̂j (j | yji − xji bj < j ≤ yji − xji bj , xji , bj )
Z
Z ∞
∞
= xji bj +
e dF̂j
dF̂j
yji −xji bj
yji −xji bj
6.9.2
regression with censored data
5
This method to calculate conditional expectations for censored observations is implemented as the default option (opt = 1) of the lsreg1
command. The tolerance, TOLP, can be specified with the tolp parameter, the maximal number of iterations with the mxit parameter.
6.10
1
alternative regression methods
6.10 Alternative Regression Methods
While least squares regression is most often used in applied work, there
are many alternative approaches, see, e.g., Birkes and Dodge [1993]. This
chapter is intended to discuss some of these alternatives. Currently, we
have the following sections.
6.10.1
L1-Norm Regression
6.10.2
Nonparametric Regression
d0610.tex
May 23, 1998
6.10.1
l1 norm regression
1
6.10.1 L1-Norm Regression
An obvious alternative to least squares regression is to minimize absolute deviations, instead of squared deviations. This is called L1 norm
regression. See, e.g., Dielman and Pfaffenberger [1982], Birkes and Dodge
[1993, ch. 4]. For a comprehensive bibliography see Dielman [1984]. Given
a sample of data, (yi , xi ), where xi is a row vector of covariates, a linear
L1 norm regression will find parameter estimates by minimizing
n
X
| yi − xi β | −→ min
i=1
The TDA command to solve this problem is called l1reg, its syntax is
shown in Box 1. The command uses a linear programming algorithm
developed by Barrodale and Roberts [1974]. For an extensive discussion
of this and related algorithms see also Späth [1987]. The syntax for the
l1reg command is
l1reg (parameter) = varlist;
The right-hand side must provide a list of variables for the regression. It
is a comma-separated list of variable names that must be given in the
following form:
Y1 , X1 , X2 , . . . , Xm
First comes the name of the dependent variable, then follow names of
independent variables. The l1reg command can only be used with crosssectional data.
The solution of the L1 norm regression problem is not necessarily
unique. Information is given in the programm’s standard output. It may
also happen that the algorithm is not able to find a solution. If successful,
the l1reg command prints a table with estimated parameters into the
standard output.
In order to calculate standard errors TDA follows the advice of Birkes
and Dodge [1993, p. 63], See also Dielman and Pfaffenberger [1982]. The
steps are as follows.
d061001.tex
December 18, 1999
6.10.1
l1 norm regression
2
Box 1 Syntax for l1reg command
l1reg (
ni=1,
tfmt=...,
df=...,
fmt=...,
ppar=...,
pcov=...,
pres=...,
mfmt=...,
mplog=...,
mppar=...,
mpcov=...,
if without intercept, def. ni=0
print format for results, def. 10.4
print data to an output file
print format for df option, def. 10.0
print estimated coefficients to output file
print unscaled covariance matrix to output file
print residuals to output file
print format for pcov and pres option, def. 12.6
write norm of residuals into matrix
write estimated parameters into matrix
write unscaled covariance matrix
into matrix
) = varlist;
Box 2 Data file l1reg4.dat
Y X1 X2
---------37
4 22
40
6 24
48
6 18
44
9 20
50 11 15
51 12
9
1. The residuals, êi = yi − ŷi , are calculated and the non-zero residuals
are sorted in ascending order. Further calculations are only done if
the number of non-zero residuals, m, is at least 2.
√
2. Let k1 and k2√denote integers corresponding to (m + 1)/2 − m and
(m + 1)/2 + m, respectively. TDA then calculates
√
τ=
m (ê(k2 ) − ê(k1 ) )
4
The value of τ is reported in the standard output.
3. If the data matrix, X, has full column rank, standard errors are
6.10.1
l1 norm regression
3
Box 3 Command file l1reg4.cf
nvar(
dfile = l1reg4.dat,
Y = c1,
X1= c2,
X2 = c3,
);
l1reg = Y,X1,X2;
Box 4 Part of standard output from l1reg4.cf
L1-norm regression.
Variables (cross-section)
------------------------Y
: Y
X1 : X1
X2 : X2
Solution is unique
Rank of data matrix: 3
Sum of absolute deviations: 9.7619
Number of non-zero residuals: 3
Tau:
3.7218 [calculated from residuals: -1.16667,7.42857]
Pseudorank of data matrix: 3
Norm of least squares residuals:
5.7642
Degrees of freedom: 3
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
32.7143
16.8621
1.9401 0.8523
2
1 X1
1.5952
0.9462
1.6860 0.8096
3
1 X2
-0.0952
0.5537
-0.1720 0.1256
calculated as
S.E.(βj ) = τ
q
diag{(X 0 X)−1 }j
4. Levels of significance are calculated by using a t-distribution with
n − k degrees of freedom where n is the number of data points and
k the number of covariates.
Example 1 To illustrate we replicate an example discussed in Birkes
6.10.1
l1 norm regression
4
and Dodge [1993, p. 66]. The command file is l1reg4.cf, see Box 3. Part
of the standard output is shown in Box 4.
6.10.2
1
nonparametric regression
6.10.2 Nonparametric Regression
This section describes the npreg command for nonparametric regression
with a single independent variable. The syntax is shown in Box 1. The
right-hand side must provide exactly two variable names; the first one
will be used for the dependent variable, the second one for the independent variable. Also required is the name of an output file, to be specified
with the df parameter, and a list of values for the independent variable
to be specified with the x parameter (see below). All other parameters
are optional.
General Approach
Consider a 2-dimensional variable (X, Y ). We seek for a regression
x −→ Representation of {Pr(Y = y | X = x)}
that assigns to each value in the range of X a representation of the
conditional distribution of Y given x. Our general approach will be to
consider intervals
[xj − d/2, xj + d/2]
for j = 1, . . . , m
and calculate a representation of the conditional distribution of Y for
each interval separately. The interval width, d > 0, can be specified
with the d parameter. The list of values, xj , must be specified with the
parameter
x = x1 , . . . , xm
Alternatively, one can use the syntax
x = a(b)c
The expression on the right-hand side will then be expanded into
xj = a + (j − 1)b
for
j = 1, 2, . . .
as long as xj ≤ c. The output file will contain a separate record for each
value xj . The information provided will be described below, separately
for each option.
d061002.tex
May 25, 1998
6.10.2
nonparametric regression
2
Box 1 Syntax for npreg command
npreg (
opt=...,
k=...,
d=...,
x=...,
df=...,
fmt=...,
dtda=...,
sel=...,
method selection, def. 1
1 : smoothed means
2 : smoothed quantiles
3 : smoothed frequencies
type of kernel (if opt=1), def. 1
1 : uniform
2 : triangle
3 : quartic
4 : Epanechnikov
band width, def. d = 1.0
list of points for x axis (required)
output file (required)
print format, def. 10.4
TDA description of output file
optional case selection
) = Y,X;
Conditional Means
If opt = 1, the npreg command calculates conditional mean values:
xj −→ E(Y | X ∈ [xj − d/2, xj + d/2])
The calculation depends on the kernel selected with the k parameter.
By default, k=1, the npreg procedure uses a uniform kernel and simply
calculates the mean of all values of Y where the corresponding value of
X is in the interval [xj − d/2, xj + d/2]. In this case, the command also
calculates the conditional standard deviation of Y . Each record of the
output file will contain the following entries:
1. The record number j (records are j = 1, . . . , m).
2. The value of xj .
3. Number of observations in [xj − d/2, xj + d/2].
4. Mean of X in [xj − d/2, xj + d/2].
5. Mean of Y in [xj − d/2, xj + d/2].
6. Standard deviation of Y in [xj − d/2, xj + d/2].
6.10.2
3
nonparametric regression
Alternative Kernels
When estimating conditional means (opt=1) one can use the k parameter
to select one of the following kernel functions.1
1. Uniform:
Kd (x, xi ) := I(| x − xi | ≤ d/2)
1
d
2. Triangle:
Kd (x, xi ) := I(| x − xi | ≤ d/2)
2
d
1−
| x − xi |
d/2
3. Quartic:
15
Kd (x, xi ) := I(| x − xi | ≤ d/2)
8d
x − xi
d/2
x − xi
d/2
1−
2 !2
4. Epanechnikov:
3
Kd (x, xi ) := I(| x − xi | ≤ d/2)
2d
1−
2
Given one of these kernels and assuming data given by (Xi , Yi ), for i =
1, . . . , n, the conditional mean is calculated as
Pn
Kd (xj , Xi )Yi
Ê(Y | X = x) = Pi=1
n
i=1 Kd (xj , Xi )
Note that calculation of the conditional standard deviation of the dependend variable is always based on a uniform kernel.
Conditional Quantiles
If opt = 2, the npreg command calculates conditional quantiles:
xj −→ Qp (Y | X ∈ [xj − d/2, xj + d/2])
This is done for
p = 0.1, 0.2, 0.25, 0.3, 0.4, 0.5, 0.6, 0.7, 0.75, 0.8, 0.9
Each record of the output file will contain the following entries:
1 See,
e.g., Härdle [1991] for a discussion of kernel-based regression methods.
6.10.2
nonparametric regression
4
1. The record number j (records are j = 1, . . . , m).
2. The value of xj .
3. Number of observations in [xj − d/2, xj + d/2].
4. Mean of X in [xj − d/2, xj + d/2].
5. Then follow the conditional quantiles (11 values corresponding to
the list of p-values given above).
Conditional Frequencies
If opt = 3, the npreg command sorts the values of the dependent variable in ascending order, say
∗
y1∗ , . . . , yK
and then calculates conditional frequencies
xj −→ Pr(Y = yk∗ | X ∈ [xj − d/2, xj + d/2])
for k = 1, . . . , K. Each record of the output file will contain the following
entries:
1. The record number j (records are j = 1, . . . , m).
2. The value of xj .
3. Number of observations in [xj − d/2, xj + d/2].
4. Mean of X in [xj − d/2, xj + d/2].
5. Then follow K entries containing the proportions
Pr(Y = yk∗ | X ∈ [x − d/2, x + d/2])
Example 1 To illustrate we replicate an example discussed by Härdle
[1991, p. 124]. Box 2 shows the command file. The first nvar command
creates the data, defined by the function
yi = sin(2πx3i )3 + i
for
i = 1, . . . , 256
where the xi are uniformely distributed in [0, 1], and i is normally distributed with σ 2 ≈ 0.1. A scatterplot of the data is shown in Figure 1.
6.10.2
5
nonparametric regression
1.8
1.3
0.8
0.3
-0.2
-0.7
-1.2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Figure 1 Plot created with npreg1.cf (Example 1).
The function used to create the data (omitting the noise) is shown by a
dashed line.
Then follows the npreg command. We have selected conditional means
(opt=1) with a quartic kernel (k=3), interval width is d = 0.1. The estimated values are written into an output file, d.out. Finally, the current
data matrix is cleared, a new data matrix is created by reading the data
file d.out, and the regression line is put as an additional plot object into
the plot output file. Figure 1 shows the estimated regression line as a
solid curve.
6.10.2
nonparametric regression
Box 2 Command file npreg1.cf
nvar(
# creates some random
noc = 256,
X = rd,
E = 0.32 * rdn,
Z = sin(2 * pi * X * X * X),
Y = Z * Z * Z + E,
);
psfile = npreg1.ps;
plot output file
psetup(
pxlen = 90,
# length of x axis
pylen = 50,
# length of y axis
pxa
= 0,1,
# user coordinates
pya
= -1.2,1.8, # user coordinates
);
plxa (sc=0.1,ic=0);
plot x axis
plya (sc=0.5,ic=0);
plot y axis
plot(
s=5,
fs = 0.5,
lw = 0,
) = X,Y;
data
in
in
on
on
mm
mm
x axis
y axis
# scatterplot
# plot the function without noise
plotf (rx=0(0.01)1,lw=0.3,lt=5) = ss = sin(2 * pi * x * x * x),
fn = ss * ss * ss;
npreg(
opt = 1,
k = 3,
d = 0.1,
df = d.out,
x = 0(0.01)1,
dtda = t,
) = Y,X;
# nonparametric regression
# conditional means
# quartic kernel
clear;
# delete data matrix
nvar(
# read data created by npreg command
dfile = d.out,
X
= c2 , # x value
YM
= c5 , # mean of Y
);
plot(lw=0.5) = X,YM;
plot regression curve
6
6.11
1
maximum likelihood estimation
6.11 Maximum Likelihood Estimation
Estimation of many models in TDA is based on maximum likelihood. In
general, this requires finding the maximum of the model’s likelihood by
an iterative procedure. This chapter begins with a short introduction and
then explains TDA’s fml command that can be used to maximize userdefined log-likelihood functions. For an introduction to the mathematical
and algorithmic background see 5.6 on function minimization.
6.11.1
Introduction
6.11.2
The fml Command
d0611.tex
April 21, 1998
6.11.1
introduction
1
6.11.1 Introduction
The starting point of ML estimation is a hypothesis involving the distribution of a random vector, say Y . The hypothesis is parametric, meaning
that it depends on an unknown parameter vector, say θ = (θ1 , . . . , θq )0 ,
but is otherwise known. This allows to derive the distribution of Y ,
depending on θ. For any choice of θ, the hypothesis implies a specific
density of Y , say f (y, θ).
Evaluation of the hypothesis is based on a set of observations of Y . We
shall assume that there is a random sample of i = 1, . . . , n independent
observations yi , realizations of corresponding sample variates, Yi . Given
the sample of observations, its likelihood and log-likelihood are defined
by
L(y1 , . . . , yn , θ)
`(y1 , . . . , yn , θ)
=
=
n
Y
i=1
n
X
f (yi , θ)
(1)
log (f (yi , θ))
i=1
The ML principle simply states that a good estimator of θ can be found
by maximizing this likelihood or, equivalently, the log-likelihood function. We will denote this estimator by θ̂n .
The existence of the maximum likelihood estimator can be shown
under very general conditions, see, e.g., White [1982]. However, the estimator is not necessarily unique. This depends on the hypothesis (model)
used to derive the likelihood function and cannot be shown in general.
If, for instance, the likelihood function contains two linear dependent
variables, there is no longer a unique maximum. In what follows we will
assume that the likelihood function has a unique global maximum.
Justification of the ML estimator is often based on asymptotic considerations. To investigate its properties, the estimator is interpreted as
a function of the sample variates
θ̂n = θ̂n (Y1 , . . . , Yn )
(2)
and it is assumed that there is a density of Y , say g(y), according to which
the observed data have been generated. The problem is to compare g(y)
with f (y, θ̂n ), the estimated density based on maximum likelihood.
d061101.tex
December 18, 1999
6.11.1
introduction
2
Standard maximum likelihood theory is based on the assumption that
the family of distributions, given by f (y, θ), contains as a special case
the unknown distribution which has generated the data. Consequently,
there is a true parameter vector, say θ◦ , so that g(y) = f (y, θ◦ ). Given
this assumption, it can be shown under very general conditions that θ̂n
is a consistent estimator of θ◦ .
However, in almost all practical work it is very difficult, or impossible,
to justify the assumption of a correct model specification. An interesting justification of the ML estimator for situations where it is unknown
whether the assumption of a correct model specification holds has been
given by White [1982]. With reference to Akaike [1973], he shows how
ML estimation can be interpreted, then, as a method to minimize our
ignorance about the distribution that has generated the data.1 The reasoning is based on the Kullback-Leibler Information Criterion (KLIC),
defined by
I(g, f, θ) = E [log(g(Y )/f (Y, θ))]
(3)
with expectation taken with respect to g(y). Given that I(g, f, θ) has a
unique minimum, say θ∗ , it can be shown that θ̂n is a strongly consistent estimator for θ∗ .2 Furthermore, if there exists a θ◦ so that g(y) =
f (y, θ◦ ), it follows that θ∗ = θ◦ .
This generalization of the classical ML approach has an important
implication for the question how to find an appropriate estimate of the
limiting distribution of θ̂n . In any case, as shown by White, the difference
θ̂n − θ∗ follows asymptotically a normal distribution with mean zero:
√
A
n(θ̂n − θ∗ ) ∼ N (0, ∆)
(4)
However, there are different ways to estimate the covariance matrix, ∆.
To show the alternatives, we shall use the following definitions.
" n
#
X ∂ log(f (Yi , θ)) ∂ log(f (Yi , θ))
Jn (θ) =
(5)
∂θj
∂θk
i=1
j,k=1,...,q
" n
#
X ∂ 2 log(f (Yi , θ))
Hn (θ) =
∂θj ∂θk
i=1
j,k=1,...,q
1 Of course, there is no way to transcend the horizon defined by assuming that at
least one member of f (y, θ) comes close to g(y).
2 Actually some additional but very mild assumptions are required, see White [1982].
6.11.1
introduction
∂ log(f (Y, θ)) ∂ log(f (Y, θ))
J(θ) = E
∂θj
∂θk
j,k=1,...,q
2
∂ log(f (Y, θ))
H(θ) = E
∂θj ∂θk
j,k=1,...,q
3
(6)
These are symmetrical (q, q) matrices. Jn (θ)/n is the sample mean, and
J(θ) is the expectation (again taken with respect to g(y)) of the outer
product of the gradient of the log-likelihood; and Hn (θ)/n is the sample
mean, and H(θ) is the expectation of the Hessian of the log-likelihood.
Given these definitions, the covariance matrix of the limiting distribution (4) is:3
∆ = H(θ∗ )−1 J(θ∗ )H(θ∗ )−1
(7)
and, furthermore,
a.s.
n Hn (θ∗ )−1 Jn (θ∗ )Hn (θ∗ )−1 −→ ∆
(8)
This shows that the left-hand side of (8) which is based on sample means
provides a consistent estimator of the covariance matrix, ∆.
This holds true even if the model is not correctly specified. Therefore, the left-hand side of (8) is sometimes called a robust covariance
estimator. On the other hand, the classical ML approach can be shown
to be a special case. One needs the assumption that there is a θ◦ so that
g(y) = f (y, θ◦ ) and some additional regularity conditions. It follows,
then, that θ∗ = θ◦ and
−H(θ◦ ) = J(θ◦ )
and
∆ = −n H(θ◦ )−1 = n J(θ◦ )−1
(9)
Summing up, there are three different ways to approximate the covariance matrix of an ML estimator. In TDA, this can be selected with the
ccov parameter as follows:4
Jn (θ̂n )−1
if ccov = 1
(10)
−Hn (θ̂n )−1
if ccov = 2
(11)
Hn (θ̂n )−1 Jn (θ̂n )Hn (θ̂n )−1
if ccov = 3
(12)
3 If the log-likelihood has a maximum it can be assumed that, at least in a neighborhood of this maximum, the matrices defined in (6) are negative definite.
4 Note that the program calculates estimated covariance matrices for θ̂ − θ ∗ .
n
6.11.1
introduction
4
For some models, the ccov option is not yet implemented and TDA
then uses the default, that is, ccov=2. In any case, the ccov parameter
is independent of the numerical algorithm used to maximize the loglikelihood. Having found a (local) maximand, θ̂n , of the log-likelihood
with whatever method, TDA uses this estimate of the parameter vector
in an additional evaluation of the log-likelihood and its derivatives to
calculate the requested covariance estimator.
6.11.2
the fml command
1
6.11.2 The fml Command
TDA offers the fml command to find parameter estimates for a user-
defined log-likelihood. The syntax and all parameters are the same as in
the fmin command described in 5.6.1. The only difference is that the
default scaling factor is -1 for the fml command, while it is +1 for the
fmin command. The fml command needs a log-likelihood function on
its right-hand side. Its defintion must follow the syntax for functions
explained in 5.3.1.
Example 1 To illustrate, we use our standard example data set for
episode data and estimate a simple exponential model (see 6.17.2.1).
The command file, fml1.cf, is shown in Box 1. The nvar command
reads the episode data file rrdat.1 and defines variables. Then comes the
edef command to define an episode data structure. The rate command
estimates an exponential transition rate model using TDA’s standard ML
procedures. Then, the same model is estimated with the fml command.
Part of the standard output is shown in Box 2. The TDA example archive
contains a second example, fml2.cf, where the same model is estimated
by using numerical integration for the cumulated transition rate.
d061102.tex
April 21, 1998
6.11.2
the fml command
Box 1 Command file fml1.cf
nvar(
dfile = rrdat.1,
# data file
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
#
#
#
#
#
#
#
#
#
#
#
#
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
# define additional variables
COHO1
COHO2
COHO3
W
=
=
=
=
ge(TB,348) & le(TB,384),
ge(TB,468) & le(TB,504),
ge(TB,588) & le(TB,624),
SEX[2],
#
#
#
#
birth
birth
birth
women
cohort 1
cohort 2
cohort 3
= 1
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
);
edef(
ts = 0,
tf = DUR,
org = 0,
des = DES,
# define single episode data
#
#
#
#
starting time
ending time
origin state
destination state
);
rate(
xa(0,1) = COHO2,COHO3,W,
) = 2;
fml (
xp = -4,0,0,0,
# starting values
) = rate = exp(a0 + COHO2 * a1 + COHO3 * a2 + W * a3),
l1
= if(DES,log(rate),0),
fn
= l1 - rate * DUR;
2
6.11.2
the fml command
Box 2 Part of standard output from fml1.cf
ML estimation of user-defined model.
Function definition:
rate
= exp(a0+COHO2*a1+COHO3*a2+W*a3)
l1
= if(DES,log(rate),0)
fn
= l1-rate*DUR
Function will be summed over 600 data matrix cases.
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 4
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Got starting values from xp parameter.
Idx Parameter Starting value
1 a0
-4.0000
2 a1
0.0000
3 a2
0.0000
4 a3
0.0000
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -2475.44
Norm of final gradient vector: 1.10499e-11
Last absolute change of function value: 3.30667e-15
Last relative change in parameters: 5.84648e-07
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 a0
-5.0114
0.0843
-59.4446 1.0000
2 a1
0.5341
0.1120
4.7686 1.0000
3 a2
0.6738
0.1152
5.8472 1.0000
4 a3
0.5065
0.0942
5.3746 1.0000
Log likelihood (starting values): -2578.9484
Log likelihood (final estimates): -2475.4383
3
6.12
quantal response models
1
6.12 Quantal Response Models
This chapter deals with models for a dependent variable with two or
more discrete categories. The models can be used with cross-sectional
and pooled panel data.
6.12.1
The qreg Command
6.12.2
Binary Logit and Probit
6.12.3
Ordinal Logit and Probit
6.12.4
Multinomial Logit
6.12.5
Multivariate Probit
d0612.tex
April 21, 1998
6.12.1
the qreg command
1
6.12.1 The qreg Command
A single command, qreg, is used to estimate all quantal response models.
The syntax is shown in Box 1. The command expects a list of variables
on the right-hand side, other parameters are optional.
Cross-section data. For cross-section data, the variables on the righthand side of the command should be specified as
qreg (parameter) = Y, X1 , X2 , ...;
First comes the name of the dependent variable, then follow the names
of independent variables. If the list of independent variables is empty,
the model is estimated with a single parameter for an intercept.
Panel data. The qreg command can also be used with panel data based
on a horizontal data organization, see 3.2. Assuming w waves, variables
must be specified in the following order:
qreg (parameter) = Y1 , . . . , Yw ,
X11 , . . . , X1w ,
X21 , . . . , X2w ,
...,
Xm1 , . . . , Xmw ;
Using panel data requires to specify the number of waves with the nw parameter. By default, nw=1, that is, the command expects cross-sectional
data.
As a general convention, cases are considered valid if the dependent
variable has a non-negative value. This allows to use unbalanced panel
data. If variable Yt for wave t has a negative value for individual i, this
wave-specific data is omitted from the likelihood. For panel data, an
additional option is provided by the pmin parameter. If pmin=k, the likelihood only includes individuals who have valid data for at least k waves.
Default is pmin=1. The number of different categories in the dependent
variable is calculated from the set of non-zero values found in at least
one of these variables. These values are always taken as integers and then
sorted in ascending order. The smallest integer is taken as the reference
d061201.tex
December 18, 1999
6.12.1
the qreg command
2
Box 1 Syntax for qreg command
qreg (
m=...,
ni=1,
nq=...,
nw=...,
pmin=...,
tfmt=...,
ppar=...,
pcov=...,
res=...,
mfmt=...,
df=...,
dtda=...,
fmt=...,
opt=...,
nhp=...,
eps=...,
mplog=...,
mppar=...,
mpcov=...,
mpgrad=...,
model selection, def. 1
1 = binary logit
2 = binary probit
3 = ordinal logit
4 = ordinal probit
5 = multinomial logit
6 = multivariate probit
7 = conditional logit
8 = simultaneous binary probits
if without intercept, def. 0
number of categories if m=5
number of waves, def. 1
minimum number of valid wave data, def. 1
print format for results, def. 10.4
print estimated coefficients to output file
print covariance matrix to output file
additional output options
print format for pcov option, def. 12.4
print data to an output file
TDA description of output file
print format for df option, def. 10.4
model-specific options, def. 1
control of numerical integration, def. 6
accuracy of numerical integration, def. 10−6
write loglikelihood value into matrix
write estimated parameters into matrix
write covariance matrix into matrix
write gradients into matrix
) = varlist;
category. The maximum number of categories can be specified with the
maxcat parameter; default is maxcat=100.
Additional options. Model estimation is done with the maximum likelihood method. Most parameters discussed in 5.6.1 for the fmin command can also be used for the qreg command. This includes the selection
of a minimization algorithm and parameter constraints.
6.12.1
the qreg command
3
1. Results of model estimation are written into the standard output.
The print format can be controlled with the tfmt parameter, default is
tfmt=10.4. For a description of the output see the examples given in
subsequent sections.
2. The parameter
ppar = name of an output file,
can be used to write the estimated model coefficients (and standard
errors) into an output file. The print format is determined by tfmt.
3. By default, the covariance matrix is calculated from the matrix of
second derivatives of the log-likelihood function. One can use the ccov
parameter to require an alternative calculation procedure; see the discussion in 5.6.5 and 6.11.1.
4. The parameter
pcov = name of an output file,
can be used to write the estimated covariance matrix into an output file.
The print format can be controlled with the mfmt parameter, default is
mfmt=12.4.
5. The data used for model estimation can be written into an output
file with the df parameter. For most models, the output file will then
also include estimated probabilities for each individual case. Using the
dtda parameter provides a description of the output file corresponding
to the syntax of TDA. The print format can be controlled with the fmt
parameter.
6. Some of the additional parameters are model-specific and will be
explained in the corresponding sections.
6.12.2
binary logit and probit
1
6.12.2 Binary Logit and Probit
This section describes simple logit and probit models for cross-sectional
or pooled panel data. It is assumed that the dependent variable, Y , has
only two possible values, Y ∈ {0, 1}. The 0-1 coding is actually only used
to ease notation, any other non-negative integer values may be used.
TDA always sorts the categories of the dependent variable in ascending order, the sorted categories are then mapped to 0 and 1. Choosing
category 0 to be the reference category, the modeling approach is
Pr (Y = 1 | x) = F (x; β)
(1)
F is a suitably chosen function depending on a parameter vector β and
a (row) vector of covariates, x. For each possible realization of these
covariates, the model implies an estimate of the probability that the
dependent variable takes the value 1.
The function F can be specified in many different ways; each specification results in a different type of model. Most commonly, the specification is in terms of a cumulative distribution function; this guarantees
its values to be in the range of zero to one, required for an interpretation
as probabilities. If the distribution function has only a single parameter, the dependence on covariates is modeled by a single predictor, most
commonly assumed to be linear, i.e., xβ. Of course, in principle it would
also be possible to use more general distribution functions with two or
more parameters and then linking covariates differently to all of these
parameters.
Two choices of F are most popular. One is the distribution function
of the logistic distribution
F (y) =
exp(y)
1 + exp(y)
Choosing this distribution function leads to a logit model:
Pr (Y = 1 | x) =
exp(xβ)
1 + exp(xβ)
(2)
Another popular choice is the standard normal distribution function
Z x
1
1
Φ(x) =
φ(u) du
with φ(u) = √
exp(− u2 )
(3)
2
2
π
−∞
d061202.tex
April 21, 1998
6.12.2
2
binary logit and probit
1
0
-4
-3
-2
-1
0
1
2
3
4
Figure 1 Normal and logistic distribution functions.
Choosing the normal distribution function to specify (1) leads to a probit
model:
Pr (Y = 1 | x) = Φ(xβ)
(4)
Both distributions are very similar as can be seen in Figure 1.1 The solid
and dashed lines show the standard logistic and the standard normal
distribution functions, respectively. The difference is mainly due to a
different variance. The standard logistic distribution has variance π 2 /3.
A normal distribution function with this variance is plotted as a dotted
line in Figure 1.
However, in both models it is not possible to include explicitly a separate parameter for the variance. Taking Φ(xβ/σ) in (4), or exp(σxβ)/(1+
exp(σxβ)) in (2), shows that σ is not identified. In both, the logit and
probit model, the parameter vector β is identified only up to a multiplicative scalar.
Maximum Likelihood Estimation. Using yi to denote observed values of the dependent variables, the likelihood of a sample of observations
can be written
L=
N
Y
Pr (Yi = 1 | xi )yi Pr (Yi = 0 | xi )1−yi
(5)
i=1
Case weights will be used if defined with the cwt command.
1 The
plot was created with command file plot-bin.cf contained in the TDA example
archive.
6.12.2
binary logit and probit
3
Logit Model. Inserting the logit specification (2) and taking logarithms
gives the log-likelihood of the logit model:
N
X
1
exp(xi β)
` =
yi log
+ (1 − yi ) log
1 + exp(xi β)
1 + exp(xi β)
i=1
=
N
X
yi xi β − log {1 + exp(xi β)}
i=1
This log-likelihood is well behaved (globally concave) and can be maximized easily with the standard Newton algorithm. The first and second
derivatives with respect to the components of the parameter vector, β,
are
N X
∂`
exp(xi β)
xij
=
yi −
∂βj
1 + exp(xi β)
i=1
∂2`
∂βj ∂βk
=
N
X
i=1
− exp(xi β)
xij xik
(1 + exp(xi β))2
Here it is assumed that there are j = 1, . . . , m covariates, xij being the
jth covariate for individual i. If the model is specified to include an
intercept (default), xi1 is a constant one.
Probit Model. Inserting the probit specification (4) into (5) and taking
logarithms gives the log-likelihood for the probit model:
`=
N
X
yi log {Φ(xi β)} + (1 − yi ) log {1 − Φ(xi β)}
i=1
The first and second derivatives with respect to the model parameters
are
N X
∂`
φ(xi β)
φ(xi β)
=
yi
− (1 − yi )
xij
∂βj
Φ(xi β)
1 − Φ(xi β)
i=1
∂2`
∂βj ∂βk
N X
φ(xi β)
φ(xi β)
−xi β −
+
=
yi
Φ(xi β)
Φ(xi β)
i=1
φ(xi β)
φ(xi β)
(1 − yi )
xi β −
xij xik
1 − Φ(xi β)
1 − Φ(xi β)
6.12.2
binary logit and probit
4
Box 1 Data file qr1.dat
D
W R
--------1
9 0
1
1 1
2
10 0
2
2 1
3
6 0
3
4 1
4
5 0
4
5 1
5
4 0
5
8 1
6
2 0
6
8 1
7
10 1
As in (3), φ denotes the standard normal density function; xij denotes
the observed value of the jth covariate for the ith individual.
Implementation. Both, the binary logit and probit model can be estimated with the qreg command; the m=1 parameter selects the logit
model, the m=2 parameter selects the probit model.
Model estimation is done by maximum likelihood as described above.
The default algorithm is mina=5 (Newton I). It is possible to use the ccov
option to select a specific type of covariance matrix calculation; default
is ccov=2, that is, the covariance matrix is create from the Hessian of
the log-likelihood. The parameter vector, and accordingly the first and
second derivatives of the log-likelihood, is organized as
β1 . . . βm
with m the number of covariates. As default, the first parameter is added
automatically as an intercept. The ni=1 option can be used to estimate a
model without an intercept. Since the log-likelihood for the simple logit
and probit models is globally concave, starting values for its iterative
maximization are, in most cases, not critical. As a default, TDA sets all
starting values to zero.
Example 1 To illustrate estimation of logit and probit models, we use
a small example data set taken from the SAS User’s Guide (Version 6, vol.
II, p. 1340). The data file, qr1.dat, is shown in Box 1. Each row of the
6.12.2
binary logit and probit
5
Box 2 command file qr1.cf
nvar(
dfile = qr1.dat,
Dose = c1,
Weight = c2,
Response = c3,
Log10Dose = log(Dose) / log(10),
);
cwt = Weight;
use case weights
qreg = Response,Log10Dose;
qreg (m=2) = Response,Log10Dose;
logit model
probit model
# fml command to estimate same logit model
fml = xb = beta0 + Log10Dose * beta1,
fn = Weight * (Response * xb - log(1 + exp(xb)));
# fml command to estimate same probit model
fml = pxb = nd (beta0 + Log10Dose * beta1),
fn = Weight * (Response * log(pxb) +
(1 - Response) * log(1 - pxb));
data file corresponds to a group of individuals with the stimulus variable
recorded in Log10Dose, the response variable recorded in Response, and
the number of individuals given in Weight.
The command file qr1.cf, shown in Box 2, estimates a logit and a
probit model. First this is done with the qreg command, then with TDA’s
fml command that requires a user-defined log-likelihood (see 6.11.2).
Part of the standard output of the qreg commands is shown in Box 3
(logit model) and Box 4 (probit model).
Standardized Effect Coefficients. As described by Long [1987], to
ease interpretation of estimation results of binary (and multinomial) logit
models, it is helpful to calculate standardized effect coefficients, defined
by
standardized effect of jth covariate = exp(β̂j σ̂(Xj ))
σ̂(Xj ) is the standard deviation of covariate Xj . A calculation of these
standardized effects can be requested with the parameter
res = 1,
6.12.2
binary logit and probit
Box 3 Part of standard output from qr1.cf (logit model)
Model: binary logit.
Variables (cross-section)
------------------------Y
: Response
X1 : Log10Dose
Checking available data (pmin=1)
Number of cases with valid data: 13
Categories of dependent variable.
Maximum number of categories: 100
Index
0
1
(Weighted)
Category
0
1 Observations
--------------------------------------------Wave 1
N
36.00
38.00
74.00
Pct
48.65
51.35
Using case weights defined by: Weight
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Number of model parameters: 2
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -37.1107
Norm of final gradient vector: 1.5979e-10
Last absolute change of function value: 9.82354e-12
Last relative change in parameters: 7.41904e-06
Idx Cat Term
Variable
Coeff
Error
C/Error Signif
----------------------------------------------------------------1
1 I
Intercept
-3.2246
0.8861
-3.6393 0.9997
2
1 X
Log10Dose
5.9702
1.4492
4.1197 1.0000
Log likelihood (starting values):
Log likelihood (final estimates):
-51.2929
-37.1107
6
6.12.2
binary logit and probit
7
Box 4 Part of standard output from qr1.cf (probit model)
Model: binary probit.
Convergence reached in 5 iterations.
Number of function evaluations: 6 (6,6)
Maximum of log likelihood: -37.2804
Norm of final gradient vector: 3.44131e-07
Last absolute change of function value: 2.0696e-08
Last relative change in parameters: 0.00030733
Idx Cat Term
Variable
Coeff
Error
C/Error Signif
----------------------------------------------------------------1
1 I
Intercept
-1.8127
0.4493
-4.0341 0.9999
2
1 X
Log10Dose
3.4181
0.7455
4.5847 1.0000
Log likelihood (starting values):
Log likelihood (final estimates):
-51.2929
-37.2804
Box 5 Standardized coefficients (logit model)
Standardized coefficients.
Idx Cat Term
Variable
Coeff
Exp(C) Exp(C*SD)
Std.Dev.
-------------------------------------------------------------------1
1 I
Intercept
-3.2246
0.0398
1.0000
0.0000
2
1 X
Log10Dose
5.9702
391.5761
1.0000
0.0000
To illustrate, adding this command to command file qr1.cf creates the
table shown in Box 5. Standardized effects are only calculated for crosssectional input data (nw=1).
Estimated Probabilities. The df parameter can be used to request an
output file that contains the data used for model estimation and, in addition, the estimated probabilities. An illustration is given in Box 6. The
columns labelled PROB0 and PROB1 contain the estimated probabilities
for Y to take the values 0 and 1, respectively, conditional on the corresponding values of the covariates, calculated as F (xβ) and 1 − F (xβ),
respectively.
The column labelled PROB is the probability for the actually observed
value of the dependent variable. The data in Box 6 have been created
6.12.2
8
binary logit and probit
Box 6 Illustration of df and dtda parameters
Contents of file requested by dtda parameter
nvar(
dfile = df,
noc = 13,
CaseID
[6.0] = c1 ,
Wave
[2.0] = c2 ,
Response [10.4] = c3 ,
Log10Dose [10.4] = c4 ,
Weight
[10.4] = c5 ,
PROB
[10.4] = c6 ,
PROB0
[10.4] = c7 ,
PROB1
[10.4] = c8 ,
);
Contents of file requested by df parameter
1
2
3
4
5
6
7
8
9
10
11
12
13
1
1
1
1
1
1
1
1
1
1
1
1
1
0.0000
1.0000
0.0000
1.0000
0.0000
1.0000
0.0000
1.0000
0.0000
1.0000
0.0000
1.0000
1.0000
0.0000
0.0000
0.3010
0.3010
0.4771
0.4771
0.6021
0.6021
0.6990
0.6990
0.7782
0.7782
0.8451
9.0000
1.0000
10.0000
2.0000
6.0000
4.0000
5.0000
5.0000
4.0000
8.0000
2.0000
8.0000
10.0000
0.9618
0.0382
0.8065
0.1935
0.5929
0.4071
0.4086
0.5914
0.2792
0.7208
0.1945
0.8055
0.8607
0.9618
0.9618
0.8065
0.8065
0.5929
0.5929
0.4086
0.4086
0.2792
0.2792
0.1945
0.1945
0.1393
0.0382
0.0382
0.1935
0.1935
0.4071
0.4071
0.5914
0.5914
0.7208
0.7208
0.8055
0.8055
0.8607
by adding the parameters
df
= name of an output file,
dtda = name of an output file,
to the qreg command for a logit model (see command file qr1a.cf, not
shown).
6.12.2.1
grouped data
1
6.12.2.1 Grouped Data
Assume that the sample consists of m groups and the ith group has ni
members. Let yi denote the number of members in group i who have
Y = 1. The likelihood may then be written as
L=
m Y
ni
i=1
yi
piyi (1 − pi )ni −yi
(1)
where pi is the probability for Y = 1 in group i, see Collett [1991, p. 57].
If one not just observes ni and yi , but also group-specific vectors of
covariates, say xi , one might go on and model the dependence of pi on
xi . For example, using a logit specification, one can assume
pi =
exp(xi β)
1 + exp(xi β)
The model is easily estimated with the qreg command using either individual or grouped data. For example, with grouped data, one can use
a data structure like
Y
W
X
1
0
..
.
yi
ni − y i
..
.
xi
xi
..
.
to be repeated for each group. The model is estimated by defining case
weights with cwt = W and then using the command
qreg (parameter) = Y, X;
Given that the data are clustered into groups it may also make sense to
estimate a multilevel model, see, e.g., Collett [1991, p. 207]. One approach
is by including a random effect into the model specification. Using again
a logit model, one might write
pi =
exp(xi β + zi γ)
1 + exp(xi β + zi γ)
d06120201.tex
April 21, 1998
6.12.2.1
grouped data
2
Box 1 Data file qr5.dat
Y N S X
----------10 39 0 0
23 62 0 0
23 81 0 0
26 51 0 0
17 39 0 0
Y N S X
----------5 6 0 1
53 74 0 1
55 72 0 1
32 51 0 1
46 79 0 1
10 13 0 1
Y N S X
----------8 16 1 0
10 30 1 0
8 28 1 0
23 45 1 0
0 4 1 0
Y
N S X
-----------3 12 1 1
22 41 1 1
15 30 1 1
32 51 1 1
3
7 1 1
where zi is a random number drawn from some distribution and γ an
additional model parameter. In order to make the model estimable, one
needs an assumption about the distribution of the random effects. Assuming a density, say φ(z), one can then try a marginal likelihood approach:
L=
m Z
Y
i=1
∞
ni y i
pi (1 − pi )ni −yi φ(zi ) d@, zi
y
i
−∞
(2)
For a discussion of this model where it is assumed that φ is a standard
normal density, see Collett [1991, pp. 207–212].
Example 1 While it is not possible to use the qreg command for multilevel models one can try to set up the likelihood directly and then use
the fml command, see 6.11.2. To illustrate this approach we use an example from Collett [1991, p. 211]. The data are shown in Box 1. There
are 21 groups. N is the number of group members, and Y is the number
of individuals with Y = 1. S and X are covariates (four categories).
The command file is qr5.cf, shown in Box 2. The first fml command
uses the likelihood (1). Since the likelihood is well-behaved, there is no
need to specify starting values (as default, all are zero), and one can use
TDA’s default Newton algorithm for function minimization. First and
second derivatives are automatically calculated as discussed in 5.3.2.
Box 3 shows part of the standard output. As can be seen from this
output, convergence is reached in four iterations.
To estimate a random effects model is far more difficult since we then
need numerical integration and do not know much about the behavior of
the likelihood. The second fml command in command file qr5.cf uses a
direct translation of the likelihood shown in (2). Creating the marginal
likelihood is done with the intn operator that uses Hermite integration
6.12.2.1
grouped data
3
Box 2 Command file qr5.cf
nvar(
dfile = qr5.dat,
Y = c1,
N = c2,
S = c3,
X = c4,
SX = S * X,
);
# standard logit model
fml = xb = beta0 + S * beta1 + X * beta2 + SX * beta3,
ee = bc(N,Y) * (exp(xb)^Y) / ((1 + exp(xb))^N) ,
fn = log(ee);
# logit model with random effects
fml(
mina = 4,
xp = -0.5582,0.1459,1.3182,-0.7781,0.1, # starting values
) = xb = beta0 + S * beta1 + X * beta2 + SX * beta3,
ee = intn(7,bc(N,Y) * (exp(xb+t*gam)^Y)/((1 + exp(xb+t*gam))^N) ),
fn = log(ee);
to calculate
Z
∞
intn(p, expression(t)) ≈
expression(t) φ(t) d@, t
−∞
where φ is a standard normal density and p is the number of Hermite
integration points. For more information on this operator, see 5.4.2.
Since the likelihood is difficult to maximize, we use starting values
taken from the estimation results of the simple logit model. Also, we
use the BFGS algorithm (mina=4) that only requires first derivatives.1
Nevertheless, having reached convergence, standard errors are calculated
from the Hessian of second derivatives. Box 4 shows the estimation results. They are basically identical with the results found by Collett [1991,
p. 212] who used the EGRET program.
1 The performance of the minimization algorithm depends, in particular, on the number of integration points. It might be necessary to begin with a low number of integration points and then using the estimated parameters as starting values for higher
number of integration points.
6.12.2.1
grouped data
Box 3 Part of standard output from qr5.cf
ML estimation of user-defined model.
Function definition:
xb
= beta0+S*beta1+X*beta2+SX*beta3
ee
= bc(N,Y)*(exp(xb)^Y)/((1+exp(xb))^N)
fn
= log(ee)
Function evaluation: sum over 21 data matrix cases.
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Number of model parameters: 4
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Idx
1
2
3
4
Parameter
beta0
beta1
beta2
beta3
Starting value
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
Convergence reached in 4 iterations.
Number of function evaluations: 5 (5,5)
Maximum of log likelihood: -54.937
Norm of final gradient vector: 7.20504e-07
Last absolute change of function value: 2.61263e-08
Last relative change in parameters: 0.000299634
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 beta0
-0.5582
0.1260
-4.4292 1.0000
2 beta1
0.1459
0.2232
0.6539 0.4868
3 beta2
1.3182
0.1775
7.4277 1.0000
4 beta3
-0.7781
0.3064
-2.5392 0.9889
Log likelihood (starting values):
Log likelihood (final estimates):
-87.8318
-54.9370
4
6.12.2.1
grouped data
Box 4 Part of standard output from qr5.cf
ML estimation of user-defined model.
Function definition:
xb
= beta0+S*beta1+X*beta2+SX*beta3
ee
= intn(7,bc(N,Y)*(exp(xb+t*gam)^Y)/((1+exp(xb+t*gam))^N))
fn
= log(ee)
Function evaluation: sum over 21 data matrix cases.
Maximum likelihood estimation.
Algorithm 4: BFGS
Number of model parameters: 5
Type of covariance matrix: 2
Maximum number of iterations: 100
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Minimum of step size value: 1e-10
Scaling factor: -1
Got 5 starting value(s) from xp parameter.
Idx
1
2
3
4
5
Parameter
beta0
beta1
beta2
beta3
gam
Starting value
-5.58200000e-01
1.45900000e-01
1.31820000e+00
-7.78100000e-01
1.00000000e-01
Convergence reached in 16 iterations.
Number of function evaluations: 19 (19,1)
Maximum of log likelihood: -53.7562
Norm of final gradient vector: 6.18713e-07
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 beta0
-0.5481
0.1668
-3.2861 0.9990
2 beta1
0.0965
0.2785
0.3464 0.2710
3 beta2
1.3369
0.2369
5.6442 1.0000
4 beta3
-0.8104
0.3853
-2.1034 0.9646
5 gam
0.2368
0.1110
2.1335 0.9671
Log likelihood (starting values):
Log likelihood (final estimates):
-54.4375
-53.7562
5
6.12.3
ordinal logit and probit
1
6.12.3 Ordinal Logit and Probit
This section describes logit and probit models for a dependent variable
with ordered categories. (For an applied introduction with social science
examples see Ludwig-Mayerhofer [1990].) It is assumed that there are at
least three such categories:
q≥3
R1 < R2 < . . . < Rq
The dependent variable is
Y ∈ {R1 , R2 , . . . , Rq }
q≥3
(1)
with observed values denoted by yi ; xi is a corresponding row vector
containing, say, m covariates. There are two mathematically equivalent
methods to extend the simple binary logit and probit models discussed
in 6.12.2 to the case of an ordinal response variable. One approach is
to formulate the model in terms of cumulative probabilities (logits or
probits), the other approach uses the concept of a latent variable.
Cumulative Logits and Probits. The definition of cumulative probabilities (logits or probits) is given by
θj = Pr (Y > Rj )
j = 1, . . . , q
Of course, θq = 0. Defining θ0 = 1, it follows that
Pr (Y = Rj ) = θj−1 − θj
j = 1, . . . , q
Using these quantities, θj , a general probability model for an ordinal
dependent variable can be written
θj = F (αj + x β)
j = 1, . . . , q − 1
(2)
where F is a suitably chosen distribution function. The basic idea is to
assume a different intercept, αj , for each of the ordered categories of
the dependent variable. In this way, the model links the covariates to
the probability of reaching a given level of the dependent variable. Since
θq = 0, only q − 1 intercepts can be estimated, and there cannot be an
additional intercept in the vector of covariates, x.
d061203.tex
April 21, 1998
6.12.3
ordinal logit and probit
2
Choosing F as a logistic distribution function leads to the logit model
for an ordinal dependent variable:
θj =
exp (αj + x β)
1 + exp (αj + x β)
j = 1, . . . , q − 1
The corresponding probit model is
θj = Φ(αj + x β)
j = 1, . . . , q − 1
with Φ denoting the standard normal distribution function.
Formulation with a Latent Variable. Another way to derive probability models for an ordered dependent variable, discussed by McKelvey
and Zavoina [1975],1 is in terms of a continuous latent variable, say Y ∗ .
It is assumed that there are unknown threshold values
−∞ = µ0 < µ1 < . . . < µq−1 < µq = ∞
so that there is a correspondence
Y = Rj
⇔
µj−1 < Y ∗ ≤ µj
j = 1, . . . , q
This approach leads to an ordinary regression model for Y ∗ that can be
written
Y ∗ = xβ + (3)
However, the model can equally well be written as a probability model,
and then it is seen to be equivalent to the formulation given in (2).
θj = Pr (Y > Rj ) = Pr (Y ∗ > µj )
(4)
= Pr (− < x β − µj ) = F (x β − µj )
with F the distribution function of the disturbance term, , in (3).2 It
follows that
αj = −µj
j = 1, . . . , q − 1
1 The same approach has been used earlier to find a scaling procedure for ordinal
categorical data, see Snell [1964].
2 It is assumed that has zero mean and that F is symmetric.
6.12.3
ordinal logit and probit
3
This assumes that the covariate vector x does not contain an intercept.
Of course, other parametrizations are possible.3
Maximum Likelihood Estimation. The likelihood of a sample of
observations can be written as
L=
q
N Y
Y
Pr (Yi = Rj | xi )wij
i=1 j=1
with indicator variables wij defined by
wij
=
1
0
if Yi = Rj
otherwise
Using cumulative logits or probits, the log-likelihood is
`=
q
N X
X
wij log(θi,j−1 − θij )
i=1 j=1
To maximize this expression and to estimate the covariance matrix, one
needs first and second derivatives with respect to αu (u = 1, . . . , q − 1)
and with respect to the components of the parameter vector, β. They
are given below for the logit and probit case.
Ordinal Logit. To simplify notation, the following abbreviations are
used:
Pij = θi,j−1 − θij , Ci0 = Ciq = 0
Cij =
θij
1 + exp (αj + xi β)
Dij =
Cij
1 + exp (αj + xi β)
3 LIMDEP,
j = 1, . . . , q − 1
for instance, sets µ0 = 0 and requires an intercept in x. The correspondence is then given by α1 = estimate of the global intercept, and αj = α1 − µj−1 .
6.12.3
ordinal logit and probit
4
With these abbreviations and using the Kronecker symbol, δij , the first
and second derivatives are
∂`
∂αu
=
∂`
∂βu
=
∂2`
∂αu ∂αv
q
m X
X
wij
δu,j−1 Ci,j−1 − δuj Cij
Pij
wij
Ci,j−1 − Cij
xiu
Pij
i=1 j=1
q
m X
X
i=1 j=1
=
q
m X
X
i=1 j=1
wij
δu,j−1 δv,j−1 (Di,j−1 − Ci,j−1 θi,j−1 )
−
Pij
δu,j δvj (Dij − Cij θij )
−
Pij
∂2`
∂βu ∂βv
(δu,j−1 Ci,j−1 − δuj Cij )(δv,j−1 Ci,j−1 − δvj Cij )
Pij Pij
q
m
XX
(Di,j−1 − Ci,j−1 θi,j−1 )
=
wij xiu xiv
−
Pij
i=1 j=1
2 )
Ci,j−1 − Cij
(Dij − Cij θij )
−
Pij
Pij
∂2`
∂βu ∂αv
=
q
m X
X
i=1 j=1
wij xiu
δv,j−1 (Di,j−1 − Ci,j−1 θi,j−1 )
−
Pij
δvj (Dij − Cij θij ) (Ci,j−1 − Cij )(δv,j−1 Ci,j−1 − δvj Cij )
−
Pij
Pij Pij
6.12.3
5
ordinal logit and probit
Ordinal Probit. Using the abbreviations
Eij = αj + xi β,
and
Φij = Φ(αi + xi β)
the first and second derivatives can be written as
∂`
∂αu
=
∂`
∂βu
=
∂2`
∂αu ∂αv
=
∂2`
∂βu ∂βv
q
m X
X
wij
δu,j−1 Φi,j−1 − δuj Φij
Pij
wij
Φi,j−1 − Φij
xiu
Pij
i=1 j=1
q
m X
X
i=1 j=1
q
m X
X
wij
i=1 j=1
(δu,j−1 Φi,j−1 − δuj Φij )(δv,j−1 Φi,j−1 − δvj Φij )
Pij Pij
q
m X
X
Eij Φij − Ei,j−1 Φi,j−1
=
wij xiu xiv
−
Pij
i=1 j=1
∂2`
∂βu ∂αv
δuj δvj Eij Φi,j − δu,j−1 δv,j−1 Ei,j−1 Φi,j−1
−
Pij
=
q
m X
X
i=1 j=1
Φi,j−1 − Φij
Pij
wij xiu
2 )
δvj Eij Φij − δv,j−1 Ei,j−1 Φi,j−1
−
Pij
(Φi,j−1 − Φij )(δv,j−1 Φi,j−1 − δvj Φij )
Pij Pij
Implementation. Both models can be estimated with the qreg command. m=3 selects the ordinal logit, m=4 selects the ordinal probit model.
Default is cross-sectional data. If there are two or more panel waves, the
data are pooled over waves. The models are parameterized using cumulative logits and probits. Estimation is done by maximum likelihood as
described above.
Both models have m + q − 1 parameters where m is the number of
covariates and q is the number of categories of the dependent variable. It
6.12.3
ordinal logit and probit
6
Box 1 Data file qr2.dat
Y
X1
X2
-----------------2
1.163 1.000
2
3.453 0.000
2
3.724 0.000
0
1.090 0.000
3
3.819 0.000
1
2.838 1.000
3
1.516 1.000
3
1.417 1.000
0
2.436 1.000
2
3.300 1.000
Y
X1
X2
-----------------1
1.058 1.000
0
2.607 1.000
0
2.162 0.000
1
2.145 0.000
0
1.103 0.000
1
1.914 0.000
3
3.610 0.000
3
3.579 1.000
3
3.379 1.000
1
3.175 0.000
Box 2 Command file qr2.cf
nvar(
dfile = qr2.dat,
Y = c1,
X1 = c2,
X2 = c3,
);
qreg (m=3) = Y,X1,X2;
qreg (m=4) = Y,X1,X2;
ordinal logit
ordinal probit
fml(
# ordinal logit with fml command
xp = -2,-3,-4,0,0,
)
= xb
theta0
theta1
theta2
theta3
ll
fn
=
=
=
=
=
=
X1 * beta1 + X2 * beta2,
1,
eexp(alpha1 + xb),
eexp(alpha2 + xb),
eexp(alpha3 + xb),
if (eq(Y,0),theta0 - theta1,
if (eq(Y,1),theta1 - theta2,
if (eq(Y,2),theta2 - theta3, theta3))),
= log(ll);
is required that q ≥ 3. A separate intercept must not be specified. The
parameter vector is organized as follows:
α1 . . . αq−1 β1 . . . βm
6.12.3
ordinal logit and probit
Box 3 Part of standard output from qr2.cf (logit model)
Model: ordinal logit.
Variables (cross-section)
------------------------Y
: Y
X1 : X1
X2 : X2
Categories of dependent variable.
Maximum number of categories: 100
Index
0
1
2
3
(Weighted)
Category
0
1
2
3 Observations
----------------------------------------------------------------Wave 1
N
5.00
5.00
4.00
6.00
20.00
Pct
25.00
25.00
20.00
30.00
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Number of model parameters: 5
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Convergence reached in 5 iterations.
Number of function evaluations: 6 (6,6)
Maximum of log likelihood: -24.4642
Norm of final gradient vector: 5.20097e-09
Last absolute change of function value: 1.43246e-10
Last relative change in parameters: 1.81865e-05
Idx Cat Term
Variable
Coeff
Error
C/Error Signif
----------------------------------------------------------------1
- I 1
Alpha 1
-1.7466
1.3020
-1.3414 0.8202
2
- I 2
Alpha 2
-3.1531
1.4312
-2.2032 0.9724
3
- I 3
Alpha 3
-4.2253
1.5565
-2.7146 0.9934
4
- X
X1
1.0121
0.4637
2.1826 0.9709
5
- X
X2
1.3100
0.8949
1.4639 0.8568
Log likelihood (starting values):
Log likelihood (final estimates):
-27.5245
-24.4642
7
6.12.3
ordinal logit and probit
8
Box 4 Part of standard output from qr2.cf (probit model)
Model: ordinal probit.
Convergence reached in 5 iterations.
Number of function evaluations: 6 (6,6)
Maximum of log likelihood: -24.5741
Norm of final gradient vector: 5.01377e-10
Last absolute change of function value: 9.6685e-12
Last relative change in parameters: 3.42967e-06
Idx Cat Term
Variable
Coeff
Error
C/Error Signif
----------------------------------------------------------------1
- I 1
Alpha 1
-1.0868
0.8207
-1.3243 0.8146
2
- I 2
Alpha 2
-1.9059
0.8692
-2.1927 0.9717
3
- I 3
Alpha 3
-2.5413
0.9193
-2.7645 0.9943
4
- X
X1
0.6122
0.2801
2.1853 0.9711
5
- X
X2
0.7563
0.5225
1.4475 0.8522
Log likelihood (starting values):
Log likelihood (final estimates):
-29.5284
-24.5741
To provide starting values, they must be given in this order. Default
starting values are calculated as
(
β=0
and
αj = log
θ̃j
1 − θ̃j
)
where θ̃j is the weighted fraction of cases with Yi > Rj . In most cases,
this is sufficient to reach convergence.
Example 1 To illustrate estimation of ordered logit and probit models with TDA we replicate an example taken from the LIMDEP manual
(version 6, p. 532). The data file is qr2.dat, shown in Box 1. The dependent variables has four categories. The command file, qr2.cf (Box
2), first uses two qreg commands to estimate an ordinal logit and an ordinal probit model, respectively. Finally, there is an fml command that
illustrates how to estimate the ordinal logit model with a user-defined
log-likelihood. Output from the qreg commands is shown in Box 3 (logit
model) and 4 (probit model).
6.12.3
ordinal logit and probit
9
The df Option. Box 5 illustrates the df and dtda parameters using the
data from Example 1 with a logit specification. The output file created
with the df parameter contains the data used for model estimation and,
in addition, estimated cumulative probabilities calculated according to
formula (4). The column labelled PROB contains the value for the actually
observed category.
6.12.3
10
ordinal logit and probit
Box 5 Illustration of df and dtda option
nvar(
dfile = df,
noc = 20,
CaseID
[6.0] = c1 ,
Wave
[2.0] = c2 ,
Y
[10.4] = c3 ,
X1
[10.4] = c4 ,
X2
[10.4] = c5 ,
PROB
[10.4] = c6 ,
PROB0
[10.4] = c7 ,
PROB1
[10.4] = c8 ,
PROB2
[10.4] = c9 ,
PROB3
[10.4] = c10,
);
result of dtda parameter
result of df paramter
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2.0000
2.0000
2.0000
0.0000
3.0000
1.0000
3.0000
3.0000
0.0000
2.0000
1.0000
0.0000
0.0000
1.0000
0.0000
1.0000
3.0000
3.0000
3.0000
1.0000
1.1630
3.4530
3.7240
1.0900
3.8190
2.8380
1.5160
1.4170
2.4360
3.3000
1.0580
2.6070
2.1620
2.1450
1.1030
1.9140
3.6100
3.5790
3.3790
3.1750
1.0000
0.0000
0.0000
0.0000
0.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
0.0000
0.0000
0.0000
0.0000
0.0000
1.0000
1.0000
0.0000
0.1495
0.3251
0.3879
0.3445
0.0000
0.7368
0.0000
0.0000
0.8838
0.6046
0.3160
0.9004
0.6086
0.2725
0.3475
0.2286
0.0000
0.0000
0.0000
0.5151
0.6771
0.8517
0.8831
0.3445
0.8927
0.9195
0.7498
0.7306
0.8838
0.9480
0.6535
0.9004
0.6086
0.6045
0.3475
0.5475
0.8707
0.9603
0.9518
0.8126
0.3394
0.5846
0.6493
0.1141
0.6709
0.7368
0.4234
0.3992
0.6508
0.8171
0.3160
0.6890
0.2759
0.2725
0.1154
0.2286
0.6226
0.8556
0.8287
0.5151
0.1495
0.3251
0.3879
0.0422
0.4109
0.4893
0.2009
0.1853
0.3894
0.6046
0.1365
0.4313
0.1154
0.1136
0.0427
0.0921
0.3609
0.6698
0.6235
0.2666
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
6.12.4
1
multinomial logit
6.12.4 Multinomial Logit
We now consider a multinomial logit model for a depending variable with
two or more unordered categories:
Y ∈ {R1 , R2 , . . . , Rq }
q≥2
Given a sample of i = 1, . . . , n individuals, observed values of Y will be
denoted by yi .
Although not essential, to describe the modeling approach we use the
parlance of discrete choice models. It is assumed, then, that with each
of the q alternatives (R1 , . . . , Rq ) is associated a value of attractiveness,
or utility, depending on characteristics of the alternative and on characteristics of the individual who perceives the alternatives. This may be
written
uj = u(x, zj ) + j
(1)
uj is the attractiveness of the jth alternative assumed to depend on
covariates x and zj . The (row) vector x comprises covariates not varying
with the alternatives; the (row) vector zj comprises covariates specific
for the jth alternative. j is a random variable to account for stochastic
influences.
The hypothesis of random utility maximization assumes that an individual chooses the jth alternative if its attractiveness is greater than the
attractiveness of all other alternatives, that is if uj > uk for all k 6= j.1
Consequently, the condition for the jth alternative to be chosen is
u(x, zj ) + j > u(x, zk ) + k
for all
k 6= j
This condition can be used to express the probability distribution of the
dependent variable as follows:
Pj (x, zj ) = Pr (Y = Rj | x, zj )
(2)
= Pr ( ∀k 6= j : u(x, zj ) + j > u(x, zk ) + k )
1 The possibility that two or more alternatives have the same attractiveness is excluded. Through introduction of the random term in (1) it has a zero probability
in the model. However, this may not be appropriate for all situations; for instance,
if something is picked from a supermarket shelf. See Ullmann-Margalit and Morgenbesser [1977] for an interesting distinction between “picking” and “choosing”.
d061204.tex
April 21, 1998
6.12.4
multinomial logit
2
These probabilities are defined as soon as the joint probability distribution for the random variables j is specified. Let f (1 , . . . , q ) be their
joint density function. Then
Z
b1
Pr (1 ≤ b1 , . . . , q ≤ bq ) =
Z
bq
···
−∞
f (1 , . . . , q ) dq . . . d1
−∞
The probabilities Pj (x, zj ) can be expressed in the same way by only
changing the limits of integration according to the condition stated in
(2). So one gets
Z
b1
Z
−∞
bk =
bq
···
Pj (x, zj ) =
f (1 , . . . , q ) dq . . . d1
(3)
−∞
u(x, zj ) − u(x, zk ) + j
∞
if k 6= j
if k = j
Based on this general choice model, a variety of different specifications
is possible. Two things must be specified. First, one has to choose a
distribution for the random vector = (1 , . . . , q ). A standard choice
is an extreme value distribution leading to a multinomial logit model.
This is further discussed in the next section. Another choice would be
the multivariate normal distribution leading to a probit model.
Secondly, one has to specify the dependence of the alternative’s attractiveness on the covariates. For simplicity, all models discussed in this
chapter assume a linear dependence, that is
u(x, zj ) = x βj + zj γ
(4)
Note that there is a separate parameter vector, βj , for each category of
the dependent variable, but only a single parameter vector, γ, for the
alternative-specific covariates.
Logit Specification. The model is derived from the assumption that
the components of the random vector = (1 , . . . , q ) are independent
and that each component follows a (type I) extreme value distribution.
The density and distribution functions are, respectively
fe (x) = exp(− exp(−x)) exp(−x)
Fe (x) = exp(− exp(−x))
(5)
6.12.4
3
multinomial logit
The general model formulation given in (3) can be rewritten, then, as
Z ∞
Y
Pj (x, zj ) =
fe (j )
Fe (bk ) dj
−∞
k6=j
Using the abbreviation ũk = u(x, zk ) and the definition of the extreme
value distribution in (5), gives
Pj (x, zj ) =
Z ∞
Y
exp {− exp(−j )} exp(−j )
exp {− exp(ũk − ũj − j )} dj =
−∞
Z
k6=j
∞
exp(−j )
−∞
Z
q
Y
exp {− exp(ũk − ũj ) exp(−j )} dj =
k=1
∞
exp(−j ) exp {− exp(−uj ) C} dj
−∞
with
C=
q
X
exp(ũk − ũj )
k=1
The integral can be solved analytically. With C 0 = log(C), one finds that
Z ∞
Pj (x, zj ) =
exp {−j − exp(−j + C 0 )} dj
−∞
1
=
C
Z
∞
exp {−j + C 0 − exp(−j + C 0 )} dj
−∞
and, since by definition of the extreme value distribution function the
last integral evaluates to unity, one gets
exp(ũj )
k=1 exp(ũk )
Pj (x, zj ) = Pq
Finally, assuming a linear dependence on covariates as defined in (4),
the model becomes
exp(x βj + zj γ)
k=1 exp(x βk + zk γ)
Pr (Y = Rj |, x, zj ) = Pq
(6)
6.12.4
4
multinomial logit
Since not all of the parameter vectors βj are estimable we add the restriction that β1 = 0.
One should note that in the general model formulation given in (6)
there is a different parameter vector, βj , for each category of the dependent variable, but only a single parameter vector, γ, associated with the
alternative-specific covariates. Therefore, all vectors zj , for j = 1, . . . , q,
must have the same length, and this in turn is the length of the parameter vector γ. Of course, the model can be specified without any
alternative-specific covariates leading to a standard multinomial logit
model.
Maximum Likelihood Estimation. For a sample of i = 1, . . . , n observations, the likelihood of model (6) is
L=
q n Y
Y
i=1 j=1
exp(xi βj + zij γ)
Pq
k=1 exp(xi βk + zik γ)
wij
with indicator variables wij defined by
wij =
1
0
if yi = Rj
otherwise
Since β1 = 0, the expression can be simplified by defining
z̃ij = zij − zi1
The log-likelihood can then be written as
`=
q
n X
X
wij log
i=1 j=1
exp(xi βj + z̃ij γ)
Pq
1 + k=2 exp(xi βk + z̃ik γ)
Using the abbreviations
eij = exp(xi βj + z̃ij γ)
and Ei =
q
X
k=2
exp(xi βk + z̃ik γ)
6.12.4
5
multinomial logit
the derivatives are
∂`
∂βr,s
∂`
∂γs
=
q
n X
X
wij
δrj −
wij
Pq
k=2 eik z̃ik,s
z̃ij,s −
1 + Ei
i=1 j=1
=
q
n X
X
i=1 j=1
∂2`
∂βr1 ,s1 ∂βr2 ,s2
=
∂2`
∂γs1 ∂γs2
=
q
n X
X
i=1 j=1
q
n X
X
eir1
wij
1 + Ei
Pq
wij
eir
1 + Ei
k=2
i=1 j=1
xi,s
eir2
− δ r1 r2
1 + Ei
k=2
=
q
n X
X
i=1 j=1
wij
eir
1 + Ei
xi,s1 xi,s2
Pq
eik z̃ik,s1 k=2 eik z̃ik,s2
−
(1 + Ei )2
Pq
∂2`
∂βr,s ∂γs1
Pq
eik z̃ik,s1 z̃ik,s2
1 + Ei
eik z̃ik,s1
− z̃ir,s1
1 + Ei
k=2
xi,s
Implementation. The model can be estimated with the qreg command, using the model selection parameter m=5. In addition, one must
use the nq parameter to specify the number of categories in the dependent variable.
The specification of covariates must follow the convention explained
at the beginning of this chapter. A special convention is used for alternative-specific covariates. Each z-variable must be specified as a set of
q variable names, q being the number of categories in the dependent
variable. Each of these sets of variable name must be enclosed in brackets
(see Example 2).
The number of model parameters is m (q − 1) + m0 . q is the number
of categories of the dependent variable, m is the number of individualspecific covariates, and m0 is the number of z-variables. The parameter vector (and accordingly the first and second derivatives of the loglikelihood) is organized in the following way:
β21 . . . β2m β31 . . . β3m βq1 . . . βqm γ1 . . . γq
6.12.4
multinomial logit
6
Box 1 Command file qr3.cf
nvar(
dfile = qr3.dat,
X
= c1,
Weight = c2,
Y
= c3,
);
cwt = Weight;
use case weights
qreg(
m = 5,
# select multinomial logit
nq = 5,
# categories of dep variable
)= Y,X;
# same model with fml command
fml = xb2
xb3
xb4
xb5
xbb
ll
fn
=
=
=
=
=
=
exp(beta20 + X * beta21),
exp(beta30 + X * beta31),
exp(beta40 + X * beta41),
exp(beta50 + X * beta51),
1 + xb2 + xb3 + xb4 + xb5,
if (eq(Y,1),1 / xbb,
if (eq(Y,2),xb2 / xbb,
if (eq(Y,3),xb3 / xbb,
if (eq(Y,4),xb4 / xbb, xb5 / xbb)))),
= Weight * log(ll);
βlj is the parameter associated with the lth category and jth covariate (l = 2, . . . , q, j = 1, . . . , m). The first (lowest) category, R1 , is the
reference. γk is the parameter associated with the kth set of alternativespecific covariates (k = 1, . . . , m0 ).
The model is estimated by maximum likelihood as described above.
All of TDA’s maximization algorithms can be used. Default starting values are currently zero for all parameters. This may result in convergence problems. One should start, then, with simple models, successively
adding covariates and using the results of simple models as starting values for more complex models.
Example 1 To illustrate estimation of multinomial logit models with
TDA, our first example uses data taken from Dixon ([1990], p. 1048). The
data file is qr3.dat (not shown, but contained in the example archive).
The command, qr3.cf, is shown in Box 1. It contains the qreg command to estimate a multinomial logit model, and also a fml command
that directly uses the corresponding log-likelihood. Part of the standard
6.12.4
multinomial logit
Box 2 Part of standard output from qr3.cf
Model: multinomial logit.
Variables (cross-section)
------------------------Y
: Y
X1 : X
Categories of dependent variable.
Maximum number of categories: 100
Index
0
1
2
3
4
(Weighted)
Category
1
2
3
4
5 Observations
----------------------------------------------------------------Wave 1
N
70.00 149.00 175.00 132.00
55.00
581.00
Pct 12.05
25.65
30.12
22.72
9.47
Using case weights defined by: Weight
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Convergence reached in 7 iterations.
Number of function evaluations: 8 (8,8)
Maximum of log likelihood: -826.748
Norm of final gradient vector: 6.22304e-12
Last absolute change of function value: 8.25065e-16
Last relative change in parameters: 2.47957e-07
Idx Cat Term
Variable
Coeff
Error
C/Error Signif
----------------------------------------------------------------1
2 I
Intercept
7.2428
1.3578
5.3342 1.0000
2
2 X
X
-0.5035
0.1024
-4.9187 1.0000
----------------------------------------------------------------3
3 I
Intercept
8.3752
1.3438
6.2326 1.0000
4
3 X
X
-0.5842
0.1014
-5.7582 1.0000
----------------------------------------------------------------5
4 I
Intercept
11.3662
1.3897
8.1788 1.0000
6
4 X
X
-0.8723
0.1073
-8.1304 1.0000
----------------------------------------------------------------7
5 I
Intercept
11.1873
1.5042
7.4373 1.0000
8
5 X
X
-0.9375
0.1202
-7.7979 1.0000
Log likelihood (starting values):
Log likelihood (final estimates):
-935.0834
-826.7484
7
6.12.4
multinomial logit
8
Box 3 Command file qr4.cf
nvar(
dfile = qr4.dat,
Z1
= c1,
Z2
= c2,
Z3
= c3,
Y
= c4,
);
qreg(
m = 5,
nq = 3,
ni = 1,
) = Y,(Z1,Z2,Z3);
# selection of logit model
# number of categories
# without an intercept
output from the qreg command is shown in Box 2.
In this example, the number of categories in the dependent variable is
q = 5 and there are no alternative-specific covariates. As indicated in the
column labeled Cat, the output contains a separate block of estimated
coefficients for each category, j = 2, . . . , q.
Example 2 To illustrate the estimation of a multinomial logit models
with alternative-specific covariates, we use example data from Daganzo
[1979, p. 61].2 The data are concerned with a choice between three different means of transportation. The data file, qr4.dat, contains four
variables. Y is the dependent variable, with three categories, reflecting
the actual choice. Variables Z1, Z2, and Z3, contain the expected transportation times associated with the three alternatives. The command file
is shown in Box 3, part of its standard output in Box 4. In this example, there is only a single alternative-specific covariate and the model is
estimated without an intercept. Since the dependent variable has three
categories, the alternative-specific covariate must be specified by using
three variable names, enclosed in brackets.
2 The same data have been used for an illustration of KALOS, a program to estimate
general multinomial logit models written by Röding, Küsters, and Arminger [1985].
However, there are minor differences in the data files. The TDA example archive
contains the original Daganzo data as qr4.dat and the slightly different KALOS
data set as qr4a.dat.
6.12.4
multinomial logit
Box 4 Part of standard output from qr4.cf
Model: multinomial logit.
Variables (cross-section)
------------------------Y
: Y
Z1
1 : Z1
2 : Z2
3 : Z3
Model without intercept.
Checking available data (pmin=1)
Number of cases with valid data: 50
Categories of dependent variable.
Maximum number of categories: 100
Index
0
1
2
(Weighted)
Category
1
2
3 Observations
------------------------------------------------------Wave 1
N
14.00
29.00
7.00
50.00
Pct
28.00
58.00
14.00
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Number of model parameters: 1
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -33.3581
Norm of final gradient vector: 8.09432e-07
Last absolute change of function value: 4.59832e-09
Last relative change in parameters: 0.000120481
Idx Cat Term
Variable
Coeff
Error
C/Error Signif
-----------------------------------------------------------------1
- Z1
Z1
-0.3568
0.0776
-4.5966 1.0000
Log likelihood (starting values):
Log likelihood (final estimates):
-54.9306
-33.3581
9
6.12.4
10
multinomial logit
Box 5 Illustration of df and dtda parameters
nvar(
dfile = df,
noc = 50,
CaseID
[6.0] = c1 ,
Wave
[2.0] = c2 ,
Y
[10.4] = c3 ,
Z1
[10.4] = c4 ,
Z2
[10.4] = c5 ,
Z3
[10.4] = c6 ,
PROB
[10.4] = c7 ,
PROB0
[10.4] = c8 ,
PROB1
[10.4] = c9 ,
PROB2
[10.4] = c10,
);
created by dtda parameter
first records of output file created with df parameter
1
2
3
4
5
6
7
8
1
1
1
1
1
1
1
1
2.0000
2.0000
2.0000
2.0000
2.0000
1.0000
1.0000
1.0000
16.4810
15.1230
19.4690
18.8470
12.5780
11.5130
10.6510
8.3590
16.1960
11.3730
8.8220
15.6490
10.6710
20.5820
15.5370
15.6750
23.8900
14.1820
20.8190
21.2800
18.3350
27.8380
17.4180
21.0500
0.5083
0.6137
0.9651
0.6880
0.6364
0.9595
0.7910
0.9223
0.4591
0.1610
0.0216
0.2198
0.3223
0.9595
0.7910
0.9223
0.5083
0.6137
0.9651
0.6880
0.6364
0.0377
0.1383
0.0678
0.0326
0.2252
0.0133
0.0922
0.0413
0.0028
0.0707
0.0100
The df Option. Box 5 illustrates the df and dtda parameters using
the data from Example 2. The output file created with the df parameter
contains the data used for model estimation and, in addition, estimated
probabilities calculated according to formula (6). The column labelled
PROB contains the value for the actually observed category.
6.12.5
1
multivariate probit
6.12.5 Multivariate Probit
We now consider a probit specification of the general model that was
decribed in 6.12.4. For an extensive discussion see Daganzo [1979]. Y ∈
{R1 , . . . , Rq } is the dependent variable. The attractiveness of category
Rj is
uj = xβj + zj γ
depending on covariate vectors x and zj . The probability for alternative
Rj may then be written as
Pj (x, zj ) = Pr (Y = Rj | x, zj )
(1)
= Pr ( ∀k 6= j : uj + j > uk + k )
= Pr ( ∀k 6= j : k − j < uj − uk )
The assumption is that
= (1 , . . . , q ) ∼ N (0, Σq )
meaning that follows a q-dimensional normal distribution with zero
mean and covariance matrix Σq . One dimension can be saved by using
(j)
(j)
k := k − j , uk := uj − uk
We may then write
(j)
(j)
Pj (x, zj ) = Pr ( ∀k 6= j : k < uk )
(2)
(j)
(j)
Let (j) denote the (q − 1)-vector consisting of 1 , . . . , q
There is then a simple linear transformation
for k 6= j.
(j) = T (j) @, where the (q − 1, q)-matrix T (j) is defined by

−1 if t0 = j



1 if t0 < j and t0 = t
(j)
Ttt0 =
1 if t0 > j and t0 = t + 1



0 otherwise
d061205.tex
April 21, 1998
6.12.5
2
multivariate probit
Then, see, e.g., Tong [1990, p. 26], (j) follows a (q − 1)-dimensional
normal distribution:
0
(j) ∼ N 0, T (j) Σq T (j) @,
(j)
(j)
Finally, let u(j) denote the (q − 1)-vector consisting of u1 , . . . , uq for
(j)
k 6= j and Φq−1 the distribution function of a (q −1)-dimensional normal
distribution with zero mean and covariance matrix
(j)
Σq−1 := T (j) Σq T (j)
0
we can simply write
(j)
Pj (x, zj ) = Φq−1 (u(j) )
(3)
Maximum Likelihood Estimation. Now assume a sample of observations, (yi , xi , zij ), for i = 1, . . . , n, where the covariate vector x has
m components. Setting β1 = 0, the model parameters are β2 , . . . , βm , γ.
Using (3), the log-likelihood can be written as
`(β2 , . . . , βm , γ) =
n
X
(yi )
log Φq−1
(u(yi ) )
(4)
i=1
Implementation. The multivariate probit model can be estimated with
the qreg command, the model selection parameter is m=6. The specification of variables on the right-hand side follows the same convention as
was explained for the multinomial logit model in 6.12.4. If there are two
or more panel waves, the data are pooled over waves. The covariance
matrix
Σq = σij
is assumed to be a correlation matrix (σii = 1), and all elements in the
lower triangle are possible model parameters. The parameter vector is
organized as
β21 . . . β2m . . . βq1 . . . βqm γ1 . . . γq σ21 σ31 σ32 . . . σq1 . . . σq,q−1
Of course, most often the model will not be estimable without further
constraints on the components of the correlation matrix. This can be
done with the con parameter for parameter constraints, see 5.6.6.
6.12.5
3
multivariate probit
Optional Parameterizations. The off-diagonal elements of the correlation matrix must satisfy −1 < σij < 1. TDA’s standard function
minimization routines cannot account for this constraint, and so it can
easily happen that the estimation procedure cannot continue because of
a violation of this constraint. For this reason, TDA offers three different
∗
parameterizations that can be selected with the opt parameter. Let σij
denote the components in the parameter vector corresponding to the
lower triangle of the correlation matrix.
1. If opt = 1 (default), the correlation matrix for model estimation is
∗
created by setting σij = σij
, without any further reparameterization.
2. If opt = 2, the transformation
σij = 2
∗
exp(σij
)
∗) −1
1 + exp(σij
is applied before creating the covariance from the σij parameters.
∗
3. If opt = 3, it is assumed that the parameters σij
are the elements of
a Cholesky decomposition of Σq (see, e.g., Tong [1990, p. 184]):

Σq = CC 0
where



C=



1
∗
σ21
∗
σ31
..
.
1
∗
σ32
..
.
1
..
.
∗
σq1
∗
σq2
∗
σq,q−1






···
1
In any case, the output file that can be requested with the ppar parameter will contain the estimated correlation matrix, Σq . It is easy, therefore,
to re-estimate the model with opt=1, given that one has achieved results
with opt=2 or opt=3.
Numerical Integration. The main difficulty is the evaluation of the
(j)
distribution function, Φq−1 , of the (q − 1)-dimensional normal distribution. Different approaches have been proposed for the evaluation of the
corresponding (q − 1)-dimensional integral. For the current implementation, TDA uses a numerical integration procedure developed by Drezner
[1992], based on a Gauss quadrature method. The procedure is computationally very intensive, and so it might not be possible to estimate
models with a large number of categories in the dependent variable. The
6.12.5
multivariate probit
4
Box 1 Command file qr6.cf
nvar(
dfile = qr3.dat,
X
= c1,
Weight = c2,
Y
= c3,
);
cwt = Weight;
use case weights
qreg(
m = 6,
# select multivariate probit
nq = 5,
# categories of dep variable
ppar = par,
# write estimated parameters and corr. matrix
con = b9 = 0,
# use identity correlation matrix
con = b10 = 0,
con = b11 = 0,
con = b12 = 0,
con = b13 = 0,
con = b14 = 0,
con = b15 = 0,
con = b16 = 0,
con = b17 = 0,
con = b18 = 0,
)= Y,X;
advantage is, however, that the numerical accuracy of results can be
controlled. There are two parameters, nhp and eps.
1. The nhp = k parameter can be used to specify the number of integration points, k, in the Gauss quadrature procedure. If 2 ≤ k ≤ 10,
the algorithm performs one Gauss quadrature with k points for each
dimension of the integral. Default is nhp = 6.
2. If 12 ≤ k ≤ 20 the algorithm performs Gauss quadratures with k − 10
points progressively until the accuracy specified with the eps parameter
is reached. Default is eps = 10−6 . Note that the value of this parameter
is only used when 12 ≤ k ≤ 20.
The algorithm may not be able to achieve the required accuracy. This will
not be regarded as an error, but the standard output will then contain a
warning message. Note that the same algorithm is also used for the mvn
operator, see 5.2.5.7. This operator can be used to become familiar with
the integration procedure and to evaulate the distribution function for
arbitrary input values.
6.12.5
multivariate probit
Box 2 Part of standard output from qr6.cf
Maximum likelihood estimation.
Algorithm 8: CES (tensor model [0])
Number of model parameters: 18
Type of covariance matrix: 2
Maximum number of iterations: 200
Convergence criterion: 6
Tolerance for scaled gradient: 1e-05
Tolerance for scaled parameter change: 1e-08
Control of integration (nhp): 6
Type of parameterization: 1
Protocol will be written to: p
Convergence reached in 5 iterations.
Number of function evaluations: 321 (0,0)
Maximum of log likelihood: -828.232
Norm of final gradient vector: 0.0031299
Final scaled gradient: 2.71293e-06
Final scaled parameter change: 0.00511032
Idx Cat Term
Variable
Coeff
Error
C/Error Signif
-----------------------------------------------------------------1
2 I
Intercept
2.8003
0.5530
5.0641 1.0000
2
2 X
X
-0.1915
0.0431
-4.4467 1.0000
-----------------------------------------------------------------3
3 I
Intercept
3.3703
0.5456
6.1771 1.0000
4
3 X
X
-0.2320
0.0426
-5.4507 1.0000
-----------------------------------------------------------------5
4 I
Intercept
4.8489
0.5610
8.6428 1.0000
6
4 X
X
-0.3740
0.0445
-8.3974 1.0000
-----------------------------------------------------------------7
5 I
Intercept
4.5686
0.6055
7.5450 1.0000
8
5 X
X
-0.3867
0.0493
-7.8409 1.0000
-----------------------------------------------------------------9
- S
Sigma 2, 1
0.0000
0.0000
----10
- S
Sigma 3, 1
0.0000
0.0000
----11
- S
Sigma 3, 2
0.0000
0.0000
----12
- S
Sigma 4, 1
0.0000
0.0000
----13
- S
Sigma 4, 2
0.0000
0.0000
----14
- S
Sigma 4, 3
0.0000
0.0000
----15
- S
Sigma 5, 1
0.0000
0.0000
----16
- S
Sigma 5, 2
0.0000
0.0000
----17
- S
Sigma 5, 3
0.0000
0.0000
----18
- S
Sigma 5, 4
0.0000
0.0000
----Log likelihood (starting values):
Log likelihood (final estimates):
-935.1167
-828.2320
5
6.12.5
multivariate probit
6
Box 3 Command file qr7.cf
nvar(
dfile = qr4.dat,
Z1
= c1,
Z2
= c2,
Z3
= c3,
Y
= c4,
);
qreg(
m = 6,
nq = 3,
ni = 1,
df = df,
dtda = t,
xp = -0.1716,0,0,0,
con = b3 = 0,
con = b4 = 0,
ppar = par,
df = df,
dtda = t,
) = Y,(Z1,Z2,Z3);
#
#
#
#
#
#
#
#
model selection
number of categories
without intercept
print data and estimated probabilities
description file
starting values taken from model with
zero correlations
keep only sigma(2,1)
# print parameter and corr. matrix
# write data and estimated probabilities
Available Minimization Algorithms. While it would be possible to
use the same integration procedure also for an evaluation of derivatives
of the log-likelihood, this has not been implemented in TDA. Only function values, no derivataives of the likelihood function are available. As a
consequence, the multivariate probit model can only be estimated with
one of the algorithms, mina=7 or mina=8 (see 5.6.2). Default is mina=8.
Both algorithms numerically approximate first and second derivatives. A
numerical approximation to the Hessian of the log-likelihood is also used
to estimate a covariance matrix for the model parameters (consequently,
always ccov = 2).
Starting Values. As default, all parameters have a zero initial value.
This will often not lead to convergence. So one should specify suitable
starting values with the xp or dsv parameter. Note that an output file
written with the ppar parameter can be used as an input file with the
dsv parameter. Of course, file names should be different when using both
parameters simultaneously.
Example 1 As a first illustration we replicate multinomial logit model
from the first example in 6.12.4, now with a multivariate probit speci-
6.12.5
multivariate probit
Box 4 Part of standard output from qr7.cf
Maximum likelihood estimation.
Algorithm 8: CES (tensor model [0])
Number of model parameters: 4
Type of covariance matrix: 2
Maximum number of iterations: 200
Convergence criterion: 6
Tolerance for scaled gradient: 1e-05
Tolerance for scaled parameter change: 1e-08
Scaling factor: -1
Control of integration (nhp): 6
Type of parameterization: 1
Got 4 starting value(s) from xp parameter.
Number of constraints: 2
Convergence reached in 4 iterations.
Number of function evaluations: 40 (0,0)
Maximum of log likelihood: -34.4166
Norm of final gradient vector: 2.21297e-06
Final scaled gradient: 5.87576e-08
Final scaled parameter change: 0.000629914
Idx Cat Term
Variable
Coeff
Error
C/Error Signif
-----------------------------------------------------------------1
- Z1
Z1
-0.1787
0.0349
-5.1178 1.0000
-----------------------------------------------------------------2
- S
Sigma 2, 1
0.4134
0.6320
0.6542 0.4870
3
- S
Sigma 3, 1
0.0000
0.0000
----4
- S
Sigma 3, 2
0.0000
0.0000
----Log likelihood (starting values):
Log likelihood (final estimates):
-34.5825
-34.4166
7
6.12.5
multivariate probit
8
fication. Box 1 shows the command file, qr6.cf. In this example, as is
often a good starting point for the estimation of a multivariate probit
model, the constraints assure that all correlations are zero.
Although the command files relies on default starting values convergence has been achieved in 5 iterations. However, due to the fact that
the minimization algorithm numerically approximates first and second
derivatives of the log-likelihood function, it needs altogether 321 evaluations of the likelihood function. Multiplying this with the number of cases
shows how often the multivariate normal integral has been evaluated.
6.14
models for count data
1
6.14 Models for Count Data
We consider a dependent variable, Y , with non-negative integer values.
Values of Y may be interpreted as the number of times that some event
occurs. Standard regression models for this type of dependent variable
include the Poisson model and several variants of a negative binomial
model. While TDA does not offer a specific command to estimate these
kinds of models, estimation is actually quite easy when using the fml
command and the operators for the Poisson and the negative binomial
distribution. This chapter shortly explains how to set up the likelihood
for some standard models and gives some examples. For a discussion of
the fml command see 6.11.2.
6.14.1
Poisson Regression
6.14.2
Compound Poisson Models
d0614.tex
April 21, 1998
6.14.1
poisson regression
1
6.14.1 Poisson Regression
The most simple model is based on the assumption that Y follows a
Poisson distribution, that is,
Pr(Y = k) =
θk
exp(−θ)
k!
θ > 0, k = 0, 1, 2, . . .
This implies that E(Y ) = Var(Y ) = θ. The standard Poisson regression
model uses the parameterization
θ = exp(xβ)
where x denotes a (row) vector of covariates. For numerical evaluation,
TDA offers the poisson
k
θ
poisson(θ, k) ≡ log
exp(−θ)
k!
This is a type I operator and allows for automatic differentiation with
respect to its first argument, θ. It is therefore quite easy to estimate
a Poisson regression model with the fml command. Given a sample of
observations, (yi , xi ), for i = 1, . . . , n, the log-likelihood is simply
`=
n
X
poisson(exp(xi β), yi )
i=1
Example 1 To illustrate model estimation we use the ship damage
data, data file cd1.dat, shown in Box 1. (These data have been used
by many authors. See, e.g., McCullagh and Nelder [1983, p. 137], Greene
[1992, p. 546], Winkelmann and Zimmermann [1991].) Box 2 shows the
command file, cd1.cf. The nvar command creates the necessary variables based on data file cd1.dat. Then follows the fml command. Since
the likelihood of the Poisson model is well behaved, we do not need any
parameters. It suffices to formulate the expression for the log-likelihood
on the right-hand side of the command. This is done in three steps. The
first step creates the intermediate expression xb, corresponding to xβ.
Note that parameter names can be arbitrary strings consisting of lower
d061401.tex
April 21, 1998
6.14.1
poisson regression
2
Box 1 Ship damage data file cd1.dat (McCullagh and Nelder [1983, p. 137]
#
#
#
#
#
Col 1
:
Col 2
:
Col 3 - 6 :
Col 7 - 9 :
Col 10
:
0
0
3
4
6
18
-1
11
39
29
58
53
12
44
-1
18
1
1
0
1
6
2
-1
1
0
0
0
0
2
11
-1
4
0
-1
7
7
5
12
-1
1
127
63
1095
1095
1512
3353
0
2244
44882
17176
28609
20370
7064
13099
0
7117
1179
552
781
676
783
1948
0
274
251
105
288
192
349
1208
0
2051
45
0
789
437
1157
2161
0
542
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
number of damage incidents
aggregate months of service
dummy variables for ship type (A - E)
dummy variables for year of construction
dummy variables for period of operation
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
6.14.1
poisson regression
3
Box 2 Command file cd1.cf
nvar(
dfile = cd1.dat,
isel = ge(c1,0),
# select valid cases
NDI
= c1,
Service = c2,
B
= c3,
C
= c4,
D
= c5,
E
= c6,
C60
= c7,
C65
= c8,
C70
= c9,
P75
= c10,
LOGS
= log(Service),
);
fml(
# we don’t need any parameters
) =
C60
exb
fn
xb = b0 + B * bb + C * bc + D * bd + E * be +
* bc60 + C65 * bc65 + C70 * bc70 + P75 * bp75 + LOGS * blogs,
= exp(xb),
= poisson(exb,NDI);
case letters and digits, but must begin with a lower case letter. The second step creates the intermediate expression exb ≡ exp(xβ). The final
step uses the poisson operator to formulate the log-likelihood. This final
expression is automatically summed over all cases in the currently active
data matrix.
Box 3 shows part of the standard output. Since the fml command
uses a default set up, the minimization algorithm is TDA’ default Newton
I algorithm, and all starting values are zero. Nevertheless, convergence
is achieved with 12 function calls. Since the poisson operator allows for
automatic differentiation (in its first argument), first and second derivatives of the log-likelihood are automatically calculated, and the second
derivatives are used for the standard errors (ccov = 2).
Example 2 As noted by McCullagh and Nelder [1983, p. 138], one can
assume that the parameter for the logarithm of aggregate months service
(blogs) should be 1. This constraint can easily be formulated by adding
con = b9 = 1,
to the fml command (see command file cd1a.cf in the example archive).
Note, however, that one cannot use the (arbitrary) name of the param-
6.14.1
poisson regression
Box 3 Part of standard output from cd1.cf (Poisson regression)
ML estimation of user-defined model.
Algorithm 5: Newton (I)
Number of model parameters: 10
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Idx
1
2
3
4
5
6
7
8
9
10
Parameter
b0
bb
bc
bc60
bc65
bc70
bd
be
blogs
bp75
Starting value
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
Convergence reached in 8 iterations.
Number of function evaluations: 12 (12,12)
Maximum of log likelihood: -67.8354
Norm of final gradient vector: 2.84694e-11
Last absolute change of function value: 5.57244e-14
Last relative change in parameters: 2.05576e-06
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 b0
-5.5940
0.8724
-6.4121 1.0000
2 bb
-0.3499
0.2702
-1.2948 0.8046
3 bc
-0.7631
0.3382
-2.2565 0.9760
4 bc60
0.6625
0.1536
4.3124 1.0000
5 bc65
0.7597
0.1777
4.2763 1.0000
6 bc70
0.3697
0.2458
1.5040 0.8674
7 bd
-0.1355
0.2971
-0.4560 0.3516
8 be
0.2739
0.2418
1.1331 0.7428
9 blogs
0.9027
0.1018
8.8672 1.0000
10 bp75
0.3703
0.1181
3.1339 0.9983
Log likelihood (starting values):
Log likelihood (final estimates):
-870.2939
-67.8354
4
6.14.1
poisson regression
5
Box 4 Part of standard output from qr1a.cf
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 b0
-6.4059
0.2174
-29.4600 1.0000
2 bb
-0.5433
0.1776
-3.0595 0.9978
3 bc
-0.6874
0.3290
-2.0891 0.9633
4 bc60
0.6971
0.1496
4.6587 1.0000
5 bc65
0.8184
0.1698
4.8207 1.0000
6 bc70
0.4534
0.2332
1.9446 0.9482
7 bd
-0.0760
0.2906
-0.2614 0.2062
8 be
0.3256
0.2359
1.3803 0.8325
9 blogs
1.0000
------10 bp75
0.3845
0.1183
3.2507 0.9988
Log likelihood (starting values): -161157.7779
Log likelihood (final estimates):
-68.2808
eter in the formulation of constraints but has to use a reference to the
number of the parameter. In this case, blogs is the 9th parameter, and
is therefore referred to by b9. Estimation results are shown in Box 4.
6.14.2
1
compound poisson models
6.14.2 Compound Poisson Models
The Poisson distribution is obviously restrictive in the assumption that
the mean equals the variance. Several less restrictive distributions have
been proposed in the literature. A standard approach thinks in terms
of a compound Poisson distribution. It is assumed, then, that the mean
of the Poisson distribution, θ, is a random variable. Let fθ (θ; θ∗ ) denote
its density function, depending on some further parameter vector, θ∗ . A
compound Poisson distribution is then given by
Z ∞ k
θ
Pr(Y = k) =
exp(−θ) fθ (θ; θ∗ ) dθ
k!
0
Following the discussion in Cameron and Trivedi [1986], one arrives at
a relatively simple model when a gamma mixture distribution ist use,
that is,
α
1 1 αθ
−αθ
fθ (θ; α, γ) =
exp
Γ(α) θ
γ
γ
having mean E(θ) = γ and variance Var(θ) = γ 2 /α. The distribution for
Y then becomes
Z ∞
(α/γ)α
α
Pr(Y = k) =
θα+k−1 exp −θ( + 1) dθ
k! Γ(α) 0
γ
Making the substitution θ → δγ/(α + γ), we find
α+1 Z ∞
(α/γ)α
γ
Pr(Y = k) =
δ α+k−1 exp(−δ) dθ
k! Γ(α) α + γ
0
The integral (Euler’s integral, see Abramowitz and Stegun [1964, p. 255])
equals Γ(α + k), so we find
k α
γ
α
Γ(α + k)
(1)
Pr(Y = k) =
Γ(k + 1) Γ(α) α + γ
α+γ
This is called a (general) negative binomial distribution. The distribution
has two parameters, α and γ. The mean is E(Y ) = γ, the variance is
Var(Y ) = γ + γ 2 /α.
d061402.tex
April 21, 1998
6.14.2
2
compound poisson models
To arrive at a regression model, one can make both parameters dependent on covariates. A particularly convenient approach has been proposed by Cameron and Trivedi [1986]:
γ = exp(xβ)
α =
1
exp(xβ)q
σ
where q is some given constant and σ is a model parameter to be estimated. The mean of Y is directly given by exp(xβ), and the variance
is
Var(Y ) = E(Y ) + σ E(Y )2−q
In particular, if q = 1, one gets
Var(Y ) = E(Y ) + σ E(Y )
called NEGBIN I model by Cameron and Trivedi. If q = 0, one gets
Var(Y ) = E(Y ) 1 + σ E(Y )
called NEGBIN II model by Cameron and Trivedi.1 In both cases, if
σ → 0, one arrives at the standard Poisson regression model.
The negbin Operator In order to evaluate the negative binomial distribution one can use TDA’s negbin operator, with syntax
negin(α, γ, k) ≡ log
(Γ(α + k)
Γ(k + 1) Γ(α)
γ
α+γ
k α
α+γ
α !
that directly evaluates the logarithm of the distribution as given in (1).
This is a type I operator and allows for automatic differentiation with
respect to its first two arguments, α and γ. The derivatives are calculated
as follows.
1 This
is the parameterization of the negative binomial model used in LIMDEP, see
Greene [1992, p. 539].
6.14.2
compound poisson models
∂ negbin(α, γ, k)
∂α
∂ negbin(α, γ, k)
∂γ
∂ 2 negbin(α, γ, k)
∂α∂α
∂ 2 negbin(α, γ, k)
∂γ∂γ
∂ 2 negbin(α, γ, k)
∂α∂γ
3
= digam(α + k) − digam(α) +
α
γ−k
log
+
α+γ
α+γ
k
α
=
−1
α+γ γ
= trigam(α + k) − trigam(α) +
γ
1
γ−k
−
α+γ α α+γ
α
1 − k/γ
k
=
− 2
α+γ α+γ
γ
=
k−γ
(α + γ)2
digam and trigam are operators that evaluate, respectively, the first and
second derivative of the logarithm of the gamma function.
The negbin operator allows an easy formulation of the log-likelihood
for a regression model with a negative binomial distribution. Given a
sample of observations (yi , xi ), for i = 1, . . . , n, the log-likelihood is
simply
`=
n
X
negbin (α, exp(xi β), yi )
i=1
where α can be made in an arbitrary way a function of the mean, i.e.,
exp(xi β).
Example 1 To illustrate, we estimate a NEGBIN II model for the ship
damage data that were used in 6.14.1. Box 1 shows part of the command file, cd2.cf (the first part is identical with the nvar command in
command file cd1.cf.) The expression
sigma = exp(sig),
is used to assure that σ is always positive. The model is estimated with
algorithm 8 that is somewhat more robust than TDA’s standard algorithm for function minimization.
6.14.2
compound poisson models
4
Box 1 Part of command file cd2.cf
fml(
mina = 8,
) = xb
C60 *
gamma
sigma
alpha
fn =
# select somewhat more robust algorithm
= b0 + B * bb + C * bc + D * bd + E * be +
bc60 + C65 * bc65 + C70 * bc70 + P75 * bp75 + LOGS * blogs,
= exp(xb),
= exp(sig),
= 1 / sigma,
negbin(alpha,gamma,NDI);
Box 2 shows the estimation results. σ is almost zero, indicating that
the NEGBIN II model does not give a better fit than the standard Poisson
model. In fact, the parameter estimates are almost the same as those
shown in Box 3. Of course, the huge standard error for the sig parameter
indicates that the estimate is not reliable.
6.14.2
compound poisson models
Box 2 Part of standard output from cd2.cf
Maximum likelihood estimation.
Algorithm 8: CES (tensor model [2])
Number of model parameters: 11
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 6
Tolerance for scaled gradient: 1e-05
Tolerance for scaled parameter change: 1e-08
Scaling factor: -1
Idx
1
2
3
4
5
6
7
8
9
10
11
Parameter
b0
bb
bc
bc60
bc65
bc70
bd
be
blogs
bp75
sig
Starting value
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
Convergence reached in 17 iterations.
Number of function evaluations: 52 (19,19)
Maximum of log likelihood: -67.8354
Norm of final gradient vector: 5.25902e-05
Final scaled gradient: 4.90683e-06
Final scaled parameter change: 0.066398
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 b0
-5.5940
0.8724
-6.4121 1.0000
2 bb
-0.3499
0.2702
-1.2948 0.8046
3 bc
-0.7631
0.3382
-2.2565 0.9760
4 bc60
0.6625
0.1536
4.3124 1.0000
5 bc65
0.7597
0.1777
4.2763 1.0000
6 bc70
0.3697
0.2458
1.5040 0.8674
7 bd
-0.1355
0.2971
-0.4560 0.3516
8 be
0.2739
0.2418
1.1331 0.7428
9 blogs
0.9027
0.1018
8.8672 1.0000
10 bp75
0.3703
0.1181
3.1339 0.9983
11 sig
-15.0643
213.8136
-0.0705 0.0562
Log likelihood (starting values):
Log likelihood (final estimates):
-270.3274
-67.8354
5
6.15
generalized linear models
1
6.15 Generalized Linear Models
This chapter deals with generalized linear models. It contains the following sections.
6.15.1
Introduction
6.15.2
The glm Command
d0615.tex
February 14, 2005
6.15.1
introduction
1
6.15.1 Introduction
Let Y denote a single dependent variable and (yi , xi ), for i = 1, . . . , n,
a sample of observations. For case i, yi is the value of the dependent
variable, and xi is a vector of covariates. To find a model for the distribution of Y , conditional on the information in covariates, one can
assume a parametric density for Y , say f (y; θ), and then make the parameter vector θ dependent on x. In general terms, this approach might
be written as
f (y; θ) with θ = u(x, β)
(1)
where the function u(x, β) depends on information given by the covariate
vector x, and a new parameter vector β. The log-likelihood for model
estimation may then be written as
`=
n
X
log f (yi ; u(xi , β))
(2)
i=1
TDA’s fml command described in 6.11.2 can be used to implement this
general approach.
However, while this approach is very general, it requires a complete
specification of the (conditional) density of the given observations. It
is often more practical to focus on the conditional expectation of the
dependent variable. The modeling approach then becomes
E(Y | x) = u(x, β)
(3)
Deriving a likelihood function depends, of course, on assuming some
density function for the observations. In general, if the expectation is
not directly given by one of the distributional parameters, this requires
a more or less complicated re-parameterization.
A relatively flexible approach is possible by using the exponential
family of distributions having the density
y θ − b(θ)
+ c(y, φ)
(4)
f (y; θ, φ) = exp
a(φ)
d061501.tex
February 14, 2005
6.15.1
introduction
2
where a(φ), b(θ) and c(y, φ) are some functions of the distributional
parameters φ and θ. The expectation is
E(Y ) = b0 (θ) =
∂b(θ)
∂θ
(5)
and the variance is given by
Var(Y ) = a(φ) b00 (θ) = a(φ)
∂ 2 b(θ)
∂θ2
(6)
b00 (φ) is often called variance function (of the distribution).
This exponential family of distributions has many familiar distributions as special cases and, furthermore, provides a comfortable starting
point for the modeling strategy (3). The required re-parameterization is
simply
E(Y | x) = b0 (θ) ≡ u(x, β)
(7)
An additional simplification is possible if one assumes that the scale
parameter, φ, does not vary across individual cases. It is possible, then,
to estimate β and φ separately.
This exponential family of distributions has become the starting point
for a broad literature on generalized linear models, see, e.g., McCullagh
and Nelder [1983], Aitkin et al. [1989] and, in particular, for the GLIM
program, see Francis et al. [1994]. Starting from (7), one needs the specification of b(θ) and u(x, β). Specification of b(θ) depends on which member of the exponential family of distributions will be used; some options
are discussed in later sections. Specification of u(x, β) can, in principle,
be arbitrary. Generalized linear models as normally defined and implemented, for instance, in the GLIM package (Francis et al., 1994), use a
two-part specification. One part is the linear predictor
η = xβ
(8)
optionally including an intercept term. The second part is the link function
η = g(µ),
µ = E(Y | x)
(9)
linking the conditional expectation, µ, to the linear predictor, η. The
link function must be invertible to allow the calculation of
µ = g −1 (η) = g −1 (xβ)
(10)
6.15.1
introduction
3
We will also assume that the link function is twice continuously differentiable. The relationship with the link function u, used in (7), is then
u(x, β) ≡ g −1 (η) = g −1 (xβ)
(11)
Secondly, when formulating generalized linear models, it is normally assumed that the scale parameter, φ, in (4) is constant across individual
cases. We then have that a(φ), but not φ, may depend on case-specific
information. In fact, most often there is a further simplification by assuming that
a(φ) = φ/w
(12)
where w is some case-specific prior weight, see Francis et al. [1994, p. 260].
Estimation Approach. Using the specification of generalized linear
models described above, maximum likelihood estimation can be achieved
with an iteratively reweighted least squares procedure, see McCullagh
and Nelder [1983, pp. 31–34], Francis et al. [1994, chap. 11]. The loglikelihood is
`=
n
X
yi θi − b(θi )
+ c(yi , φ)
φ/wi
i=1
(13)
In general, maximization requires an iterative procedure. Given a current
parameter vector, β, the Newton-Raphson method would calculate a new
estimate
β ∗ = β − H −1
∂`
∂β
(14)
where H is the Hessian matrix
∂2`
H=
∂βj ∂βk
Assuming m components in the parameter vector, this is an (m, m) matrix. A somewhat simpler approach uses the expectation of H which can
be substituted by
∂` ∂`
= −E (H)
H̃ = E
∂βj ∂βk
6.15.1
introduction
4
Considering first the gradient for the ith component of the log-likelihood,
we find
∂`i
∂`i ∂θi ∂µi ∂ηi
=
∂βj
∂θi ∂µi ∂ηi ∂βj
Using the model specification defined above, this becomes
∂`i
yi − µi 1
1
xij
=
∂βj
ai (φ) b00 (θi ) g 0 (µi )
(15)
Using the abbreviations
di = g 0 (µi )
and ui =
wi
φ b00 (θi ) d2i
this may also be written as
∂`i
= (yi − µi )ui di xij
∂βj
(16)
Since ∂`i /∂ηj = 0 if i 6= j, we find
H̃jk = E
∂` ∂`
∂βj ∂βk
=
n
X
u2i d2i E (yi − µi )2 xij xik
i=1
However, as already noted in (6),
φ 00
1
E (yi − µi )2 =
b (θi ) =
wi
ui d2i
and the expectation of the outer product simplifies into
X
n
∂` ∂`
H̃jk = E
=
ui xij xik
∂βj ∂βk
i=1
Written in matrix notation, we finally have
H̃ = X 0 U X ,
U = diag {u1 , , . . . , un }
(14) can then be rewritten as
β ∗ = β + (X 0 U X)−1
∂`
∂`
= (X 0 U X)−1 X 0 U Xβ +
∂β
∂β
(17)
6.15.1
5
introduction
This can be further simplified. Using (16) we can write
n
X
∂`
= (X 0 U z)j =
xij ui (yi − µi )di
∂βj
i=1
where z is an n-vector with components
zi = (yi − µi ) di
Consequently,
∂`
= X 0U z
∂β
and inserting this into (17), we find
β ∗ = (X 0 U X)−1 X 0 U [Xβ + z]
(18)
This is the basic equation for the iteratively re-weighted least squares
procedure. The steps are as follows:
1. Given a parameter vector β of starting values and an invertible link
function ηi = g(µi ) we can calculate
ηi = xi β
and µi = g −1 (ηi )
Since the link function is assumed to be differentiable, we can also calculate di = g 0 (µi ) and the components of the z vector, zi = (yi − µi )di .
While this can be done quite generally, we furthermore need the weights,
ui , depending on the scaling and variance functions. As noted above, we
shall assume that the scaling function is φ/wi , where wi is some casespecific weight. It is possible, therefore, to ignore the scaling function
when estimating the parameter vector β.1 On the other hand, we need
the variance function, b00 (θ), that depends on which member of the exponential family of distributions has been selected for model specification.
Several standard options will be described below. Given such a specification, we know the variance function and can calculate the working
weights
ũi =
1 Of
wi
b00 (θi ) d2i
course, we need this function for estimating a covariance matrix, see below.
6.15.1
introduction
6
2. The next step is to solve the weighted least squares problem defined in
(18) using the working weights Ũ = diag {ũ1 , . . . , ũn }. This least squares
problem may be written as
0
(Xβ + z − Xβ ∗ ) Ũ (Xβ + z − Xβ ∗ ) −→ min
with β the given parameter vector for the current iteration and β ∗ the
solution vector. For solving this least squares problem, TDA uses the
Hanson and Haskell [1982] algorithm described in Chapter 6.9.1. As an
option, this algorithm allows equality and inequality constraints as described below.
3. The procedure is iterated until the difference between β and β ∗ becomes small, or a maximum number of iterations has been performed.
TDA assumes convergence if
(
)
| βj∗ − βj |
max
≤ TOLSP
(19)
j
max{βj∗ , 1}
where TOLSP is some small value.
4. If the procedure successfully converged, one can estimate a covariance
matrix
Cov(β̂) = (X 0 U X)−1 = φ̂ (X 0 Ũ X)−1
(20)
using some estimate of the scale parameter, φ. These are the basic steps
performed by TDA to estimate generalized linear models. As indicated,
the procedure needs a variance function depending on which member of
the exponential family of distributions has been selected.
Estimating the Scale Parameter. Not all members of the exponential family of distributions do have a scale parameter. If there is a scale
parameter, estimation can be done separately, using an estimate of the
structural parameter vector, β̂. The log-likelihood for estimating φ is
`(φ) =
n
X
i=1
wi
yi θ̂i − b(θ̂i )
+ c(yi , φ)
φ
(21)
Solutions of maximizing this log-likelihood will be called ML-based estimates of the scale factor. Alternatively, one can use the concept of
6.15.1
introduction
7
deviance to estimate a scale parameter, see Francis et al. [1994, p. 270].
This estimate is defined by
φ̂d = D/df
(22)
where D denotes the deviance (see below) and df = n − rank(X) is the
number of degrees of freedom. φ̂d will be called deviance-based estimate
of the scale parameter.
Goodness-of-Fit Indicators. Two types of goodness-of-fit indicators
are often used when estimating generalized linear models. First, a generalized Pearson statistic, defined by
X2 =
n
X
(yi − µ̂i )2
,
V (µ̂i )
i=1
µ̂i = g −1 (xi β̂),
V (µ̂i ) = b00 (θ̂i )
(23)
The second indicator is called deviance and is derived from considering
likelihood ratios, see Francis et al. [1994, p. 274]. If M1 ⊂ M2 are two
nested models, the likelihood ratio is 2(`2 − `1 ); `1 and `2 being the
maxima of the corresponding log-likelihood functions. The idea behind
the concept of deviance is to compare the log-likelihood of the current
model, `c , with the log-likelihood of a saturated model, `s , defined as
a model having sufficient parameters to reproduce the observed data
exactly.
Ds = 2 ( `s − `c )
is called the scaled deviance (of the current model). Let θ̂i,c and θ̂i,s
denote estimates for the current and saturated model, respectively, the
latter being defined by b0 (θ̂i,s ) = yi . Also assume that we use the same
estimate, φ̂, for the current and for the saturated model. Then
Ds = 2
n
X
wi
yi (θ̂i,s − θ̂i,c ) − (b(θ̂i,s ) − b(θ̂i,c ))
i=1
φ̂
(24)
Without scaling,
D = φ̂ Ds
(25)
is called simply deviance. Detailed information about how TDA calculates both goodness-of-fit indicators will be given below, separately for
each type of distribution.
6.15.2
the glm command
1
6.15.2 The glm Command
The TDA command to estimate generalized linear models is glm. The
syntax is shown in Box 1. The command allows to select one of five
members of the exponential family of distributions, default is d=1, selecting a normal distribution. Details will be described in later sections,
separately for each type of distribution.
1. A list of variable names must be given with the v parameter, the
syntax is
v = Y,X1,...XM,
The right-hand side is a comma-separated list of variable names, optionally containing namelists. The first variable on the right-hand side is
used as dependent variable. The remaining variables are used to create
the linear predictor. If the dependent variable is followed by m variable
names, say X1 , . . . , Xm , the linear predictor is
ηi = β0 +
m
X
Xij βj
j=1
By default, the linear predictor contains an intercept term, β0 . The ni=1
parameter can be used to suppress this term. In this case, there must be
at least one independent variable.
2. An additional parameter, yw, is used to specify a dependent variable as
proportions. This will only be recognized if the distribution is binomial,
see 6.15.2.2.
3. The d parameter can be used to select a specific type of distribution.
Default is d = 1, i.e, the normal distribution. For a discussion of specific
distributions see subsequent sections.
4. There are two possibilities to specify a link function. One can use one
of the pre-defined link functions, or one can explicitly define a link function on the right-hand side of the command. Pre-defined link functions
can be selected with the link parameter. For available options and defaults, see Box 2. If a link function is specified on the right-hand side of
d061502.tex
February 14, 2005
6.15.2
the glm command
Box 1 Syntax for glm command
glm (
d=...,
link=...,
v=...,
yw=...,
ni=...,
lsecon=...,
lsicon=...,
mxit=...,
tolsp=...,
xp=...,
dsv=...,
tfmt=...,
ppar=...,
pcov=...,
mfmt=...,
pres=...,
fmt=...,
dtda=...,
prot=...,
pfmt=...,
ab=...,
type of distribution, def. 1
1 = Normal
2 = Binomial
3 = Poisson
4 = Gamma
5 = Inverse Gaussian
type of link function (if not user-defined)
1 = identity
2 = log
3 = logit
4 = reciprocal
5 = probit
6 = complementary log-log
7 = square root
8 = quadratic inverse
list of variables
additional variable to define proportions
1 if linear predictor without intercept, def. 0
equality constraints
inequality constraints
maximum number of iterations, def. 20
tolerance for parameter change, def. 1.e-8
starting values
input file with starting values
print format for results, def. tfmt=10.4
print estimated coefficients to output file
print covariance matrix to output file
print format for pcov option, def. 12.4
print data and estimated values to output file
print format for pres option, def. 10.4
TDA description of file written with pres
request protocol file
print format for protocol file, def. 19.11
domain of user-defined link function
def. ab=0,1
) = user-defined link function;
2
6.15.2
the glm command
3
Box 2 Built-in link functions
link
1
2
3
4
5
6
7
8
function
η
η
η
η
η
η
η
η
=µ
= log(µ)
= log(µ/(1 − µ))
= 1/µ
= Φ−1 (µ)
= log(− log(1 − µ))
√
= µ
= 1/µ2
default if
d=1
d=3
d=2
d=4
d=5
the command, the syntax must follow the conventions for the definition
of functions in TDA explained in 5.3. The function must contain exactly
one parameter and must not refer to data matrix variables. An arbitrary
name can be used for the function argument; in the examples below we
will use the name mue.
The algorithm for estimating generalized linear models requires the
link function to be invertible (monotone) and differentiable. If the link
function is η = g(µ), the algorithm needs µ = g −1 (η) and g 0 (µ). Derivatives are calculated by automatic differentiation. For calculating µ, given
η, for a user-defined link function, the algorithm also needs to know the
domain of the link function, that is, some interval µa < µ < µb . In
general, the domain will depend on the selected distribution and link
function. Default is 0 < µ < 1. To change this default, one can use the
parameter
ab = µa , µb ,
To illustrate user-defined link functions, here are some standard examples:
Identity
glm(...) = mue;
Log
glm(...) = log(mue);
Logit
glm(...) = log(mue / (1 - mue));
Reciprocal
glm(...) = 1 / mue;
Probit
glm(...) = ndi(mue);
Compl. log-log
glm(...) = log(-log(1 - mue));
Square root
glm(...) = sqrt(mue);
6.15.2
the glm command
4
5. As explained in 6.15.1, estimation is done with an iteratively reweighted least squares procedure. The behavior of the algorithm can be
controlled with two parameters. The mxit parameter can be used to define the maximum number of iterations, default is mxit=20. The tolsp
parameter can be used to define the convergence tolerance according to
(19), default is tolsp = 1.e-8. While the command performs the iterations, a message about the current value of the scaled parameter change
is written into the standard error output. To get additional information
about the iterations, one can request a protocol file with the parameter
prot = name of an output file,
The print format can be controlled with the pfmt parameter.
6. There are two options to define starting values for the model parameters, β. First, one can use
xp = b1,b2,...,
to provide starting values directly in the command file. Alternatively,
one can use
dsv = fname,
where fname is the name of a free-format input file containing starting
values in the first numerical column. If the model has m parameters, up
to m values are taken from the right-hand side of the xp parameter, or
read from the input file defined with the dsv parameter. By default, all
parameters get the starting value zero.
7. Basic estimation results are written into the standard output. Examples will be given below. The print format can be controlled with the
tfmt parameter, default is tfmt=10.4.
8. As an option, estimated parameters and standard errors can be written into an output file using the parameter
ppar = name of an output file,
The print format is the same as used for the standard output and may
be changed with the tfmt parameter.
9. Also optionally, one can write the estimated covariance matrix into
an output file using the parameter
pcov = name of an output file,
6.15.2
the glm command
5
The print format can be controlled with the mfmt parameter, default is
mfmt=12.4.
10. As a further option, one can use the parameter
pres = name of an output file,
to request an output file containing the following entries:
Case
Y
Ŷ
η
X1
...
Xm
W
Ũ
1
2
..
.
y1
y2
..
.
ŷ1
ŷ2
..
.
η1
η2
..
.
x11
x21
..
.
···
···
x1m
x2m
..
.
w1
w2
..
.
ũ1
ũ2
..
.
11. Finally, one can define equality and inequality constraints with the
lsecon and lsicon parameters, respectively. For an explanation of the
syntax, see 6.9.1. Note that a covariance matrix for the estimated parameters is only calculated if there are no inequality constraints.
6.15.2.1
normal distribution
1
6.15.2.1 Normal Distribution
As a first member of the exponential family of distributions, one gets
the normal distribution by specifying
1 2
1 y2
a(φ) = φ, b(θ) = θ , c(y, φ) = −
+ log(2πφ)
2
2 φ
Inserting these expressions into (4) and using
µ ≡ θ,
σ2 ≡ φ
one gets the normal density function
(
2 )
1
1 y−µ
2
f (y; µ, σ ) = √
exp −
2
σ
2πσ
The variance function is simply b00 (θ) = 1. Given an invertible link function ηi = g(µi ), the relation between the mean value of the distribution
and the linear predictor is
µi = g −1 (xi β)
If g(µi ) is the identity function, the resulting model is an ordinary linear
regression model. Of course, many other link functions can be used.
Note that, in general, the default domain for the link function will not
be appropriate for the normal distribution and should be redefined with
the ab parameter when user defined link functions are used.
The normal distribution has a scale parameter equal to its variance.
The log-likelihood for estimating this scale parameter is
`(φ) =
n
X
i=1
wi
yi µ̂i − µ̂2i /2 1
−
φ
2
yi2
+ log(2πφ)
φ
Maximizing this log-likelihood gives the estimate
Pn
wi (yi − µ̂i )2
φ̂ = i=1Pn
i=1 wi
d06150201.tex
February 14, 2005
6.15.2.1
normal distribution
2
Box 1 Command file glm1.cf
nvar(
dfile = lsreg1.dat,
# data file
Height = c1,
Weight = c2,
);
glm(
# use default normal distribution, identity link
v = Weight,Height,
# variables
pcov = cov,
ppar = par,
pres = res,
# write covariance matrix to cov
# write parameter to par
# write residuals to res
);
TDA uses this formula to calculate an ML estimate for the scale parameter. Note that this is different from the deviance-based estimate of the
scale parameter, see below.
For the normal distribution, the variance function is b00 (θ) = 1 and
the generalized Pearson statistic is
X2 =
n
X
(yi − µ̂i )2
i=1
The deviance is calculated according to (24) and (25). The saturated
model is defined by θ̂i,s = yi and b(θ̂i,s ) = yi2 /2. Inserting this into (24),
one gets
D=
n
X
wi (yi − µ̂i )2
i=1
TDA uses this formula to calculate the deviance, and the deviance-based
estimate of the scale factor, φ̂d = D/df .
Example 1 To illustrate the glm command, we begin with a replication of the simple linear regression model discussed in the first example
in 6.9.1. The command file, glm1.cf, is shown in Box 1, part of the
standard output is shown in Box 2. The command file uses a normal distribution and an identity link function (both being defaults). The same
results can be achieved with a user-defined link function, for example,
by using the command
glm (..., ab=1.e-6,1.e6) = mue;
6.15.2.1
normal distribution
Box 2 Part of standard output from glm1.cf
Distribution: normal.
Link function: identity.
Variables (cross-section)
------------------------Y
: Weight
X1 : Height
Estimation with iteratively re-weighted least squares.
Number of model parameters: 2
Maximum number of iterations: 20
Tolerance for scaled parameter change: 1e-06
Data and estimated values written to: res
Convergence reached in 2 iterations.
Final scaled parameter change: 0
Rank of data matrix: 2
Degrees of freedom: 17
Deviance
Pearson statistic
ML-based scaling factor
Deviance-based scaling factor
2142.4880
2142.4880
112.7625
126.0287
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept -143.0269
32.2746
-4.4316 0.9996
2
1 Height
3.8990
0.5161
7.5549 1.0000
Parameter estimates written to: par
Covariance matrix written to: cov
Box 3 Command file glm2.cf
nvar(
noc = 100,
X = rd(-3,3),
Y = exp(3 + X) + rd,
);
glm(
link = 2,
# logarithmic link function
v = Y,X,
# variables
xp = 5,5,
# some bad starting values
);
3
6.15.2.1
normal distribution
4
Box 4 Part of standard output from glm2.cf
Distribution: normal.
Link function: log.
Variables (cross-section)
------------------------Y
: Y
X1 : X
Estimation with iteratively re-weighted least squares.
Number of model parameters: 2
Maximum number of iterations: 20
Tolerance for scaled parameter change: 1e-06
Got 2 starting value(s) from xp parameter.
Convergence reached in 19 iterations.
Final scaled parameter change: 4.13314e-10
Rank of data matrix: 2
Degrees of freedom: 98
Deviance
Pearson statistic
ML-based scaling factor
Deviance-based scaling factor
19.9937
19.9937
0.1999
0.2040
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
3.0128
0.0020 1524.0440 1.0000
2
1 X
0.9958
0.0008 1299.1800 1.0000
Of course, we need to re-define the domain for the link function with
the ab parameter. The modified command file is supplied as glm1a.cf
in the example archive.
Example 2 To illustrate a logarithmic link function we use some random data, see command file glm2.cf in Box 3. Data generation is by
yi = exp(3 + xi ) + ri
where ri and xi are random numbers, equally distributed in (0, 1) and
in (−3, 3), respectively. Results are shown in Box 4. The parameters
are quite well recovered. (The example archive contains a command file,
glm2a.cf, that estimates the same model with a user-defined link function.)
6.15.2.2
binomial distribution
1
6.15.2.2 Binomial Distribution
If the dependent variable, Y , has possible values in {0, 1, . . . , m}, where
m ≥ 1, we may assume a binomial distribution with density function
m
f (y) =
π y (1 − π)m−y
0≤π≤1
(1)
y
This is again a special case of the exponential family of distributions.
This is easily seen by using the definitions
exp(θ)
π
,π =
θ = log
1−π
1 + exp(θ)
φ = 1
b(θ) = m log(1 + exp(θ))
m
c(y, φ) = log
y
Inserting these expressions into 6.15.1-4 will give (1). The expectation
is
E(Y ) = b0 (θ) = mπ
and the variance is
Var(Y ) = mπ(1 − π)
This shows that models for a dependent variable with a binomial distribution can be specified as special cases of generalized linear models. In
the glm command, the d=2 parameter selects the binomial distribution.
Proportions. We speak of individual-level data if m = 1. The dependent variable is then expected to take values in {0, 1}. Note that, contrary
to TDA’s qreg command, the glm command expects that the variable Y
actually has values in {0, 1}, there is no internal recoding.
On the other hand, if m > 1, we speak of grouped data, or proportions. In order to tell the glm command that it should use proportions,
d06150202.tex
February 14, 2005
6.15.2.2
binomial distribution
2
instead of individual-level data, one needs to specify an additional variable, say M, that contains the values of m. This is done with the additional
parameter
yw = M,
M must be the name of a variable that is defined in the currently active data matrix and must contain only positive integer values. And, of
course, if Y is the name of the dependent variable, we need to have
0 ≤ Yi ≤ Mi
and 1 ≤ Mi
Deviance. Since φ = 1, deviance and scaled deviance are identical. Calculation follows formula 6.15.1-24. For the saturated model, we assume
that
yi = µ̂i,s = b0 (θ̂i,s ) = mi π̂i,s
and, consequently,
π̂i,s
yi
θ̂i,s = log
= log
1 − π̂i,s
mi − y i
For the current model we find analoguously
π̂i,c
µ̂i
θ̂i,c = log
= log
1 − π̂i,c
mi − µ̂i
By the same reasoning, we also find
1
b(θ̂i,s ) = mi log
1 − yi /mi
1
b(θ̂i,c ) = mi log
1 − µ̂i /mi
It might happen that the expressions for the saturated model cannot be
evaluated. However, inserting into 6.15.1-24 gives
n
X
yi
mi − y i
Ds = 2
wi yi log
+ (mi − yi ) log
µ̂i
mi − µ̂i
i=1
TDA uses this formula for calculating the deviance in case of a binomial
distribution.
6.15.2.2
binomial distribution
3
Box 1 Command file glm3.cf
nvar(
dfile = qr1.dat,
Dose = c1,
Weight = c2,
Response = c3,
Log10Dose = log(Dose) / log(10),
);
cwt = Weight;
use case weights
glm(
d = 2,
# binomial distribution
v = Response,Log10Dose, # variables
);
Logit Link Function. We arrive at a standard logit model if we assume
that
µi
exp(ηi )
=
mi
1 + exp(ηi )
with the linear predictor ηi = xi β. This corresponds to a logit link
function that is the default for the binomial distribution. In order that
individual-level and grouped data can be treated in the same way, all
link functions for the binomial case will be interpreted as
g(µi /mi ) = ηi
If g is then specified as a logit function we have
µi /mi
ηi = log
1 − µi /mi
Example 1 To illustrate we replicate example 6.12.2-1. Box 1 shows
the command file, Box 2 shows part of the standard output. The case
weights defined with the cwt command are used by the glm command.
Estimation results are identical with the results from the qreg command.
The only difference is in the column Signif due to the fact that the qreg
command ues a normal distribution and the glm command, instead, a
t-distribution.
Example 2 To illustrate estimation of logit model with grouped data
we use the data file glm3.dat shown in Box 3. The data are taken from
6.15.2.2
binomial distribution
4
Box 2 Part of standard output from glm3.cf
Distribution: binomial.
Link function: logit.
Variables (cross-section)
------------------------Y
: Response
X1 : Log10Dose
Estimation with iteratively re-weighted least squares.
Using weights defined by: Weight
Number of model parameters: 2
Maximum number of iterations: 20
Tolerance for scaled parameter change: 1e-06
Convergence reached in 6 iterations.
Final scaled parameter change: 2.75435e-16
Rank of data matrix: 2
Degrees of freedom: 11
Deviance
Fixed scaling factor
74.2213
1.0000
Idx Wave Variable
Coeff
Error
Coeff/E Signif
-------------------------------------------------------------1
- Intercept
-3.2246
0.8861
-3.6393 0.9961
2
1 Log10Dose
5.9702
1.4492
4.1197 0.9983
Box 3 Data file glm3.dat
D
N
Y
------------0.01 49
0
2.60 50
6
3.80 48 16
5.10 46 24
7.70 49 42
10.20 50 44
Francis et al. [1994, p. 440]. The command file, glm4.cf (Box 4) contains two glm commands. Both estimate the same logit model. The first
command uses the build-in logit link function (default with the binomial
distribution), the second one uses a user-defined link function. Estimation results from the first command are shown in Box 5. Of course, the
6.15.2.2
binomial distribution
5
Box 4 Command file glm4.cf
nvar(
dfile = glm3.dat,
D = log(c1),
N = c2,
Y = c3,
);
glm(
d = 2,
# binomial distribution
v = Y,D,
# variables
yw = N,
# count
);
glm(
d = 2,
# binomial distribution
v = Y,D,
# variables
yw = N,
# count
) = log(mue / (1 - mue));
# user-defined link function
second glm command should give the same results.
Probit Link Function. Substituting the logit link function by a probit
function, i.e.,
ηi = Φ−1 (µi /mi )
results in a probit model. Box 6 shows the command file glm4a.cf that
replicates example 2 with a probit model. The first glm command uses
the link=5 parameter to select the in-build probit function, the second glm command uses an equivalent user-defined link function. (The
ndi operator evaluates the inverse of the standard normal distribution
function.)
6.15.2.2
binomial distribution
Box 5 Part of standard output from glm4.cf
Distribution: binomial.
Link function: logit.
Variables (cross-section)
------------------------Y
: Y
X1 : D
Variable used to define proportions: N
Estimation with iteratively re-weighted least squares.
Number of model parameters: 2
Maximum number of iterations: 20
Tolerance for scaled parameter change: 1e-06
Convergence reached in 8 iterations.
Final scaled parameter change: 2.86182e-16
Rank of data matrix: 2
Degrees of freedom: 4
Deviance
Fixed scaling factor
1.4241
1.0000
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
-4.8869
0.6429
-7.6010 0.9984
2
1 D
3.1035
0.3877
8.0047 0.9987
Box 6 Command file glm4a.cf
nvar(...);
glm(
d = 2,
link = 5,
v = Y,D,
yw = N,
);
glm(
d = 2,
v = Y,D,
yw = N,
) = ndi(mue);
# same as in glm4.cf
#
#
#
#
binomial distribution
probit link function
variables
count
#
#
#
#
binomial distribution
variables
count
user-defined link function
6
6.15.2.3
poisson distribution
1
6.15.2.3 Poisson Distribution
The Poisson distribution has density
f (y) = e−λ
λy
y!
λ>0
(1)
where y takes values in {0, 1, 2, . . .}. It is therefore often used to model
count data. The Poisson distribution is a special case of the exponential
family with
θ = log(λ) , λ = eθ
φ = 1
b(θ) = eθ
c(y, φ) = − log(y !)
The expectation is
E(Y ) = b0 (θ) = λ = eθ
and the variance is
Var(Y ) = λ = eθ
Thus the variance is equal to the expectation. The Poisson distribution
can be specified with the d=3 parameter in the glm command. The default link function is the logarithm (link=2). The glm command expects
the variable Y to have values in {0, 1, 2, . . .}.
Deviance. Since φ = 1, deviance and scaled deviance are identical.
Calculation follows formula (24). For the saturated model, we assume
that
yi = µ̂i,s = λ̂i,s
Thus
θ̂i,s = log λ̂i,s
d06150203.tex
February 14, 2005
6.15.2.3
poisson distribution
Box 1 Data file glm5.dat
X
Y
--------0.844
7
0.603
5
0.316
6
0.245
2
0.944
6
-0.322
1
0.365
3
-0.386
3
-0.795
1
-0.609
0
Box 2 Command file glm5.cf
nvar(
dfile = glm5.dat,
X
= c1,
# covariate
Y
= c2,
);
glm(
d = 3,
v = Y,X,
# Poisson distribution
# variables
);
For the current model we find
θ̂i,c = log λ̂i,c
Consequently,
b(θ̂i,s ) = λ̂i,s = yi
b(θ̂i,c ) = λ̂i,c
Inserting these expressions into 6.15.1-24 gives
(
!
)
n
X
yi
D=2
wi yi log
− (yi − λ̂i,c )
λ̂i,c
i=1
where we set 0 log(0) = 0.
2
6.15.2.3
poisson distribution
3
Box 3 Part of standard output from glm5.cf
Distribution: Poisson.
Link function: log.
Variables (cross-section)
------------------------Y
: Y
X1 : X
Estimation with iteratively re-weighted least squares.
Number of model parameters: 2
Maximum number of iterations: 20
Tolerance for scaled parameter change: 1e-06
Convergence reached in 9 iterations.
Final scaled parameter change: 2.22045e-16
Rank of data matrix: 2
Degrees of freedom: 8
Deviance
6.1902
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
0.8673
0.2384
3.6382 0.9934
2
1 X
1.1754
0.3565
3.2974 0.9891
Example 1 To illustrate we use the data in Box 1 with the command
file glm5.cf (Box 2). Part of the output is shown in Box 3.
6.15.2.3
poisson distribution
4
Box 4 Command file glm6.cf
nvar(
dfile = cd1.dat,
isel = ge(c1,0),
NDI
=
Service=
B
=
C
=
D
=
E
=
C60
=
C65
=
C70
=
P75
=
LOGS
=
);
# select valid cases
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
log(Service),
glm( d = 3,
v = NDI, B, C, D, E, C60, C65, C70, P75, LOGS,
lsecon = b9 = 1,
# constraint: blogs = 1
mxit = 100,
);
Example 2 In this example we use the ship damage data from McCullagh and Nelder [1983, p. 136–140]. The command file is glm6.cf (Box
4). It demonstrates the use of equality constraints: The effect of the aggregate months of service should be proportional to the expected number
of damage incidents. The coefficient of the logarithm of service time is
thus constraint to be 1. Also the number of iterations must be increased
to achieve convergence. Part of the output is shown in (Box 5). It replicates the results in McCullagh and Nelder [1983, p. 139] except for the
standard errors. McCullagh and Nelder use a factor of approximately 1.3
from an overdispersion model.
6.15.2.3
poisson distribution
Box 5 Part of standard output from glm6.cf
Distribution: Poisson.
Link function: log.
Variables (cross-section)
------------------------Y
: NDI
X1 : B
X2 : C
X3 : D
X4 : E
X5 : C60
X6 : C65
X7 : C70
X8 : P75
X9 : LOGS
Estimation with iteratively re-weighted least squares.
Number of model parameters: 10
LSECon: 1 b9 = 1
Equality constraints: 1
Inequality constraints: 0
Maximum number of iterations: 100
Tolerance for scaled parameter change: 1e-06
Convergence reached in 41 iterations.
Final scaled parameter change: 1.66533e-15
Rank of data matrix: 9
Degrees of freedom: 25
Deviance
38.6950
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
-6.4059
0.2174
-29.4600 1.0000
2
1 B
-0.5433
0.1776
-3.0595 0.9948
3
1 C
-0.6874
0.3290
-2.0891 0.9530
4
1 D
-0.0760
0.2906
-0.2614 0.2041
5
1 E
0.3256
0.2359
1.3803 0.8203
6
1 C60
0.6971
0.1496
4.6587 0.9999
7
1 C65
0.8184
0.1698
4.8207 0.9999
8
1 C70
0.4534
0.2332
1.9446 0.9368
9
1 P75
0.3845
0.1183
3.2507 0.9967
10
1 LOGS
1.0000
-------
5
6.15.2.4
gamma distribution
1
6.15.2.4 Gamma Distribution
The gamma distribution has density
f (y) =
1
Γ(ν)
ν
ν
νy
y ν−1 exp −
µ
µ
ν, µ > 0
(1)
where y takes values in the non-negative reals R+ . The gamma distribution is a special case of the exponential family with
θ = −
1
1
,µ = −
µ
θ
1
ν
b(θ) = − log(−θ)
φ =
c(y, φ) = (log(y) − log(φ)) /φ − log(Γ(1/φ))
The expectation is
E(Y ) = b0 (θ) = −
1
=µ
θ
and the variance is
00
Var(Y ) = b (θ)φ =
µ2
ν
Thus the variance is proportional to the square of the expectation. In
other words, the coefficient of variation is constant. The gamma distribution can be specified with the d=4 parameter in the glm command.
The default link function is the inverse link (link=4). This is also the
canonical link. But note that this link will not necessarily respect the
condition µ > 0. Thus, very often also the log link or the identity link
are considered.
Deviance. For the saturated model, we assume that
yi = µ̂i,s
d06150204.tex
February 14, 2005
6.15.2.4
gamma distribution
2
Box 1 Command file glm7.cf
nvar(
X = rd,
Y = -exp(1+X)*(log(rd) + log(rd))/2, # Gamma with form parameter 2
# and expectation exp(1+X)
);
glm(
d = 4,
link = 2,
v = Y,X,
);
clear;
nvar(
X = rd,
Y = -(log(rd) + log(rd))/((1+X)*2), # Gamma with form parameter 2
# and expectation 1/(1+X)
);
glm(
d = 4,
xp = 1,1,
# starting values required
mxit = 50,
v = Y,X,
);
If ν is taken to be a constant, the log likelihood reduces to
X yi
ν
wi − − log(µi )
µi
i
Since
P twice the log likelihood of the saturated model takes the value
−2 wi (1 + log(yi )), the deviance can be written as
n
X
yi
yi − µ̂i
D = −2
wi log
−
µ̂i
µ̂i
i=1
Note that this is only defined for yi > 0.
Estimating ν. Not implemented.
Example 1 We use simulated data as a first illustration. Box 1 shows
the command file. It generates gamma distributed variables with covariate effects first with the log link and then with the canonical inverse link.
Results are shown in Box 2 and 3.
6.15.2.4
gamma distribution
Box 2 Part of standard output from glm7.cf
Distribution: gamma.
Link function: log.
Estimation with iteratively re-weighted least squares.
Number of model parameters: 2
Maximum number of iterations: 20
Tolerance for scaled parameter change: 1e-06
Convergence reached in 10 iterations.
Final scaled parameter change: 1.72821e-08
Rank of data matrix: 2
Degrees of freedom: 998
Deviance
519.1471
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
0.9620
0.0624
15.4239 1.0000
2
1 X
1.0947
0.1070
10.2332 1.0000
Box 3 Part of standard output from glm7.cf
Distribution: gamma.
Link function: reciprocal.
Estimation with iteratively re-weighted least squares.
Number of model parameters: 2
Maximum number of iterations: 50
Tolerance for scaled parameter change: 1e-06
2 starting value(s) from xp parameter.
Convergence reached in 3 iterations.
Final scaled parameter change: 4.12683e-15
Rank of data matrix: 2
Degrees of freedom: 998
Deviance
550.6416
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
1.0082
0.0748
13.4819 1.0000
2
1 X
1.0223
0.1604
6.3736 1.0000
3
6.15.2.4
gamma distribution
4
Box 4 Command file glm8.cf
nvar(
dfile = glm8a.dat,
U = c1,
Y = c2,
L = c3,
X = log(U),
LX = L * X,
);
tsel=eq(L,0);
glm(
d = 4,
v = Y,X,
xp = -0.02,0.02,
# starting values required
);
tsel=eq(L,1);
glm(
d = 4,
v = Y,X,
xp = -0.02,0.02,
# starting values required
);
Example 2 In this example we use the blood clutting data from McCullagh and Nelder [1983, p. 163]. The command file is glm8.cf (Box 4).
It uses the canonical (inverse) link separately for the two lots. Part of
the output is shown in Box 5. It replicates the results in McCullagh and
Nelder [1983, p. 163], except for the estimated standard errors. The latter are computed using a moment estimate of the dispersion parameter
by McCullagh and Nelder.
6.15.2.4
gamma distribution
Box 5 Part of standard output from glm8.cf
Distribution: gamma.
Link function: reciprocal.
Estimation with iteratively re-weighted least squares.
Number of model parameters: 2
Maximum number of iterations: 20
Tolerance for scaled parameter change: 1e-06
2 starting value(s) from xp parameter.
Convergence reached in 5 iterations.
Final scaled parameter change: 1.48388e-12
Rank of data matrix: 2
Degrees of freedom: 7
Deviance
0.0167
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
-0.0166
0.0188
-0.8827 0.5933
2
1 X
0.0153
0.0084
1.8287 0.8898
tsel=eq(L,1)
Distribution: gamma.
Link function: reciprocal.
Convergence reached in 4 iterations.
Final scaled parameter change: 2.45984e-15
Rank of data matrix: 2
Degrees of freedom: 7
Deviance
0.0127
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
-0.0239
0.0311
-0.7675 0.5321
2
1 X
0.0236
0.0135
1.7423 0.8750
5
6.15.2.5
inverse gaussian distribution
1
6.15.2.5 Inverse Gaussian Distribution
The inverse Gaussian distribution has density
)
(
2
1
(y − µ)
µ, σ 2 > 0
f (y) = p
exp − 2 2
2σ µ y
2πσ 2 y 3
(1)
where y takes values in the positive reals. The inverse Gaussian distribution is a special case of the exponential family with
θ = −
1
1
,µ = √
2µ2
−2θ
φ = σ2
√
b(θ) = − −2θ
1
1
c(y, φ) = −
log(2πφy 3 ) +
2
φy
The expectation is
E(Y ) = b0 (θ) = − √
1
=µ
−2θ
and the variance is
00
Var(Y ) = b (θ)φ = µ3 σ 2
Thus the variance is proportional to the cube of the expectation. The
inverse Gaussian distribution can be specified with the d=5 parameter
in the glm command. The default link function is the quadratic inverse
link (link=5). This is also the canonical link.
Deviance. For the saturated model, we assume that
yi = µ̂i,s
If σ 2 is taken to be a constant, the log likelihood reduces to
yi
1
1 X
wi − 2 +
σ2 i
2µi
µ
d06150205.tex
February 14, 2005
6.15.2.5
inverse gaussian distribution
2
Since
P twice the log likelihood of the saturated model takes the value
− wi /(yi σ 2 ), the deviance can be written as
)
(
n
2
X
(yi − µ̂)
D=
wi
yi µ̂2
i=1
6.15.2.5
inverse gaussian distribution
3
Box 1 Command file glm9.cf
nvar(
X = rd,
Mu = 1 + X,
Z = rdn^2,
X1 = Mu + Mu^2 * Z - Mu * sqrt(2 * Mu * Z + Mu^2 * Z^2),
Y = if le(rd,Mu / (Mu + X1)) then X1 else Mu^2 / X1,
);
glm(
d = 5,
link = 1,
xp = 0.5,0.5,
v = Y,X,
);
Box 2 Part of standard output from glm9.cf
Distribution: inverse Gaussian.
Link function: identity.
Estimation with iteratively re-weighted least squares.
Number of model parameters: 2
Maximum number of iterations: 20
Tolerance for scaled parameter change: 1e-06
2 starting value(s) from xp parameter.
Convergence reached in 6 iterations.
Final scaled parameter change: 7.00738e-09
Rank of data matrix: 2
Degrees of freedom: 998
Deviance
1933.9975
Idx Wave Variable
Coeff
Error
Coeff/E Signif
------------------------------------------------------------1
- Intercept
1.0702
0.0867
12.3497 1.0000
2
1 X
0.8148
0.1909
4.2690 1.0000
Example 1 We use simulated data as an illustration. Box 1 shows
the command file. It generates inverse Gaussian variables with covariate
effects with the identity link using a simulation method described in
Seshadri [1993, p. 203–204]. In this case, non-default starting values
must be given. Results are shown in Box 2.
6.16
nonlinear regression
1
6.16 Nonlinear Regression
This chapter deals with nonlinear regression. It includes linear regression
as a special case. Currently, there is only a single section.
6.16.1
d0616.tex
The freg Command
April 21, 1998
6.16.1
the freg command
1
6.16.1 The freg Command
If Yi is a dependent variable for cases i = 1, . . . , n and Xi is a corresponding vector of covariates, a nonlinear regression model can be written as
f (θ) =
n
X
(Yi − g(Xi , θ))2 −→ min
i=1
While it would be possible to use the fmin command to find a solution,
there are somewhat different conventions for calculating a covariance matrix for parameter estimates. TDA therefore provides a complementary
command, freg, for nonlinear regression. The syntax and parameters
for this command are identical with the fmin command. The only difference is that, when using the freg command, the covariance matrix is
calculated as
2
f (θ̂)
H(θ̂)−1
n−p
n is the number of data matrix cases, p the number of model parameters,
H(θ) is the Hessian of f (θ), and θ̂ denotes the parameter estimates. This
convention makes linear regression a special case of nonlinear regression.
d061601.tex
April 21, 1998
6.16.1
the freg command
2
Box 1 Command file freg1.cf
nvar(
dfile = freg1.dat,
Y = c2,
X1 = c3,
X2 = c4,
X3 = c5,
);
freg (
xp = -0.1,1,-1,-1,
# data file (Gallant, p. 4)
# starting values
) = ax = a1 * X1 + a2 * X2 + a4 * exp(a3 * X3),
fn = (Y - ax)^2;
Example 1 To illustrate the freg command we use an example from
Gallant [1987, p. 4]. The command file, freg1.cf, is shown in Box 1. It
creates four variables, Y, X1, X2, X3, based on the data file freg1.dat
containing Gallant’s example data. The model is
Y = X1 a1 + X2 a2 + a4 exp(X3 a3 ) + The freg command specifies the corresponding function to be minimized. Its standard output is shown in Box 2.
For most applications, the freg command, like the fmin and fml
commands, requires reasonable starting values.
6.16.1
the freg command
Box 2 Output from command file freg1.cf
Nonlinear regression. Current memory: 148695 bytes.
Function definition:
ax
= a1*X1+a2*X2+a4*exp(a3*X3)
fn
= (Y-ax)^2
Function will be summed over 30 data matrix cases.
Nonlinear regression.
Algorithm 5: Newton (I)
Number of model parameters: 4
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: 1
Got starting values from xp parameter.
Idx Parameter Starting value
1 a1
-0.1000
2 a2
1.0000
3 a3
-1.0000
4 a4
-1.0000
Convergence reached in 9 iterations.
Number of function evaluations: 11 (11,11)
Minimum of function: 0.0304955
Norm of final gradient vector: 3.95162e-11
Last absolute change of function value: 4.75342e-11
Last relative change in parameters: 4.7722e-06
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 a1
-0.0259
0.0126
-2.0511 0.9597
2 a2
1.0157
0.0099
102.4481 1.0000
3 a3
-1.1157
0.1607
-6.9423 1.0000
4 a4
-0.5049
0.0255
-19.7862 1.0000
Function (starting values):
Function (final estimates):
1.3728
0.0305
3
6.17
transition rate models
1
6.17 Transition Rate Models
This chapter deals with transition rate models. They require an episode
data structure as explained in 3.3. The sections are:
6.17.1
Introduction
6.17.2
Exponential Models
6.17.3
Parametric Duration Dependence
6.17.4
Mixture Models
6.17.5
User-defined Rate Models
6.17.6
Discrete Time Rate Models
6.17.7
Semi-parametric Rate Models
d0617.tex
April 21, 1998
6.17.1
introduction
1
6.17.1 Introduction
This section provides introductory remarks about specification and estimation of transition rate models. Specific types of models are dealt with
in subsequent sections. Here we shortly discuss TDA’s general approach
to the formulation of parametric transition rate models.
6.17.1.1 Parametric Rate Models
6.17.1.2 Maximum Likelihood Estimation
6.17.1.3 Time-varying Covariates
6.17.1.4 The rate Command
6.17.1.5 Relative Risks
6.17.1.6 Generalized Residuals
d061701.tex
April 21, 1998
6.17.1.1
1
parametric models
6.17.1.1 Parametric Rate Models
For the discussion in this section we assume a set of single episodes:
(oi , di , si , ti , xi )
i = 1, . . . , N
(1)
where si is the starting time of the episode, ti is the ending time, oi
is the origin state, di is the destination state, and xi is a (row) vector
of covariates which, for the moment, are assumed to have fixed values
during the episode to which they are associated. Some of the episode
may be right censored, but we assume that they are not left censored.
Formulation (1) allows for different origin and destination states and,
therefore, covers the case of multi-episode data. Model estimation generally assumes that the individual episodes are statistically independent
and this might not be true for multi-episode data where each individual
can contribute more than one episode. But this requirement is conditional on covariates, and can often be met by using covariates to capture
information about the past history of the individual episodes.
We use O and Dj (j ∈ O) to refer to the origin and destination
state spaces, respectively. Any pair (j, k), j ∈ O and k ∈ Dj , is called a
transition from origin state j to destination state k.
The modeling approach will be based on a process time axis, that
is, we assume that all episodes begin at time zero (si = 0 for all i =
1, . . . , N ). It is then possible to describe the episodes in terms of durations as discussed in 3.3.1.
The general setup of the models will be explained in two steps. First
we look at the case of a single transition. There is a single random variable, T , for the duration in the origin state until a transition to the
destination state occurs. To derive a model, one assumes a known parametric distribution for this duration variable T , depending on some unknown parameters a, b, c, . . . The distribution may be given by a density
function f (t ; a, b, c, . . .) or, likewise, by a distribution function
Z t
F (t ; a, b, c, . . .) =
f (τ ; a, b, c, . . .) dτ
(2)
0
or by a survivor function
G(t ; a, b, c, . . .) = 1 − F (t ; a, b, c, . . .)
d06170101.tex
(3)
April 21, 1998
6.17.1.1
parametric models
2
or by a transition rate
r(t ; a, b, c, . . .) =
f (t ; a, b, c, . . .)
G(t ; a, b, c, . . .)
(4)
Second, if episodes can end in one of several different destination states
then, for each origin state j ∈ O, there is a two-dimensional random
variable (Tj , Dj ), with Tj the duration in the origin state and Dj the
destination state after leaving the origin state. Generalization of transition rate models to this case is based on the concept of transition-specific
rates defined by
Pr (t ≤ Tj < t + ∆t, Dj = k | Tj ≥ t)
∆t→0
∆t
rjk (t) = lim
(5)
For the formulation of parametric transition rate models we assume that
all transition-specific rate rjk (t) have the same mathematical form. In
general, we begin by assuming a model for the single-transition case
and simply extend this model to a situation with alternative destination
states. The general approach is then
rjk (t ; ajk , bjk , cjk , . . .) ≡ r(t ; ajk , bjk , cjk , . . .)
(6)
with r(t ; . . .) taken from the single-transition case, but providing the
possibility of transition-specific parameters.
To formulate the likelihood for the model one needs also survivor
functions, Gj (t), for the duration in the given origin state j ∈ O. They
can easily be derived from the transition-specific rates. One can first
define transition-specific pseudo-survivor functions
Z t
G̃jk (t) = exp −
rjk (τ ) dτ
(7)
0
As a consequence of assuming identical rates for the different transitions,
all pseudo-survivor functions have the same mathematical form as the
corresponding survivor function in the single-transition case, and Gj (t)
may be written as
Y
Gj (t) =
G̃jk (t)
(8)
k∈Dj
Summing up: For the formulation of parametric transition rate models in
TDA, we start with an expression for the rate in the single-transition case.
6.17.1.1
parametric models
3
This is then generalized for the case of several transitions by assuming
the same mathematical form for each transition-specific rate. Finally,
the survivor function for the duration in a given origin states is derived
from the transition-specific rates. As is seen in the next section, this is
all what is needed for maximum likelihood estimation of the models.
There is, of course, a broad range of possibilities for the construction
of transition rate models. One can start with some familiar parametric distribution and then derive the implied transition rate; or one can
immediately start with some expression for the transition rates. Both
approaches will be used in subsequent chapters. In any case, there are
some unknown parameters, denoted by ajk , bjk , and cjk , generally assumed to have different values for each transition. The models described
in later chapters have, in fact, different numbers of unknown parameters (for each transition). There are simple models with only one set of
parameters, and more flexible models with up to three different sets of
parameters. Using the terminology of the rate program (Tuma 1980),
they are called, respectively, the A-terms, the B-terms, and the C-terms
of a model.
The final step in model formulation is to specify how covariates shall
be linked to the transition rate functions that primarily define the model.
This, sometimes called the parameterization of a model, is done by linking covariates to the unknown parameters of the transition-specific rates.
In principle, many different link functions are possible. Here we follow
the most common approach: if parameters can only take positive values
an exponential link function is used, otherwise a linear link function. The
exponential link functions are
n
o
ajk = exp A(j,k) α(j,k)
bjk
=
n
o
exp B (j,k) β (j,k)
cjk
=
n
o
exp C (j,k) γ (j,k)
(9)
A(j,k) , B (j,k) , and C (j,k) , are (row) vectors of covariates, and in most
cases it is assumed that the first component is equal to a constant one
to provide an intercept term. α(j,k) , β (j,k) , and γ (j,k) , are the associated
coefficients. Therefore, if no covariates are specified, TDA estimates a
so-called null model , i.e. a model with only constant effects.
6.17.1.2
1
maximum likelihood estimation
6.17.1.2 Maximum Likelihood Estimation
For model estimation TDA uses the method of maximum likelihood. To
explain the setup of the likelihood, we proceed in three steps: first we
consider the case of a single transition, then situations with one origin
state but possibly two or more destination states, and finally situations
with more than one origin state.
The notation to set up the likelihood is as follows. O is the set of
origin states, and Dj is the set of possible destination states for episodes
with origin j ∈ O. Nj and Zj are, respectively, the set of all episodes and
the set of all right censored episodes, having an origin state j. Ejk is the
set of all episodes with origin state j and destination state k which have
an event (j 6= k). To simplify notation the dependence on parameters is
not explicitly shown.
In the case of a single transition (j, k), the likelihood may be written
as
Y
Y
Y
Y
f (ti )
Ljk =
G(ti ) =
r(ti )
G(ti )
(1)
i∈Ejk
i∈Zj
i∈Ejk
i∈Nj
where f (t) is the density function and G(t) is the survivor function for the
single transition (j, k). The contribution to the likelihood of an episode
with an event at ti is the density function, evaluated at the ending time
ti and with appropriate covariate values. The contribution of a censored
episode is the survivor function evaluated at the censored ending time
ti , but possibly depending on covariates changing their values during the
episode. Remember that, for the moment, we assume all episodes have
starting time zero. The likelihood can be expressed, then, by using only
the transition rate and the survivor function.
The next step is to look at the case with a single origin state j but
possibly more than one destination state. The destination state space is
Dj . Using the notation introduced in 3.3.1, the likelihood may be written
as
Y Y
Y
Lj =
f˜jk (ti )
Gj (ti )
(2)
k∈Dj i∈Ejk
i∈Zj
The contribution of an episode with event at ti is again the density
function, but now transition-specific, according to the underlying model.
d06170102.tex
April 21, 1998
6.17.1.2
maximum likelihood estimation
2
As shown in 3.3.1, this density is given by the subdensity function f˜jk (t),
with the interpretation that f˜jk (t) dt is the probability of leaving the
origin state j to the destination state k in a small time interval at t. The
contribution to the likelihood of a right censored episode is the value of
the survivor function at ti , denoted by Gj (ti ), i.e. the probability that
the episode starting in state j has no event until ti .
It is possible to factor the likelihood into a product of transitionspecific terms.1 First, the likelihood (2) may be rewritten as
Y Y
Y
Lj =
rjk (ti )
Gj (ti )
(3)
k∈Dj i∈Ejk
i∈Nj
Secondly, using 6.17.1.1-(8), this may be rewritten as
Y Y
Y Y
G̃jk (ti )
Lj =
rjk (ti )
k∈Dj i∈Ejk
or, with terms rearranged:



Y Y
Y
Lj =
rjk (ti )
G̃jk (ti )


k∈Dj
i∈Ejk
(4)
k∈Dj i∈Nj
(5)
i∈Nj
The third step, to account for more than a single origin state, is again
simple. One assumes that the model for each type of episode is conditional on a given origin state. Then also the total likelihood is the
product of the likelihood for each of the origin states, Lj , and can be
written as
Y Y Y
Y
L=
rjk (ti )
G̃jk (ti )
(6)
j∈O k∈Dj i∈Ejk
i∈Nj
In the case of a single origin state this likelihood reduces to (5), and in
the case of a single transition it reduces to (1).
Estimation of parametric transition rate models in TDA is always
based on the likelihood (6), by using the corresponding log-likelihood
n
o
X X X
X
`=
log {rjk (ti )} +
log G̃jk (ti )
(7)
j∈O k∈Dj i∈Ejk
1 This
i∈Nj
assumes transition rates defined in continuous time. For discrete-time models
this factoring is not possible as will be discussed in 6.17.6.
6.17.1.2
maximum likelihood estimation
3
The fact that the total likelihood can be expressed as a product of
transition-specific factors implies that, in the case of several transitions,
a model for each transition can be estimated separately.2 Estimation for
a transition (j, k) is then done using all episodes starting in origin state
j; the episodes ending in destination state k are regarded as having an
event, and all other episodes are included as censored.
The possibility of factoring the total likelihood opens an easy way to
estimate models with a different specification of transition rates for different transitions. A different model can be assumed for each transition,
and then each of these models can be estimated separately.
2 Of course, this is only possible if there are no constraints on parameters across
different transitions. To provide for this possibility is the main reason for using the
likelihood (6).
6.17.1.3
1
time-varying covariates
6.17.1.3 Time-varying Covariates
For practical applications one should be able to include time-varying
covariates into the specification of transition rate models. There are basically three different approaches.
1. The first approach develops a special model formulation that accounts for time-dependent covariates. For each application, this method
requires a new formulation of the model and, accordingly, new equations
for maximum likelihood estimation must be derived. This is impossible
with a general purpose program and not supported by TDA.
2. A second method was proposed by Tuma (1980). The idea is to split
the time axis into predefined time intervals and then to specify possibly
different covariates with associated coefficients for each of these intervals. This method is supported by one version of the piecewise-constant
exponential model discussed in 6.17.2.3.
3. In practice, the most useful method is known as episode splitting,
and this is the preferred method for handling time-varying covariates in
TDA. The basic idea is very simple. At all time points when at least one
of the covariates changes its value, the original episode is split into subepisodes, called splits (of an episode), and each split is given appropriate
values of the covariates. All splits have the same origin state as the
original episode and, except the last split, are regarded as right censored.
Only the last split is given the same destination state as the original
episode. To be used with split episodes, the likelihood expression for
transition rate models needs some modification. We need an expression
based on split episodes that might have starting times greater than zero.
Fortunately, the necessary modifications in the likelihood construction
are simple. Consider one of the original episodes (si , ti ), with si = 0, and
assume it is split into li sub-episodes
(l)
(l)
(si , ti ) ≡ {(si , ti ) | l = 1, . . . , li }
(1)
The pseudo-survivor functions defined in (7) can then be written as
G̃jk (ti ) =
li
Y
(l)
(l)
G̃jk (ti | si )
(2)
l=1
d06170103.tex
April 21, 1998
6.17.1.3
time-varying covariates
2
where, on the right-hand side, one has conditional pseudo-survivor functions1 defined by
( Z (l)
)
(l)
ti
G̃jk (ti )
(l)
(l)
G̃jk (ti | si ) =
= exp −
rjk (τ ) dτ
(3)
(l)
(l)
si
G̃jk (si )
In the integral on the right-hand side, rjk (τ ) is the rate appropriate for
(l) (l)
the split (si , ti ) and may depend on the covariate values for this split.
Now it would be possible to rewrite again the likelihood function (6)
by inserting (3). But one does not really need the subscripts, l. One can
simply write the likelihood function as
Y Y Y
Y
G̃jk (ti | si )
(4)
L=
rjk (ti )
j∈O k∈Dj i∈Ejk
i∈Nj
Written this way, the likelihood can be used as well with a sample of
original (not split) episodes where all starting times are zero, and with
a sample of splits. Of course, it is the responsibility of the user to do
any episode splitting in such a way that the splits add up to a sample of
meaningful episodes.
1 In
the case of a single transition it is the probability of staying in the origin state
(l)
(l)
at least until ti given that this state was not exited until si .
6.17.1.4
the rate command
1
6.17.1.4 The rate Command
TDA’s command to request estimation of a transition rate model is rate.
The syntax is
rate (parameter) = model number;
Currently available model numbers are shown in Box 1; parameter options are summarized in Box 2. Also, most options available in the fmin
command for function minimization can be use with the rate command,
see 5.6.1. These additional options are not shown in Box 2.
1. For most models all parameters are optional and the rate command
can be used as
rate = model number;
resulting in the estimation of a null model withou covariates. An exception are the piecewise constant exponential models where it is required
to use the tp parameter to define time periods.
2. Possibilities to include covariates depend on the model terms provided
by the selected model (see 6.17.1.1). In general, covariates are linked to
model terms by using expressions like
xa(j,k) = list of variables,
meaning that the list of variables to be given on the right-hand side
is linked to transition from origin state j to destination state k in the
model term A. If a model also provides a model term B, one can use
analoguously the expression
xb(j,k) = list of variables,
Some models also provide model terms C and D. The available possibilities depend on the type of model and will be explained separtely for
each model.
3. For some models, additional parameters can be used for model specification. If a model contains polynomial terms, one can use the deg
d06170104.tex
December 18, 1999
6.17.1.4
the rate command
2
Box 1 Models available with the rate command
Model
rate =
1
2
3
4
5
6
7
8
9
10
12
13
14
16
20
21
Cox model
Exponential model
Piecewise constant exponential model
Polynomial rates, I
Polynomial rates, II
Gompertz-Makeham models
Weibull model
Sickle model
Log-logistic model, type I
Log-logistic model, type II
Log-normal model
Generalized gamma model
Inverse Gaussian model
Piecewise constant exponential model
with period-specific effects
Logistic regression model
Complementary log-log model
parameter to specify the degree. The kgam parameter is used for the
generalized gamma model. The grp parameter is used to define groups
for stratified Cox model estimation. Additional explanations for these
parameters will be given in separate sections.
4. Basic results of model estimation are always written into the standard output. This includes information about the results of maximum
likelihood estimtion, in particular whether the selected minimization algorithm has reached convergence. In any case, one gets a table with
estimated parameters and, if the algorithm terminated successfully, also
estimated standard errors. The print format can be controlled with the
tfmt option, default is tfmt=10.4. In addition, for some models, one
can use the rrisk option to request a table containing relative risks; see
6.17.1.5.
5. The ppar option can be used to request an output file containing the
estimated parameters and standard errors. The syntax is
ppar = name of an output file,
6.17.1.4
the rate command
3
Box 2 Syntax for rate command
rate (
tp=...,
xa(j,k)=...,
xb(j,k)=...,
xc(j,k)=...,
xd(j,k)=...,
deg=...,
kgam=...,
mix=...,
grp=...,
rrisk,
ppar=...,
pcov=...,
pres=...,
prate=...,
tfmt=...,
mfmt=...,
mplog=...,
mppar=...,
mpcov=...,
mpgrad=...,
dsv=...,
con=...,
time periods
variables for A-term of model
variables for B-term of model
variables for C-term of model
variables for D-term of model
degree of polynomial rate, def. 0
coefficient for gamma model, def. 1
selection of mixture model, see 6.17.4.1
groups for stratified Cox model
print table with relative risks
output file with estimated parameters
output file with covariance matrix
generalized residuals, see 6.17.1.6
calculation of estimated rates
print format for results, def. 10.4
print format for pcov, pres and prate, def. 12.4
write loglikelihood value into matrix
write estimated parameters into matrix
write covariance matrix into matrix
write gradients into matrix
input file with starting values
linear parameter constraints
in addition, one can use all parameters to
control maximum likelihood estimation.
) = model number;
The print format is the same as used for the table in the standard output
and can be changed with the tfmt option.
6. As mentioned, standard errors are only calculated if the minimization
algorithm terminates successfully. In this case it is also possible to request
an output file containing the full covariance matrix with the pcov option.
The print format can be controlled with the mmfmt option, default is
mfmt=12.4.
7. For many models TDA can calculate so-called generalized residuals,
see 6.17.1.6. If supported, one can use the option
pres = name of an output file,
6.17.1.4
the rate command
4
to request an output file containing the estimated generalized residuals.
The print format can be controlled with the mfmt option, default is
mfmt=12.4.
8. One often wishes to tabulate, or plot, the transition rates, survivor
and density functions resulting from estimation of a model, depending
on certain constellations of covariates. This can be done with the prate
option, the syntax is
prate (tab=a(d)b, V1=..., V2=...,...) = fname,
The right-hand side must provide the name of an output file. Except for
the Cox model (see below), the tab parameter must be used to specify
a time axis, running from a to b with increments d; a must be nonnegative, b must be greater than a, and d must be positive. Then follows,
optionally, a list of variable names each with syntax
VName = v,
where VName is the name of a variable used in the model specification
and v is the value to be used for this variable when estimating rates. All
variable not explicitly mentioned will get the value zero. Consider, for
example, the expression
prate (tab=0(1)100, X1=1, X2=2) = r.dat,
The output file, r.dat, will tabulate the estimated rates for the time
axis 0, 1, 2, . . . , 100 and uses covariate values X1=1 and X2=2; all other
covariates (if any) will be zero. We mention that the prate option can be
used more than once in the same rate command. Each prate expression
will create a new table. If the output file name is the same as previously
used, the table will be appended to the end of the file.
For the Cox model, the prate option can be used to request tables
with estimated baseline rates; this will be explained in 6.17.7.6. For
other parametric models, the output file will contain the estimated rates,
survivor and density functions. The calculation depends on the type of
model and will be described separately for each type of model. The print
format can be controlled with the mfmt option, default is mfmt=12.4. For
an example, see 6.17.2.1.
9. The dsv parameter can be used to provide starting values for model
estimation. The syntax is
dsv = name of an input file,
6.17.1.4
the rate command
5
The input file is assumed to be a free-format file. TDA uses the first
numerical entry in each record assuming an ordering of parameters as
used in model estimation and shown in the table of estimated parameters.
In particular, it is possible to use an output file created with the ppar
option as an input file for the dsv option. Alternatively, one can use the
xp parameter to provide starting values, see 5.6.4.
10. The con option can be used to specify linear constraints for the
model parameters, see the discussion in 5.6. Also most other options,
discussed there, to control maximum likelihood estimation can be used
as additional parameters in the rate command.
11. For a few parametric rate models one can use the mix parameter
to request model estimation based on a gamma mixing distribution, see
the discussion in 6.17.4.1.
6.17.1.5
relative risks
1
6.17.1.5 Relative Risks
Many of TDA’s transition rate models can be written as proportional rate
models:
r(t) = rb (t; θ) exp
m
X
Ai αi
(1)
i=1
Here the transition rate, r(t), is a product of two parts. The first part,
rb (t; θ), is called the baseline transition rate. In general, with the exception of the simple exponential model, the baseline rate depends on
time and on a parameter vector θ. The second part links the transition
rate to individual-specific covariates; the formulation in (1) assumes m
covariates, A1 , . . . , Am , with associated parameters α1 , . . . , αm .
Models which can be given the representation (1) are called proportional rate models because the effects of individual-specific covariates
are independent of time. Only the baseline transition rate is dependent
on time; different individual-specific covariates only lead to proportional
shifts of the baseline rate.
Of course, this interpretations assumes that the baseline rate’s parameter vector, θ, is not also linked to individual-specific covariates. In
fact, most transition rate models which can be estimated with TDA also
provide the opportunity to link some or all components of θ to individualspecific covariates; and if this is done the resulting model is no longer a
proportional rate model.
The main advantage of proportional rate models is their simple interpretation. There is a single baseline rate, independent of individualspecific covariates and therefore easily assessed, for instance by plotting
this rate. Then, having a picture of the baseline rate, the influence of
individual-specific covariates can be represented by their contribution to
shifting the baseline rate. This is easily seen by rewriting (1) as
r(t) = rb (t; θ)
m
Y
ri (Ai )
with ri (Ai ) = exp(Ai αi )
(2)
i=1
For instance, if a1 and a2 are two values of the covariate Ai , the proportional shift of the baseline rate due to a change from a1 to a2 is
d06170105.tex
April 21, 1998
6.17.1.5
relative risks
2
ri (a2 )/ri (a1). In particular, if Ai is a dummy variable with possible values 0 and 1, the whole information about this variable’s influence on the
transition rate can be given by ri (1), i.e. by exp(αi ).
To aid in this interpretation of proportional transition rate models,
TDA provides the option to request a table containing the estimated relative risks, ri (1). To request this table, simply use rrisk as a parameter
in the rate command. Note, however, that for some transition rate models, this option is not available. Whether it is, and how the calculations
are performed, is described separately for each type of model.
6.17.1.6
generalized residuals
1
6.17.1.6 Generalized Residuals
Since estimation of transition rate models is based on maximum likelihood, it is difficult to assess goodness of fit. As proposed by Cox and
Snell [1968], it might be helpful to calculate so-called generalized residuals. For applying this idea to transition rate models see, for instance,
Blossfeld et al. [1989, p. 82] and Lancaster [1985].
The definition is as follows. Let r̂(t; x) denote the estimated rate,
depending on time t and on a vector of covariates, x. The estimation is
based on a random sample of individuals i = 1, . . . , N , with duration
ti and covariate vectors xi . Generalized residuals are then defined as
cumulative transition rates, evaluated for the given sample observations,
i.e.
Z ti
êi =
r̂(τ ; xi ) dτ
i = 1, . . . , N
(1)
0
The reasoning behind this definition is that, if the model is appropriate,
and if there were no censored observations, the set of residuals should
approximately behave like a sample from a standard exponential distribution. If some of the observations are censored, the residuals may be
regarded as a censored sample from a standard exponential distribution.
In any case, one can calculate a product-limit estimate of the survivor
function of the residuals, say Gê (e), and a plot of − log(Gê (êi )) against
êi can be used to check whether the residuals actually follow a standard
exponential distribution.
For most of TDA’s continuous time transition rate models one can
request a table with generalized residuals by using the option
pres = name of an output file,
as part of the rate command for model estimation. Depending on the
estimated model, residuals are calculated and then written into the specified output file. The print format can be controlled with the mfmt option,
default is mfmt=12.4. The variables (columns) in the output file are as
follows:
1. Case number.
2. Origin state of the episode.
d06170106.tex
April 21, 1998
6.17.1.6
generalized residuals
2
3. Destination state of the episode.
4. Starting time of the episode.
5. Ending time of the episode.
6. Estimated transition rate, evaluated at the ending time of the episode.
Note that the calculation is independent of whether the case is right
censored or not. The information about origin and destination states
can be used to select censored or uncensored cases.
7. Estimated survivor function, evaluated at the ending time of the
episode.
8. Residual.
9. Case weight.
Calculation of residuals is conditional on given starting times which may
be greater than zero. There will then be no straightforward interpretation. In any case, residuals are calculated according to
Z ti
êi =
r̂(τ ; xi ) dτ
i = 1, . . . , N
(2)
si
with si and ti denoting the starting and ending time of the episode,
respectively. Also the survivor functions printed into the output file are
conditional on starting times, meaning that the program calculates and
prints
Z ti
Ĝ(ti | si ; xi ) = exp −
r̂(τ ; xi ) dτ
i = 1, . . . , N
(3)
si
Therefore, if the starting times are not zero, in particular if the method of
episode splitting is applied, the output file will not contain proper information about residuals. We also mention that calculation of generalized
residuals is only supported for single episode data without alternative
destination states.1 It is also not supported for Cox models, mixture
models, and discrete time transition rate models.
1 If
one really needs these residuals for models with alternative destination states one
should estimate a separate model for each transition.
6.17.2
exponential models
1
6.17.2 Exponential Models
This section describes transition rate models with a constant, or piecewise constant, rate. The sections are:
6.17.2.1 The Basic Exponential Model
6.17.2.2 Models with Time Periods
6.17.2.3 Period-specific Effects
d061702.tex
April 21, 1998
6.17.2.1
1
basic exponential model
6.17.2.1 The Basic Exponential Model
The most simple transition rate model is the exponential where it is
assumed that the transition rates can vary with different constellations
of covariates but do not depend on time. In the case of a single transition
the model is derived by assuming that the duration variable T can be
described by an exponential distribution with density, survivor function,
and transition rates, respectively, given by
f (t)
= a exp(−a t)
G(t)
=
r(t)
exp(−a t)
= a
a>0
(1)
(2)
(3)
The transition rates are constants, not varying with the duration of the
episode. In most applications this might be not realistic. But the model is
not only simple, it can serve as a reference for more complicated models,
and some more flexible extensions of this model are described in later
sections.
Implementation. The simple exponential model has model number 2;
the command to request model estimation is
rate (parameter) = 2;
For a description of the syntax and options see 6.17.1.4.
The exponential distribution has only one parameter, so there is only
one possibility to include covariates (there is only an A-term in this
model). TDA uses an exponential link function. Taking into account the
possibility of different transitions one gets the model formulation
n
o
rjk = exp A(jk) α(jk)
(4)
rjk is the constant transition rate from origin state j to destination
state k. A(jk) is a (row) vector of covariates, having a constant one in
its first component to provide an intercept term; and α(jk) is a vector of
associated coefficients to be estimated.
The command rate=2 estimates a null model without covariates. For
linking covariates to the model for transition from j to k one should use
d06170201.tex
December 18, 1999
6.17.2.1
basic exponential model
2
the parameter
xa(j,k) = list of variables,
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify notation we omit
indices for transitions. Using the conditional survivor function
G(ti | si ) = exp {−(ti − si ) exp(Ai α)}
(5)
the log-likelihood is
X
X
`=
Ai α +
(si − ti ) exp(Ai α)
i∈E
i∈N
Using the same notation, the first and second derivatives of the loglikelihood with respect to the α coefficients are derived to be
∂`
∂αja
=
∂2`
∂αja ∂αka
=
X
Ai,ja +
i∈E
X
X
(si − ti ) exp(Ai α)Ai,ja
(6)
i∈N
(si − ti ) exp(Ai α)Ai,ja Ai,ka
i∈N
The notation Ai,j is used to denote the jth component of the covariate
vector Ai associated with the ith episode.
Starting Values. In most cases there are no difficulties in reaching a
ML solution for the exponential model. The calculation of default starting values by TDA is build on the fact that the ML solution of an exponential null model can be found directly and does not need an iterative
procedure.1 In the single transition case the rate of an exponential null
model is given by
rn =
Wu
Tw
(7)
where Wu is the number of uncensored episodes, and Tw is the summed
duration of all episodes. Default starting values for exponential models
are calculated as the parameters of a corresponding null model.
1 Cf.
Lawless [1982, p. 101].
6.17.2.1
basic exponential model
3
Box 1 Command file rt1.cf for simple exponential model
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
COHO1
COHO2
COHO3
W
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
ge(TB,348) & le(TB,384), # birth
ge(TB,468) & le(TB,504), # birth
ge(TB,588) & le(TB,624), # birth
SEX[2],
# women
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
cohort 1
cohort 2
cohort 3
= 1
# destination state
);
edef(
ts = 0,
tf = DUR,
org = 0,
des = DES,
);
#
#
#
#
starting time
ending time
origin state
destination state
rate (
xa(0,1) = COHO2,COHO3,W,
) = 2;
exponential model
Example 1 To illustrate estimation of an exponential model we use
our main example data, rrdat.1 (see 3.3.3). Box 1 shows the command
file, rt1.cf. Most commands are used to specify variables and episode
data. The rate command at the end of the file requests model estimation.
Three covariates are linked to the model for transition from 0 to 1,
the only transition in the input data. For all other options of the rate
command, default values are used. The output from the rate command
is shown in Box 2. Convergence is reached with 5 iterations using TDA’s
6.17.2.1
basic exponential model
4
Box 2 Output from command file rt1.cf
Transition rate models.
Model: Exponential
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 4
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 5 iterations.
Number of function evaluations: 6 (6,6)
Maximum of log likelihood: -2475.44
Norm of final gradient vector: 9.66315e-08
Last absolute change of function value: 2.62535e-11
Last relative change in parameters: 3.26799e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Constant
-5.0114
0.0843
-59.4446 1.0000
2 1
0
1 A COHO2
0.5341
0.1120
4.7686 1.0000
3 1
0
1 A COHO3
0.6738
0.1152
5.8472 1.0000
4 1
0
1 A W
0.5065
0.0942
5.3746 1.0000
Log likelihood (starting values): -2514.0201
Log likelihood (final estimates): -2475.4383
default algorithm for ML estimation. The table shows the estimated
parameter values and standard errors.
Example 2 If the input data contain several transitions covariates can
be linked separately to each transition. To illustrate this option we use
command file rt1m.cf, shown in Box 4. There are now three transitions,
and we link some covariates to each of them. While it would be possible
to use different sets of covariates for each transition, we have not done
6.17.2.1
basic exponential model
5
Box 3 Command file rt1m.cf
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
COHO1
COHO2
COHO3
W
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
ge(TB,348) & le(TB,384), # birth
ge(TB,468) & le(TB,504), # birth
ge(TB,588) & le(TB,624), # birth
SEX[2],
# women
cohort 1
cohort 2
cohort 3
= 1
DES [1.0] = if eq(TF,TI) then 0 else
if ge(PRESN/PRES - 1,0.2) then 1 else
if lt(PRESN/PRES - 1,0.0) then 3 else 2,
DUR [3.0] = TF - TS + 1,
);
edef(
ts = 0,
tf = DUR,
org = 0,
des = DES,
);
#
#
#
#
starting time
ending time
origin state
destination state
rate (
xa(0,1) = COHO2,COHO3,W,
xa(0,2) = COHO2,COHO3,W,
xa(0,3) = COHO2,COHO3,W,
) = 2;
exponential model
so in this example. Part of TDA’s standard output from this command
file is shown in Box 4.
Example 3 To control model estimation, one can use all the options
6.17.2.1
basic exponential model
6
Box 4 Part of standard output from file rt1m.cf
Convergence reached in 5 iterations.
Number of function evaluations: 6 (6,6)
Maximum of log likelihood: -2935.58
Norm of final gradient vector: 2.6882e-05
Last absolute change of function value: 6.16377e-09
Last relative change in parameters: 0.000460972
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Constant
-6.3412
0.1736
-36.5229 1.0000
2 1
0
1 A COHO2
0.3012
0.2618
1.1504 0.7500
3 1
0
1 A COHO3
0.4999
0.2651
1.8857 0.9407
4 1
0
1 A W
-0.0957
0.2342
-0.4084 0.3170
------------------------------------------------------------------5 1
0
2 A Constant
-5.7548
0.1236
-46.5439 1.0000
6 1
0
2 A COHO2
0.6924
0.1630
4.2486 1.0000
7 1
0
2 A COHO3
0.8088
0.1684
4.8039 1.0000
8 1
0
2 A W
0.2977
0.1381
2.1556 0.9689
------------------------------------------------------------------9 1
0
3 A Constant
-6.3686
0.1578
-40.3698 1.0000
10 1
0
3 A COHO2
0.4424
0.1924
2.2996 0.9785
11 1
0
3 A COHO3
0.5875
0.1980
2.9676 0.9970
12 1
0
3 A W
1.1156
0.1659
6.7254 1.0000
Log likelihood (starting values): -2986.9500
Log likelihood (final estimates): -2935.5833
described in 5.6.1. For example, one can use the ppar option to write
estimated model parameters into an output file, the pcov option for the
covariance matrix, and the dsv option for reading starting values.
It is also possible to estimate models with linear constraints on a
subset of parameters. Assume, for example, we want to estimate the
model defined in command file rt1m.cf with the additional constraint
that the effects of COHO3 are equal for the transitions (0, 1) and (0, 3).
The corresponding model parameters have indices 3 and 11, respectively.
So the additional parameter for the rate command should be
con = b3 - b11 = 0,
Relative Risks. The rrisk option can be used as part of the rate command to request a table with relative risks. Since the exponential model
is a proportional transition rate model, the estimated model parameters
6.17.2.1
basic exponential model
7
Box 5 Relative risks for the model estimated by command file rt1m.cf
Idx SN Org Des MT Variable
R.Risk
------------------------------------1 1
0
1 A Constant
0.0018
2 1
0
1 A COHO2
1.3514
3 1
0
1 A COHO3
1.6485
4 1
0
1 A W
0.9088
------------------------------------5 1
0
2 A Constant
0.0032
6 1
0
2 A COHO2
1.9985
7 1
0
2 A COHO3
2.2452
8 1
0
2 A W
1.3468
------------------------------------9 1
0
3 A Constant
0.0017
10 1
0
3 A COHO2
1.5564
11 1
0
3 A COHO3
1.7994
12 1
0
3 A W
3.0514
Box 6 Generalized residuals estimated with rt1.cf
# Generalized residuals.
#
Survivor
# Case Org Des
TS
TF
Rate Function Residual Weight
# -----------------------------------------------------------1
0
0 0.0 428.0
0.0067
0.0578
2.8513
1.0000
2
0
1 0.0
46.0
0.0111
0.6014
0.5086
1.0000
3
0
1 0.0
34.0
0.0111
0.6867
0.3759
1.0000
4
0
1 0.0 220.0
0.0111
0.0878
2.4322
1.0000
5
0
1 0.0
12.0
0.0189
0.7975
0.2263
1.0000
...........................................................
595
0
1 0.0 119.0
0.0067
0.4526
0.7928
1.0000
596
0
1 0.0
66.0
0.0067
0.6442
0.4397
1.0000
597
0
0 0.0 112.0
0.0067
0.4742
0.7461
1.0000
598
0
0 0.0 103.0
0.0131
0.2603
1.3461
1.0000
599
0
1 0.0
22.0
0.0189
0.6604
0.4149
1.0000
600
0
1 0.0
13.0
0.0189
0.7826
0.2452
1.0000
can be changed into relative risks by taking their anti-logarithm. Box 5
shows the effect of the rrisk option when added to the rate command
in command file rt1m.cf (Box 3).
Generalized Residuals. If the exponential model is estimated for only
a single transition, one can use the pres option to request a table with
generalized residuals. For each episode (oi , di , si , ti , Ai ) in the input data,
the output file specified with the pres option will contain the estimated
6.17.2.1
basic exponential model
8
rate
r̂i = exp(Ai α̂)
the estimated (conditional) survivor function
Ĝ(ti | si ) = exp(−(ti − si ) r̂i )
and the generalized residual
êi = − log(Ĝ(ti | si ))
To illustrate the structure of the output file, the parameter pres=res.d
is added to the rate command in command file rt1.cf. Part of the
resulting output file, res.d, is shown in Box 6.
6.17.2.1
basic exponential model
9
Box 7 Table with estimated rates based on rt1.cf
# Time axis: 0 (5) 100
# Idx SN Org Des MT Variable
Coeff Covariate
# -----------------------------------------------#
1 1
0
1 A Constant
-5.0114
1.0000
#
2 1
0
1 A COHO2
0.5341
0.0000
#
3 1
0
1 A COHO3
0.6738
1.0000
#
4 1
0
1 A W
0.5065
1.0000
# ID
Time
Surv.F
Density
Rate
# -----------------------------------------------------0
0.0000
1.0000
0.0217
0.0217
0
5.0000
0.8972
0.0195
0.0217
0
10.0000
0.8050
0.0175
0.0217
0
15.0000
0.7223
0.0157
0.0217
0
20.0000
0.6481
0.0141
0.0217
0
25.0000
0.5815
0.0126
0.0217
0
30.0000
0.5217
0.0113
0.0217
0
35.0000
0.4681
0.0102
0.0217
0
40.0000
0.4200
0.0091
0.0217
0
45.0000
0.3768
0.0082
0.0217
0
50.0000
0.3381
0.0073
0.0217
0
55.0000
0.3034
0.0066
0.0217
0
60.0000
0.2722
0.0059
0.0217
0
65.0000
0.2442
0.0053
0.0217
0
70.0000
0.2191
0.0048
0.0217
0
75.0000
0.1966
0.0043
0.0217
0
80.0000
0.1764
0.0038
0.0217
0
85.0000
0.1583
0.0034
0.0217
0
90.0000
0.1420
0.0031
0.0217
0
95.0000
0.1274
0.0028
0.0217
0
100.0000
0.1143
0.0025
0.0217
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations. To illustrate this option, we use command file rt1.cf and add the option
prate (tab=0(5)100, COHO3=1,W=1) = r.dat,
to the rate command. The resulting output file, r.dat, is shown in Box
7. The header shows the parameter and covariate values used to create
the table. The calculation of the rate, survivor and density functions is
based on the estimated model parameters and the specified covariate
values using the formulas for the exponential model. The first column of
6.17.2.1
basic exponential model
10
each table, labelled by ID, can be used to identify the table. If the prate
option is used more than once in the same rate command then, using
the same output file name, it will contain several tables which can be
identified by their ID column.
6.17.2.2
models with time periods
1
6.17.2.2 Models with Time Periods
For most applications, the basic exponential model is not appropriate.
However, a small generalization leads to a very useful and flexible model,
called the piecewise constant exponential model. The idea is to split the
time axis into periods and assume that transition rates are constant in
each period but may vary across periods.
If there are m time periods, the distribution of durations has m parameters. To include covariates one has two different options. The first
is to assume that only a baseline rate, given by period-specific constants,
can vary across time periods but the covariates have the same (proportional) effects in each period. The second option would be to allow for
period-specific effects of covariates. TDA’s standard exponential model
with time periods follows the first option; a model with period specific
effects is described in 6.17.2.3.
Implementation. The exponential model with time periods has model
number 3; the command to request model estimation is
rate (tp=..., additional parameters) = 3;
The tp parameter must be used to provide time periods defined by split
points on the time axis:
0 = τ1 < τ2 < τ3 < . . . < τm
(1)
With τm+1 = ∞, one gets m time periods
Il = {t | τl ≤ t < τl+1 }
l = 1, . . . , m
(2)
The syntax of the tp parameter is
tp = τ1 , τ2 , . . . , τm ,
This parameter must be used with τ1 = 0 when requesting an exponential
model with time periods.
Given these time periods, the transition rate from origin state j to
destination state k is
n
o
(jk)
rjk (t) = exp ᾱl + A(jk) α(jk)
if t ∈ Il
(3)
d06170202.tex
April 21, 1998
6.17.2.2
models with time periods
2
(jk)
For each transition (j, k), ᾱl
is a constant coefficient associated with
the lth time period. A(jk) is a (row) vector of covariates, and α(jk) is an
associated vector of coefficients assumed not to vary across time periods.
The model must not contain a separate constant. If specified without
covariates, TDA estimates a null model, containing only parameters for
the baseline rate. Covariates can be added to the A-term of the model
with the option
xa(j,k) = list of variables,
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify notation we omit
indices for transitions and define l[t] to be the index of the time period
containing t (so always t ∈ Il[t] ). Also the following notation is helpful:
1
if t ∈ Il
δ[t, l] =
(4)
0
else

t − τl
if s ≤ τl , τl < t < τl+1



τl+1 − τl
if s ≤ τl , t ≥ τl+1
∆[s, t, l] =
(5)
τl+1 − s
if t ≥ τl+1 , τl < s < τl+1



0
else
The conditional survivor function may then be written as
( m
)
X
G(t | s) = exp −
∆[s, t, l] exp(ᾱl + Aα)
(6)
l=1
Using this expression, the log-likelihood can be written as
`=
X
(ᾱl[ti ] + Ai α) −
i∈E
m
XX
∆[si , ti , l] exp(ᾱl + Ai α)
(7)
i∈N l=1
and the first and second derivatives are1
X
X
∂`
=
δ[ti , j] +
−∆[si , ti , j] exp(ᾱj + Ai α)
∂ ᾱj
i∈E
i∈N
m
X
XX
∂`
=
Ai,j +
−∆[si , ti , l] exp(ᾱl + Ai α) Ai,j
∂αj
i∈E
1δ
i∈N l=1
jk , here and in later sections, denotes the Kronecker symbol which is equal to one,
if j = k, and is otherwise zero.
6.17.2.2
models with time periods
3
X
∂2`
=
−∆[si , ti , j] exp(ᾱj + Ai α) δjk
∂ ᾱj ∂ ᾱk
i∈N
m
XX
∂2`
=
−∆[si , ti , l] exp(ᾱl + Ai α)Ai,j Ai,k
∂αj ∂αk
i∈N l=1
X
∂2`
=
−∆[si , ti , j] exp(ᾱj + Ai α)Ai,k
∂ ᾱj ∂αk
i∈N
Starting Values. TDA calculates default starting values for the piecewise constant exponential model by assuming the same rate in each time
period. As an initial estimate the constant rate of a correspondingly defined exponential null model is used. In most practical situations this
is sufficient to reach convergence of the maximum likelihood iterations.
Alternatively, one can use the dsv option to provide starting values.
Example 1 To illustrate estimation of an exponential model with time
periods we continue with command file rt1.cf (see 6.17.2.1), and modify the rate command. The model number is now 3, instead of 2; and
we add the parameter
tp = 0 (12) 96,
to specify time periods. The new command file is rt2.cf (not shown).
The output from this command is shown in Box 1. It begins with a table
showing the time periods and the distribution of starting and ending
times of the episodes used for model estimation. In particular, the table
shows the number of events occurring in each of the time periods. This
is useful information because the model is only estimable if each time
period contains sufficient events.
Example 2 The piecewise constant exponential model contains the basic exponential model as a special case. Consequently, adding constraints
to the parameters in the piecewise constant model should result in estimates for the basic exponential model. The rate command in Box 2
(command file rt2c.cf) illustrates this by adding 8 constraints in order to have the same baseline rate in all time periods. The result of
this command should be identical to the output from rt1.cf shown in
6.17.2.1.
6.17.2.2
models with time periods
Box 1 Part of output from command file rt2.cf
Model: Exponential with time-periods
Time period
Starting times Ending times
Events
----------------------------------------------------------0.0000 12.0000
600
76
63
12.0000 24.0000
0
104
96
24.0000 36.0000
0
108
95
36.0000 48.0000
0
55
44
48.0000 60.0000
0
44
38
60.0000 72.0000
0
38
29
72.0000 84.0000
0
22
17
84.0000 96.0000
0
9
6
96.0000 0
144
70
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 12
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Convergence reached in 5 iterations.
Number of function evaluations: 6 (6,6)
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Period-1
-5.2070
0.1520
-34.2464 1.0000
2 1
0
1 A Period-2
-4.5473
0.1312
-34.6574 1.0000
3 1
0
1 A Period-3
-4.2675
0.1296
-32.9389 1.0000
4 1
0
1 A Period-4
-4.7784
0.1690
-28.2758 1.0000
5 1
0
1 A Period-5
-4.7300
0.1779
-26.5924 1.0000
6 1
0
1 A Period-6
-4.7862
0.1987
-24.0857 1.0000
7 1
0
1 A Period-7
-5.1366
0.2523
-20.3568 1.0000
8 1
0
1 A Period-8
-6.0783
0.4137
-14.6918 1.0000
9 1
0
1 A Period-9
-5.4198
0.1281
-42.3123 1.0000
10 1
0
1 A COHO2
0.4367
0.1133
3.8534 0.9999
11 1
0
1 A COHO3
0.5162
0.1182
4.3688 1.0000
12 1
0
1 A W
0.4336
0.0949
4.5688 1.0000
Log likelihood (starting values): -2514.0201
Log likelihood (final estimates): -2433.3757
4
6.17.2.2
models with time periods
5
Box 2 Illustration of constraints (rt2c.cf)
rate (
tp = 0 (12) 96,
xa(0,1) = COHO2,COHO3,W,
con
con
con
con
con
con
con
con
=
=
=
=
=
=
=
=
b1
b2
b3
b4
b5
b6
b7
b8
-
b2
b3
b4
b5
b6
b7
b8
b9
=
=
=
=
=
=
=
=
0,
0,
0,
0,
0,
0,
0,
0,
#
#
#
#
adding constraints
to request the same
baseline rate in
all periods
) = 3;
Relative Risks. Since the exponential model is a proportional transition rate model one can use the rrisk option as part of the rate
command to request a table with relative risks. Calculation is done by
simply taking the anti-logarithm of the estimated model parameters.
Generalized Residuals. If the model is specified for only a single
transition, one can use the pres option to request generalized residuals.
For each episode in the input data, the rate is calculated with formula (3),
the (conditional) survivor function with formula (6), and the generalized
residuals by taking the anti-logarithm of the survivor function.
For example, adding the pres option to the rate command in Box
2 would result in the same output as shown in 6.17.2.1 for the basic
exponential model.
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations. For each point on the specified time
axis, the rate is calculated according to (3) using the corresponding time
period. Calculation of the survivor function is based on (6), always starting with time zero but integrating over all time periods up to the given
time point. The density function is calculated by multiplying the rate
and the survivor function.
6.17.2.3
period-specific effects
1
6.17.2.3 Period-specific Effects
The piecewise constant exponential model described in 6.17.2.2 is based
on the assumption that the effects of covariates do not change across time
periods. Of course, the covariates may be time-dependent, but not their
effects. An essentially different approach would allow also the effects of
covariates to vary across time periods. This model with period-specific
effects will be explained in the current section. We mention that this
is different from a standard exponential model with interaction effects
between covariates and time periods.
Implementation. The exponential model with time periods has model
number 16; the command to request model estimation is
rate (tp=..., additional parameters) = 16;
The tp parameter must be used to provide time periods defined by split
points on the time axis in the same way as described in 6.17.2.2 for
the standard piecewise constant exponential model. Given these time
periods, and using the same notation as in 6.17.2.2, the transition rate
from origin state j to destination state k is
n
o
(jk)
(jk)
rjk (t) = exp ᾱl + A(jk) αl
if t ∈ Il
(1)
(jk)
For each transition (j, k), ᾱl
is a constant coefficient associated with
(jk)
the lth time period. A(jk) is a (row) vector of covariates, and αl
is
an associated vector of coefficients, also associated with the lth time
period. TDA estimates a null model, containing only parameters for the
baseline rate, if specified without covariates. Covariates can be added to
the A-term of the model with the option
xa(j,k) = list of variables,
The standard exponential model with time periods described in 6.17.2.2
is, of course, a special case of the model defined in (1). In fact, estimating
(jk)
the latter model with constraints so that the αl
parameters are equal
across time periods, would give the same result as a standard exponential
model with time periods.
d06170203.tex
April 21, 1998
6.17.2.3
2
period-specific effects
Box 1 Part of command file rt3.cf
rate (
tp = 0 (12) 96,
xa(0,1) = COHO2,COHO3,W,
) = 16;
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. Using the same notation as
in 6.17.2.2, the conditional survivor function can now be written as
( m
)
X
G(t | s) = exp −
∆[s, t, l] exp(ᾱl + Aαl )
(2)
l=1
Using this expression, the log-likelihood is
`=
X
(ᾱl[ti ] + Ai αl[ti ] ) −
i∈E
m
XX
∆[si , ti , l] exp(ᾱl + Ai αl )
(3)
i∈N l=1
and the first and second derivatives are
X
X
∂`
=
δ[ti , l] +
−∆[si , ti , l] exp(ᾱl + Ai αl )
∂ ᾱl
∂`
∂αl,j
=
∂2`
∂ ᾱl ∂ ᾱl0
=
∂2`
∂αl,j ∂αl0 ,k
=
∂2`
∂ ᾱl ∂αl0 ,k
=
i∈E
i∈N
X
Ai,j δ[ti , l] +
i∈E
X
X
(4)
−∆[si , ti , l] exp(ᾱl + Ai αl ) Ai,j
i∈N
−∆[si , ti , l] exp(ᾱl + Ai αl ) δll0
i∈N
m
XX
−∆[si , ti , l] exp(ᾱl + Ai αl ) Ai,j Ai,k δll0
i∈N l=1
X
−∆[si , ti , l] exp(ᾱl + Ai αl ) Ai,k δll0
i∈N
Starting Values. Default starting values are calculated in the same way
as for the piecewise constant exponential model described in 6.17.2.2,
based on the results of a simple exponential null model.
Example 1 To illustrate estimation of an exponential model with periodspecific effects we use again command file rt1.cf. The rate command
6.17.2.3
period-specific effects
Box 2 Part of the output from command file rt3.cf
Model: Exponential with period-specific effects
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Idx SN Org Des MT P Variable
Coeff
Error
C/Error Signif
---------------------------------------------------------------------1 1
0
1 A 1 Period-1
-5.0470
0.2696
-18.7220 1.0000
2 1
0
1 A 2 Period-2
-4.5147
0.2121
-21.2865 1.0000
3 1
0
1 A 3 Period-3
-4.1413
0.1949
-21.2431 1.0000
4 1
0
1 A 4 Period-4
-5.1107
0.3160
-16.1711 1.0000
5 1
0
1 A 5 Period-5
-4.9954
0.3226
-15.4869 1.0000
6 1
0
1 A 6 Period-6
-4.5366
0.2934
-15.4603 1.0000
7 1
0
1 A 7 Period-7
-5.5069
0.5127
-10.7403 1.0000
8 1
0
1 A 8 Period-8
-6.0808
0.7231
-8.4090 1.0000
9 1
0
1 A 9 Period-9
-5.5356
0.1732
-31.9616 1.0000
10 1
0
1 A 1 COHO2
0.4474
0.3362
1.3308 0.8168
11 1
0
1 A 2 COHO2
0.5783
0.2509
2.3050 0.9788
12 1
0
1 A 3 COHO2
0.4240
0.2411
1.7581 0.9213
13 1
0
1 A 4 COHO2
0.2850
0.3616
0.7881 0.5694
14 1
0
1 A 5 COHO2
0.6357
0.4089
1.5546 0.8800
15 1
0
1 A 6 COHO2
-0.0546
0.4292
-0.1273 0.1013
16 1
0
1 A 7 COHO2
1.5141
0.6137
2.4671 0.9864
17 1
0
1 A 8 COHO2
-0.8695
1.0971
-0.7925 0.5719
18 1
0
1 A 9 COHO2
0.2966
0.3020
0.9822 0.6740
19 1
0
1 A 1 COHO3
0.7676
0.3161
2.4284 0.9848
20 1
0
1 A 2 COHO3
0.5335
0.2609
2.0446 0.9591
21 1
0
1 A 3 COHO3
0.2488
0.2590
0.9606 0.6633
22 1
0
1 A 4 COHO3
0.3212
0.3684
0.8717 0.6166
23 1
0
1 A 5 COHO3
0.8420
0.3946
2.1337 0.9671
24 1
0
1 A 6 COHO3
-0.7163
0.5605
-1.2779 0.7987
25 1
0
1 A 7 COHO3
1.1975
0.6762
1.7708 0.9234
26 1
0
1 A 8 COHO3
-8.4412
42.1447
-0.2003 0.1587
27 1
0
1 A 9 COHO3
1.2489
0.3227
3.8701 0.9999
28 1
0
1 A 1 W
-0.1421
0.2620
-0.5422 0.4123
29 1
0
1 A 2 W
0.2518
0.2054
1.2259 0.7798
30 1
0
1 A 3 W
0.3492
0.2060
1.6958 0.9101
31 1
0
1 A 4 W
1.2122
0.3188
3.8026 0.9999
32 1
0
1 A 5 W
0.6285
0.3255
1.9310 0.9465
33 1
0
1 A 6 W
0.7757
0.3749
2.0691 0.9615
34 1
0
1 A 7 W
-0.2436
0.5366
-0.4539 0.3501
35 1
0
1 A 8 W
1.7909
0.8673
2.0650 0.9611
36 1
0
1 A 9 W
0.5607
0.2580
2.1729 0.9702
Log likelihood (starting values): -2514.0201
Log likelihood (final estimates): -2411.9909
3
6.17.2.3
period-specific effects
4
is changed as shown in Box 1; the new command file is rt3.cf. For each
covariate one gets m parameters, with m the number of time periods.
So the total number of model parameters can easily become very large.
To reduce the number of parameters one can use constraints. As already mentioned, constraining the parameters to get equal values across
time periods would be equivalent to estimating the standard piecewise
constant exponential model described in 6.17.2.2. The example archive
contains another command file, rt3c.cf, that illustrates how to use constraints.
Generalized Residuals. If the model is specified for only a single
transition, one can use the pres option to request generalized residuals.
For each episode in the input data, the rate is calculated with formula (1),
the (conditional) survivor function with formula (2), and the generalized
residuals by taking the anti-logarithm of the survivor function.
Printing Estimated Rates. The prate option is not supported for
models with period-specific effects. This option might be added in a
future version of TDA.
6.17.3
1
parametric duration dependence
6.17.3 Parametric Duration Dependence
This section describes transition rate models with parametric duration
dependence. All models can be estimated with the rate command, see
6.17.1.4. All examples use the episode data set rrdat.1 that was discussed in 3.3.3.
6.17.3.1 Polynomial Rates
6.17.3.2 Gompertz-Makeham Models
6.17.3.3 Weibull Models
6.17.3.4 Sickle Models
6.17.3.5 Log-logistic Models
6.17.3.6 Log-normal Models
6.17.3.7 Generalized Gamma Models
6.17.3.8 Inverse Gaussian Models
d061703.tex
April 21, 1998
6.17.3.1
polynomial rates
1
6.17.3.1 Polynomial Rates
This section described two types of models where duration dependence
is specified by a polynomial. In a direct approach, one can specify the
transition rate by
r(t) = a + b1 t + b2 t2 + . . . bn tn
The corresponding survivor function is
b1
b2
bn n+1
G(t) = exp −at − t2 − t3 − . . . −
t
2
3
n+1
(1)
(2)
This model, sometimes called a generalized Rayleigh model, is described
shortly by Lawless [1982, p. 252]. An obvious drawback is that not all
possible parameter values will give sensible results.
With a small modification this problem can be avoided. If we take
the approach
r(t) = exp a + b1 t + b2 t2 + . . . bn tn
(3)
estimated rates can only take positive values. The corresponding conditional survivor function is
Z t
G(t | s) = exp − exp(a)
exp(b1 τ + b2 τ 2 + . . . bn τ n ) dτ
(4)
s
Unfortunately, there is no analytic solution; the integral must be evaluated numerically.
Implementation. Both types of models can be estimated with TDA.
They are called models with polynomial rates, type I and II, respectively.
The commands to request model estimation are
rate (parameter) = 4;
for the type I model, and
rate (parameter) = 5;
d06170301.tex
April 21, 1998
6.17.3.1
polynomial rates
2
for the type II model. To specify the degree of the polynomial one can
use the parameter
deg = degree of polynomial,
The default value is deg=0, both models are then identical to a simple
exponential model. For type II model, one can also select a method for
numerical integration with the niset command, see 5.4.1. Default is
method 1 (QNG algorithm) with relative accuracy 0.0001.
1. The type I model provides for the possibility to link covariates to the
parameter a via an exponential link function, the other parameters are
estimated directly. The degree of the polynomial can be defined by the
user, so there is the possibility of testing how many degrees are needed
for a reasonably model fit. One gets the following formulation for the
transition rate from origin state j to destination state k.
rjk (t)
= ajk +
n
X
(jk) l
bl
t
(5)
l=1
ajk
=
n
o
exp A(jk) α(jk)
2. In the type II model covariates can be linked to the parameter a by
a linear link function, the other parameters are again estimated directly.
Also the degree of the polynomial can be defined by the user. One then
gets the following formulation for the transition rate from origin state j
to destination state k.
n
n
o
X
(jk)
(6)
rjk (t) = exp ajk +
b l tl
l=1
ajk
(jk) (jk)
= A
α
In both cases, A(jk) is a (row) vector of covariates, and α(jk) is the
associated vector of coefficients. The first component of A(jk) is supplied
automatically by TDA as a constant equal to one. The model parameters
to be estimated are the vectors
α(jk)
and
(jk)
b(jk) = (b1
0
, . . . , b(jk)
n )
In both cases the null model is defined as a simple exponential model,
that is a model with polynomial degree zero. We mention that the type I
6.17.3.1
3
polynomial rates
model is not a proportional transition rate model, and the rrisk option
is not available for this model.
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify notation we omit
indices for transitions.
I. Using the conditional survivor function
n
n
X
G(t | s) = exp a (s − t) +
o
bl
sl+1 − tl+1
l+1
l=1
(7)
the log-likelihood for the type I model can be written as
`=
X
i∈E
n
n
n
o Xn
X
X
o
bl
sl+1
− tl+1
log a +
bl tli +
a (si − ti ) +
i
i
l+1
i∈N
l=1
l=1
The first and second derivatives with respect to the α and b coefficients
are
X
X
∂`
a
P
=
A
+
a (si − ti ) Ai,ja
i,j
a
∂αja
a + l bl tli
i∈E
i∈N
jb
∂`
∂bjb
=
X
i∈E
a+
1
P
tjb
l i
l b l ti
+
X
si
ti
(si − ti )
jb + 1
i∈N
tji b
n
o
X
∂2`
a
a
P
P
=
1
−
Ai,ja Ai,ka +
∂αja ∂αka
a + l bl tli
a + l bl tli
i∈E
X
a (si − ti ) Ai,ja Ai,ka
i∈N
∂2`
∂bjb ∂bkb
=
∂2`
∂αja ∂bkb
=
X
i∈E
a+
−1
P
X
i∈E
a+
l
l bl t i
−a
P
l
l b l ti
tji b tki b
Ai,ja tki b
6.17.3.1
polynomial rates
4
II. For the type II model the integral needed in the formulation of (4)
must be evaluated numerically. Assuming that a method for the evaluation of
Z t
n
o
nX
bl τ l dτ
I(s, t) =
exp
s
∂In (s, t)
∂bj
∂ 2 In (si , ti )
∂bj ∂bk
Z
l=1
t
=
exp
n
nX
s
Z
=
o
bl τ l τ j dτ
l=1
t
exp
n
nX
s
o
bl τ l τ j+k dτ
l=1
is available, the log-likelihood for the type II model can be written as
`=
n
o X
Xn
X
bl tli −
a+
exp(a) I(si , ti )
i∈E
i∈N
l=1
and the first and second derivatives are
X
X
∂`
=
Ai,ja −
exp(a) I(si , ti ) Ai,ja
∂αja
i∈E
∂`
∂bjb
=
∂2`
∂αja ∂αka
=
∂2`
∂bjb ∂bkb
=
∂2`
∂αja ∂bkb
=
X
i∈E
X
i∈N
tji b −
X
i∈N
exp(a)
∂I(si , ti )
∂bjb
− exp(a) I(si , si ) Ai,ja Ai,ka
i∈N
X
− exp(a)
∂ 2 I(si , ti )
∂bjb ∂bkb
− exp(a)
∂I(si , ti )
Ai,ja
∂bkb
i∈N
X
i∈N
To evaluate the integrals, one can use the algorithms described in the
chapter about numerical integration, see 5.4.
Starting Values. Both types of the polynomial model contain the simple exponential model as a special case. Therefore it is possible to use the
parameter estimates of correspondingly defined exponential null models
6.17.3.1
polynomial rates
5
Box 1 Illustration of polynomial model, type I (rt4.cf)
Model: Polynomial (I) with degree 2
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Convergence reached in 7 iterations.
Number of function evaluations: 10 (10,10)
Maximum of log likelihood: -2454.04
Norm of final gradient vector: 0.00705957
Last absolute change of function value: 9.30148e-14
Last relative change in parameters: 2.72933e-06
Numerical problems: 1
Idx SN Org Des MT Variable
Coeff
Error
C/Error
-----------------------------------------------------------------1 1
0
1 A Constant -4.4325e+00
1.0308e-01 -4.2999e+01
2 1
0
1 A COHO2
2.4523e-01
8.9275e-02
2.7468e+00
3 1
0
1 A COHO3
3.4108e-01
9.9069e-02
3.4429e+00
4 1
0
1 A W
3.1746e-01
7.5806e-02
4.1878e+00
5 1
0
1 B Beta-1
-7.6085e-05
1.5607e-05 -4.8750e+00
6 1
0
1 B Beta-2
1.4274e-07
4.3506e-08
3.2809e+00
Log likelihood (starting values):
Log likelihood (final estimates):
-2.5140e+03
-2.4540e+03
as starting values. The default starting values used by TDA are calculated this way, i.e. the constant terms in the model parameters α(jk)
are initially set as in a simple exponential model, and all other model
parameters are set to zero.
In most cases this is sufficient for low degree polynomials. If the
degree of the polynomial is greater than three, the default starting values
can result in convergence difficulties. In many cases this can be avoided
if a series of models with successively higher degrees of the polynomial is
estimated, and for each model the parameter estimates of the previous
one are used as starting values.
Another source of convergence difficulties arises because of numerical
overflow if the degree of the polynomials becomes large since TDA does
not automatically scale variables of different magnitude.
Example 1 To illustrate the type I model, we use command file rt4.cf
(not shown). Part of the standard output is shown in Box 1, the first line
shows the rate command used for model estimation. In this example,
6.17.3.1
polynomial rates
6
Box 2 Illustration of polynomial model, type II (rt5.cf)
Model: Polynomial (II) with degree 2
Numerical integration with Method 1 (QNG).
Relative error: 1.00000e-04
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Convergence reached in 7 iterations.
Number of function evaluations: 8 (8,8)
Maximum of log likelihood: -2454.54
Norm of final gradient vector: 5.24649e-07
Last absolute change of function value: 3.25666e-15
Last relative change in parameters: 3.81352e-06
Idx SN Org Des MT Variable
Coeff
Error
C/Error
-----------------------------------------------------------------1 1
0
1 A Constant -4.5911e+00
1.1656e-01 -3.9389e+01
2 1
0
1 A COHO2
3.8307e-01
1.1322e-01
3.3834e+00
3 1
0
1 A COHO3
4.5832e-01
1.1780e-01
3.8908e+00
4 1
0
1 A W
4.1726e-01
9.4840e-02
4.3996e+00
5 1
0
1 B Beta-1
-3.6942e-03
2.1718e-03 -1.7010e+00
6 1
0
1 B Beta-2
-7.4989e-06
9.6591e-06 -7.7635e-01
Log likelihood (starting values):
Log likelihood (final estimates):
-2.5140e+03
-2.4545e+03
we have selected a second-degree polynomial. Box 2 shows part of the
output for an analogously specified type II model. The command file is
rt5.cf.
Generalized Residuals. If one of the polynomial models is estimated
for only a single transition, one can use the pres option to request a
table with generalized residuals. For a type II model, the calculation
uses again numerical integration.
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations.
6.17.3.2
1
Gompertz-Makeham Models
6.17.3.2 Gompertz-Makeham Models
This section describes the Gompertz-Makeham model. The transition
rate is given by the expression
r(t) = a + b exp(c t)
a, b ≥ 0
(1)
If a = 0, one gets the Gompertz model without a Makeham term. If b = 0,
the model reduces to the simple exponential model. The corresponding
survivor and density functions are
b
f (t) = exp −at − [exp(c t) − 1] [a + b exp(c t)]
(2)
c
b
G(t) = exp −at − [exp(c t) − 1]
(3)
c
In the case of c = 0, it is assumed that these expressions reduce to the
density and survivor function for a simple exponential model. Figure 1
shows some graphs of the transition rate function for a = 0, b = 1, and
some different values of c.
Implementation. The Gompertz-Makeham model has three parameters to include covariates. TDA uses exponential link functions with the
A- and B-term of the model, and a linear link function for the C-term.
The model formulation for the transition rate from origin state j to
destination state k is
rjk (t)
ajk
= ajk + bjk exp(cjk t)
n
o
= exp A(jk) α(jk)
bjk
=
cjk
= C (jk) γ (jk)
(4)
n
o
exp B (jk) β (jk)
It is assumed that the first component of each of the covariate (row)
vectors A(jk) , B (jk) , and C (jk) , is a constant equal to one. The associated
coefficient vectors, α(jk) , β (jk) , and γ(jk) , are the model parameters to
be estimated. Both, the Gompertz and the Gompertz-Makeham model
d06170302.tex
April 21, 1998
6.17.3.2
2
Gompertz-Makeham Models
3
c = 0.2
2
c = 0.0
1
c = -0.5
0
0
1
2
3
4
5
Figure 1 Gompertz transition rates. Plot created with command file plot-gm.cf.
have model number 6. If no covariates are linked to the A-term of the
model, a simple Gompertz model is estimated. If at least one variable
is linked to the A-term, the estimated model is of Gompertz-Makeham
type. To estimate a Gompertz-Makeham model with the A-term a single
constant, one can use the parameter ni=1 to specify a single constant
for the A term.
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify notation, we omit
indices for transitions and use the abbreviations a, b, and c, as defined
in (4). Providing for the possibility of episode splitting by using the
conditional survivor function
b
G(t | s) = exp a (s − t) + [exp(c s) − exp(c t)]
(5)
c
one gets the log-likelihood
`=
X
log{a + b exp(c ti )} +
i∈E
X
i∈N
a(si − ti ) +
b
[ exp(c si ) − exp(c ti ) ]
c
The first and second derivatives with respect to the α, β, and γ coefficients are
X
X
∂`
a
=
Ai,ja +
a (si − ti ) Ai,ja
∂αja
a + b exp(c ti )
i∈E
i∈N
6.17.3.2
∂`
∂βjb
=
∂`
∂γjc
=
3
Gompertz-Makeham Models
X
i∈E
X b
b exp(c ti )
Bi,jb +
{exp(c si ) − exp(c ti )}
a + b exp(c ti )
c
i∈N
b exp(c ti )
ti Ci,jb +
a + b exp(c ti )
i∈E
X b 1
1
exp(c si ) si −
− exp(c ti ) ti −
c
c
c
X
i∈N
∂2`
∂αja ∂αka
=
X
i∈E
X
a
a + b exp(c ti )
1−
a
a + b exp(c ti )
b exp(c ti )
a + b exp(c ti )
Ai,ja Ai,ka +
a (si − ti ) Ai,ja Ai,ka
i∈N
∂2`
∂βjb ∂βkb
=
X
i∈E
b exp(c ti )
a + b exp(c ti )
1−
Bi,jb Bi,kb +
X b
{exp(c si ) − exp(c ti )} Bi,jb Bi,kb
c
i∈N
∂2`
∂γjb ∂γkb
=
∂2`
∂αja ∂βkb
=
∂2`
∂αja ∂γkb
=
∂2`
∂βja ∂γkb
=
b exp(c ti )
b exp(c ti )
1−
t2 Ci,jb Ci,kb +
a + b exp(c ti )
a + b exp(c ti ) i
i∈E
X b
1
1
exp(c si ) (si − )2 + 2 −
c
c
c
i∈N
1
1
Ci,jb Ci,kb
exp(c ti ) (ti − )2 + 2
c
c
X
X
−a b exp(c ti )
i∈E
[a + b exp(c ti )]
X
−a b exp(c ti )
i∈E
[a + b exp(c ti )]
2
Ai,ja Bi,kb
2 ti
Ai,ja Ci,kb
b exp(c ti )
b exp(c ti )
1−
ti Bi,jb Ci,kc +
a + b exp(c ti )
a + b exp(c ti )
i∈E
X b
1
1
exp(c si ) si −
− exp(c ti ) ti −
Bi,jb Ci,kc
c
c
c
X
i∈N
6.17.3.2
Gompertz-Makeham Models
4
Box 1 Output from command file rt6.cf
Model: Gompertz-Makeham
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 8
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 6 iterations.
Number of function evaluations: 8 (8,8)
Maximum of log likelihood: -2453.38
Norm of final gradient vector: 1.83153e-06
Last absolute change of function value: 2.81327e-12
Last relative change in parameters: 8.91402e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 B Constant
-4.5114
0.1198
-37.6503 1.0000
2 1
0
1 B COHO2
0.4595
0.1546
2.9716 0.9970
3 1
0
1 B COHO3
0.4147
0.1657
2.5025 0.9877
4 1
0
1 B W
0.2860
0.1280
2.2351 0.9746
5 1
0
1 C Constant
-0.0060
0.0013
-4.6239 1.0000
6 1
0
1 C COHO2
-0.0016
0.0022
-0.7275 0.5331
7 1
0
1 C COHO3
0.0011
0.0028
0.3986 0.3098
8 1
0
1 C W
0.0026
0.0018
1.4299 0.8472
Log likelihood (starting values): -2557.3645
Log likelihood (final estimates): -2453.3828
Starting Values. In most cases, good initial estimates are required
to estimate Gompertz-Makeham models. It is difficult, however, to determine such estimates since it depends on whether a positive or negative time dependence should be assumed. In particular the addition of
a Gompertz A-term in the model specification can give rise to conver-
6.17.3.2
Gompertz-Makeham Models
5
gence difficulties. Default starting values used with TDA are calculated
by using the constant rate of a correspondingly defined exponential null
model. The constant coefficients in the A- and the B-terms of the model
are taken to be the logarithm of this rate, the constant coefficient in the
C-term is taken to be minus this rate. The idea is to assume negative
time dependence as the “normal case”, and to start with some estimate
of the mean duration. However, the starting values calculated this way
are not very good initial estimates. So it may be necessary to try other
ones.
Example 1 As is seen from Figure 1, the Gompertz-Makeham model
is not appropriate for our main example data. As a consequence we find
severe convergence difficulties when trying a full Gompertz-Makeham
specification. No problems arise, however, if we try a model without the
Makeham term. To illustrate, we use command file rt6.cf, identical
with rt1.cf except for the model number that has been changed into 6.
Box 1 shows the resulting parameter estimates.
Generalized Residuals. For the Gompertz-Makeham models, one can
use the pres option to request a table with generalized residuals.
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations.
6.17.3.3
weibull models
1
6.17.3.3 Weibull Models
This section describes the Weibull model. In the single transition case
it is derived by assuming the duration variable T to follow a Weibull
distribution. Density and survivor function, and the transition rate, are
given respectively by
f (t) = b ab tb−1 exp −(at)b
a, b > 0
(1)
G(t) = exp −(at)b
(2)
r(t)
= b ab tb−1
(3)
Figure 1 shows graphs of the transition rate for a = 1 and some different
values of b.
Implementation. The Weibull model has two parameters, so one has
two possibilities to include covariates. Using standard exponential link
functions, the model formulation for the transition rate from origin state
j to destination state k is
b
ajk
= bjk ajkjk tbjk −1
n
o
= exp A(jk) α(jk)
bjk
=
rjk (t)
(4)
n
o
exp B (jk) β (jk)
It is assumed that the first component of each of the covariate (row)
vectors A(jk) and B (jk) is a constant equal to one. The associated coefficient vectors, α(jk) and β (jk) , are the model parameters to be estimated.
The Weibull model has model number 7.
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify the notation we
omit indices for transitions and use the abbreviations a and b as defined
in (4). Using the conditional survivor function
G(t | s) =
d06170303.tex
exp(−(at)b )
exp(−(as)b )
(5)
April 21, 1998
6.17.3.3
2
weibull models
3
b = 1.5
2
b = 1.2
b = 0.0
1
b = 0.7
b = 0.5
0
0
1
2
3
Figure 1 Weibull transition rates. Plot created with command
file plot-wei.cf.
the log-likelihood is
X
X
`=
log b ab tib−1 +
(asi )b − (ati )b
i∈E
(6)
i∈N
and the first and second derivatives with respect to the α and β coefficients are
X
X ∂`
=
b Ai,ja +
b (asi )b − (ati )b Ai,ja
∂αja
∂`
=
∂βjb
i∈E
i∈N
X
{1 + b log(ati )} Bi,jb +
i∈E
X b log(asi )(asi )b − log(ati )(ati )b Bi,jb
i∈N
X ∂2`
=
b2 (asi )b − (ati )b Ai,ja Ai,ka
∂αja ∂αka
i∈N
2
X
∂ `
=
b log(ati )Bi,jb Bi,kb +
∂βjb ∂βkb
i∈E
X b log(asi )(asi )b [1 + b log(asi )] −
i∈N
log(ati )(ati )b [1 + b log(ati )] Bi,jb Bi,kb
6.17.3.3
weibull models
3
Box 1 Output from command file rt7.cf
Model: Weibull
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 5
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -2473.54
Norm of final gradient vector: 4.34528e-08
Last absolute change of function value: 5.00438e-12
Last relative change in parameters: 4.56362e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Constant
-5.0055
0.0906
-55.2452 1.0000
2 1
0
1 A COHO2
0.5339
0.1201
4.4448 1.0000
3 1
0
1 A COHO3
0.6671
0.1235
5.4003 1.0000
4 1
0
1 A W
0.5191
0.1012
5.1274 1.0000
5 1
0
1 B Constant
-0.0700
0.0366
-1.9119 0.9441
Log likelihood (starting values): -2514.0201
Log likelihood (final estimates): -2473.5432
X
∂2`
=
b Ai,ja Bi,kb +
∂αja ∂βkb
i∈E
X b (asi )b [1 + b log(asi )] −
i∈N
(ati )b [1 + b log(ati )] Ai,ja Bi,kb
Starting Values. Estimation of Weibull models is relatively robust
with respect to starting values. Therefore initial estimates used by TDA
are taken to be the constant rates of an accordingly defined exponential
6.17.3.3
weibull models
4
null model. All model coefficients, without the constant coefficient of the
A-term, are set to zero, implying a unit shape parameter of the duration
distribution, i.e. an exponential model.
Example 1 Although the Weibull model does not seem appropriate for
our main example data, just to illustrate its estimation we use command
file rt7.cf. Estimation results are shown in Box 1.
Generalized Residuals. For the Weibull model, one can use the pres
option to request a table with generalized residuals.
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations. Note that, for t = 0, the rate and
density is always set to zero, and the survivor function is set to 1; this
might not be correct.
6.17.3.4
1
sickle models
6.17.3.4 Sickle Models
This section describes the Sickle model proposed by Diekmann and Mitter ([1983], [1984]). Formulated first for the single transition case, it
starts with the assumption of a transition rate given by the expression
t
r(t) = a t exp −
a, b > 0
(1)
b
The corresponding survivor and density functions are
t
t
f (t) = exp −a b [ b − (t + b) exp(− )] a t exp −
b
b
t
G(t) = exp −a b [ b − (t + b) exp(− )]
b
(2)
(3)
As is easily seen, this distribution is defective. The survivor function goes
not to zero if t → ∞, instead one finds that
lim G(t) = exp(−a b2 )
t→∞
(4)
Figure 1 shows graphs of the rate function for a = 1 and some values of
b. It looks like a sickle, and this has given the name for the model.
Implementation. The Sickle model has two parameters, so there are
two possibilities to include covariates. Because both of the parameters
must be positive, TDA uses standard exponential link functions. One
then gets the following model formulation for the transition rate from
origin state j to destination state k.
t
(5)
rjk (t) = ajk t exp −
bjk
n
o
ajk = exp A(jk) α(jk)
bjk
=
n
o
exp B (jk) β (jk)
It is assumed that the first component of each of the covariate (row)
vectors A(jk) and B (jk) is a constant equal to one. The associated coefficient vectors, α(jk) and β (jk) , are the model parameters to be estimated.
The sickle model has model number 8.
d06170304.tex
April 21, 1998
6.17.3.4
2
sickle models
0.8
0.4
b = 0.5
b = 2.0
b = 1.0
0
0
1
2
3
4
5
6
7
8
9
10
11
12
Figure 1 Sickle transition rates. Plot created with command
file plot-sic.cf.
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify the notation we
omit indices for transitions and use the abbreviations a and b as defined
in (5). Providing for the possibility of episode splitting by using the
conditional survivor function
s t
G(t | s) = exp a b (t + b) exp(− ) − (s + b) exp −
b
b
the log-likelihood is
` =
X
i∈E
ti
+
b
s X
ti
i
a b (ti + b) exp −
− (si + b) exp −
b
b
log(a ti ) −
i∈N
and the first and second derivatives with respect to the α and β coefficients are
X
∂`
=
Ai,ja +
∂αja
i∈E
X
ti
si
a b (ti + b) exp(− ) − (si + b) exp(− ) Ai,ja
b
b
i∈N
6.17.3.4
sickle models
3
t2i
ti
2(ti + b) +
exp(− )−
b
b
i∈E
i∈N
s2i
si
2(si + b) +
exp(− ) Bi,jb
b
b
X
si
ti
=
a b (ti + b) exp(− ) − (si + b) exp(− ) Ai,ja Ai,ka
b
b
X ti
X
∂`
=
Bi,jb +
ab
∂βjb
b
∂2`
∂αja ∂αka
i∈N
2
X ti
∂ `
=
− Bi,jb Bi,kb +
∂βjb ∂βkb
b
i∈E
X
ti
ti t2i
exp(− )−
a b 4(ti + b) + (2 + )
b b
b
i∈N
si s2
si
4(si + b) + (2 + ) i exp(− ) Bi,jb Bi,kb
b b
b
2
2
X
∂ `
ti
t
=
a b 2(ti + b) + i exp(− )−
∂αja ∂βkb
b
b
i∈N
s2i
si
2(si + b) +
exp(− ) Ai,ja Bi,kb
b
b
Starting Values. The choice of starting values for the Sickle model
seems not to be critical. In most applications it seems to be sufficient to
set initially all coefficients to zero. Somewhat more efficient calculations
are possible if one chooses as initial estimates α0 = r2 and β0 = − log(r)
with r the constant rate of a correspondingly defined exponential null
model. The starting values used by TDA are defined this way.
Example 1 To illustrate estimation of the Sickle model, we use our
main example data; the command file is rt8.cf, identical with rt1.cf,
only the model number has been changed into 8. Estimation results are
shown in Box 1. Note that the program has given an information about
numerical problems. Number 4 signals that the Hessian matrix was at
least for one time not positive definite during the iterations to maximize
the likelihood. However, in the final iteration steps it was positive definite, and so there is no real problem. For more details about the iteration
process one can request a protocol file with the pprot parameter.
6.17.3.4
sickle models
4
Box 1 Output from command file rt8.cf
Model: Sickle
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 5
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 7 iterations.
Number of function evaluations: 10 (10,10)
Maximum of log likelihood: -2466.1
Norm of final gradient vector: 4.84093e-06
Last absolute change of function value: 1.11023e-09
Last relative change in parameters: 0.00044869
Numerical problems.
4 : Hessian not positive definite.
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Constant
-7.0645
0.1140
-61.9618 1.0000
2 1
0
1 A COHO2
0.3926
0.1123
3.4947 0.9995
3 1
0
1 A COHO3
0.4363
0.1162
3.7556 0.9998
4 1
0
1 A W
0.4232
0.0944
4.4808 1.0000
5 1
0
1 B Constant
3.7046
0.0517
71.6511 1.0000
Log likelihood (starting values): -2897.7621
Log likelihood (final estimates): -2466.1005
Generalized Residuals. For the sickle model, one can use the pres
option to request a table with generalized residuals.
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations.
6.17.3.5
log-logistic models
1
6.17.3.5 Log-logistic Models
Another model with time-varying transition rates is the log-logistic. This
section describes the standard log-logistic model and an extension.
I. In the single transition case the standard log-logistic model is based
on the assumption that the duration variable follows a log-logistic distribution. The density, survivor, and rate functions for this distribution
are
f (t) =
b ab tb−1
[1 + (at)b ]2
G(t)
=
1
1 + (at)b
(2)
r(t)
=
b ab tb−1
1 + (at)b
(3)
a, b > 0
(1)
Figure 1 shows graphs of the rate function for a = 1 and some values of
b. The time tmax when the rate reaches its maximum, rmax , is given by
tmax =
1
1
(b − 1) b
a
1
rmax = a (b − 1)1− b
(4)
II. A sometimes useful extension of the standard log-logistic model was
proposed by Brüderl [1991]; see also Brüderl and Diekmann [1995]. The
idea is to include a third parameter to allow for varying levels of the
transition rate. The model definition is
r(t) = c
b (a t)b−1
1 + (at)b
a, b, c > 0
(5)
The associated density and survivor functions are
f (t) = c
G(t) =
d06170305.tex
b (a t)b−1
c
[1 + (at)b ] a +1
1
c
[1 + (at)b ] a
April 21, 1998
6.17.3.5
2
log-logistic models
2
1
2.0
1.5
1.0
0.7
0.4
0
0
1
2
3
Figure 1 Log-logistic transition rates. Plot created with
command file plot-ll.cf.
Implementation. Both versions of the log-logistic transition rate model
can be estimated, the model numbers are 9 and 10, respectively.
I. The standard log-logistic model has two parameters, so there are two
possibilities to include covariates. TDA uses exponential link functions,
so one gets the following model formulation for the transition rate from
origin state j to destination state k.
b
rjk (t) =
ajk
bjk ajkjk tbjk −1
1 + (ajk t) bjk
n
o
= exp A(jk) α(jk)
n
o
bjk = exp B (jk) β (jk)
II. The extended log-logistic model has three parameters, providing three
possibilities to include covariates. Using again standard exponential link
functions, one gets the following model formulation for the transition
rate from origin state j to destination state k.
bjk (ajk t)bjk −1
1 + (ajk t)bjk
n
o
= exp A(jk) α(jk)
rjk (t) = cjk
ajk
6.17.3.5
3
log-logistic models
n
o
bjk = exp B (jk) β (jk)
n
o
cjk = exp C (jk) γ (jk)
Again it is assumed that the first component of each of the covariate
vectors A(jk) , B (jk) , and C (jk) , is a constant equal to one. The coefficients
α(jk) , β (jk) , and γ (jk) , are the model parameters to be estimated.
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify the notation we
omit indices for transitions and use some already defined abbreviations.
I. The conditional survivor function for the standard log-logistic model
may be written as
G(t | s) =
1 + (as)b
1 + (at)b
(6)
Using these conditional survivor functions, the log-likelihood is
(
`=
X
i∈E
log
b ab tib−1
1 + (ati )b
)
+
X
i∈N
log
1 + (asi )b
1 + (ati )b
(7)
with first and second derivatives
X
∂`
=
b
∂αja
i∈E
(ati )b
Ai,ja +
1 + (ati )b
X (asi )b
(ati )b
b
−
Ai,ja
1 + (asi )b
1 + (ati )b
1−
i∈N
X
∂`
log(ati )
=
1+b
Bi,jb +
∂βjb
1 + (ati )b
i∈E
X (asi )b
(ati )b
b log(asi )
−
log(at
)
Bi,jb
i
1 + (asi )b
1 + (ati )b
i∈N
6.17.3.5
log-logistic models
4
b
X
∂2`
(ati )b
2 (ati )
=
b
− 1 Ai,ja Ai,ka +
∂αja ∂αka
1 + (ati )b 1 + (ati )b
i∈E
X (asi )b (asi )b
2
b
1−
−
1 + (asi )b
1 + (asi )b
i∈N
(ati )b
(ati )b
1−
Ai,ja Ai,ka
1 + (ati )b
1 + (ati )b
X log(ati ) (ati )b
∂2`
1 − b log(ati )
Bi,jb Bi,kb +
=
b
∂βjb ∂βkb
1 + (ati )b
1 + (ati )b
i∈E
X log(asi )
(asi )b
1
+
b
−
b log(asi )
1 + (asi )b
1 + (asi )b
i∈N
(ati )b
log(ati )
log(ati )
1
+
b
Bi,jb Bi,kb
1 + (ati )b
1 + (ati )b
X (ati )b
b log(ati )
∂2`
=
b 1−
1
+
Ai,ja Bi,kb +
∂αja ∂βkb
1 + (ati )b
1 + (ati )b
i∈E
X (asi )b b log(asi )
1+
−
b
1 + (asi )b
1 + (asi )b
i∈N
(ati )b
b log(ati )
1
+
Ai,ja Bi,kb
1 + (ati )b
1 + (ati )b
Starting Values. Starting values for estimating the standard log-logistic
model are not critical, and in most situations it is sufficient to set the
shape parameter b = 1 and to use for the parameter a the constant rate
of an accordingly defined exponential null model. TDA uses this method
to calculated default starting values.
For the extended models the choice of starting values is critical. In
many cases the best way to reach convergence is to use as starting values the parameter estimates of a standard log-logistic model, and to set
the C-term coefficients to zero. Because of the overhead involved in this
method it is not practical as a default. Default starting values are therefore calculated by TDA in another way: the A- and B-term coefficients
of the model are set to zero, and the C-term coefficients are defined by
the constant rate of an accordingly defined exponential null model. For
many situations this method seems to provide sufficient initial parameter
estimates.
6.17.3.5
log-logistic models
5
Box 1 Output from command file rt9.cf
Model: Log-logistic (I)
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 5
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 5 iterations.
Number of function evaluations: 8 (8,8)
Maximum of log likelihood: -2442.3
Norm of final gradient vector: 3.28158e-05
Last absolute change of function value: 6.67562e-09
Last relative change in parameters: 0.000446382
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Constant
-4.3546
0.0998
-43.6183 1.0000
2 1
0
1 A COHO2
0.4792
0.1257
3.8138 0.9999
3 1
0
1 A COHO3
0.5927
0.1295
4.5752 1.0000
4 1
0
1 A W
0.3989
0.1065
3.7463 0.9998
5 1
0
1 B Constant
0.3307
0.0385
8.5823 1.0000
Log likelihood (starting values): -2521.7850
Log likelihood (final estimates): -2442.3017
Example 1 To illustrate estimation of the standard (type I) log-logistic
model, we use command file rt9.cf. Parameter estimates are shown in
Box 1.
Generalized Residuals. For both types of the log-logistic model, one
can use the pres option to request a table with generalized residuals.
6.17.3.5
log-logistic models
6
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations. Tor t = 0, the rate and density is
always set to zero, and the survivor function is set to 1; this might not be
correct. The rrisk option cannot be used with the standard log-logistic
models. For the extended (type II) log-logistic model, the rrisk option
shows relative risks for covariates linked to the C-term of the model.
6.17.3.6
log-normal models
1
6.17.3.6 Log-normal Models
This section describes two versions of the log-normal model, a standard
log-normal model and a model with an additional shift-parameter. The
models correspond to the two-parameter and the three-parameter lognormal distributions as described, for instance, by Aitchison and Brown
[1957]. (Descriptions of log-normal rate models are given by Lawless
[1982], p. 313, Lancaster [1990], p. 47.)
In the single transition case, the standard (two-parameter) log-normal
model is derived by assuming that the logarithm of the duration variable, T , follows a normal distribution or, equivalently, that T follows a
log-normal distribution with density function
1
log(t) − a
f (t) =
φ
b>0
(1)
bt
b
φ and Φ are used, respectively, to denote the standard normal density
and distribution function:
1
τ2
φ(τ ) = √ exp(− )
2
2π
Z t
Φ(t) =
φ (τ ) d τ
0
The survivor function corresponding to (1) is
log(t) − a
G(t) = 1 − Φ
b
(2)
and the transition rate can be written as
r(t) =
1
φ (zt )
b t 1 − Φ (zt )
where
zt =
log(t) − a
b
Figure 1 shows graphs of the rate function for a = 0 and some different
values of b. As is seen, the graphs are very similar for the log-normal and
the log-logistic model, provided that b > 1 in the latter case.
A simple, sometimes useful extension of the standard log-normal
model is derived by adding a shift parameter c > 0. The assumption,
d06170306.tex
April 21, 1998
6.17.3.6
2
log-normal models
1.5
1
b = 0.7
0.5
b = 1.0
b = 1.3
0
0
1
2
3
4
5
6
7
Figure 1 Log-normal transition rates. Plot created with
command file plot-ln.cf.
then, is that not T , but T −c follows a log-normal distribution. Of course,
this assumption implies that no episode ends before c, meaning that c
can be thought of as a basic waiting time until the risk of leaving the
origin state becomes positive. The resulting model formulation is very
similar to the standard log-normal model. The density, survivor and rate
functions have the same expressions as given above for the standard case,
one only has to substitute t by t − c. The result is a simple shift of the
functions to the right by an amount of c.
Implementation. The standard log-normal model has two parameters,
so there are two possibilities to include covariates. Following Lawless
([1982], p. 313), TDA uses a linear link function for a, but additionally
provides the possibility to link covariates to the dispersion parameter,
b, via an exponential link function. So one gets the following model
formulation for the transition rate from origin state j to destination
state k.
(jk)
φ zt
rjk (t) =
bjk t 1 − Φ z (jk)
t
1
(jk)
zt
=
log(t) − ajk
bjk
ajk = A(jk) α(jk)
6.17.3.6
log-normal models
3
n
o
bjk = exp B (jk) β (jk)
The extended three-parameter model is defined analogously by
(jk)
φ zt
1
rjk (t) =
bjk (t − cjk ) 1 − Φ z (jk)
(3)
t
(jk)
zt
=
log(t − cjk ) − ajk
bjk
ajk = A(jk) α(jk)
n
o
bjk = exp B (jk) β (jk)
n
o
cjk = exp C (jk) γ (jk)
In both cases, it is assumed that the first component of each of the
covariate (row) vectors A(jk) , B (jk) , and C (jk) in the extended model, is
a constant equal to one. The associated coefficient vectors α(jk) , β (jk) ,
and γ (jk) , are the model parameters to be estimated.
Both versions of the log-normal model have the same model numbers,
12. The two versions are distinguished by the definition of a C-term of
the model. If this term is defined, TDA estimates an extended model,
otherwise a standard log-normal model. To request estimation of a lognormal model of type II with a single constant in the C-term, one can
use the parameter ni=1 to link a single constant to the C term.
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. Since the standard model is
a special case of the extended model, when c = 0, we only give formulas
for the extended model. To simplify the notation we omit indices for
transitions and use the abbreviations a, b, c, and zt , as defined in (3).
Furthermore, we define
Q(z) = 1 − Φ(z)
and V (z) =
φ(z)
Q(z)
(4)
Providing for the possibility of episode splitting by using the conditional
survivor function, the log-likelihood is
X
X
V (zti )
Q(zti )
`=
log
+
log
(5)
b (ti − c)
Q(zsi )
i∈E
i∈N
6.17.3.6
log-normal models
4
I. The first and second derivatives for the A and B-terms of the model
may then be written as
X1
X1
∂`
{zti − V (zti } Ai,ja +
{V (zti ) − V (zsi )} Ai,ja
=
∂αja
b
b
i∈N
i∈E
∂`
=
∂βjb
X
{zti [zti − V (zti )] − 1} Bi,jb +
i∈E
X
{zti V (zti ) − zsi V (zsi )} Bi,jb
i∈N
X 1
∂2`
=
{V (zti ) [V (zti ) − zti ] − 1} Ai,ja Ai,ka +
∂αja ∂αka
b2
i∈E
X 1
{V (zti ) [zti − V (zti )] − V (zsi ) [zsi − V (zsi )]} Ai,ja Ai,ka
b2
i∈N
X
∂2`
=
zti V (zti ) [1 + zti (V (zti ) − zti )] − 2zt2i Bi,jb Bi,kb +
∂βjb ∂βkb
i∈E
X
{zti V (zti ) [zti (zti − V (zti )) − 1] −
i∈N
zsi V (zsi ) [zsi (zsi − V (zsi )) − 1]} Bi,jb Bi,kb
X1
∂2`
=
{(1 + zti V (zti )) [V (zti ) − zti ] − zti } Ai,ja Bi,kb +
∂αja ∂βkb
b
i∈E
X1
{V (zti ) [zti (zti − V (zti )) − 1] −
b
i∈N
V (zsi ) [zsi (zsi − V (zsi )) − 1]} Ai,ja Bi,kb
II. In the case of an extended model, if c > 0, one also needs the derivatives with respect to the γ coefficients. They can be written as follows.
X
∂`
c
=
{b + zti − V (zti } Ci,ja +
∂γja
b (ti − c)
i∈E
X
c
V (zti ) Ci,ja
b (ti − c)
i∈N
6.17.3.6
log-normal models
5
X
∂2`
c
{V (zti ) [V (zti ) − zti ] − 1} Ai,ja Ci,ka +
=
∂αja ∂γka
b2 (ti − c)
i∈E
X c V (zt ) [zt − V (zt )] V (zs ) [zs − V (zs )] i
i
i
i
i
i
−
Ai,ja Ci,
b2
ti − c
si − c
i∈N
2
X
∂ `
c
{V (zti ) [1 + zti (V (zti ) − zti )] − 2zti } Bi,jb Ci,kb
=
∂βjb ∂γkb
b (ti − c)
i∈E
X c V (zt )
i
[zti (zti − V (zti )) − 1] −
b ti − c
i∈N
V (zsi )
[zsi (zsi − V (zsi )) − 1] Bi,jb Ci,kb
si − c
X
∂2`
c
{ti [b + zti − V (zti )]−
=
∂γja ∂γkb
b (ti − c)2
i∈E
o
c
(1 − V (zti ) [V (zti ) − zti ]) Ci,ja Ci,kb +
b
i
X c V (zt ) h
c
i
(V
(z
)
−
z
)
−
t
−
t
t
i
i
i
b (ti − c)2
b
i∈N
i
V (zsi ) h
c
si − (V (zsi ) − zsi ) Ci,ja Ci,kb
(si − c)2
b
Starting Values. In most situations, estimating standard log-normal
models does not need very accurate initial estimates. Therefore as default
starting values used by TDA all model coefficients are set to zero, with
the exception of the constant in the A-term of the model with is set
to the negative of the logarithm of the constant rate of an accordingly
defined exponential null model. On the other hand, initial estimates for
the extended log-normal model are critical, in particular the estimates
for the C-term of the model. This is implied by the shift operation done
with the c parameter. However, simple estimates of useful starting values
are not available, and therefore TDA uses as a default the same starting
values as with the standard log-normal model, and in addition sets all
C-term coefficients to zero. In many cases this will result in convergence
difficulties, in particular when episode splitting is applied. One way out
of such difficulties is first to estimate a standard log-normal model, and
then to use the resulting parameter estimates as starting values for the
6.17.3.6
log-normal models
6
Box 1 Output from command file rt12.cf
Model: Log-normal
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 5
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -2437.52
Norm of final gradient vector: 0.000491553
Last absolute change of function value: 8.24072e-08
Last relative change in parameters: 0.00239838
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Constant
4.3688
0.0960
45.5055 1.0000
2 1
0
1 A COHO2
-0.4608
0.1255
-3.6707 0.9998
3 1
0
1 A COHO3
-0.6110
0.1272
-4.8036 1.0000
4 1
0
1 A W
-0.3821
0.1062
-3.5992 0.9997
5 1
0
1 B Constant
0.2036
0.0342
5.9508 1.0000
Log likelihood (starting values): -2603.1677
Log likelihood (final estimates): -2437.5243
extended model, with the additional C-term coefficients set to zero or,
if available, to some better estimates.
Example 1 The standard log-normal model is similar to the log-logistic
model. One can see this by comparing the results from command file
rt12.cf, shown in Box 1 with the estimation results in 6.17.3.5.
6.17.3.6
log-normal models
7
Generalized Residuals. For the log-normal model, one can use the
pres option to request a table with generalized residuals.
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations. The rrisk option cannot be used
with the log-normal model.
6.17.3.7
1
generalized gamma models
6.17.3.7 Generalized Gamma Models
This section describes a model where, in the single transition case, the
duration is assumed to follow a generalized gamma distribution. The
model is useful since it contains as special cases the exponential, the
Weibull, and the log-normal models. Therefore it can be used to discriminate between these models.
We follow the explanation of this model given by Lawless ([1982],
p. 237). Lawless shows that, if the duration variable T has a generalized gamma distribution, then by re-parameterization, log(T ) can be
expressed as
log(T ) = a + b W
b>0
where W has density function
√
κκ−1/2
w
fw (w) =
exp
κ w − κ exp √
Γ(κ)
κ
κ>0
Γ(κ) is the gamma function. Now, given this re-parameterization, the
density of T can be written as
f (t ; a, b, κ) =
√
κκ−1/2
exp
κ zt − qt
b t Γ(κ)
log(t) − a
b
zt
qt = κ exp √
κ
zt =
Of course, the density is only defined for t > 0. The associated cumulative
distribution function, F (t), can be derived as the integral of the density
to be
Z qt κ−1
τ
F (t ; a, b, κ) = I(qt , κ) =
exp(−τ ) dτ
Γ(κ)
0
I(qt , κ) is called the incomplete gamma integral. Using this expression,
the survivor function is
G(t ; a, b, κ) = 1 − I(qt , κ)
d06170307.tex
April 21, 1998
6.17.3.7
2
generalized gamma models
2
a = 0.0
b = 1.0
k = 0.5
k = 1.0
1
k = 10.0
0
0
1
2
3
4
5
Figure 1 Gamma transition rates. Plot created with command
file plot-g1.cf.
3
a = 0.0
k = 1.0
b = 0.7
2
b = 1.0
1
b = 2.0
0
0
1
2
3
4
5
Figure 2 Gamma transition rates. Plot created with command
file plot-g2.cf.
and the transition rate for this model can finally be written as
√
f (t ; a, b, κ)
κκ−1/2 exp { κ zt − qt }
r(t ; a, b, κ) =
=
G(t ; a, b, κ)
b t Γ(κ) (1 − I(qt , κ))
The model has three parameters, a, b, and κ. a can take arbitrary values,
b and κ must be positive. Special cases are the exponential model, if
b = 1 and κ = 1; the Weibull model, if κ = 1; and a log-normal model is
reached if κ → ∞. Figures 1 and 2 show graphs of the rate function for
some parameter constellations.
6.17.3.7
generalized gamma models
3
Implementation. The generalized gamma model has three parameters
to be estimated. However, the current implementation in TDA follows a
proposal of Lawless to estimate only two of them, a and b, by maximum
likelihood, and to treat the κ-parameter as externally given. This very
much simplifies the calculations. The model approach implemented in
TDA may then be written as
κκ−1/2 exp
rjk (t) =
(jk)
zt
=
n√
(jk)
− qt
(jk)
bjk t Γ(κ) (1 − I(qt
o
(1)
, κ))
log(t) − ajk
bjk
(jk)
(jk)
qt
(jk)
κ zt
= κ exp
zt
√
!
κ
ajk = A(jk) α(jk)
bjk = exp B (jk) β (jk)
κ>0
externally given
It is assumed that the first component of each of the covariate (row)
vectors A(jk) and B (jk) is a constant equal to one. The associated coefficient vectors, α(jk) and β (jk) , are the model parameters to be estimated.
κ is assumed to be given externally, so there are in effect only two model
terms.
The generalized gamma model has model number 13. The κ coefficient
can be specified with the parameter
kgam = κ,
Default is kgam=1. Note that this coefficient must be positive.
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify the notation we
omit indices for transitions and use the abbreviations defined in (1).
Providing for the possibility of episode splitting by using the conditional
6.17.3.7
4
generalized gamma models
survivor function the log-likelihood is
`=
X
log
i∈E
X
√
κκ−1/2
exp κ zt − qt +
b t Γ(κ)
log {1 − I(qt , κ)} −
i∈Z
X
log {1 − I(qs , κ)}
i∈N
Using the notation
Qt = √
qtκ exp(−qt )
κ Γ(κ) [1 − I(qt , κ)]
Qs = √
qsκ exp(−qs )
κ Γ(κ) [1 − I(qs , κ)]
the first and second derivatives with respect to the α- and β-coefficients
are
X
∂`
=
∂αja
i∈E
√
q
√t − κ
κ
X Qt
X Qs
1
Ai,ja +
Ai,ja −
Ai,ja
b
b
b
i∈Z
X qt
√
∂`
√ − κ zt − 1 Bi,jb +
=
∂βjb
κ
i∈E
X
X
Qt zt Bi,jb −
Qs zs Bi,jb
i∈Z
i∈N
X
∂2`
qt
=
− 2 Ai,ja Ai,ka +
∂αja ∂αka
κb
i∈E
X qt
√
Qt
√ − κ − Qt
Ai,ja Ai,ka −
b2
κ
i∈Z
X qs
√
Qs
√ − κ − Qs
Ai,ja Ai,ka
b2
κ
i∈N
i∈N
6.17.3.7
generalized gamma models
5
X √
∂2`
zt
qt
=
κ− √ +1 √
zt Bi,jb Bi,kb +
∂βjb ∂βkb
κ
κ
i∈E
X qt
√
√ − κ − Qt zt − 1 Qt zt Bi,jb Bi,kb −
κ
i∈Z
X
√
qs
√ − κ − Qs zs − 1 Qs zs Bi,jb Bi,kb
κ
i∈N
2
X √
zt
qt
∂ `
1
κ− √ +1 √
=
Ai,ja Bi,kb +
∂αja ∂βkb
κ
κ b
i∈N
X qt
√
Qt
√ − κ − Qt zt − 1
Ai,ja Bi,kb −
b
κ
i∈Z
X qs
√
Qs
√ − κ − Qs zs − 1
Ai,ja Bi,kb
b
κ
i∈N
Example 1 To illustrate, we use command file rt13.cf. The parameter estimates are shown in Box 1.
Generalized Residuals. For the gamma models, one can use the pres
option to request a table with generalized residuals.
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations. Note that, for t = 0, the rate and
density is always set to zero, and the survivor function is set to 1; this
might not be correct. The rrisk option cannot be used with the gamma
model.
6.17.3.7
generalized gamma models
Box 1 Output from command file rt13.cf
Model: Generalized Gamma (kgam=1).
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 5
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -2473.54
Norm of final gradient vector: 9.1704e-08
Last absolute change of function value: 2.03216e-12
Last relative change in parameters: 4.57906e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Constant
5.0055
0.0906
55.2452 1.0000
2 1
0
1 A COHO2
-0.5339
0.1201
-4.4448 1.0000
3 1
0
1 A COHO3
-0.6671
0.1235
-5.4003 1.0000
4 1
0
1 A W
-0.5191
0.1012
-5.1274 1.0000
5 1
0
1 B Constant
0.0700
0.0366
1.9119 0.9441
Log likelihood (starting values): -2514.0199
Log likelihood (final estimates): -2473.5432
6
6.17.3.8
inverse gaussian models
1
6.17.3.8 Inverse Gaussian Models
This section describes the Inverse Gaussian model introduced for duration analysis by Chhikara and Folks [1977]. We follow the description
given by Lancaster ([1990], p. 49). In the single transition case the model
is derived by assuming that the duration variable, T , follows an Inverse
Gaussian distribution with density function
at − 1
1
√
f (t) = 3/2 φ
b>0
(1)
bt
b t
As in 6.17.3.6, φ and Φ are used, respectively, to denote the standard
normal density and distribution function. The model has two parameters.
The parameter a can take arbitrary values, the parameter b generates
dispersion and should be positive. The survivor function corresponding
to (1) is
1 − at
2a
−1 − a t
√
√
G(t) = Φ
− exp
Φ
b2
b t
b t
Figure 1 shows graphs of the rate function, defined as r(t) = f (t)/G(t),
for b = 1 and some values of a. As is seen, the graphs are similar to
log-normal rates.
Implementation. The Inverse Gaussian model has two parameters,
so there are two possibilities to include covariates. Similar to the lognormal model, TDA uses a linear link function for the parameter a, and
additionally provides the possibility to link covariates to the dispersion
parameter b via an exponential link function. So one gets the following
model formulation for the transition rate from origin state j to destination state k.
ajk t−1
1
√
φ
3/2
b t
b
t
jk
jk rjk (t) = 1−ajk t
2 ajk
−1−a t
Φ b √t − exp b2
Φ b √jkt
jk
jk
jk
(jk) (jk)
ajk = A
bjk
α
n
o
= exp B (jk) β (jk)
d06170308.tex
April 21, 1998
6.17.3.8
2
inverse gaussian models
2.5
b = 1.0
2
a = 1.5
1.5
1
a = 1.0
0.5
a = 0.1
0
0
1
2
3
4
5
6
Figure 1 Inverse Gaussian transition rates. Plot created with
command file plot-ig.cf.
It is assumed that the first component of each of the covariate (row)
vectors A(jk) and B (jk) is a constant equal to one. The associated coefficient vectors α(jk) and β (jk) are the model parameters to be estimated.
The transition rate model with inverse Gaussian duration distribution
has model number 14.
Maximum Likelihood Estimation. The model is estimated following
the outline given in 6.17.1.2 and 6.17.1.3. To simplify the formulas we
omit indices for transitions and use the following abbreviations:
at − 1
z1 (t) = √
b t
at + 1
z2 (t) = √
b t
E = exp
1
φ (z1 (t))
b t3/2
√
z1 (t) t
fa (t) = −
f (t)
b
fb (t) = z1 (t)2 − 1 f (t)
f (t) =
t
faa (t) = z1 (t)2 − 1 2 f (t)
b
2
fbb (t) = z1 (t) − 1 fb (t) − 2 z1 (t)2 f (t)
√
z1 (t) t
fab (t) =
{2 f (t) − fb (t)}
b
2a
b2
6.17.3.8
inverse gaussian models
3
G(t) = Φ (−z1 (t)) − E Φ (−z2 (t))
√
t
2E
Ga (t) =
{E φ (z2 (t)) − φ (z1 (t))} − 2 Φ (−z2 (t))
b
b
4a
Gb (t) = z1 (t) φ (z1 (t)) + E
Φ (−z2 (t)) − z2 (t) φ (z2 (t))
b2
√
4
1
t
Gaa (t) = 2 E
φ (z2 (t)) − 2 Φ (−z2 (t)) +
b
b
b
t
{z1 (t) φ (z1 (t)) − z2 (t) E φ (z2 (t))}
b2
Gbb (t) = z1 (t) z1 (t)2 − 1 φ (z1 (t)) +
8a
z2 (t) 1 − z2 (t)2 + 2 E φ (z2 (t)) −
b
2a 8a
E Φ (−z2 (t))
1+ 2
b
b2
√
t 1 − z1 (t)2 φ (z1 (t))
Gab (t) =
b
4a
1 − z2 (t)2 + 2 E φ (z2 (t)) +
b
2
4a
Φ
(−z
(t))
−
z
(t)
φ
(z
(t))
E
2
+
2
2
2
b2
b2
Using these abbreviations and providing for the possibility of episode
splitting by using conditional survivor functions, the log-likelihood is
X
X
X
`=
log (f (ti )) +
log (G(ti )) −
log (G(si ))
i∈E
i∈Z
i∈N
E, Z, and N , are the sets of episodes with an event, of censored episodes,
and of all episodes, respectively. The subscript i indexes single episodes
with starting time si and ending time ti . Omitting terms for starting
times, the first and second derivatives are
X fa (ti )
X Ga (ti )
∂`
=
Ai,ja +
Ai,ja
∂αja
f (ti )
G(ti )
i∈E
i∈Z
6.17.3.8
inverse gaussian models
4
X fb (ti )
X Gb (ti )
∂`
=
Bi,jb +
Bi,jb
∂βjb
f (ti )
G(ti )
i∈E
X
∂2`
=
∂αja ∂αka
i∈Z
(
i∈E
2 )
fa (ti )
faa (ti )
−
Ai,ja Ai,ka +
f (ti )
f (ti )
(
X
i∈Z
X
∂2`
=
∂βjb ∂βkb
(
2 )
fb (ti )
fbb (ti )
−
Bi,jb Bi,kb +
f (ti )
f (ti )
(
2 )
Gb (ti )
Gbb (ti )
−
Bi,jb Bi,kb
G(ti )
G(ti )
fab (ti ) fa (ti ) fb (ti )
−
f (ti )
f (ti ) f (ti )
i∈E
X
i∈Z
2
X
∂ `
=
∂αja ∂βkb
i∈E
2 )
Ga (ti )
Gaa (ti )
−
Ai,ja Ai,ka
G(ti )
G(ti )
X Gab (ti )
i∈Z
Ga (ti ) Gb (ti )
−
G(ti )
G(ti ) G(ti )
Ai,ja Bi,kb +
Ai,ja Bi,kb
Starting Values. For models which are not too complicated, maximum
likelihood estimation seems to be relatively robust with respect to initial
estimates. TDA takes the following default values: The constant of the
A-term of the model is taken to be the rate of an accordingly defined
exponential model, the constant of the B-term of the model is defined
as the logarithm of the square root of this rate. All other coefficients are
set to zero.
Example 1 As an illustration we use the main example data set. The
command file is rt14.cf, estimation results are shown in Box 1.
Generalized Residuals. For the inverse Gaussian model, one can use
the pres option to request a table with generalized residuals.
Printing Estimated Rates. The prate option can be used to request
tables with estimated rates, survivor and density functions, depending
on selected covariate constellations. The rrisk option cannot be used
with the inverse Gaussian model.
6.17.3.8
inverse gaussian models
Box 1 Output from command file rt14.cf
Model: Inverse Gaussian
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 5
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -2434.34
Norm of final gradient vector: 9.30888e-05
Last absolute change of function value: 8.96273e-10
Last relative change in parameters: 0.00415467
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A Constant
0.0007
0.0015
0.4225 0.3273
2 1
0
1 A COHO2
0.0067
0.0024
2.8243 0.9953
3 1
0
1 A COHO3
0.0088
0.0026
3.4284 0.9994
4 1
0
1 A W
0.0093
0.0021
4.4894 1.0000
5 1
0
1 B Constant
-1.7092
0.0311
-55.0104 1.0000
Log likelihood (starting values): -2699.6178
Log likelihood (final estimates): -2434.3446
5
6.17.4
mixture models
1
6.17.4 Mixture Models
This section deals with transition rate models where the distribution of
the waiting times for an event is a mixture distribution. Currently we
have only a single subsection that describes parametric models with a
gamma mixing distribution.
6.17.4.1 Gamma Mixture Models
d061704.tex
April 21, 1998
6.17.4.1
gamma mixture models
1
6.17.4.1 Gamma Mixture Models
Introduction
It is assumed that the transition rate, r(t | x, v), depends on a vector, x,
of observed time-invariant variables and on a scalar stochastic term, v,
which is not observed. The question is how the variables comprised in
x influence the transition rate, but this can not be observed directly. In
any observed sample of episodes, the transition rate depends also on the
values of the stochastic term v.
The same is true for the associated density and survivor functions.
The relation between transition rate, density and survivor function must
therefore be written as
r(t | x, v) =
G(t | x, v)
=
f (t | x, v)
G(t | x, v)
Z t
exp −
r(τ | x, v) dτ
(1)
(2)
0
To find an estimation approach, we shall make some simplifying assumptions. First, that the transition rate can be expressed as
r(t | x, v) = ru (t | x) v
v≥0
(3)
In fact, we assume that the component ru (t | x) of this expression,
called the underlying rate, is parametrically given according to one of
the models discussed in 6.17.3. The cumulative transition rate may then
be written as
Z t
H(t | x, v) = v H u (t | x) = v
ru (τ | x) dτ
(4)
0
The second basic assumption is that the stochastic term, v, follows a
gamma distribution with expectation E(v) = 1. Implied by this assumption, the density function of v can be written as
fv (v) =
d06170401.tex
κκ v κ−1
exp(−κ v)
Γ(κ)
κ>0
April 21, 1998
6.17.4.1
gamma mixture models
2
The variance of this distribution is Var(v) = 1/κ. The next step is to
calculate the resulting mixture distribution for the observed durations.
First, for the density and survivor functions, one gets
Z ∞
f (t | x) =
f (t | x, v) fv (v) dv
0
Z
G(t | x)
∞
G(t | x, v) fv (v) dv
=
0
The mixtures are expectations according to the distribution of v. The
calculation is easy because the gamma distribution implies the basic
equality
−κ
Z ∞
1
(5)
exp(−v s(t)) fv (v) dv = 1 + s(t)
κ
0
which holds for any real valued function s(t) (see, e.g., Lancaster [1990,
p. 328]). Therefore, using (2) and (4), the unconditional survivor function
is
−κ
Z ∞
1
G(t | x) =
exp (−v H u (t | x)) fv (v) dv = 1 + H u (t | x)
κ
0
(6)
The unconditional density function can be found by differentiating the
negative value of the unconditional survivor function, resulting in
−κ−1
1
f (t | x) = ru (t | x) 1 + H u (t | x)
(7)
κ
Finally, the unconditional transition rate is
−1
f (t | x)
1 u
u
r(t | x) =
= r (t | x) 1 + H (t | x)
G(t | x)
κ
(8)
Note that the unconditional transition rate cannot be derived directly
as an expectation according to the mixing distribution since, in general,
Z ∞
r(t | x) 6=
r(t | x, v) fv (v) dv
0
Also, there is no linear relation between r(t | x) and ru (t | x). Instead
(cf. Lancaster [1990, p. 63f]), the relation (8) may be expressed as
r(t | x) = ru (t | x) E(v | x, T ≥ t)
6.17.4.1
gamma mixture models
3
where E(v | x, T ≥ t) is the expectation of v for the part of the population which stays in the origin state at least until t. This expectation
is decreasing with t if the variance of the mixing distribution is positive. For example, the underlying rate ru (t | x) may be assumed to be
time-independent, a standard exponential model, but if there is unobserved heterogeneity, this is compatible with an decreasing observed rate
r(t | x).
Several Transitions. Extension to a situation with several transitions
is straightforward. To simplify notation, we omit dependence on covariates. Following the basic idea formulated in (3), for each transition from
origin state j ∈ O to destination state k ∈ Dj , transition-specific rates
can be defined by
u
rjk (t | vjk ) = rjk
(t) vjk
vjk ≥ 0
(9)
u
rjk
(t) is the underlying rate used for model formulation. The stochastic
term vjk , to provide for the possibility of unobserved heterogeneity, is
assumed to be specific for each transition, and stochastically independent
across transitions. And as before, it is assumed that these stochastic
terms are gamma distributed with unit expectation. The overall exit
rate from origin state j is then
X
u
rj (t | vj ) =
rjk
(t) vjk
(10)
k∈Dj
where vj is a vector of the stochastic terms vjk , k ∈ Dj . To simplify
notation we also define
Z t
u
u
Hjk
(t) =
rjk
(τ ) dτ
0
The next step is to define transition-specific sub-density functions f˜jk (t |
vjk ), and the overall survivor function Gj (t | vj ), in order to express
transition-specific rates as
rjk (t | vjk ) =
f˜jk (t | vjk )
Gj (t | vj )
(11)
Then, analogously to the single transition case, one has to derive the mixture distributions describing the data on the observational level. First,
to derive the unconditional survivor function Gj (t), one can take the expectation of Gj (t | vj ) over all components of the vector vj . This is easy
6.17.4.1
gamma mixture models
4
because we have assumed transition-specific stochastic terms. Therefore,
using (10), one finds
Z t
Y
u
Gj (t | vj ) =
exp −vjk
rjk (τ ) dτ
(12)
0
k∈Dj
Obviously, the expectation can be taken for each term separately, and
using (5), one gets
−κjk
Y 1
u
Gj (t) =
1+
H (t)
(13)
κjk jk
k∈Dj
which may be written as
Gj (t) =
Y
G̃jk (t)
where
k∈Dj
−κjk
1
u
G̃jk (t) = 1 +
H (t)
κjk jk
(14)
Secondly, one has to derive the expectation of f˜jk (t | vjk ). This can be
done by starting with (9) and (11), resulting in
u
f˜jk (t) = rjk
(t) vjk Gj (t | vj )
Using then (12), the expectation of the right-hand side is a product
of transition-specific expectations. Except for the transition (j, k), the
expectation is directly given by (5). The kth term contains vjk , but
u
differentiating the mixing integral with respect to Hjk
gives
−κjk −1
Z ∞
1
u
u
vjk exp(−vjk Hjk
(t)) fvjk (vjk ) dvjk = 1 +
Hjk
(t)
κjk
0
The expectation of f˜jk (t | vjk ) is therefore given by
−1 Y −κjk
1
1
u
u
u
˜
fjk (t) = rjk (t) 1 +
H (t)
1+
H (t)
κjk jk
κjk jk
k∈Dj
(15)
Finally, taking together (13) and (15), the unconditional transitionspecific rates are
−1
f˜jk (t)
1
u
u
rjk (t) =
= rjk
(t) 1 +
Hjk
(t)
(16)
Gj (t)
κjk
6.17.4.1
gamma mixture models
5
Maximum Likelihood Estimation. The approach to derive transition rate models with an additional gamma distributed stochastic term,
discussed in the previous section, can be summarized as follows. One
starts with a parametrically given transition rate, as in (3), and assumes
that the unobserved heterogeneity is gamma distributed with unit mean
and variance 1/κ. Then one derives the observed mixture distribution,
described by f (t | x), G(t | x), and r(t | x), given by (6), (7), and (8),
respectively.
It follows that the log-likelihood can be set up in the usual way. First,
for a single transition (with E and N denoting the sets of episodes with
an event, and of censored episodes, respectively) the log-likelihood may
be written as
X
X
` =
(17)
log(r(ti )) +
log(G(ti ))
i∈E
=
i∈N
1
log (ru (ti )) − log 1 + H u (ti ) −
κ
i∈E
X
1
κ log 1 + H u (ti )
κ
X
i∈N
Unfortunately, the standard method of episode splitting cannot be applied. Assume an episode (0, t), starting at time zero, and ending at
time t, and assume there is a covariate that changes its value at tx
(0 < tx < t). Then it would be quite possible to include this information
into the calculation of the cumulated rate H u (t | x) which is needed for
the calculation of the density and survivor function of the unconditional
distribution. But the method of episode splitting only works when this
calculation can be separated into two steps, the first step has information about the split (0, tx ), and the second step has information about
the split (tx , t). But obviously, the unconditional density and survivor
function can not be calculated in this two distinct steps. Therefore, all
models described in the following subsections assume that covariates do
not change during the given episodes. Of course, it would be possible to
include information about time-dependent variables in the description
of episodes in other ways; but this is currently not supported by tda.
Secondly, one has to consider a situation with two or more transitions.
In the same way as was shown in 6.17.1.2 for models without unobserved
heterogeneity, it is possible to factor the total likelihood into transitionspecific terms. Using the pseudo-survivor functions defined in (14), the
6.17.4.1
gamma mixture models
6
total likelihood can be written
Y Y Y
Y
L=
rjk (ti )
G̃j (ti )
j∈O k∈Dj i∈Ejk
i∈Nj
Then, by inserting the expressions given in (16) and (14), and by taking
the logarithm, the total log-likelihood becomes
X X X
1 u
u
` =
log rjk (ti ) − log 1 + Hjk (ti ) −
κ
j∈O k∈Dj i∈Ejk
X
1
u
(18)
κ log 1 +
Hjk
(ti )
κjk
i∈Nj
As already mentioned, the models described in this paper are extensions
of parametric transition rate models discussed in 6.17.3. The underlying
u
rate, rjk
(t), always corresponds to one of those models. Therefore, as
seen by (17) or (18), maximum likelihood estimation requires only little
modifications.
In fact, the formulation of log-likelihoods and derivatives can use
most of the formulas already given in 6.17.3. For this reason, we will use
the following notation. First we define new variables, djk , denoting the
transition-specific variances of the gamma distributed stochastic terms
in the models. These model parameters are analogously called the Dterms of the models. The possibility to link covariates to these model
parameters is provided by exponential link functions, that is
n
o
1
= exp D(jk) δ (jk)
djk =
kjk
D(jk) is a (row) vector of covariates with the first component always a
constant one. δ (jk) is the vector of associated coefficients to be estimated.
Secondly, to simplify the log-likelihood formulation, we define indicator variables
( 1
−d − 1
if i ∈ E
ui =
1
−d
otherwise
where E is the set of uncensored episodes. Furthermore, it is possible
to omit indices for transitions since the total likelihood factors into
transition-specific terms. The log-likelihood can be written, then, as
X
X
`=
log (ru (ti )) +
ui log (1 + d H u (ti ))
i∈E
i∈N
6.17.4.1
7
gamma mixture models
with ru (t) and H u (t), respectively, the rate and the cumulative rate of
the underlying model.
Finally, one can write the first and second derivatives of this loglikelihood in a way that can be used for all of the following models.
∂`
∂αja
=
X ∂ log(ru (ti ))
∂`
∂δjd
=
∂2`
∂αja ∂αka
=
X
i∈N
i∈E
i∈N
∂2`
∂αja ∂δkd
i∈N
ui
d
∂H u (ti )
u
1 + d H (ti ) ∂αja
log(1 + d H u (ti ))
d H u (ti )
+ ui
d
1 + d H u (ti )
X ∂ 2 log(ru (ti ))
X
∂2`
∂δjd ∂δkd
∂αja
i∈E
X
+
∂αja ∂αka
Di,jd
+
∂ 2 log(H u (ti ))
−
∂αja ∂αka
d
∂H u (ti ) ∂H u (ti )
1 + d H u (ti ) ∂αja
∂αka
d
ui
1 + d H u (ti )
X 2 H u (ti )
log(1 + d H u (ti ))
=
−
+
1 + d H u (ti )
d
i∈N
d H u (ti )
d H u (ti )
ui
1−
Di,jd Di,kd
1 + d H u (ti )
1 + d H u (ti )
=
d
1 + d H u (ti )
i∈N
d H u (ti )
∂H u (ti )
1
+ ui 1 −
Di,kd
d
1 + d H u (ti )
∂αja
X
These formulas, appropriate for a model with a single model term, can
obviously be generalized for models with two or more terms. Consequently, maximum likelihood estimation of the models described in section 6.17.3, with a stochastic term added, can use most of the expressions
for the log-likelihood and its derivatives given there.
Exponential Models
This section describes the exponential model with a gamma-distributed
stochastic term, v. The transition rate, now conditional on v, may be
6.17.4.1
gamma mixture models
8
written as
r(t | v) = a v
a, v ≥ 0
The underlying model, already described in 6.17.2.1, is the exponential,
implying that
ru (t) = a
H u (t) = a t
and
The observed mixture distribution is described by
1
−d
G(t)
=
[1 + d a t]
f (t)
= a [1 + d a t]
r(t)
= a [1 + d a t]
1
−d
−1
−1
d is the variance of the gamma distributed stochastic term. Obviously, if
d > 0, there will be a negative time-dependence, although the underlying
rate is a time-independent constant.
The model has two parameters. a, the constant transition rate of the
underlying exponential model, and d, the variance of the mixing gamma
distribution. In the tda implementation of this model both parameters,
the A- and D-term of the model, can be linked to covariates by an
exponential link function. The model formulation for the transition rate
from origin state j to destination state k is
ajk
rjk (t) =
1 + djk ajk t
ajk = exp A(jk) α(jk)
djk
=
exp D(jk) δ (jk)
It is assumed that the first component of each of the covariate (row)
vectors, A(jk) and D(jk) , is a constant equal to one. The associated coefficient vectors, α(jk) and δ (jk) , are the model parameters to be estimated.
The model can be requested by adding the mix=1 parameter to the
rate command for an exponential model. Maximum likelihood estimation is done following the outline given above. Using the notation introduced there, and dropping indices for transitions, the log-likelihood may
be written as
X
X
`=
log(a) +
ui log(1 + d a ti )
i∈E
i∈N
6.17.4.1
gamma mixture models
9
The first and second derivatives are easily found following the remarks
given above.
Weibull Models
This section describes the Weibull model with a gamma-distributed
stochastic term, v. The transition rate, now conditional on v, is
r(t | v) = ru (t) v
v≥0
with an underlying rate and cumulative rate given by
ru (t) = b ab tb−1
and
H u (t) = (a t)b
(see 6.17.3.3). The observed mixture distribution is described by
− 1
G(t) = 1 + d (a t)b d
f (t)
r(t)
− 1 −1
= b ab tb−1 1 + d (a t)b d
−1
= b ab tb−1 1 + d (a t)b
The model has three parameters. a, and b, are the parameters of the
underlying Weibull distribution, and d represents the variance of the
mixing gamma distribution. In the tda implementation of this model
all three parameters, the A-, B-, and D-term of the model, can be linked
to covariates by an exponential link function. The model formulation for
the transition rate from origin state j to destination state k is
b
bjk ajkjk tbjk −1
rjk (t)
=
ajk
=
1 + djk (ajk t)bjk
exp A(jk) α(jk)
bjk
=
exp B (jk) β (jk)
djk
=
exp D(jk) δ (jk)
It is assumed that the first component of each of the covariate (row) vectors, A(jk) , B (jk) , and D(jk) , is a constant equal to one. The associated
coefficient vectors, α(jk) , β (jk) , and δ (jk) , are the model parameters to
be estimated.
6.17.4.1
gamma mixture models
10
The model can be requested by adding the mix=1 parameter to the
rate command for a Weibull model. Maximum likelihood estimation is
done following the outline given in the introduction to this section. Using
the notation introduced there, and dropping indices for transitions, the
log-likelihood may be written as
X
X
`=
log(b ab tb−1
)+
ui log(1 + d (a ti )b )
i
i∈E
i∈N
First and second derivatives are easily found following the remarks given
above, and using the formulas for derivatives given in 6.17.3.3.
Sickle Models
This section describes the sickle model with a gamma-distributed stochastic term, v. The transition rate, now conditional on v, is
r(t | v) = ru (t) v
v≥0
with an underlying rate and cumulative rate defined by
t
u
r (t) = a t exp −
b
t
u
H (t) = a b b − (t + b) exp −
b
(see 6.17.3.4). The observed mixture distribution is described by
− d1
t
G(t) =
1 + d a b b − (t + b) exp −
b
− d1 −1
t
t
1 + d a b b − (t + b) exp −
f (t) = a t exp −
b
b
−1
t
t
r(t) = a t exp −
1 + d a b b − (t + b) exp −
b
b
The model has three parameters. a, and b, are the parameters of the
underlying sickle model, and d represents the variance of the mixing
gamma distribution. In the tda implementation of this model all three
parameters, the A-, B-, and D-term of the model, can be linked to covariates by an exponential link function. The model formulation for the
6.17.4.1
11
gamma mixture models
transition rate from origin state j to destination state k is
t
rjk (t) = ajk t exp −
bjk
−1
t
1 + djk ajk bjk bjk − (t + bjk ) exp −
bjk
ajk = exp A(jk) α(jk)
bjk
=
exp B (jk) β (jk)
djk
=
exp D(jk) δ (jk)
It is assumed that the first component of each of the covariate (row) vectors, A(jk) , B (jk) , and D(jk) , is a constant equal to one. The associated
coefficient vectors, α(jk) , β (jk) , and δ (jk) , are the model parameters to
be estimated.
The model can be requested by adding the mix=1 parameter to the
rate command for a sickle model. Maximum likelihood estimation is
done following the outline given in the introduction. Using the notation
introduced there, and dropping indices for transitions, the log-likelihood
may be written as
X
X
`=
log(b ab tb−1
)+
ui log(1 + d (a ti )b )
i
i∈E
i∈N
First and second derivatives are easily found following the remarks given
above, and using the formulas for derivatives given in 6.17.3.4.
Log-Logistic Models
This section describes the log-logistic model with a gamma-distributed
stochastic term, v. The transition rate, now conditional on v, is
r(t | v) = ru (t) v
v≥0
with an underlying rate and cumulative rate defined by
ru (t) =
b ab tb−1
1 + (a t)b
and
H u (t) = log 1 + (a t)b
6.17.4.1
gamma mixture models
12
(see 6.17.3.5). The observed mixture distribution is described by
G(t)
=
− 1
1 + d log(1 + (a t)b ) d
f (t)
=
− 1 −1
b ab tb−1 1 + d log(1 + (a t)b ) d
b
1 + (a t)
r(t)
=
−1
b ab tb−1 1 + d log(1 + (a t)b )
b
1 + (a t)
The model has three parameters. a, and b, are the parameters of the
underlying log-logistic distribution, and d represents the variance of the
mixing gamma distribution. In the tda implementation of this model all
three parameters, the A-, B-, and D-term of the model, can be linked
to covariates by an exponential link function. The model formulation for
the transition rate from origin state j to destination state k is
b
rjk (t)
=
ajk
=
bjk ajkjk tbjk .1 bjk −1
1
+
d
log(1
+
(a
t)
)
jk
jk
b
1 + (ajk t) jk
exp A(jk) α(jk)
bjk
=
exp B (jk) β (jk)
djk
=
exp D(jk) δ (jk)
It is assumed that the first component of each of the covariate (row) vectors, A(jk) , B (jk) , and D(jk) , is a constant equal to one. The associated
coefficient vectors, α(jk) , β (jk) , and δ (jk) , are the model parameters to
be estimated.
The model can be requested by adding the mix=1 parameter to the
rate command for a log-logistic model. Maximum likelihood estimation
is done following the outline given in the introduction. Using the notation
introduced there, and dropping indices for transitions, the log-likelihood
may be written as
X
X
`=
log(b ab tb−1
)+
ui log(1 + d (a ti )b )
i
i∈E
i∈N
First and second derivatives are easily found following the remarks given
above, and using the formulas for derivatives given in 6.17.3.5.
6.17.4.1
gamma mixture models
13
Log-Normal Models
This section describes the log-normal model with a gamma-distributed
stochastic term, v. The transition rate, now conditional on v, is
r(t | v) = ru (t) v
v≥0
with an underlying rate and cumulative rate defined by
ru (t)
H u (t)
=
φ (zt )
1
b t 1 − Φ (zt )
zt =
log(t) − a
b
= − log (1 − Φ (zt ))
(see 6.17.3.6). The observed mixture is described by
1
−d
G(t)
=
[1 − d log {1 − Φ (zt )}]
f (t)
=
φ (zt )
1
− 1 −1
[1 − d log {1 − Φ (zt )}] d
b t 1 − Φ (zt )
r(t)
=
φ (zt )
1
−1
[1 − d log {1 − Φ (zt )}]
b t 1 − Φ (zt )
The model has three parameters. a, and b, are the parameters of the
underlying log-normal model, and d represents the variance of the mixing gamma distribution. In the tda implementation of this model all
three parameters, the A-, B-, and D-term of the model, can be linked
to covariates by an exponential link function. The model formulation for
the transition rate from origin state j to destination state k is
1
φ (zjk (t))
−1
[1 − djk log {1 − Φ (zjk (t))}]
bjk t 1 − Φ (zjk (t))
rjk (t)
=
zjk (t)
=
ajk
=
log(t) − ajk
bjk
exp A(jk) α(jk)
bjk
=
exp B (jk) β (jk)
djk
=
exp D(jk) δ (jk)
with
6.17.4.1
gamma mixture models
14
It is assumed that the first component of each of the covariate (row) vectors, A(jk) , B (jk) , and D(jk) , is a constant equal to one. The associated
coefficient vectors, α(jk) , β (jk) , and δ (jk) , are the model parameters to
be estimated.
The model can be requested by adding the mix=1 parameter to the
rate command for an log-normal model. Maximum likelihood estimation
is done following the outline given in the introduction. Using the notation
introduced there, and dropping indices for transitions, the log-likelihood
may be written as
X
X
1
φ (zt )
`=
log
+
ui log {1 − d log (1 − Φ (zt ))}
b t 1 − Φ (zt )
i∈E
i∈N
First and second derivatives are easily found following the remarks given
above, and using the formulas for derivatives given in 6.17.3.6.
6.17.5
user-defined rate models
1
6.17.5 User-defined Rate Models
This chapter describes the frml command that can be used to estimate
transition rate models based on a user-defined likelihood. In addition,
the chapter discusses some examples. We thank Francesco Billari who
developed these applications of the frml command. The sections are:
6.17.5.1 The frml Command
6.17.5.2 The Hernes Model
6.17.5.3 The Coale-McNeil Model
6.17.5.4 Model with Several Domains
d061705.tex
April 21, 1998
6.17.5.1
1
the frml command
6.17.5.1 The frml Command
The command for estimating user-defined rate models is
frml (parameters) = log-likelihood function;
This command is basically identical with the fml command described
in 6.11.2. The main difference is that the frml command requires an
episode data structure, and the log-likelihood function is summed over
all episodes (or episode splits) which are currently defined; furthermore,
one can use the episode data operators (org, des, ts, tf) and type 5
variables (defined inside the edef command) in the definition of the
log-likelihood function.
The right-hand side of the command must provide the log-likelihood
function for a transition rate model. All other parameters (on the lefthand side) are optional and can be used to select an algorithm for function minimization, to define starting values, to change the print format,
and so on. In principle, one can use all parameters to control maximum
likelihood estimation which have been explained in 5.6. It should also be
possible to use the con parameter to define equality constraints.
All minimization algorithms discussed in 5.6.2 can be used; default is
the Newton I algorithm (mina=5). If required, Wengert’s method is used
to calculate first and second derivatives of the log-likelihood function.
The remainder of this section provides some elementary examples to
illustrate the frml command. Additional discussion and examples will
be given in subsequent sections. All examples will use the episode data
set rrdat.1 described in 3.3.3.
Example 1 Box 1 shows the command file frml1.cf that can be used
to replicate the estimation of a simple exponential model (see the example in 6.17.2.1). The log-likelihood function is
`=
X
i∈E
log(r(ti )) +
X
log(G(ti ))
i∈N
In our example, the rate does not depend on time,
r(ti ) = r = exp(α0 + COHO2α1 + COHO3α2 + Wα3 )
d06170501.tex
April 21, 1998
6.17.5.1
the frml command
Box 1 Command file frml1.cf (exponential model)
nvar(
dfile
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
= rrdat.1,
[3.0] = c1,
[2.0] = c2,
[3.0] = c3,
[3.0] = c4,
[2.0] = c5,
[3.0] = c6,
[3.0] = c7,
[3.0] = c8,
[3.0] = c9,
[3.0] = c10,
[3.0] = c11,
[2.0] = c12,
#
#
#
#
#
#
#
#
#
#
#
#
#
data file
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
# define additional variables
COHO1
COHO2
COHO3
W
=
=
=
=
ge(TB,348) & le(TB,384),
ge(TB,468) & le(TB,504),
ge(TB,588) & le(TB,624),
SEX[2],
#
#
#
#
birth
birth
birth
women
cohort 1
cohort 2
cohort 3
= 1
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
);
edef(
# define single episode data
ts = 0,
# starting time
tf = DUR,
# ending time
org = 0,
# origin state
des = DES, # destination state
);
frml (
xp = -4,0,0,0,
# starting values
) = rate = exp(a0 + COHO2 * a1 + COHO3 * a2 + W * a3),
l1
= if(DES,log(rate),0),
fn
= l1 - rate * DUR;
and the survivor function is simply
G(ti ) = exp(−r ti )
yielding the log-likelihood function
X
X
`=
log(r) −
rti
i∈E
i∈N
2
6.17.5.1
the frml command
Box 2 Part of standard output from frml1.cf
ML estimation of user-defined rate model.
Function definition:
rate
= exp(a0+COHO2*a1+COHO3*a2+W*a3)
l1
= if(DES,log(rate),0)
fn
= l1-rate*DUR
Function will be summed over 600 data matrix cases.
Using episode data.
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Number of model parameters: 4
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Got 4 starting value(s) from xp parameter.
Idx
1
2
3
4
Parameter
a0
a1
a2
a3
Starting value
-4.00000000e+00
0.00000000e+00
0.00000000e+00
0.00000000e+00
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -2475.44
Norm of final gradient vector: 1.10499e-11
Last absolute change of function value: 3.30667e-15
Last relative change in parameters: 5.84648e-07
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 a0
-5.0114
0.0843
-59.4446 1.0000
2 a1
0.5341
0.1120
4.7686 1.0000
3 a2
0.6738
0.1152
5.8472 1.0000
4 a3
0.5065
0.0942
5.3746 1.0000
Log likelihood (starting values): -2578.9484
Log likelihood (final estimates): -2475.4383
3
6.17.5.1
the frml command
4
Box 3 Part of Command file frml1m.cf
frml (
xp = -4,0,0,0,
-4,0,0,0,
-4,0,0,0,
) = rate1
rate2
rate3
rate
l1
fn
# starting values
=
=
=
=
=
exp(a10 + COHO2 * a11 + COHO3 * a12 + W * a13),
exp(a20 + COHO2 * a21 + COHO3 * a22 + W * a23),
exp(a30 + COHO2 * a31 + COHO3 * a32 + W * a33),
rate1 + rate2 + rate3,
if(eq(DES,1),log(rate1),
if(eq(DES,2),log(rate2),
if(eq(DES,3),log(rate3),0))),
= l1 - rate * DUR;
This expression is used in the command file frml1.cf. Part of the standard output is shown in Box 2. The estimated model parameters and
standard errors are identical with the results shown in 6.17.2.1.
Example 2 To replicate the estimation of an exponential model with
three alternative destination states (see 6.17.2.1), one can use the frml
command shown in Box 3. This command is part of command file frml1m.cf
(not shown) which is basically identical to command file rt1m.cf to set
up an episode data structure for three alternative destination states.
Example 3 The frml command in Box 4 illustrate the definition of a
log-likelihood function for a piecewise constant exponential model with 9
time periods. The specification is identical with the example discussed in
6.17.2.2 and should provide identical results. Of course, there are many
different possibilities to define this log-likelihood function.
This example can also be used to illustrate equality constraints. If
adding the parameters
con = bi − bi+1 = 0,
(i = 1, . . . , 8)
to the frml command in frml2.cf, one should get the estimation results for a simple exponential model (see command file frml2c.cf in
the example archive).
Example 4 The frml command in Box 5 estimates a Weibull model.
It is a replication of the example discussed in 6.17.3.3. The log-likelihood
can easily be modified to provide alternative specifications of the Weibull
model. For example, removing the line bb=exp(b0), would result in a
6.17.5.1
the frml command
Box 4 Part of Command file frml2.cf
frml (
xp = -4,-4,-4,-4,-4,-4,-4,-4,-4,0,0,0,
) = bb
p1
p2
p3
p4
p5
p6
p7
p8
p9
rr
=
=
=
=
=
=
=
=
=
=
=
COHO2 * b1 + COHO3 * b2 + W * b3,
ge(tf, 0) & lt(tf,12),
ge(tf,12) & lt(tf,24),
ge(tf,24) & lt(tf,36),
ge(tf,36) & lt(tf,48),
ge(tf,48) & lt(tf,60),
ge(tf,60) & lt(tf,72),
ge(tf,72) & lt(tf,84),
ge(tf,84) & lt(tf,96),
ge(tf,96),
if(p1,a1,
if(p2,a2,
if(p3,a3,
if(p4,a4,
if(p5,a5,
if(p6,a6,
if(p7,a7,
if(p8,a8,
if(p9,a9,0))))))))),
s1
s2
s3
s4
s5
s6
s7
s8
=
=
=
=
=
=
=
=
12
s1
s2
s3
s4
s5
s6
s7
*
+
+
+
+
+
+
+
exp(a1 + bb),
12 * exp(a2 +
12 * exp(a3 +
12 * exp(a4 +
12 * exp(a5 +
12 * exp(a6 +
12 * exp(a7 +
12 * exp(a8 +
lsurv = if(p1,tf
if(p2,s1
if(p3,s2
if(p4,s3
if(p5,s4
if(p6,s5
if(p7,s6
if(p8,s7
if(p9,s8
l1
fn
*
+
+
+
+
+
+
+
+
# starting values
bb),
bb),
bb),
bb),
bb),
bb),
bb),
exp(a1 + bb),
(tf - 12) * exp(a2
(tf - 24) * exp(a3
(tf - 36) * exp(a4
(tf - 48) * exp(a5
(tf - 60) * exp(a6
(tf - 72) * exp(a7
(tf - 84) * exp(a8
(tf - 96) * exp(a9
= if(DES,rr + bb,0),
= l1 - lsurv;
+
+
+
+
+
+
+
+
bb),
bb),
bb),
bb),
bb),
bb),
bb),
bb),0))))))))),
5
6.17.5.1
the frml command
6
Box 5 Part of Command file frml3.cf (Weibull model)
frml (
xp = -4,0,0,0,0,
) = aa
bb
l1
fn
=
=
=
=
exp( a0 + COHO2 * a1 + COHO3 * a2 + W * a3 ),
exp( b0 ),
if(DES,log(bb * aa^bb * tf^(bb - 1)),0),
l1 - (aa * tf)^bb;
Box 6 Part of Command file frml4.cf (log-normal model)
frml (
xp = 4,0,0,0,0,
) = aa
bb
zz
qz
vz
l1
fn
=
=
=
=
=
a0 + COHO2 * a1 + COHO3 * a2 + W * a3,
exp( b0 ),
(log(tf) - aa) / bb,
1 - nd(zz),
ndf(zz) / qz,
= if(DES,log(vz / (bb * tf)),0),
= l1 + log(qz);
direct estimate of bb and its standard error. (One should change, then,
the starting value for this parameter into a 1.)
Example 5 Box 6 shows an frml command to estimate a log-normal
model. It is a replication of the example discussed in 6.17.3.6. The definition of the log-likelihood function uses the operators nd and ndf for
the standard normal distribution and density function, respectively. This
is possible since these operators allow for automatic differentiation.
Example 6 To replicate the estimation of a generalized gamma model
(see 6.17.3.7), we use command file frml5.cf. Part of this command
file is shown in Box 7. As discussed in 6.17.3.7, the log-likelihood requires the gamma function and the incomplete gamma integral. Since
TDA automatically provides derivatives for the logarithm of the gamma
function (lgam operator) and for the incomplete gamma integral (icg
operator), the log-likelihood can be directly written, without numerical
integration. See Box 7. The resulting parameter estimates are identical
to those achieved with the rate command in 6.17.3.7. Note that we have
used the BFGS algorithm (mina=4), since the default Newton algorithm
would need better starting values. However, the final covariance matrix
6.17.5.1
the frml command
7
Box 7 Part of Command file frml5.cf (gamma model)
frml (
mina = 4,
xp = 5,0,0,0,0,
) = kk
aa
bb
zz
qt
rr
=
=
=
=
=
=
1,
a0 + COHO2 * a1 + COHO3 * a2 + W * a3,
exp(b0),
(log(tf) - aa) / bb,
kk * exp(zz / sqrt(kk)),
(kk - 0.5) * log(kk) + (sqrt(kk) * zz - qt) (b0 + log(tf) + lgam(kk)),
icc = icg(qt,kk),
fn = if (DES,rr,log(1 - icc));
is based on (automatically calculated) second derivatives.
Episode Splitting. The frml command expects the definition of a
log-likelihood function on the right-hand side. In general, this will be a
function of data matrix variables (including possibly type 5 variables)
and model parameters. This function is summed over all episodes, or
episode splits, which are available in the currently defined episode data
structure. Consequently, if the episode data structure contains episode
splits (given by the input data or by using the split option in the
edef command), summation is over episode splits. It is easy, therefore,
to apply the method of episode splitting to user-defined transition rate
models.
Example 7 To illustrate, we use the simple exponential model (Example 1). The new command file is frml1s.cf, shown in Box 8. Purely for
illustration, we define a variable SV=50 to be used in the edef command
for episode splitting. Every episode which is longer than 50 is then split
at t = 50, resulting in 845 episode splits.
To get the same results as in Example 1, the definition of the loglikelihood must be modified. Instead of using the duration (previously
DUR), we have to use starting and ending times of the splits. This information is available by the ts and tf operators. Also, instead of using
the exit status of the episodes (previously given by DES), we now have to
use the exit status of the episode splits; available by the operator des.
Using this modified set up of the log-likelihood function, we get identical
results as seen in Box 9.
6.17.5.1
the frml command
Box 8 Command file frml1s.cf
nvar(
dfile = rrdat.1,
# data file
ID
[3.0] = c1,
# identification number
SN
[2.0] = c2,
# spell number
TS
[3.0] = c3,
# starting time
TF
[3.0] = c4,
# ending time
SEX
[2.0] = c5,
# sex (1 men, 2 women)
TI
[3.0] = c6,
# interview date
TB
[3.0] = c7,
# birth date
TE
[3.0] = c8,
# entry into labor market
TMAR [3.0] = c9,
# marriage date (0 if no marriage)
PRES [3.0] = c10,
# prestige of current job
PRESN [3.0] = c11,
# prestige of next job
EDU
[2.0] = c12,
# highest educational attainment
COHO1 = ge(TB,348) & le(TB,384), # birth cohort 1
COHO2 = ge(TB,468) & le(TB,504), # birth cohort 2
COHO3 = ge(TB,588) & le(TB,624), # birth cohort 3
W
= SEX[2],
# women = 1
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
SV = 50,
);
edef(
# define single episode data
ts = 0,
# starting time
tf = DUR,
# ending time
org = 0,
# origin state
des = DES, # destination state
split = SV, # episode splitting at t = 50
);
frml (
xp = -4,0,0,0,
# starting values
) = rate = exp(a0 + COHO2 * a1 + COHO3 * a2 + W * a3),
l1
= if(des,log(rate),0),
fn
= l1 - rate * (tf - ts);
8
6.17.5.1
the frml command
Box 9 Part of standard output from command file frml1s.cf
ML estimation of user-defined rate model.
Function definition:
rate
= exp(a0+COHO2*a1+COHO3*a2+W*a3)
l1
= if(des,log(rate),0)
fn
= l1-rate*(tf-ts)
Function will be summed over 600 data matrix cases.
Using episode data (845 episode splits).
Convergence reached in 6 iterations.
Number of function evaluations: 7 (7,7)
Maximum of log likelihood: -2475.44
Norm of final gradient vector: 1.08916e-11
Last absolute change of function value: 1.83704e-16
Last relative change in parameters: 5.84648e-07
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 a0
-5.0114
0.0843
-59.4446 1.0000
2 a1
0.5341
0.1120
4.7686 1.0000
3 a2
0.6738
0.1152
5.8472 1.0000
4 a3
0.5065
0.0942
5.3746 1.0000
Log likelihood (starting values): -2578.9484
Log likelihood (final estimates): -2475.4383
9
6.17.5.2
the hernes model
1
6.17.5.2 The Hernes Model
To illustrate estimation of the Hernes model (originally proposed by
Hernes [1972], see also Diekmann [1990], Wu [1990]), we follow the model
formulation given by Wu [1990]. The (defective) distribution function is
given by
F (t) =
1
1 + σ exp(βλt )
with parameters σ > 0, β > 0, and 0 < λ < 1. The corresponding
survivor function is
G(t) =
1
1 + exp(−βλt )/σ
and the density function is
f (t) =
−σ exp(βλt )βλt log(λ)
(1 + σ exp(βλt ))2
The log-likelihood for model estimation can be written
X
X
`=
log(f (ti ; σ, β, λ) +
log(G(ti ; σ, β, λ)
i∈E
(1)
i∈Z
In general, one can link covariates to all three model parameters. It seems
reasonable to use the following link functions:
β
σ
λ
= exp(b)
= exp(s)
= exp(l)/(1 + exp(l))
(2)
Example 1 To illustrate, we begin with a model formulation without
covariates. A suitable command file to be used with our standard episode
data set is frml8.cf. Part of this command file (the frml command) is
shown in Box 1. The log-likelihood formulation follows (1) and uses the
link functions proposed in (2). Part of the standard output is shown in
Box 2. Some numerical problems occurred since we have not specified
suitable starting values.
d06170502.tex
April 21, 1998
6.17.5.2
the hernes model
2
Box 1 Part of command file frml8.cf
frml (
mina =
) = sigma
lambda
beta
ff
ff1
dens
surv
fn
4,
= exp(s),
= exp(l)/(1 + exp(l)),
= exp(b),
= sigma * exp(beta * lambda^tf),
= ff + 1,
= -(ff * beta * lambda^tf * log(lambda)) / (ff1 * ff1),
= 1 - 1 / ff1,
= if(des,log(dens),log(surv));
Example 2 The TDA example archive contains another command file,
frml8a.cf, that can be used to estimate a Hernes model with covariates
linked to the β parameter of the model. To achieve convergence, we have
used the parameter estimates from Example 1 as starting values.
6.17.5.2
the hernes model
Box 2 Part of standard output from command file frml8.cf
ML estimation of user-defined rate model.
Function definition:
sigma
= exp(s)
lambda = exp(l)/(1+exp(l))
beta
= exp(b)
ff
= sigma*exp(beta*lambda^tf)
ff1
= ff+1
dens
= -(ff*beta*lambda^tf*log(lambda))/(ff1*ff1)
surv
= 1-1/ff1
fn
= if(des,log(dens),log(surv))
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Number of model parameters: 3
Type of covariance matrix: 2
Idx
1
2
3
Parameter
b
l
s
Starting value
0.00000000e+00
0.00000000e+00
0.00000000e+00
Convergence reached in 10 iterations.
Number of function evaluations: 25 (25,25)
Numerical problems.
1 : overflow/underflow in exp() or log().
3 : log-likelihood positive.
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 b
1.5856
0.0378
41.9814 1.0000
2 l
3.8236
0.0599
63.8607 1.0000
3 s
-1.8379
0.1305
-14.0821 1.0000
Log likelihood (starting values): -16552.9633
Log likelihood (final estimates): -2526.3743
3
6.17.5.3
1
the coale-mcneil model
6.17.5.3 The Coale-McNeil Model
Another model sometimes used in demographical research is the CoaleMcNeil model, originally proposed by Coale and McNeil [1972]. For a
discussion of the model, see Huinink and Henz [1993], Wu [1990]. The
density function is
f (t) = σλ exp {−β(t − µ) − exp(−λ(t − µ)) − log(Γ(β/λ))}
with parameters µ > 0, β > 0, and λ > 0. Slightly more general, one can
also consider the defective density
fσ (t) = σ f (t)
0≤σ≤1
To derive the distribution function
Z t
Fσ (t) =
fσ (τ ) d τ
−∞
one can apply the substitution τ = µ − log(u)/λ, to get
Fσ (t) = σ (1 − I(exp(−λ(t − µ)), β/λ))
where I denotes the incomplete gamma integral. To estimate this model,
one often assumes that β/λ is constant. As proposed by Rodriguez and
Trussell, β/λ = 0.604 (see Wu [1990], p. 192). It seems reasonable, then,
to use the parameterization
β/λ
λ
µ
σ
= constant
= exp(l)
= exp(m)
= exp(s)/(1 + exp(s))
Example 1 To illustrate, we begin with β/λ = 0.604 and σ = 1, and
use our standard episode data set. The frml command for model estimation (part of command file frml9.cf) is shown in Box 1. The loglikelihood uses TDA’s operators for the logarithm of the gamma function
(lgam) and the incomplete gamma integral (icg). We use the BFGS algorithm (mina=4) since the default Newton algorithm is very sensitive
d06170503.tex
April 21, 1998
6.17.5.3
the coale-mcneil model
2
Box 1 Part of command file frml9.cf
frml (
mina = 4,
xp = -3,2,
)= bg
sigma
lambda
mue
beta
tl
ldens
surv
fn
=
=
=
=
=
=
=
=
=
0.604,
exp(s) / (1 + exp(s)),
exp(l),
exp(m),
bg * lambda,
exp(-lambda * (tf - mue)),
log(sigma) + l - beta * (tf - mue) - tl - lgam(bg),
1 - sigma * (1 - icg(tl,bg)),
if (DES,ldens,log(surv));
to starting values. (In fact, the BFGS algorithm could also be used with
default zero starting values.) Estimation results are shown in Box 2.
It should also be possible to link covariates to the model parameters.
As an illustration, the TDA example archive contains the command file
frml9a.cf where our standard covariates (COHO2, COHO3, and W) are
linked to the λ parameter. This results in a significantly better fit, the
log-likelihood is −2322.96.
6.17.5.3
the coale-mcneil model
Box 2 Part of standard output from command file frml9.cf
ML estimation of user-defined rate model.
Function definition:
bg
= 0.604
sigma
= exp(s)/(1+exp(s))
lambda = exp(l)
mue
= exp(m)
beta
= bg*lambda
tl
= exp(-lambda*(tf-mue))
ldens
= log(sigma)+l-beta*(tf-mue)-tl-lgam(bg)
surv
= 1-sigma*(1-icg(tl,bg))
fn
= if(DES,ldens,log(surv))
Function evaluation: sum over 600 data matrix cases.
Using episode data.
Maximum likelihood estimation.
Algorithm 4: BFGS
Idx
1
2
3
Parameter
l
m
s
Starting value
-3.00000000e+00
2.00000000e+00
0.00000000e+00
Convergence reached in 13 iterations.
Number of function evaluations: 22 (22,1)
Maximum of log likelihood: -2529.59
Norm of final gradient vector: 8.34764e-07
Idx Parameter
Value
Error
Value/E Signif
-------------------------------------------------------1 l
-3.0653
0.0454
-67.4629 1.0000
2 m
2.8902
0.0685
42.1764 1.0000
3 s
1.8505
0.1421
13.0217 1.0000
Log likelihood (starting values): -2699.6069
Log likelihood (final estimates): -2529.5933
3
6.17.5.4
1
models with several domains
6.17.5.4 Models with Several Domains
In unpublished papers, Francesco Billari has proposed to consider transition rate models with two (or more) domains. In the most simple case,
the modeling approach is
ra (t)
if t ≤ ta
r(t) =
ra (t) + rb (t) if t > ta
where ra (t) is the rate for the first domain, [0, ta ], and ra (t) + rb (t) is
the rate for the second domain, (ta , ∞). A possible application, proposed
by Billari, is to investigate age at first marriage. Until a certain age ta ,
a very low constant rate ra (t) is assumed, then, beginning at age ta ,
individuals enter the ”‘essential”’ risk period for becoming married and
the rate is given by ra (t) + rb (t).
In principle, it should be possible to regard ta as a model parameter
that can be estimated. The problem is, of course, that the log-likelihood
is not continuously differentiable in ta . As a possible strategy, one can try
an algorithm that does not require derivatives. Another strategy would
be to estimate the model with a series of differently fixed values of ta . Of
course, the best strategy would be to apply an algorithm for non-smooth
function optimization; however, this is not yet available in TDA.
Since this modeling approach is not well suited for our standard example data set, we do not give an illustration. However, the TDA example archive contains the command file frml10.cf which can be used
for some experiments. It uses a simple exponential model for ra (t) and
a log-normal model for rb (t).
d06170504.tex
April 21, 1998
6.17.6
discrete time rate models
1
6.17.6 Discrete Time Rate Models
This chapter discusses discrete time transition rate models. Basic references are Allison [1982] and Hamerle and Tutz [1989]. The sections
are:
6.17.6.1 Introduction
6.17.6.2 Logistic Regression Models
6.17.6.3 Complementary Log-Log Models
d061706.tex
April 21, 1998
6.17.6.1
introduction
1
6.17.6.1 Introduction
Discrete time models begin with the assumption that transitions from
a given origin state to a new destination state can only occur at some
discrete points in time. It is possible then to represent the time axis by
the natural numbers (1, 2, 3, . . .). To describe episodes in a discrete time
framework we can use the same notation as introduced in 3.3.1 for a
continuous time framework:
(oi , di , tsi , tfi , xi (t))
i = 1, . . . , N
tsi is the starting time of the episode, tfi is the ending time. oi is the origin
state of the episode, and di is the destination state. xi (t) is a vector of
covariates, possibly time-varying, connected with the episode.
The only important difference is in the interpretation of starting and
ending times. We first concentrate on ending times and assume that all
starting times are zero. As already mentioned, values of ending times
are natural numbers indexing the discrete points in time when events
can take place. Consequently, the duration of episodes is represented by
a discrete random variable T ∗ . (To distinguish discrete time concepts
from the corresponding continuous time concepts we mark all symbols
by an asterisk.) For all practical purposes we can restrict T ∗ to be finite
with possible values 1, . . . , q.
Ti∗ = ti means that in the ith episode an event takes place at ti , or
the episode is censored at this point in time. To understand this, it may
be helpful to imagine an underlying continuous time axis with intervals
defined by
τ0 < τ1 < τ2 < . . . < τq
with τ0 = 0
A continuous variable T , defined on this time axis, may then be used to
define a one-to-one relation
T ∗ = t ⇔ T ∈ [τt−1 , τt )
t = 1, . . . , q
That an event takes place at the discrete point in time t may be interpreted, then, as an event taking place in the tth interval on the underlying continuous time axis. The meaning of a right censored episode at
d06170601.tex
April 21, 1998
6.17.6.1
introduction
2
the discrete point in time t is, accordingly, that no event has happened
up to the end of the tth interval.
With these guidelines for interpretation, we can define some concepts
for a statistical description; first assuming a single possible event. We
may start with the discrete analogon to the density function for a continuous time variable which is the probability function for the discrete
time variable T ∗ . It is simply defined as
f ∗ (t) = Pr (T ∗ = t)
t = 1, . . . , q
with
q
X
f ∗ (t) = 1
t=1
Next we can define a discrete analogon for the survivor function. To be
consistent with the interpretation of right censored episodes, the definition is
G∗ (t) = Pr (T ∗ > t)
(1)
meaning that at least at t the individual is still in the origin state. Finally,
a discrete analogon for the transition rate can be defined as
r∗ (t) = Pr (T ∗ = t | T ∗ ≥ t)
Obviously, this is a conditional probability, meaning that the episode
ended with an event at t given that no event occurred before t. This
implies that
0 ≤ r∗ (t) ≤ 1
what is different from the continuous time case. In that case the transition rate can assume values greater than one; now the rate is restricted
to the range zero to one, and of course, the formulation of models must
be consistent with this fact.
Another difference to the continuous time case is in the expression
of the transition rate by the density and survivor functions. As a consequence of the definition in (1), the formulation is
r∗ (t) =
f ∗ (t)
f ∗ (t)
= ∗
(1 − r∗ (t))
− 1)
G (t)
G∗ (t
6.17.6.1
introduction
3
The right-hand side of this equation follows from
1 − r∗ (t) = 1 −
Pr (T ∗ = t)
Pr (T ∗ > t)
G∗ (t)
=
= ∗
∗
∗
Pr (T ≥ t)
Pr (T ≥ t)
G (t − 1)
(2)
This equation also implies an expression for the survivor function in
terms of transition rates:
G∗ (t) =
t
Y
(1 − r∗ (l))
l=1
We can now extend these concepts to situations with several transitions.
The terminology is analogous to the continuous time case. There are
origin states j ∈ O with destination states k ∈ Dj ; and with each origin
state is associated a two-dimensional discrete random variable (Tj∗ , Dj ).
The variable Tj∗ describes the duration in the origin state j until an
event occurs; and the variable Dj gives the destination state if an event
occurs.
A description of this two-dimensional variable is given by the two-dimensional probability function
∗
fjk
(t) = Pr (Tj∗ = t, Dj = k)
expressing the probability that a transition from origin state j to destination state k takes place at the discrete point in time t. Next, as in the
continuous time case, transition-specific rates can be defined by
∗
rjk
(t) = Pr (Tj∗ = t, Dj = k | Tj∗ ≥ t)
This is the conditional probability that a transition from origin state j to
destination state k occurs at t, given the origin state j and that no event
occurred until this point in time. It follows that the transition-specific
rates add together to the overall exit rate
X
∗
rj∗ (t) =
rjk
(t)
(3)
k∈Dj
The associated survivor function now depends on the given origin state
j and is defined by
G∗j (t) = Pr (Tj∗ > t)
(4)
6.17.6.1
introduction
4
This is the probability that no event occurs up to, and including, the
discrete point in time t. The transition rate from the given origin state
j to destination state k may finally be written as
∗
rjk
(t) =
∗
fjk
(t)
Gj (t − 1)
Having introduced the basic concepts, the idea of transition rate models
in discrete time can be described analogously to the continuous time case
by
∗
rjk
(t) = gjk (t, xjk , βjk , jk )
The basic idea is again to make the transition rate from origin state
j to destination state k dependent on a vector of possibly transitionspecific covariates xjk with associated coefficients βjk , and possibly on
some not observed stochastic influences comprised in the variables jk .
The dependence is formulated by assuming a function gjk ; each different
specification of this function gives a different discrete time rate model.
Maximum Likelihood Estimation. Before dealing with some possible model specifications we shortly describe maximum likelihood estimation in the discrete time framework. The approach is basically the same
as in the continuous time case, but there are some small differences. The
most important one is that the likelihood in the case of several transitions
does not factorize into transition-specific terms.
1. In the case of a single transition, the likelihood may be written as
Y
Y
L=
f ∗ (ti )
G∗ (ti )
i∈E
i∈Z
where E denotes the set of episodes ending with an event, and Z denotes
the set of censored episodes. The contribution to the likelihood of an
episode that has an event at ti is simply the probability that an event
takes place at this point in time. The contribution of an episode that is
censored at ti is the value of the survivor function at ti , the probability
that no event has occurred until, and including, ti .
2. In the case of a given origin state j, but possibly more than one
destination state, the likelihood is
Y Y
Y
∗
Lj =
fjk
(ti )
G∗j (ti )
(5)
k∈Dj i∈Ejk
i∈Zj
6.17.6.1
introduction
5
where Ejk is the set of episodes with origin state j and destination state
k (implying that these episodes are not censored), and Zj is the set of
censored episodes with origin state j. The contribution of an uncensored
episode to the likelihood is now the value of the tow-dimensional prob∗
ability function fjk
(ti ); the contribution of a censored episode is again
the overall survivor function as defined in (4).
3. If there are several origin states, the total likelihood is simply the
product of the likelihoods for each given origin state as defined in (5):
Y Y Y
Y
∗
L=
fjk
(ti )
G∗j (ti )
(6)
j∈O k∈Dj i∈Ejk
i∈Zj
This may be rewritten to reach a somewhat more convenient expression.
By using (2) and (4) we get
∗
fjk
(t) =
∗
rjk
(t)
G∗ (t)
1 − rj∗ (t) j
and
G∗j (t) =
t
Y
(1 − rj∗ (l))
l=1
By substituting these expressions in (6) and taking the logarithm, the
overall log-likelihood becomes
(
)
ti
∗
X X X
XX
rjk
(ti )
`=
log
+
log 1 − rj∗ (l) (7)
∗
1 − rj (ti )
j∈O k∈Dj i∈Ejk
i∈Nj l=1
where Nj denotes the set of all episodes with origin state j. This formulation shows that we only need expressions for the transition rates to
derive maximum likelihood estimates.
Episode Splitting. To include covariates, transition rates must be conditioned on their values. If covariates do not change during episodes the
log-likelihood (7) can be used without modification. If covariates change
their values at some discrete points in time the method of episode splitting can be used.
In order to apply this method, TDA uses conditional survivor functions, defined by
G∗j (t | s) = Pr (Tj∗ > t | Tj∗ > s) =
G∗j (t)
G∗j (s)
6.17.6.1
introduction
6
with the convention that G∗j (0) = 1. Substituting these conditional survivor functions in (7), the log likelihood becomes
(
)
ti
∗
X X X
X X
rjk
(ti )
`=
log
+
log 1 − rj∗ (l)
∗
1 − rj (ti )
j∈O k∈Dj i∈Ejk
i∈Nj l=si +1
This modified log likelihood is used by TDA and provides for the possibility to use split episodes and to condition on given starting times not
necessarily equal to zero.
6.17.6.2
1
logistic regression models
6.17.6.2 Logistic Regression Models
This section describes the logistic regression model. It was discussed
by several authors in somewhat different versions. Here we follow the
exposition given by Allison [1982]. In the single transition case the model
can be written as
r∗ (t) =
exp(Aα + β1 t + β2 t2 + . . . + βn tn )
1 + exp(Aα + β1 t + β2 t2 + . . . + βn tn )
t = 1, . . . , q
A is a (row) vector of covariates with the first component equal to one,
and α is a vector of associated coefficients. The time dependence is assumed to be a polynomial of order n in t. Therefore, the model is quite
flexible, and one can test how many terms of the polynomial are needed
for a sufficient fit.
Implementation. The logistic regression model has model number 20;
the command to request model estimation is
rate (parameter) = 20;
For a description of the syntax and options see 6.17.1.4. Implementation
of the model provides for possibly more than one transition and can be
described by
ajk bjk (t)
1 + ajk bjk (t)
n
o
= exp A(jk) α(jk)
∗
rjk
(t) =
ajk
(1)
o
n
(jk)
(jk)
bjk (t) = exp β1 t + β2 t2 + . . . + βn(jk) tn
There are two model terms. The product of a (row) vector A(jk) of possibly transition-specific covariates (with first component equal to one) and
of associated coefficients α(jk) is linked via an exponential link function
to the first, the A-term, of the model. In addition, a transition-specific
polynomial in t, with coefficient vector β (jk) , is linked to the second, the
B-term, of the model. Linking is again via an exponential link function.
d06170602.tex
April 21, 1998
6.17.6.2
logistic regression models
2
The degree of the polynomial, n, can be specified with the deg parameter
in the rate command; default is deg=0.
Maximum Likelihood Estimation. Model estimation follows the
outline given in 6.17.6.1. Since the likelihood does not factor into transition-specific terms, mixed derivatives across different destination states
must be included. First, following (3), the overall exit rate from origin
state j is
X
∗
rj∗ (t) =
rjk
(t)
k∈Dj
∗
where rjk
(t) is given by (1). The log-likelihood for a given origin state j
is
(
)
ti
∗
X X
X X
rjk
(ti )
`j =
log
log 1 − rj∗ (l) (2)
+
∗
1 − rj (ti )
k∈Dj i∈Ejk
i∈Nj l=si +1
A small simplification is possible if we define a variable
(
ti − 1
if i ∈ Ej
∗
ti =
ti
if i ∈ Zj
where Ej and Zj are, respectively, the set of uncensored and of censored
episodes with origin state j. (2) may then be rewritten as
∗
`j =
X X
log
∗
rjk
(ti )
k∈Dj i∈Ejk
ti
X X
+
log 1 − rj∗ (l)
(3)
i∈Nj l=si +1
This log-likelihood is used for the formulation of derivatives. We denote
the left-hand part of the right side, that is the additional part for unR
censored episodes, by `L
j , the right-hand part, for all episodes, by `j . To
write down the derivatives we will use the following abbreviations.
L
∗
Ujk
(t) = 1 − rjk
(t)
L
∗
∗
Vjk
(t) = −rjk
(t) 1 − rjk
(t)
R
Ujk
(t) = −
∗
∗
(t))
rjk
(t)(1 − rjk
∗
1 − rj (t)
6.17.6.2
logistic regression models
R
Vjk
(t)
3
"
#
∗
∗
∗
∗
rjk
(t)(1 − rjk
(t))
rjk
(t)(1 − rjk
(t))
∗
=
2 rjk (t) − 1 −
1 − rj∗ (t)
1 − rj∗ (t)
R
Wjkk
0 (t) =
∗
∗
∗
∗
−rjk
(t)(1 − rjk
(t)) rjk
0 (t)(1 − rjk 0 (t))
(1 − rj∗ (t))2
The derivatives of the left part on the right-hand side of (3), `L
j , with
respect to α(jk) and β (jk) may then be written as
∂`L
j
(jk)
∂αja
∂`L
j
(jk)
∂βjb
∂ 2 `L
j
(jk) (jk)
∂αja αka
∂ 2 `L
j
(jk) (jk)
∂βjb βkb
∂ 2 `L
j
(jk) (jk)
∂αja βkb
=
X X
(jk)
L
Ujk
(ti ) Aja
k∈Dj i∈Ejk
=
X X
L
Ujk
(ti ) tji b
k∈Dj i∈Ejk
=
X X
(jk)
(jk)
L
Vjk
(ti ) Aja Aka
k∈Dj i∈Ejk
=
X X
L
Vjk
(ti ) tji b +kb
k∈Dj i∈Ejk
=
X X
(jk)
L
Vjk
(ti ) Aja tki b
k∈Dj i∈Ejk
(jk)
The derivatives of the right-hand side of (3), `R
j , with respect to α
(jk)
and β
, are
∂`R
j
(jk)
∂αja
∂`R
j
(jk)
∂βjb
∂ 2 `R
j
(jk) (jk)
∂αja αka
∗
=
ti
X X
(jk)
R
Ujk
(l) Aja
i∈Nj l=si +1
∗
=
ti
X X
R
Ujk
(l) ljb
i∈Nj l=si +1
∗
=
ti
X X
i∈Nj l=si +1
(jk)
(jk)
R
Vjk
(l) Aja Aka
6.17.6.2
logistic regression models
∂ 2 `R
j
(jk) (jk)
∗
=
∂βjb βkb
∂ 2 `R
j
(jk) (jk)
∂αja βkb
4
ti
X X
R
Vjk
(l) ljb +kb
i∈Nj l=si +1
∗
=
ti
X X
(jk)
R
Vjk
(l) Aja lkb
i∈Nj l=si +1
Finally, the mixed derivatives, k 6= k 0 , are
∂ 2 `R
j
(jk) (jk0 )
∂αja αka
∂ 2 `R
j
(jk) (jk0 )
∂βjb βkb
∂ 2 `R
j
(jk) (jk0 )
∂αja βkb
∗
=
ti
X X
(jk)
(jk0 )
R
Aka
Wjkk
0 (l) Aj
a
i∈Nj l=si +1
∗
=
ti
X X
R
jb +kb
Wjkk
0 (l) l
i∈Nj l=si +1
∗
=
ti
X X
(jk)
R
l kb
Wjkk
0 (l) Aj
a
i∈Nj l=si +1
Initial Estimates. Without duration-dependence (deg=0), the logistic regression model is similar to the exponential model and TDA uses
identical default starting values. If the polynomials to capture durationdependence have higher degrees these default starting values can easily lead to numerical problems and convergence difficulties. A sensible
strategy is to estimate a series of models with increasing degrees of the
polynomial and to use the parameter estimates of previously estimated
models to define starting values.
Example 1 To illustrate the logistic regression model, we use our main
episode data set (rrdat.1) described in 3.3.3. Box 1 shows the command
file, rtd.1, to estimate a logistic regression model without durationdependence. Standard output is shown in Box 2.
To estimate models with duration dependence, one can add the parameter
deg = degree of polynomial,
to the rate command. As we have mentioned, higher degree polynomials
can lead to numerical difficulties due to very different magnitudes of the
model parameters. Also, if there are many time points, the estimation
process can be very slow.
6.17.6.2
logistic regression models
5
Box 1 Command file rtd1.cf for logistic regression
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
COHO1
COHO2
COHO3
W
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
ge(TB,348) & le(TB,384), # birth
ge(TB,468) & le(TB,504), # birth
ge(TB,588) & le(TB,624), # birth
SEX[2],
# women
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
cohort 1
cohort 2
cohort 3
= 1
# destination state
);
edef(
ts = 0,
# starting time
tf = DUR,
# ending time
org = 0,
# origin state
des = DES, # destination state
);
rate (
xa(0,1) = COHO2,COHO3,W,
) = 20;
Example 2 The TDA example archive contains another command file,
rtd1m.cf, that can be used to estimate logistic regression models for
episodes with three alternative destination states. It is basically identical
with command file rt1m.cf described in 6.17.2.1.
Additional Remarks. TDA cannot calculate generalized residuals for
the logistic regression model and the pres option will be ignored. Also
the rrisk and prate parameters cannot be used with this model.
6.17.6.2
logistic regression models
Box 2 Standard output from command file rtd1.cf
Model: Discrete time logistic regression, degree 0.
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Number of model parameters: 4
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Changed scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 5 iterations.
Number of function evaluations: 6 (6,6)
Maximum of log likelihood: -2472.39
Norm of final gradient vector: 1.04739e-07
Last absolute change of function value: 2.93992e-11
Last relative change in parameters: 3.51238e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
--------------------------------------------------------------1 1
0
1 A Constant -5.0056
0.0847
-59.1117 1.0000
2 1
0
1 A COHO2
0.5402
0.1127
4.7944 1.0000
3 1
0
1 A COHO3
0.6822
0.1160
5.8808 1.0000
4 1
0
1 A W
0.5133
0.0949
5.4097 1.0000
Log likelihood (starting values): -2511.4674
Log likelihood (final estimates): -2472.3890
6
6.17.6.3
1
complementary log-log models
6.17.6.3 Complementary Log-Log Models
Following again the exposition by Allison [1982], this section describes
the complementary log-log model. In the single transition case, the transition rate is
r∗ (t) = 1 − exp − exp(Aα + β1 t + β2 t2 + . . . + βn tn )
Implementation. The command to request model estimation is
rate (parameter) = 21;
For a description of the syntax and options see 6.17.1.4. Implementation
of the model provides for possibly more than one transition and can be
described by
∗
rjk
(t) = 1 − exp {−ajk bjk (t)}
n
o
ajk = exp A(jk) α(jk)
n
o
(jk)
(jk)
bjk (t) = exp β1 t + β2 t2 + . . . + βn(jk) tn
Again, there are two model terms. The product of a (row) vector A(jk) of
possibly transition-specific covariates (with the first component equal to
one), and of associated coefficients α(jk) , is linked via an exponential link
function to the first, the A-term, of the model. In addition, a transitionspecific polynomial in t, with coefficient vector β (jk) , is linked to the
second, the B-term, of the model. The degree of the polynomial can be
specified with the deg parameter in the rate command; default is deg=0.
Maximum Likelihood Estimation. Model estimation follows the
outline given in 6.17.6.1 and is similar to the estimation procedure for
the logistic regression model. Using the notation introduced in 6.17.6.2,
the log-likelihood for the complementary log-log model can be written
as
∗
`j =
X X
k∈Dj i∈Ejk
d06170603.tex
log
∗
rjk
(ti )
+
ti
X X
log 1 − rj∗ (l)
i∈Nj l=si +1
April 21, 1998
6.17.6.3
complementary log-log models
2
Also the expressions for the derivatives are the same as for the logistic
regression model, one only has to change the abbreviations defined in
6.17.6.2 into
"
#
1
L
Ujk
(t) = ajk bjk (t)
∗ (t) − 1
rjk
#"
#
"
ajk bjk (t)
1
L
−1 1−
Vjk (t) = ajk bjk (t) ∗
∗ (t)
rjk (t)
rjk
R
Ujk
(t) = ajk bjk (t)
R
Vjk
(t)
∗
rjk
(t) − 1
1 − rj∗ (t)
"
#
∗
∗
1 − rjk
(t)
rjk
(t) − rj∗ (t)
= ajk bjk (t)
ajk bjk (t)
−1
1 − rj∗ (t)
1 − rj∗ (t)
R
Wjkk
0 (t) =
∗
∗
−ajk bjk (t)(1 − rjk
(t)) ajk0 bjk0 (t)(1 − rjk
0 (t))
∗
2
(1 − rj (t))
Using these abbreviations, the first and second derivatives of the log
likelihood have the same formulation as described in 6.17.6.2.
Initial Estimates. By default, TDA uses the default starting values for
an exponential null model. Similar to the logistic regression model, there
can occur numerical difficulties if duration dependence is specified by a
higher degree polynomial.
Example 1 To illustrate estimation of a complementary log-log model
we use command file rtd2.cf (not shown). This command file identical
to rtd1.cf (see 6.17.6.2), only the model number was changed into 21.
Part of the standard output is shown in Box 1.
The TDA example archive contains another command file, rtd2m.cf,
which can be used to estimate a complementary log-log model for episodes
with alternative destination states.
Additional Remarks. As with the logistic regression mode, TDA cannot calculate generalized residuals for the complementary log-log model
and the pres option will be ignored. Also the rrisk and prate parameters are ignored.
6.17.6.3
complementary log-log models
Box 1 Part of standard output from command file rtd2.cf
Model: Discrete time complementary log-log, degree 0.
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Number of model parameters: 4
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Changed scaling factor for log-likelihood: -0.001
Using default starting values.
Convergence reached in 5 iterations.
Number of function evaluations: 6 (6,6)
Maximum of log likelihood: -2472.42
Norm of final gradient vector: 1.31763e-07
Last absolute change of function value: 3.60281e-11
Last relative change in parameters: 3.83396e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
--------------------------------------------------------------1 1
0
1 A Constant -5.0081
0.0843
-59.4116 1.0000
2 1
0
1 A COHO2
0.5371
0.1120
4.7959 1.0000
3 1
0
1 A COHO3
0.6773
0.1152
5.8768 1.0000
4 1
0
1 A W
0.5094
0.0942
5.4044 1.0000
Log likelihood (starting values): -2511.4459
Log likelihood (final estimates): -2472.4212
3
6.17.7
1
semi-parametric rate models
6.17.7 Semi-parametric Rate Models
This chapter describes a semi-parametric transition rate model proposed
by Cox [1972].
6.17.7.1 Partial Likelihood Estimation
6.17.7.2 Time-dependent Covariates
6.17.7.3 Episode Splitting
6.17.7.4 The Proportionality Assumption
6.17.7.5 Stratification
6.17.7.6 The Baseline Rate
d061707.tex
April 21, 1998
6.17.7.1
1
partial likelihood estimation
6.17.7.1 Partial Likelihood Estimation
The transition rate models described in 6.17.3 are based on parametrical
assumptions about the distribution of episode durations. This allows
for straightforward maximum likelihood estimation. However, one not
always has sufficient reasons to choose a specific model. The best strategy
seems then to use a variety of different model specifications and to look
for similar outcomes.
An alternative was proposed by Cox [1972] and subsequently discussed by many other authors. For the single transition case, and based
on a continuous time variable t ≥ 0, the model, generally called a Cox
model, is
r(t) = h(t) exp (A(t) α)
(1)
The transition rate r(t) depends on an unspecified baseline rate, h(t),
and on a vector of covariates A(t) with coefficients α.1 The covariates
may depend on the process time, t.
Implementation. Implementation of the Cox model in TDA is based
on the following model formulation.
n
o
rjk (t) = hjk (t) exp A(jk) (t) α(jk)
(2)
rjk (t) is the transition rate at time t for the transition from origin state
j to destination state k. hjk (t) is the unspecified baseline rate for the
same transition.2 A(jk) (t) is a (row) vector of covariates, specified for
the transition (j, k), and α(jk) is a vector of associated coefficients. The
covariates may have time-dependent values, examples will be given in
6.17.7.2 and 6.17.7.3.
Furthermore, it is possible to explain each transition by a specific set
of covariates. One should note, however, that model estimation requires
that all covariates are defined for all members of a sample of episodes.
1 We use the convention to write covariates as row vectors, the accompanying coefficients as column vectors.
2 The formulation implies that the baseline rates are allowed to vary between different
transitions. A special type of Cox models arises if the baseline rates are constrained
to be identical for all transitions; cf. Kalbfleisch and Prentice [1980], p. 170.
d06170701.tex
April 21, 1998
6.17.7.1
partial likelihood estimation
2
This is necessary to calculate the risks for all transitions, regardless of
the transition that actually happens at the end of an episode. One should
also note that it is necessary to include at least one covariate for each
transition because Cox models do not contain any intercept terms; all
constant effects are included in the baseline rates.
Model estimation is based on the method of partial likelihood (Cox
[1972], [1975]). To explain this method, we will first consider the case
of a single transition (j, k) from origin state j to destination state k.
Indices referring to transitions are dropped to simplify notation. With
the assumption of no ties, i.e. all ending times in the sample are different,
the partial likelihood, Lp , may be written as
Lp =
Y
exp(Ai (ti ) α)
l∈R(ti ) exp(Al (ti ) α)
P
i∈E
(3)
E denotes the set of all not censored episodes belonging to the (single)
transition (j, k), i.e. having an event at their ending time. R(ti ) is the
risk set at the ending time, say ti , of the ith episode contained in E. The
definition of the risk set is exactly the same as was given for the productlimit estimator: the set of all episodes, exiting with an event or censored,
with starting time less than ti and ending time equal to or greater than
ti . Ai (ti ) is the (row) vector of covariates for the ith episode (evaluated
at ti ), and α is the vector of associated coefficients to be estimated.
Given this notation, the calculation of the partial likelihood is easily
understood: One looks at the ending times of episodes ending with an
event, then, for each of these points in time, the risk set is created,
and finally the expression on the right-hand side of (3) is evaluated.
This expression may be interpreted as the probability that it is just
the ith individual to have an event at this point in time, given the risk
set containing all individuals who could have an event. Note that in
the calculation of these probabilities time-dependent covariates can be
accounted for in a simple way: at each successive point in time the actual
covariate values can be used.
As was shown by Cox and other authors, the partial likelihood defined in (3) may be treated as if it was a standard likelihood. Estimates
of model coefficients reached by maximizing the partial likelihood have
properties similar to those of standard maximum likelihood estimates.
The only difficulty arises if there are tied ending times. It is complicated, then, to calculate the partial likelihood exactly. Therefore several
simplifying approximations have been proposed. As is done, for instance,
6.17.7.1
partial likelihood estimation
3
by BMDP and SAS, the TDA algorithm for the partial likelihood calculation uses an approximation proposed by Breslow [1974].3
Assume there are di episodes, with indices contained in a set Di , all
having the same ending time, ti . The partial likelihood factor for this
point in time is approximated, then, by
exp(Si (ti ) α)
Y
exp(Aj (ti ) α)
P
idi =
l∈R(ti ) exp(Al (ti ) α)
j∈Di
l∈R(ti ) exp(Al (ti ) α)
hP
with Si (ti ) defined as the sum of the covariate vectors for the di episodes
with events at ti (evaluated at ti ). As is easily seen, using this approximation is already accounted for in the formulation of the partial likelihood
in (3), because in this formulation the product goes over all episodes,
tied or not, contained in E.
As is generally assumed, this approximation is sufficient if there are
only “relatively few” ties in a sample of episode data. If there are “too
many” ties it seems preferable to use a discrete time model. Box 1 shows
how the partial likelihood calculation is done in TDA. The algorithm
includes the possibility of stratification variables and time-dependent
covariates explained in later sections.
Alternative Destination States. To extend the Cox model for sets
of episode data with several transitions, we follow the general remarks in
3.3.1. First it should be restated that, in the context of duration analysis, all estimations are performed conditional on given origin states.
Therefore, with two or more origin states, the (partial) likelihood may
be built separately for all origin states; each single term may be maximized separately, or the product of all origin-specific terms is taken and
maximized simultaneously.4
Also the case of two or more transitions, from a given origin state,
is simple.5 One only needs an appropriate definition of the risk set. The
risk set is now conditional on a given origin state, so we shall write Rj (t);
but one has to include all episodes, with given origin state j, that are at
3 We follow the formulation given by Lawless [1982], p. 346. An introduction to Breslow’s approximation may also be found in Namboodiri and Suchindran [1987], p. 213.
4 The algorithm in TDA maximizes the partial likelihood simultaneously for all transitions to provide for the possibility of constraints on model parameters across transitions. In this case the transition-specific terms of the likelihood are no longer independent, and the maximization must be done simultaneously.
5 Cf. Kalbfleisch and Prentice [1980], p. 169.
6.17.7.1
partial likelihood estimation
Box 1 Partial Likelihood Algorithm in TDA
For all transitions: tran = 1,...,NT {
org = origin state of transition tran
des = destination state of transition tran
For all groups: g = 1,...,NG {
time = 0
For all episodes: ep = 1,...,NE (according to their ending times) {
org1 = origin state of episode ep
des1 = destination state of episode ep
tf = ending time of episode ep
If (org1 = org and des1 = des and ep belongs to group g) {
If (no time-dependent covariates) {
Add to the risk set all episodes which belong to
group G, have origin state org, and starting time
less than tf.
}
If (tf greater than time) {
If (no time-dependent covariates) {
Eliminate from the risk set all episodes which
belong to group g, have origin state org,
and have ending time less than tf.
}
If (time-dependent covariates) {
Create a new risk set: all episodes that belong
to group g, have origin state org, starting
time less than tf and ending time greater or
equal to tf. While creating this new risk
set, use the values of time-dependent covariates
at tf.
}
time = tf
}
Update partial likelihood (and derivatives) with
episode ep.
}
}
}
}
Note: If no groups (strata) are defined, this algorithm assumes that there is
exactly one group containing all episodes.
4
6.17.7.1
partial likelihood estimation
5
risk for a transition to state k at time t, regardless of which destination
state they actually will reach at some later point in time. Therefore, the
risk set Rj (t) is defined as the set of all episodes with origin state j
provided that their starting time is less than t and their ending time is
equal to or greater than t.
The partial likelihood in the case of possibly more than one origin
and/or destination state may then be written as
Y Y Y
exp A(jk) (ti ) α(jk)
p
P
L =
(4)
(jk) (t ) α(jk)
l
l∈Rj (ti ) exp A
j∈O k∈D i∈E
j
jk
Box 1 shows the implementation of this partial likelihood calculation in
TDA. If there is only one transition this reduces to the partial likelihood
given in (3). Also the approximation, if there are tied ending times, is
the same as in the single transition case.
Maximizing the Partial Likelihood. Maximum partial likelihood
estimates are found by maximizing (4). Dropping indices referring to
transitions and time-dependence of covariates, the logarithm of (4) may
be written as
(
)
X
X
p
` =
Ai α − log
exp(Al α)
(5)
i∈E
l∈Ri
Denoting by Ai,j the jth element of the covariate vector for the episode
i ∈ E, the first and second derivatives are
P
X
exp(Al α) Al,ja
∂`p
Pi
=
Ai,ja − l∈R
∂αja
l∈Ri exp(Al α)
i∈E
P
P
X
∂ 2 `p
l∈Ri exp(Al α) Al,ja
l∈Ri exp(Al α) Al,ka
P
P
=
−
∂αja ∂αka
l∈Ri exp(Al α)
l∈Ri exp(Al α)
i∈E
P
l∈Ri exp(Al α) Al,ja Al,ka
P
l∈Ri exp(Al α)
Example 1 To illustrate partial likelihood estimation with TDA, the
first example will be based on our main example data set, rrdat.1,
described in 3.3.3. For this example we only consider a single transition
and select two covariates: birth cohort and sex. Estimation is done with
command file pl1.cf, basically identical with the command files used in
previous chapters. Estimation results are shown in Box 3.
6.17.7.1
partial likelihood estimation
6
Box 2 Command file pl1.cf
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
COHO1
COHO2
COHO3
W
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
ge(TB,348) & le(TB,384), # birth
ge(TB,468) & le(TB,504), # birth
ge(TB,588) & le(TB,624), # birth
SEX[2],
# women
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
cohort 1
cohort 2
cohort 3
= 1
# destination state
);
edef(
ts = 0,
tf = DUR,
org = 0,
des = DES,
);
#
#
#
#
starting time
ending time
origin state
destination state
rate (
xa(0,1) = COHO2,COHO3,W,
) = 1;
model number 1 for Cox model
Example 2 For a second example, we distinguish three destination
states. The command file is pl1m.cf, estimation results are shown in
Box 4. Actually, one would get the same result if estimating separate
models for each of the three destination states taking into account all
other episodes as censored.
Example 3 Lawless ([1982], p. 367) has given an example for the Cox
model with some biometrical data. This example can be replicated with
6.17.7.1
partial likelihood estimation
Box 3 Output from command file pl1.cf
Model: Cox (partial likelihood)
Maximum Likelihood Estimation.
Algorithm 5: Newton (I)
Number of model parameters: 3
Type of covariance matrix: 2
Maximum number of iterations: 20
Convergence criterion: 1
Tolerance for norm of final gradient: 1e-06
Mue of Armijo condition: 0.2
Minimum of step size value: 1e-10
Scaling factor: -1
Log-likelihood of exponential null model: -2514.02
Scaling factor for log-likelihood: -0.001
Using default starting values.
Sorting episodes according to ending times.
Convergence reached in 4 iterations.
Number of function evaluations: 5 (5,5)
Maximum of log likelihood: -2565.43
Norm of final gradient vector: 8.94115e-09
Last absolute change of function value: 1.45822e-11
Last relative change in parameters: 5.14129e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A COHO2
0.3797
0.1134
3.3501 0.9992
2 1
0
1 A COHO3
0.4585
0.1177
3.8937 0.9999
3 1
0
1 A W
0.4078
0.0949
4.2988 1.0000
Log likelihood (starting values): -2584.5701
Log likelihood (final estimates): -2565.4257
command file pl2.cf, contained in the TDA example archive.
7
6.17.7.1
partial likelihood estimation
Box 4 Part of output from command file pl1m.cf
Model: Cox (partial likelihood)
Convergence reached in 4 iterations.
Number of function evaluations: 5 (5,5)
Maximum of log likelihood: -2550.89
Norm of final gradient vector: 1.07487e-06
Last absolute change of function value: 1.9399e-09
Last relative change in parameters: 0.000489307
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------------1 1
0
1 A COHO2
0.0893
0.2640
0.3381 0.2647
2 1
0
1 A COHO3
0.1624
0.2696
0.6024 0.4531
3 1
0
1 A W
-0.2411
0.2352
-1.0254 0.6948
------------------------------------------------------------------4 1
0
2 A COHO2
0.5017
0.1641
3.0564 0.9978
5 1
0
2 A COHO3
0.5487
0.1714
3.2010 0.9986
6 1
0
2 A W
0.1879
0.1387
1.3547 0.8245
------------------------------------------------------------------7 1
0
3 A COHO2
0.3726
0.1964
1.8966 0.9421
8 1
0
3 A COHO3
0.4909
0.2039
2.4069 0.9839
9 1
0
3 A W
1.0651
0.1678
6.3482 1.0000
Log likelihood (starting values): -2584.5701
Log likelihood (final estimates): -2550.8900
8
6.17.7.2
1
time-dependent covariates
6.17.7.2 Time-dependent Covariates
There is an easy way to include time-dependent covariates into Cox
models. The method is based on the fact that the partial likelihood
calculation goes gradually through all points in time where at least one of
the uncensored episodes has an event. It is possible, then, to re-evaluate
the values of time-dependent covariates at these time points. Actually,
this is already provided for in the partial likelihood formulas given in
6.17.7.1. As shown by the algorithm in Box 6.17.7.1-1 the risk set is
re-calculated at every new point in time.
To define time-dependent covariates, TDA provides a special operator, time. Note that this operator is only valid for type 5 variables defined
inside the edef command. All type 5 variables are re-evaluated during
partial likelihood estimation substituting time by the current process
time.
Example 1 To illustrate time-dependent covariates we continue with
Example 6.17.7.1-1 and investigate the hypothesis that marriage has
some influence on job duration. Box 1 shows command file pl3.cf using
a time-dependent dummy variable MARR. Estimation results are shown
in Box 2.
One should note that estimation results, when using time-dependent
covariates, can depend critically on the point in time when the variable
changes its value, in particular when the data are heavily tied. In general,
one should expect different results if using the gt or the ge operator
in defining the time-dependent variable.1 For practical applications, one
should also consider the introduction of lags, see the discussion of “effect
shapes” in Blossfeld et al. [1996].
1 Since process time in TDA is treated as a floating point number, there is also a
problem of rounding errors. Different from previous versions of TDA, the evaluation
of the logical operators ge, gt, etc. now uses a “safe” comparison that tries to minimize
rounding errors.
d06170702.tex
April 21, 1998
6.17.7.2
time-dependent covariates
Box 1 Command file pl3.cf
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
COHO1
COHO2
COHO3
W
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
ge(TB,348) & le(TB,384), # birth
ge(TB,468) & le(TB,504), # birth
ge(TB,588) & le(TB,624), # birth
SEX[2],
# women
cohort 1
cohort 2
cohort 3
= 1
MDATE = if le(TMAR,0) then 10000 else TMAR - TS, # marriage date
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
# destination state
);
edef(
ts = 0,
# starting time
tf = DUR,
# ending time
org = 0,
# origin state
des = DES, # destination state
MARR = gt(time,MDATE), # time-dependent dummy for married
);
rate (
xa(0,1) = COHO2,COHO3,W,MARR,
) = 1;
2
6.17.7.2
time-dependent covariates
Box 2 Part of standard output from command file pl3.cf
Model: Cox (partial likelihood)
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Maximum of log likelihood: -2562.51
Norm of final gradient vector: 1.73888e-08
Last absolute change of function value: 2.16271e-11
Last relative change in parameters: 7.73306e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
----------------------------------------------------------------1 1
0
1 A COHO2
0.3920
0.1135
3.4548 0.9994
2 1
0
1 A COHO3
0.4668
0.1179
3.9604 0.9999
3 1
0
1 A W
0.3947
0.0950
4.1543 1.0000
4 1
0
1 A MARR
-0.2408
0.1000
-2.4083 0.9840
Log likelihood (starting values): -2584.5701
Log likelihood (final estimates): -2562.5089
3
6.17.7.3
episode splitting
1
6.17.7.3 Episode Splitting
A disadvantage of the standard method to include time-dependent variables into Cox models is that the partial likelihood calculation becomes
very time-consuming. The reason is that the risk set must be re-calculated, and the values of the covariates must be re-evaluated (including
comparisons with the process time), for every point in time where at
least one event occurs.
If time-dependent covariates change their values only at some discrete
points in time, one can use, instead, the method of episode splitting. The
original episodes are split at every time point where one of the timedependent covariates changes its value. Each of the original episodes is
substituted by a set of sub-episodes (splits) with appropriate values of
the covariates. The last of these splits has the same exit status as the
original episode, all other splits are regarded as right censored.
To use this method with partial likelihood maximization, it is only
necessary to be precise in the definition of the risk set. At every time
point t, the risk set should contain all episodes, or splits, which have
a starting time less than t and ending time greater than, or equal to t,
regardless of their exit status. Consequently, the risk set at every point in
time contains only a single split—associated with appropriate values of
the time-dependent covariates—for each of the original episodes. Clearly,
it is necessary that the partial likelihood algorithm takes into account
both the different ending and starting times of the episodes (splits). As
shown in Box 1 in 6.17.7.1, this is a general feature of TDA’s partial
likelihood algorithm. (The method is inspired by the program RATE
(Tuma [1980]) which, we presume, was the first using starting and ending
times for updating the risk set.)
Unfortunately, one cannot use TDA’s method of internal episode
splitting with the partial likelihood algorithm. This is due to the fact
that, to save storage space, episode splits are only virtually created. However, to be sorted according to starting and ending times, they must be
present in the program’s data matrix. Consequently, to use the method
of episode splitting to estimate Cox models, or analogously for productlimit estimates, one has to proceed in two steps.
1. The first step is to create a new data file containing split episodes
d06170703.tex
April 21, 1998
6.17.7.3
episode splitting
2
Box 1 Command file pl4.cf for episode splitting
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
COHO1
COHO2
COHO3
W
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
ge(TB,348) & le(TB,384), # birth
ge(TB,468) & le(TB,504), # birth
ge(TB,588) & le(TB,624), # birth
SEX[2],
# women
cohort 1
cohort 2
cohort 3
= 1
MDATE = if le(TMAR,0) then 10000 else TMAR - TS, # marriage date
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
# destination state
);
edef(
ts = 0,
# starting time
tf = DUR,
# ending time
org = 0,
# origin state
des = DES,
# destination state
split = MDATE, # split with MDATE
);
epdat(
v = COHO1,COHO2,COHO3,W,MDATE,
# additional variables
dtda = pl4.tda,
# description file
) = pl4.dat;
instead of the original ones. This can be done with the split parameter
in the edef command and then writing the new data file with the epdat
command. Time-dependent covariates can be generated during this first
step, but equally well based on the resulting output file containing the
episode splits.
2. In a second step, the data file containing the episode splits can be used
6.17.7.3
episode splitting
3
Box 2 First records in data file pl4.dat
ID SN NSPL SPL ORG DES
TS
TF
COHO1 COHO2 COHO3 W MDATE
-------------------------------------------------------------1
1
2
1
0
0
0.00 124.00
1
0
0
0 124
1
1
2
2
0
0 124.00 428.00
1
0
0
0 124
2
1
1
1
0
1
0.00 46.00
1
0
0
1 169
3
1
1
1
0
1
0.00 34.00
1
0
0
1 123
4
1
2
1
0
0
0.00 89.00
1
0
0
1
89
4
1
2
2
0
1 89.00 220.00
1
0
0
1
89
5
1
1
1
0
1
0.00 12.00
0
1
0
1 182
6
1
1
1
0
1
0.00 30.00
0
1
0
1 170
7
1
1
1
0
1
0.00 12.00
0
1
0
1 140
8
1
1
1
0
1
0.00 75.00
0
1
0
1 128
9
1
1
1
0
1
0.00 12.00
0
1
0
1
53
10
1
1
1
0
1
0.00 55.00
0
0
1
1
0
11
1
1
1
0
1
0.00 68.00
1
0
0
0 118
12
1
2
1
0
0
0.00 50.00
1
0
0
0
50
12
1
2
2
0
1 50.00 137.00
1
0
0
0
50
as if it was a set of “original” episodes. Based on the new input data, one
can estimate Cox models and, of course, all of the parametric transition
rate models discussed in 6.17.2 and 6.17.3. It is also possible to request
product-limit estimates based on a set of episode splits; one should get
exactly the same results as if one had used the original episodes.
Example 1 To illustrate the method of episode splitting, we replicate
Example 1 in 6.17.7.2. The first step is to create a new data file containing episode splits, with splitting at the date of marriage. This is
done with the command file pl4.cf shown in Box 1. There are three
commands. The first command (nvar) reads the data file and creates
the basic variables. The following edef command defines episode data
and requests internal episode splitting with variable MDATE. Finally, the
epdat command writes the new episode data into the output file pl4.dat
and creates a data description file with the dtda parameter. The first
records of the new data file, pl4.dat, are shown in Box 2. This new
data file is then used by command file pl5.cf shown in Box 3.1 The
nvar command reads the data file, creates the basic variables and, in
addition, the time-dependent dummy variable MARR. This variable gets
the value 1 for each split where the marriage data is less than, or equal
to the starting time of the split. The edef command specifies the episode
data structure and creates the time-dependent dummy variable MARR1 as
1 This
command file was created by using the description file pl4.tda.
6.17.7.3
episode splitting
4
Box 3 Command file pl5.cf
nvar(
dfile = pl4.dat,
noc = 761,
ID
[6.0] = c1, # id number
SN
[3.0] = c2, # spell number
NSPL [3.0] = c3, # number of splits
SPL [3.0] = c4, # split number
ORG [3.0] = c5, # origin state
DES [3.0] = c6, # destination state
TS
[6.2] = c7, # starting time
TF
[6.2] = c8, # ending time
COHO1 [0.0] = c9,
COHO2 [0.0] = c10,
COHO3 [0.0] = c11,
W [0.0] = c12,
MDATE [0.0] = c13,
MARR = le(MDATE,TS),
);
edef(
ts = TS,
# starting time
tf = TF,
# ending time
org = ORG, # origin state
des = DES, # destination state
MARR1 = le(MDATE,ts),
);
rate (
# without time-dep covariates
xa(0,1) = COHO2,COHO3,W,
) = 1;
rate (
# using episode splitting
xa(0,1) = COHO2,COHO3,W,MARR,
) = 1;
rate (
# using standard approach
xa(0,1) = COHO2,COHO3,W,MARR1,
) = 1;
a type 5 variable, meaning that this variable will be re-evaluated when
TDA estimates a Cox model which contains this variable. Then follow
three rate commands.
1. The first one re-estimates the model discussed in Example 6.17.7.11. Although model estimation now uses split episodes, one should get
identical estimation results.
2. The second rate command re-estimates the model discussed in Example 6.17.7.2-1, using now the method of episode splitting for the time-
6.17.7.3
episode splitting
5
dependent variable MARR.
3. The third rate command estimates the same model, but uses the
conventional method for time-dependent covariates in Cox models; the
model specification contains now MARR1 instead of MARR. Since MARR1 is a
type 5 variable, it is re-evaluated whenever the risk set must be updated.
Consequently, the estimation procedure is much more time consuming.
6.17.7.4
1
proportionality assumption
6.17.7.4 The Proportionality Assumption
A basic feature of the Cox model is that transition rates for different
values of covariates are proportional. If, for instance, we compare the
transition rates r1 (t) and r2 (t), corresponding to the values of the ith
covariate, Ai and A0i , then
r1 (t)
= exp {(Ai − A0i ) αi }
r2 (t)
Models having this feature are called proportional transition rate models.
This is, of course, not specific for the Cox model but also implied in
at least some versions of most of the parametric models discussed in
6.17.3. This section discusses three methods to check whether a sample
of episode data allows for this proportionality assumption.
Graphical Methods. If there are only a few categorical covariates in
a model, it is possible to perform simple graphical checks. Assume that
this is the case, and that A and A0 are two different vectors with covariate values. The sample can then be split into two groups, and one can
calculate nonparametric Kaplan-Meier estimates of the survivor functions, say G(t) and G0 (t), for both of these groups. The proportionality
assumption implies that
log (G(t))
= exp {(A − A0 ) α}
log (G0 (t))
Or, taking logarithms once more, this becomes
log {− log (G(t))} = log {− log (G0 (t))} + (A − A0 ) α
(1)
Therefore, a plot of the logarithms of the survivor functions may be used
to check if the proportionality assumption is (nearly) admissible. Similar
ideas can be used to check parametric model assumptions, see Wu [1990].
Testing Time-dependence. Another method to check the proportionality assumption is to look at possible interaction effects between
covariates and process time. Assuming that the ith component of the
covariate vector A shall be tested, a model like the following could be
d06170704.tex
April 21, 1998
6.17.7.4
proportionality assumption
2
Box 1 Command file pl6.cf
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
COHO1
COHO2
COHO3
W
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
ge(TB,348) & le(TB,384), # birth
ge(TB,468) & le(TB,504), # birth
ge(TB,588) & le(TB,624), # birth
SEX[2],
# women
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
cohort 1
cohort 2
cohort 3
= 1
# destination state
);
edef(
ts = 0,
# starting time
tf = DUR,
# ending time
org = 0,
# origin state
des = DES, # destination state
WTEST = W * (log(time) - 4.22),
);
rate (
xa(0,1) = COHO2,COHO3,W,WTEST,
) = 1;
set up and estimated.
r(t) = h(t) exp {A α + Ai (log(time) − M ) αi0 }
(2)
The constant M is added to facilitate the estimation. It should be the
logarithm of the mean (or median) duration of the sample episodes used
for the estimation. If the added variable gives a significant coefficient, i.e.,
if there are significant interaction effects, the proportionality assumption
for this covariate is probably not in order.
6.17.7.4
proportionality assumption
3
Box 2 Part of standard output from command file pl6.cf
Maximum of log likelihood: -2562.9
Norm of final gradient vector: 6.79524e-07
Last absolute change of function value: 1.71305e-09
Last relative change in parameters: 0.00104722
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
-------------------------------------------------------------1 1
0
1 A COHO2
0.3816
0.1133
3.3682 0.9992
2 1
0
1 A COHO3
0.4381
0.1180
3.7136 0.9998
3 1
0
1 A W
0.5840
0.1215
4.8074 1.0000
4 1
0
1 A WTEST
0.2196
0.0974
2.2541 0.9758
Log likelihood (starting values): -2584.5701
Log likelihood (final estimates): -2562.9031
Example 1 To illustrate this procedure, we check variable W (sex=2,
women) that was used in Example 1 in 6.17.7.1. The command file is now
pl6.cf shown in Box 1. It is mostly identical to command file pl1.cf,
but we have added the time-dependent covariate WTEST. Simply following
(4), we use the definition
WTEST = W * (log(time) - 4.22),
where 4.22 is approximately the logarithm of the mean duration in our
example data set. Estimation results are shown in Box 2. Obviously,
variable W creates some conflicts with the proportionality assumption.
A Goodness-of-Fit Test. As proposed by Moreau et al. [1985], the
idea of checking for interactions between covariates and the process time
can be generalized for a global goodness-of-fit test for the Cox model.
We briefly describe this proposal and its implementation in TDA. Is is
assumed that all episodes start at the origin of the process time, i.e.
have starting time zero. The process time axis is divided into intervals
according to
0 ≤ τ1 < τ2 < . . . < τq ,
τ0 = 0,
τq+1 = ∞
The lth time interval is defined as
Il = {t | τl ≤ t < τl+1 }
l = 1, . . . , q
Moreau et al. [1985] propose to look at a general model formulation that
allows possibly different covariate effects in the time intervals. In the
6.17.7.4
4
proportionality assumption
single transition case, the model can be formulated as
r(t) = h(t) exp{A(α + γl )}
if t ∈ Il
(3)
In the lth time interval, the coefficients associated with the covariate
vector A are given by α + γl . If the proportionality assumption holds,
then all γl vectors should be zero. Since one of these vectors is obviously
redundant, we set γ1 = 0, and the null hypothesis becomes
γ2 = γ3 = . . . = γq = 0
(4)
For testing this hypothesis, Moreau et al. [1985] propose a score test.
The test statistic, say S, is defined by
S = U 0 V −1 U
where U is a vector of first derivatives of the log-likelihood of model
(3), evaluated under the null hypothesis (4) and with maximum partial
likelihood estimates of the vector α. And V is the observed information
matrix, i.e. minus the matrix of second derivatives of the log-likelihood
of (3), evaluated the same way. Under the null hypothesis, the statistic
S is asymptotically χ2 distributed with degrees of freedom equal to the
number of parameters of model (3), that is (q − 1) p where p is the
dimension of α.
As shown by Moreau et al. [1985], the calculation of S can be greatly
simplified. It is possible to write the test statistic as
S=
q
X
Ul0 Vl−1 Ul
l=1
with Ul and Vl restricted to the lth time interval. The elements of Ul
and Vl (s, s1 , s2 = 1, . . . , p) may then be written as
P
X
j∈R exp (Aj α̂) Aj,s
Ul,s =
Ai,s − P i,l
j∈Ri,l exp (Aj α̂)
i∈El
P
Vl,s1 s2
=
X
j∈Ri,l
exp (Aj α̂) Aj,s1 Aj,s2
P
j∈Ri,l
i∈El
P
j∈Ri,l
P
exp (Aj α̂) Aj,s1
j∈Ri,l
−
exp (Aj α̂)
exp (Aj α̂)
P
j∈Ri,l
P
exp (Aj α̂) Aj,s2
j∈Ri,l
exp (Aj α̂)
6.17.7.4
proportionality assumption
5
These expressions are analogous to the derivatives of (5) given above,
restricted to single time periods. El is the set of episodes with events
in the lth time interval; Ri,l is the risk set at the ending time of the
ith episode in El , and α̂ is the maximum partial likelihood estimate of
α under the null hypothesis. As proposed by Moreau et al. [1985], the
formulation in (5) accounts for tied ending times by using Breslow’s
approximation.
A generalization to the case of multiple transitions is simple since
the log-likelihood factors into separate components for each transition.
Therefore, a separate test statistics can be calculated for each transition, say (j, k), taking into account all episodes with origin state j and
regarding all episodes as censored that do not end with destination state
k.
To request a calculation of the test statistics, one has to add a definition of time periods to the specification of a Cox model. The test
statistics is printed, then, into the standard output. In addition to the
test statistics and associated degrees of freedom, TDA prints the corresponding value of the χ2 distribution function.
If a time interval is empty, meaning that there are no episodes with
ending times in this interval, it is not used for the calculation of the
test statistic and not counted in the degrees of freedom. Sometimes,
mainly depending on the definition of time intervals, one or more of the
Vl matrices is rank-deficient and cannot be inverted. The test statistic is
not calculated, then, but some information about the underlying problem
is given, on request, in the protocol file.
Example 2 To illustrate the test procedure, we first reproduce an example given by Moreau et al. [1985]. The data, rrdat.5, are survival
times of gastric carcinoma patients who are given two different therapies. The command file, pl7.cf, is shown in Box 3. Part of the standard
output is shown in Box 4. The test statistic is significant, indicating that
the effect of the two therapies (groups) is not proportional.
Example 3 For a second illustration, we add the parameter
tp = 0 (12) 96,
to command files pl1.cf pl1m.cf, see the first two examples in 6.17.7.1.
The new command files are pl8.cf and pl8m.cf, not shown, but contained in the example archive. Box 5 shows the test statistics created
by these command files. In the first example, without alternative destination states, the test statistic is 37.108 with 24 degrees of freedom.
6.17.7.4
proportionality assumption
6
Box 3 Command file pl7.cf
nvar(
dfile = rrdat.5,
TF
DES
G1
G2
=
=
=
=
c1,
c2,
c3[1],
c3[2],
# data file
# ending time = duration
# destination state (= status)
# definition of two groups
);
edef(
ts = 0,
# starting time = 0
tf = TF,
# ending time = duration
org = 0,
# origin state = 0
des = DES,
# destination state (= status)
);
rate(
xa(0,1) = G1,
# included: group 1
tp = 0,170.1,354.1,535.1,
# time periods
) = 1;
Box 4 Part of standard output from command file pl7.cf
Maximum of log likelihood: -282.744
Norm of final gradient vector: 9.13651e-08
Last absolute change of function value: 3.13773e-07
Last relative change in parameters: 0.0116559
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
------------------------------------------------------------1 1
0
1 A G1
-0.2666 0.2332
-1.1428 0.7469
Log likelihood (starting values):
Log likelihood (final estimates):
Global Goodness-of-fit.
-283.3952
-282.7441
SN Org Des
TStat
DF
Signif
---------------------------------------1
0
1
10.2351
3
0.9833
As expected, it shows a significant deviation from the proportionality
assumption. Again, this is mainly due to the W variable. Omitting this
variable from the model gives a test statistic of 21.04 with 16 degrees of
freedom which is no longer significant.
6.17.7.4
proportionality assumption
Box 5 Test statistics created by command files pl8.cf and pl8m.cf
pl8.cf (corresponding to pl1.cf)
---------------------------------------SN Org Des
TStat
DF
Signif
---------------------------------------1
0
1
37.1084
24
0.9573
pl8m.cf (corresponding to pl1m.cf)
---------------------------------------SN Org Des
TStat
DF
Signif
---------------------------------------1
0
1
15.9066
24
0.1086
1
0
2
25.5454
24
0.6234
1
0
3
38.5791
24
0.9698
7
6.17.7.5
stratification
1
6.17.7.5 Stratification
If the proportionality assumption is not acceptable, it is sometimes sensible to estimate a stratified model. This is possible if the covariates that
cause conflicts with the proportionality assumption are categorical. The
whole sample can then be split into groups (strata), one group for each
of the possible combinations of categories. In the simplest case, with only
a single dummy variable, there will be only two groups.
Let G denote the set of groups. The model should be specified such
that the baseline rate can be different for each group. There are two
possibilities. One possibility is to define a different model for each group;
then not only the baseline rates, but also the covariate effects can vary
across groups.
This approach requires fairly large samples. When data sets are relatively small, or when there are no significant interaction effects, it can be
sensible to build on the assumption that covariate effects are the same
in all groups. (See Kalbfleisch and Prentice [1980], p. 87; Lawless [1982],
p. 365; Blossfeld et al. [1989].) Based on this assumption, a model for the
single transition case can be written as
rg (t) = hg (t) exp(A α)
g∈G
To estimate this model, the partial likelihood has to be calculated as
the product of the group-specific likelihoods. For the multiple transition
case, this may be written as
Y Y Y Y
exp A(jk) (ti ) α(jk)
p
P
L =
(jk) (t ) α(jk)
l
l∈Rj,g (ti ) exp A
j∈O k∈D g∈G i∈E
j
j
jk,g
with Gj denoting the set of groups of episodes with origin state j. Ejk,g is
the set of all not censored episodes having origin state j and destination
state k and belong to group g. Accordingly, Rj,g (t) is the risk set at
time t containing all episodes with origin state j, starting time less than
t, ending time equal to or greater than t, and belonging to group g. Box
1 in 6.17.7.1 shows how this is implemented in TDA’s partial likelihood
algorithm.
Example 1 To illustrate stratified estimation, we continue with the
first example in 6.17.7.1 and use the variable SEX for stratification. Box
d06170705.tex
April 21, 1998
6.17.7.5
2
stratification
Box 1 Command file pl9.cf for stratified estimation
nvar(
dfile = rrdat.1,
ID
SN
TS
TF
SEX
TI
TB
TE
TMAR
PRES
PRESN
EDU
[3.0]
[2.0]
[3.0]
[3.0]
[2.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[3.0]
[2.0]
COHO1
COHO2
COHO3
G1
G2
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
c1,
c2,
c3,
c4,
c5,
c6,
c7,
c8,
c9,
c10,
c11,
c12,
# data file
#
#
#
#
#
#
#
#
#
#
#
#
identification number
spell number
starting time
ending time
sex (1 men, 2 women)
interview date
birth date
entry into labor market
marriage date (0 if no marriage)
prestige of current job
prestige of next job
highest educational attainment
ge(TB,348) & le(TB,384), # birth cohort 1
ge(TB,468) & le(TB,504), # birth cohort 2
ge(TB,588) & le(TB,624), # birth cohort 3
SEX[1],
# group1 (men)
SEX[2],
# group2 (women)
DES [1.0] = if eq(TF,TI) then 0 else 1,
DUR [3.0] = TF - TS + 1,
# destination state
);
edef(
ts = 0,
# starting time
tf = DUR,
# ending time
org = 0,
# origin state
des = DES, # destination state
);
rate (
xa(0,1) = COHO2,COHO3,
grp = G1,G2,
# groups for stratification
) = 1;
1 shows the command file pl9.cf. The additional variables G1 and G2
define two groups, men and women.1 To request stratified estimation,
one has to use the parameter
grp = G1,G2,...,
in the rate command. TDA then assumes that the right-hand side is
1 In
TDA, groups are always defined by indicator variables. Nonzero values indicate
that a case belongs to the respective group.
6.17.7.5
stratification
3
Box 2 Part of standard output from command file pl9.cf
Model: Cox (partial likelihood)
Stratified with 2 groups.
Maximum likelihood estimation.
Algorithm 5: Newton (I)
Convergence reached in 4 iterations.
Number of function evaluations: 5 (5,5)
Maximum of log likelihood: -2250.58
Norm of final gradient vector: 8.82782e-09
Last absolute change of function value: 1.43084e-11
Last relative change in parameters: 4.27658e-05
Idx SN Org Des MT Variable
Coeff
Error
C/Error Signif
-------------------------------------------------------------1 1
0
1 A COHO2
0.3840
0.1133
3.3878 0.9993
2 1
0
1 A COHO3
0.4428
0.1183
3.7433 0.9998
Log likelihood (starting values): -2259.5669
Log likelihood (final estimates): -2250.5832
a list of indicator variables specifying groups and performs the stratified estimation procedure based on these groups. Estimation results are
shown in Box 2. As is seen the cohort effects have not changed very
much. This is consistent with that both variables do not give significant
results when tested for proportionality.
6.17.7.6
1
baseline rate
6.17.7.6 The Baseline Rate
The partial likelihood method provides estimates of the parameters of
a Cox model but no direct estimate of the underlying baseline rate.
Clearly, this would be useful in order to provide some information about
the structure of time-dependence and could be used, for instance, to
check whether also a fully parametric model could be appropriate.
There are different proposals for estimators of the baseline rate.
TDA’s approach is based on a proposal by Breslow [1974], discussed also
in Blossfeld et al. [1989]. To explain the method, we begin with assuming
a single transition where all episodes have starting time zero. Then, in
the usual notation, E denotes the set of, say q, not censored episodes,
and the ordered ending times may be given by
τ1 < τ2 < . . . < τq
Let Ei be the number of events at τi , let Ri be the risk set at τi . Furthermore, let α̂ be the partial likelihood estimate of the model parameters.
Then, defining τ0 = 0, we may regard
ĥbi =
(τi − τi−1 )
E
P i
l∈Ri
exp(Al α̂)
i = 1, . . . , q
as an estimate of the baseline rate of the model. It is a constant during
each interval (τi−1 , τi ], resulting in a step function with steps at the
points in time where at least one event occurs. This step function can
be integrated to provide an estimate of the cumulative baseline rate.
Ĥ b (t) =
i
X
(τl − τl−1 ) ĥl + (t − τi ) ĥi+1
τi < t ≤ τi+1
l=1
The resulting estimate of the baseline survivor function is
n
o
Ĝb (t) = exp −Ĥ b (t)
Finally, because of the assumption of proportional effects, the cumulative
transition rate for an arbitrary covariate vector A can be estimated by
Ĥ(t) = Ĥ b (t) exp (Aα̂)
d06170706.tex
(1)
April 21, 1998
6.17.7.6
baseline rate
2
Box 1 rate command in command file pl1r.cf
rate (
xa(0,1) = COHO2,COHO3,W,
prate (COHO3=1) = r.dat,
prate (COHO3=1,W=1) = r.dat,
) = 1;
and the corresponding survivor function estimate is
n
o
exp(Aα̂)
Ĝ(t) = exp −Ĥ b (t) exp (Aα̂) = Ĝb (t)
(2)
TDA uses these formulas to provide estimates of the cumulative base-
line transition rate and the corresponding baseline survivor function.
A generalization to the case of multiple transitions is done analogously
to a product-limit estimation for several transitions: each transition is
treated separately. For instance, with respect to the transition (j, k),
the procedure outlined above is applied to all episodes with origin state
j; all episodes not ending in destination state k are regarded as censored. The resulting estimates are then pseudosurvivor functions and
transition-specific cumulative rates, respectively.
To request an estimation of cumulated (baseline) rates, one can use
the prate option described in 6.17.1.4. Note that TDA calculates cumulated rates. To get estimates of the rates, one should first smooth the
cumulated rates and then use numerical differentiation.
Example 1 To illustrate the calculation of cumulative baseline rates,
we use command file pl1r.cf (not shown). It is basically identical with
command file pl1.cf (Example 1 in 6.17.7.1). Two prate parameters
have been added to the rate command as shown in Box 1. The first
prate parameter calculates an estimate of the cumulated rate with the
covariate values COHO2=0, COHO3=1, and W=0. The table is written into the
output file r.dat. The second prate parameter creates a second table,
written to the same output file, for covariate values COHO2=0, COHO3=1,
and W=1. Part of the first table in output file r.dat is shown in Box 2.
6.17.7.6
baseline rate
Box 2 Part of output file r.dat
# Cox Model. Baseline Rate Calculation.
# Idx SN Org Des MT Variable
Coeff Covariate
# -----------------------------------------------#
1 1
0
1 A COHO2
0.3797
0.0000
#
2 1
0
1 A COHO3
0.4585
1.0000
#
3 1
0
1 A W
0.4078
0.0000
# Transition: SN 1
Org 0
Des 1
# ID
Time
Events
Censored
Risk Set
Surv.F.
Cum.Rate
# ---------------------------------------------------------------0
0.0000
0.0000
0.0000
600.0000
1.0000
0.0000
0
2.0000
2.0000
0.0000
600.0000
0.9967
0.0033
0
3.0000
5.0000
1.0000
597.0000
0.9885
0.0115
0
4.0000
9.0000
2.0000
590.0000
0.9737
0.0266
0
5.0000
3.0000
0.0000
581.0000
0.9688
0.0317
0
6.0000 10.0000
1.0000
577.0000
0.9523
0.0488
3
6.18
regression models for events
1
6.18 Regression Models for Events
This chapter describes a simple approach to model conditions for the
occurrence of events, based on sequence data (see 3.4). The sections are:
6.18.1
A Simple Modelling Approach
6.18.2
Investigating Events
6.18.3
Data Generation
d0618.tex
April 21, 1998
6.18.1
1
a simple modeling approach
6.18.1 A Simple Modeling Approach
We assume a K-dimensional sequence data structure (see 3.4). (yik )
denotes the sequence of the ith individual (sample member) in the kth
sequence data structure. We are interested in a model that shows how
the occurrence of events depends on certain conditions. Such a model
can be constructed for each type of event. To simplify the approach, we
focus on just one event, say [k, s1 , s2 ], that is, a transition from state
s1 to state s2 in the kth state space (sequence data structure). In the
following, this will be called the target event. We can then construct an
indicator variable
1 if yi,t−1 = s1 and yi,t = s2
Zit =
0 otherwise
Zit is 1 if, for individual i, the target event occurs at t. In general, we
will say that an event occurs at t if the corresponding transition occurs
between t − 1 and t. This implies that we can observe events only in a
time interval [ta , tb ] with
Tkmin < ta ≤ tb ≤ Tkmax
(Tkmin and Tkmax define the time axis for the kthe sequence data structure,
see 3.4.2.) Of course, to set up a model we are free to choose any subset
Tm ⊂ [ta , tb ]. Note that Tm can be a calendar time axis or a process time
axis.
Focus of the model is the conditional probability for the occurrence of
the target event. For a time point t ∈ Tm , a general parametric approach
can be written as
Pr (Zit = 1 | xit ) = g(t, xt , θt )
(1)
g(.) is some function, constrained to 0 ≤ g(.) ≤ 1, depending on time
t, a time-dependent vector of covariates, xt , and a parameter vector,
θt . Model specification will then consist of two parts. First, to choose
a specification of the function g(.). The most simple choice would be a
logistic regression specification. Second, and in a sense more important,
is then a specification of the covariates. This will be discussed below.
d061801.tex
April 21, 1998
6.18.1
a simple modeling approach
2
To estimate the model, one needs the concept of a risk set. Consistent
with our definition of events, the risk set Rt for the target event [k, s1 , s2 ]
contains all sample members who are in state s1 at time t − 1 and have
some valid state at t. Using zit to denote realized values of Zit , the
log-likelihood of model (1) can then be written as
X
`t =
zit g(t, xt , θt ) + (1 − zit )(1 − g(t, xt , θt ))
(2)
i∈Rt
So far, we have separate models for each time point t ∈ Tm . To arrive at
a single model for the whole time axis Tm , we can simply consider and
estimate the models simultaneously. The log-likelihood then becomes
X X
`=
zit g(t, xt , θt ) + (1 − zit )(1 − g(t, xt , θt ))
(3)
t∈Tm i∈Rt
This allows to consider constraints for the time-specific parameter vectors θt and should be seen as an essential part of the model specification.
Models of this kind can be easily estimated if one uses some standard
specification, e.g., a logit or probit model. The only difficulty consists in
the construction of sensible covariates. A causal modeling approach can
begin with the idea that, what happens at some time point t, can, in
principle, depend on any facts earlier than t. Here, we only shortly indicate some different kinds of covariates. How to construct these covariates
will be discussed in 6.18.3.
1. Without any covariates, (3) will give an estimate of a mean probability
for the occurrence of the target event during the time axis Tm . In many
applications one would like to consider also period effects. This can be
achieved by constructing dummy variables for each time point and then
estimating a single coefficient for each of these variables.
2. Also time-independent covariates can be specified as an identical part
in each of the covariate vectors xit , and it would normally suffice to
estimate a single effect for each of these variables.
3. More interesting are events in the past history of the process. Given
a time point t, one should consider all possibly important events which
occurred earlier than t. Let E denote the set of possibly important events.
A simple way would be to define, for each e ∈ E, a variable ei,t that
simply counts how often the event e occurred before t. Depending on the
kind of event and its frequency of occurrence, these variables can be used
directly as part of xit , or transformed into suitable dummy variables.
6.18.1
a simple modeling approach
3
4. In addition, one can consider the duration between the occurrence of
an event, say at t0 , and the current process time t. This could make sense
if the event should have happened for each individual before entering the
time axis of the model, Tm . The most important example is birth, and
the duration until t is simply age.
5. However, duration between occurrence of an event and current process
time t is only defined if the event has, in fact, occurred in the past of
t. In general, events in E are contingent and may, or may not, occur. It
seems preferable, then, to use the concept of an effect shape (instead of
duration).1 To provide a formulation, we will use (e, τ ) to denote events;
e ∈ E is the type of the event, and τ is the time when it occurred.2 For
each event (e, τ ), we can define an effect shape as a function of time in
the following way:
he (t − τ ; θe ) if t > τ
h(t; e, τ, θe ) =
0 otherwise
The idea is, given an event of type e occurred at τ < t, its possible
impact on the occurrence of the target event at t is given by the value of
its effect shape at t. In this way, one can consider the combined impact of
any number of events that happened before t; and, of course, this impact
dynamically changes while t runs through the time axis, Tm .
Unfortunately, we most often cannot justify a priori assumptions about
the functional form, he (.), of an effect shape. Instead, we would like to
get some evidence about effect shapes from the data. However, since
our time axis is discrete, we can simply consider, for each type of event
e ∈ E, a set of d + 1 dummy variables ei,t,(j) , j = 0, . . . , d, d ≥ 0; using
the definition
1 if there is an event (e, τ ) and t = τ + j
ei,t,(j) =
0 otherwise
The effects of these event-specific dummy variables can then provide
1 For a discussion of this concept, see Blossfeld and Rohwer [1995]. In a certain sense,
the approach can be viewed as a generalization of conventional transition rate modeling based on the duration of episodes. For some background discussion, see Mayer
and Huinink [1990], Rohwer [1996].
2 Notice that we should always distinguish between kinds of events, here given by
the set E, and ”‘concrete”’ events that actually occurred. For a discussion of this
distinction see, e.g., Galton [1984], ch. 2.
6.18.1
a simple modeling approach
4
some evidence about the effect shapes.3
How to construct these different kinds of covariates, based on a Kdimensional sequence data structure, will be discussed in 6.18.3.
3 For
an application of this approach to estimate the effect shape of pregnancies for
the target event marriage (in consensual unions), see Blossfeld et al. [1996].
6.18.2
1
investigating events
6.18.2 Investigating Events
Before trying to construct regression models for events, one should investigate the events actually occurring in a given sample of sequence data.
TDA provides two simple commands. The first command is seqev with
syntax shown in the following box.
seqev (
sn=...,
sel=...,
number of sequence data structure, def. 1
expression for sequence selection
) = fname;
All parameters are optional and have their usual meaning. If no file
name is given on the right-hand side, the command prints information
about the events in the selected sequence data structure into the standard output; otherwise into the file specified on the right-hand side. The
information about events is a table with three columns and a separate
line for each type of event. The first two columns indicate the type of
event (origin and destination state), and the third column shows how
often this event occurs in the selected sequence data structure.
Example 1 To illustrate the seqev command, we use the data file
seq.d4 shown in Box 1. There are two sequences for each of three individuals, both defined on the same time axis, t = 0, 1, . . . , 5; and in
addition, two variables, V1 and V2, which will be used for some illustrations in 6.18.3. Box 2 shows the command file, seq8.cf, that was used
to read the data, define two sequence data structures, and count the
number of events in the first one. Part of the standard output is shown
in Box 3. There are two types of events. The first one, (1, 2), occurs four
times, the second one, (2, 1) occurs three times.
d061802.tex
April 21, 1998
6.18.2
investigating events
Box 1 Sequence data file seq.d4
ID
-1
2
3
0 1 2 3 4 5
----------1 1 2 2 1 1
1 2 1 1 1 2
2 2 1 1 2 2
0 1 2 3 4 5
----------1 3 3 1 1 3
1 1 1 3 3 3
1 1 3 1 3 3
V1 V2
-- -1 -10
2 -15
2 -20
Box 2 Command file seq8.cf
nvar(
dfile = seq.d4,
ID
= c1,
Y{0,5} = c2,
S{0,5} = c8,
V1
= c14,
V2
= c15,
);
seqdef(sn=1) = Y0,,Y5;
seqdef(sn=2) = S0,,S5;
seq;
info about current sequence data
seqev;
count events in first sequence data structure
Box 3 Part of standard output from seq8.cf
Sequence
State
Time axis
Number
Structure Type Variables Minimum Maximum of States States
-------------------------------------------------------------1
1
6
0
5
2
1 2
2
1
6
0
5
2
1 3
Range of common time axis: 0 to 5.
Counting events.
Using sequence data structure 1.
Event type
Number
-------------------1
2
4
2
1
3
2
6.18.2
investigating events
3
The seqevd command. The second command is seqevd with syntax
shown in the following box.
seqevd (
sn=...,
sel=...,
dtda=...,
number of sequence data structure, def. 1
expression for sequence selection
request TDA description file
) = fname;
A file name must be given on the right-hand side, all other parameters
are optional and have their usual meaning. The command creates the
specified output file and writes into this output file a table showing,
for each time point, the number of events occurring at that time point.
We say that an event occurs at t if there is a corresponding transition
between t − 1 and t.
6.18.2
investigating events
4
Box 4 Illustration of seqevd command
Output file from command: seqevd(dtda=t)=d
Time Cases EV1_2 EV2_1 NEV
--------------------------1
3
1
0
1
2
3
1
2
3
3
3
0
0
0
4
3
1
1
2
5
3
1
0
1
Output file created by dtda=t
nvar(
dfile = d,
noc = 5,
TIME <5>[6.0] = c1 ,
NCAS <5>[6.0] = c2 ,
EV1_2 <5>[4.0] = c3 ,
EV2_1 <5>[4.0] = c4 ,
NEV <5>[6.0] = c5 ,
);
#
#
#
#
#
time
number of cases
number of events (1,2)
number of events (2,1)
total number of events
Example 2 To illustrate the structure of the output file, we add the
command
seqevd (dtda = t) = d;
to command file seq8.cf. Box 4 shows the resulting output file. The
column labelled Cases contains the number of sequences having a valid
state at t − 1 and t. The additional output file created by the dtda
parameter provides an nvar command to read the data file. This is useful,
for instance, if one wishes to create plots of the frequency distributions
of events.
6.18.3
data generation
1
6.18.3 Data Generation
This section describes how to create a data file which can subsequently be
used for estimating regression models for events, following the approach
discussed in 6.18.1. Having created such a data file, model estimation
can use any of the models for binary response variables described in 6.12.
The command to create a data file is seqmd with syntax shown in the
following box.
seqmd (
ev=...,
sel=...,
tp=...,
v=...,
xe=...,
dtda=...,
definition of target event
expression for sequence selection
definition of time axis
time-independent covariates
event-specific covariates
request TDA description file
) = fname;
With the exception of ev, all parameters are optional. The ev parameter
must be used to define a target event. The syntax is
ev = [k, s1 , s2 ],
where k is the number of a sequence data structure and s1 and s2 are
the origin and destination state of the event.
1. The parameter
sel = expression,
can be used to select a subset of sequences. The command then uses only
sequences (data matrix cases) where expression evaluates to a nonzero
value.
2. The command uses a time axis [ta , tb ] with ta = Tkmin + 1 and tb =
Tkmax by default. Alternatively, one can specify a time axis with the tp
parameter. The syntax is
tp = t1 , t2 , . . . , tn
d061803.tex
or
tp = t1 (d) t2
April 21, 1998
6.18.3
data generation
2
Box 1 Illustration of seqmd command
Command: seqmd (ev=[1,1,2]);
Selected target event: [1,1,2]
Using sequence data structure 1.
Selected time axis for events: 1 -- 5
Time Risk Set Events
---------------------1
2
1
2
1
1
3
2
0
4
2
1
5
2
1
or a mixture of both expressions. Assuming d > 0, the second expression
is expanded into the sequence t1 + id (i = 0, 1, 2, . . .), as long as the
result is less than or equal to t2 . Note that a time axis defined with the
tp parameter must be a subset of the default time axis. As in 6.18.1, we
will use Tm to denote the selected time axis.
3. If the seqmd command is used without providing a file name on the
right-hand side, the command only displays a table showing the risk set
and the number of events for the specified time axis. To illustrate, we
use the data file seq.d4 (see 6.18.2) and add the following command to
command file seq8.cf:
seqmd(ev = [1,1,2]);
This command specifies the target event [1, 2] in sequence data structure
1 and uses the default time axis. The resulting table is shown in Box 2. In
this example, the risk set Rt contains all individuals (sample members)
being in state 1 at t − 1 and having a valid state at t.
4. The command only creates a data file if a name is given on the righthand side. The data file will contain n = Σni records where ni is the
number of time points the individual i belongs to the risk set. Without
any additional parameters specifying covariates, the data file will contain
the following variables:
• An ID variable which is equal to the case number of the sequence
in the data matrix.
• The current time point, t.
6.18.3
data generation
3
Box 2 Illustration of seqmd command
Command: seqmd(ev = [1,1,2]) = out.d;
Output file:
period
ID Time Z
dummies
----------------------1
1
0
1 0 0 0 0
1
2
1
0 1 0 0 0
1
5
0
0 0 0 0 1
2
1
1
1 0 0 0 0
2
3
0
0 0 1 0 0
2
4
0
0 0 0 1 0
2
5
1
0 0 0 0 1
3
3
0
0 0 1 0 0
3
4
1
0 0 0 1 0
Box 3 Illustration of seqmd command
Command: seqmd(ev = [1,1,2], v = V1,V2) = out.d;
Output file:
period
ID Time Z
dummies
V1 V2
------------------------------1
1
0
1 0 0 0 0
1 -10
1
2
1
0 1 0 0 0
1 -10
1
5
0
0 0 0 0 1
1 -10
2
1
1
1 0 0 0 0
2 -15
2
3
0
0 0 1 0 0
2 -15
2
4
0
0 0 0 1 0
2 -15
2
5
1
0 0 0 0 1
2 -15
3
3
0
0 0 1 0 0
2 -20
3
4
1
0 0 0 1 0
2 -20
• An indicator variable, Zit , being 1 if the target event occurs for
individual i at time t, and otherwise zero.
• For each j ∈ Tm , a dummy variable Pit,(j) , being 1 if j = t and zero
otherwise. These dummy variables can be used to model period effects.
To illustrate this option, we add the command
seqmd(ev = [1,1,2]) = out.d;
to command file seq8.cf. The new command file is seq8a.cf. The output file, out.d, is shown in Box 3.
6.18.3
data generation
4
5. The v parameter can be used to add time-independent covariates to
the output file. The syntax is
v = varlist,
where varlist is a list of variables which must be available in the currently defined TDA data matrix. To illustrate this option, we use the
two variables, V1 and V2, from the data file seq.d4 (see 6.18.2). The
command is
seqmd(ev = [1,1,2], v=V1,V2) = out.d;
The new command file is seq8b.cf. The resulting output file is shown
in Box 3. For writing these variables, TDA uses their associated print
formats. In our example, it is a free format.
6. We have finally to discuss how to create event-specific covariates. This
can be done with the xe parameter. The syntax is
xe = [....],[....],...,
where each pair of square brackets specifies one set of covariates, related
to one type of event. There are two possibilities to define the eventspecific covariates.
7. As a first option, one can use variables contained in TDA’s currently
defined data matrix. For example, xe=[V], where V is the name of a
variable. It is assumed, then, that V records the dates of an event. The xe
parameter then creates a covariate, named V D, containing the duration
between the date in V and the current process time, t.
To illustrate this option, we use the variables V1 and V2 in data file
seq.d4 and add the command
seqmd(ev = [1,1,2], v=V1,V2, xe=[V1],[V2]) = out.d;
The new command file is seq8c.cf. The resulting output file is shown in
Box 4. Note that the values of these covariates will be negative if the date
is greater than the current process time. In general, using this option only
makes sense with variables recording the dates of events which occurred
for all individuals before the process time begins. A typical example is
date of birth with age as the corresponding duration variable.
8. As a second option, one can refer to events occurring in the currently
defined sequence data structures. The syntax is
xe = [k 0 , s01 , s02 , d],...,
6.18.3
data generation
5
Box 4 Illustration of seqmd command
Command: seqmd(ev = [1,1,2], v=V1,V2, xe=[V1],[V2]) = out.d;
Output file:
period
ID Time Z
dummies
V1 V2 V1_D V2_D
-----------------------------------------1
1
0
1 0 0 0 0
1 -10
0
11
1
2
1
0 1 0 0 0
1 -10
1
12
1
5
0
0 0 0 0">