Department of Probability and Mathematical Statistics NMSA407 Linear Regression Course Notes 2015–16 Arnošt Komárek Last modified on January 14, 2016. These course notes contain an overview of notation, definitions, theorems and comments covered by the course “NMSA407 Linear Regression”, which is a part of the curriculum of the Master’s programs “Probability, Mathematical Statistics and Econometrics” and “Financial and Insurance Mathematics”. This document undergoes continuing development. This version is dated January 14, 2016. Arnošt Komárek [email protected] On Řečička, in Karlín, in Zvůle from May 2015, partially based on lecture overheads used in fall 2013 and 2014. Contents 1 Linear Model 1 1.1 Regression analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Basic setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.2 Probabilistic model for the data . . . . . . . . . . . . . . . . . . . . . . . . . 2 Linear model: Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.1 Definition of a linear model . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.2 Definition of a linear model using the error terms . . . . . . . . . . . . . . 3 1.2.3 Rank of the model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.4 Independent observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.5 Linear model with i.i.d. errors . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2.6 Regression function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.7 Transformations of covariates . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.8 Linear model with intercept . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2.9 Interpretation of regression coefficients . . . . . . . . . . . . . . . . . . . . . 8 1.2.10 Fixed or random covariates . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.2.11 Limitations of a linear model . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2 2 Least Squares Estimation 2.1 11 Regression and residual space, projections . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.1 Regression and residual space . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.2 Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 Fitted values, residuals, Gauss–Markov theorem . . . . . . . . . . . . . . . . . . . . 16 2.3 Normal equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4 Estimable parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.5 Parameterizations of a linear model . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.5.1 Equivalent linear models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.5.2 Full-rank parameterization of a linear model . . . . . . . . . . . . . . . . . . 30 iii CONTENTS 2.6 iv Matrix algebra and a method of least squares . . . . . . . . . . . . . . . . . . . . . 33 2.6.1 QR decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.6.2 SVD decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3 Normal Linear Model 4 5 35 3.1 Normal linear model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.2 Properties of the least squares estimators under the normality . . . . . . . . . . . . 37 3.2.1 Statistical inference in a full-rank normal linear model . . . . . . . . . . . . 38 3.2.2 Statistical inference in a general rank normal linear model . . . . . . . . . . 40 3.3 Confidence interval for the model based mean, prediction interval . . . . . . . . . . 42 3.4 Distribution of the linear hypotheses test statistics under the alternative . . . . . . 44 Basic Regression Diagnostics 46 4.1 (Normal) linear model assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.2 Standardized residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.3 Graphical tools of regression diagnostics . . . . . . . . . . . . . . . . . . . . . . . . 50 4.3.1 (A1) Correctness of the regression function . . . . . . . . . . . . . . . . . . . 50 4.3.2 (A2) Homoscedasticity of the errors . . . . . . . . . . . . . . . . . . . . . . . 50 4.3.3 (A3) Uncorrelated errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.3.4 (A4) Normality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Submodels 53 5.1 Submodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 5.1.1 Projection considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 5.1.2 Properties of submodel related quantities . . . . . . . . . . . . . . . . . . . 56 5.1.3 Series of submodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.1.4 Statistical test to compare nested models . . . . . . . . . . . . . . . . . . . 58 5.2 Omitting some covariates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.3 Linear constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.3.1 F-statistic to verify a set of linear constraints . . . . . . . . . . . . . . . . . 65 5.3.2 t-statistic to verify a linear constraint . . . . . . . . . . . . . . . . . . . . . . 65 Coefficient of determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.4.1 Intercept only model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.4.2 Models with intercept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.4.3 Evaluation of a prediction quality of the model . . . . . . . . . . . . . . . . 68 5.4.4 Coefficient of determination . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.4.5 Overall F-test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.4 6 General Linear Model 73 CONTENTS 7 v Parameterizations of Covariates 77 7.1 Linearization of the dependence of the response on the covariates . . . . . . . . . . 77 7.2 Parameterization of a single covariate . . . . . . . . . . . . . . . . . . . . . . . . . . 78 7.2.1 Parameterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 7.2.2 Covariate types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Numeric covariate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 7.3.1 Simple transformation of the covariate . . . . . . . . . . . . . . . . . . . . . 81 7.3.2 Raw polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 7.3.3 Orthonormal polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 7.3.4 Regression splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Categorical covariate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 7.4.1 Link to a G-sample problem . . . . . . . . . . . . . . . . . . . . . . . . . . 88 7.4.2 Linear model parameterization of one-way classified group means . . . . . 90 7.4.3 ANOVA parameterization of one-way classified group means . . . . . . . . . 91 7.4.4 Full-rank parameterization of one-way classified group means . . . . . . . . 97 7.3 7.4 8 Additivity and Interactions 8.1 8.2 8.3 8.4 8.5 8.6 105 Additivity and partial effect of a covariate . . . . . . . . . . . . . . . . . . . . . . . 105 8.1.1 Additivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 8.1.2 Partial effect of a covariate . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 8.1.3 Additivity, partial covariate effect and conditional independence . . . . . . 106 Additivity of the effect of a numeric covariate . . . . . . . . . . . . . . . . . . . . . 107 8.2.1 Partial effect of a numeric covariate . . . . . . . . . . . . . . . . . . . . . . 107 Additivity of the effect of a categorical covariate . . . . . . . . . . . . . . . . . . . . 108 8.3.1 Partial effects of a categorical covariate . . . . . . . . . . . . . . . . . . . . 108 8.3.2 Interpretation of the regression coefficients . . . . . . . . . . . . . . . . . . 109 Effect modification and interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 8.4.1 Effect modification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 8.4.2 Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 8.4.3 Interactions with the regression spline . . . . . . . . . . . . . . . . . . . . . 112 8.4.4 Linear model with interactions . . . . . . . . . . . . . . . . . . . . . . . . . 113 8.4.5 Rank of the interaction model . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Interaction of two numeric covariates . . . . . . . . . . . . . . . . . . . . . . . . . . 116 8.5.1 Mutual effect modification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 8.5.2 Mutual effect modification with regression splines . . . . . . . . . . . . . . 117 Interaction of a categorical and a numeric covariate . . . . . . . . . . . . . . . . . . 120 8.6.1 Categorical effect modification . . . . . . . . . . . . . . . . . . . . . . . . . 120 8.6.2 Categorical effect modification with regression splines . . . . . . . . . . . . 124 CONTENTS 8.7 8.8 vi Interaction of two categorical covariates . . . . . . . . . . . . . . . . . . . . . . . . . 126 8.7.1 Linear model parameterization of two-way classified group means . . . . . 127 8.7.2 ANOVA parameterization of two-way classified group means . . . . . . . . . 130 8.7.3 Full-rank parameterization of two-way classified group means . . . . . . . . 133 8.7.4 Relationship between the full-rank and ANOVA parameterizations . . . . . . 135 8.7.5 Additive model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 8.7.6 Interpretation of model parameters for selected choices of (pseudo)contrasts 137 Hierarchically well-formulated models, ANOVA tables . . . . . . . . . . . . . . . . . 143 8.8.1 Model terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 8.8.2 Model formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 8.8.3 Hierarchically well formulated model . . . . . . . . . . . . . . . . . . . . . . 146 8.8.4 ANOVA tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 9 Analysis of Variance 9.1 9.2 152 One-way classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 9.1.1 Parameters of interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 9.1.2 One-way ANOVA model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 9.1.3 Least squares estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 9.1.4 Within and between groups sums of squares, ANOVA F-test . . . . . . . . . 157 Two-way classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 9.2.1 Parameters of interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 9.2.2 Two-way ANOVA models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 9.2.3 Least squares estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 9.2.4 Sums of squares and ANOVA tables with balanced data . . . . . . . . . . . 170 10 Checking Model Assumptions 173 10.1 Model with added regressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 10.2 Correct regression function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 10.2.1 Partial residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 10.2.2 Test for linearity of the effect . . . . . . . . . . . . . . . . . . . . . . . . . . 181 10.3 Homoscedasticity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 10.3.1 Tests of homoscedasticity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 10.3.2 Score tests of homoscedasticity . . . . . . . . . . . . . . . . . . . . . . . . . 183 10.3.3 Some other tests of homoscedasticity . . . . . . . . . . . . . . . . . . . . . . 185 10.4 Normality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 10.4.1 Tests of normality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 10.5 Uncorrelated errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 10.5.1 Durbin-Watson test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 CONTENTS vii 10.6 Transformation of response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.1 Prediction based on a model with transformed response . . . . . . . . . . . 193 10.6.2 Log-normal model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 11 Consequences of a Problematic Regression Space 11.1 11.2 196 Multicollinearity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 11.1.1 Singular value decomposition of a model matrix . . . . . . . . . . . . . . . 197 11.1.2 Multicollinearity and its impact on precision of the LSE . . . . . . . . . . . 198 11.1.3 Variance inflation factor and tolerance . . . . . . . . . . . . . . . . . . . . . 200 11.1.4 Basic treatment of multicollinearity . . . . . . . . . . . . . . . . . . . . . . . 204 Misspecified regression space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 11.2.1 Omitted and irrelevant regressors . . . . . . . . . . . . . . . . . . . . . . . . 205 11.2.2 Prediction quality of the fitted model . . . . . . . . . . . . . . . . . . . . . . 209 11.2.3 Omitted regressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 11.2.4 Irrelevant regressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 11.2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 12 Simultaneous Inference in a Linear Model 12.1 193 219 Basic simultaneous inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 12.2 Multiple comparison procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 12.2.1 Multiple testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 12.2.2 Simultaneous confidence intervals . . . . . . . . . . . . . . . . . . . . . . . 223 12.2.3 Multiple comparison procedure, P-values adjusted for multiple comparison 12.2.4 Bonferroni simultaneous inference in a normal linear model . . . . . . . . . 226 224 12.3 Tukey’s T-procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 12.3.1 Tukey’s pairwise comparisons theorem . . . . . . . . . . . . . . . . . . . . . 228 12.3.2 Tukey’s honest significance differences (HSD) . . . . . . . . . . . . . . . . . 230 12.3.3 Tukey’s HSD in a linear model . . . . . . . . . . . . . . . . . . . . . . . . . . 233 12.4 Hothorn-Bretz-Westfall procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 12.4.1 Max-abs-t distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 12.4.2 General multiple comparison procedure for a linear model . . . . . . . . . 239 12.5 Confidence band for the regression function . . . . . . . . . . . . . . . . . . . . . . 243 13 Asymptotic Properties of the LSE and Sandwich Estimator 13.1 247 Assumptions and setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 13.2 Consistency of LSE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 13.3 Asymptotic normality of LSE under homoscedasticity . . . . . . . . . . . . . . . . . 255 13.3.1 Asymptotic validity of the classical inference under homoscedasticity but non-normality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 CONTENTS viii 13.4 Asymptotic normality of LSE under heteroscedasticity . . . . . . . . . . . . . . . . . 258 13.4.1 Heteroscedasticity consistent asymptotic inference . . . . . . . . . . . . . . 265 14 Unusual Observations 14.1 267 Leave-one-out and outlier model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 14.2 Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 14.3 Leverage points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 14.4 Influential diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 14.4.1 DFBETAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 14.4.2 DFFITS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 14.4.3 Cook distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.4 COVRATIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 14.4.5 Final remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 A Matrices 281 285 A.1 Pseudoinverse of a matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 A.2 Kronecker product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 A.3 Additional theorems on matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 B Distributions 290 B.1 Non-central univariate distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 B.2 Multivariate distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 C Asymptotic Theorems 295 Bibliography 299 Preface • R software (R Core Team, 2015). • Basic literature: Khuri (2010); Zvára (2008). • Supplementary literature: Seber and Lee (2003); Draper and Smith (1998); Sun (2003); Weisberg (2005); Anděl (2007); Cipra (2008); Zvára (1989). ix Notation and general conventions • Vectors are understood as column vectors (matrices with one column). Nevertheless, to simplify notation, standalone vectors will also be written as rows, e.g., z = z1 , . . . , zp . The transposition symbol (>) will only be used if it is necessary to highlight the fact that the vector must be used as a row vector (matrix with one row). This mainly points to situations when (i) vectors are used within the matrix multiplication, e.g., µ = z > γ, z = z1 , . . . , zp , γ = γ1 , . . . , γp , means that µ is a product of a row vector z > and a column vector γ (i.e., a scalar product of the two vectors); (ii) vectors are used to indicate rows of a matrix, e.g., Z> Z 1 = Z1,1 , . . . , Z1,p , 1 . .. . Z= . . , > Zn Z n = Zn,1 , . . . , Zn,p , > means that the matrix Z has rows given by the (row) vectors Z > 1 , . . ., Z n , i.e., Z1,1 . . . Z1,p . .. .. .. Z= . . . Zn,1 . . . Zn,p • Statements concerning equalities between two random quantities are understood as equalities almost surely even if “almost surely” is not explicitely stated. x Chapter 1 Linear Model 1.1 Regression analysis Start of Linear is a basic method of so called regression which covers a variety of Lecture #1 methods to model on how distribution of one variable depends on one or more other variables. (05/10/2015) A principal tool of linear regression is then so called linear model 3 which will be a main topic of this lecture. regression1 1.1.1 analysis2 Basic setup Most methods of regression analysis assume that the data can be represented by n random vectors Yi , X i , i = 1, . . . , n, where X i = Xi,0 , . . . , Xi,k−1 , i = 1, . . . , n, have all the same number k < n components. This will also be a basic assumption used throughout the whole lecture. Notation and terminology (Response, covariates, response vector, model matrix). • Yi is called response 4 or dependent variable 5 . • The components of X i are called covariates6 , explanatory variables7 , predictors8 , or independent variables9 . • The sample space10 of the covariates will be denoted as X . That is, X ⊆ Rk , and among other things, P(X i ∈ X ) = 1, i = 1, . . . , n. Further, let Y1 . . Y = . , Yn X> X1,0 . . . X1,k−1 1 . . .. .. . . X= . . . = . . X> Xn,0 . . . Xn,k−1 n • Vector Y is called the response vector 11 . • The n × k matrix X is called the model matrix 12 or the regression matrix 13 . 1 lineární regrese 2 regresní analýza 3 lineární model 4 odezva 5 závisle proměnná 6 Nepřekládá se. Výraz „kovariáty“ nepoužívat! 7 vysvětlující proměnné 8 prediktory 9 nezávisle proměnné 10 výběrový prostor 11 vektor odezvy 12 matice modelu 13 regresní matice 1 1.1. REGRESSION ANALYSIS 2 Notation. Letter Y (or y) will always denote a response related quantity. Letters X (or x) and Z (or z) will always denote a quantity related to the covariates. This lecture: • Response Y is continuous. • Interest in modelling dependence of only the expected value (the mean) of Y on the covariates. • Covariates can be of any type (numeric, categorical). 1.1.2 Probabilistic model for the data Any statistical analysis is based on specifying a stochastic mechanism which is assumed to generate the data. In our situation, with data Yi , X i , i = 1, . . . , n, the data generating mechanism cor- responds to a joint distribution of a “long” random vector Y1 , . . . , Yn , X 1 , . . . , X n ≡ Y , X which can be given by a joint density (1.1) fY ,X (y1 , . . . , yn , x1 , . . . , xn ) ≡ fY ,X (y, x) (with respect to some σ-finite product measure λY × λX ). Note. For the purpose of this lecture, λY will always be a Lebesgue measure on Rn , Bn . Further, it is known from basic lectures on probability that any joint density can be decomposed into a product of a conditional and a marginal density as (1.2) fY ,X (y, x) = fY |X y x fX (x). With the regression analysis, and with the linear regression in particular, the interest lies in revealing certain features of the conditional distribution Y X (given by the density fY |X ) while considering the marginal distribution of the covariates X (given by the density fX ) as nuisance. It will be shown during the lecture that a valid statistical inference is possible for suitable characteristics of the conditional distribution of the response given the covariates while leaving the covariates distribution fX practically unspecified. Moreover, to infer characteristics of certain on the conditional distribution Y X, e.g., on the conditional mean E Y X , even the density fY |X might be left practically unspecified for many tasks. 1.2. LINEAR MODEL: BASICS 1.2 1.2.1 3 Linear model: Basics Definition of a linear model Definition 1.1 Linear model. The data Yi , X i , i = 1, . . . , n, E Y satisfy a linear model if X = Xβ, var Y X = σ 2 In , where β ∈ Rk and 0 < σ 2 < ∞ are unknown parameters. Terminology (Regression coefficients, residual variance and standard deviation). • β = β0 , . . . , βk−1 is called the vector of regression coefficients14 or regression parameters.15 • σ 2 is called the residual variance.16 √ • σ = σ 2 is called the residual standard deviation.17 Note. The linear model as specified by Definition 1.1 deals with specifying only the first two moments of the conditional distribution Y X. For the rest, both the density fY |X and the density fX from (1.2) can be arbitrary. 1.2.2 Definition of a linear model using the error terms The linear model can equivalently be defined as follows. Alternative to Definition 1.1 The data Yi , X i , i = 1, . . . , n satisfy a linear model if Y = Xβ + ε, where ε = ε1 , . . . , εn is a random vector such that ε and X are independent, E(ε) = 0n , var(ε) = σ 2 In , and β ∈ Rk and 0 < σ 2 < ∞ are unknown parameters. Terminology (Error terms). Random variables ε1 , . . . , εn are called the error terms (disturbances).18 Notation. • To indicate that a random vector Y follows some distribution with mean µ and a covariance matrix Σ, we will write Y ∼ µ, Σ . 14 regresní koeficienty odchylky 15 regresní parametry 16 reziduální rozptyl 17 reziduální směrodatná odchylka 18 náhodné 1.2. LINEAR MODEL: BASICS 4 • The fact that data Yi , X i , i = 1, . . . , n, follow a linear model can now be indicated by writing Y X ∼ Xβ, σ 2 In . • When using the error term vector ε, the fact that data Yi , X i , i = 1, . . . , n, follow a linear model can be written as Y = Xβ + ε, ε ∼ 0n , σ 2 In . 1.2.3 Rank of the model The k-dimensional covariate vectors X 1 , . . . , X n (the n × k model matrix X) are in general generated by some (n · k)-dimensional joint distribution with a density fX (x1 , . . . , xn ) = fX (x). Next to assuming that n > k, we will additionally assume in the whole lecture that for a fixed r ≤ k, P rank(X) = r = 1. (1.3) That is, we will assume that the (column) rank of the model matrix is fixed rather than being random. It should gradually become clear throughout the lecture that this assumption is not really restrictive for most of the practical applications of a linear model. Convention. In the reminder of the lecture, we will only write rank X = r which will mean that P rank(X) = r = 1 if randomness of the covariates should be taken into account. Notation. To indicate that data Yi , X i , i = 1, . . . , n follow a linear model where the k dimensional covariates satisfy the condition (1.3) will be denoted as Y X ∼ Xβ, σ 2 In , rank Xn×k = r, or as Y = Xβ + ε, ε ∼ 0n , σ 2 In , rank Xn×k = r. Definition 1.2 Full-rank linear model. A full-rank linear model19 is such a linear model where r = k. Note. In a full-rank linear model, columns of the model matrix X are linearly independent vectors in Rn (almost surely). 1.2.4 Independent observations In many areas of regression modelling, the data Yi , X i , i = 1, . . . , n, correspond to independently behaving units (experimental units, individuals sampled randomly from a certain population, . . . ). If it is so, the random vectors Yi , X i are independent for i = 1, . . . , n, and the joint density (1.1) takes a product form fY ,X (y, x) = n Y i=1 19 lineární model o plné hodnosti fYi ,X i (yi , xi ), 1.2. LINEAR MODEL: BASICS 5 where fYi ,X i is a joint density of a random vector Yi , X i . This can again be decomposed into a product of a conditional and a marginal density as i = 1, . . . , n, fYi ,X i (yi , xi ) = fYi |X i yi xi fX i (xi ), leading to the joint density of all data points of a form fY ,X (y, x) = n nY fYi |X i n o o nY yi xi fX i (xi ) . i=1 | (1.4) i=1 {z fY |X y x } | {z fX (x) } Due to the fact that the Q covariate distribution is in fact a nuisance, it is not really necessary to assume that fX (x) = ni=1 fX i (xi ) to derive most results that will be presented in this lecture. In other words, the form of fX in the expression (1.4) might be left in its general form which leads to the joint density of the data points of the form fY ,X (y, x) = n nY | i=1 o fYi |X i yi xi fX (x). {z fY |X y x } Definition 1.3 Independent observations in a regression context. In the regression context, we say that we deal with independent observations if the conditional density of the response vector Y given the covariates X takes the form n Y fY |X y x = fYi |X i yi xi . (1.5) i=1 Note. When dealing with independent observations in a regression context according to Defini- tion 1.3, no independence assumptions are imposed on the joint distribution fX of the covariates. 1.2.5 Linear model with i.i.d. errors If we deal with independent observations in a regression context and a form (1.5) of the conditional distribution of Y given X can be assumed, it is often useful to assume not only a certain form of the first two moments of each conditional distribution fYi |X i , i = 1, . . . , n, but even a common functional form for all those conditional distributions. Definition 1.4 Linear model with i.i.d. errors. The data Yi , X i , i = 1, . . . , n, satisfy a linear model with i.i.d. errors20 if the joint conditional density of Y given X takes a form fY |X (y, x) = n Y i=1 20 lineární model s nezávislými stejně rozdělenými chybami fY |X yi xi , 1.2. LINEAR MODEL: BASICS where 6 2 fY |X yi xi = fe (yi − x> i β; σ ), i = 1, . . . , n, β ∈ Rk and 0 < σ 2 < ∞ are unknown parameters and fe (·; σ 2 ) is some density of a continuous distribution with zero mean and a variance σ 2 . 2 Note. In a linear model with i.i.d. errors, fY |X yi xi = fe (yi − x> i β; σ ), i = 1, . . . , n and since fe (·; σ 2 ) is a density of zero mean distribution with variance σ 2 , we still have 2 E Yi X i = X > i = 1, . . . , n, i β, var Yi X i = σ , var Y X = σ 2 In . E Y X = Xβ, That is, the linear model with i.i.d. errors indeed implies a linear model according to the basic definition (Definition 1.1). Analogously to Section 1.2.2, also a linear model with i.i.d. errors can alternatively be defined while using the (i.i.d.) error terms as follows. Definition 1.4 using the error terms The data Yi , X i , i = 1, . . . , n, satisfy a linear model with i.i.d. errors if Yi = X > i β + εi , i = 1, . . . , n, where ε1 , . . . , εn are independent and identically distributed (i.i.d.) random variables with a continuous distribution with zero mean and a variance σ 2 , εi and X j independent for each i, j = 1, . . . , n. Parameters β ∈ Rk and 0 < σ 2 < ∞ are unknown parameters. Example 1.1 (Normal linear model). So called normal linear model that we will extensively work with starting from Chapter 3 is obtained by considering fe (·; σ 2 ) as a density of a normal distribution N (0, σ 2 ). We then have 2 Yi X i ∼ N (X > i = 1, . . . , n, i β, σ ), Y X ∼ Nn (Xβ, σ 2 In ), i.i.d. ε1 , . . . , εn ∼ N (0, σ 2 ). 1.2.6 Regression function Let Y, X denote a (generic) random vector that correspond to a new data point being generated by the same random mechanism as data at hand. When using the linear model, we do not specify any characteristic of the random mechanism that generates the covariate value X. Nevertheless, as soon as a covariate value x ∈ X is given, and i.i.d. errors are assumed, the linear model specifies (models) the first two moments of the conditional distribution Y X = x (with a density fY |X · x)). Namely, the linear model claims that E Y X = x = x> β, var Y X = x = σ 2 , 1.2. LINEAR MODEL: BASICS 7 for some β ∈ Rk and some σ 2 . Terminology (Regression function). A function m : Rk −→ R given as m(x) = x> β = β0 x0 + · · · + βk−1 xk−1 , x = x0 , . . . , xk−1 ∈ X , is called the regression function.21 Note. The regression function is an assumed model for evolution of the response expectation given the covariates if those are changing. That is, m(x) = E Y X = x , 1.2.7 x ∈ X. Transformations of covariates In most practical situations,p we primarily observe data Yi , Z i , i = 1, . . . , n, where Z i = Zi,1 , . . . , Zi,p ∈ Z ⊆ R , i = 1, . . . , n. The main interest lies in modelling the conditional expectations E Yi Z i , or generically the conditional expectation E Y Z = z , z ∈ Z. To allow for use of a linear model, original covariates Z i must often be transformed into X i = Xi,0 , . . . , Xi,k−1 as Xi,0 = t0 (Z i ), ... , Xi,k−1 = tk−1 (Z i ), where tj : Rp −→ R, j = 0, . . . , k − 1, are some functions. When applying classical linear model methodology, it is assumed that the functions t0 , . . . , tk−1 are known. In the following, let for z ∈ Z: x = x0 , . . . , xk−1 = t0 (z), . . . , tk−1 (z) = t(z). The linear model then specifies (i = 1, . . . , n): E Yi Z i = t> (Z i )β = X > i β = E Yi X i , or generically E Y Z = z = t> (z)β = x> β = E Y X = x . The model matrix takes the form X1,0 . . . X1,k−1 t0 (Z 1 ) . . . tk−1 (Z 1 ) t> (Z 1 ) X> 1 . . . .. .. .. .. = ... = .. , .. = .. X= . . . . > (Z ) X> X . . . X t (Z ) . . . t (Z ) t n n,0 0 n n k−1 n,k−1 n and the corresponding regression function is m(z) = E Y Z = z = t> (z)β = β0 t0 (z) + · · · + βk−1 tk−1 (z), z = z1 , . . . , zp ∈ Z. (1.6) Note. Even though some of the transformations t0 , . . . , tk−1 are often non-linear functions, a model with the regression function (1.6) still belongs to the area of linear regression. The reason is that the regression function is linear with respect to the unknown parameters, i.e., in the regression coefficients β. 21 regresní funkce 1.2. LINEAR MODEL: BASICS 1.2.8 8 Linear model with intercept Often, Xi,0 is taken to be constantly equal to one. That is, the first element of the (transformed) covariate vectors corresponds to a random variable which is almost surely equal to one. In this case, the model matrix and the regression function take the forms 1 X1,1 . . . X1,k−1 m(x) = β0 + β1 x1 + · · · + βk−1 xk−1 , . .. .. , .. X= . . x = x0 , . . . , xk−1 ∈ X ⊆ Rk . 1 Xn,1 . . . Xn,k−1 Definition 1.5 Linear model with intercept. Linear model in which the first column of the model matrix X is almost surely equal to a vector 1n of ones is called the linear model with intercept22 . Terminology (Intercept). In the linear model with intercept, the first column of the model matrix, X1,0 , . . . , Xn,0 which is almost surely equal to a vector 1n of ones is called the intercept column. The regression coefficient β0 is called the intercept term23 (or just the intercept) of the regression function. 1.2.9 Interpretation of regression coefficients The regression parameters express influence of the covariates on response. Let for a chosen j ∈ 0, 1, . . . , k − 1 x = x0 , . . . , xj . . . , xk−1 ∈ X , and xj(+1) := x0 , . . . , xj + 1 . . . , xk−1 ∈ X . We then have E Y X = xj(+1) − E Y X = x = E Y X0 = x0 , . . . , Xj = xj + 1, . . . , Xk−1 = xk−1 − E Y X0 = x0 , . . . , Xj = xj , . . . , Xk−1 = xk−1 = β0 x0 + · · · + βj (xj + 1) + · · · + βk−1 xk−1 − β0 x0 + · · · + βj xj + · · · + βk−1 xk−1 = βj . That is, the regression coefficient βj expresses a change of the response expectation corresponding to a unity change of the jth regressor while keeping the remaining covariates unchanged. Further, let for a fixed δ ∈ R xj(+δ) := x0 , . . . , xj + δ . . . , xk−1 ∈ X , 22 lineární model s absolutním členem 23 absolutní člen 1.2. LINEAR MODEL: BASICS 9 we then have E Y X = xj(+δ) − E Y X = x = E Y X0 = x0 , . . . , Xj = xj + δ, . . . , Xk−1 = xk−1 − E Y X0 = x0 , . . . , Xj = xj , . . . , Xk−1 = xk−1 = βj δ. It is thus implied by the assumed linear model: (i) The change of the response expectation corresponding to a constant change δ of the jth regressor does not depend on the value xj of that regressor which is changed by δ. (ii) The change of the response expectation corresponding to a constant change δ of the jth regressor does not depend on the values of the remaining regressors. Terminology (Effect of the regressor). The regression coefficient βj is also called the effect of the jth regressor. Linear model with intercept In a model with intercept where Xi,0 is almost surely equal to one, it does not make sense to consider a change of this covariate by any fixed value. The intercept β0 has then the following interpretation. If x0 , x1 , . . . , xk−1 = 1, 0, . . . , 0 ∈ X , that is, if the non-intercept covariates may all attain zero values, we have β0 = E Y X1 = 0, . . . , Xk−1 = 0 . 1.2.10 Fixed or random covariates In certain application areas (e.g., designed experiments), the covariates can all (or some of them) be fixed rather than random variables. This means that the covariate values are determined/set by the analyst rather than being observed on (randomly selected) subjects. For majority of the theory presented throughout this course, it does not really matter whether the covariates are considered as random or as fixed quantities. Many proofs proceed almost exactly in the same way in both situations. Nevertheless, especially when dealing with asymptotic properties of the estimators used in the context of a linear model (see Chapter 13), care must be taken on whether the covariates are considered as random or as fixed. End of Lecture #1 Convention. For majority of the lecture, most expectations that we will work with, will be (05/10/2015) conditional expectations with respect to the conditional distribution of the response Y given the Start of covariate values we will use E(·) Lecture #2 matrix X). To simplify notation, X 1 , . . . , X n (given the model X . That is, if g = g1 , . . . , gm : Rn·(1+k) −→ Rm is also for E · X and var(·) also for var · (08/10/2015) a measurable function then E g(Y , X) and var g(Y , X) means E g(Y , X) := E g(Y , X) X Z = gj (y, X) fY |X y X dλY (y) , Rn var g(Y , X) := var g(Y , X) X . j=1,...,m 1.2. LINEAR MODEL: BASICS 10 Further, for x ∈ Rn·k , we will use E g(Y , x) := E g(Y , X) X = x , var g(Y , x) := var g(Y , X) X = x . (Unconditional) expectations and covariance matrices with respect to the joint distribution of Y and X will be indicated by a subscript Y , X, that is, Z EY , X g(Y , X) = gj (y, x) fY ,X (y, x) dλY (y) dλX (x) . Rn(1+k) j=1,...,m Analogously, expectations with respect to the marginal distribution of X will also be indicated by an appropriate subscript. That is, if h = h1 , . . . , hm : Rn·k −→ Rm is a measurable function then Z hj (x) fX (x) dλX (x) EX h(X) = . Rn·k j=1,...,m Note. To calculate EY , X , a basic relationship EY , X h i g(Y , X) = EX E g(Y , X) (1.7) known from probability courses can conveniently be used. Furthermore, it trivially follows from n·k (1.7) that if E g(Y , x) = c for λX -almost all x ∈ R then also EY , X g(Y , X) = c. 1.2.11 Limitations of a linear model “Essentially, all models are wrong, but some are useful. The practical question is how wrong do they have to be to not be useful.” George E. P. Box (1919 – 2013) Linear model is indeed only one possibility (out of infinitely many) on how to model dependence of the response on the covariates. The linear model as defined by Definition 1.1 is (possibly seriously) wrong if, for example, • The expected value E Y X = x , x ∈ X , cannot be expressed as a linear function of x. ⇒ Incorrect regression function. • The conditional variance var Y X = x , x ∈ X , is not constant. It may depend on x as well, it may depend on other factors. ⇒ Heteroscedasticity. • Response random variables are not conditionally uncorrelated/independent (the error terms are not uncorrelated/independent). This is often the case if response is measured repeatedly (e.g., over time) on n subjects included in the study. Additionally, the linear model deals with modelling of only the first two (conditional) moments of the response. In many application areas, other characteristics of the conditional distribution Y X are of (primary) interest. Chapter 2 Least Squares Estimation We keep considering a set of n random vectors Yi , X i , X i = Xi,0 , . . . , Xi,k−1 , i = 1, . . . , n, that satisfy a linear model. That is, Y X ∼ Xβ, σ 2 In , rank Xn×k = r ≤ k < n, (2.1) > where Y = Y1 , . . . , Yn , X is a matrix with vectors X > 1 , . . . , X n in its rows and β = β0 , . . . , k 2 βk−1 ∈ R and σ > 0 are unknown parameters. In this chapter, we introduce a method of least squares1 to estimate the unknown parameters of the linear model (2.1). All results in this chapter will be derived without imposing any (parametric) distributional assumptions concerning the conditional distribution of the response given the covariates and without assuming independent observations in the regression context or i.i.d. errors. 1 metoda nejmenších čtverců 11 2.1. REGRESSION AND RESIDUAL SPACE, PROJECTIONS 2.1 2.1.1 12 Regression and residual space, projections Regression and residual space Notation (Linear span of columns of the model matrix and its orthogonal complement). For given dataset and a linear model, the model matrix X is a real n×k matrix. Let x0 , . . . , xk−1 ∈ Rn denote its columns, i.e., X = x0 , . . . , xk−1 . 0 k−1 will be • The linear span2 of columns of X, i.e., a vector space generated by vectors x , . . . , x denoted as M X , that is, M X = v: v= k−1 X βj xj , β = β0 , . . . , βk−1 ∈ Rk . j=0 ⊥ • The orthogonal complement to M X will be denoted as M X , that is, M X ⊥ = u : u ∈ Rn , v > u = 0 for all v ∈ M X . Note. We know from linear algebra lectures that the linear span of column of X, M X , is a vector subspace of dimension r of the n-dimensional Euclidean space Rn . Similarly, M X a vector subspace of dimension n − r of the n-dimensional Euclidean space Rn . We have ⊥ is ⊥ ⊥ M X ∪ M X = Rn , M X ∩ M X = 0n , ⊥ for any v ∈ M X , u ∈ M X v > u = 0. Definition 2.1 Regression and residual space of a linear model. Consider a linear modelY X ∼ Xβ, σ 2 In , rank(X) = r. The regression space3 of the model is a vector space M X . The residual space4 of the model is the orthogonal complement of the ⊥ regression space, i.e., a vector space M X . Notation (Orthonormal vector bases of the regression and residual space). Let q 1 , . . . , q r be (any) orthonormal vector basis of the regression space M X and let n1 , . . . , nn−r ⊥ be (any) orthonormal vector basis of the residual space M X . That is, q 1 , . . . , q r , n1 , . . . , nn−r is an orthonormal vector basis of the n-dimensinal Euclidean space Rn . We will denote • Qn×r = q 1 , . . . , q r . • Nn×(n−r) = n1 , . . . , nn−r . 2 lineární obal 3 regresní prostor 4 reziduální prostor 2.1. REGRESSION AND RESIDUAL SPACE, PROJECTIONS 13 • Pn×n = q 1 , . . . , q r , n1 , . . . , nn−r = Q, N . It follows from the linear algebra lectures. • Properties of the columns of the Q matrix: • q> j = 1, . . . , r; j q j = 1, j, l = 1, . . . , r, j 6= l. • q> j q l = 0, Notes. • Properties of the columns of the N matrix: • n> j = 1, . . . , n − r; j nj = 1, > • nj nl = 0, j, l = 1, . . . , n − r, j 6= l. • Mutual properties of the columns of the Q and N matrix: > j = 1, . . . , r, l = 1, . . . , n − r. • q> j nl = nl q j = 0, • Above properties written in a matrix form: Q> Q = Ir , N> N = In−r , Q> N = 0r×(n−r) , N> Q = 0(n−r)×r , P> P = In . (2.2) • It follows from (2.2) that P> is inverse to P and hence ! Q> > In = P P = Q, N = Q Q> + N N> . N> • It is also useful to remind M X =M Q , M X ⊥ =M N , Rn = M P . Notation. In the following, let H = Q Q> , M = N N> . Note. Matrices H and M are symmetric and idempotent: > H> = Q Q> = Q Q> = H, > M> = N N> = N N> = M, 2.1.2 H H = Q Q> Q Q> = Q Ir Q> = Q Q> = H, M M = N N> N N> = N In−r N> = N N> = M. Projections Let y ∈ Rn . We can then write (while using identity in Expression 2.2) y = In y = Q Q> + N N> y = (H + M)y = Hy + My. We have 2.1. REGRESSION AND RESIDUAL SPACE, PROJECTIONS 14 b := Hy = Q Q> y ∈ M X . • y ⊥ • u := My = N N> y ∈ M X . • y > u = y > Q Q> N N> y = y > Q 0r×(n−r) N> y = 0. That is, we have decomposition of any y ∈ Rn into b + u, y=y ⊥ b∈M X , u∈M X , y b ⊥ u. y ⊥ b and u are projections of y into M X and M X , respectively, and H and M In other words, y are corresponding projection matrices. It follows from the linear algebra lectures. b + u is unique. • Decomposition y = y Notes. • Projection matrices H, M are unique. That is H = Q Q> does not depend on a choice of the orthonormal vector basis of M X included in the Q matrix and M = N N> does not depend ⊥ on a choice of the orthonormal vector basis of M X included in the N matrix. b = yb1 , . . . , ybn is the closest point (in a Euclidean metric) in the regression space • Vector y M(X) to a given vector y = y1 , . . . , yn , that is, e = ye1 , . . . , yen ∈ M(X) ∀y n n X X 2 bk = e k2 . ky − y (yi − ybi ) ≤ (yi − yei )2 = ky − y 2 i=1 i=1 Definition 2.2 Hat matrix, residual projection matrix. Consider a linear model Y X ∼ Xβ, σ 2 In , where Q and N are the orthonormal bases of the regression and the residual space, respectively. 1. The hat matrix5 of the model is the matrix Q Q> which is denoted as H. 2. The residual projection matrix6 of the model is the matrix N N> which is denoted as M. Lemma 2.1 Expressions of the projection matrices using the model matrix. The hat matrix H and the residual projection matrix M can be expressed as H = X X> X − X> , − M = In − X X> X X> . Proof. − • Five matrices rule (Theorem A.2): n X X> X Xo> X − In − X X> X X> X 5 regresní projekční matice, lze však užívat též výrazu „hat matice“ 6 = X, = 0n×k . reziduální projekční matice 2.1. REGRESSION AND RESIDUAL SPACE, PROJECTIONS e = X X> X • Let H 15 − X> , e = In − X X> X − X> = In − H. e M e = 0n×k , both H e and M e are symmetric. • We have MX • We now have: e + In − H e y = Hy e + My. e y = In y = H e =X • Clearly, Hy n X> X − o X> y ∈ M X . e = b> X> My = y > MX b = 0. Hence My e ∈ M X ⊥. • For any z = Xb ∈ M X : z > My |{z} 0n ⇒ Uniqueness of projections and projection matrices e = X X> X H=H − X> , e = In − X X> X − X> . M=M k Notes. − − • Expression X X> X X> does not depend on a choice of the pseudoinverse matrix X> X . • If r = rank Xn×k = k then −1 H = X X> X X> , −1 M = In − X X> X X> . 2.2. FITTED VALUES, RESIDUALS, GAUSS–MARKOV THEOREM 2.2 16 Fitted values, residuals, Gauss–Markov theorem Before starting to deal with estimation of the principal parameters of the linear model, which are the regression coefficients β, we will deal with estimation of the full (conditional) mean E(Y ) = Xβ of the response vector Y and its (conditional) covariance matrix var(Y ) = σ 2 In for which it is sufficient to estimate the residual variance σ 2 . Notation. We denote µ := Xβ = E(Y ). By saying that we are now interested in estimation of the full (conditional) expectation E(Y ) we mean that we want to estimate the parameter vector µ on its own without necessity to know its decomposition into Xβ. Definition 2.3 Fitted values, residuals, residual sum of squares. Consider a linear model Y X ∼ Xβ, σ 2 In . 1. The fitted values7 or the vector of fitted values of the model is a vector HY which will be denoted as Yb . That is, Yb = Yb1 , . . . , Ybn = HY . 2. The residuals8 or the vector of residuals of the model is a vector MY which will be denoted as U . That is, U = U1 , . . . , Un = MY = Y − Yb . 2 3. The residual sum of squares9 of the model is a quantity U which will be denoted as SSe . That is, n X 2 SSe = U = U > U = Ui2 i=1 = n X Yi − Ybi 2 = Y − Yb > 2 Y − Yb = Y − Yb . i=1 Notes. • The fitted values Yb and the residuals U are projections of the response vector Y into the ⊥ regression space M X and the residual space M X , respectively. • Using different quantities and expressions introduced in Section 2.1, we can write − Yb = HY = Q Q> Y = X X> X X> Y , n − o U = MY = N N> Y = In − X X> X X> Y = Y − Yb . 7 vyrovnané hodnoty 8 rezidua 9 reziduální součet čtverců 2.2. FITTED VALUES, RESIDUALS, GAUSS–MARKOV THEOREM 17 > • It follows from the projection properties that the vector Yb = Yb1 , . . . , Ybn is the nearest point > of the regression space M X to the response vector Y = Y1 , . . . , Yn , that is, ∀ Ye = Ye1 , . . . , Yen > ∈ M(X) n n X X 2 Y − Yb 2 = (Yi − Ybi )2 ≤ (Yi − Yei )2 = Y − Ye . (2.3) i=1 i=1 • The Gauss–Markov theorem introduced below shows that Yb is a suitable estimator of µ = Xβ. Owing to (2.3), it is also called the least squares estimator (LSE).10 The method of estimation is then called the method of least squares11 or the method of ordinary least squares (OLS). Theorem 2.2 Gauss–Markov. Assume a linear model Y X ∼ Xβ, σ 2 In . Then the vector of fitted values Yb is the best linear unbiased estimator (BLUE)12 of a vector parameter µ = E(Y ). Further, − var Yb = σ 2 H = σ 2 X X> X X> . Proof. First, remind our notational convention that E(·) = E · X and var(·) = var · X . Linearity means that Yb is a linear function of the response vector Y which is clear from the expression Yb = HY . Unbiasedness. Let us calculate E Yb . E Yb = E(HY ) = H E(Y ) = H Xβ = Xβ = µ. The pre-last equality holds due to the fact that HX is a projection of each column of X into M X which is generated by those columns. That is HX = X. Optimality. Let Ye = a + BY be some other linear unbiased estimator of µ = Xβ. • That is, ∀ β ∈ Rk E Ye = Xβ, ∀ β ∈ Rk a + B E(Y ) = Xβ, ∀ β ∈ Rk a + BXβ = Xβ. It follows from here, by using above equality with β = 0k , that a = 0k . • That is, from unbiasedness, we have that ∀ β ∈ Rk B Xβ = Xβ. Take now β = > 0, . . . , 1, . . . , 0 while changing a position of one. From here, it follows that BX = X. • We now have: Ye = a + BY unbiased estimator of µ =⇒ a = 0k & BX = X. Trivially (but we will not need it here), also the opposite implication holds (if Ye = BY with BX = X then Ye is the unbiased estimator of µ = Xβ). In other words, Ye = a + BY is unbiased estimator of µ 10 odhad metodou nejmenších čtverců 11 ⇐⇒ metoda nejmenších čtverců (MNČ) 12 a = 0k & BX = X. nejlepší lineární nestranný odhad 2.2. FITTED VALUES, RESIDUALS, GAUSS–MARKOV THEOREM 18 • Let us now explore what can be concluded from the equality BX = X. · X> X − X> BX = X, − − BX X> X X> = X X> X X> , BH = H, (2.4) H> B> = H> , HB> = H. (2.5) • Let us calculate var Yb : var Yb = var(HY ) = H var(Y ) H> = H (σ 2 In ) H> − = σ 2 HH> = σ 2 H = σ 2 X X> X X> . • Analogously, we calculate var Ye for Ye = BY , where BX = X: var Ye = var BY = B var(Y ) B> = B (σ 2 In ) B> = σ 2 BB> = σ 2 (H + B − H) (H + B − H)> > > > > + H(B − H) = σ 2 HH + (B − H)H + (B − H) (B − H) | {z } | {z } | {z } H 0n 0n = σ 2 H + σ 2 (B − H) (B − H)> , where H(B − H)> = (B − H)H> = 0n follow from (2.4) and (2.5) and from the fact that H is symmetric and idempotent. • Hence finally, var Ye − var Yb = σ 2 (B − H) (B − H)> , which is a positive semidefinite matrix. That is, the estimator Yb is not worse than the estimator Ye . k Note. It follows from the Gauss–Markov theorem that Yb X ∼ Xβ, σ 2 H . Historical remarks • The method of least squares was used in astronomy and geodesy already at the beginning of the 19th century. • 1805: First documented publication of least squares. Adrien-Marie Legendre. Appendix “Sur le méthode des moindres quarrés” (“On the method of least squares”) in the book Nouvelles Méthodes Pour la Détermination des Orbites des Comètes (New Methods for the Determination of the Orbits of the Comets). 2.2. FITTED VALUES, RESIDUALS, GAUSS–MARKOV THEOREM 19 • 1809: Another (supposedly independent) publication of least squares. Carl Friedrich Gauss. In Volume 2 of the book Theoria Motus Corporum Coelestium in Sectionibus Conicis Solem Ambientium (The Theory of the Motion of Heavenly Bodies Moving Around the Sun in Conic Sections). • C. F. Gauss claimed he had been using the method of least squares since 1795 (which is probably true). • The Gauss–Markov theorem was first proved by C. F. Gauss in 1821 – 1823. • In 1912, A. A. Markov provided another version of the proof. • In 1934, J. Neyman described the Markov’s proof as being “elegant” and stated that Markov’s contribution (written in Russian) had been overlooked in the West. ⇒ The name Gauss–Markov theorem. Theorem 2.3 Basic properties of the residuals and the residual sum of squares. Let Y = Xβ + ε, ε ∼ 0n , σ 2 In , where rank Xn×k = r ≤ k < n. The following then holds: (i) U = Mε. (ii) SSe = Y > MY = ε> Mε. (iii) E U = 0n , var U = σ 2 M. (iv) E SSe = EY , X SSe = (n − r) σ 2 . Notes. • Point (i) of Theorem 2.3 says that the residuals can be obtained not only by projecting the ⊥ response vector Y into M X but also by projecting the vector of the error terms of the linear ⊥ model into M X . • Point (iii) of Theorem 2.3 can also be briefly written as U X ∼ 0n , σ 2 M . Proof. (i) U = MY = M (Xβ + ε) = |{z} MX β + Mε = Mε. 0n (ii) SSe = U > U = (MY )> MY > > = Y>M | {zM} Y = Y MY M (i) = ε> M> M ε = ε> M ε. 2.2. FITTED VALUES, RESIDUALS, GAUSS–MARKOV THEOREM (iii) 20 E(U ) = E(MY ) = |{z} MX β = 0n . 0n var(U ) = var(MY ) = M var(Y ) M> = M σ 2 In M> = σ 2 MM> = σ 2 M. (iv) E SSe n n n o o o = E ε> M ε = E tr ε> M ε = E tr Mεε> = tr E Mεε> n o = tr M E εε> = tr M σ 2 In = tr σ 2 M = σ 2 tr(M) | {z } var(ε) 2 = σ tr NN> = σ 2 tr N> N = σ 2 tr In−r = σ 2 (n − r), where (to remind) N denotes an n × (n − r) matrix whose columns form an orthonormal ⊥ vector basis of a residual space M X . Z Further, EY , X SSe = E SSe fX (x) dλX (x) Rn·k Z = Rn·k σ 2 (n − r) fX (x) dλX (x) = σ 2 (n − r). k Definition 2.4 Residual mean square and residual degrees of freedom. Consider a linear model Y X ∼ Xβ, σ 2 In , rank(X) = r. 1. The residual mean square13 of the model is a quantity SSe /(n − r) and will be denoted as MSe . That is, SSe . MSe = n−r 2. The residual degrees of freedom14 of the model is the dimension of the residual space and will be denotes as νe . That is, νe = n − r. Theorem 2.4 Unbiased estimator of the residual variance. The residual mean square MSe is an unbiased estimator (both conditionally given X and also with joint distribution of Y and X) of the residual variance σ 2 in a linear model respect to the Y X ∼ Xβ, σ 2 In , rank Xn×k = r ≤ k < n. Proof. Direct consequence of Theorem 2.3, point (iv). 13 reziduální střední čtverec 14 reziduální stupně volnosti k End of Lecture #2 (08/10/2015) 2.3. NORMAL EQUATIONS 2.3 21 Normal equations Start of The vector of fitted values Yb = HY is a projection of the response vector into M X . Hence, it Lecture #3 must be possible to write Yb as a linear combination of the columns of the model matrix X. That (08/10/2015) is, there exists b ∈ Rk such that Yb = Xb. (2.6) Notes. • In a full-rank model (rank Xn×k = k), linearly independent columns of X form a vector basis of M X . Hence b ∈ Rk such that Yb = Xb is unique. • If rank Xn×k = r < k, a vector b ∈ Rk such that Yb = Xb is not unique. We already know from the Gauss-Markov theorem (Theorem 2.2) that E Yb = Xβ. Hence if we manage to express Yb as Yb = Xb and b will be unique, we have a natural candidate for an estimator of the regression coefficients β. Nevertheless, before we proceed to estimation of β, we derive conditions that b ∈ Rk must satisfy to fulfill also (2.6). Definition 2.5 Sum of squares. Consider a linear model Y X ∼ Xβ, σ 2 In . The function SS : Rk −→ R given as follows 2 > Y − Xβ , SS(β) = Y − Xβ = Y − Xβ β ∈ Rk will be called the sum of squares15 of the model. Theorem 2.5 Least squares and normal equations. Assume a linear model Y X ∼ Xβ, σ 2 In . The vector of fitted values Yb equals to Xb, b ∈ Rk if and only if b solves a linear system X> Xb = X> Y . (2.7) Proof. Yb = Xb, is a projection of Y into M X ⇔ ⇔ Yb = Xb is the closest point to Y in M X 2 Yb = Xb, where b minimizes SS(β) = Y − Xβ over β ∈ Rk . Let us find conditions under which the term SS(β) attains its minimal value over β ∈ Rk . To this end, a vector of the first derivatives (a gradient) and a matrix of the second derivatives (a Hessian) of SS(β) are needed. 2 > SS(β) = Y − Xβ = Y − Xβ Y − Xβ = Y > Y − 2Y > Xβ + β > X> Xβ. ∂SS (β) = −2 X> Y + 2 X> Xβ. ∂β 15 součet čtverců 2.3. NORMAL EQUATIONS 22 ∂ 2 SS (β) = 2 X> X. ∂β∂β > (2.8) For any β ∈ Rk , the Hessian (2.8) is a positive semidefinite matrix and hence b minimizes SS(β) over β ∈ Rk if and only if ∂SS (b) = 0k , ∂β that is, if and only if X> Xb = X> Y . k Definition 2.6 Normal equations. Consider a linear model Y X ∼ Xβ, σ 2 In . The system of normal equations16 or concisely normal equations17 of the model is the linear system X> Xb = X> Y , or equivalently, the linear system X> (Y − Xb) = 0k . Note. In general, the linear system (2.7) of normal equations would not have to have a solution. Nevertheless, in our case, existence of the solution follows from the fact that it corresponds to b the projection Y of Y into the regression space M X and existence of the projection Yb is guaranteed by the projection properties known from the linear algebra lectures. On the other hand, we can also show quite easily that there exists a solution to the normal equations by using the following lemma. Lemma 2.6 Vector spaces generated by the rows of the model matrix. Let Xn×k be a real matrix. Then M X> X = M X> . Note. A vector space M X> is a vector space generated by the columns of the matrix X> , that is, it is a vector space generated by the rows of the matrix X. ⊥ ⊥ Proof. First note that M X> X = M X> is equivalent to M X> X = M X> . We will ⊥ ⊥ show this by showing that for any a ∈ Rk a ∈ M X> if and only if a ∈ M X> X . (i) a ∈ M X> ⊥ > > > ⇒ a> X> = 0> n ⇒ a X X = 0k ⊥ ⇔ a ∈ M X> X 16 systém normálních rovnic 17 normální rovnice 2.3. NORMAL EQUATIONS (ii) a ∈ M X> X ⊥ 23 ⇒ a> X> X = 0> ⇒ a> X> X a = 0 k ⇒ kXak = 0 ⇔ Xa = 0n ⇔ a> X> = 0> n ⊥ ⇔ a ∈ M X> k Notes. • Existence of a solution to normal equations (2.7) follows from the fact that its right-hand side > > > X Y ∈ M X and M X is (by Lemma 2.6) the same space as a vector space generated by the columns of the matrix of the linear system (X> X). • By Theorem A.1, all solutions to normalequations, i.e., a set of points that minimize the sum of − − squares SS(β) are given as b = X> X X> Y , where X> X is any pseudoinverse to X> X (if rank Xn×k = r < k, this pseudoinverse is not unique). − • We also have that for any b = X> X X> Y : SSe = SS(b). Notation. • In the following, symbol b will be exclusively used to denote any solution to normal equations, that is, − b = b0 , . . . , bk−1 = X> X X> Y . • For a full-rank linear model, (rank Xn×k = k), the following holds: − − −1 • The only pseudoinverse X> X is X> X = X> X . −1 > • The only solution of normal equations is b = X> X X Y which is also a unique minimizer of the sum of squares SS(β). b That is, In this case, we will denote the unique solution to normal equations as β. b = βb0 , . . . , βbk−1 = X> X −1 X> Y . β 2.4. ESTIMABLE PARAMETERS 2.4 24 Estimable parameters We have seen in the previous section that the sum of squares SS(β) does not necessarily attain a unique minimum. This happens if the model matrix Xn×k has linearly dependent columns (its rank r < k) and hence there exist (infinitely) many possibilities on how to express the vector of the fitted values Yb ∈ M X as a linear combination of the columns of the model matrix X. In other words, there exist (infinitely) many vectors b ∈ Rk such that Yb = Xb. This could also be interpreted as that there are (infinitely) many estimators of the regression parameters β leading to the (unique) unbiased estimator of the response mean µ = E(Y ) = Xβ. It then does not make much sense to talk about estimation of the regression parameters β. To avoid such situations, we now define a notion of an estimable parameter 18 of a linear model. Definition 2.7 Estimable parameter. Consider a linear model Y X ∼ Xβ, σ 2 In . Let l ∈ Rk . We say that a parameter θ = l> β is an estimable parameter of the model if for all µ ∈ M X the expression l> β does not depend on a choice of a solution to the linear system Xβ = µ. Notes. • Definition of an estimable parameter is equivalent to the requirement ∀β 1 , β 2 ∈ Rk Xβ 1 = Xβ 2 ⇒ l> β 1 = l> β 2 . That is, the estimable parameter is such a linear combination of the regression coefficients β whichdoes not depend on a choice of the β leading to the same vector in the regression space M X (leading to the same vector of the response expectation µ). • In a full-rank model (rank Xn×k = k), columns of the model matrix X form a vector basis of the regression space M X . It then follows from the properties of a vector basis that for any µ ∈ M X there exist a unique β such that Xβ = µ. Trivially, for any l ∈ Rk , the expression l> β then does not depend on a choice of a solution to the linear system Xβ = µ since there is only one such solution. In other words, in a full-rank model, any linear function of the regression coefficients β is estimable. Definition 2.8 Estimable vector parameter. Consider a linear model Y X ∼ Xβ, σ 2 In . Let l ∈ Rk . Let l1 , . . . , lm ∈ Rk . Let L be an m × k > matrix having vectors l> 1 , . . . , lm in its rows. We say that a vector parameter θ = Lβ is an estimable vector parameter of the model if all parameters θj = l> j β, j = 1, . . . , m, are estimable. 18 odhadnutelný parametr 2.4. ESTIMABLE PARAMETERS 25 Notes. • Definition of an estimable parameter is equivalent to the requirement ∀β 1 , β 2 ∈ Rk Xβ 1 = Xβ 2 ⇒ Lβ 1 = Lβ 2 . • Trivially, a vector parameter µ = E(Y ) = Xβ is always estimable. We also already know its BLUE which is the vector of fitted values Yb . • In a full-rank model (rank Xn×k = k), the regression coefficients vector β is an estimable vector parameter. Example 2.1 (Overparameterized two-sample problem). Consider a two-sample problem, where Y1 , . . . , Yn1 are assumed to be identically distributed random variables with E(Y1 ) = · · · = E(Yn1 ) = µ1 and var(Y1 ) = · · · = var(Yn1 ) = σ 2 (sample 1), Yn1 +1 , . . . , Yn1 +n2 are also assumed to be identically distributed random variables with E(Yn1 +1 ) = · · · = E(Yn1 +n2 ) = µ2 and var(Yn1 +1 ) = · · · = var(Yn1 +n2 ) = σ 2 (sample 2) and Y1 , . . . , Yn1 , Yn1 +1 , . . . , Yn1 +n2 are assumed to be independent. This situation can be described by a linear model Y ∼ Xβ, σ 2 In , n = n1 + n2 , where Y1 µ1 β0 + β1 1 1 0 . . . . . . .. .. .. .. .. .. β0 Yn1 β0 + β1 µ1 1 1 0 = . Y = µ = Xβ = µ , X = 1 0 1 , β = β1 , Y β + β n1 +1 2 0 2 β2 . . . . . .. . . . . . . . . . . . Yn1 +n2 1 0 1 β0 + β2 µ2 (i) Parameters µ1 = β0 + β1 and µ2 = β0 + β2 are (trivially) estimable. > = 0, 1, 0 and (ii) None of the elements of the vector β is estimable. For example, take β 1 β 2 = 1, 0, −1 . We have Xβ 1 = Xβ 2 = 1, . . . , 1, 0, . . . , 0 but none of the elements of β 1 and β 2 is equal. This corresponds to the fact that two means µ1 and µ2 can be expressed in infinitely many ways using three numbers β0 , β1 , β2 as µ1 = β0 + β1 and µ2 = β0 + β2 . (iii) A non-trivial estimable parameter is, e.g., θ = µ2 − µ1 = β2 − β1 = l> β, l = 0, −1, 1 . > We have for β 1 = β1,0 , β1,1 , β1,2 ∈ R3 and β 2 = β2,0 , β2,1 , β2,2 ∈ R3 : Xβ 1 = Xβ 2 ⇔ β1,0 + β1,1 = β2,0 + β2,1 , β1,0 + β1,2 = β2,0 + β2,2 ⇒ β1,2 − β1,1 = β2,2 + β2,1 ⇔ l> β 1 = l> β 2 . Definition 2.9 Contrast. Consider a linear model Y X ∼ Xβ, σ 2 In . An estimable parameter θ = c> β, given by a real > vector c = c0 , . . . , ck−1 which satisfies c> 1k = 0, i.e., k−1 X j=0 cj = 0, 2.4. ESTIMABLE PARAMETERS 26 is called contrast19 . Definition 2.10 Orthogonal contrasts. Consider a linear model Y X ∼ Xβ, σ 2 In . Contrasts θ = c> β and η = d> β given by orthogo> > nal vectors c = c0 , . . . , ck−1 and d = d0 , . . . , dk−1 , i.e., given by vectors c and d that satisfy c> d = 0, are called (mutually) orthogonal contrasts. Theorem 2.7 Estimable parameter, necessary and sufficient condition. Assume a linear model Y X ∼ Xβ, σ 2 In . (i) Let l ∈ Rk . Parameter θ = l> β is an estimable parameter if and only if l ∈ M X> . (ii) A vector θ = Lβ is an estimable vector parameter if and only if M L> ⊂ M X> . Proof. (i) θ = l> β is estimable ⇔ ∀ β 1 , β 2 ∈ Rk Xβ 1 = Xβ 2 ⇒ l> β 1 = l> β 2 ⇔ ∀ β 1 , β 2 ∈ Rk X(β 1 − β 2 ) = 0n ⇒ l> (β 1 − β 2 ) = 0 ⇔ ∀ γ ∈ Rk Xγ = 0n ⇒ l> γ = 0 ⇔ ∀ γ ∈ Rk γ orthogonal to all rows of X ⇒ l> γ = 0 ⊥ ∀ γ ∈ Rk γ ∈ M X> ⇒ l> γ = 0 l ∈ M X> . ⇔ ⇔ (ii) Direct consequence of point (i). k Note. In a full-rank model (rank Xn×k = k < n), M X> = Rk . That is, any linear function of β is indeed estimable (statement that we already concluded from the definition of an estimable parameter). 19 kontrast 2.4. ESTIMABLE PARAMETERS 27 Theorem 2.8 Gauss–Markov for estimable parameters. Let θ = l> β be an estimable parameter of a linear model Y X ∼ Xβ, σ 2 In . Let b be any solution to the normal equations. The statistic θb = l> b then satisfies: (i) θb does not depend on a choice of the solution b of the normal equations, i.e., it does not depend − on a choice of a pseudoinverse in b = X> X X> Y . (ii) θb is the best linear unbiased estimator (BLUE) of the parameter θ. − (iii) var θb = σ 2 l> X> X l , that is, − θb | X ∼ θ, σ 2 l> X> X l , where l> X> X − l does not depend on a choice of the pseudoinverse X> X − If additionally l 6= 0k then l> X> X l > 0. − . > Let further θ1 = l> 1 β and θ2 = l2 β be estimable parameters. Let θb1 = l> 1 b, Then cov θb1 , θb2 > where l> 1 X X − θb2 = l> 2 b. > = σ 2 l> 1 X X − l2 , l2 does not depend on a choice of the pseudoinverse X> X − . Proof. (i) Let b1 , b2 be two solutions to normal equations, that is, X> Y = X> X b1 = X> X b2 . By Theorem 2.5 (Least squares and normal equations): ⇔ Yb = Xb1 & Yb = Xb2 , that is, Xb1 = Xb2 . Estimability of θ: ⇒ l> b1 = l> b2 . (ii) Parameter θ = l> β is estimable. By Theorem 2.7: ⇔ l ∈ M X> ⇔ l = X> a for some a ∈ Rn ⇒ θb = a> Xb = a> Yb . That is, θb is a linear function of Yb which is the BLUE of µ = Xβ. It then follows that θb is the BLUE of the parameter a> µ = a> Xβ = l> β = θ. 2.4. ESTIMABLE PARAMETERS 28 (iii) Proof/calculations were available on the blackboard in K1. > The last part “Let further θ1 = l> 1 β and θ2 = l2 β be. . . ”: Proof/calculations were available on the blackboard in K1. k Theorem 2.9 Gauss–Markov for estimable vector parameter. Let θ = Lβ be an estimable vector parameter of a linear model Y X ∼ Xβ, σ 2 In . Let b be any solution to normal equations. The statistic b = Lb θ then satisfies: b does not depend on a choice of the solution b of the normal equations. (i) θ b is the best linear unbiased estimator (BLUE) of the vector parameter θ. (ii) θ b = σ 2 L X> X − L> , that is, (iii) var θ b | X ∼ θ, σ 2 L X> X − L> , θ − − where L X> X L> does not depend on a choice of the pseudoinverse X> X . If additionally m ≤ r and the rows of the matrix L are linearly independent then L X> X is a positive definite (invertible) matrix. Proof. Direct consequence of Theorem 2.8, except positive definiteness of L X> X situations when L has linearly independent rows. − Positive definiteness of L X> X L> if Lm×k has linearly independent rows: Proof/calculations were available on the blackboard in K1. Consequence of Theorem 2.9. Assume a full-rank linear model Y X ∼ Xβ, σ 2 In , rank Xn×k = k < n. The statistic b = X> X β −1 X> Y then satisfies: b is the best linear unbiased estimator (BLUE) of the regression coefficients β. (i) β − − L> L> in k 2.4. ESTIMABLE PARAMETERS 29 b = σ 2 X> X −1 , that is, (ii) var β b X ∼ β, σ 2 X> X −1 . β Proof. Use L = Ik in Theorem 2.9. k 2.5. PARAMETERIZATIONS OF A LINEAR MODEL 2.5 2.5.1 30 Parameterizations of a linear model Equivalent linear models Definition 2.11 Equivalent linear models. Assume two linear models: M1 : Y X1 ∼ X1 β, σ 2 In , where X1 is an n × k matrix with rank X = r and M2 : Y X2 ∼ X2 γ, σ 2 In , where X2 is an n × l matrix with rank X2 = r. We say that models M1 and M2 are equivalent if their regression spaces are the same. That is, if M X1 = M X2 . Notes. • The two equivalent models: − − • have the same hat matrix H = X X> X X> = Z Z> Z Z> and a vector of fitted values Yb = HY ; • have the same residual projection matrix M = In − H and a vector of residuals U = MY ; • have the same value of the residual sum of squares SSe = U > U , residual degrees of freedom νe = n − r and the residual mean square MSe = SSe /(n − r). • The two equivalent models provide two different parameterizations of one situation. Nevertheless, practical interpretation of the regression coefficients β ∈ Rk and γ ∈ Rl in the two models might be different. In practice, both parameterizations might be useful and this is also the reason why it often makes sense to deal with both parameterizations. 2.5.2 Full-rank parameterization of a linear model Any linear model can be parameterized such that the model matrix has linearly independent 2 columns, i.e., is of a full-rank. To see this, consider a linear model Y X ∼ Xβ, σ In ,where rank Xn×k = r ≤ k < n. If Qn×r is a matrix with the orthonormal vector basis of M X in its columns (that is, rank(Q) = r), the linear model Y Q ∼ Qγ, σ 2 In (2.9) is equivalent to the original model with the model matrix X. Nevertheless, parameterization of a model using the orthonormal basis and the Q matrix is only rarely used in practice since the interpretation of the regression coefficients γ in model (2.9) is usually quite awkward. Parameterization of a linear model using the orthonormal basis matrix Q is indeed not the only full-rank parameterization of a given linear model. There always exist infinitely many full-rank parameterizations and in reasonable practical analyses, it should always be possible to choose such a full-rank parameterization or even parameterizations that also provide practically interpretable regression coefficients. Example 2.2 (Different parameterizations of a two-sample problem). Let us again consider a two-sample problem (see also Example 2.1). That is, > Y = Y1 , . . . , Yn1 , Yn1 +1 , . . . , Yn1 +n2 , 2.5. PARAMETERIZATIONS OF A LINEAR MODEL 31 where Y1 , . . . , Yn1 are identically distributed random variables with E(Y1 ) = · · · = E(Yn1 ) = µ1 and var(Y1 ) = · · · = var(Yn1 ) = σ 2 (sample 1), Yn1 +1 , . . . , Yn1 +n2 are also identically distributed random variables with E(Yn1 +1 ) = · · · = E(Yn1 +n2 ) = µ2 and var(Yn1 +1 ) = · · · = var(Yn1 +n2 ) = σ 2 (sample 2) and Y1 , . . . , Yn1 , Yn1 +1 , . . . , Yn1 +n2 are assumed to be independent. This situation can be described by differently parameterized linear models Y X ∼ Xβ, σ 2 In , n = n1 + n2 where the model matrix X is always divided into two blocks as ! X1 X= , X2 where X1 is an n1 × k matrix having n1 identical rows x> 1 and X2 is an n2 × k matrix having n2 identical rows x> . The response mean vector µ = E(Y ) is then 2 x> β µ1 1 . . .. .. ! > x1 β µ1 X1 β µ = Xβ = = x> β = µ . X2 β 2 2 . . . . . . x> µ2 2β That is, parameterization of the model is given by choices of vectors x1 6= x2 , x1 6= 0k , x2 6= 0k leading to expressions of the means of the two samples as µ 1 = x> 1 β, µ2 = x> 2 β. The rank of the model is always r = 2. Overparameterized model x1 = 1, 1, 0 , x2 = 1, 0, 1 : 1 1 0 . . . .. .. .. 1 1 0 X= 1 0 1 , . . . . . . . . . 1 0 1 √ β0 β = β1 , β2 µ1 = β0 + β1 , µ2 = β0 + β2 . √ Orthonormal basis x1 = 1/ n1 , 0 , x2 = 0, 1/ n2 : √1 n1 . . . √1 X = Q = n1 0 . .. 0 0 .. . 0 , √1 n2 .. . √1 n2 β= β1 β2 ! , 1 µ 1 = √ β1 , n1 1 µ2 = √ β 2 , n2 β1 = √ n 1 µ1 , β2 = √ n 2 µ2 . 2.5. PARAMETERIZATIONS OF A LINEAR MODEL 32 Group means x1 = 1, 0 , x2 = 0, 1 : 1 0 . . .. .. 1 0 , X= 0 1 . . . . . . 0 1 β= β1 β2 ! , µ1 = β1 , µ2 = β2 . This could also be viewed as the overparameterized model constraint by a condition β0 = 0. Group differences x1 = 1, 1 , x2 = 1, 0 : 1 1 . . .. .. 1 1 X= 1 0 , . . . . . . 1 0 β= β0 β1 ! , µ 1 = β0 + β1 , β1 = µ1 − µ2 . µ 2 = β0 , This could also be viewed as the overparameterized model constraint by a condition β2 = 0. Deviations from the mean of the means x1 = 1, 1 , x2 = 1, −1 : 1 1 . .. .. . 1 1 , X= 1 −1 . .. . . . 1 −1 µ1 = β0 + β1 , β= β0 β1 ! , µ2 = β0 − β1 , µ1 + µ2 , 2 µ1 + µ2 β1 = µ 1 − 2 µ1 + µ2 = − µ2 . 2 β0 = This could also be viewed as the overparameterized model constraint by a condition β1 + β2 = 0. Except the overparameterized model, all above parameterizations are based on a model matrix having full-rank r = 2. End of Lecture #3 (08/10/2015) 2.6. MATRIX ALGEBRA AND A METHOD OF LEAST SQUARES 2.6 33 Matrix algebra and a method of least squares Start of 2 Lecture #4 We have seen in Section 2.5 that any linear model Y X ∼ Xβ, σ In can be reparameterized such that the model matrix X has linearly independent columns, that is, rank Xn×k = k. Remind (15/10/2015) now expressions of some quantities that must be calculated when dealing with the least squares estimation of parameters of the full-rank linear model: −1 −1 H = X X> X X> , M = In − H = In − X X> X X> , −1 −1 Yb = HY = X X> X X> Y , var Yb = σ 2 H = σ 2 X X> X X> , n −1 U = MY = Y − Yb , var U = σ 2 M = σ 2 In − X X> X X> , b = X> X −1 X> Y , β b = σ 2 X> X −1 . var β −1 The only non-trivial calculation involved in above expressions is calculation of the inverse X> X . Nevertheless, all above expressions (and many others needed in a context of the least squares estimation) can be calculated without explicit evaluation of the matrix X> X. Some of above ex−1 pressions can even be evaluated without knowing explicitely the form of the X> X matrix. To this end, methods of matrix algebra can be used (and are used by all reasonable software routines dealing with the least squares estimation). Two methods, known from the course Fundamentals of Numerical Mathematics (NMNM201), that have direct usage in the context of least squares are: • QR decomposition; • Singular value decomposition (SVD) applied to the model matrix X. Both of them can be used, among other things, to find the orthonormal vector basis of the regression space M X and to calculate expressions mentioned above. 2.6.1 QR decomposition QR decomposition of the model matrix is used, for example, by the R software (R Core Team, 2015) to estimate a linear model by the method of least squares. If Xn×k is a real matrix with rank X = k < n then we know from the course Fundamentals of Numerical Mathematics (NMNM201) that it can be decomposed as X = QR, where Qn×k = q 1 , . . . , q k , q j ∈ Rk , j = 1, . . . , k, q 1 , . . . , q k is an orthonormal basis of M X and Rk×k is upper triangular matrix. That is, Q> Q = Ik , We then have QQ> = H. X> X = R> Q> Q R = R> R. | {z } Ik (2.10) That is, R> R is a Cholesky (square root) decomposition of the symmetric matrix X> X. Note that this is a special case of an LU decomposition for symmetric matrices. Decomposition (2.10) −1 can now be used to get easily (i) matrix X> X , (ii) a value of its determinant or a value of determinant of X> X, (iii) solution to normal equations. 2.6. MATRIX ALGEBRA AND A METHOD OF LEAST SQUARES 34 −1 (i) Matrix X> X . X> X −1 = R> R −1 = R−1 R> −1 = R−1 R−1 > . That is, to invert the matrix X> X, we only have to invert the upper triangular matrix R. −1 (ii) Determinant of X> X and X> X . Let r1 , . . . , rk denote diagonal elements of the matrix R. We then have k 2 2 Y det X> X = det R> R = det(R) = rj , j=1 n −1 o n o−1 . det X> X = det X> X b = X> X (iii) Solution to normal equations β b by solving: We can obtain β −1 X> Y . X> X b = X> Y R> R b = R> Q> Y R b = Q> Y . (2.11) b it is only necessary to solve a linear system with the upper triangular That is, to get β, system matrix which can easily be done by backward substitution. > Further, the right-hand-side c = c1 , . . . , ck := Q> Y of the linear system (2.11) additionally serves to calculate the vector of fitted values. We have Yb = HY = QQ> Y = Q c = k X cj q j . j=1 That is, the vector c provides coefficients of the linear combination of the orthonormal vector basis of the regression space M X that provide the fitted values Yb . 2.6.2 SVD decomposition Use of the SVD decomposition for the least squares will not be explained in detail in this course. It is covered by the Fundamentals of Numerical Mathematics (NMNM201) course. Chapter 3 Normal Linear Model Until now, all proved theorems did not pose any distributional assumptions on the random vectors Yi , X i , X i = Xi,0 , . . . , Xi,k−1 , i = 1, . . . , n, that represent the data. We only assumed a certain form of the (conditional) expectation and the (conditional) covariance matrix of Y = Y1 , . . . , Yn given X 1 , . . . , X n (given the model matrix X). In this chapter, we will additionally assume that the response is conditionally normally distributed given the covariates which will lead us to the normal linear model. 35 3.1. NORMAL LINEAR MODEL 3.1 36 Normal linear model Definition 3.1 Normal linear model. The data Yi , X i , i = 1, . . . , n, satisfy a normal linear model1 if they satisfy a linear model with i.i.d. errors with 2 Yi X i ∼ N X > i β, σ , where β ∈ Rk and 0 < σ 2 < ∞ are unknown parameters. Notation. It follows from Definition 3.1, from definition of a linear model with i.i.d. errors (Definition 1.4) and from properties of a normal distribution that a joint conditional distribution of Y given X is multivariate normal with a mean Xβ and a covariance matrix σ 2 In . Hence the fact that data follow a normal linear model will be indicated by notation Y X ∼ Nn Xβ, σ 2 In . Definition 3.1 using the error terms The data Yi , X i , i = 1, . . . , n, satisfy a normal linear model if Yi = X > i β + εi , i = 1, . . . , n, i.i.d. where ε1 , . . . , εn ∼ N (0, σ 2 ), ε = ε1 , . . . , εn and X independent, and β ∈ Rk and 0 < σ 2 < ∞ are unknown parameters. Notation. To indicate that data follow a normal linear model, we will also use notation Y = Xβ + ε, 1 normální lineární model ε ∼ Nn (0n , σ 2 In ). 3.2. PROPERTIES OF THE LEAST SQUARES ESTIMATORS UNDER THE NORMALITY 3.2 37 Properties of the least squares estimators under the normality Theorem 3.1 Least squares estimators under the normality. Let Y X ∼ Nn Xβ, σ 2 In , rank Xn×k = r. Let Lm×k is a real matrix with non-zero rows > > > > l> = l> = Lβ is an estimable parameter. 1 , . . . , lm such that θ = θ1 , . . . , θm 1 β, . . . , lm β > > > > b b b Let θ = θ1 , . . . , θm = l b, . . . , l b = Lb be its least squares estimator. Further, let 1 m − V = L X> X L> = vj,t j,t=1,...,m , 1 1 , ..., √ , D = diag √ v1,1 vm,m θbj − θj Tj = p , j = 1, . . . , m, MSe vj,j > 1 b−θ . T = T1 , . . . , Tm = √ D θ MSe The following then holds. (i) Yb X ∼ Nn Xβ, σ 2 H . (ii) U X ∼ Nn 0n , σ 2 M . b X ∼ Nm θ, σ 2 V . (iii) θ (iv) Statistics Yb and U are conditionally, given X, independent. b and SSe are conditionally, given X, independent. (v) Statistics θ Yb − Xβ 2 (vi) ∼ χ2r . σ2 (vii) SSe ∼ χ2n−r . σ2 (viii) For each j = 1, . . . , m, Tj ∼ tn−r . (ix) T | X ∼ mvtm,n−r DVD . (x) If additionally rank Lm×k = m ≤ r then the matrix V is invertible and > −1 1 b b − θ ∼ Fm, n−r . θ−θ MSe V θ m Proof. Proof/calculations were available on the blackboard in K1. k b = X> X −1 X> Y and under the normality assumption, In a full-rank linear model, we have β b of the regression coefficients β. Theorem 3.1 can be used to state additional properties of the LSE β 3.2. PROPERTIES OF THE LEAST SQUARES ESTIMATORS UNDER THE NORMALITY 38 Consequence of Theorem 3.1: Least squares estimator of the regression coefficients in a full-rank normal linear model. Let Y X ∼ Nn Xβ, σ 2 In , rank Xn×k = k. Further, let −1 V = X> X = vj,t j,t=0,...,k−1 , 1 1 D = diag √ , ..., √ . v0,0 vk−1,k−1 The following then holds. b X ∼ Nk β, σ 2 V . (i) β b and SSe are conditionally, given X, independent. (ii) Statistics β βbj − βj (iii) For each j = 0, . . . , k − 1, Tj := p ∼ tn−k . MSe vj,j > 1 b − β ∼ mvtk,n−k DVD . D β (iv) T := T0 , . . . , Tk−1 = √ MSe > 1 b > b (v) β − β MS−1 e X X β − β ∼ Fk, n−k . k Proof. Use L = Ik in Theorem 3.1 and realize that the only pseudoinverse to the matrix X> X in −1 a full-rank model is the inverse X> X . k Theorem 3.1 and its consequence can now be used to perform principal statistical inference, i.e., calculation of confidence intervals and regions, testing statistical hypotheses, in a normal linear model. 3.2.1 Statistical inference in a full-rank normal linear model Assume a full-rank normal linear model Y X ∼ Nn Xβ, σ 2 In , rank Xn×k = k and keep −1 denoting V = X> X = vj,t j,t=0,...,k−1 . Inference on a chosen regression coefficient First, take a chosen j ∈ 0, . . . , k − 1 . We then have the following. • Standard error of βbj and confidence interval for βj We have var βbj = σ 2 vj,j (Consequence of Theorem 2.9) which is unbiasedly estimated as MSe vj,j (Theorem 2.4). The square root of this quantity, i.e., estimated standard deviation of βbj is then called as standard error 2 of the estimator βbj . That is, p S.E. βbj = MSe vj,j . (3.1) 2 směrodatná, příp. standardní chyba 3.2. PROPERTIES OF THE LEAST SQUARES ESTIMATORS UNDER THE NORMALITY 39 The standard error (3.1) is also the denominator of the t-statistic Tj from point (iii) of Consequence of Theorem 3.1. Hence the lower and the upper bounds of the Wald-type (1 − α) 100% confidence interval for βj based on the statistic Tj are α βbj ± S.E. βbj tn−k 1 − . 2 Analogously, also one-sided confidence interval can be calculated. • Test on a value of βj Suppose that for a given βj0 ∈ R, we aim in testing H0 : H1 : βj = βj0 , βj 6= βj0 . The Wald-type test based on point (iii) of Consequence of Theorem 3.1 proceeds as follows: βbj − βj0 βbj − βj0 . Test statistic: Tj,0 = =p MSe vj,j S.E. βbj Reject H0 if α |Tj,0 | ≥ tn−k 1 − . 2 P-value when Tj,0 = tj,0 : p = 2 CDFt, n−k − |tj,0 | . Analogously, also one-sided tests can be conducted. End of Lecture #4 (15/10/2015) Simultaneous inference on a vector of regression coefficients Start of When the interest lies in the inference for the full vector of the regression coefficients β, the Lecture #6 (22/10/2015) following procedures can be used. • Simultaneous confidence region3 for β It follows from point (v) of Consequence of Theorem 3.1 that the simultaneous (1 − α) 100% confidence region for β is the set n o b > MS−1 X> X β − β b < k Fk,n−k (1 − α) , β ∈ Rk : β − β e which is an ellipsoid with center: shape matrix: diameter: b β, −1 b , c β MSe X> X = var p k Fk,n−k (1 − α). Remember from the linear algebra and geometry lectures that the shape matrix determines the principal directions of the ellipsoid as those are given by the eigen vectors of this matrix. In this case, the principal directions of the confidence ellipsoid are given by the eigen vectors of the b . c β estimated covariance matrix var • Test on a value of β Suppose that for a given β 0 ∈ Rk , we aim in testing H0 : H1 : β = β0 , β 6= β 0 . The Wald-type test based on point (v) of Consequence of Theorem 3.1 proceeds as follows: > 1 b > 0 b Test statistic: Q0 = β − β 0 MS−1 e X X β−β . k 3 Reject H0 if Q0 ≥ Fk,n−k (1 − α). P-value when Q0 = q0 : p = 1 − CDFF , k,n−k q0 . simultánní konfidenční oblast 3.2. PROPERTIES OF THE LEAST SQUARES ESTIMATORS UNDER THE NORMALITY 3.2.2 40 Statistical inference in a general rank normal linear model Let us now assume a geneal rank normal linear model Y X ∼ Nn Xβ, σ 2 In , rank Xn×k = r ≤ k. Inference on an estimable parameter Let θ = l> β, l 6= 0k , be an estimable parameter and let θb = l> b be its least squares estimator. • Standard error of θb and confidence interval for θ − − We have var θb = σ 2 l> X> X l (Theorem 2.8) which is unbiasedly estimated as MSe l> X> X l (Theorem 2.4). Hence the standard error of θb is − q b S.E. θ = MSe l> X> X l. (3.2) The standard error (3.2) is also the denominator of the appropriate t-statistic from point (viii) of Theorem 3.1. Hence the lower and the upper bounds of the Wald-type (1 − α) 100% confidence interval for θ based on this t-statistic are α . θb ± S.E. θb tn−r 1 − 2 Analogously, also one-sided confidence interval can be calculated. • Test on a value of θ Suppose that for a given θ0 ∈ R, we aim in testing H0 : H1 : θ = θ0 , θ 6= θ0 . The Wald-type test based on point (viii) of Theorem 3.1 proceeds as follows: θb − θ0 θb − θ0 Test statistic: T0 = =q − . S.E. θb MSe l> X> X l α Reject H0 if |T0 | ≥ tn−r 1 − . 2 P-value when T0 = t0 : p = 2 CDFt, n−r − |t0 | . Analogously, also one-sided tests can be conducted. Simultaneous inference on an estimable vector parameter Finally, let θ = Lβ be an estimable parameter, where L is an m × k matrix with m ≤ r linearly b = Lb be the least squares estimator of θ. independent rows. Let θ • Simultaneous confidence region for θ It follows from point (x) of Theorem 3.1 that the simultaneous (1 − α) 100% confidence region for θ is the set n o > n − > o−1 m > b b θ ∈R : θ−θ MSe L X X L θ − θ < m Fm,n−r (1 − α) , which is an ellipsoid with center: shape matrix: diameter: b θ, − b , c θ MSe L X> X L> = var p m Fm,n−r (1 − α). 3.2. PROPERTIES OF THE LEAST SQUARES ESTIMATORS UNDER THE NORMALITY • Test on a value of θ Suppose that for a given θ 0 ∈ Rm , we aim in testing H0 : H1 : 41 θ = θ0 , θ 6= θ 0 . The Wald-type test based on point (x) of Theorem 3.1 proceeds as follows: > n − o−1 1 b b − θ0 . Test statistic: Q0 = θ θ − θ0 MSe L X> X L> m Reject H0 if Q0 ≥ Fm,n−r (1 − α). P-value when Q0 = q0 : p = 1 − CDFF , m,n−r q0 . Note. Assume again a full-rank model (r = k) and take L as a submatrix of the identity matrix Ik by selecting some of its rows. The above procedures can then be used to infer simultaneously on a subvector of the regression coefficients β. Note. All tests, confidence intervals and confidence regions derived in this Section were derived under the assumption of a normal linear model. Nevertheless, we show in Chapter 13 that under certain conditions, all those methods of statistical inference remain asymptotically valid even if normality does not hold. 3.3. CONFIDENCE INTERVAL FOR THE MODEL BASED MEAN, PREDICTION INTERVAL 3.3 42 Confidence interval for the model based mean, prediction interval We keep assuming that the data Yi , X i , i = 1, . . . , n, follow a normal linear model. That is, Yi = X > i β + εi , In other words, i.i.d. εi ∼ N (0, σ 2 ). 2 Yi X i ∼ N (X > i β, σ ) and Y1 , . . . , Yn are conditionally independent given X 1 , . . . , X n . Remember that X ⊆ Rk denote a sample space of the covariate random vectors X 1 , . . . , X n . Let xnew ∈ X and let Ynew = x> new β + εnew , > where εnew ∼ N (0, σ 2 ) is independent of ε = ε1 , . . . , εn . A value of Ynew is thus a value of a “new” observation sampled from the conditional distribution 2 Ynew X new = xnew ∼ N (x> new β, σ ) independently of the “old” observations. We will now tackle two important problems: (i) Interval estimation of µnew := E Ynew X new = xnew = x> new β. (ii) Interval estimation of the value of the random variable Ynew itself, given the covariate vector X new = xnew . Solution to the outlined problems will be provided by the following theorem. Theorem 3.2 Confidence interval for the model based mean, prediction interval. Let Y = Xβ + ε, ε ∼ Nn (0n , σ 2 In ), rank Xn×k = r. Let xnew ∈ X ∩ M X> , xnew 6= 0k . Let εnew ∼ N (0, σ 2 ) is independent of ε. Finally, let Ynew = x> new β + εnew . The following then holds: (i) µnew = x> new β is estimable, µ bnew = x> new b is its best linear unbiased estimator (BLUE) with the standard error of q − > S.E. µ bnew = MSe x> xnew new X X and the lower and the upper bound of the (1 − α) 100% confidence interval for µnew are α µ bnew ± S.E. µ bnew tn−r 1 − . (3.3) 2 (ii) A (random) interval with the bounds α µ bnew ± S.E.P. xnew tn−r 1 − , (3.4) 2 where r S.E.P. xnew = n o >X −x , MSe 1 + x> X new new covers with the probability of (1 − α) the value of Ynew . (3.5) 3.3. CONFIDENCE INTERVAL FOR THE MODEL BASED MEAN, PREDICTION INTERVAL 43 Proof. Proof/calculations were available on the blackboard in K1. k Terminology (Confidence interval for the model based mean, prediction interval, standard error of prediction). • The interval with the bounds (3.3) is called the confidence interval for the model based mean. • The interval with the bounds (3.4) is called the prediction interval. • The quantity (3.5) is called the standard error of prediction. Terminology (Fitted regression function). b of the regression Suppose that the corresponding linear model is of full-rank with the LSE β coefficients. The function b m(x) b = x> β, x ∈ X, which, by Theorem 3.2, provides BLUE’s of the values of µ(x) := E Ynew X new = x = x> β and also provides predictions for Ynew = x> β + εnew , is called the fitted regression function.4 Terminology (Confidence band around the regression function, prediction band). As was explained in Section 1.2.7, the covariates X i ∈ X ⊆ Rk used in the linear model are often obtained by transforming some original covariates Z i ∈ Z ⊆ Rp . Common situation is that Z ⊆ R is an interval and > > X i = Xi,0 , . . . , Xi,k−1 = t0 (Zi ), . . . , tk−1 (Zi ) = t(Zi ), i = 1, . . . , n, where t : R −→ Rk is a suitable transformation such that E Yi Zi = t> (Zi )β = X > i β. b of the regression Suppose again that the corresponding linear model is of full-rank with the LSE β coefficients. Confidence intervals for the model based mean or prediction intervals can then be calculated for an (equidistant) sequence of values znew,1 , . . . , znew,N ∈ Z and then drawn over a scatterplot of observed data Y1 , Z1 , . . . , Yn , Zn . In this way, two different bands with a fitted regression function b m(z) b = t> (z)β, z ∈ Z, going through the middle of both the bands, are obtained. In this context, (i) The band based on the confidence intervals for the model based mean (Eq. 3.3) is called the confidence band around the regression function;5 (ii) The band based on the prediction intervals (Eq. 3.4) is called the prediction band.6 4 odhadnutá regresní funkce 5 pás spolehlivosti okolo regresní funkce 6 predikční pás 3.4. DISTRIBUTION OF THE LINEAR HYPOTHESES TEST STATISTICS UNDER THE ALTERNATIVE 3.4 44 Distribution of the linear hypotheses test statistics under the alternative Beginning of Section 3.2 provided classical tests of the linear hypotheses (hypotheses on the values of estimable skipped part parameters). To allow for power or sample size calculations, we additionally need distribution of the test statistics under the alternatives. Theorem 3.3 Distribution of the linear hypothesis test statistics under the alternative. Let Y X ∼ Nn Xβ, σ 2 In , rank(Xn×k ) = r ≤ k. Let l 6= 0k such that θ = l> β is estimable. Let θb = l> b be its LSE. Let θ0 , θ1 ∈ R, θ0 = 6 θ1 and let T0 = q θb − θ0 MSe l > − X> X l . Then under the hypothesis θ = θ1 , T0 X ∼ tn−r (λ), θ1 − θ0 λ= q − . σ 2 l > X> X l Note. The statistic T0 is the test statistic to test the null hypothesis H0 : θ = θ0 using point (viii) of Theorem 3.1. Proof. Proof/calculations were skipped and are not requested for the exam. k Theorem 3.4 Distribution of the linear hypotheses test statistics under the alternative. Let Y X ∼ Nn Xβ, σ 2 In , rank(Xn×k ) = r ≤ k. Let Lm×k be a real matrix with m ≤ r linearly b = Lb be its LSE. Let θ 0 , θ 1 ∈ Rm , θ 0 6= θ 1 independent rows such that θ = Lβ is estimable. Let θ and let > n − o−1 1 b b − θ0 . Q0 = θ − θ0 MSe L X> X L> θ m Then under the hypothesis θ = θ 1 , Q0 X ∼ Fm,n−r (λ), λ = θ1 − θ0 > n 2 − o−1 1 σ L X> X L> θ − θ0 . Note. The statistic Q0 is the test statistic to test the null hypothesis H0 : θ = θ0 using point (x) of Theorem 3.1. 3.4. DISTRIBUTION OF THE LINEAR HYPOTHESES TEST STATISTICS UNDER THE ALTERNATIVE Proof. Proof/calculations were skipped and are not requested for the exam. 45 k Note. We derived only a conditional (given the covariates) distribution of the test statistics at hand. This corresponds to the fact that power and sample size calculations for linear models are mainly used in the area of designed experiments7 where the covariate values, i.e., the model matrix X is assumed to be fixed and not random. A problem of the sample size calculation then involves not only calculation of needed sample size n but also determination of the form of the model matrix X. More can be learned in the course Experimental Design (NMST436).8 7 navržené experimenty 8 Návrhy experimentů (NMST436) End of skipped part Chapter 4 Basic Regression Diagnostics We will now assume that data are represented by n random vectors Yi , Z i , Z i = Zi,1 , . . . , Zi,p ∈ Z ⊆ Rp i = 1, . . . , n. We keep considering that the principal aim of the statistical analysis is to find a suitable model to express the (conditional) response expectation E Y := E Y Z , where Z is a matrix with vectors Z 1 , . . ., Z n in its rows. Suppose that t : Z −→ X ⊆ Rk is a transformation of covariates leading to the model matrix X> t> (Z 1 ) 1 . . . . X= rank Xn×k = r ≤ k. . = . =: t(Z), X> t> (Z n ) n 46 4.1. (NORMAL) LINEAR MODEL ASSUMPTIONS 4.1 47 (Normal) linear model assumptions Basis for statistical inference shown for the by now was derived whilek assuming a linear model 2 data, i.e., while assuming that E Y Z = Xβ for some β ∈ R and var Y Z = σ In . For the data Yi , X i , i = 1, . . . , n, where we directly work with the transformed covariate vectors, this means the following assumptions (i = 1, . . . , n): (A1) E Yi X i = x = x> β for some β ∈ Rk and any x ∈ X . ≡ Correct regression function m(z) = t> (z)β, z ∈ Z, correct choice of transformation t of the original covariates leading to linearity of the (conditional) response expectation. (A2) var Yi X i = x = σ 2 for some σ 2 irrespective of the value of x ∈ X . ≡ The conditional response variance is constant (does not depend on the covariates or other factors) ≡ homoscedasticity 1 of the response. (A3) cov Yi , Yl X = 0, i 6= l. ≡ The responses are conditionally uncorrelated. Some of our results (especially those shown in Chapter 3) were derived while additionally assuming normality of the response, i.e., while assuming (A4) Yi | X i = x ∼ N x> β, σ 2 , x ∈ X . ≡ Normality of the response. If we use a definition of the linear model using the error terms, i.e., while assuming that Y = Xβ + ε for some β ∈ Rk , the linear model assumptions are all transferred into assumptions on the error terms ε = ε1 , . . . , εn . Namely (i = 1, . . . , n): (A1) E εi = 0. ≡ This again means that a structural part of the model stating that E Y X = Xβ for some β ∈ Rk is correctly specified, or in other words, that the regression function of the model is correctly specified. (A2) var εi = σ 2 for some σ 2 which is constant irrespective of the value if i. ≡ The error variance is constant ≡ homoscedasticity of the errors. (A3) cov εi , εl = 0, i 6= l. ≡ The errors are uncorrelated. Possible assumption of normality is transferred into the errors as (A4) εi ∼ N 0, σ 2 i.i.d. ≡ The errors are normally distributed and owing to previous assumptions, ε1 , . . . , εn ∼ N (0, σ 2 ). Remember now that many important results, especially those already derived in Chapter 2, are valid even without assuming normality of the response/errors. Moreover, we shall show in Chapter 13 that also majority of inferential tools based on results of Chapters 3 and 5 are, under certain conditions, asymptotically valid even if normality does not hold. 1 homoskedasticita 4.1. (NORMAL) LINEAR MODEL ASSUMPTIONS 48 In general, if inferential tools based on a statistical model with certain properties (assumptions) are to be used, we should verify, at least into some extent, validity of those assumptions with a particular dataset. In a context of regression models, the tools to verify the model assumptions are usually referred to as regression diagnostic 2 tools. In this chapter, we provide only the most basic graphical methods. Additional, more advanced tools of the regression diagnostics will be provided in Chapters 10 and 14. As already mentioned above, the assumptions (A1)–(A4) are not equally important. Some of them are not needed to justify usage of a particular inferential tool (estimator, statistical test, . . . ), see assumptions and proofs of corresponding Theorems. This should be taken into account when using the regression diagnostics. It is indeed not necessary to verify those assumptions that are not needed for a specific task. It should finally be mentioned that with respect to the importance of the assumptions (A1)–(A4), far the most important is assumption (A1) concerning a correct specification of the regression function. Remember that practically all Theorems in this lecture that are related to the inference of a linear model use in their proofs, in some sense, the on the parameters assumption E Y X ∈ M X . Hence if this is not satisfied, majority of the traditional statistical inference is not correct. In other words, special attention in any data analysis should be devoted to verifying the assumption (A1) related to a correct specification of the regression function. As we shall show, the assumptions of the linear model are basically checked through exploration of the properties of the residuals U of the model, where − U = MY , M = In − X X> X X> = mi,l i,l=1,...,n . When doing so, it is exploited that each of assumptions (A1)–(A4) implies a certain property of the residuals stated earlier in Theorems 2.3 (Basic properties of the residuals and the residual sum of squares) or will be stated later in Theorem 3.1 (Properties of the LSE under the normality). It follows from those theorems (or their proofs) the following: 1. (A1) =⇒ E U X =: E U = 0n . 2. (A1) & (A2) & (A3) =⇒ var U X =: var U = σ 2 M. 3. (A1) & (A2) & (A3) & (A4) =⇒ U | X ∼ Nn 0n , σ 2 M . Usually, the right-hand side of the implication is verified and if it is found not to be satisfied, we know that also the left-hand side of the implication (a particular assumption or a set of assumptions) is not fulfilled. Clearly, if we conclude that the right-hand side of the implication is fulfilled, we still do not know whether the left-hand side (a model assumption) is valid. Nevertheless, it is common to most of the statistical diagnostic tools that they are only able to reveal unsatisfied model assumptions but are never able to confirm their validity. An uncomfortable property ofthe residuals of the linear model is the fact that even if the errors (ε) are homoscedastic (var εi = σ 2 for all i = 1, . . . , n), the residuals U are, in general, heteroscedastic (having unequal variances). Indeed, even if the assumption (A2) if fulfilled, we have var U = σ 2 M, var Ui = σ 2 mi,i (i = 1, . . . , n), where note that the residual projection matrix M, in general, does not have a constant diagonal m1,1 , . . . , mn,n . Moreover, the matrix M is even not a diagonal matrix. That is, even if the errors ε1 , . . . , εn are uncorrelated, the residuals U1 , . . . , Un are, in general, correlated. This must be taken into account when the residuals U are used to check validity of assumption (A2). The problem of heteroscedasticity of the residuals U is then partly solved be defining so called standardized residuals. 2 regresní diagnostika 4.2. STANDARDIZED RESIDUALS 4.2 49 Standardized residuals Consider a linear model Y X ∼ Xβ, σ 2 In , with the vector or residuals U = U1 , . . . , Un , the residual mean square MSe , and the residual projection matrix M having a diagonal m1,1 , . . . , mn.n . The following definition is motivated by the facts following the properties of residuals shown in Theorem 2.3: E U = 0n , var U = σ 2 M, ! ! Ui Ui E p = 0, var p = 1, if mi,i > 0, i = 1, . . . , n. (4.1) σ 2 mi,i σ 2 mi,i Definition 4.1 Standardized residuals. The standardized residuals3 or the vector of standardized residuals of the model is a vector U std = U1std , . . . , Unstd , where Ui , mi,i > 0, p MSe mi,i Uistd = i = 1, . . . , n. undefined, m = 0, i,i Notes. • It will be shown in Section 10.4 that if a normal linear model is assumed, i.e., if Y X ∼ Nn Xβ, σ 2 In and if for given i ∈ {1, . . . , n}, mi,i > 0 then, analogously to (4.1), E Uistd = 0, var Uistd = 1. • Unfortunately, even in a normal linear model, the standardized residuals U1std , . . . , Unstd are, in general, • neither normally distributed; • nor uncorrelated. • In some literature (and some software packages), the standardized residuals are called studentized residuals4 . • In other literature including those course notes (and many software packages including R), the term studentized residuals is reserved for a different quantity which we shall define in Chapter 14. 3 standardizovaná rezidua 4 studentizovaná rezidua 4.3. GRAPHICAL TOOLS OF REGRESSION DIAGNOSTICS 4.3 50 Graphical tools of regression diagnostics In the whole section, the columns of the model matrix X (the regressors), will be denoted as X 0 , . . . , X k−1 , i.e., X = X 0 , . . . , X k−1 . Remember that usually X 0 = 1, . . . , 1 is an intercept column. Further, in many situations, see Section 5.2 dealing with a submodel obtained by omitting some regressors, the current model matrix X is the model matrix of just a candidate submodel (playing the role of the model matrix X0 in Section regressors are available to model the response expectation 5.2) and perhaps additional 1 E Y Z . Let us denote them as V , . . . , V m . That is, in the notation of Section 5.2, X1 = V 1 , . . . , V m . The reminder of this section provides purely an overview of basic residual plots that are used as basic diagnostic tools in the context of a linear regression. More explanation on use of those plots will be/was provided during the lecture and the exercise classes. 4.3.1 (A1) Correctness of the regression function To detect: Overall inappropriateness of the regression function ⇒ scatterplot Yb , U of residuals versus fitted values. Nonlinearity of the regression function with respect to a particular regressor X j ⇒ scatterplot X j , U of residuals versus that regressor. Possibly omitted regressor V ⇒ scatterplot V , U of residuals versus that regressor. For all proposed plots, a slightly better insight is obtained if standardized residuals U std are used instead of the raw residuals U . 4.3.2 (A2) Homoscedasticity of the errors To detect Residual variance that depends on the response expectation ⇒ scatterplot Yb , U of residuals versus fitted values. Residual variance that depends on a particular regressor X j ⇒ scatterplot X j , U of residuals versus that regressor. Residual variance that depend on a regressor V not included in the model ⇒ scatterplot V , U of residuals versus that regressor. 4.3. GRAPHICAL TOOLS OF REGRESSION DIAGNOSTICS 51 For all proposed plots, a better insight is obtained if standardized residuals U std are used instead of the raw residuals U . This due to the fact that even if homoscedasticity of the errors is fulfilled, 2 the raw residuals U are not necessarily homoscedastic (var U Z = σ M), but the standardized residuals are homoscedastic having all a unity variance if additionally normality of the response holds. So called scale-location plots are obtained, if on the above proposed plots, the vector of raw residuals U is replaced by a vector q q U std , . . . , Unstd . 1 4.3.3 (A3) Uncorrelated errors End of Lecture #6 (22/10/2015) Start of Assumption of uncorrelated errors is often justified by the used data gathering mechanism (e.g., Lecture #8 observations/measurements performed on clearly independently behaving units/individuals). In (29/10/2015) that case, it does not make much sense to verify this assumption. Two typical situation when uncorrelated errors cannot be taken for granted are (i) repeated observations performed on N independently behaving units/subjects; (ii) observations performed sequentially in time where the ith response value Yi is obtained in time ti and the observational occasions t1 < · · · < tn form an equidistant sequence. In the following, we will not discuss any further the case (i) of repeated observations. In that case, a simple linear model is in most cases fully inappropriate for a statistical inference and more advanced models and methods must be used, see the course Advanced Regression Models (NMST432). In case (ii), the errors ε1 , . . . , εn can be considered as a time series5 . The assumptions (A1)–(A3) of the linear model then states that this time series (the errors of the model) forms a white noise 6 . Possible serial correlation (autocorrelation) between the error terms is then usually considered as possible violation of the assumption (A3) of uncorrelated errors. As stated above, even if the errors are uncorrelated and assumption (A3) is fulfilled, the residuals U are in general correlated. Nevertheless, the correlation is usually rather low and the residuals are typically used to check assumption (A3) and possibly to detect a form of the serial correlation present in data at hand. See Stochastic Processes 2 (NMSA409) course for basic diagnostic methods that include: • Autocorrelation and partial autocorrelation plot based on residuals U . • Plot of delayed residuals, that is a scatterplot based on points (U1 , U2 ), (U2 , U3 ), . . ., (Un−1 , Un ). 4.3.4 (A4) Normality To detect possible non-normality of the errors, standard tools used to check normality of a random sample known from the course Mathematical Statistics 1 (NMSA331) are used, now with the vector of residuals U or standardized residuals U std in place of the random sample which normality is to be checked. A basic graphical tool to check the normality of a sample is then • the normal probability plot (the QQ plot). Usage of both the raw residuals U and the standardized residuals U std to check the normality assumption (A4) bears certain inconveniences. If all assumptions of the normal linear model are fulfilled, then 5 časová řada 6 bílý šum 4.3. GRAPHICAL TOOLS OF REGRESSION DIAGNOSTICS 52 The raw residuals U satisfy U | Z ∼ Nn 0m , σ 2 M . That is, they maintain the normality, nevertheless, they are, in general, not homoscedastic (var Ui = σ 2 mi,i , i = 1, . . . , n). Hence seeming non-normality of a “sample” U1 , . . . , Un might be caused by the fact that the residuals are imposed to different variability. The standardized residuals U std satisfy E Uistd Z = 0, var Uistd Z = 1 for all i = 1, . . . , n. That is, the standardized residuals are homoscedastic (with a known variance of one), nevertheless, they are not necessarily normally distributed. On the other hand, deviation of the distributional shape of the standardized residuals from the distributional shape of the errors ε is usually rather minor and hence the standardized residuals are usually useful in detecting non-normality of the errors. Chapter 5 Submodels In this chapter, we will data being represented by n random vectors Yi , Z i , again consider Z i = Zi,1 , . . . , Zi,p ∈ Z ⊆ Rp , i = 1, . . . , n. The main aim is still to find a suitable model to express the (conditional) response expectation E Y := E Y Z , where Z is a matrix with vectors Z 1 , . . ., Z n in its rows. Suppose that t0 : Rp −→ Rk0 and t : Rp −→ Rk are two transformations of covariates leading to model matrices > X 01 X 01 = t0 (Z 1 ), X> X 1 = t(Z 1 ), 1 . . . .. 0 . .. . X = X= (5.1) . . , . , > X 0n = t0 (Z n ), X> X n = t(Z n ). X 0n n Briefly, we will write Let X0 = t0 (Z), X = t(Z). rank(X0 ) = r0 , rank(X) = r, (5.2) where 0 < r0 ≤ k0 < n, 0 < r ≤ k < n. We will now deal with a situation when the matrices X0 and X determine two linear models: Model M0 : Y | Z ∼ X0 β 0 , σ 2 In , Model M : Y | Z ∼ Xβ, σ 2 In , and the task is to decide on whether one of the two models fits “better” the data. In this chapter, we limit ourselves to a situation when M0 is so called submodel of the model M. 53 5.1. SUBMODEL 5.1 54 Submodel Definition 5.1 Submodel. We say that the model M0 is the submodel1 (or the nested model2 ) of the model M if M X0 ⊂ M X with r0 < r. Notation. Situation that a model M0 is a submodel of a model M will be denoted as M0 ⊂ M. Notes. • Submodel provides a more parsimonious expression of the response expectation E Y . • The fact that the submodel M0 holds means E Y ∈ M X0 ⊂ M X . That is, if the submodel M0 holds then also the larger model M holds. That is, there exist β 0 ∈ Rk0 and β ∈ Rk such that E Y = X0 β 0 = Xβ. • The fact submodel M0 does not hold but the model M holds means that E Y ∈ that the M X \ M X0 . That is, there exist no β 0 ∈ Rk0 such that E Y = X0 β 0 . 5.1.1 Projection considerations Decomposition of the n-dimensional Euclidean space Since M X0 ⊂ M X ⊆ Rn , it is possible to construct an orthonormal vector basis Pn×n = p1 , . . . , pn of the n-dimensional Euclidean space as P = Q0 , Q1 , N , where • Q0n×r0 : orthonormal vector basis of a submodel regression space, i.e., M X0 = M Q0 . • Q1n×(r−r0 ) : orthonormal vectors such that Q := Q0 , Q1 is an orthonormal vector basis of a model regression space, i.e., M X = M Q = M Q0 , Q1 . 1 podmodel 2 vnořený model 5.1. SUBMODEL 55 • Nn×(n−r) : orthonormal vector basis of a model residual space, i.e., M X ⊥ =M N . Further, • N0n×(n−r0 ) := Q1 , N : orthonormal vector basis of a submodel residual space, i.e., M X0 ⊥ = M N0 = M Q1 , N . It follows from the orthonormality of columns of the matrix P: > > In = P> P = P P> = Q0 Q0 + Q1 Q1 + N N> = Q Q> + N N> > > = Q0 Q0 + N0 N0 . Notation. In the following, let > H0 = Q0 Q0 , > > M0 = N0 N0 = Q1 Q1 + N N> . Notes. • Matrices H0 and M0 which are symmetric and idempotent, are projection matrices into the regression and residual space, respectively, of the submodel. • The hat matrix and the residual projection matrix of the model can now also be written as > > > H = Q Q> = Q0 Q0 + Q1 Q1 = H0 + Q1 Q1 , > M = N N> = M0 − Q1 Q1 . Projections into subspaces of the n-dimensional Euclidean space Let y ∈ Rn . We can then write y = In y = Q0 Q0 > + Q1 Q1 > > > > > + NN> y = Q0 Q0 y + Q1 Q1 y + NN> y | {z } | {z } u b y = Q0 Q0 y + Q1 Q1 y + NN> y. | {z } | {z } 0 0 u b y We have b = Q0 Q0 • y > + Q1 Q1 > y = Hy ∈ M X . 5.1. SUBMODEL 56 • u = N N> y = My ∈ M X ⊥ . > b 0 := Q0 Q0 y = H0 y ∈ M X0 . • y ⊥ > • u0 := Q1 Q1 + N N> y = M0 y ∈ M X0 . > b−y b 0 = u0 − u. • d := Q1 Q1 y = y 5.1.2 Properties of submodel related quantities Notation (Quantities related to a submodel). When dealing with a pair of a model and a submodel, quantities related to the submodel will be denoted by a superscript (or by a subscript) 0. In particular: 0 > • Yb = H0 Y = Q0 Q0 Y : fitted values in the submodel (projection of Y into the submodel regression space. 0 > • U 0 = Y − Yb = M0 Y = Q1 Q1 + NN> Y : residuals of the submodel. 2 • SS0e = U 0 : residual sum of squares of the submodel. • νe0 = n − r0 : submodel residual degrees of freedom. • MS0e = SS0e : submodel residual mean square. νe0 Additionally, as D, we denote projection of the response vector Y into the space M Q1 , i.e., > 0 D = Q1 Q1 Y = Yb − Yb = U 0 − U . (5.3) Theorem 5.1 On a submodel. Consider two linear models M : Y | Z ∼ Xβ, σ 2 In and M0 : Y | Z ∼ X0 β 0 , σ 2 In such that M0 ⊂ M. Let the submodel M0 holds, i.e., let E Y ∈ M X0 . Then 0 (i) Yb is the best linear unbiased estimator (BLUE) of a vector parameter µ0 = X0 β 0 = E Y . (ii) The submodel residual mean square MS0e is the unbiased estimator of the residual variance σ 2 . 0 (iii) Statistics Yb and U 0 are conditionally, given Z, uncorrelated. 0 (iv) A random vector D = Yb − Yb = U 0 − U satisfies 2 D = SS0e − SSe . (v) If additionally, a normal linear model is assumed, i.e., if Y | Z ∼ Nn X0 β 0 , σ 2 In then the 0 statistics Yb and U 0 are conditionally, given Z, independent and SS0e − SSe SS0e − SSe νe0 − νe r − r0 F0 = = ∼ Fr−r0 , n−r = Fνe0 −νe , νe . SSe SSe n−r νe (5.4) 5.1. SUBMODEL 57 Proof. Proof/calculations were available on the blackboard in K1. k 5.1.3 Series of submodels When looking for a suitable model to express E Y , often a series of submodels is considered. Let us now assume a series of models Model M0 : Y | Z ∼ X0 β 0 , σ 2 In , Model M1 : Y | Z ∼ X1 β 1 , σ 2 In , Model M : Y | Z ∼ Xβ, σ 2 In , where, analogously to (5.1), an n × k1 matrix X1 is given as > X 11 X 11 = t1 (Z 1 ), . .. . X1 = . . , > 1 X n = t1 (Z n ), X 1n for some transformation t1 : Rp −→ Rk1 of the original covariates Z 1 , . . . , Z n , which we briefly write as X1 = t1 (Z). Analogously to (5.2), we will assume that for some 0 < r1 ≤ k1 < n, rank(X1 ) = r1 . Finally, we will assume that the three considered models are mutually submodels. That is, we will assume that M X0 ⊂ M X1 ⊂ M X with r0 < r1 < r, which we denote as M0 ⊂ M1 ⊂ M. Notation. Quantities derived while assuming a particular model will be denoted by the corresponding superscript (or by no superscript in case of the model M). That is: 0 • Yb , U 0 , SS0e , νe0 , MS0e : quantities based on the (sub)model M0 : Y | Z ∼ X0 β 0 , σ 2 In ; 1 • Yb , U 1 , SS1e , νe1 , MS1e : quantities based on the (sub)model M1 : Y | Z ∼ X1 β 1 , σ 2 In ; • Yb , U , SSe , νe , MSe : quantities based on the model M: Y | Z ∼ Xβ, σ 2 In . Theorem 5.2 On submodels. Consider three normal linear models M : Y | Z ∼ Nn Xβ, σ 2 In , M1 : Y | Z ∼ Nn X1 β 1 , σ 2 In , 5.1. SUBMODEL 58 0 M0 : Y | Z∼ Nn X0 β , σ 2 In such that M0 ⊂ M1 ⊂ M. Let the (smallest) submodel M0 hold, i.e., let E Y ∈ M X0 . Then F0,1 SS0e − SS1e SS0e − SS1e νe0 − νe1 r1 − r0 = = ∼ Fr1 −r0 , n−r = Fνe0 −νe1 , νe . SSe SSe n−r νe Proof. Proof/calculations were available on the blackboard in K1. (5.5) k Note. Both F-statistics (5.4) and (5.5) contain • In the numerator: a difference in the residual sums of squares of the two models where one of them is a submodel of the other divided by the difference of the residual degrees of freedom of those two models. • In the denominator: a residual sum of squares of the model which is larger or equal to any of the two models whose quantities appear in the numerator, divided by the corresponding degrees of freedom. • To obtain an F-distribution of the F-statistics (5.4) or (5.5), the smallest model whose quantities appear in that F-statistic must hold which implies that any other larger model holds as well. Notation (Differences when dealing with a submodel). Let MA and MB are two models distinguished by symbols “A” and “B” such that MA ⊂ MB . Let A B B Yb and Yb , U A and U B , SSA e and SSe denote the fitted values, the vectors of residuals and the residual sums of squares based on models MA and MB , respectively. The following notation will be used if it becomes necessary to indicate which are the two model related to the vector D or to the difference in the sums of squares: B A D MB MA = D B A := Yb − Yb = U A − U B . B SS MB MA = SS B A := SSA e − SSe . Notes. • Both F-statistics (5.4) and (5.5) contain certain SS B A in their numerators. • Point (iv) of Theorem 5.1 gives 2 SS B A = D B A . 5.1.4 Statistical test to compare nested models End of Theorems 5.1 and 5.2 provide a way to compare two nested models by the mean of a statistical Lecture #8 test. (29/10/2015) Start of Lecture #10 (05/11/2015) 5.1. SUBMODEL 59 F-test on a submodel based on Theorem 5.1 Consider two normal linear models: Model M0 : Y | Z ∼ Nn X0 β 0 , σ 2 In , Model M: Y | Z ∼ Nn Xβ, σ 2 In , where M0 ⊂ M, and a set of statistical hypotheses: H0 : E Y ∈ M X0 H1 : E Y ∈ M X \ M X0 , that aim in answering the questions: • Is model M significantly better than model M0 ? • Does the (larger) regression space M X provide a significantly better expression for E Y over the (smaller) regression space M X0 ? The F-statistic (5.4) from Theorem 5.1 now provides a way to test the above hypotheses as follows: SS M M0 SS0e − SSe r − r0 r − r0 = . Test statistic: F0 = SSe SSe n−r n−r Reject H0 if F0 ≥ Fr−r0 ,n−r (1 − α). P-value when F0 = f0 : p = 1 − CDFF , r−r0 ,n−r f0 . F-test on a submodel based on Theorem 5.2 Consider three normal linear models: Model M0 : Y | Z ∼ Nn X0 β 0 , σ 2 In , Model M1 : Y | Z ∼ Nn X1 β 1 , σ 2 In , Model M: Y | Z ∼ Nn Xβ, σ 2 In , where M0 ⊂ M1 ⊂ M, and a set of statistical hypotheses: H0 : E Y ∈ M X0 H1 : E Y ∈ M X1 \ M X0 , that aim in answering the questions: • Is model M1 significantly better than model M0 ? • Does the (larger) regression space M X1 provide a significantly better expression for E Y over the (smaller) regression space M X0 ? The F-statistic (5.5) from Theorem 5.2 now provides a way to test the above hypotheses as follows: SS M1 M0 SS0e − SS1e r1 − r0 r1 − r0 = . Test statistic: F0,1 = SSe SSe n−r n−r Reject H0 if F0,1 ≥ Fr1 −r0 ,n−r (1 − α). P-value when F0,1 = f0,1 : p = 1 − CDFF , r1 −r0 ,n−r f0,1 . 5.2. OMITTING SOME COVARIATES 5.2 60 Omitting some covariates The most common couple (model – submodel) is Model M: Submodel M0 : Y | Z ∼ Xβ, σ 2 In , Y | Z ∼ X0 β 0 , σ 2 In , where the submodel matrix X0 is obtained by omitting selected columns from the model matrix X. In other words, some covariates are omitted from the original covariate vectors X 1 , . . . , X n to get the submodel and the matrix X0 . In the following, without the loss of generality, let X = X0 , X1 , 0 < rank X0 = r0 < r = rank X < n. The corresponding submodel F-test then evaluates whether, given the knowledge of the covariates included in the submodel matrix X0 , the covariates included in the matrix X1 has an impact on the response expectation. Theorem 5.3 Effect of omitting some covariates. Consider a couple (model – submodel), where the submodel is obtained by omitting some covariates from the model. Then (i) D 6= 0n and SS0e − SSe > 0. (ii) If M X1 ⊥ M X0 then > D = X1 X1 X1 − > 1 X1 Y =: Yb , which are the fitted values from a linear model Y | Z ∼ X1 β 1 , σ 2 In . Proof. Proof/calculations were available on the blackboard in K1. k Note. If we take the residual sum of squares as a measure of a quality of the model, point (i) of Theorem 5.3 says that the model is always getting worse if some covariates are removed. Nevertheless, in practice, it is always a question whether this worsening is statistically significant (the submodel F-test answers this) or practically important (additional reasoning is needed). 5.3. LINEAR CONSTRAINTS 5.3 61 Linear constraints Suppose that a linear model Y | Z ∼ Xβ, σ2 In , rank Xn×k = r is given and it is our aim to verify whether the response expectation E Y lies in a constrained regression space M X; Lβ = θ 0 := v : v = Xβ, β ∈ Rk , Lβ = θ 0 , (5.6) where Lm×k is a given real matrix and θ 0 ∈ Rm is a given vector. In other words, verification of whether the response expectation lies in the space M X; Lβ = θ 0 corresponds to verification of whether the regression coefficients satisfy a linear constraint Lβ = θ 0 . Lemma 5.4 Regression space given by linear constraints. Consider a linear model Y | Z ∼ Xβ, σ 2 In , rank Xn×k = r ≤ k < n. Let Lm×k be a real matrix with m ≤ r rows such that (i) rank L = m (i.e., L is a matrix with linearly independent rows); (ii) θ = Lβ is estimable parameter of the considered linear model. The space M X; Lβ = 0m is then a vector subspace of dimension r − m of the regression space M X . Proof. Proof/calculations were available on the blackboard in K1. k Notes. 0 • The space M X; Lβ = θ is a vector space only if θ 0 = 0m since otherwise, 0n ∈ / 0 M X; Lβ = θ . Nevertheless, for the purpose of the statistical analysis, it is possible (and in practice also necessary) to work also with θ 0 6= 0m . • With m = r, M X; Lβ = 0m = 0n . Definition 5.2 Submodel given by linear constraints. We say that the model given by linear constraints3 Lβ = θ 0 of model M: M0 is a submodel 2 Y | Z ∼ Xβ, σ In , rank Xn×k = r, if matrix L satisfies conditions of Lemma 5.4, m < r and the response expectation E Y under the model M0 is assumed to lie in a space M X; Lβ = θ 0 . Notation. A submodel given by linear constraints will be denoted as M0 : Y | Z ∼ Xβ, σ 2 In , Lβ = θ 0 . 3 podmodel zadaný lineárními omezeními 5.3. LINEAR CONSTRAINTS 62 Since with θ 0 6= 0m , the space M X; Lβ = 0m is not a vector space, we in general cannot talk about projections in a sense of linear algebra when deriving the fitted values, the residuals and other quantities related to the submodel given by linear constraints. Hence we introduce the following definition. Definition 5.3 Fitted values, residuals, residual sum of squares, rank of the model and residual degrees of freedom in a submodel given by linear constraints. 2 Let b0 ∈ Rk minimize SS(β) = Y − Xβ over β ∈ Rk subject to Lβ = θ 0 . For the submodel M0 : Y | Z ∼ Xβ, σ 2 In , Lβ = θ 0 , the following quantities are defined as follows: 0 Fitted values: Yb := Xb0 . 0 Residuals: U 0 := Y − Yb . 2 Residual sum of squares: SS0e := U 0 . Rank of the model: r0 = r − m. Residual degrees of freedom: νe0 := n − r0 . Note. The fitted values could also be defined as 0 Yb = argmin e ∈M X; Lβ=θ 0 Y Y − Ye 2 . That is, the fitted values are (still) the closest point to Y in the constrained regression space M X; Lβ = θ 0 . Theorem 5.5 On a submodel given by linear constraints. Let M0 : Y | Z ∼ Xβ, σ 2 In , Lβ = θ 0 be a submodel given by linear constraints of a model M : Y | Z ∼ Xβ, σ 2 In . Then 0 (i) The fitted values Yb and consequently also the residuals U 0 and the residual sum of squares SS0e are unique. 2 (ii) b0 minimizes SS(β) = Y − Xβ subject to Lβ = θ 0 if and only if b0 = b − X> X where b = X> X − − n − o−1 L> L X> X L> Lb − θ 0 , X> Y is (any) solution to a system of normal equations X> Xb = X> Y . 0 (iii) The fitted values Yb can be expressed as 0 Yb = Yb − X X> X − n − o−1 L> L X> X L> Lb − θ 0 . 0 (iv) The vector D = Yb − Yb satisfies n o−1 2 D = SS0e − SSe = (Lb − θ 0 )> L X> X − L> (Lb − θ 0 ). (5.7) 5.3. LINEAR CONSTRAINTS 63 Proof. First mention that under our assumptions, the matrix L X> X − L> is (i) invertible; (ii) does not depend on a choice of the pseudoinverse X> X − . This follows from Theorem 2.9 (Gauss–Markov for estimable vector parameter). 2 0 Second, try to look for Yb = Xb0 such that b0 minimizes SS(β) = Y − Xβ over β ∈ Rk subject to Lβ = θ 0 by a method of Lagrange multipliers. Let 2 ϕ(β, λ) = Y − Xβ + 2λ> Lβ − θ 0 > = Y − Xβ Y − Xβ + 2λ> Lβ − θ 0 , where a factor of 2 in the second part of expression of the Lagrange function ϕ is only included to simplify subsequent expressions. The first derivatives of ϕ are as follows: ∂ϕ (β, λ) = −2 X> Y − Xβ + 2 L> λ, ∂β ∂ϕ (β, λ) = 2 Lβ − θ 0 . ∂λ Realize now that ∂ϕ (β, λ) = 0k if and only if ∂β X> Xβ = X> Y − L> λ. (5.8) Note that the linear system (5.8) is consistent for any λ ∈ Rm and any Y ∈ Rn . This follows from > > the fact that due to estimability of a parameter Lβ, we have M L ⊂ M X (Theorem 2.7). Hence the right-hand-side of the system (5.8) lies in M X> , for any λ ∈ Rm and any Y ∈ Rn . The left-hand-side of the system (5.8) lies in M X> X , for any β ∈ Rk . We already know that > > M X = M X X (Lemma 2.6) which proves that there always exist a solution to the linear system (5.8). Let b0 (λ) be any solution to X> Xβ = X> Y − L> λ. That is, − − b0 (λ) = X> X X> Y − X> X L> λ − = b − X> X L> λ, which depends on a choice of X> X Further, − . ∂ϕ (β, λ) = 0m if and only if ∂λ Lb0 (λ) = θ 0 − Lb − L X> X L> λ = θ 0 − L X> X L> λ = Lb − θ 0 . | {z } invertible as we already know 5.3. LINEAR CONSTRAINTS That is, 64 n o−1 − λ = L X> X L> Lb − θ 0 . Finally, 0 > b = b− X X − n − > o−1 > (Lb − θ 0 ), L L X X L 0 Yb = Xb0 = Yb − X X> X > − n − o−1 (Lb − θ 0 ). L> L X> X L> Realize again that M L> ⊂ M X> . That is, there exist a matrix A such that L> = X> A> , L = AX. 0 Under our assumptions, matrix A is even unique. The vector Yb can now be written as n − o−1 − 0 A> L X> X L> Lb −θ 0 . Yb = |{z} Yb − X X> X X> |{z} |{z} {z } | {z } unique unique | unique unique unique (5.9) 0 To show point (iv), use (5.9) in expressing the vector D = Yb − Yb : n − o−1 − (Lb − θ 0 ). D = X X> X X> A> L X> X L> That is, o−1 n 2 − − D = (Lb − θ 0 )> L X> X − L> A X X> X X> X X> X X> A> | {z } X by the five matrices rule n − o−1 L X> X L> (Lb − θ 0 ) = (Lb − θ 0 ) > n n − o−1 − o−1 − (Lb − θ 0 ) L X> X L> AX X> X X> A> L X> X L> = (Lb − θ 0 ) > n n − o−1 − − o−1 (Lb − θ 0 ) L X> X L> L X> X L> L X> X L> = (Lb − θ 0 ) > n − o−1 L X> X L> (Lb − θ 0 ). 2 It remains to be shown that D = SS0e − SSe . We have n o−1 0 2 0 2 b + X X> X − L> L X> X − L> SS0e = Y − Yb = Y − Y (Lb − θ ) | {z } {z } ⊥ | U ∈M X D∈M X 2 2 2 = U + D = SSe + D . k 5.3. LINEAR CONSTRAINTS 5.3.1 65 F-statistic to verify a set of linear constraints Let us take the expression (5.7) for the difference between the residual sums of squares of the model and the submodel given by linear constraints and derive the submodel F-statistic (5.4): > (Lb − θ 0 ) SS0e − SSe r − r0 F0 = = SSe n−r n − o−1 L X> X L> (Lb − θ 0 ) m SSe n−r = n − o−1 1 > (Lb − θ 0 ) MSe L X> X L> (Lb − θ 0 ) m = n − o−1 > 1 b b − θ 0 ), (θ − θ 0 ) MSe L X> X L> (θ m (5.10) b = Lb is the LSE of the estimable vector parameter θ = Lβ in the linear model Y X ∼ where θ Xβ, σ 2 In without constraints. Note now that (5.10) is exactly equal to the Wald-type statistic 0 Q0 (see page 41) that we used in Section 3.2.2 to test the null hypothesis2 H0 : θ = θ on an estimable vector parameter θ in a normal linear model Y Z ∼ Nn Xβ, σ In . If normality can be assumed, point (x) of Theorem 3.1 then provided that under the null hypothesis H0 : θ = θ 0 , that is, under the validity of the submodel given by linear constraints Lβ = θ 0 , the statistic F0 follows the usual F-distribution Fm,n−r . This shows that the Wald-type test on the estimable vector parameter in a normal linear model based on Theorem 3.1 is equivalent to the submodel F-test based on Theorem 5.1. 5.3.2 t-statistic to verify a linear constraint > k Consider L = l> , l ∈ R , l 6= 02k such that 0θ = l β is an estimable parameter of the normal linear model Y Z ∼ Nn Xβ, σ In . Take θ ∈ R and consider the submodel given by m = 1 linear constraint l> β = θ0 . Let θb = l> b, where b is any solution to the normal equations in the model without constraints. The statistic (5.10) then takes the form !2 b − θ0 − o−1 n 1 b θ 0 > > 0 = T02 , MSe l X X l F0 = θ−θ θb − θ = q − m MSe l> X> X l where θb − θ0 T0 = q − MSe l> X> X l is the Wald-type test statistic introduced in Section 3.2.2 (on page 40) to test the null hypothesis H0 : θ = θ0 in a normal linear model Y Z ∼ Nn Xβ, σ 2 In . Point (viii) of Theorem 3.1 provided that under the null hypothesis H0 : θ = θ0 , the statistic T0 follows the Student t-distribution tn−r which is indeed in agreement with the fact that T02 = F0 follows the F-distribution F1,n−r . 5.4. COEFFICIENT OF DETERMINATION 5.4 5.4.1 66 Coefficient of determination Intercept only model Notation (Response sample mean). The sample mean over the response vector Y = Y1 , . . . , Yn > will be denoted as Y . That is, n 1X 1 Y = Yi = Y > 1n . n n i=1 Definition 5.4 Regression and total sums of squares in a linear model. Consider a linear model Y X ∼ Xβ, σ 2 In , rank(Xn×k ) = r ≤ k. The following expressions define the following quantities: (i) Regression sum of squares4 and corresponding degrees of freedom: n 2 X 2 SSR = Yb − Y 1n = Ybi − Y , νR = r − 1, i=1 (ii) Total sum of squares5 and corresponding degrees of freedom: n 2 X 2 Yi − Y , SST = Y − Y 1n = νT = n − 1. i=1 Lemma 5.6 Model with intercept only. Let Y ∼ 1n γ, ζ 2 In . Then (i) Yb = Y 1n = Y , . . . , Y > . (ii) SSe = SST . Proof. This is a full-rank model with X = 1n . Further, X> X Hence γ b= 4 1 n Pn i=1 Yi regresní součet čtverců 5 −1 = 1> n 1n −1 = 1 , n X > Y = 1> nY = = Y and Yb = Xb γ = 1n Y = Y 1n . celkový součet čtverců n X Yi . i=1 k 5.4. COEFFICIENT OF DETERMINATION 5.4.2 67 Models with intercept Lemma 5.7 Identity in a linear model with intercept. Let Y X ∼ Xβ, σ 2 In where 1n ∈ M X . Then 1> nY = n X Yi = n X i=1 b Ybi = 1> nY . i=1 Proof. • Follows directly from the normal equations if 1n is one of the columns of X matrix. • General proof: b b> 1> n Y = Y 1n = HY > 1n = Y > H1n = Y > 1n , since H1n = 1n due to the fact that 1n ∈ M X . k Theorem 5.8 Breakdown of the total sum of squares in a linear model with intercept. Let Y X ∼ Xβ, σ 2 In where 1n ∈ M X . Then SST n X i=1 Yi − Y = 2 = SSe n X Yi − Ybi + 2 + i=1 SSR n X Ybi − Y 2 . i=1 Proof. The identity SST = SSe + SSR follows trivially if r = rank X = 1 since then M X = M 1n and hence (by Lemma 5.6) Yb = Y 1n . Then SST = SSe , SSR = 0. 0, σ2I In the following, let r = rank X > 1. Then, model Y | X ∼ 1 β is a submodel of the n n model Y X ∼ Xβ, σ 2 In and by Lemma 5.6, SST = SS0e . Further, from definition of SSR , 2 0 it equals to SSR = D , where D = Yb − Yb . By point (iv) of Theorem 5.1 (on a submodel), 2 D = SS0e − SSe . In other words, SSR = SST − SSe . k 5.4. COEFFICIENT OF DETERMINATION 68 The identity SST = SSe + SSR can also be shown directly while using a little algebra. We have SST = n X Yi − Y 2 = i=1 = n X n X Yi − Ybi + Ybi − Y 2 i=1 Yi − Ybi 2 + i=1 n X Ybi − Y 2 +2 = SSe + SSR + 2 Yi − Ybi Ybi − Y i=1 i=1 n nX n X Yi Ybi − Y i=1 n X Yi + Y i=1 n X i=1 Ybi − n X Ybi2 i=1 {z 0 | o } = SSe + SSR P P since ni=1 Yi = ni=1 Ybi and additionally n X Yi Ybi = Y > Yb = Y > HY , i=1 5.4.3 n X > Ybi2 = Yb Yb = Y > HHY = Y > HY . i=1 k Evaluation of a prediction quality of the model End of Lecture #10 (05/11/2015) Start of One of the usual aims of regression modelling is so called prediction in which case the model based Lecture #12 mean is used as the predicted response value. In such situations, it is assumed that data Yi , X i , (12/11/2015) i = 1, . . . , n, are a random sample from some joint distribution of a generic random vector Y, X and theconditional distribution Y | X can be described by a linear model Y | X ∼ Xβ, σ 2 In , rank X = r, for the data. That is, the mean and the variance of the conditional distribution Y | X are given as E Y X = X > β and var Y X = σ 2 , respectively. In the following, let EY and varY , respectively, denote expectation and variance, respectively, with 2 respect to the marginal distribution of Y . The only intercept model Y ∼ 1n γ, ζ In for the data then corresponds to the marginal distribution of the response Y with EY Y = γ and varY Y = ζ 2 . Suppose now that notyet observed random vector Ynew , X new is also distributed as a generic random vector Y, X and assume that all parameters of considered models are known. The aim is to provide the prediction Ybnew of the value of Ynew . To this end, the quality of the prediction is, most classically, evaluated by the mean squared error of prediction6 (MSEP) defined as 2 MSEP Ybnew = EY, X Ybnew − Ynew , (5.11) where symbol EY, X denotes expectation with respect to the joint distribution of the random vector Y, X . To predict the value of Ynew , we have basically two options depending on whether a value of X new (covariates for the new observation) is or is not available to construct the prediction. 6 střední čtvercová chyba predikce 5.4. COEFFICIENT OF DETERMINATION 69 (i) If the value of X new is not available, prediction can only be based on the marginal (intercept M of only) model where the model based mean equals to γ, which is also the prediction Ybnew Ynew . That is, M Ybnew = γ, and we get 2 2 M MSEP Ybnew = EY, X γ − Ynew = EY γ − Ynew = varY Ynew = ζ 2 . (ii) If the value of X new is available, the conditional (regression) model can be used leading to the prediction of Ynew of the form C Ybnew = X> new β. Let EX denote the expectation with respect to the marginal distribution of the covariates X. The MSEP then equals h i 2 2 C > > b MSEP Ynew = EY, X X new β − Ynew = EX E X new β − Ynew X new n o = EX var Ynew X new = EX σ 2 = σ 2 . Finally, we get C MSEP Ybnew σ2 = 2. M ζ MSEP Ybnew C based on the regression That is, the ratio σ 2 /ζ 2 quantifies advantage of using the prediction Ybnew M which is equal to model and the covariate values X new compared to using the prediction Ybnew the marginal response expectation. 5.4.4 Coefficient of determination To estimate the ratio σ 2 /ζ 2 between the conditional and the marginal response variances, we can straightforwardly consider the following. First, n 2 1 1 X SST = Yi − Y νT n−1 i=1 is a standard sample variance based on the random sample Y1 , . . . , Yn . That is, it is an unbiased estimator of the marginal variance ζ 2 . Note that it is also the residual mean square from the only 2 intercept model 1n γ, ζ In . Further, n 2 1 1 X SSe = Yi − Ybi , νe n−r i=1 which is the residual mean square from the considered linear model Xβ, σ 2 In , is an unbiased estimator of the conditional variance σ 2 . That is, a suitable estimator of the ratio σ 2 /ζ 2 is 1 n−r SSe 1 n−1 SST = n − 1 SSe . n − r SST (5.12) 5.4. COEFFICIENT OF DETERMINATION 70 Alternatively, if Y ∼ N (γ, ζ 2 ), that is, if Y1 , . . . , Yn is a random sample from N (γ, ζ 2 ), it can be (it was) easily derived that a quantity n 2 1 1 X SST = Yi − Y n n i=1 is the maximum-likelihood estimator7 (MLE) of the marginal variance ζ 2 . Analogously, if Y | X ∼ > N X β, σ 2 , it can be derived (see the exercise class) that a quantity n 2 1 1 X SSe = Yi − Yb n n i=1 is the MLE of the conditional variance σ 2 . Alternative estimator of the ratio σ 2 /ζ 2 is then 1 n SSe 1 n SST = SSe . SST (5.13) Remember that in the model Y | X ∼ Xβ, σ 2 In with intercept (1n ∈ M X ), we have, n X Yi − Y 2 Yi − Ybi 2 i=1 i=1 | = n X {z SST } | + n X Ybi − Y 2 , i=1 {z SSe } | {z SSR } where the three sums of squares represent different sources of the response variability: SST (total sum of squares): original (marginal) variability of the response, SSe (residual sum of squares): variability not explained by the regression model, (residual variability, conditional variability) SSR (regression sum of squares): variability explained by the regression model. Expressions (5.12) and (5.13) then motivate the following definition. Definition 5.5 Coefficients of determination. Consider a linear model Y X ∼ Xβ, σ 2 In , rank(X) = r where 1n ∈ M X . A value R2 = 1 − SSe SST is called the coefficient of determination8 of the linear model. A value 2 Radj =1− n − 1 SSe n − r SST is called the adjusted coefficient of determination9 of the linear model. 7 maximálně věrohodný odhad 8 koeficient determinace 9 upravený koeficient determinace 5.4. COEFFICIENT OF DETERMINATION 71 Notes. • By Theorem 5.8, SST = SSe + SSR and at the same time SST ≥ 0. Hence 0 ≤ R2 ≤ 1, and R2 can also be expressed as R2 = 2 ≤ 1, 0 ≤ Radj SSR . SST 2 are often reported as R2 · 100% and R2 · 100% which can be interpreted • Both R2 and Radj adj as a percentage of the response variability explained by the regression model. 2 quantify a relative improvement of the quality of prediction if the regression • Both R2 and Radj model and the conditional distribution of response given the covariates is used compared to the prediction based on the marginal distribution of the response. • Both coefficients of determination only quantifies the predictive ability of the model. They do not say much about the quality of the model with respect to the possibility 2to capture correctly ) might be useful the conditional mean E Y X . Even a model with a low value of R2 (Radj with respect to modelling the conditional mean E Y X . The model is perhaps only useless for prediction purposes. 5.4.5 Overall F-test Lemma 5.9 Overall F-test. Assume a normal linear model Y | X ∼ Nn Xβ, σ 2 In , rank Xn×k = r > 1 where 1n ∈ M X . Let R2 be its coefficient of determination. The submodel F-statistic to compare model M : Y | X ∼ Nn Xβ, σ 2 In and the only intercept model M0 : Y | X ∼ Nn 1n γ, σ 2 In takes the form F0 = R2 n−r · . 2 1−R r−1 (5.14) Proof. • R2 = 1 − SSe SST and according to Lemma 5.6: SST = SS0e . • Hence R2 = 1 − SSe SS0e − SSe = , SS0e SS0e 1 − R2 = SSe . SS0e • At the same time F0 = SS0e −SSe r−1 SSe n−r n − r SS0e − SSe n−r = = r−1 SSe r−1 SS0e −SSe SS0e SSe SS0e = n − r R2 . r − 1 1 − R2 k 5.4. COEFFICIENT OF DETERMINATION 72 Note. The F-test with the test statistic (5.14) is sometimes (especially in some software packages) referred to as an overall goodness-of-fit test. Nevertheless be cautious when interpreting the results of such test. It says practically nothing about the quality of the model and the “goodness-of-fit”! Chapter 6 General Linear Model We still assume that data are represented by a set of n random vectors Yi , X i , X i = Xi,0 , . . . , Xi,k−1 , i = 1, . . . , n, and use symbols Y for a vector Y1 , . . . , Yn and X for an n × k matrix with rows given by vectors X 1 , . . . , X n . In this chapter, we mildly extend a linear model by allowing for a covariance matrix having different form than σ 2 In assumed by now. Definition 6.1 General linear model. The data Yi , X i , i = 1, . . . , n, satisfy a general linear model1 if E Y X = Xβ, var Y X = σ 2 W−1 , where β ∈ Rk and 0 < σ 2 < ∞ are unknown parameters and W is a known positive definite matrix. Notes. • The fact that data follow a general linear model is denoted as Y X ∼ Xβ, σ 2 W−1 . • General linear model should not be confused with a generalized linear model 2 which is something different (see Advanced Regression Models (NMST432) course). In the literature, abbreviation “GLM” is used for (unfortunately) both general and generalized linear model. It must be clear from context which of the two is meant. Example 6.1 (Regression based on sample means). Suppose that data are represented by random vectors Ye1,1 , . . . , Ye1,w1 , X 1 , ..., Yen,1 , . . . , Yen,wn , X n such that for each i = 1, . . . , n, the random variables Yei,1 , . . . , Yei,wi are uncorrelated with a common conditional (given X i ) variance σ 2 . 1 obecný lineární model 2 zobecněný lineární model 73 74 Suppose that we are only able to observe the sample means of the “Ye ” variables leading to the response variables Y1 , . . . , Yn , where Y1 = w1 1 X Ye1,j , w1 ..., Yn = j=1 wn 1 X Yen,j . wn j=1 The covariance matrix (conditional given X) of a random vector Y = Y1 , . . . , Yn is then 1 w1 . . var Y := var Y X = σ 2 . 0 | ... .. . ... {z W−1 0 .. . . 1 wn } Theorem 6.1 Generalized least squares. Assume a general linear model Y X ∼ Xβ, σ 2 W−1 , where rank Xn×k = r ≤ k < n. The following then holds: (i) A vector Yb G := X X> WX − X> WY is the best linear unbiased estimator (BLUE) of a vector parameter µ := E Y = Xβ, and − var Yb G = σ 2 X X> WX X> . − Both Yb G and var Yb G do not depend on a choice of the pseudoinverse X> WX . If further Y X ∼ Nn Xβ, σ 2 W−1 then − Yb G X ∼ Nn Xβ, σ 2 X X> WX X> . (ii) Let l ∈ Rk , l 6= 0k , be such that θ = l> β is an estimable parameter of the model and let bG := X> WX − X> WY . Then θbG = l> bG does not depend on a choice of the pseudoinverse used to calculate bG and θbG is the best linear unbiased estimator (BLUE) of θ with − var θbG = σ 2 l> X> WX l, which also does not depend on a choice of the pseudoinverse. If further Y X ∼ Nn Xβ, σ 2 W−1 then − θbG X ∼ N θ, σ 2 l> X> WX l . 75 (iii) If further r = k (full-rank general linear model), then b := X> WX −1 X> WY β G is the best linear unbiased estimator (BLUE) of β with b = σ 2 X> WX −1 . var β G If additionally Y X ∼ Nn Xβ, σ 2 W−1 then b X ∼ Nk β, σ 2 X> WX −1 . β G (iv) The statistic MSe,G := where SSe,G , n−r 1 > 2 SSe,G := W 2 Y − Yb G = Y − Yb G W Y − Yb G , is the unbiased estimator of the residual variance σ 2 . If additionally Y X ∼ Nn Xβ, σ 2 W−1 then SSe,G ∼ χ2n−r , σ2 and the statistics SSe,G and Yb G are conditionally, given X, independent. Proof. Proof/calculations were available on the blackboard in K1. k Note. Mention also that as consequence of the above theorem, all classical tests, confidence intervals etc. work in the same way as in the OLS case. Terminology (Generalized fitted values, residual sum of squares, mean square, least square estimator). − • The statistic Yb G = X X> WX X> WY is called the vector of the generalized fitted values.3 1 > 2 • The statistic SSe,G = W 2 Y − Yb G = Y − Yb G W Y − Yb G is called the generalized residual sum of squares.4 SSe,G is called the generalized mean square.5 n−r b = X> WX −1 X> WY in a full-rank general linear model is called the gener• The statistic β G alized least squares (GLS) estimator 6 of the regression coefficients. • The statistic MSe,G = 3 zobecněné vyrovnané hodnoty zobecněných nejmenších čtverců 4 zobecněný reziduální součet čtverců 5 zobecněný střední čtverec 6 odhad metodou 76 Note. The most common use of the generalized least squares is the situation described in Example 6.1, where 1 w1 . . W−1 = . 0 ... .. . ... 0 .. . . 1 wn We then get X> WY = n X wi Yi X i , i=1 SSe,G = X> WX = n X wi X i X > i , i=1 n X 2 wi Yi − YbG,i . i=1 The method of the generalized least squares is then usually referred to as the method of the weighted least squares (WLS).7 7 vážené nejmenší čtverce End of Lecture #12 (12/11/2015) Chapter 7 Parameterizations of Covariates 7.1 Linearization of the dependence of the response on the covariates Start of As it is usual in this lecture, we represent data by n random vectors Yi , Z i , Z i = Zi,1 , . . . , Lecture #5 (15/10/2015) Zi,p ∈ Z ⊆ Rp , i = 1, . . . , n. The principal problem we consider is to find a suitable model to express the (conditional) response expectation E Y := E Y Z , where Z is a matrix with vectors Z 1 , . . ., Zn in its rows. To this end, we consider a linear model, where E Y can be expressed as E Y = Xβ for some β ∈ Rk , where X> X 1 = X1,0 , . . . , X1,k−1 = t(Z 1 ), 1 . .. . X= . . , > Xn X n = Xn,0 , . . . , Xn,k−1 = t(Z n ), and t : Z −→ X ⊆ Rk , t(z) = t0 (z), . . . , tk−1 (z) = x0 , . . . , xk−1 = x, is a suitable transformation of the original covariates that linearize the relationship between the response expectation and the covariates. The corresponding regression function is then m(z) = β0 t0 (z) + · · · + βk−1 tk−1 (z), z ∈ Z. (7.1) One of the main problems of a regression analysis is to find a reasonable form of the transformation function t to obtain a model that is perhaps but at least useful to capture sufficiently the wrong form of E Y and in general to express E Y Z = z , z ∈ Z, for a generic response Y being generated, given the covariate value Z = z, by the same probabilistic mechanism as the original data. 77 7.2. PARAMETERIZATION OF A SINGLE COVARIATE 7.2 78 Parameterization of a single covariate In this and two following sections, we first limit ourselves to the situation of a single covariate, i.e., p = 1, Z ⊆ R, and show some classical choices of the transformations that are used in practical analyses when attempting to find a useful linear model. 7.2.1 Parameterization Our aim is to propose transformations t : Z −→ Rk , t(z) = t0 (z), . . . , tk−1 (z) such that a regression function (7.1) can possibly provide a useful model for the response expectation E Y Z = z . Furthermore, in most cases, we limit ourselves to transformations that lead to a linear model with intercept. In such cases, the regression function will be m(z) = β0 + β1 s1 (z) + · · · + βk−1 sk−1 (z), z ∈ Z, (7.2) where the non-intercept part of the transformation t will be denoted as s. That is, for z ∈ Z, j = 1, . . . , k − 1, s(z) = s1 (z), . . . , sk−1 (z) = t1 (z), . . . , tk−1 (z) . sj (z) = tj (z), s : Z −→ Rk−1 , Definition 7.1 Parameterization of a covariate. Let Z1 , . . . , Zn be values of a given univariate covariate Z ∈ Z ⊆ R. By a parameterization of this covariate we mean (i) a function s : Z −→ Rk−1 , s(z) = s1 (z), . . . , sk−1 (z) , z ∈ Z, where all s1 , . . . , sk−1 are non-constant functions on Z, and (ii) an n × (k − 1) matrix S, where s> (Z1 ) s1 (Z1 ) . . . sk−1 (Z1 ) . . .. .. . .. = .. S= . . s> (Zn ) s1 (Zn ) . . . sk−1 (Zn ) Terminology (Reparameterizing matrix, regressors). Matrix S from Definition 7.1 is called reparameterizing matrix 1 of a covariate. Its columns, i.e., vectors sk−1 (Z1 ) s1 (Z1 ) . .. .. , . . . , X k−1 = X1 = . s1 (Zn ) sk−1 (Zn ) are called regressors.2 1 reparametrizační matice 2 regresory 7.2. PARAMETERIZATION OF A SINGLE COVARIATE 79 Notes. • A model matrix X of the model with 1 X> 1 .. .. X = 1n , S = . = . > 1 Xn X i = s(Zi ), the regression function (7.2) is 1 X1,1 . . . X1,k−1 . .. .. .. 1 k−1 , .. = 1 , X , . . . , X n . . . 1 Xn,1 . . . Xn,k−1 Xi,j = sj (Zi ), i = 1, . . . , n, j = 1, . . . , k − 1. • Definition 7.1 is such that an intercept vector 1n (or a vector c 1n , c ∈ R) is (with a positive probability) not included in the reparameterizing matrix S. Nevertheless, it will be useful in some situations to consider such parameterizations that (almost surely) include an intercept term in the corresponding regression space. That is, for some parameterizations (see the regression splines in Section 7.3.4), we will have 1n ∈ M S . 7.2.2 Covariate types The covariate space Z and the corresponding univariate covariates Z1 , . . . , Zn are usually of one of the two types and different parameterizations are useful depending on the covariate type which are the following. Numeric covariates Numeric 3 covariates are such covariates where a ratio of the two covariate values makes sense and a unity increase of the covariate value has an unambiguous meaning. The numeric covariate is then usually of one of the two following subtypes: (i) continuous, in which case Z is mostly an interval in R. Such covariates have usually a physical interpretation and some units whose choice must be taken into account when interpreting the results of the statistical analysis. The continuous numeric covariates are mostly (but not necessarily) represented by continuous random variables. (ii) discrete, in which case Z is infinite countable or finite (but “large”) subset of R. The most common situation of a discrete numeric covariate is a count 4 with Z ⊆ N0 . The numeric discrete covariates are represented by discrete random variables. Categorical covariates Categorical 5 covariates (in the R software referred to as factors), are such covariates where the ratio of the two covariate values does not necessarily make sense and a unity increase of the covariate value does not necessarily have an unambiguous meaning. The sample space Z is a finite (and mostly “small”) set, i.e., Z = ω1 , . . . , ωG , where the values ω1 < · · · < ωG are somehow arbitrarily chosen labels of categories purely used to obtain a mathematical representation of the covariate values. The categorical covariate is always represented by a discrete random variable. Even for categorical covariates, it is useful to distinguish the two subtypes: 3 numerické, příp. kvantitativní 4 počet 5 kategoriální, příp. kvalitativní 7.2. PARAMETERIZATION OF A SINGLE COVARIATE 80 (i) nominal 6 where from a practical point of view, chosen values ω1 , . . . , ωG are completely arbitrary. Consequently, practically interpretable results and conclusions of any sensible statistical analysis should be invariant towards the choice of ω1 , . . . , ωG . The nominal categorical covariate mostly represents a pertinence to some group (a group label), e.g., region of residence. (ii) ordinal 7 where ordering ω1 < · · · < ωG makes sense also from a practical point of view. An example is a school grade. Notes. • From the practical point of view, it is mainly important to distinguish numeric and categorical covariates. • Often, ordinal categorical covariate can be viewed also as a discrete numeric. Whatever in this lecture that will be applied to the discrete numeric covariate can also be applied to the ordinal categorical covariate if it makes sense to interprete, at least into some extent, its unity increase (and not only the ordering of the covariate values). 6 nominální 7 ordinální 7.3. NUMERIC COVARIATE 7.3 81 Numeric covariate It is now assumed that Zi ∈ Z ⊆ R, i = 1, . . . , n, are numeric covariates. Our aim is now to propose their sensible parameterizations. 7.3.1 Simple transformation of the covariate The regression function is m(z) = β0 + β1 s(z), z ∈ Z, (7.3) where s : Z −→ R is a suitable non-constant function. The corresponding reparameterizing matrix is s(Z1 ) . . S= . . s(Zn ) Due to interpretability issues, “simple” functions like: identity, logarithm, exponential, square root, reciprocal, . . . , are considered in place of the transformation s. Evaluation of the effect of the original covariate Advantage of a model with the regression function (7.3) is the fact that a single regression coefficient β1 (the slope in a model with the regression line in x = s(z)) quantifies the effect of the covariate on the response expectation which can then be easily summarized by a single point estimate and a confidence interval. Evaluation of a statistical significance of the effect of the original covariate on the response expectation is achieved by testing the null hypothesis H0 : β1 = 0. A possible test procedure was introduced in Section 3.2. Interpretation of the regression coefficients Disadvantage is the fact that the slope β1 expresses the change of the response expectation that corresponds to a unity change of the transformed covariate X = s(Z), i.e., for z ∈ Z: β1 = E Y X = s(z) + 1 − E Y X = s(z) , which is not always easily interpretable. Moreover, unless that corresponds E Y the transformation s is a linear function, the change in the response expectation to a unity change of the original covariate is a function of that covariate: Z = z + 1 − E Y Z = z = β1 s(z + 1) − s(z) , z ∈ Z. In other words, a model with the regression function (7.3) and a non-linear transformation s expresses the fact that the original covariate has different influence on the response expectation depending on the value of this covariate. Note. It is easily seen that if n > k = 2, the transformation s is strictly monotone and the data contain at least two different values among Z1 , . . . , Zn (which has a probability of one if the covariates Zi are sampled from a continuous distribution), the model matrix X = 1n , S is of a full-rank r = k = 2. 7.3. NUMERIC COVARIATE 7.3.2 82 Raw polynomials The regression function is polynomial of a chosen degree k − 1, i.e., m(z) = β0 + β1 z + · · · + βk−1 z k−1 , z ∈ Z. (7.4) The parameterization is s : Z −→ Rk−1 , s(z) = z, . . . , z k−1 , z ∈ Z and the corresponding reparameterizing matrix Z1 . . S= . Zn is . . . Z1k−1 .. .. . . . k−1 . . . Zn Evaluation of the effect of the original covariate The effect of the original covariate on the response expectation is now quantified by a set of k − 1 Z regression coefficients β := β1 , . . . , βk−1 . To evaluate a statistical significance of the effect of the original covariate on the response expectation we have to test the null hypothesis H0 : β Z = 0k−1 . An appropriate test procedure was introduced in Section 3.2. Interpretation of the regression coefficients With k > 2 (at least a quadratic regression function), the single regression coefficients β1 , . . . , βk−1 only occasionally have a direct reasonable interpretation. Analogously to simple non-linear transformation of the covariate, the change in the response expectation that corresponds to a unity change of the original covariate is a function of that covariate: E Y Z = z + 1 − E Y Z = z = β1 + β2 (z + 1)2 − z 2 + · · · + βk−1 (z + 1)k−1 − z k−1 , z ∈ Z. Note. It is again easily seen that if n > k and the data contain at least k different values among among Z1 , . . . , Zn (which has a probabilityof one if the covariates Zi are sampled from a continuous distribution), the model matrix 1n , S is of a full-rank r = k. Degree of a polynomial Test on a subset of regression coefficients (Section 3.2) or a submodel test (Section 5.2) can be used to infer on the degree of a polynomial in the regression function (7.4). The null hypothesis expressing, for d < k, belief that the regression function is a polynomial of degree d−1 corresponds to the null hypothesis H0 : βd = 0 & . . . & βk−1 = 0. 7.3. NUMERIC COVARIATE 7.3.3 83 Orthonormal polynomials The regression function is again polynomial of a chosen degree k − 1, nevertheless, a different basis of the regression space, i.e., a different parameterization of the polynomial is used. Namely, the regression function is m(z) = β0 + β1 P 1 (z) + · · · + βk−1 P k−1 (z), z ∈ Z, (7.5) where P j is an orthonormal polynomial of degree j, j = 1, . . . , k − 1 built above a set of the covariate datapoints Z1 , . . . , Zn . That is, P j (z) = aj,0 + aj,1 z + · · · + aj,j z j , j = 1, . . . , k − 1, (7.6) and the polynomial coefficients aj,l , j = 1, . . . , k − 1, l = 0, . . . , j are such that vectors P j (Z1 ) . . Pj = . , j = 1, . . . , k − 1, P j (Zn ) are all orthonormal and also orthogonal to an intercept vector P 0 = 1, . . . , 1 . The corresponding reparameterizing matrix is P 1 (Z1 ) . . . P k−1 (Z1 ) .. .. .. S = P 1 , . . . , P k−1 = (7.7) . . . P 1 (Zn ) . . . P k−1 (Zn ), which leads to the model matrix X = 1n , S which have all columns mutually orthogonal and the non-intercept columns having even a unity norm. For methods of calculation of the coefficients of the polynomials (7.6), see lectures on linear algebra. It can only be mentioned here that as soon as the data contain at least k different values among Z1 , . . . , Zn , those polynomial coefficients exist and are unique. Note. For given dataset and given polynomial degree k −1, the model matrix X = 1n , S based on the orthonormal polynomial provide the same regression space as the model matrix based on the raw polynomials. Hence, the two model matrices determine two equivalent linear models. Advantages of orthonormal polynomials compared to raw polynomials • All non-intercept columns of the model matrix have the same (unity) norm. Consequently, all non-intercept regression coefficients β1 , . . . , βk−1 have the same scale. This may be helpful when evaluating a practical (not statistical!) importance of higher-order degree polynomial terms. b • Matrix X> X is a diagonal matrix diag(n, 1, . . . , 1). Consequently, the covariance matrix var β is also a diagonal matrix, i.e., the LSE of the regression coefficients are uncorrelated. Evaluation of the effect of the original covariate The effect of the original covariate on the response expectation is again quantified by a set of k − 1 Z regression coefficients β := β1 , . . . , βk−1 . To evaluate a statistical significance of the effect of the original covariate on the response expectation we have to test the null hypothesis H0 : β Z = 0k−1 . See Section 3.2 for a possible test procedure. 7.3. NUMERIC COVARIATE 84 Interpretation of the regression coefficients The single regression coefficients β1 , . . . , βk−1 do not usually have a direct reasonable interpretation. Degree of a polynomial Test on a subset of regression coefficients/test on submodels (were introduced in Sections 3.2 and 5.2) can again be used to infer on the degree of a polynomial in the regression function (7.5) in the same way as with the raw polynomials. The null hypothesis expressing, for d < k, belief that the regression function is a polynomial of degree d − 1 corresponds to the null hypothesis H0 : βd = 0 & . . . & βk−1 = 0. 7.3.4 Regression splines Basis splines The advantage of a polynomial regression function introduced in Sections 7.3.2 and 7.3.3 is that it is smooth (have continuous derivatives of all orders) on the whole real line. Nevertheless, with the least squares estimation, each data point affects globally the fitted regression function. This often leads to undesirable boundary when effects the fitted regression function only poorly approximates the response expectation E Y Z = z for the values of z being close to the boundaries of the covariate space Z. This can be avoided with so-called regression splines. Definition 7.2 Basis spline with distinct knots. Let d ∈ N0 and λ = λ1 , . . . , λd+2 ∈ Rd+2 , where −∞ < λ1 < · · · < λd+2 < ∞. The basis spline of degree d with distinct knots8 λ is such a function B d (z; λ), z ∈ R that (i) B d (z; λ) = 0, for z ≤ λ1 and z ≥ λd+2 ; (ii) On each of the intervals (λj , λj+1 ), j = 1, . . . , d + 1, B d (·; λ) is a polynomial of degree d; (iii) B d (·; λ) has continuous derivatives up to an order d − 1 on R. Notes. • The basis spline with distinct knots is piecewise 9 polynomial of degree d on (λ1 , λd+2 ). • The polynomial pieces are connected smoothly (of order d − 1) at inner knots λ2 , . . . , λd+1 . • On the boundary (λ1 and λd+2 ), the polynomial pieces are connected smoothly (of order d − 1) with a constant zero. Definition 7.3 Basis spline with coincident left boundary knots. Let d ∈ N0 , 1 < r < d + 2 and λ = λ1 , . . . , λd+2 ∈ Rd+2 , where −∞ < λ1 = · · · = λr < · · · < λd+2 < ∞. The basis spline of degree d with r coincident left boundary knots10 λ is such a function B d (z; λ), z ∈ R that 8 bazický spline [čti splajn] stupně d se vzájemně různými uzly se levými uzly 9 po částech 10 bazický spline stupně d s r překrývajícími 7.3. NUMERIC COVARIATE 85 (i) B d (z; λ) = 0, for z ≤ λr and z ≥ λd+2 ; (ii) On each of the intervals (λj , λj+1 ), j = r, . . . , d + 1, B d (·; λ) is a polynomial of degree d; (iii) B d (·; λ) has continuous derivatives up to an order d − 1 on (λr , ∞); (iv) B d (·; λ) has continuous derivatives up to an order d − r in λr . Notes. • The only qualitative difference between the basis spline with coincident left boundary knots and the basis spline with distinct knots is the fact that the basis spline with coincident left boundary knots is at the left boundary smooth of order only d − r compared to order d − 1 in case of the basis spline with distinct knots. • By mirroring Definition 7.3 to the right boundary, basis spline with coincident right boundary knots is defined. Basis B-splines There are many ways on how to construct the basis splines that satisfy conditions of Definitions 7.2 and 7.3, see Fundamentals of Numerical Mathematics (NMNM201) course. In statistics, so called B-splines have proved to be extremely useful for regression purposes. It goes beyond the scope of this lecture to explain in detail their construction which is fully covered by two landmark books de Boor (1978, 2001); Dierckx (1993) or in a compact way, e.g., by a paper Eilers and Marx (1996). For the purpose of this lecture it is assumed that a routine is available to construct the basis B-splines of given degree with given knots (e.g., the R function bs from the recommended package splines). An important property of the basis B-splines is that they are positive inside their support interval (general basis splines is, if can also attain negative values inside the support interval). That d λ = λ1 , . . . , λd+2 is a set of knots (either distinct or coincident left or right) and B (·, λ) is a basis B-spline of degree d built above the knots λ then B d (z, λ) > 0, λ1 < z < λd+2 , B d (z, λ) = 0, z ≤ λ1 , z ≥ λd+2 . Spline basis Definition 7.4 Spline basis. Let d ∈ N0 , k ≥ d+1 and λ = λ1 , . . . , λk−d+1 ∈ Rk−d+1 , where −∞ < λ1 < . . . < λk−d+1 < ∞. The spline basis11 of degree d with knots λ is a set of basis splines B1 , . . . , Bk , where for z ∈ R, B1 (z) = B d (z; λ1 , . . . , λ1 , λ2 ), | {z } (d+1)× B2 (z) = B d (z; λ1 , . . . , λ1 , λ2 , λ3 ), | {z } d× .. . 11 splinová báze 7.3. NUMERIC COVARIATE 86 Bd (z) = B d (z; λ1 , λ1 , λ2 , . . . , λd+1 ), | {z } 2× Bd+1 (z) = B d (z; λ1 , λ2 , . . . , λd+2 ), Bd+2 (z) = B d (z; λ2 , . . . , λd+3 ), .. . Bk−d (z) = B d (z; λk−2d , . . . , λk−d+1 ), Bk−d+1 (z) = B d (z; λk−2d+1 , . . . , λk−d+1 , λk−d+1 ), {z } | 2× .. . Bk−1 (z) = B d (z; λk−d−1 , λk−d . . . , λk−d+1 , . . . , λk−d+1 ), {z } | d× Bk (z) = B d (z; λk−d . . . , λk−d+1 , . . . , λk−d+1 ). {z } | (d+1)× Properties of the B-spline basis If k ≥ d + 1, a set of knots λ = λ1 , . . . , λk−d+1 , −∞ < λ1 < . . . < λk−d+1 < ∞ is given and B1 , . . . , Bk is the spline basis of degree d with knots λ composed of basis B-splines k ≥ d + 1, a set of knots λ = λ1 , . . . , λk−d+1 , −∞ < λ1 < . . . < λk−d+1 < ∞ is given and B1 , . . . , Bk is the spline basis of degree d with knots λ composed of basis B-splines then (a) k X Bj (z) = 1 for all z ∈ λ1 , λk−d+1 ; (7.8) j=1 (b) for each m ≤ d there exist a set of coefficients γ1m , . . . , γkm such that k X γjm Bj (z) is on (λ1 , λk−d+1 ) a polynomial in z of degree m. (7.9) j=1 Regression spline It will now be assumed that the covariate space is a bounded interval, i.e., Z = zmin , zmax , −∞ < zmin < zmax < ∞. The regression function that exploits the regression splines is m(z) = β1 B1 (z) + · · · + βk Bk (z), z ∈ Z, (7.10) where B1 , . . . , Bk is the spline basis of chosen degree d ∈ N0 composed of basis B-splines built above a set of chosen knots λ = λ1 , . . . , λk−d+1 , zmin = λ1 < . . . < λk−d+1 = zmax . The corresponding reparameterizing matrix coincided with the model matrix and is B1 (Z1 ) . . . Bk (Z1 ) . .. .. .. X=S= (7.11) . . =: B. B1 (Zn ) . . . Bk (Zn ) End of Lecture #5 (15/10/2015) 7.3. NUMERIC COVARIATE 87 Start of Lecture #7 (22/10/2015) Notes. • It follows from (7.8) that 1n ∈ M B . This is also the reason why we do not explicitely include the intercept term in the regression function since it is implicitely included in the regression space. Due to clarity of notation, the regression coefficients are now indexed from 1 to k. That is, the vector of regression coefficients is β = β1 , . . . , βk . • It also follows from (7.9) that for any m ≤ d, a linear model with the regression function based on either raw or orthonormal polynomials of degree m is a submodel of the linear model with the regression function given by a regression spline and the model matrix B. • With d = 0, the regression spline (7.10) is simply a piecewise constant function. • In practice, not much attention is paid to the choice of the degree d of the regression spline. Usually d = 2 (quadratic spline) or d = 3 (cubic spline) is used which provides continuous first or second derivatives, respectively, of the regression function inside the covariate domain Z. • On the other hand, the placement of knots (selection of the values of λ1 , . . . , λk−d+1 ) is quite important to obtain function that sufficiently well approximates the response the regression expectations E Y Z = z , z ∈ Z. Unfortunately, only relatively ad-hoc methods towards selection of the knots will be demonstrated during this lecture as profound methods of the knots selection go far beyond the scope of this course. Advantages of the regression splines compared to raw/orthogonal polynomials • Each data point influences the LSE of the regression coefficients and hence the fitted regression function only locally. Indeed, only the LSE of those regression coefficients that correspond to the basis splines whose supports cover a specific data point are influenced by those data points. • Regression splines of even a low degree d (2 or 3) are, with a suitable choice of knots, able to approximate sufficiently well even functions with a highly variable curvature and that globally on the whole interval Z. Evaluation of the effect of the original covariate To evaluate a statistical significance of the effect of the original covariate on the response expectation we have to test the null hypothesis H0 : β1 = · · · = βk . Due to the property (7.8), this null hypothesis corresponds to assuming that E Y Z ∈ M 1n ⊂ M B . Consequently, it is possible to use a test on submodel that was introduced in Section 5.1 to test the above null hypothesis. Interpretation of the regression coefficients The single regression coefficients β1 , . . . , βk do not usually have a direct reasonable interpretation. 7.4. CATEGORICAL COVARIATE 7.4 88 Categorical covariate In this Section, it is assumed that Zi ∈ Z, i = 1, . . . , n, are categorical covariates. That is, the covariate sample space Z is finite and its elements are only understood as labels. Without loss of generality, we will use, unless stated otherwise, a simple sequence 1, . . . , G for those labels, i.e., Z = 1, . . . , G . Unless explicitely stated (in Section 7.4.4), even the ordering of the labels 1 < · · · < G will not be used for any but notational purposes and the methodology described below is then suitable for both nominal and ordinal categorical covariates. The regression function, m : Z −→ R is now a function defined on a finite set aiming in parameterizing just G (conditional) response expectations E Y Z = 1 , . . . , E Y Z = G . For some clarity in notation, we will also use symbols m1 , . . . , mG for those expectations, i.e., m(1) = E Y Z = 1 =: m1 , .. .. . . m(G) = E Y Z = G =: mG . Notation and terminology (One-way classified group means). Since a categorical covariate often indicates pertinence to one of G groups, we will call m1 , . . . , mG as group means12 or one-way classified group means A vector m = m1 , . . . , mG will be called a vector of group means,13 or a vector of one-way classified group means. Note. Perhaps appealing simple regression function of the form m(z) = β0 + β1 z, z = 1, . . . , G, is in most cases fully inappropriate. First, it orders ad-hoc the group means to form a monotone sequence (increasing if β1 > 0, decreasing if β1 < 0). Second, it ad-hoc assumes a linear relationship between the group means. Both those properties also depend on the ordering or even the values of the labels (1, . . . , G in our case) assigned to the G categories at hand. With a nominal categorical covariate, none of it is justifiable, with an ordinal categorical covariate, such assumptions should, at least, never be taken for granted and used without proper verification. 7.4.1 Link to a G-sample problem For following considerations, we will additionally assume (again without loss of generality) that the data Yi , Zi , i = 1, . . . , n, are sorted according to the covariate values Z1 , . . . , Zn . Furthermore, we will also exchangeably use a double subscript with the response where the first subscript will 12 skupinové střední hodnoty 13 vektor skupinových středních hodnot 7.4. CATEGORICAL COVARIATE indicate the covariate value, i.e., 1 Z1 . . .. .. 1 Zn1 − − − −− . .. = .. Z= . − − − −− Z n−nG +1 G . .. . . . Zn G , 89 n1 -times nG -times Y1 .. . Yn1 − − −− .. = Y = . − − −− Y n−nG +1 .. . Yn Y1,1 .. . Y1,n1 −−− .. . . −−− YG,1 .. . YG,nG Finally, let Y g = Yg,1 , . . . , Yg,ng , g = 1, . . . , G, denote a subvector of the response vector that corresponds to observations with the covariate value being equal to g. That is, Y = Y1 , . . . , Yn = Y 1 , . . . , Y G . A related regression model written using above introduced notation and the error terms is then Yg,j = mg + εg,j , E ε = 0n , ε := ε1,1 , . . . , εG,nG , var ε = σ 2 In . (7.12) Notes. • If the covariates Z1 , . . . , ZG are random then also n1 , . . . , nG are random. • If the covariates Z1 , . . . , ZG are fixed and errors ε1,1 , . . . , εG,nG in (7.12) are assumed to be independent (possibly also identically distributed) then (7.12) is a “regression” parameterization of a classical G-sample problem, where n1 , . . . , nG are the sample sizes of each sample. Observation of the sample g, g = 1, . . . , G, are given by a vector Y g , the expected value in the sample g is given by mg and the variance in the sample g is given by σ 2 . Note that if model (7.12) is assumed then it is assumed that the G samples are homoscedastic, i.e., have all the same variance. • In the following, it is always assumed that n1 > 0, . . . , nG > 0. 7.4. CATEGORICAL COVARIATE 7.4.2 90 Linear model parameterization of one-way classified group means As usual, let µ be the (conditional) µ1,1 .. . µ1,n1 −− .. E Y Z = µ := . −− µ G,1 .. . µG,nG response expectation, i.e., m1 . .. n1 -times m1 −− m1 1n1 . .. = .. . = . −− mG 1nG m G . . nG -times . m (7.13) G Notation and terminology (Regression space of a categorical covariate). A vector space m 1 1 n 1 . : m1 , . . . , mG ∈ R ⊆ Rn .. m 1 G nG will be called the regression space of a categorical covariate (factor) with levels frequencies n1 , . . . , nG and will be denoted as MF (n1 , . . . , nG ). Note. Obviously, with n1 > 0, . . . , nG > 0, a vector dimension of MF (n1 , . . . , nG ) is equal to G and a possible (orthogonal) vector basis is 1 ... 0 . . . .. .. .. n1 -times 1 ... 0 − − −− 1n1 ⊗ 1, . . . , . . . .. .. .. Q= = . .. − − −− 1nG ⊗ 0, . . . , 0 ... 1 . . . . . . nG -times . . . 0 ... 1 0 . 1 (7.14) When using the linear model, we are trying to allow for expressing the response expectation µ, i.e., a vector from MF (n1 , . . . , nG ) as a linear combination of columns of a suitable n × k matrix X, i.e., as µ = Xβ, β ∈ Rk . It is obvious that any model matrix that parameterizes the regression space MF (n1 , . . . , nG ) 7.4. CATEGORICAL COVARIATE 91 must have at least G columns, i.e., k ≥ G x> 1 . .. > x1 −− . X= .. −− x> G . . . x> G and must be of the type n1 -times 1n1 ⊗ x> 1 .. = . 1nG ⊗ x> G nG -times , (7.15) where x1 , . . . , xG ∈ Rk are suitable vectors. Problem of parameterizing a categorical covariate with G levels thus simplifies into selecting a G×k e such that matrix X x> 1 . e . X= . . > xG Clearly, e . rank X = rank X Hence to be able to parameterize the regression space MF (n1 , . . . , nG ) which has a vector e must satisfy dimension of G, the matrix X e = G. rank X The group means then depend on a vector β = β0 , . . . , βk−1 of the regression coefficients as mg = x> g β, g = 1, . . . , G, e m = Xβ. A possible (full-rank) linear model parameterization of regression space of a categorical covariate e = IG and we have uses matrix Q from (7.14) as a model matrix X. In that case, X µ = Q β, m = β. (7.16) Even though parameterization (7.16) seems appealing since the regression coefficients are directly equal to the group means, it is only rarely considered in practice for reasons that will become clear later on. Still, it is useful for some of theoretical derivations. 7.4.3 ANOVA parameterization of one-way classified group means In practice and especially in the area of designed experiments, the group means are parameterized as mg = α0 + αg , g = 1, . . . , G, (7.17) m = 1G , IG α = α0 1G + αZ , 7.4. CATEGORICAL COVARIATE 92 where α = α0 , α1 , . . . , αG is a vector of regression coefficients and αZ = α1 , . . . , αG is its e is non-intercept subvector. That is, the matrix X e = 1G , IG . X The model matrix is then 1 1 ... 0 . . .. .. .. .. . . 1 1 ... 0 − − − − −− . . .. .. X= . . .. .. − − − − −− 1 0 ... 1 . . .. .. . . . . . . 1 0 ... 1 n1 -times 1n1 ⊗ 1, 1, . . . , 0 .. , = . 1nG ⊗ 1, 0, . . . , 1 (7.18) nG -times which has G + 1 columns but its rank is G (as required). That is, thelinear model Y | Z ∼ Xα, σ 2 In is less-than-full rank. In other words, for given µ ∈ M X = MF (n1 , . . . , nG ) a vector α ∈ RG+1 such that µ = Xα is not unique. Consequently, also a solution to related normal equations is not unique. Nevertheless, unique solution can be obtained if suitable indetifying constraints14 are imposed on the vector of regression coefficients α. Terminology (Effects of a categorical covariate). Values of α1 , . . . , αG (a vector αZ ) are called effects of a categorical covariate. Note. Effects of a categorical covariate are not unique. Hence their interpretation depends on chosen identifying constraints. Identification in less-than-full-rank linear model In the following, a linear model Y X ∼ Xβ, σ 2 In , rank(Xn×k ) = r will be assumed (in our general notation), where r < k. We shall consider linear constraints on a vector of regression coefficients, i.e., constraints of the type Aβ = 0m , where A is an m × k matrix. Definition 7.5 Identifying constraints. We say that a constraint Aβ = 0m identifies a vector β in a linear model Y X ∼ Xβ, σ 2 In if and only if for each µ ∈ M X there exists only one vector β which satisfies at the same time µ = Xβ 14 identifikační omezení and Aβ = 0m . 7.4. CATEGORICAL COVARIATE 93 Note. If a matrix A determines the identifying constraints, then, due to Theorem 2.5 (least squares and normal equations), it also uniquely determines the solution to normal equations. That is, there b that jointly solves linear systems is a unique solution b = β X> Xb = X> Y , Ab = 0m , or written differently, there is a unique solution to a linear system ! ! X> X X> Y . b= A 0m The question is now, what are the conditions for a matrix A to determinean identifying constraint. Remember (Theorem 2.7): If a matrix Lm×k satisfies M L> ⊂ M X> then a parameter vector θ = Lβ is estimable which also means that for all real vectors β 1 , β 2 the following holds: Xβ 1 = Xβ 2 =⇒ Lβ 1 = Lβ 2 . That is, if two different solutions of normal equations are taken and one of them satisfies the constraint then do the both. It was also shown in Section 5.3 that if further L has linearly independent rows then a set of linear constraints Lβ = 0 determines a so called submodel (Lemma 5.4). It follows from above that for identification, we cannot use such a matrix L for identification. Theorem 7.1 Scheffé on identification in a linear model. X ∼ Constraint Aβ = 0 with a real matrix A identifies a vector β in a linear model Y m m×k 2 Xβ, σ In , rank(Xn×k ) = r < k ≤ n if and only if M A> ∩ M X> = 0 , rank(X) + rank(A) = k. Proof. We have to show that, for any µ ∈ M X , the conditions stated in the theorem are equivalent to existence of the unique solution to a linear system Xβ = µ that satisfies Aβ = 0m . Existence of the solution ⇔ ∀µ ∈ M X there exists a vector β ∈ Rk such that Xβ = µ & Aβ = 0m . ⇔ ∀µ ∈ M X there exists a vector β ∈ Rk such that ! ! Xn×k µ β = . 0m Am×k | {z } D 7.4. CATEGORICAL COVARIATE ⇔ ⇔ ⇔ ⇔ ∀µ ∈ M X there exists a vector β ∈ Rk such that ! µ . Dβ = 0m n > o ⊆M D . µ> , 0> , µ ∈ M X m n o⊥ n > > > o⊥ ⊆ µ , 0m , µ ∈ M X . M D ∀v 1 ∈ Rn , v 2 ∈ Rm ∀µ ∈ M X v> 1, ⇔ > v2 D = ∀v 1 ∈ Rn , v 2 ∈ Rm v> 1, ⇔ 94 v> 2 n ∀v 1 ∈ Rn , v 2 ∈ Rm 0> k =0 . ! =0 . ⇒ v> 1, v2 ⇒ v> 1, v> 2 Xβ 0m ∀β ∈ Rk D= 0> k ∀β ∈ Rk > v> 1 X = −v 2 A ⇔ n ∀v 1 ∈ Rn , v 2 ∈ Rm n ∀v 1 ∈ Rn , v 2 ∈ Rm ⇔ n ∀u ∈ Rk ⇔ M A> ∩ M X> = 0 . ⇔ ! µ 0m > ⇒ o v> Xβ = 0 . 1 > v> 1 X = −v 2 A ⇒ o > v> X = 0 1 k . o X> v 1 = −A> v 2 ⇒ X> v 1 = 0k . | {z } o u > u∈M X ∩ M A> ⇒ u = 0k . Uniqueness of the solution ⇔ ∀µ ∈ M X there exists a unique vector β ∈ Rk such that Xβ = µ & Aβ = 0m . ⇔ ⇔ ⇔ ∀µ ∈ M X there exists a unique vector β ∈ Rk such that ! ! Xn×k µ β = . Am×k 0m | {z } D rank(D) = k. n o A has rows such that dim M A> = k − r (since rank(X) = r) and all rows in A are linearly independent with rows in X. ⇔ rank(A) = k − r (since we already have a condition M X> ∩ M A> = 0} needed for existence of the solution). 7.4. CATEGORICAL COVARIATE 95 k Notes. 1. Matrix Am×k used for identification must satisfy rank(A) = k − r. In practice, the number of identifying constraints (the number of rows of the matrix A) is usually the lowest possible, i.e., m = k − r. 2. Theorem 7.1 further states that the matrix A must be such that a vector parameter θ = Aβ is not estimable in a given model. 3. In practice, a vector µ, for which we look for a unique β such that µ = Xβ, Aβ = 0m b ∈ Rk (that, since is equal to the vector of fitted values Yb . That is, we look for a unique β being unique, can be considered as the LSE of the regression coefficients) such that b = Yb Xβ & b = 0m . Aβ b if and only if β b solves By Theorem 2.5 (Least squares and normal equations), Yb = Xβ > > b =X Y. normal equations X Xβ Suppose now that rank A = m = k − r, i.e., the regression parameters are identified by b we have to solve a set of m = k − r linearly independent linear constraints. To get β, a linear system b = X> Y , X> Xβ b = 0m , Aβ which can be solved by solving b = X> Y , X> Xβ b = 0m , A> Aβ or using a linear system which written differently is b = X> Y , X> X + A> A β b = X> Y , D> Dβ with D= ! X . A Matrix D> D is now an invertible k × k matrix and hence the unique solution is b = D> D β −1 X> Y . 7.4. CATEGORICAL COVARIATE 96 Identification in a one-way ANOVA model As example of use of Scheffé’s Theorem 7.1, consider a model matrix X given by (7.18) that provides an ANOVA parameterization of a single categorical covariate, i.e., a linear model for the one-way classified group means parameterized as mg = α0 + αg , g = 1, . . . , G. We have rank Xn×(G+1) = G with a vector α = α0 , α1 , . . . , αG of the regression coefficients. The smallest matrix Am×(G+1) that identifies α with respect to the regression space M X = MF (n1 , . . . , nG ) is hence a non-zero matrix with m = 1 row, i.e., A = a> = a0 , a1 , . . . , aG 6= 0G+1 such that a ∈ / M X> , i.e., such that θ = a> α is not estimable in the linear model Y X ∼ Xα, σ 2 In . It is seen from a structure of the matrix X given by (7.18) that a ∈ M X> ⇐⇒ a= G X cg , c1 , . . . , cG g=1 for some c = c1 , . . . , cG ∈ RG , c 6= 0G . That is, for identification of α in the linear model Y X ∼ Xα, σ 2 In with the model matrix (7.18), we can use any vector a = a0 , a1 , . . . , aG 6= 0G−1 that satisfy G X a0 6= ag . g=1 Commonly used identifying constraints include: Sum constraint: A1 = a> 1 = 0, 1, . . . , 1 ⇐⇒ G X αg = 0 g=1 that imply the following interpretation of the model parameters: G α0 = 1 X mg G =: m, g=1 α1 = m1 − m, .. . αG = mG − m. Weighted sum constraint: A2 = a> 2 = 0, n1 , . . . , nG ⇐⇒ G X ng αg = 0 g=1 that implies G α0 = 1X ng mg n g=1 α1 = m1 − mW , .. . αG = mG − mW . =: mW , 7.4. CATEGORICAL COVARIATE 97 Reference group constraint (l ∈ {1, . . . , G}): A3 = a> . . . , 1, . . . , 0 3 = 0, 0, | {z } 1 on lth place ⇐⇒ αl = 0, which corresponds to omitting one of the non-intercept columns in the model matrix X given by (7.18) and using the resulting full-rank parameterization. It implies α0 = m l , α1 = m 1 − m l , .. . αG = mG − ml . No intercept: A4 = a> 4 = 1, 0, . . . , 0 ⇐⇒ α0 = 0, which corresponds to omitting the intercept column in the model matrix X given by (7.18) and using the full-rank parameterization with the matrix Q given by (7.14). That is, α0 = 0, α1 = m 1 , .. . αG = mG . Note. Identifying constraints given by vectors a1 , a2 , a3 (sum, weighted sum and reference group constraint) correspond to one of commonly used full-rank parameterizations that will be introduced in Section 7.4.4 where we shall also discuss interpretation of the effects αZ = α1 , . . . , αG if different identifying constraints are used. 7.4.4 End of Lecture #7 (22/10/2015) Full-rank parameterization of one-way classified group means Start of In the following, we limit ourselves to full-rank parameterizations that involve an intercept column. Lecture #9 That is, the model matrix will be an n × G matrix (29/10/2015) 1 c> 1 . .. .. n1 -times . 1 c> 1 −−− 1n1 ⊗ 1, c> 1 . .. .. , . = X= . . . > −−− 1nG ⊗ 1, cG 1 c> G . . . . nG -times . . 1 c> G where c1 , . . . , cG ∈ are suitable vectors. In the following, let C be an G × (G − 1) matrix with those vectors as rows, i.e., c> 1 . . C= . . > cG RG−1 7.4. CATEGORICAL COVARIATE 98 e is thus a G × G matrix A matrix X e = 1G , C . X If β = β0 , . . . , βG−1 ∈ RG denote, as usual, a vector of regression coefficients, the group means m are parameterized as where β Z know, Z mg = β0 + c> g = 1, . . . , G, gβ , (7.19) e = 1G , C β = β0 1G + Cβ Z , m = Xβ = β1 , . . . , βG−1 is a non-intercept subvector of the regression coefficients. As we e = rank (1G , C) . rank X = rank X Hence, to get the model matrix X of a full-rank (rank X = G), the matrix C must satisfy rank C = G − 1 and 1G ∈ / M C . That is, the columns of C must be (i) (G − 1) linearly independent vectors from RG ; (ii) being all linearly independent with a vector of ones 1G . Definition 7.6 Full-rank parameterization of a categorical covariate. Full-rank parameterization of a categorical covariate with G levels (G = card(Z)) is a choice of the G × (G − 1) matrix C that satisfies rank C = G − 1, 1G ∈ /M C . Terminology ((Pseudo)contrast matrix). Columns of matrix C are often chosen to form a set of G − 1 linearly independent contrasts from RG . In this case, we will call the matrix C as a contrast matrix.15 In other cases, the matrix C will be called as a pseudocontrast matrix.16 Note. The (pseudo)contrast matrix C also determines parameterization of a categorical covariate according to Definition 7.1. Corresponding function s : Z −→ RG−1 is s(z) = cz , z = 1, . . . , G, and the reparameterizing matrix S is an n × (G − 1) matrix c> 1 . .. n1 -times > c1 −− 1n1 ⊗ c> 1 . . .. = S = .. −− 1nG ⊗ c> G c> G . . nG -times . c> G 15 kontrastová matice 16 pseudokontrastová matice . 7.4. CATEGORICAL COVARIATE 99 Evaluation of the effect of the categorical covariate With a given full-rank parameterization of a categorical covariate, evaluation of a statistical significance of its effect on the response expectation corresponds to testing the null hypothesis H0 : β1 = 0 & · · · & βG−1 = 0, or written concisely (7.20) H0 : β Z = 0G−1 . This null hypothesis indeed also corresponds to a submodel where only intercept is included in the model matrix. Finally, it can be mentioned that the null hypothesis (7.20) is indeed equivalent to the hypothesis of equality of the group means H0 : m1 = · · · = mG . (7.21) If normality of the response is assumed, equivalently an F-test on a submodel (Theorem 5.1) or a test on a value of a subvector of the regression coefficients (F-test if G ≥ 2, t-test if G = 2, see Theorem 3.1) can be used. The following can be shown with only a little algebra: • If G = 2, β = β0 , β1 . The (usual) t-statistic to test the hypothesis H0 : β1 = 0 using point (viii) of Theorem 3.1, i.e., the statistic based on the LSE of β, is the same as a statistic of a standard two-sample t-test. Notes. • If G ≥ 2, the (usual) F-statistic to test the null hypothesis (7.20) using point (x) of Theorem 3.1 which is the same as the (usual) F-statistic on a submodel, where the submodel is the onlyintercept model, is the same as an F-statistic used classically in one-way analysis of variance (ANOVA) to test the null hypothesis (7.21). Reference group pseudocontrasts 0 ... 0 1 . . . 0 = C = . . . . ... .. 0 ... 1 0> G−1 IG−1 ! (7.22) The regression coefficients have the following interpretation m1 = β0 , m2 = β0 + β1 , .. . β0 = m1 , β 1 = m2 − m1 , .. . (7.23) βG−1 = mG − m1 . mG = β0 + βG−1 , That is, the interceptβ0 is equal to the mean of the first (reference) group, the elements of β Z = β1 , . . . , βG−1 (the effects of Z) provide differences between the means of the remaining groups and the reference one. The regression function can be written as m(z) = β0 + β1 I(z = 2) + · · · + βG−1 I(z = G), z = 1, . . . , G. It is seen from (7.23) that the full-rank parameterization using the reference group pseudocontrasts is equivalent to the less-than-full-rank (ANOVA) parameterization mg = α0 + αg , g = 1, . . . , G, where α = α0 , α1 , . . . , αG is identified by the reference group constraint α1 = 0. 7.4. CATEGORICAL COVARIATE 100 Notes. • With the pseudocontrast matrix C given by (7.22), a group labeled by Z = 1 is chosen as a reference for which the intercept β0 provides the group mean. In practice, any other group can be taken as a reference by moving the zero row of the C matrix. • In the R software, the reference group pseudocontrasts with the C matrix being of the form (7.22) are used by default to parameterize categorical covariates (factors). Explicitely this choice is indicated by the contr.treatment function. Alternatively, the contr.SAS function provides a pseudocontrast matrix in which the last Gth group serves as the reference, i.e., the C matrix has zeros on its last row. Sum contrasts 1 ... 0 . . .. .. .. . = C= 1 0 ... −1 . . . −1 Let IG−1 − 1> G−1 ! (7.24) G 1 X m= mg . G g=1 The regression coefficients have the following interpretation β0 = m, β1 = m1 − m, m1 = β0 + β1 , mG−1 .. . = β0 + βG−1 , mG = β0 − G−1 X βG−1 .. . = mG−1 − m. (7.25) βg , g=1 The regression function can be written as m(z) = β0 + β1 I(z = 1) + · · · + βG−1 I(z = G − 1) − G−1 X βg I(z = G), g=1 z = 1, . . . , G. If we consider the less-than-full-rank ANOVA parameterization of the group means as mg = α0 +αg , g = 1, . . . , G, it is seen from (7.25) that the full-rank parameterization using the contrast matrix (7.24) links the regression coefficients of the two models as = m, = µ1 − m, .. . α0 = β0 α1 = β1 .. . αG−1 = βG−1 G−1 X αG = − βg g=1 = µG−1 − m, = µG − m. (7.26) 7.4. CATEGORICAL COVARIATE 101 At the same time, the vector α satisfies G X (7.27) αg = 0. g=1 That is, the full-rank parameterization using the sum contrasts (7.25) is equivalent to the lessthan-full-rank ANOVA parameterization, where the regression coefficients are identified by the sum constraint (7.27). The intercepts α0 = β0 equal to the mean of the group means and the elements of β Z = β1 , . . . , βG−1 = α1 , . . . , αG−1 are equal to the differences between the corresponding group mean and the means of the P group means. The same quantity for the last, Gth group, αG is calculated from β Z as αG = − G−1 g=1 βg . Note. In the R software, the sum contrasts with the C matrix being of the form (7.24) can be used by the mean of the function contr.sum. Weighted sum contrasts 1 .. . C= 0 n1 − nG Let ... .. . ... ... − 0 .. . 1 nG−1 (7.28) nG G mW 1X = ng mg . n g=1 The regression coefficients have the following interpretation β0 = mW , β1 = m1 − mW , m1 = β0 + β1 , mG−1 .. . = β0 + βG−1 , mG = β0 − G−1 X g=1 βG−1 .. . = mG−1 − mW . (7.29) ng βg , nG The regression function can be written as m(z) = β0 + β1 I(z = 1) + · · · + βG−1 I(z = G − 1) − G−1 X ng βg I(z = G), nG g=1 z = 1, . . . , G. If we consider the less-than-full-rank ANOVA parameterization of the group means as mg = α0 +αg , g = 1, . . . , G, it is seen from (7.29) that the full-rank parameterization using the contrast matrix 7.4. CATEGORICAL COVARIATE 102 (7.28) links the regression coefficients of the two models as α0 = β0 α1 = β1 .. . = mW , = m1 − mW , .. . = mG−1 − mW , αG−1 = βG−1 G−1 X ng βg αG = − nG = mG − mW . g=1 At the same time, the vector α satisfies G X (7.30) ng αg = 0. g=1 That is, the full-rank parameterization using the weighted sum pseudocontrasts (7.29) is equivalent to the less-than-full-rank ANOVA parameterization, where the regression coefficients are identified by the weighted sum constraint (7.30). The intercepts α0 = β0 equal to the weighted mean of the group means and the elements of β Z = β1 , . . . , βG−1 = α1 , . . . , αG−1 are equal to the differences between the corresponding group mean and the weighted means of the group means. PG−1 n Z The same quantity for the last, Gth group, αG is calculated from β as αG = − g=1 nGg βg . Helmert contrasts −1 −1 . . . −1 1 −1 . . . −1 2 ... −1 C= 0 . . . . .. .. .. .. 0 0 ... G − 1 The group means are obtained from the regression coefficients as m1 = β0 − G−1 X βg , g=1 m2 = β0 + β1 − G−1 X βg , g=2 G−1 X m3 = β0 + 2 β2 − βg , g=3 mG−1 .. . = β0 + (G − 2) βG−2 − βG−1 , mG = β0 + (G − 1) βG−1 . (7.31) 7.4. CATEGORICAL COVARIATE 103 Inversely, the regression coefficients are linked to the group means as G β0 = β1 = β2 = β3 = .. . βG−1 = 1 X mg G g=1 1 (m2 − m1 ), 2 o 1 1n m3 − (m1 + m2 ) , 3 2 n o 1 1 m4 − (m1 + m2 + m3 ) , 4 3 =: m, G−1 o 1n 1 X mG − mg . G G−1 g=1 which provide their (slightly awkward) interpretation: βg , g = 1, . . . , G − 1, is 1/(g + 1) times the difference between the mean of group g + 1 and the mean of the means of the previous groups 1, . . . , g. Note. In the R software, the Helmert contrasts with the C matrix being of the form (7.31) can be used by the mean of the function contr.helmert. Orthonormal polynomial contrasts P 1 (ω1 ) 1 P (ω2 ) C= .. . P 2 (ω1 ) P 2 (ω2 ) .. . ... ... .. . P G−1 (ω1 ) P G−1 (ω2 ) , .. . (7.32) P 1 (ωG ) P 2 (ωG ) . . . P G−1 (ωG ) where ω1 < · · · < ωG is an equidistant (arithmetic) sequence of the group labels and P j (z) = aj,0 + aj,1 z + · · · + aj,j z j , j = 1, . . . , G − 1, are orthonormal polynomials of degree 1, . . . , G − 1 built above a sequence of the group labels. Note. It can be shown that the polynomial coefficients aj,l , j = 1, . . . , G − 1, l = 0, . . . , j and hence the C matrix (7.32) is for given G invariant towards the choice of the group labels as soon 7.4. CATEGORICAL COVARIATE 104 as they form an equidistant (arithmetic) sequence. For example, for G = 2, 3, 4 the C matrix is G=2 G=3 1 √ − 2 C= 0 1 √ 2 1 −√ 2 C= 1 , √ 2 G=4 3 − √ 2 5 1 − √ 2 5 C= 1 √ 2 5 3 √ 2 5 1 2 1 − 2 1 − 2 1 2 1 √ 6 2 −√ , 6 1 √ 6 1 − √ 2 5 3 √ 2 5 . 3 − √ 2 5 1 √ 2 5 The group means are then obtained as m1 = m(ω1 ) = β0 + β1 P 1 (ω1 ) + · · · + βG−1 P G−1 (ω1 ), m2 = m(ω2 ) = β0 + β1 P 1 (ω2 ) + · · · + βG−1 P G−1 (ω2 ), .. . mG = m(ωG ) = β0 + β1 P 1 (ωG ) + · · · + βG−1 P G−1 (ωG ), where m(z) = β0 + β1 P 1 (z) + · · · + βG−1 P G−1 (z), z ∈ ω1 , . . . , ωG is the regression function. The regression coefficients β now do not have any direct interpretation. That is why, even though the parameterization with the contrast matrix (7.32) can be used with the categorical nominal covariate, it is only rarely done so. Nevertheless, in case of the categorical ordinal covariate where the ordered group labels ω1 < · · · < ωG have also practical interpretability, parameterization (7.32) can be used to reveal possible polynomial trends in the evolution of the group means m1 , . . . , mG and to evaluate whether it may make sense to consider that covariate as numeric rather than categorical. Indeed, for d < G, the null hypothesis H0 : βd = 0 & . . . & βG−1 = 0 corresponds to the hypothesis that the covariate at hand can be considered as numeric (with values ω1 , . . . , ωG of the form of an equidistant sequence) and the evolution of the group means can be described by a polynomial of degree d − 1. Note. In the R software, the orthonormal polynomial contrasts with the C matrix being of the form (7.32) can be used by the mean of the function contr.poly. It is also a default choice if the covariate is coded as categorical ordinal (ordered). End of Lecture #9 (29/10/2015) Chapter 8 Additivity and Interactions 8.1 Additivity and partial effect of a covariate Suppose now that the covariate vectors are Z1 , V 1 , . . . , Zn , V n ∈ Z × V, Z ⊆ R, V ⊆ Rp−1 , As usual, let Z, V ∈ Z × V denote a generic covariate, and let Z1 V> 1 . . .. .. , Z= V = Zn V> n Start of Lecture #11 (05/11/2015) p ≥ 2. be matrices covering the observed values of the two sets of covariates. 8.1.1 Additivity Definition 8.1 Additivity of the covariate effect. We say that a covariate Z ∈ Z acts additively in the regression model with covariates Z, V ∈ Z×V, if the regression function is of the form m(z, v) = mZ (z) + mV (v), z, v ∈ Z × V, (8.1) where mZ : Z −→ R and mV : V −→ R are some functions. 8.1.2 Partial effect of a covariate If the effect of Z ∈ Z acts additively in a regression model, we have for any fixed v ∈ V: E Y Z = z + 1, V = v − E Y Z = z, V = v = mZ (z + 1) − mZ (z), z ∈ Z. (8.2) That is, the influence (effect) of the covariate Z on the response expectation is the same with any value of V ∈ V. 105 8.1. ADDITIVITY AND PARTIAL EFFECT OF A COVARIATE 106 Terminology (Partial effect of a covariate). If a covariate Z ∈ Z acts additively in the regression model with covariates (Z, V ) ∈ Z × V, quantity (8.2) expresses so called partial effect 1 of the covariate Z on the response given the value of V . 8.1.3 Additivity, partial covariate effect and conditional independence In a context of a linear model, both mZ and mV are chosen to be linear in unknown (regression) parameters and the corresponding model matrix is decomposed as X = XZ , XV , where XZ corresponds to the regression function mZ and depends only on the covariate values Z, and XV corresponds to the regression function mV and depends only on the covariate values V. That is, the response expectation is assumed to be E Y Z, V = XZ β + XV γ, for some real vectors of regression coefficients β and γ. Matrix XZ and the regression function mZ then correspond to parameterization of a single covariate for which any choice out of those introduced in Sections 7.3 and 7.4 (or others notdiscussed here) can be used. Further, matrix XZ is/can be usually chosen such that 1n ∈ M XZ in which case a hypothesis of no partial effect of the Z covariate corresponds to testing a submodel with the model matrix X 0 = 1n , X V againts a model with the model matrix X = XZ , XV . Note that if it can be assumed that the covariates at hand influence only the (conditional) response expectation and not other characteristics of the conditional distribution of the response given the covariates, then testing a submodel with the model matrix X0 against a model with the model matrix X corresponds to testing a conditional independence of the response and the Z covariate given the remaining covariates V . 1 parciální efekt 8.2. ADDITIVITY OF THE EFFECT OF A NUMERIC COVARIATE 8.2 107 Additivity of the effect of a numeric covariate Assume that Z is a numeric covariate with Z ⊆ R. While limiting ourselves to parameterizations discussed in Section 7.3, the matrix XZ can be (i) XZ = 1n , SZ , where SZ is a reparameterizing matrix of a parameterization sZ = s1 , . . . , sk−1 : Z −→ Rk−1 having a form of either (a) a simple transformation (Section 7.3.1); (b) raw polynomials (Section 7.3.2); (c) orthonormal polynomials (Section 7.3.3). Z If we denote the regression coefficients related to the model matrix X as β = β0 , β1 , . . . , βk−1 , the regression function is m(z, v) = β0 + β1 s1 (z) + · · · + βk−1 sk−1 (z) + mV (v), z, v ∈ Z × V, (8.3) which can also be interpreted as m(z, v) = γ0 (v) + β1 s1 (z) + · · · + βk−1 sk−1 (z), z, v ∈ Z × V, where γ0 (v) = β0 + mV (v). In other words, if a certain covariate acts additively and its effect on the response is described by parameterization sZ then the remaining covariates V only modify an intercept term in the relationship between the response and the covariate Z. (ii) XZ = BZ , where BZ is a model matrix (7.11) of the regression splines B1 , . . . , Bk . With the regression coefficients related to the model matrix BZ being denoted as β = β1 , . . . , βk , the regression function becomes m(z, v) = β1 B1 (z) + · · · + βk Bk (z) + mV (v), z, v ∈ Z × V, (8.4) where the term mV (v) can again be interpreted as an intercept γ0 (v) = mV (v) in the relationship between response and the covariate Z whose value depends on the remaining covariates V . 8.2.1 Partial effect of a numeric covariate With the regression function (8.3), the partial effect of the Z covariate on the response is determined by a set of the non-intercept regression coefficients β Z := β1 , . . . , βk−1 . The null hypothesis H0 : β Z = 0k−1 then expresses the hypothesis that the covariate Z has, conditionally given a fixed (even though arbitrary) value of V , no effect on the response expectation. That is, it is a hypothesis of no partial effect of the covariate Z on the response expectation. With the spline-based regression function (8.4), the partial effect of the Z covariate is expressed by (all) spline-related regression coefficients β1 , . . . , βk . Nevertheless, due to the B-splines property (7.8), the null hypothesis of no partial effect of the Z covariate is now H0 : β 1 = · · · = β k . 8.3. ADDITIVITY OF THE EFFECT OF A CATEGORICAL COVARIATE 8.3 108 Additivity of the effect of a categorical covariate Assume that Z is a categorical covariate with Z = {1, . . . , G} where Z = g, g = 1, . . . , G, is repeated ng -times in the data which are assumed to be sorted according to the values of this covariate. The group means used in Section 7.4 must now be understood as conditional group means, given a value of the covariates V , and the regression function (8.1) parameterizes those conditional group means, i.e., for v ∈ V: m(1, v) = E Y Z = 1, V = v =: m1 (v), .. .. . . m(G, v) = E Y Z = G, V = v =: mG (v). (8.5) Let m(v) = m1 (v), . . . , mG (v) be a vector of those conditional group means. The matrix XZ can be any of the model matrices discussed in Section 7.4. If we restrict ourselves to the full-rank parameterizations introduced in Section 7.4.4, the matrix XZ is 1n1 ⊗ c> 1 .. , XZ = 1n , SZ , SZ = . > 1nG ⊗ cG where c1 , . . . , cG ∈ RG−1 are rows of a chosen (pseudo)contrast matrix c> 1 . .. . C= c> G If β = β0 , β1 , . . . , βG−1 denotes the regression coefficients related to the model matrix XZ = 1n , SZ and we further denote β Z = β1 , . . . , βG−1 , the conditional group means are, for v ∈ V, given as Z > Z m1 (v) = β0 + c> 1 β + mV (v) = γ0 (v) + c1 β , .. .. . . Z Z > mG (v) = β0 + cG β + mV (v) = γ0 (v) + c> Gβ , (8.6) where γ0 (v) = β0 + mV (v), v ∈ V. In a matrix notation, (8.6) becomes m(v) = γ0 (v) 1G + Cβ Z . 8.3.1 (8.7) Partial effects of a categorical covariate In agreement with a general expression (8.2), we have for arbitrary v ∈ V and arbitrary g1 , g2 ∈ Z: E Y Z = g1 , V = v − E Y Z = g2 , V = v = mg1 (v) − mg2 (v) > = cg1 − cg2 β Z , (8.8) 8.3. ADDITIVITY OF THE EFFECT OF A CATEGORICAL COVARIATE 109 which does not depend on a value of V = v. That is, the difference between the two conditional group means is the same for all values of the covariates in V . Terminology (Partial effects of a categorical covariate). If additivity of a categorical Z covariate and V covariates can be assumed, a vector of coefficients β Z from parameterization of the conditional group means (Eqs. 8.6, 8.7) will be referred to as partial effects of the categorical covariate. Note. It should be clear from (8.8) that interpretation of the partial effects of a categorical covariate depends on chosen parameterization (chosen (pseudo)contrast matrix C). If the Z covariate acts additively with the V covariate, it makes a sense to ask a question whether all G conditional group means are, for a given v ∈ V, equal. That is, whether all partial effects of the Z covariate are equal to zero. In general, this corresponds to the null hypothesis H0 : m1 (v) = · · · = mG (v), v ∈ V. (8.9) If the regression function is parameterized as (8.6), the null hypothesis (8.9) is expressed using the partial effects as H0 : β Z = 0G−1 . 8.3.2 Interpretation of the regression coefficients Note that (8.6) and (8.7) are basically the same expressions as those in (7.19) in Section 7.4.4. The only difference is dependence of the group means and the intercept term on the value of the covariates V . Hence interpretation of the individual coefficients β0 and β Z = β1 , . . . , βG−1 depends on the chosen pseudocontrast matrix C, nevertheless, it is basically the same as in case of a single categorical covariate in Section 7.4.4 with the only difference that (i) The non-intercept coefficients in β Z have the same interpretation as in Section 7.4.4 but always conditionally, given a chosen (even though arbitrary) value v ∈ V. (ii) The intercept β0 has interpretation given in Section 7.4.4 only for such v ∈ V for which mV (v) = 0. This follows from the fact that, again, for a chosen v ∈ V, the expression (8.6) of the conditional group means is the same as in Section 7.4.4. Nevertheless only for v such that mV (v) = 0, we have β0 = γ0 (v). Example 8.1 (Reference group pseudocontrasts). If C is the reference group pseudocontrasts matrix (7.22), we obtain analogously to (7.23), but now for a chosen v ∈ V, the following β0 + mV (v) = γ0 (v) = m1 (v), β1 = m2 (v) − m1 (v), .. . βG−1 = mG−1 (v) − m1 (v). 8.3. ADDITIVITY OF THE EFFECT OF A CATEGORICAL COVARIATE 110 Example 8.2 (Sum contrasts). If C is the sum contrasts matrix (7.24), we obtain analogously to (7.25), but now for a chosen v ∈ V, the following β0 + mV (v) = γ0 (v) = m(v), β1 = m1 (v) − m(v), .. . βG−1 = mG−1 (v) − m(v), where G m(v) = 1 X mg (v), G v ∈ V. g=1 If we additionally define αG = − PG−1 g=1 αG = − βg , we get, in agreement with (7.26), G−1 X g=1 βg = mG (v) − m(v). 8.4. EFFECT MODIFICATION AND INTERACTIONS 8.4 111 Effect modification and interactions 8.4.1 Effect modification Suppose now that the covariate vectors are Z1 , W1 , V 1 , . . . , Zn , Wn , V n ∈ Z ×W ×V, As usual, let Z1 . . Z= . , Zn Z ⊆ R, W ⊆ R, V ⊆ Rp−2 , W1 . . W= . , Wn p ≥ 2. V> 1 . . V= . V> n denote matrices collecting observed covariate values and finally, let Z, W, V ∈ Z × W × V denote a generic covariate. Suppose now that the regression function is m(z, w, v) = mZW (z, w) + mV (v), z, w, v ∈ Z × W × V, (8.10) where mV : V −→ R is some function and mZW : Z × W −→ R is a function that cannot be factorized as mZW (z, w) = mZ (z) + mW (w). We then have for any fixed v ∈ V. E Y Z = z + 1, W = w, V = v − E Y Z = z, W = w, V = v = mZW (z + 1, w) − mZW (z, w), (8.11) E Y Z = z, W = w + 1, V = v − E Y Z = z, W = w, V = v = mZW (z, w + 1) − mZW (z, w), (8.12) z ∈ Z, w ∈ W, where (8.11), i.e., the effect of a covariate Z on the response expectation possibly depends on the value of W = w and also (8.12), i.e., the effect of a covariate W on the response expectation possibly depends on the value of Z = z. We then say that the covariates Z and W are mutual effect modifiers.2 In a context of a linear model, both mZW and mV are chosen to be linear in unknown (regression) parameters and the corresponding model matrix is decomposed as X = XZW , XV , (8.13) where XZW corresponds to the regression function mZW and depends only on the covariate values Z and W and XV corresponds to the regression function mV and depends only on the covariate values V. In the rest of this section and in Sections 8.5, 8.6 and 8.7, we show classical choices for the matrix XZW based on so called interactions derived from covariate parameterizations that we introduced in Sections 7.3 and 7.4. End of Lecture #11 (05/11/2015) 8.4.2 Interactions Suppose that the covariate Z is parameterized using a parameterization Z k−1 sZ = sZ , 1 , . . . , sk−1 : Z −→ R 2 modifikátory efektu (8.14) Start of Lecture #13 (12/11/2015) 8.4. EFFECT MODIFICATION AND INTERACTIONS and the covariate W is parameterized using a parameterization W l−1 sW = sW , 1 , . . . , sl−1 : W −→ R 112 (8.15) and let SZ and SW be the corresponding reparameterizing matrices: > (W ) s> (Z ) s 1 1 Z. W . W = S 1W , . . . , S l−1 . .. = S 1Z , . . . , S k−1 .. SZ = , S = Z W s> s> Z (Zn ) W (Wn ) Definition 8.2 Interaction terms. Let Z1 , W1 , . . . , Zn , Wn ∈ Z × W ⊆ R2 be values of two covariates being parameterized using the reparameterizing matrices SZ and SW . By interaction terms3 based on the reparameterizing matrices SZ and SW we mean columns of a matrix SZW := SZ : SW . Note. See Definition A.5 for a definition of the columnwise product of two matrices. We have SZW = SZ : SW = k−1 : S l−1 S 1Z : S 1W , . . . , S k−1 : S 1W , . . . , S 1Z : S l−1 W Z W , . . . , SZ > s> W (W1 ) ⊗ sZ (Z1 ) .. . = . > (Z ) s> (W ) ⊗ s n n W Z 8.4.3 Interactions with the regression spline Either the Z covariate or/and the W covariate can also be parameterized by the regression splines. In that case, the interaction terms are defined in the same way as in Definition 8.2. For example, if the Z covariate is parameterized by the regression splines B Z = B1Z , . . . , BkZ with the related model matrix B> (Z ) 1 Z .. = B 1Z , . . . , B kZ , BZ = . B> Z (Zn ) and the W covariates by the parameterization (8.15) and the reparameterizing matrix SW as above, we will mean by the interaction terms columns of a matrix l−1 k BZW = BZ : SW = B 1Z : S 1W , . . . , B kZ : S 1W , . . . , B 1Z : S l−1 W , . . . , BZ : SW > s> (W ) ⊗ B (Z ) 1 1 Z W .. . = . > > sW (Wn ) ⊗ B Z (Zn ) 3 interakční členy 8.4. EFFECT MODIFICATION AND INTERACTIONS 8.4.4 113 Linear model with interactions Interaction terms are used in a linear model to express a certain form of the effect modification. If 1n ∈ / SZ and 1n ∈ / SW , the matrix XZW from (8.13) is usually chosen as XZW = 1n , SZ , SW , SZW , (8.16) which, as will be shown, corresponds to a certain form of the effect modification. Let the related regression coefficients be denoted as W ZW ZW ZW ZW Z , β1,1 , . . . , βk−1,1 , . . . , β1,l−1 , . . . , βk−1, β = β0 , β1Z , . . . , βk−1 , β1W , . . . , βl−1 l−1 . {z } | {z } | | {z } Z W ZW =: β =: β =: β Main and interaction effects Terminology (Main and interaction effects). Coefficients in β Z and β W are called the main effects4 of the covariate Z and W , respectively. Coefficients in β ZW are called the interaction effects.5 The related regression function (8.10) is Z > W ZW m(z, w, v) = β0 + s> + s> + mV (v), Z (z)β + sW (w)β ZW (z, w)β z, w, v ∈ Z × W × V, (8.17) where sZW (z, w) : Z × W −→ R, sZW (z, w) = sW (w) ⊗ sZ (z) W Z W Z W W Z = sZ 1 (z)s1 (w), . . . , sk−1 (z)s1 (w), . . . , s1 (z)sl−1 1(w), . . . , sk−1 (z)sl−1 (w) , z, w ∈ Z × W. The effects of the covariates Z or W , given the remaining covariates are then expressed as E Y Z = z + 1, W = w, V = v − E Y Z = z, W = w, V = v > > = sZ (z + 1) − sZ (z) β Z + sZW (z + 1, w) − sZW (z, w) β ZW , (8.18) E Y Z = z, W = w + 1, V = v − E Y Z = z, W = w, V = v > > = sW (w + 1) − sW (w) β W + sZW (z, w + 1) − sZW (z, w) β ZW , (8.19) z ∈ Z, w ∈ W. That is, the effect (8.18) of the covariate Z is determined by the main effects β Z of this covariate as well as by the interaction effects β ZW . Analogously, the effect (8.19) of the covariate W is determined by its main effects β W as well as by the interaction effects β ZW . 4 hlavní efekty 5 interakční efekty 8.4. EFFECT MODIFICATION AND INTERACTIONS 114 Hypothesis of no effect modification If factor mZW (z, w) of the regression function (8.10) is parameterized by matrix XZW given by (8.16) then the hypothesis of no effect modification is expressed by considering a submodel in which matrix XZW is replaced by a matrix XZ+W = 1n , SZ , SW . Interaction model with regression splines Suppose that matrix SZ = BZ where BZ corresponds to the parameterization of the Z covariate Z W using the regression splines. We then have 1n ∈ M B and also M S ⊆ M BZW for BZW = BZ : SW (see Section 8.5.2). That is, M 1n , BZ , SW , BZW = M BZ , BZW . It is thus sufficient (with respect to obtained regression space) to choose the matrix XZW as XZW = BZ , BZW . (8.20) Hypothesis of no effect modification then corresponds to a submodel in which matrix (8.20) is replaced by a matrix XZ+W = BZ , SW . 8.4.5 Rank of the interaction model Lemma 8.1 Rank of the interaction model. (i) Let rank SZ , SW = k + l − 2, i.e., all columns from the matrices SZ and SW are linearly independent and the matrix SZ , SW is of full-rank. Then the matrix SZW = SZ : SW is of full-rank as well, i.e., rank SZW = (k − 1) (l − 1). (ii) Let additionally 1n ∈ / M SZ , 1n ∈ / M SW . Then also a matrix XZW = 1n , SZ , SW , SZW is of full-rank, i.e., rank XZW = 1 + (k − 1) + (l − 1) + (k − 1) (l − 1) = k l. Proof. Left as exercise in linear algebra. Proof/calculations were skipped and are not requested for the exam. k 8.4. EFFECT MODIFICATION AND INTERACTIONS 115 Note (Hypothesis of no effect modification). Under the conditions of Lemma 8.1, we have for XZW = 1n , SZ , SW , SZW 1n , SZ , SW : rank XZW = k l, and XZ+W = rank XZ+W = 1 + (k − 1) + (l − 1) = k + l − 1, M XZ+W ⊂ M XZW . If the hypothesis of no effect modification is tested by a submodel F-test then its numerator degrees of freedom are kl − k − l + 1 = (k − 1) · (l − 1). The corresponding null hypothesis can also be specified as a hypothesis on the zero value of an estimable vector of all interaction effects: H0 : β ZW = 0(k−1)·(l−1) . 8.5. INTERACTION OF TWO NUMERIC COVARIATES 8.5 116 Interaction of two numeric covariates Let us now consider a situation when both Z and W are numeric covariates with Z ⊆ R, W ⊆ R. 8.5.1 Mutual effect modification Linear effect modification Suppose first that SZ is a reparameterizing matrix that corresponds to a simple identity transformation of the covariate Z. For the second covariate, W , assume that the matrix SW is an n × (l − 1) reparameterizing matrix that corresponds to the general parameterization (8.14), e.g., any of reparameterizing matrices discussed in Sections 7.3.1, 7.3.2 and 7.3.3. That is, z ∈ Z, W sW (w) = sW 1 (w), . . . , sl−1 (w) , w ∈ W, sZ (z) = z, Z1 . . SZ = . , Zn SW W (W ) . . . sW (W ) s> (W ) s 1 1 1 1 k−1 W . .. .. .. = , .. = . . . W (W ) . . . sW (W ) s> (W ) s n n n 1 W k−1 The matrix XZW , Eq. (8.16) is then W W Z1 s W 1 Z1 sW 1 (W1 ) . . . Z1 sl−1 (W1 ) 1 (W1 ) . . . sl−1 (W1 ) .. .. .. .. .. .. .. XZW = ... . . . . . . . W W W W Zn s1 (Wn ) . . . Zn sl−1 (Wn ) s1 (Wn ) . . . sl−1 (Wn ) 1 Zn {z } | {z } |{z} | {z } | 1n SZ SW SZW (8.21) and the related regression coefficients are W ZW β = β0 , β Z , β1W , . . . , βl−1 , β1ZW , . . . , βl−1 . | {z } | {z } βW β ZW The regression function (8.17) then becomes W W m(z, w, v) = β0 + β Z z + β1W sW 1 (w) + · · · + βl−1 sl−1 (w) ZW W + β1ZW z sW 1 (w) + · · · + βl−1 z sl−1 (w) + mV (v) W ZW = β0 + s> + mV (v) + (β Z + s> )z W (w)β W (w)β | {z } | {z } =: γ1Z (w) =: γ0Z (w, v) (8.22) (8.23) 8.5. INTERACTION OF TWO NUMERIC COVARIATES 117 = β0 + β Z z + mV (v) | {z } W =: γ0 (z, v) W ZW (w) + · · · + (βl−1 + βl−1 z) sW + (β1W + β1ZW z) sW (w), {z } 1 | | {z } l−1 W =: γ1W (z) =: γl−1 (z) (8.24) z, w, v ∈ Z × W × V. The regression function (8.22) can be interpreted twofold. (i) Expression (8.23) shows that for any fixed w ∈ W, the covariates Z and V act additively and the effect of Z on the response expectation is expressed by a line. Nevertheless, both intercept γ0Z and the slope γ1Z of this line depend on w and this dependence is described by the parameterization sW . The intercept is further additively modified by a factor mV (v). With respect to interpretation, this shows that the main effect β Z has an interpretation of the slope of the line that for a given V = v describes the influence of Z on the response if W = w is such that sW (w) = 0l−1 . This also shows that a test of the null hypothesis H0 : β Z = 0 does not evaluate a statistical significance of the influence of the covariate Z on the response expectation. It only evaluates it for values of W = w for which sW (w) = 0l−1 . (ii) Analogously, expression (8.24) shows that for any fixed z ∈ Z, the covariates W and V act additively and the effect of W on the response expectation is expressed by its parameterizaW ) depend in a linear way tion sW . Nevertheless, the related coefficients (γ0W , γ1W , . . . , γl−1 W on z. The intercept term is γ0 further additively modified by a factor mV (v). With respect to interpretation, this shows that the main effects β W has an interpretation of the coefficients of the influence of the W covariate on the response if Z = 0. This also shows that a test of the null hypothesis H0 : β W = 0l−1 does not evaluate a statistical significance of the influence of the covariate W on the response expectation. It only evaluates it under the condition of Z = 0. More complex effect modification More complex effect modifications can be obtained by choosing a more complex reparameterizing matrix SZ for the Z covariate. Interpretation of such a model is then straightforward generalization of the above situation. 8.5.2 Mutual effect modification with regression splines Linear effect modification with regression splines Let us again assume that the covariate Z is parameterized using a simple identity transformation and the reparameterizing matrix SZ is given as in (8.21). Nevertheless, for the covariate W , let us assume its parameterization using the regression splines B W = B1W , . . . , BlW 8.5. INTERACTION OF TWO NUMERIC COVARIATES 118 with the related model matrix BW B> B1W (W1 ) . . . BlW (W1 ) W (W1 ) .. .. .. .. = . = . . . . > W W B W (Wn ) B1 (Wn ) . . . Bl (Wn ) Analogously to previous usage of a matrix SZW , let a matrix BZW be defined as Z1 B1W (W1 ) . . . Z1 BkW (W1 ) .. .. .. . BZW = SZ : BW = . . . W W Zn B1 (Wn ) . . . Zn Bk (Wn ) P Remember that for any w ∈ W, lj=1 BjW (w) = 1 from which it follows that (i) 1n ∈ BW ; (ii) M SZ ⊆ M BZW . That is, M 1n , SZ , BW , BZW = M BW , BZW . (8.25) Hence if a full-rank linear model is to be obtained, where interaction between a covariate parameterized using the regression splines and a covariate parameterized using the reparameterizing > matrix SZ = Z1 , . . . , Zn is included, the model matrix XZW must be of the form Z1 B1W (W1 ) . . . Z1 BlW (W1 ) B1W (W1 ) . . . BlW (W1 ) .. .. .. .. .. .. XZW = BW , BZW = . . . . . . . Zn B1W (Wn ) . . . Zn BlW (Wn ) B1W (Wn ) . . . BlW (Wn ) {z } | {z } | W ZW B B If we denote the related regression coefficients as β = β1W , . . . , βlW , β1ZW , . . . , βlZW , | {z } | {z } W ZW =: β =: β the regression function (8.17) becomes m(z, w, v) = β1W B1W (w) + · · · + βlW BlW (w) + β1ZW z B1W (w) + · · · + βlZW z BlW (w) + mV (v) (8.26) W = B> + mV (v) + B > (w)β ZW z W (w)β | W {z } | {z } Z Z =: γ1 (w) =: γ0 (w, v) (8.27) = (β1W + β1ZW z) B1W (w) + · · · + (βlW + βlZW z) BlW (w) + mV (v), | {z } | {z } W W =: γ1 (z) =: γl (z) (8.28) z, w, v ∈ Z × W × V. The regression function (8.26) can again be interpreted twofold. 8.5. INTERACTION OF TWO NUMERIC COVARIATES 119 (i) Expression (8.27) shows that for any fixed w ∈ W, the covariates Z and V act additively and the effect of W on the response expectation is expressed by a line. Nevertheless, both intercept γ0Z and the slope γ1Z of this line depend on w and this dependence is described by the regression splines B W . The intercept is further additively modified by the factor mV (v). (ii) Analogously, expression (8.28) shows that for any fixed z ∈ Z, the covariates W and V act additively and the effect of W on the response expectation is expressed by the regression splines B W . Nevertheless, related spline coefficients (γ1W , . . . , γlW ) depend in a linear way on z. With respect to interpretation, this shows that the main effects β W has an interpretation of the coefficients of the influence of the W covariate on the response if Z = 0. More complex effect modification with regression splines Also with regression splines, a more complex reparameterizing matrix SZ can be chosen. The property (8.25) still holds and the matrix XZW can still be chosen as BW , BZW , BZW = SZ : BW . Interpretation of the model is again a straightforward generalization of the above situation. With respect to the rank of the resulting model, analogous statements hold as those given in Z Z Lemma 8.1. Suppose that the matrix S has k − 1 columns. If rank S , BW = k − 1 + l, i.e., Z , BW is all columns from the matrices SZ and BW are linearly independent and the matrix S of full-rank, then both BZW = SZ : BW and XZW = BW , BZW are of full-rank, i.e., rank BZW = (k − 1) l, rank XZW = l + (k − 1)l = k l. 8.6. INTERACTION OF A CATEGORICAL AND A NUMERIC COVARIATE 8.6 120 Interaction of a categorical and a numeric covariate Consider now a situation when Z is a categorical covariate with Z = {1, . . . , G} where Z = g, g = 1, . . . , G, is repeated ng -times in the data and W is a numeric covariate with W ⊆ R. We will assume (without loss of generality) that data are sorted according to the values of the categorical covariate Z. For the clarity of notation, we will analogously to Section 7.4 use also a double subscript to number the individual observations where the first subscript will indicate the value of the covariate Z. That is, we will use Z1 Z1,1 1 W1 W1,1 . .. .. .. .. .. . . . . Z Z 1 W W n 1,n n 1,n 1 1 1 1 − − −− − − − −− − − −− − − − . .. .. . . . .. .. . . = = .. , = − − −− − − − −− − − −− − − − Z W n−nG +1 ZG,1 G n−nG +1 WG,1 . .. .. .. .. . . . . . . Zn ZG,nG G Wn WG,nG If the categorical covariate Z can be interpreted as a label that indicates pertinence to one of G groups, the regression function (8.17) in which the value of z is fixed at z = g, g = 1, . . . , G, can be viewed as a regression function that parameterizes dependence of the response expectation on the numeric covariate W and possibly other covariates V in group g. We have for w ∈ W, v ∈ V: =: m1 (w, v), m(1, w, v) = E Y Z = 1, W = w, V = v .. .. . . m(G, w, v) = E Y Z = G, W = w, V = v =: mG (w, v). (8.29) Functions m1 , . . . , mG are then conditional (given a value of Z) regression functions describing dependence of the response expectation on the covariates W and V . Alternatively, for fixed w ∈ W and v ∈ V, a vector m(w, v) = m1 (w, v), . . . , mG (w, v) can be interpreted as a vector of conditional (given W and V ) group means. In the following assume that the categorical covariate Z is parameterized by the mean of a chosen (pseudo)contrast matrix c = c , . . . , c , 1 1,1 1,G−1 c> 1n1 ⊗ c> 1 1 . .. Z .. . .. , C= that is, S = . . > c> 1 ⊗ c nG cG = cG,1 , . . . , cG,G−1 , G G 8.6.1 Categorical effect modification First suppose that the numeric covariate W is parameterized using a parameterization W l−1 sW = sW , 1 , . . . , sl−1 : W −→ R 8.6. INTERACTION OF A CATEGORICAL AND A NUMERIC COVARIATE 121 W SW is the corresponding n × (l − 1) reparameterizing matrix. Let SW 1 , . . . , SG be the blocks of W the reparameterizing matrix S that correspond to datapoints with Z = 1, . . . , Z = G. That is, for g = 1, . . . , G, matrix SW g is an ng × (l − 1) matrix, s> sW . . . sW SW 1 (Wg,1 ) 1 W (Wg,1 ) l−1 (Wg,1 ) .. .. .. .. .. W W Sg = = and S = . . . . . . W s> sW SW 1 (Wg,ng ) . . . sl−1 (Wg,ng ) W (Wg,ng ) G The matrix XZW that parameterizes the term mZW (z, w) in the regression function (8.10) is again XZW = 1n , SZ , SW , SZW , where the interaction matrix SZW = SZ : SW is an n × (G − 1)(l − 1) matrix: SZW = ... c1,1 sW ... c1,G−1 sW c1,1 sW ... c1,G−1 sW 1 (W1,1 ) 1 (W1,1 ) l−1 (W1,1 ) l−1 (W1,1 ) .. .. .. .. .. .. .. . . . . . . . W W W ) ) . . . c s (W c1,1 sW (W ) . . . c s . . . c s (W 1,G−1 1,n 1,n 1,G−1 1,1 1,n 1 1 l−1 (W1,n1 ) l−1 1 1 1 − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −− .. .. .. .. .. .. .. . . . . . . . − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −− . . . cG,1 sW . . . cG,G−1 sW cG,1 sW . . . cG,G−1 sW 1 (WG,1 ) 1 (WG,1 ) l−1 (WG,1 ) l−1 (WG,1 ) .. .. .. .. .. .. .. . . . . . . . cG,1 sW . . . cG,G−1 sW . . . cG,1 sW . . . cG,G−1 sW 1 (WG,nG ) 1 (WG,nG ) l−1 (WG,nG ) l−1 (WG,n1 ) > > sW . . . sW 1 (W1,1 ) c1 l−1 (W1,1 ) c1 .. .. .. . . . W > s1 (W1,n1 ) c> . . . sW 1 l−1 (W1,n1 ) c1 − − − − − − − − − − − − − − −− .. .. .. = . . . − − − − − − − − − − − − − − −− sW (W ) c> . . . sW (W ) c> 1 G,1 G,1 G G l−1 .. .. .. . . . W W > s1 (WG,nG ) cG . . . sl−1 (WG,nG ) c> G SW 1 = SW G ⊗ c> 1 .. . . > ⊗ cG The regression coefficients related to the model matrix XZW are now Z W ZW ZW ZW ZW β = β0 , β1Z , . . . , βG−1 , β1W , . . . , βl−1 , β1,1 , . . . , βG−1,1 , . . . , β1,l−1 , . . . , βG−1,l−1 . | {z } | {z } | {z } =: β Z =: β W =: β ZW For following considerations, it will be useful to denote subvectors of the interaction effects β ZW as ZW ZW ZW ZW β ZW = β1,1 , . . . , βG−1,1 , . . . , β1,l−1 , . . . , βG−1,l−1 . | {z } | {z } =: β ZW =: β ZW •1 •l−1 The values of the regression function (8.17) for z = g, g = 1, . . . , G, w ∈ W and v ∈ V , i.e., the values of the conditional regression functions (8.29) can then be written as Z W W mg (w, v) = β0 + c> + β1W sW gβ 1 (w) + · · · + βl−1 sl−1 (w) > ZW W > ZW + sW 1 (w) cg β •1 + · · · + sl−1 (w) cg β •l−1 + mV (v). (8.30) 8.6. INTERACTION OF A CATEGORICAL AND A NUMERIC COVARIATE 122 A useful interpretation of the regression function (8.30) is obtained if we view mg (w, v) as a conditional (given Z = g) regression function that describes dependence of the response expectation on the covariates W and V in group g and write it as Z mg (w, v) = β0 + c> g β + mV (v) | {z } W =: γg,0 (v) W W W ZW ZW s1 (w) + · · · + βl−1 + c> + β1W + c> g β •l−1 sl−1 (w). (8.31) g β •1 | | {z } {z } W W =: γg,1 =: γg,l−1 Expression (8.31) shows that a linear model with an interaction between a numeric covariate parameterized using a parameterization sW and a categorical covariate can be interpreted such that for any fixed Z = g, the covariates W and V act additively and the effect of W on the response expectation is expressed by its parameterization sW . Nevertheless, the related coefficients depend on a value of a categorical covariate Z. The intercept term is further additively modified by a factor mV (v). In other words, if the categorical covariate Z expresses pertinence of a subject/experimental unit into one of G groups (internally labeled by numbers 1, . . . , G), the regression function (8.31) of the interaction model parameterizes a situation when, given the remaining covariates V , dependence of the response expectation on the numeric covariate W can be in each of the G groups expressed by the same linear model (parameterized by the parameterization sW ), nevertheless, the regression coefficients of the G linear models may differ. It follows from (8.31) that given Z = g (and given V = v), the regression coefficients for the dependence of the response on the numeric covariate W expressed by the parameterization sW are W Z γg,0 (v) = β0 + c> + mV (v), gβ W γg,j = βjW + ZW c> g β •j , (8.32) j = 1, . . . , l − 1. (8.33) Chosen (pseudo)contrasts that parameterize a categorical covariate Z then determine interpretation of the intercept β0 , both sets of main effect β Z and β W and also the interaction effects β ZW . This interpretation is now a straightforward generalization of derivations shown earlier in Sections 7.4.4 and 8.3. • Interpretation of the intercept term β0 and the main effects β Z of the categorical covariate Z is obtained by noting correspondence between the expression of the group specific intercepts W (v), . . . , γ W (v) given by (8.32) and the conditional group means (8.6) in Section 8.3. γ1,0 G,0 • Analogously, interpretation of the main effects β W and the interaction effects β ZW is obtained W , . . . , γ W given by by noting that for each j = 1, . . . , l − 1, the group specific “slopes” γ1,j G,j (8.33) play a role of the group specific means (7.19) in Section 7.4.4. Example 8.3 (Reference group pseudocontrasts). Suppose that C is the reference group pseudocontrast matrix (7.22). While viewing the group specific intercepts (8.32) as the conditional (given V = v) group means (8.6), we obtain, analogously to Example 8.1, the following interpretation of the intercept term β0 and the main effects Z β Z = β1Z , . . . , βG−1 of the categorical covariate: W (v), β0 + mV (v) = γ1,0 β1Z W (v) − γ W (v), = γ2,0 1,0 .. . Z W W (v). βG−1 = γG−1,0 (v) − γ1,0 8.6. INTERACTION OF A CATEGORICAL AND A NUMERIC COVARIATE 123 W , . . . , γ W given by (8.33) are viewed If for given j = 1, . . . , l − 1, the group specific “slopes” γ1,j G,j as the group specific means (7.19) in Section 7.4.4, interpretation of the jth main effect βjW of the ZW ZW numeric covariate and the jth set of the interaction effects β ZW •j = β1,j , . . . , βG−1,j is analogous to expressions (7.23): W, βjW = γ1,j ZW β1,j ZW βG−1,j W − γW , = γ2,j 1,j .. . W W. = γG−1,j − γ1,j Example 8.4 (Sum contrasts). Suppose now that C is the sum contrast matrix (7.24). Again, while viewing the group specific intercepts (8.32) as the conditional (given V = v) group means (8.6), we obtain, now analogously to Example 8.2, Z the following interpretation of the intercept term β0 and the main effects β Z = β1Z , . . . , βG−1 of the categorical covariate: β0 + mV (v) = γ0 W (v), W (v) − γ W (v), = γ1,0 0 .. . β1Z W Z (v) − γ0 W (v), = γG−1,0 βG−1 where G γ0 W (v) = 1 X W γg,0 (v), G v ∈ V. g=1 W , . . . , γ W given by (8.33) are viewed If for given j = 1, . . . , l − 1, the group specific “slopes” γ1,j G,j as the group specific means (7.19) in Section 7.4.4, interpretation of the jth main effect βjW of the ZW ZW numeric covariate and the jth set of the interaction effects β ZW •j = β1,j , . . . , βG−1,1 is analogous to expression (7.26): βjW = γj W , ZW β1,j ZW βG−1,j W − γ W, = γ1,j j .. . W = γG−1,j − γj W , where G γj W = 1 X W γg,j . G g=1 Alternative, interpretation of the regression function (8.30) is obtained if for a fixed w ∈ W and v ∈ V , the values of mg (w, v), g = 1, . . . , G, are viewed as conditional (given W and V ) group means. Expression (8.30) can then be rewritten as Z W W ZW W ZW mg (w, v) = β0 + s> + mV (v) + c> W (w)β g β + s1 (w)β •1 + · · · + sl−1 (w)β •l−1 . | | {z } {z } Z Z? =: γ0 (w, v) γ (w) (8.34) 8.6. INTERACTION OF A CATEGORICAL AND A NUMERIC COVARIATE 124 That is, the vector m(w, v) is parameterized as m(w, v) = γ0Z (w, v)1G + Cγ Z? (w). And the related coefficients γ0Z (w, v), γ Z? (w) depend on w by a linear model parameterized by the parameterization sW , the intercept term is further additively modified by mV (v). Expression (8.34) perhaps provide a way for the interpretation of the intercept term β0 and the main effects β Z . Nevertheless, attempts to use (8.34) for interpretation of the main effects β W and the interaction effects β ZW are usually quite awkward. 8.6.2 Categorical effect modification with regression splines Suppose now that the numeric covariate W is parameterized using the regression splines B W = B1W , . . . , BlW W with the related model matrix BW that we again factorize into blocks BW 1 , . . . , BG that correspond W to datapoints with Z = 1, . . . , Z = G. That is, for g = 1, . . . , G, matrix Bg is an ng × l matrix, B> B1W (Wg,1 ) . . . BlW (Wg,1 ) BW W (Wg,1 ) 1 . .. .. .. .. = . BW and BW = . . . . g = . . B> B1W (Wg,ng ) . . . BlW (Wg,ng ) BW W (Wg,ng ) G Let BZW = SZ : BW which is an n × (G − 1)l matrix: BZW = ... c1,1 BlW (W1,1 ) ... c1,G−1 BlW (W1,1 ) c1,1 B1W (W1,1 ) ... c1,G−1 B1W (W1,1 ) .. .. .. .. .. .. .. . . . . . . . c1,1 B1W (W1,n1 ) . . . c1,G−1 B1W (W1,n1 ) . . . c1,1 BlW (W1,n1 ) . . . c1,G−1 BlW (W1,n1 ) − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −− .. .. .. .. .. .. .. . . . . . . . − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − − −− cG,1 B1W (WG,1 ) . . . cG,G−1 B1W (WG,1 ) . . . cG,1 BlW (WG,1 ) . . . cG,G−1 BlW (WG,1 ) .. .. .. .. .. .. .. . . . . . . . cG,1 B1W (WG,nG ) . . . cG,G−1 B1W (WG,nG ) . . . cG,1 BlW (WG,nG ) . . . cG,G−1 BlW (WG,n1 ) B1W (W1,1 ) c> . . . BlW (W1,1 ) c> 1 1 .. .. .. . . . W W B1 (W1,n1 ) c> . . . Bl (W1,n1 ) c> 1 1 − − − − − − − − − − − − − − −− .. .. .. = . . . − − − − − − − − − − − − − − −− B W (W ) c> . . . B W (W ) c> G,1 G,1 1 G G l . . . .. .. .. W > B1W (WG,nG ) c> G . . . Bl (WG,nG ) cG As in Section 8.5.2, remember that for any w ∈ W, (i) 1n ∈ BW ; BW 1 = BW G Pl W j=1 Bj (w) ⊗ c> 1 .. . . > ⊗ cG = 1 from which it follows that 8.6. INTERACTION OF A CATEGORICAL AND A NUMERIC COVARIATE 125 (ii) M SZ ⊆ M BZW . Hence M 1n , SZ , BW , BZW = M BW , BZW and if a full-rank linear model is to be obtained that includes an interaction between a numeric covariate parameterized using the regression splines and a categorical covariate parameterized by the reparameterizing matrix SZ derived from a (pseudo)contrast matrix C, the model matrix XZW that parameterizes the term mZW (z, w) in the regression function (8.10) is W > BW 1 , B1 ⊗ c1 . .. . .. XZW = BW , BZW = . W ⊗ c> BW , B G G G The regression coefficients related to the model matrix XZW are ZW ZW ZW ZW , . . . , βG−1,1 , . . . , β1,k , . . . , βG−1,k . β = β1W , . . . , βlW , β1,1 {z } | | {z } | {z } =: β W =: β ZW =: β ZW •1 •k | {z } ZW =: β The value of the regression function (8.17) for z = g, g = 1, . . . , G, w ∈ W, and v ∈ V , i.e., the values of the conditional regression functions (8.29) can then be written as mg (w, v) = β1W B1W (w) + · · · + βlW BlW (w) > ZW ZW W + mV (v). + B1W (w) c> g β •1 + · · · + Bl (w) cg β •l Its useful interpretation is obtained if we write it as W W ZW ZW mg (w, v) = β1W + c> B1 (w) + · · · + βlW + c> Bl (w) + mV (v), g β •1 g β •l | | {z } {z } W W =: γg,1 =: γg,k which shows that the underlying linear model assumes that given Z = g, the covariates W and V act additively and the effect of the numeric covariate W on the response expectation is described W , . . . , γ W , however, depend on the value of the by the regression spline whose coefficients γg,1 g,k categorical covariate Z. Analogously to Section 8.6.1, interpretation of the regression coefficients β W and β ZW depends on chosen (pseudo)contrasts used to parameterize the categorical covariate Z. End of Lecture #13 (12/11/2015) 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 8.7 126 Interaction of two categorical covariates Start of Lecture #14 (19/11/2015) Finally consider a situation when both Z and W are categorical covariates with Z = {1, . . . , G}, W = {1, . . . , H}. Let a combination (Z, W ) = (g, h) be repeated ng,h -times in the data, g = 1, . . . , G, h = 1, . . . , H and assume that ng,h > 0 for each g and h. For the clarity of notation, we will now use also a triple subscript to index the individual observations. The first subscript will indicate a value of the covariate Z, the second subscript will indicate a value of the covariate W and the third subscript will consecutively number the observations with the same (Z, W ) combination. Finally, without loss of generality, we will assume that data are sorted primarily with respect to the value of the covariate W and secondarily with respect to the value of the covariate Z. That is, we assume and denote Z1 .. . .. . .. . .. . .. . Zn W1 .. . .. . .. . .. . .. . Wn = Z1,1,1 W1,1,1 .. .. . . W1,1,n1,1 Z1,1,n1,1 − − − − − − − − −− .. .. . . − − − − − − − − −− ZG,1,1 WG,1,1 .. .. . . ZG,1,nG,1 WG,1,nG,1 − − − − − − − − −− .. .. . . .. .. . . − − − − − − − − −− Z1,H,1 W1,H,1 .. .. . . Z1,H,n1,H W1,H,n1,H − − − − − − − − −− .. .. . . − − − − − − − − −− ZG,H,1 WG,H,1 .. .. . . ZG,H,nG,H WG,H,nG,H = 1 1 .. .. . . 1 1 −−− .. .. . . −−− G 1 .. .. . . G 1 −−− .. .. . . .. .. . . −−− 1 H .. .. . . 1 H −−− .. .. . . −−− G H .. .. . . G H , Y = Y1 .. . .. . .. . .. . .. . Yn = Y1,1,1 .. . Y1,1,n1,1 − − −− .. . − − −− YG,1,1 .. . YG,1,nG,1 − − −− .. . .. . − − −− Y1,H,1 .. . Y1,H,n1,H − − −− .. . − − −− YG,H,1 .. . YG,H,nG,H . In the same way, a triple subscript will also be used with the covariates V , i.e., V 1 , . . . , V n ≡ V 1,1,1 , . . . , V G,H,nG,H . Further, let ng• = H X ng,h , g = 1, . . . , G h=1 denote the number of datapoints with Z = g and similarly, let n•h = G X g=1 ng,h , h = 1, . . . , H 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 127 denote the number of datapoints with W = h. Analogously to Section 8.3, it will be useful to view, for a fixed v ∈ V, the values of the regression function (8.17) for z ∈ {1, . . . , G} and w ∈ {1, . . . , H} as a set of G · H conditional (given V ) group means: m(g, h, v) = E Y Z = g, W = h, V = v =: mg,h (v), g = 1, . . . , G, h = 1, . . . , H. (8.35) Let, for v ∈ V, m(v) be a vector with elements (8.35): m(v) = m1,1 (v), . . . , mG,1 (v), . . . . . . , m1,H (v), . . . , mG,H (v) . Note that under our assumptions concerning the factorization (8.10) of the regression function and an additive effect of the covariates V , we have mg,h (v) = mg,h + mV (v), g = 1, . . . , G, h = 1, . . . , H, (8.36) m(v) = m + mV (v), where mg,h := mZW (g, h) in (8.10) and m = m1,1 , . . . , mG,1 , . . . . . . , m1,H , . . . , mG,H . Due to the fact that m = m(v) if v ∈ V is such that mV (v) = 0, we will call in this section the vector m as a vector of baseline group means. Further, since they relate to division of the data according to values of two covariates (factors), we will also call them as two-way classified (baseline) group means. Notation. In the following, we will additionally use notation: G H 1 XX m := mg,h , G·H g=1 h=1 mg• := m•h := 8.7.1 1 H 1 G H X h=1 G X mg,h , g = 1, . . . , G, mg,h , h = 1, . . . , H. g=1 Linear model parameterization of two-way classified group means Let, as usual, µ denote a vector of the response expectations where again a triple subscript will be used, i.e., µ1,1,1 .. . µ = E Y Z, W, V = . µG,H,nG,H 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES Due to (8.36), this response expectation is factorized as m1,1 (V 1,1,1 ) mV (V 1,1,1 ) m1,1 . .. . .. .. . m1,1 (V 1,1,n1,1 ) mV (V 1,1,n1,1 ) m1,1 −−−−−−− −−−−−−− −−− .. .. .. . . . −−−−−−− −−−−−−− −−− m (V m (V m G,1 G,1,1 ) G,1,1 ) G,1 V .. .. .. . . . mG,1 (V G,1,nG,1 ) mV (V G,1,nG,1 ) mG,1 −−−−−−− −−−−−−− −−− . .. . .. .. . µ = = + .. .. .. . . . −−−−−−− −−−−−−− −−− m (V m (V m 1,H 1,H,1 ) 1,H,1 ) 1,H V . . . .. .. .. m1,H (V 1,H,n1,H ) mV (V 1,H,n1,H ) m1,H −−−−−−− −−−−−−− −−− . . .. .. .. . −−−−−−− −−−−−−− −−− mG,H (V G,H,1 ) mV (V G,H,1 ) mG,H . .. . .. .. . mG,H (V G,H,nG,H ) mV (V G,H,nG,H ) mG,H {z } {z | | =: µZW =: µV 128 . (8.37) } Note that a vector µZW from (8.37) corresponds to the factor mZW in the regression function (8.10). It is now our aim to parameterize µZW for use in a linear model, i.e., as µZW = XZW β, 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 129 where β is a vector of regression coefficients. Further note that only the baseline group means appear in the expression of µZW and we have m1,1 1n1,1 .. . mG,1 1nG,1 − − − − − . .. µZW = . . .. − − − − − m 1,H 1n1,H .. . mG,H 1nG,H The situation is basically the same as in case of a single categorical covariate in Section 7.4 if we view each of the G · H combinations of the Z and W covariates as one of the values of a new categorical covariate with G · H levels labeled by double indeces (1, 1), . . . , (G, H). The following facts then directly follow from Section 7.4 (given our assumption that ng,h > 0 for all (g, h)). • Matrix XZW must have a rank of G · H, i.e., at least k = G · H columns and its choice simplifies e ZW such that into selecting an (G · H) × k matrix X > x> 1 ⊗ x n 1,1 1,1 1,1 . .. .. . > > xG,1 1nG,1 ⊗ xG,1 − − − − − − − −− . .. .. . e ZW = X . , leading to XZW = . .. .. . − − − − − − − −− x> 1 > ⊗ x 1,H n1,H 1,H . .. . . . > x> 1 ⊗ x nG,H G,H G,H e ZW . • rank XZW = rank X e ZW parameterizes the baseline group means as • Matrix X mg,h = x> g,h β, g = 1, . . . , G, h = 1, . . . , H, e ZW β. m = X e can If purely parameterization of a vector m of the baseline group means is of interest, matrix X be chosen using the methods discussed in Section 7.4 applied to a combined categorical covariate with G · H levels. Nevertheless, it is often of interest to decompose in a certain sense the influence of the original covariates Z and W on the response expectation and this will be provided, as we shall show, by the interaction model built using common guidelines introduced in Section 8.4. 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 8.7.2 130 ANOVA parameterization of two-way classified group means Nevertheless, we first mention parameterization of two-way classified group means which only partly corresponds to common guidelines of Section 8.4, nevertheless, it can be encoutered in practice and it directly generalizes the ANOVA parameterization of one-way classified group means introduced in Section 7.4.3. The ANOVA parameterization of two-way classified group means is given as ZW mg,h = α0 + αgZ + αhW + αg,h , g = 1, . . . , G, h = 1, . . . , H, (8.38) where a vector of regression coefficients α = α0 , αZ , αW , αZW is composed of • intercept term α0 ; Z of the covariate Z; • main effects αZ = α1Z , . . . , αG W of the covariate W ; • main effects αW = α1W , . . . , αH ZW , . . . , αZW , . . . . . . , αZW , . . . , αZW . • interaction effects αZW = α1,1 G,1 1,H G,H Interaction effects and no effect modification The interaction effects αZW allow for possible mutual effect modification of the two categorical covariates Z and W . With parameterization (8.38) of the baseline group means, we have for any g1 , g2 ∈ {1, . . . , G}, h ∈ {1, . . . , H} and any v ∈ V : E Y Z = g1 , W = h, V = v − E Y Z = g2 , W = h, V = v = mg1 , h (v) − mg2 , h (v) = mg1 , h − mg2 , h = αgZ1 − αgZ2 . + αgZW − αgZW 2 ,h 1 ,h That is, if the value of the Z covariate is changed, the change of the response expectation possibly depends, through the interaction terms, on a value of the W covariate. Similarly, if we express a change in the response expectation due to a change in the second categorical covariate W . For any h1 , h2 ∈ {1, . . . , H}, g ∈ {1, . . . , G} and any v ∈ V : E Y Z = g, W = h1 , V = v − E Y Z = g, W = h2 , V = v = mg, h1 (v) − mg, h2 (v) = mg, h1 − mg, h2 = αhW1 − αhW2 ZW ZW + αg,h − αg,h . 1 2 Above expressions also show that hypothesis of no effect modification is given by equality of all interaction effects, i.e., ZW ZW H0 : α1,1 = · · · = αG,H . (8.39) 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 131 Linear model parameterization In a matrix form, parameterization (8.38) is m1,1 .. . mG,1 −−− IG . . . . . . 0G×G 1G IG 1G . . . . . . 0G . . . . . . .. . .. .. .. .. .. .. .. .. . . m= = . . . . .. . .. . . .. .. .. .. .. .. . . .. −−− IG G 0G×G . . . . . . | 1G IG 0G . . . . . . 1{z m 1,H e ZW X α .. . mG,H α0 αZ αW αZW } , e ZW is an (G · H) × (1 + G + H + G · H) matrix and its rank is hence at most where matrix X α e ZW provides less-than-full rank G · H (it is indeed precisely equal to G · H). That is, matrix X α e ZW can concisely be parameterization of the two-way classified group means. Note that matrix X α written as e ZW = 1H ⊗ 1G 1H ⊗ IG IH ⊗ 1G IH ⊗ IG (8.40) X α e ⊗ 1G D e ⊗C e , e D = 1H ⊗ 1G 1H ⊗ C (8.41) | {z } 1G·H where e = IG , C e = IH . D That is, we have e αZ + D e ⊗ 1G α W + D e ⊗C e αZW . m = α0 1G·H + 1H ⊗ C (8.42) Lemma 8.2 Column rank of a matrix that parameterizes two-way classified group means. e ZW being divided into blocks as Matrix X e ZW = 1H ⊗ 1G 1H ⊗ C e D e ⊗ 1G D e ⊗C e X e and 1H , D e . That is, has the column rank given by a product of column ranks of matrices 1G , C e ZW = col-rank 1G , C e e . col-rank X · col-rank 1H , D Proof. Proof/calculations below are shown only for those who are interested. 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 132 e ZW is upon suitable reordering of columns (which does not By point (x) of Theorem A.3, matrix X have any influence on the rank of the matrix) equal to a matrix ZW e e e ⊗ 1G , C e . Xreord = 1H ⊗ 1G , C D Further, by point (ix) of Theorem A.3: e ZW = 1H , D e ⊗ 1G , C e . X reord Finally, by point (xi) of Theorem A.3: e e . e ZW = col-rank X e ZW = col-rank 1G , C · col-rank 1 , D col-rank X H reord k e ZW given by (8.40) is indeed Lemma 8.2 can now be used to get easily that the rank of the matrix X α G · H and hence it can be used to parameterize G · H two-way classified group means. Sum constraints identification e ZW is 1 + G + H. By Scheffé’s theorem on identification in Deficiency in the rank of the matrix X α a linear model (Theorem 7.1), (1+G+H) (or more) linear constraints on the regression coefficients α are needed to identify the vector α in the related linear model. In practice, the following set of (2 + G + H) constraints is often used: G X H X αgZ = 0, g=1 H X ZW αg,h = 0, h=1 G X g = 1, . . . , G, αhW = 0, (8.43) ZW αg,h = 0, h = 1, . . . , H, g=1 h=1 which in matrix notation is written as Aα = 02+G+H , 0 1> G 0> H 0> 1> 0 G H 0G 0G×G 0G×H A= 0 0> 0> G H .. .. .. . . . > 0 0G 0> H > 0> G . . . 0G > 0> G . . . 0G IG . . . IG . > > 1G . . . 0G .. .. .. . . . > 0> G . . . 1G We leave it as an exercise in linear algebra to verify that an (2 + G + H) × (1 + G + H + G · H) matrix A satisfies conditions of Scheffé’s theorem, i.e., e ZW > = 0 . rank A = 1 + G + H, M A> ∩ M X α The coefficients α identified by a set of constraints (8.43) have the following (easy to see using simple algebra with expressions (8.38) while taking into account the constraints) useful interpretation. α0 = m, 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 133 αgZ = mg• − m, g = 1, . . . , G, αhW = m•h − m, h = 1, . . . , H, ZW αg,h = mg,h − mg• − m•h + m, g = 1, . . . , G, h = 1, . . . , H, from which it also follows αgZ1 − αgZ2 = mg1 • − mg2 • , g1 , g2 = 1, . . . , G, (8.44) αhW1 − αhW2 = m•h1 − m•h2 , h1 , h2 = 1, . . . , H. (8.45) That is, the difference between the two main effects of the Z covariate, Eq. (8.44) provides the mean effect of changing the Z covariate if the mean is taken over possible values of its effect modifier W . Analogously, the difference between the two main effects of the W covariate, Eq. (8.45) provides the mean effect of changing the W covariate if the mean is taken over possible values of its effect modifier Z. Both (8.44) and (8.45) are important quantities especially in the area of designed (industrial, agricultural, . . . ) experiments aiming in evaluating an effect of two factors on the response. 8.7.3 Full-rank parameterization of two-way classified group means Suppose that the categorical covariate Z is parameterized by the mean of a chosen G × (G − 1) (pseudo)contrast matrix c1 = c1,1 , . . . , c1,G−1 , c> 1 . .. . C= . , . > cG cG = cG,1 , . . . , cG,G−1 , and the categorical covariate W is parameterized by the mean of a chosen H × (H − 1) (pseudo)contrast matrix d = d , . . . , d , 1 1,1 1,H−1 d> 1 . .. . D= . , . > dH dH = dH,1 , . . . , dH,H−1 . Note that we do not require that matrices C and D are based on (pseudo)contrasts of the same type. Let e ZW = 1H ⊗ 1G 1H ⊗ C D ⊗ 1G D ⊗ C (8.46) X β = 1 .. . c> 1 d> 1 d> 1 ⊗ .. . c> 1 1 .. . .. . .. . c> G .. . .. . .. . d> 1 .. . .. . > d> 1 ⊗ cG .. . .. . 1 .. . c> 1 .. . d> H .. . > d> G ⊗ c1 .. . 1 c> G d> H > d> G ⊗ cG , (8.47) 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 134 which is a matrix with G · H rows and 1 + (G − 1) + (H − 1) + (G − 1)(H − 1) = G · H columns and its structure is the same as a structure of the matrix (8.41), where we now take e = C, C e = D. D Using Lemma 8.2 and properties of (pseudo)contrast matrices, we have e ZW = col-rank 1G , C col-rank X · col-rank 1 , D = G · H. H β e ZW is of full-rank G · H and hence can be used to parameterize the two-way That is, the matrix X β classified group means as e ZW β, m=X β = β0 , β Z , β W , β ZW , β where Z β Z = β1Z , · · · , βG−1 , W β W = β1W , · · · , βH−1 , ZW ZW ZW ZW β ZW = β1,1 , . . . , βG−1,1 , . . . . . . , β1,H−1 , . . . , βG−1,H−1 . We can also write m = β0 1G·H + 1H ⊗ C β Z + D ⊗ 1G β W + D ⊗ C β ZW , ZW > W Z , + d> mg,h = β0 + c> + d> gβ h ⊗ cg β hβ g = 1, . . . , G, h = 1, . . . , H. (8.48) Different choices of the (pseudo)contrast matrices C and D lead to different interpretations of the regression coefficients β. Link to general interaction model e ZW , it is directly seen that it can also be written as If we take expression (8.47) of the matrix X e ZW = 1G·H , e X SZ , e SW , e SZW , β where e SZ = 1H ⊗ C, e SW = D ⊗ 1G , e SZW = e SZ : e SW . Similarly, a matrix XZW which parameterizes a vector µZW (in which a value mg,h is repeated β ng,h -times) is factorized as XZW = 1n , SZ , SW , SZW , (8.49) β where SZ and SW is obtained from matrices e SZ and e SW , respectively, by appropriately repeating ZW Z W their rows and S = S : S . That is, the model matrix (8.49) is precisely of the form given in (8.16) that we used to parameterize a linear model with interactions. In context of this chapter, the (pseudo)contrast matrices C and D, respectively, play the role of parameterizations sZ in (8.14) and sW in (8.15), respectively. 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 8.7.4 135 Relationship between the full-rank and ANOVA parameterizations With the full-rank parameterization of the two-way classified group means, expression (8.48) shows that we can also write ZW mg,h = α0 + αgZ + αhW + αg,h , g = 1, . . . , G, h = 1, . . . , H. (8.50) where α0 := β0 , αgZ Z := c> gβ , g = 1, . . . , G, αhW W := d> hβ , h = 1, . . . , H, ZW αg,h := ZW > , d> h ⊗ cg β (8.51) g = 1, . . . , G, h = 1, . . . , H. That is, chosen full-rank parameterization of the two-way classified group means corresponds to the ANOVA parameterization (8.50) in which (1 + G + H + G · H) regression coefficients α is uniquely obtained from G·H coefficients β of the full-rank parameterization using the relationships (8.51). In other words, expressions (8.51) correspond to identifying constraints on α in the less-than-full-rank parameterization. Note also, that in matrix notation, (8.51) can be written as α0 := β0 , αZ := C β Z , αW := D β W , αZW (8.52) := (D ⊗ C) β ZW . Interaction effects and no effect modification ZW = · · · = αZW , Hypothesis of no effect modification for covariates Z and W is given as H0 : α1,1 G,H see (8.39), which can also be written as H0 : αZW = a 1G·H for some a ∈ R. ZW Taking into account (8.52), αZW = = a 1. Due a 1 for some a ∈ R, if and only if (D ⊗ C) β to the fact that 1 ∈ / M D ⊗ C (if both C and D are (pseudo)contrast matrices), this is only possible with a = 0 and β ZW = 0. Hence with the full-rank parameterization of the two-way classified group means, the hypothesis of no effect modification is H0 : β ZW = 0(G−1)(H−1) . 8.7.5 Additive model Suppose that the categorical covariates Z and W act additively. That is, it can be assumed that αZW = a 1 for some a ∈ R and hence the baseline group means can be parameterized as mg,h = α0 + αgZ + αhW , g = 1, . . . , G, h = 1, . . . , H, (8.53) Z is a vector of main effects of the covariate Z and αW = where, as before αZ = α1Z , . . . , αG W is a vector of main effects of the covariate W , and α1W , . . . , αH α = α0 , αZ , αW 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 136 is a vector of regression coefficients. Even though this situation was in fact treated in a more general way in Section 8.3 (covariate W in this section plays a role of covariate vector V in Section 8.3), it will be useful to revisit it in this specific case when both covariates at hand are categorical. Parameterization (8.53) written in a matrix notation is e Z+W α, m=X α e Z+W = X α 1H ⊗ 1G 1H ⊗ IG IH ⊗ 1G . e Z+W has 1 + G + H columns but its rank is only Matrix X α e Z+W = 1 + (G − 1) + (H − 1) = G + H − 1. rank X α Analogously to the interaction model and also analogously to Section 8.3, a full-rank parameterization of the two-way classified group means (8.53) under additivity assumption, is Z W mg,h = β0 + c> + d> gβ hβ , (8.54) g = 1, . . . , G, h = 1, . . . , H, where c1 > , . . . , c> G are rows of an G × (G − 1) (pseudo)contrast matrix C that parameterizes the categorical covariate Z and d1 > , . . . , d> H are rows of an H × (H − 1) (pseudo)contrast matrix D that parameterizes the categorical covariate W and Z W β = β0 , β1Z , . . . , βG−1 , β1W , . . . , βH−1 | {z } | {z } Z W β β are regression coefficients. Parameterization (8.54) written in a matrix form is (8.55) e Z+W β, m=X β e Z+W = X β 1H ⊗ 1G 1H ⊗ C D ⊗ 1G . e Z+W is a matrix with 1 + (G − 1) + (H − 1) = G + H − 1 columns obtained from Since X β e ZW , Eq. (8.46), by omitting its last (G − 1)(H − 1) columns, matrix X e Z+W is a full-rank matrix X β β indeed of a full-rank G + H − 1. Analogously to the interaction model, coefficients β from the full-rank parameterization correspond to certain identifying constraints imposed on the coefficients α of the less-than-full-rank parameterization. Their mutual relationship is basically the same as in Section 8.7.4 and is given by the first three expressions from (8.51). Partial effects If two-way classified baseline group means satisfy additivity, we have for any g1 , g2 ∈ {1, . . . , G}, h ∈ {1, . . . , H} and any v ∈ V : E Y Z = g1 , W = h, V = v − E Y Z = g2 , W = h, V = v = = mg1 , h (v) − mg2 , h (v) αgZ1 − αgZ2 = = mg1 , h − mg2 , h > Z c> g1 − cg2 β . In agreement with Section 8.3.1, main effects αZ or β Z are referred to as partial effects of Z depend on a categorical covariate Z. Remind that interpretation of particular values α1Z , . . . , αG 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 137 Z considered identifying constraints, interpretation of particular values β1Z , . . . , βG−1 depend on a choice of a (pseudo)contrast matrix C. Due to additivity, we also have that, still for arbitrary h ∈ {1, . . . , H}: mg1 , h − mg2 , h = mg1 • − mg2 • = αgZ1 − αgZ2 = Z > c> g1 − cg2 β . Interpretation of the invididual coefficients is then the same as already explained in Section 8.3.2. Similarly, we have for any h1 , h2 ∈ {1, . . . , H}, g ∈ {1, . . . , G} and any v ∈ V : E Y Z = g, W = h1 , V = v − E Y Z = g, W = h2 , V = v 8.7.6 = mg, h1 (v) − mg, h1 (v) = αhW1 − αhW2 = d> h1 mg, h2 − mg, h2 W − d> h2 β . = = m•h2 − m• h2 Interpretation of model parameters for selected choices of (pseudo)contrasts Chosen (pseudo)contrast matrices C and D determine interpretation of the coefficients β of the full-rank parameterization (8.48) of the two-way classified group means and also of the coefficients α given by (8.51) and determine an ANOVA parameterization with certain identification. For interpretation, it is useful to view the two-way classified group means as entries in the G × H matrix m1,1 . . . m1,H . .. . M= ... . . . mG,1 . . . mG,H Corresponding sample sizes ng,h , g = 1, . . . , G, h = 1, . . . , H, form a G × H contingency table based on the values of the two categorical covariates Z and W . With the ANOVA parameterization (8.50), the main effects αZ = Cβ Z and αW = Dβ W can be interpreted as the row and column effects in matrix M, respectively. The interaction effects αZW = D ⊗ C β ZW can be put into a matrix > > ZW . . . αZW > β ZW . . . > β ZW α1,1 d ⊗ c d ⊗ c 1 H 1 1 1,H . .. .. .. ZW . A = . ... . = . ... . > > ZW . . . αZW > β ZW . . . > β ZW αG,1 d ⊗ c d ⊗ c 1 H G,H G G whose entries could be interpreted as cell effects in matrix M. In other words, each of the values in M is obtained as a sum of the intercept term α0 and corresponding row, column and cell effects. As was already mentioned, the two (pseudo)contrast matrices C and D can be both of different type, e.g., matrix C being the reference group pseudocontrast matrix and matrix D being the sum contrast matrix. Nevertheless, in practice, both of them are mostly chosen as being of the same type. In the reminder of this section, we shall discuss interpretation of the model parameters for two most common choices of the (pseudo)contrasts which are (i) the reference group pseudocontrasts and (ii) sum contrasts. 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 138 Reference group pseudocontrasts Suppose that both C and D are reference group pseudocontrasts, i.e., > > 0 ... 0 c1 0 ... 0 d1 ! > 1 . . . 0 c2 1 . . . 0 d> 0> 2 G−1 C = . . = . = , D = . . = . = . . ... . . ... IG−1 .. .. .. .. 0 ... 1 c> 0 ... 1 d> H G We have, Z α1 0 Z Z α2 β . = αZ = Cβ Z = .1 , . . . . Z Z αG βG−1 W α1 0 W W α2 β . = αW = Dβ W = 1. , . . . . W W αH βH−1 0> H−1 IH−1 ! (8.56) To get the link between the full-rank interaction terms β ZW and their ANOVA counterparts αZW , > we have to explore the form of vectors d> h ⊗ cg , g = 1, . . . , G, h = 1, . . . , H. With the reference group pseudocontrasts, we easily see that > = 0, d> for all h = 1, . . . , H, h ⊗ c1 > > d1 ⊗ cg = 0, for all g = 1, . . . , G, > > dh ⊗ cg = 0, . . . , 1, . . . , 0 , if g 6= 1 & h 6= 1, ZW > ZW , 1 on a place that in dh ⊗ c> multiplies βg−1,h−1 g β which leads to A ZW ZW α1,1 ZW α2,1 = .. . ZW α1,2 ZW α2,2 .. . ZW αG,1 ZW αG,2 ZW . . . α1,H 0 ZW . . . α2,H 0 .. = .. ... . . ... ZW αG,H 0 0 ZW β1,1 .. . ZW βG−1,1 ... ... 0 ZW β1,H−1 .. . ... ZW . . . βG−1,H−1 . In summary, the ANOVA coefficients are identified by a set of 3 + (H − 1) + (G − 1) = G + H + 1 constraints α1Z = 0, α1W = 0, ZW α1,1 = 0, ZW α1,h = 0, h = 2, . . . , H, ZW αg,1 g = 2, . . . , H. = 0, The first two constraints come from (8.56), remaining ones correspond to zeros in the matrix AZW . ZW , g = 1, . . . , G, h = While taking into account parameterization mg,h = α0 + αgZ + αhW + αg,h 1, . . . , H, of the two-way classified group means where the parameters satisfy above constraints, we get their following interpretation: α0 = β 0 = m1,1 , Z αgZ = βg−1 = mg,1 − m1,1 , g = 2, . . . , G, . 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES W αhW = βh−1 = m1,h − m1,1 , 139 h = 2, . . . , H, ZW ZW αg,h = βg−1,h−1 = mg,h − mg,1 − m1,h + m1,1 , g = 2, . . . , G, h = 2, . . . , H. Note (Reference group pseudocontrasts in the additive model). If the additive model is assumed where mg,h = α0 + αgZ + αhW , g = 1, . . . , G, h = 1, . . . , H and the reference group pseudocontrasts are used in the full-rank parameterization (8.55), the ANOVA coefficients α = α0 , αZ , αW are obtained from the full-rank coefficients β = β0 , β Z , β W again by (8.56), that is, they are identified by two constraints α1Z = 0, α1W = 0. Their interpretation becomes α0 = β0 = m1,1 , Z αgZ = βg−1 = mg,h − m1,h , g = 2, . . . , G, arbitrary h ∈ {1, . . . , H} h = 2, . . . , H, arbitrary g ∈ {1, . . . , G} = mg• − m1• , W = mg,h − mg,1 , αhW = βh−1 = m•h − m•1 . Sum contrasts Suppose now that both C and D 1 . .. C= 0 −1 are sum contrasts, i.e., > ... 0 c1 . .. .. . . . = . = > ... 1 cG−1 . . . −1 c> G > 1 ... 0 d1 . . . . .. .. .. . = . = D= > 1 0 ... dH−1 −1 . . . −1 d> H IG−1 − 1> G−1 ! , We have, IH−1 − 1> H−1 β1Z α1Z .. . .. . = αZ = Cβ Z = , Z Z βG−1 αG−1 PG−1 Z Z αG − g=1 βg ! . α1W β1W .. . .. . = αW = Dβ W = , W W βH−1 αH−1 P W H−1 W αH − h=1 βh (8.57) 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES 140 > The form of the vectors d> h ⊗ cg , g = 1, . . . , G, h = 1, . . . , H, needed to calculate the interaction ZW terms αg,h is the following > = d> h ⊗ cg 0, . . . , 1, . . . , 0 , ZW > 1 on a place that in d> h ⊗ cg β for g = 1, . . . , G − 1, h = 1, . . . , H − 1, ZW multiplies βg,h , > d> h ⊗ cG = 0G−1 , . . . , − 1G−1 , . . . , 0G−1 , for all h = 1, . . . , H − 1, ZW > > ZW , multiply β•h − 1G−1 block on places that in dh ⊗ cG β > = d> H ⊗ cg 0, . . . , −1, . . . , 0, . . . . . . , 0, . . . , −1, . . . , 0 , for all g = 1, . . . , G − 1, ZW > > ZW −1’s on places that in dH ⊗ cg β multiply βg• , > d> H ⊗ cG = 1, . . . , 1 = 1(G−1)(H−1) . This leads to AZW ZW ZW ZW α1,1 ... α1,H−1 α1,H . .. .. .. .. . . . = ZW ZW ZW αG−1,1 . . . αG−1,H−1 αG−1,H ZW ZW ZW αG,H . . . αG,H−1 αG,1 ZW β1,1 .. . = ... ... ... ZW βG−1,1 − PG−1 g=1 ZW βg,1 ... − ZW β1,H−1 .. . ZW βG−1,H−1 PG−1 g=1 ZW βg,H−1 − − PH−1 h=1 .. . PH−1 h=1 ZW β1,h ZW βG−1,h PG−1 PH−1 g=1 h=1 ZW βg,h . Note that the entries in each row of the matrix AZW and also in each of its column sum up to zero. Similarly, (8.57) shows that the elements of the main effects αZ and also elements of the main effects αW sum up to zero. Identifying constraints for the ANOVA coefficients α that correspond to considered sum contrast full-rank parameterization are hence G X αgZ = 0, g=1 H X αhW = 0, h=1 H X (8.58) ZW αg,h = 0, for each g = 1, . . . , G, ZW αg,h = 0, for each h = 1, . . . , H. h=1 G X g=1 ZW , one constraint is redundant Note that in a set of G + H constraints on the interaction terms αg,h and the last two rows could also be replaced by a set of (G − 1) + (H − 1) + 1 = G + H − 1 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES constraints: H X 141 ZW αg,h = 0, for each g = 1, . . . , G − 1, ZW αg,h = 0, for each h = 1, . . . , H − 1, h=1 G X g=1 G−1 X H−1 X ZW ZW αg,h = αG,H . g=1 h=1 We see that the set of equations (8.58) exactly corresponds to identification by the sum constraints (8.43), see Section 8.7.2. Hence the interpretation of the regression coefficients is the same as derived there, namely, α0 = m, αgZ = mg• − m, g = 1, . . . , G, αhW = m•h − m, h = 1, . . . , H, ZW = mg,h − mg• − m•h + m, αg,h g = 1, . . . , G, h = 1, . . . , H. Additionally, αgZ1 − αgZ2 = mg1 • − mg2 • , g1 , g2 = 1, . . . , G, αhW1 − αhW2 = m•h1 − m•h2 , h1 , h2 = 1, . . . , H. Note (Sum contrasts in the additive model). If the additive model is assumed where mg,h = α0 + αgZ + αhW , g = 1, . . . , G, h = 1, . . . , H and the sum contrasts are used in the full-rank parameterization (8.55), the ANOVA coefficients α = α0 , αZ , αW are obtained from the full-rank coefficients β = β0 , β Z , β W again by (8.57), that is, they are identified by two constraints G X g=1 αgZ = 0, H X αhW = 0. h=1 Their interpretation becomes α0 = m, αgZ = mg,h − m•h , g = 1, . . . , G, arbitrary h ∈ {1, . . . , H} h = 1, . . . , H, arbitrary g ∈ {1, . . . , G} = mg• − m, αhW = mg,h − mg• , = m•h − m. Additionally, αgZ1 − αgZ2 = mg1 ,h − mg2 ,h , = mg1 • − mg2 • , g1 , g2 = 1, . . . , G, arbitrary h ∈ {1, . . . , H} 8.7. INTERACTION OF TWO CATEGORICAL COVARIATES αhW1 − αhW2 = mg,h1 − mg,h2 , h1 , h2 = 1, . . . , H, 142 arbitrary g ∈ {1, . . . , G} = m•h1 − m•h2 . End of Lecture #14 (19/11/2015) 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES 8.8 8.8.1 143 Hierarchically well-formulated models, ANOVA tables Start of Lecture #15 (19/11/2015) Model terms In majority of applications of a linear model, a particular covariate Z ∈ Z ⊆ R enters the regression function using one of the parameterizations described in Sections 7.3 and 7.4 or inside an interaction (see Defition 8.2) or inside a so called higher order interaction (will be defined in a while). As a summary, depending on whether the covariate is numeric or categorical, several parameterizations s were introduced in Sections 7.3 and 7.4 that with the covariate values Z1 , . . . , Zn in the data lead to a reparameterizing matrix s> (Z1 ) X> 1 . . . . S= . = . , > > s (Zn ) Xn where X 1 = s(Z1 ), . . ., X n = s(Zn ) are the regressors used in the linear model. The considered parameterizations were the following. Numeric covariate (i) Simple transformation: s = s : Z −→ R with s(Z1 ) X 1 = X1 = s(Z1 ), . .. S = .. = S , . s(Zn ) X n = Xn = s(Zn ). (8.59) (ii) Polynomial: s = s1 , . . . , sk−1 such that sj (z) = P j (z) is polynomial in z of degree j, j = 1, . . . , k − 1. This leads to P 1 (Z1 ) . . . P k−1 (Z1 ) . .. .. 1 k−1 , .. S= = (8.60) P , . . . , P . . P 1 (Zn ) . . . P k−1 (Zn ) X 1 = P 1 (Z1 ), . . . , P k−1 (Z1 ) , .. . X n = P 1 (Zn ), . . . , P k−1 (Zn ) . For a particular form of the basis polynomials P 1 , . . . , P k−1 , raw or orthonormal polynomials have been suggested in Sections 7.3.2 and 7.3.3. Other choices are possible as well. (iii) Regression spline: s = s1 , . . . , sk such that sj (z) = Bj (z), j = 1, . . . , k, where B1 , . . . , Bk is the spline basis of chosen degree d ∈ N0 composed of basis B-splines built above a set of chosen knots λ = λ1 , . . . , λk−d+1 . This leads to B1 (Z1 ) . . . Bk (Z1 ) . .. .. 1 k , .. S=B= = (8.61) B , ..., B . . B1 (Zn ) . . . Bk (Zn ) 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES X1 = .. . B1 (Z1 ), . . . , Bk (Z1 ) , Xn = B1 (Zn ), . . . , B k (Zn ) . 144 Categorical covariate with Z = 1, . . . , G . The parameterization s is s(z) = cz , z ∈ Z, where c1 , . . . , cG ∈ RG−1 are the rows of a chosen (pseudo)contrast matrix CG×G−1 . This leads to c> X 1 = cZ1 , Z. 1 .. 1 G−1 . S= (8.62) , . . = C , ..., C c> X n = cZn . Zn Main effect model terms In the following, we restrict ourselves only into situations when the considered covariates are parameterized by one of above mentioned ways. The following definitions define sets of the columns of a possible model matrix which will be called the model terms and which are useful to be always considered “together” when proposing a linear model for a problem at hand. Definition 8.3 The main effect model term. Depending on a chosen parameterization, the main effect model term6 (of order one) of a given covariate Z is defined as a matrix T with columns: Numeric covariate (i) Simple transformation: (the only) column S of the reparameterizing matrix S given by (8.59), i.e., T= S . (ii) Polynomial: the first column P 1 of the reparameterizing matrix S (given by Eq. 8.60) that corresponds to the linear transformation of the covariate Z, i.e., T = P1 . (iii) Regression spline: (all) columns B 1 , . . . , B k of the reparameterizing matrix S = B given by (8.61), i.e., T = B1, . . . , Bk . Categorical covariate: (all) columns C 1 , . . . , C G−1 of the reparameterizing matrix S given by (8.62), i.e., T = C 1 , . . . , C G−1 . Definition 8.4 The main effect model term of order j. If a numeric covariate Z is parameterized using the polynomial of degree k − 1 then the main effect model term of order j, j = 2, . . . , k − 1, means a matrix Tj whose the only column is the jth 6 hlavní efekt 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES 145 column P j of the reparameterizing matrix S (given by Eq. 8.60) that corresponds to the polynomial of degree j, i.e., Tj = P j . Note. The terms T, . . ., Tj−1 are called as lower order terms included in the term Tj . Two-way interaction model terms In the following, consider two covariates Z and W and their main effect model terms TZ and TW . Definition 8.5 The two-way interaction model term. The two-way interaction7 model term means a matrix TZW , where TZW := TZ : TW . Notes. • The main effect model term TZ and/or the main effect model term TW that enter the two-way interaction may also be of a degree j > 1. • Both the main effect model terms TZ and TW are called as lower order terms included in the two-way interaction term TZ : TW . Higher order interaction model terms In the following, consider three covariates Z, W and V and their main effect model terms TZ , TW , TV . Definition 8.6 The three-way interaction model term. The three-way interaction8 model term means a matrix TZW V , where TZW V := TZ : TW : TV . Notes. • Any of the main effect model terms TZ , TW , TV that enter the three-way interaction may also be of a degree j > 1. • All main effect terms TZ , TW and TV and also all two-way interaction terms TZ : TW , TZ : TV and TW : TV are called as lower order terms included in the three-way interaction term TZW V . • By induction, we could define also four-way, five-way, . . . , i.e., higher order interaction model terms and a notion of corresponding lower order nested terms. 7 dvojná interakce 8 trojná interakce 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES 8.8.2 146 Model formula To write concisely linear models based on several covariates, the model formula will be used. The following symbols in the model formula have the following meaning: • 1: intercept term in the model if this is the only term in the model (i.e., intercept only model). • Letter or abbreviation: main effect of order one of a particular covariate (which is identified by the letter or abbreviation). It is assumed that chosen parameterization is either known from context or is indicated in some way (e.g., by the used abbreviation). Letters or abbreviations will also be used to indicate a response variable. • Power of j, j > 1 (above a letter or abbreviation): main effect of order j of a particular covariate. • Colon (:) between two or more letters or abbreviations: interaction term based on particular covariates. • Plus sign (+): a delimiter of the model terms. • Tilde (∼): a delimiter between the response and description of the regression function. Further, when using a model formula, it is assumed that the intercept term is explicitely included in the regression function. If the explicit intercept should not be included, this will be indicated by writing −1 among the model terms. 8.8.3 Hierarchically well formulated model Definition 8.7 Hierarchically well formulated model. Hierarchically well formulated (HWF) model9 is such a model that contains an intercept term (possibly implicitely) and with each model term also all lower order terms that are nested in this term. Notes. • Unless there is some well-defined specific reason, models used in practice should be hierarchically well formulated. • Reason for use of the HWF models is the fact that the regression space of such models is invariant towards linear (location-scale) transformations of the regressors where invariance is meant with respect to possibility to obtain the equivalent linear models. Example 8.5. Consider a quadratic regression function mx (x) = β0 + β1 x + β2 x2 and perform a linear transformation of the regressor: x = δ (t − ϕ), 9 hierarchicky dobře formulovaný model t=ϕ+ x , δ (8.63) 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES 147 where δ 6= 0 and ϕ 6= 0 are pre-specified constants and t is a new regressor. The regression function in t is mt (t) = γ0 + γ1 t + γ2 t2 , where γ0 = β0 − β1 δϕ + β2 δ 2 ϕ2 , γ1 = β1 δ − 2β2 δ 2 ϕ, γ2 = β2 δ 2 . With at least three different x values in the data, both regression functions lead to two equivalent linear models of rank 3. Suppose now that the initial regression function mx did not include a linear term, i.e., it was mx (x) = β0 + β2 x2 which leads to a linear model of rank 2 (with at least three or even two different covariate values in data). Upon performing the linear transformation (8.63) of the regressor x, the regression function becomes mt (t) = γ0 + γ1 t + γ2 t2 with γ0 = β0 + β2 δ 2 ϕ2 , γ1 = −2β2 δ 2 ϕ, γ2 = β 2 δ 2 . With at least three different covariate values in data, this leads to the linear model of rank 3. To use a non-HWF model in practice, there should always be a (physical, . . . ) reason for that. For example, • No intercept in the model ≡ it can be assumed that the response expectation is zero if all regressors in a chosen parameterization take zero values. • No linear term in a model with a quadratic regression function m(x) = β0 + β2 x2 ≡ it can be assumed that the regression function is a parabola with the vertex in a point (0, β0 ) with respect to the x parameterization. • No main effect of one covariate in an interaction model with two numeric covariates and a regression function m(x, z) = β0 + β1 z + β2 x z ≡ it can be assumed that with z = 0, the response expectation does not depend on a value of x, i.e., E Y X = x, Z = 0 = β0 (a constant). 8.8.4 ANOVA tables For a particular linear model, so called ANOVA tables are often produced to help the analyst to decide which model terms are important with respect to its influence on the response expectation. Similarly to well known one-way ANOVA table (see any of introductory statistical courses), ANOVA tables produced in a context of linear models provide on each row input of a certain F-statistic, now that based on Theorem 5.2. The last row of the table (labeled often as Residual, Error or Within) provides (i) residual degrees of freedom νe of the considered model; (ii) residual sum of squares SSe of the considered model; (iii) residual mean square MSe = SSe /νe of the considered model. 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES 148 Each of the remaining rows of the ANOVA table provides input for the numerator of the F-statistic that corresponds to comparison of certain two models M1 ⊂ M2 which are both submodels of the considered model (or M2 is the considered model itself) and which have ν1 and ν2 degrees of freedom, respectively. The following quantities are provided on each of the remaining rows of the ANOVA table: (i) degrees of freedom for the numerator of the F-statistic (effect degrees of freedom νE = ν1 −ν2 ); (ii) difference in the residual sum of squares of the two models (effect sum of squares SSE = SS M2 M1 ); (iii) ratio of the above two values which is the numerator of the F-statistic (effect mean square MSE = SSE /νE ); (iv) value of the F-statistic FE = MSE /MSe ; (v) a p-value based on the F-statistic FE and the FνE , νe distribution. Several types of the ANOVA tables are distinguished which differ by definition of a pair of the two models M1 and M2 that are being compared on a particular row. Consequently, interpretation of results provided by the ANOVA tables of different type differs. Further, it is important to know that in all ANOVA tables, the lower order terms always appear on earlier rows in the table than the higher order terms that include them. Finally, for some ANOVA tables, different interpretation of the results is obtained for different ordering of the rows with the terms of the same hierarchical level, e.g., for different ordering of the main effect terms. We introduce ANOVA tables of three types which are labeled by the R software (and by many others as well) as tables of type I, II or III (arabic numbers can be used as well). Nevertheless, note that there exist software packages and literature that use different typology. In the reminder of this section we assume that intercept term is included in the considered model. In the following, we illustrate each type of the ANOVA table on a linear model based on two covariates whose main effect terms will be denoted as A and B. Next to the main effects, the model will include also an interaction term A : B. That is, the model formula of the considered model, denoted as MAB is ∼ A + B + A : B. In total, the following (sub)models of this model will appear in the ANOVA tables: M0 : ∼ 1, MA : ∼ A, MB : ∼ B, MA+B : ∼ A + B, MAB : ∼ A + B + A : B. The symbol SS F2 F1 will denote a difference in the residual sum of squares of the models with model formulas F1 and F2 . Type I (sequential) ANOVA table Example 8.6 (Type I ANOVA table for model MAB :∼ A + B + A : B). In the type I ANOVA table, the presented results depend on the ordering of the rows with the terms of the same hierarchical level. In this example, those are the rows that correspond to the main effect terms A and B. 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES 149 Order A + B + A:B Effect (Term) Degrees of freedom A ? Effect sum of squares SS A 1 ? ? ? ? SS A + B A ? ? ? A:B ? SS A + B + A : B A + B ? ? ? Residual νe SSe MSe B Effect mean square F-stat. P-value Order B + A + A:B Effect (Term) Degrees of freedom B ? Effect sum of squares SS B 1 ? ? ? ? SS A + B B ? ? ? A:B ? SS A + B + A : B A + B ? ? ? Residual νe SSe MSe A Effect mean square F-stat. P-value The row of the effect (term) E in the type I ANOVA table has in general the following interpretation and properties. • It compares two models M1 ⊂ M2 , where • M1 contains all terms included in the rows that precede the row of the term E. • M2 contains the terms of model M1 and additionally the term E. • The sum of squares shows increase of the explained variability of the response due to the term E on top of the terms shown on the preceding rows. • The p-value provides a significance of the influence of the term E on the response while controlling (adjusting) for all terms shown on the preceding rows. • Interpretation of the F-tests is different for rows labeled equally A in the two tables in Example 8.6. Similarly, interpretation of the F-tests is different for rows labeled equally B in the two tables in Example 8.6. • The sum of all sums of squares shown in the type I ANOVA table gives the total sum of squares SST of the considered model. This follows from the construction of the table where the terms are added sequentially one-by-one and from a sequential use of Theorem 5.8 (Breakdown of the total sum of squares in a linear model with intercept). Type II ANOVA table Example 8.7 (Type II ANOVA table for model MAB :∼ A + B + A : B). In the type II ANOVA table, the presented results do not depend on the ordering of the rows with the terms of the same hierarchical level as should become clear from subsequent explanation. 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES Degrees of freedom Effect sum of squares Effect mean square F-stat. P-value A ? ? ? ? B ? SS A + B B SS A + B A ? ? ? A:B ? SS A + B + A : B A + B ? ? ? Residual νe SSe MSe Effect (Term) 150 The row of the effect (term) E in the type II ANOVA table has in general the following interpretation and properties. • It compares two models M1 ⊂ M2 , where • M1 is the considered (full) model without the term E and also all higher order terms than E that include E. • M2 contains the terms of model M1 and additionally the term E (this is the same as in type I ANOVA table). • The sum of squares shows increase of the explained variability of the response due to the term E on top of all other terms that do not include the term E. • The p-value provides a significance of the influence of the term E on the response while controlling (adjusting) for all other terms that do not include E. • For practical purposes, this is probably the most useful ANOVA table. Type III ANOVA table Example 8.8 (Type III ANOVA table for model MAB :∼ A + B + A : B). Also in the type III ANOVA table, the presented results do not depend on the ordering of the rows with the terms of the same hierarchical level as should become clear from subsequent explanation. Degrees of freedom Effect sum of squares Effect mean square F-stat. P-value A ? ? ? ? B ? ? ? ? A:B ? SS A + B + A : B B + A : B SS A + B + A : B A + A : B SS A + B + A : B A + B ? ? ? Residual νe SSe MSe Effect (Term) The row of the effect (term) E in the type III ANOVA table has in general the following interpretation and properties. • It compares two models M1 ⊂ M2 , where • M1 is the considered (full) model without the term E. • M2 contains the terms of model M1 and additionally the term E (this is the same as in type I and type II ANOVA table). Due to the construction of M1 , the model M2 is always equal to the considered (full) model. 8.8. HIERARCHICALLY WELL-FORMULATED MODELS, ANOVA TABLES 151 • The submodel M1 is not necessarily hierarchically well formulated. If M1 is not HWF, interpretation of its comparison to model M2 depends on a parameterization of the term E. Consequently, also the interpretation of the F-test depends on the used parameterization. • For general practical purposes, most rows of the type III ANOVA table are often useless. Chapter 9 Analysis of Variance In this chapter, we examine several specific issues of linearmodels where all covariates are categorical. That is, the covariate vector Z is Z = Z1 , . . . , Zp , Zj ∈ Zj , j = 1, . . . , p, and each Zj is a finite set (with usually a “low” cardinality). The corresponding linear models are traditionally used in the area of designed (industrial, agricultural, . . . ) experiments or controlled clinical studies. The elements of the covariate vector Z then correspond to p factors whose influence on the response Y is of interest. The values of those factors for experimental units/subjects are typically within the control of an experimenter in which case the covariates are fixed rather than being random. Nevertheless, since the whole theory presented in this chapter is based on statements on the conditional distribution of the response given the covariate values, everything applies for both fixed and random covariates. 152 9.1. ONE-WAY CLASSIFICATION 9.1 153 One-way classification One-way classification corresponds to situation of one categorical covariate Z ∈ Z = {1, . . . , G}, see also Section 7.4. A linear model is then used to parameterize a set of G (conditional) response expectations E Y Z = 1 , . . ., E Y Z = G that we call as one-way classified group means: g = 1, . . . , G. m(g) = E Y Z = g =: mg , Without loss of generality, we can assume that the response random variables Y1 , . . . , Yn are sorted such that Z1 = · · · = Zn1 = 1, Zn1 +1 = · · · = Zn1 +n2 = 2, . . ., Zn1 +···+nG−1 +1 = · · · = Zn = G. For notational clarity in theoretical derivations, it is useful to use a double subscript to index the individual observations and to merge responses with a common covariate value Z = g, g = 1, . . . , G, into response subvectors Y g : Z=1: .. . Y 1 = Y1,1 , . . . , Y1,n1 , .. . Z = G : Y G = YG,1 , . . . , YG,nG . > The full response vector is Y = Y 1 , . . . , Y G and its (conditional, given Z = Z1 , . . . , Zn ) mean is m1 1n1 .. . E Y Z = µ = . mG 1nG A standard linear model then additionally assumes var Y Z = σ 2 In . If moreover, a linear model with i.i.d. errors is assumed, we get Yg,j = mg + εg,j , g = 1, . . . , G, j = 1, . . . , ng , i.i.d. where εg,j ∼ D(0, σ 2 ). Finally, if moreover the covariates are fixed rather than random, the data form G independent random samples (with common variances): Sample 1 : .. . Y 1 = Y1,1 , . . . , Y1,n1 , .. . i.i.d. Y1,j ∼ D(m1 , σ 2 ), .. . j = 1, . . . , n1 , i.i.d. Sample G : Y G = YG,1 , . . . , YG,nG , YG,j ∼ D(mG , σ 2 ), j = 1, . . . , nG . A linear model and related methodology can then be used for inference on the group means m1 , . . . , mG or on their linear combinations. 9.1.1 Parameters of interest Differences between the group means The principal inferential interest with one-way classification lies in estimation of and tests on parameters θg,h = mg − mh , g, h = 1, . . . , G, g 6= h, 9.1. ONE-WAY CLASSIFICATION 154 which are the differences between the means. Since each θg,h is a linear combination of the group elements of the mean vector µ = E Y Z , it is trivially an estimable parameter of the underlying linear model irrespective of its parameterization. The LSE of each θg,h is then a difference between the corresponding fitted values. The principal null hypothesis being tested in context of the one-way classification is the null hypothesis on equality of the group means, i.e., the null hypothesis H0 : m1 = · · · = mG , which written in terms of the differences between the group means is H0 : θg,h = 0, g, h = 1, . . . , G, g 6= h. Factor effects One-way classification often corresponds to a designed experiment which aims in evaluating the effect of a certain factor on the response. In that case, the following quantities, called as factor effects, are usually of primary interest. Definition 9.1 Factor effects in a one-way classification. By factor effects in case of a one-way classification we understand the quantities η1 , . . . , ηG defined as G 1 X ηg = mg − mh , g = 1, . . . , G. G h=1 Notes. • The effects are again linear combinations of the elements of the mean vector µ = factor E Y Z and hence all are estimable parameters of the underlying linear model with the LSE being equal to the appropriate linear combination of the fitted values. • The null hypothesis H0 : ηg = 0, g = 1, . . . , G, is equivalent to the null hypothesis H0 : m1 = · · · = mG on the equality of the group means. 9.1.2 One-way ANOVA model As a reminder from Section 7.4.2, the regression space of the one-way classification is m 1 1 n1 . : m1 , . . . , mG ∈ R ⊆ Rn . .. m 1 G nG While assuming ng > 0, g = 1, . . . , G, n > G, its vector dimension is G. In Sections 7.4.3 and 7.4.4, we introduced two classical classes of parameterizations of this regression space and of the response mean vector µ as µ = Xβ, β ∈ Rk . 9.1. ONE-WAY CLASSIFICATION 155 ANOVA (less-than-full rank) parameterization mg = α0 + αg , with k = G + 1, β =: α = α0 , |{z} αZ . α1 , . . . , αG g = 1, . . . , G Full-rank parameterization Z mg = β0 + c> gβ , g = 1, . . . , G with k = G, β = β0 , β Z , where |{z} β1 , . . . , βG−1 c> 1 . . C= . c> G is a chosen G × (G − 1) (pseudo)contrast matrix. Note. If the parameters in the ANOVA parameterization are identified by the sum constraint P G g=1 αg = 0, we get G α0 = 1 X mg , G g=1 αg = ηg = mg − H 1 X mh , H h=1 that is, parameters α1 , . . . , αG are then equal to the factor effects. Terminology. Related linear model is referred to as one-way ANOVA model 1 Notes. • Depending on chosen parameterization (ANOVA or full-rank) the differences between the group means, parameters θg,h , are expressed as θg,h = αg − αh = cg − ch > βZ , g 6= h. The null hypothesis H0 : m1 = · · · mG on equality of the group means is the expressed as (a) H0 : α1 = · · · = αG . (b) H0 : β1 = 0 & . . . & βG−1 = 0, i.e., H0 : β Z = 0G−1 . • If a normal linear model is assumed, test on a value of the estimable vector parameter or a submodel test which compares the one-way ANOVA model with the intercept-only model can be used to test the above null hypotheses. The corresponding F-test is indeed a well known one-way ANOVA F-test. End of Lecture #15 (19/11/2015) 9.1. ONE-WAY CLASSIFICATION 9.1.3 156 Least squares estimation Start of In case of a one-way ANOVA linear model, explicit formulas for the LSE related quantities can easily Lecture #16 be derived. (26/11/2015) Theorem 9.1 Least squares estimation in one-way ANOVA linear model. The fitted values and the LSE of the group means in a one-way ANOVA linear model are equal to the group sample means: ng 1 X b m b g = Yg,j = Yg,l =: Y g• , ng g = 1, . . . , G, j = 1, . . . , ng . l=1 That is, Y 1• m b1 . . . . c := m . = . , Y G• m bG Y 1• 1n1 .. . Yb = . Y G• 1nG c | Z ∼ NG m, σ 2 V , where If additionally normality is assumed then m 1 . . . 0 n.1 . .. . .. V= . . . 0 . . . n1G Proof. Use a full-rank parameterization µ = Xβ with 1n1 . . . 0n1 . .. .. .. , X= β = m , . . . , m . . . 1 G .. 0nG . 1nG We have n1 . . . 0 . . .. . .. X> X = . . , 0 . . . nG P n1 j=1 Y1,j .. X> Y = . PnG , X> X −1 j=1 YG,j 1 n1 . . = . 0 ... .. . ... 0 .. . , 1 nG −1 > b=m c= m β b 1, . . . , m b G = X> X X Y = Y 1• , . . . , Y G• . Finally, m b 1 1n1 Y 1• 1n1 .. b = ... = . Yb = Xβ . Y G• 1nG m b G 1nG c follows from a general LSE theory. Normality and the form of the covariance matrix of m 1 model analýzy rozptylu jednoduchého třídění k 9.1. ONE-WAY CLASSIFICATION 157 LSE of regression coefficients and estimable parameters With a full-rank parameterization, a vector m is linked to the regression coefficients β = β0 , β Z , β Z = β1 , . . . , βG−1 , by the relationship m = β0 1G + Cβ Z . b where X is a model matrix derived from the (pseudo)contrast matrix Due to the fact that Yb = Xβ, Z b = βb0 , β b of the regression coefficients in a full-rank parameterization satisfy C, the LSE β bZ , c = βb0 1G + Cβ m which is a regular linear system with the solution βb0 bZ β ! Y 1• −1 ... . = 1G , C Y G• That is, the LSE of the regression coefficients is always a linear combination of the group sample means. The same then holds for any estimable parameter. For example, the LSE of the differences between the group means θg,h = mg − mh , g, h = 1, . . . , G, are θbg,h = Y g• − Y h• , g, h = 1, . . . , G. P Analogously, the LSE of the factor effects ηg = mg − G1 G h=1 mh , g = 1, . . . , G, are G ηbg = Y g• − 1 X Y h• , G g = 1, . . . , G. h=1 9.1.4 Within and between groups sums of squares, ANOVA Ftest Sums of squares Let as usual, Y denote a sample mean based on the response vector Y , i.e., ng G G 1 XX 1X Y = ng Y g• . Yg,j = n n g=1 j=1 g=1 In a one-way ANOVA linear model, the residual and the regression sums of squares and corresponding degrees of freedom are n n g g G X G X 2 X 2 X 2 SSe = Y − Yb = Yg,j − Ybg,j = Yg,j − Y g• , g=1 j=1 g=1 j=1 νe = n − G, n g G X G 2 X 2 X 2 SSR = Yb − Y 1n = Ybg,j − Y = ng Y g• − Y , g=1 j=1 g=1 νR = G − 1. In this context, the residual sum of squares SSe is also called the within groups sum of squares2 , the regression sum of squares SSR is called the between groups sum of squares3 . 2 vnitroskupinový součet čtverců 3 meziskupinový součet čtverců 9.1. ONE-WAY CLASSIFICATION 158 One-way ANOVA F-test Let us assume normality of the response and consider a submodel Y | Z ∼ Nn 1n β0 , σ 2 In of the one-way ANOVA model. A residual sum of squares of the submodel is n SS0e g G X 2 X 2 = SST = Y − Y 1n = Yg,j − Y . g=1 j=1 Breakdown of the total sum of squares (Theorem 5.8) gives SSR = SST − SSe = SS0e − SSe and hence the statistic of the F-test on a submodel is F = SSR G−1 SSe n−G = MSR , MSe (9.1) where SSR SSe , MSe = . G−1 n−G The F-statistic (9.1) is indeed a classical one-way ANOVA F-statistics which under the null hypothesis of validity of a submodel, i.e., under the null hypothesis of equality of the group means, follows an FG−1, n−G distribution. Above quantities, together with the P-value derived from the FG−1, n−G distribution are often recorded in a form of the ANOVA table: MSR = Effect (Term) Degrees of freedom Effect sum of squares Effect mean square F-stat. P-value Factor G−1 SSR MSR F p Residual n−G SSe MSe Consider a terminology introduced in Section 8.8, and denote as Z main effect terms that correspond to the covariate Z. We have SSR = SS Z 1 and the above ANOVA table is now type I as well as type II ANOVA table. If intercept is explicitely included in the model matrix then it is also the type III ANOVA table. 9.2. TWO-WAY CLASSIFICATION 9.2 159 Two-way classification Two-way classification corresponds to situation of two categorical covariates Z ∈ Z = {1, . . . , G} and W ∈ W = {1, . . . , H}, see also Section 8.7. A linear model is then used to parameterize a set of G · H (conditional) response expectations E Y Z = g, W = h , g = 1, . . . , G, h = 1, . . . , H that we will call as two-way classified group means: m(g, h) = E Y Z = g, W = h =: mg,h , g = 1, . . . , G, h = 1, . . . , H. Analogously to Section 8.7 and without loss of generality, we can assume that the response variables Y1 , . . . , Yn are sorted as indicated in that section. That is, the first n1,1 responses correspond to (Z, W ) = (1, 1), the following n2,1 responses to (Z, W ) = (2, 1) etc. Analogously to one-way classification, we will now use a triple subscript to index the individual observations and merge responses with a common value of the two covariates into response subvectors Y g,h , g = 1, . . . , G, h = 1, . . . , H as indicated in the following table: Z 1 1 .. . Y 1,1 = Y1,1,1 , . . . , Y1,1,n1,1 .. . G Y G,1 = YG,1,1 , . . . , YG,1,nG,1 W ... H .. . Y 1,H = Y1,H,1 , . . . , Y1,H,n1,H .. .. . . .. . Y G,H = YG,H,1 , . . . , YG,H,nG,H The overall response vector Y is assumed to be taken from columns of the above table, i.e., Y = Y 1,1 , . . . , Y G,1 , . . . , Y 1,H , . . . , Y G,H . In the same way, we define a vector m being composed of the two-way classified group means as m = m1,1 , . . . , mG,1 , . . . , m1,H , . . . , mG,H . Finally, we keep using a dot notation for collapsed sample sizes or means of the group means. That is, n= G X H X ng,h , g=1 h=1 ng• = H X ng,h , g = 1, . . . , G, n•h = H 1 X mg,h , H h = 1, . . . , H, G g = 1, . . . , G, m•h = 1 X mg,h , G h = 1, . . . , H, g=1 h=1 m = ng,h , g=1 h=1 mg• = G X 1 G·H G X H X mg,h = g=1 h=1 which can be summarized in a tabular form as 1 G G X g=1 mg• = H 1 X m•h , H h=1 9.2. TWO-WAY CLASSIFICATION 160 Group means Sample sizes W Z 1 mG,1 ... .. . .. . .. . 1 .. . m1,1 .. . G • m•1 ... W H • Z 1 m1,H .. . m1• .. . 1 .. . n1,1 .. . mG,H mG• G m•H m • nG,1 ... .. . .. . .. . n•1 ... H • n1,H .. . n1• .. . nG,H nG• n•H n Note. The above defined quantities mg• , m•h , m are the means of the group means which are not weighted by the corresponding sample sizes (which are moreover random if the covariates are random). As such, all above defined means are always real constants and never random variables (irrespective of whether the covariates are considered as being fixed or random). > > The (conditional, given Z = Z1 , . . . , Zn and W = W1 , . . . , Wn ) mean of the response vector Y is m1,1 1n1,1 .. . E Y Z, W = µ = . mG,H 1nG,H A standard linear model then additionally assumes var Y Z, W = σ 2 In . If moreover, a linear model with i.i.d. errors is assumed, we get Yg,h,j = mg,h + εg,h,j , g = 1, . . . , G, h = 1, . . . , H, j = 1, . . . , ng,h , i.i.d. where εg,h,j ∼ D(0, σ 2 ). Finally, if moreover the covariates are fixed rather than random, the data form G · H independent random samples (with common variances): Sample (1, 1) : Y 1,1 = Y1,1,1 , . . . , Y1,1,n1,1 , i.i.d. Y1,1,j ∼ D(m1,1 , σ 2 ), .. . j = 1, . . . , n1,1 , .. . Sample (G, H) : Y G,H = YG,H,1 , . . . , YG,H,nG,H , i.i.d. YG,H,j ∼ D(mG,H , σ 2 ), j = 1, . . . , nG,H . A linear model and related methodology can then be used for inference on the group means m1,1 , . . . , mG,H or on their linear combinations. On top of that, a linear model can reveal a structure of the relationship in which the two covariates – the two factors influence the two-way classified group means. 9.2.1 Parameters of interest Means of the means and their differences Various quantities, all being linear combinations of the two-way classified group means, i.e., all being estimable in any parameterization of the two-way classification, are clasically of interest. They include: 9.2. TWO-WAY CLASSIFICATION 161 (i) The mean of the group means m. (ii) The means of the means by the first or the second factor, i.e., parameters m1• , . . . , mG• , and m•1 , . . . , m•H . (iii) Differences between the means of the means by the first or the second factor, i.e., parameters θg1 ,g2 • := mg1 • − mg2 • , g1 , g2 = 1, . . . , G, g1 6= g2 , θ•h1 ,h2 := m•h1 − m•h2 , h1 , h2 = 1, . . . , H, h1 6= h2 . Those, in a certain sense quantify the mean effect of the first or the second factor on the response. Main effects Analogously to one-way classification, also the two-way classification often corresponds to a designed experiment, in this case aiming in evaluating the effect of two factors represented by the covariates Z and W . The following quantities, called as main effects of factor Z and W , respectively, are then usually of primary interest. Definition 9.2 Main effects in a two-way classification. Consider a two-way classification based on factors Z and W . By main effects of the factor Z, we Z defined as understand quantities η1Z , . . . , ηG ηgZ := mg• − m, g = 1, . . . , G. W defined as By main effects of the factor W , we understand quantities η1W , . . . , ηH ηhW := m•h − m, h = 1, . . . , H. Note. Differences between the means of the means are also equal to the differences between the main effects: θg1 ,g2 • = mg1 • − mg2 • = ηgZ1 − ηgZ2 , θ•h1 ,h2 = m•h1 − m•h2 = ηhW1 − ηhW2 , g1 , g2 = 1, . . . , G, g1 6= g2 , h1 , h2 = 1, . . . , H, h1 6= h2 . Interaction effects Suppose now that the factors Z and W act additively on the response expectation. In this case, differences mg1 ,h − mg2 ,h , g1 , g2 = 1, . . . , G do not depend on a value of h = 1, . . . , H. Consequently, for any h mg1 ,h − mg2 ,h = mg1 • − mg2 • , g1 , g2 = 1, . . . , G. 9.2. TWO-WAY CLASSIFICATION 162 This implies mg1 ,h − mg1 • = mg2 ,h − mg2 • , g1 , g2 = 1, . . . , G, h = 1, . . . , H. In other words, additivity implies that for any h = 1, . . . , H, the differences ∆(g, h) = mg,h − mg• do not depend on a value of g = 1, . . . , G. Then (for any g = 1, . . . , G and h = 1, . . . , H) mg,h − mg• = ∆(g, h) = G 1 X ∆(g ? , h) G ? g =1 G 1 X (mg? ,h − mg? • ) = G ? g =1 = m•h − m. Clearly, we would arrive at the same comclusion if we started differently from assuming that mg,h1 − mg,h2 , h1 , h2 = 1, . . . , H do not depend on a value of g = 1, . . . , G. That is, additivity implies mg,h − mg• − m•h + m = 0, g = 1, . . . , G, h = 1, . . . , H. Easily, we find that this is also a sufficient condition for additivity. Definition 9.3 Interaction effects in a two-way classification. Consider a two-way classification based on factors Z and W . By interaction effects of the two factors, ZW , . . . , η ZW defined as we understand quantities η1,1 G,H ZW := mg,h − mg• − m•h + m, ηg,h g = 1, . . . , G, h = 1, . . . , H. Notes. • While taking into account definition of the main and interaction effects, the two-way classified group means are given as ZW , mg,h = m + ηgZ + ηhW + ηg,h g = 1, . . . , G, h = 1, . . . , H. • Hypothesis of additivity of the effect of the factors Z and W on the response expectation is given as ZW H0 : ηg,h = 0, g = 1, . . . , G, h = 1, . . . , H. 9.2.2 Two-way ANOVA models The following linear models, referred to as two-way ANOVA models,4 are traditionally considered. Each of them corresponds to different structure for the two-way classified group means. 4 modely analýzy rozptylu dvojného třídění 9.2. TWO-WAY CLASSIFICATION 163 Interaction model No structure is imposed on the group means that can in general be written as ZW mg,h = α0 + αgZ + αhW + αg,h , g = 1, . . . , G, h = 1, . . . , H, (9.2) where α0 , Z αZ = α1Z , . . . , αG , αZW W αW = α1W , . . . , αH , ZW ZW ZW ZW = α1,1 , . . . , αG,1 , . . . , α1,H , . . . , αG,H are the regression parameters. If ng,h > 0 for all g, h, the rank of the related linear model is G · H, see Section 8.7. This explains why the interaction model is also called as the saturated model 5 . The reason is that its regression space has maximal possible vector dimension equal to the number of the group means. Identification of the regression coefficients is possibly achieved by the sum constraints (see Section 8.7.2) G X αgZ = 0, g=1 H X H X αhW = 0, h=1 (9.3) ZW = 0, g = 1, . . . , G, αg,h h=1 G X ZW = 0, h = 1, . . . , H. αg,h g=1 Having imposed the sum constraints (9.3), the regression coefficients coincide with the mean of the means m, with the main and interaction effects respectively (see Section 8.7.2 for corresponding derivations), that is, α0 = m, αgZ = ηgZ = mg• − m, αhW ZW αg,h = = g = 1, . . . , G, ηhW = m•h − m, ZW = mg,h − mg• ηg,h h = 1, . . . , H, − m•h + m, g = 1, . . . , G, h = 1, . . . , H. Section 8.7 also explains possible full-rank parameterization of the underlying linear model, which parameterizes the two-way classified group means as ZW > Z > W mg,h = β0 + c> + d> g = 1, . . . , G, h = 1, . . . , H, (9.4) g β + dh β h ⊗ cg β where CG×(G−1) c> 1 . .. , = c> G DH×(H−1) d> 1 . .. , = d> H are chosen (pseudo)contrast matrices, and β0 , β ZW Z β Z = β1Z , . . . , βG−1 , W β W = β1W , . . . , βH−1 , ZW ZW ZW ZW = β1,1 , . . . , βG−1,1 , . . . , β1,H−1 , . . . , βG−1,H−1 are the regression parameters. 5 saturovaný model 9.2. TWO-WAY CLASSIFICATION 164 In the following, let symbols Z and W denote the terms in the model matrix that correspond to the main effects β Z of the covariate Z, and β W of the covariate W , respectively. Let further Z : W denote the terms corresponding to the interaction effects β ZW . The interaction model will then symbolically be written as MZW : ∼ Z + W + Z : W. Additive model It is obtained as a submodel of the interaction model (9.2) where it is requested ZW ZW α1,1 = · · · = αG,H , which in the full-rank parameterization (9.4) corresponds to requesting β ZW = 0(G−1)·(H−1) . Hence the group means can be written as mg,h = α0 + αgZ + αhW , g = 1, . . . , G, h = 1, . . . , H, Z > W = β0 + c> g β + dh β , (9.5) (9.6) In Section 8.7.5, we showed that if ng,h > 0 for all g, h, the rank of the linear model with the two-way classified group means that satisfy (9.5), is G+H −1. The additive model will symbolically be written as MZ+W : ∼ Z + W. Note. It can easily be shown that ng• for all g = 1, . . . , G and n•h for all h = 1, . . . , H suffice to get a rank of the related linear model being still G+H −1. This guarantees, among other things, that all parameters that are estimable in the additive model with ng,h > 0 for all g, h, are still estimable under a weaker requirement ng• for all g = 1, . . . , G and n•h for all h = 1, . . . , H. That is, if the additive model can be assumed, it is not necessary to have observations for all possible combinations of the values of the two covariates (factors) and the same types of the statistical inference are possible. This is often exploited in the area of designed experiments where it might be impractical or even impossible to get observations under all possible covariate combinations. See Section 8.7.5 what the additive model implies for the two-way classified group means. Most importantly, (i) For each g1 6= g2 , g1 , g2 ∈ {1, . . . , G}, the difference mg1 ,h − mg2 ,h does not depend on a value of h ∈ {1, . . . , H} and is equal to the difference between the corresponding means of the means by the first factor, i.e., mg1 ,h − mg2 ,h = mg1 • − mg2 • = θg1 ,g2 • , which is expressed using the parameterizations (9.5) and (9.6) as > θg1 ,g2 • = αgZ1 − αgZ2 = cg1 − cg2 β Z . (ii) For each h1 6= h2 , h1 , h2 ∈ {1, . . . , H}, the difference mg,h1 − mg,h2 does not depend on a value of g ∈ {1, . . . , G} and is equal to the difference between the corresponding means of the means by the second factor, i.e., mg,h1 − mg,h2 = m•h1 − m•h2 = θ•h1 ,h2 , which is expressed using the parameterizations (9.5) and (9.6) as > θ•h1 ,h2 = αhW1 − αhW2 = dh1 − dh2 β W . 9.2. TWO-WAY CLASSIFICATION 165 Model of effect of Z only It is obtained as a submodel of the additive model (9.5) by requesting W α1W = · · · = αH , which in the full-rank parameterization (9.6) corresponds to requesting β W = 0H−1 . Hence the group means can be written as mg,h = α0 + αgZ , g = 1, . . . , G, h = 1, . . . , H, (9.7) Z = β0 + c> gβ , This is in fact a linear model for the one-way classified (by the values of the covariate Z) group means whose rank is G as soon as ng• > 0 for all g = 1, . . . , G. The model of effect of Z only will symbolically be written as MZ : ∼ Z. The two-way classified group means then satisfy (i) For each g = 1, . . . , G, mg,1 = · · · = mg,H = mg• . (ii) m•1 = · · · = m•H . Model of effect of W only It is the same as the model of effect of Z only with exchaged meaning of Z and W . That is, the model of effect of W only is obtained as a submodel of the additive model (9.5) by requesting Z α1Z = · · · = αG , which in the full-rank parameterization (9.6) corresponds to requesting β Z = 0G−1 . Hence the group means can be written as mg,h = α0 + αhW , g = 1, . . . , G, h = 1, . . . , H, W = β0 + d> hβ . The model of effect of W only will symbolically be written as MW : ∼ W. Intercept only model This is a submodel of either the model (9.7) of effect of Z only where it is requested Z α1Z = · · · = αG , (9.8) 9.2. TWO-WAY CLASSIFICATION 166 or the model (9.8) of effect of W only where it is requested W α1W = · · · = αH , Hence the group means can be written as mg,h = α0 , g = 1, . . . , G, h = 1, . . . , H. As usual, this model will symbolically be denoted as M0 : ∼ 1. Summary In summary, we consider the following models for the two-way classification: Model Rank Requirement for Rank MZW : ∼ Z + W + Z : W G·H ng,h > 0 for all g = 1, . . . , G, h = 1, . . . , H G+H −1 ng• > 0 for all g = 1, . . . , G, n•h > 0 for all h = 1, . . . , H MZ : ∼ Z G ng• > 0 for all g = 1, . . . , G MW : ∼ W H n•h > 0 for all h = 1, . . . , H M0 : ∼ 1 1 n>0 MZ+W : ∼ Z + W The considered models form two sequence of nested submodels: (i) M0 ⊂ MZ ⊂ MZ+W ⊂ MZW ; (ii) M0 ⊂ MW ⊂ MZ+W ⊂ MZW . Related submodel testing then corresponds to evaluating whether the two-way classified group means satisfy a particular structure invoked by the submodel at hand. If normality of the error terms is assumed, the testing can be performed by the methodology of Chapter 5 (F-tests on submodels). 9.2.3 Least squares estimation Also with the two-way classification, explicit formulas for some of the LSE related quantities can be derived and then certain properties of the least squares based inference drawn. Notation (Sample means in two-way classification). Y g,h• := ng,h 1 X ng,h j=1 Yg,h,j , g = 1, . . . , G, h = 1, . . . , H, 9.2. TWO-WAY CLASSIFICATION Y g• H ng,h H 1 XX 1 X := Yg,h,j = ng,h Y g,h• , ng• ng• g = 1, . . . , G, G ng,h G 1 XX 1 X := Yg,h,j = ng,h Y g,h• , n•h n•h h = 1, . . . , G, h=1 j=1 Y •h 167 g=1 j=1 G Y := H ng,h h=1 g=1 G H g=1 h=1 1 XXX 1X 1X ng• Y g• = Yg,h,j = n•h Y •h . n n n g=1 h=1 j=1 As usual, m b g,h , g = 1, . . . , G, h = 1, . . . , H, denote the LSE of the two-way classified group means c= m and m b 1,1 , . . . , m b G,H . Theorem 9.2 Least squares estimation in two-way ANOVA linear models. The fitted values and the LSE of the group means in two-way ANOVA linear models are given as follows (always for g = 1, . . . , G, h = 1, . . . , H, j = 1, . . . , ng,h ). (i) Interaction model MZW : ∼ Z + W + Z : W m b g,h = Ybg,h,j = Y g,h• . (ii) Additive model MZ+W : ∼ Z + W m b g,h = Ybg,h,j = Y g• + Y •h − Y , but only in case of balanced data6 (ng,h = J for all g = 1, . . . , G, h = 1, . . . , H). (iii) Model of effect of Z only MZ : ∼ Z m b g,h = Ybg,h,j = Y g• . (iv) Model of effect of W only MW : ∼ W m b g,h = Ybg,h,j = Y •h . (v) Intercept only model M0 : ∼ 1 m b g,h = Ybg,h,j = Y . Note. There exists no simple expression to calculate the fitted values in the additive model in case of unbalanced data. See Searle (1987, Section 4.9) for more details. Proof. Only the fitted values in the additive model must be derived now. 6 vyvážená data 9.2. TWO-WAY CLASSIFICATION 168 Models MZW , MZ , MW are, in fact, one-way ANOVA models where we already know that the fitted values are equal to the corresponding group means. Also model M0 is nothing new. Fitted values in the additive model can be calculated by solving the normal equations corresponding to the parameterization mg,h = α0 + αgZ + αhW , g = 1, . . . , G, h = 1, . . . , H. while imposing the identifying constraints G X H X αgZ = 0, g=1 αhW = 0. h=1 For the additive model with the balanced data (ng,h = J for all g = 1, . . . , G, h = 1, . . . , H): • Sum of squares to be minimized SS(α) = XXX g h Yg,h,j − α0 − αgZ − αhW 2 . j • Normal equations ≡ derivatives of SS(α) divided by (−2) and set to zero: XXX X X Yg,h,j − GHJα0 − HJ αgZ − GJ αhW = 0, g h g j XX h Yg,h,j − HJα0 − HJαgZ − J αhW = 0, g = 1, . . . , G, αgZ − GJαhW = 0, h = 1, . . . , H. j h XX g h X Yg,h,j − GJα0 − J X g j • After exploiting the identifying constraints: XXX Yg,h,j − GHJα0 = 0, g h XX h Yg,h,j − HJα0 − HJαgZ = 0, g = 1, . . . , G, Yg,h,j − GJα0 − GJαhW = 0, h = 1, . . . , H. j XX g j j • Hence α b0 = Y , α bgZ = Y g• − Y , α bhW = Y •h − Y , g = 1, . . . , G, h = 1, . . . , H. • And then m b g,h = α b0 + α bgZ + α bhW = Y g• + Y •h − Y , g = 1, . . . , G, h = 1, . . . , H. k End of Lecture #16 (26/11/2015) 9.2. TWO-WAY CLASSIFICATION 169 Start of Lecture #17 Consequence of Theorem 9.2: LSE of the means of the means in the interaction (26/11/2015) and the additive model with balanced data. With balanced data (ng,h = J for all g = 1, . . . , G, h = 1, . . . , H), the LSE of the means of the means by the first factor (parameters m1• , . . . , mG• ) or by the second factor (parameters m•1 , . . . , m•H ) satisfy in both the interaction and the additive two-way ANOVA linear models the following: b g• = Y g• , m b •h = Y •h , m g = 1, . . . , G, h = 1, . . . , H. cZ := m cW := m b 1• , . . . , m b G• and m b •1 , . . . , m b •H If additionally normality is assumed then m satisfy cZ | Z, W ∼ NG mZ , σ 2 VZ , cW | Z, W ∼ NH mW , σ 2 VW , m m where m1• . . mZ = . , mG• mW m•1 . . = . , m•H 1 JH . . VZ = . 0 1 JG . . VW = . 0 ... .. . ... ... .. . ... 0 .. . , 1 JH 0 .. . . 1 JG Proof. All parameters mg• , g = 1, . . . , G, and m•h , h = 1, . . . , H are linear combinations of the group means (of the response mean vector µ = E Y Z, W ) and hence are estimable with the LSE being an appropriate linear combination of the LSE of the group means. With balanced data, we get for the the considered models (calculation shown only for LSE of mg• , g = 1, . . . , G): (i) Interaction model H H H H 1 X 1 X 1 X 1 X b mg• = m b g,h = Y g,h• = J Y g,h• = ng,h Y g,h• = Y g• . H H HJ ng• h=1 h=1 h=1 h=1 (ii) Additive model H H 1 X 1 X b mg• = m b g,h = Y g• + Y •h − Y H H h=1 h=1 H H 1 X 1 X G JY •h − Y = Y g• + Y •h − Y = Y g• + H H GJ h=1 h=1 H = Y g• + 1X n•h Y •h −Y = Y g• . n | h=1 {z } Y 9.2. TWO-WAY CLASSIFICATION 170 Further, E Y g• Z, W = mg• follows from properties of the LSE which are unbiased or from direct calculation. Next, var Y g• Z, W = var H J 1 XX σ2 Yg,h,j Z, W = JH JH h=1 j=1 follows from the linear model assumption var Y Z, W = σ 2 In . Finally, normality of Y g• in case of a normal linear model, follows from the general LSE theory. 9.2.4 k Sums of squares and ANOVA tables with balanced data Sums of squares As already mentioned in Section 9.2.2, the considered models form two sequence of nested submodels: (i) M0 ⊂ MZ ⊂ MZ+W ⊂ MZW ; (ii) M0 ⊂ MW ⊂ MZ+W ⊂ MZW . Corresponding differences in the residual sums of squares (that enter the numerator of the respective F-statistic) are given as squared Euclidean norms of the fitted values from the models being compared (Section 5.1). In particular, in case of balanced data (ng,h = J, g = 1, . . . , G, h = 1, . . . , H), we get G X H X 2 SS Z + W + Z : W Z + W = J Y g,h• − Y g• − Y •h + Y , g=1 h=1 G X H G X H X 2 X 2 SS Z + W W = J Y g• + Y •h − Y − Y •h = J Y g• − Y , g=1 h=1 g=1 h=1 G X G X H H X 2 X 2 SS Z + W Z = J Y g• + Y •h − Y − Y g• = J Y •h − Y , g=1 h=1 G X H X 2 SS Z 1 = J Y g• − Y , g=1 h=1 G X H X 2 SS W 1 = J Y •h − Y . g=1 h=1 We see, SS Z + W W = SS Z 1 , SS Z + W Z = SS W 1 . Nevertheless, note that this does not hold in case of unbalanced data. g=1 h=1 9.2. TWO-WAY CLASSIFICATION 171 Notation (Sums of squares in two-way classification). In case of two-way classification and balanced data, we will use the following notation. SSZ := SSW := SSZW := G X H X g=1 h=1 G X H X g=1 h=1 G X H X J Y g• − Y 2 J Y •h − Y 2 , , J Y g,h• − Y g• − Y •h + Y 2 , g=1 h=1 SST := G X H X J X Yg,h,j − Y 2 , g=1 h=1 j=1 SSZW := e G X H X J X 2 Yg,h,j − Y g,h• . g=1 h=1 j=1 Notes. • Quantities SSZ , SSW , SSZW are differences of the residual sums of squares of two models that differ by terms Z, W or Z : W, respectively. • Quantity SST is a classical total sum of squares. • Quantity SSZW is a residual sum of squares from the interaction model. e Lemma 9.3 Breakdown of the total sum of squares in a balanced two-way classification. In case of a balanced two-way classification, the following identity holds SST = SSZ + SSW + SSZW + SSZW . e Proof. Decomposition in the lemma corresponds to the numerator sum of squares of the F -statistics when testing a series of submodels M0 ⊂ MZ ⊂ MZ+W ⊂ MZW or a series of submodels M0 ⊂ MW ⊂ MZ+W ⊂ MZW . Let M0 , MZ , MW , MZ+W , MZW be the regression spaces of the models M0 , MZ , MW , MZ+W , MZW , respectively. That is, SST = kU 0 k2 , where U 0 are residuals of model M0 and U 0 = D 1 + D 2 + D 3 + U ZW , where D 1 , D 2 , D 3 , U ZW are mutually orthogonal projections of Y into subspaces of Rn : 9.2. TWO-WAY CLASSIFICATION 172 (i) D 1 : projection into MZ \ M0 , kD 1 k2 = SSZ . (ii) D 2 : projection into MZ+W \ MZ , kD 2 k2 = SSZ+W . (iii) D 3 : projection into MZW \ MZ+W , kD 3 k2 = SSZW . (iv) U ZW : projection into Rn \ MZW (residual space of MZW ). From orthogonality: SST = SSZ + SSW + SSZW + SSZW . e k ANOVA tables As consequence of the above considerations, it holds for balanced data: (i) Equally labeled rows in the type I ANOVA table are the same irrespective of whether the table is formed in the order Z + W + Z:W or in the order W + Z + Z:W. (ii) Type I and type II ANOVA tables are the same. Both type I and type II ANOVA table then take the form Effect (Term) Degrees of freedom Effect sum of squares Effect mean square F-stat. P-value Z G−1 SSZ ? ? ? W H −1 SSW ? ? ? Z:W GH − G − H + 1 SSZW ? ? ? Residual n − GH SSZW e ? Chapter 10 Checking Model Assumptions In Chapter 4, we introduced some basic, mostly graphical methods to check the model assumptions. Now, we introduce some additional methods, mostly based onstatistical tests. As in Chapter 4, we assume that data are represented by n random vectors Yi , Z i , Z i = Zi,1 , . . . , Zi,p ∈ Z ⊆ Rp i = 1, . . . , n. Possibly two sets of regressors are available: (i) X i , i = 1, . . . , n, where X i = tX (Z i ) for some transformation tX : Rp −→ Rk . They give rise to the model matrix X> 1 . 0 .. = X , . . . , X k−1 . Xn×k = > Xn For most practical problems, X 0 = 1, . . . , 1 (almost surely). (ii) V i , i = 1, . . . , n, where V i = tV (Z i ) for some transformation tV : Rp −→ Rl . They give rise to the model matrix V> 1 . 1 l . Vn×l = . = V , . . . , V . > Vn Primarily, we will assume that the model matrix X is sufficient to be able to assume that E Y Z = k E Y X = Xβ for some β = β0 , . . . , βk−1 ∈ R . That is, we will arrive from assuming Y Z ∼ Xβ, σ 2 In , or even from assuming normality, i.e., Y Z ∼ Nn Xβ, σ 2 In . The task is now to verify appropriateness of those assumptions that, in principle, consist of four subassumptions outlined in Chapter 4: (A1) Correct regression function (errors with a zero mean). (A2) Homoscedasticity of errors. (A3) Uncorrelated/independent errors. (A4) Normal errors. 173 10.1. MODEL WITH ADDED REGRESSORS 10.1 174 Model with added regressors In this section, we technically derive some expressions that will be useful in latter sections of this chapter and also in Chapter 14. We will deal with two models: (i) Model M: Y Z ∼ Xβ, σ 2 In . (ii) Model Mg : Y Z ∼ Xβ + Vγ, σ 2 In , where the model matrix is an n × (k + l) matrix G, G = X, V . Notation (Quantities derived under the two models). (i) Quantities derived while assuming model M will be denoted as it is usual. In particular: − • (Any) solution to normal equations: b = X> X X> Y . In case of a full-rank model matrix X: b = X> X −1 X> Y β is the LSE of a vector β in model M; • Hat matrix (projection matrix into the regression space M X ): H = X X> X − X> = hi,t i,t=1,...,n ; • Fitted values Yb = HY = Yb1 , . . . , Ybn ; • Projection matrix into the residual space M X ⊥ M = In − H = mi,t : i,t=1,...,n ; • Residuals: U = Y − Yb = MY = U1 , . . . , Un ; 2 • Residual sum of squares: SSe = U . (ii) Analogous quantities derived while assuming model Mg will be indicated by a subscript g: − • (Any) solution to normal equations: bg , cg = G> G G> Y . In case of a full-rank model matrix G: −1 > > b ,γ β G Y g bg = G G provides the LSE of vectors β and γ in model Mg ; • Hat matrix (projection matrix into the regression space M G ): − Hg = G G> G G> = hg,i,t i,t=1,...,n ; • Fitted values Yb g = Hg Y = Ybg,1 , . . . , Ybg,n ; ⊥ • Projection matrix into the residual space M G : Mg = In − Hg = mg,i,t i,t=1,...,n • Residuals: U g = Y − Yb g = Mg Y = Ug,1 , . . . , Ug,n ; 2 • Residual sum of squares: SSe,g = U g . ; 10.1. MODEL WITH ADDED REGRESSORS 175 Lemma 10.1 Model with added regressors. Quantities derived while assuming model M : Y Z ∼ Xβ, σ 2 In and quantities derived while assuming model Mg : Y Z ∼ Xβ + Vγ, σ 2 In are mutually in the following relationship. Yb g = Yb + MV V> MV − V> U for some bg ∈ Rk , cg ∈ Rl . = Xbg + Vcg , Vector bg and cg such that Yb g = Xbg + Vcg satisfy: cg = − V> MV V> U , − bg = b − X> X X> Vcg Finally for some b = X> X − X> Y . 2 SSe − SSe,g = MVcg . Proof. • Yb g is a projection of Y into M X, V = M X, MV . − • Use “H = X X> X X> ”: − > X MV} {z | 0 Hg = X, MV V> MX V> MV | {z } 0 = X, MV = X X> X − X> X X> X 0 − 0 − > V MV X> + MV V> MV − X> V> M ! ! X> V> M ! V> M. • So that, − − Yb g = Hg Y = X X> X X> Y + MV V> MV V> MY |{z} | {z } U Yb − = Yb + MV V> MV V> U ® • Theorem 2.5: It must be possible to write Yb g as Yb g = Xbg + Vcg , where bg , cg solves normal equations based on a model matrix X, V . • We rewrite ® to see what bg and cg could be. 10.1. MODEL WITH ADDED REGRESSORS • Remember that Yb = Xb for any b = X> X 176 − X> Y . Take now ® and further calculate: − − Yb g = |{z} Xb + In − X X> X X> V V> MV V> U {z } | Yb M − − − = Xb + V V> MV V> U − X X> X X> V V> MV V> U − − − = X b − X> X X> V V> MV V> U + V V> MV V> U . | {z } | {z } cg bg − • That is, cg = V> MV V> U , − bg = b − X> X X> Vcg . • Finally 2 2 2 − SSe − SSe,g = Yb g − Yb = MV V> MV V> U = MVcg . k End of Lecture #17 (26/11/2015) 10.2. CORRECT REGRESSION FUNCTION 10.2 177 Correct regression function We are now assuming a linear model M : Y Z ∼ Xβ, σ 2 In , that written using the error terms is M : Y = Xβ + ε, E ε Z = E ε = 0n , var ε Z = var ε = σ 2 In . The assumption (A1) of a correct regression function is, in particular, E Y Z = Xβ for some β ∈ Rk , E Y Z ∈ M X , E ε Z = E ε = 0n . As (also) explained in Section 4.1, assumption (A1) implies E U Z = 0n and this property is exploited by a basic diagnostic tool which is a plot of residuals against possible factors derived from the covariates Z that may influence the residuals expectation. Factors traditionally considered are (i) Fitted values Yb ; (ii) Regressors included in the model M (columns of the model matrix X); (iii) Regressors not included in the model M (columns of the model matrix V). Assumptions. For the rest of this section, we assume that model M is a model of general rank r with intercept, that is, rank X = r ≤ k < n, X = X 0 , . . . , X k−1 , X 0 = 1n . In the following, we develop methods to examine whether for given j (j ∈ 1, . . . , k − 1 ) the jth regressor, i.e., the column X j , is correctly included in the model matrix X. In other words, we will aim in examining whether the jth regressor is possibly responsible for violation of the assumption (A1). 10.2.1 Partial residuals Notation (Model with a removed regressor). For j ∈ 1, . . . , k − 1 , let X(−j) denote the model matrix X without the column X j and let β (−j) = β0 , . . . , βj−1 , βj , . . . , βk−1 denote the regression coefficients vector without the jth element. Model with a removed jth regressor will be a linear model M(−j) : Y Z ∼ X(−j) β (−j) , σ 2 In . Start of Lecture #18 (03/12/2015) 10.2. CORRECT REGRESSION FUNCTION 178 All quantities related to the model M(−j) will be indicated by a superscript (−j). In particular, − > > M(−j) = In − X(−j) X(−j) X(−j) X(−j) is a projection matrix into the residual space M X(−j) ⊥ ; U (−j) = M(−j) Y is a vector of residuals of the model M(−j) . Assumptions. We will assume rank X(−j) = r − 1 which implies that (i) X j ∈ / M X(−j) ; (ii) X j 6= 0n ; (iii) X j is not a multiple of a vector 1n . Derivations towards partial residuals Model M is now a model with one added regressor to a model M(−j) and the two models form a pair (model–submodel). Let b = b0 , . . . , bj−1 , bj , bj+1 , . . . , bk−1 > be (any) solution to normal equations in model M. Lemma 10.1 (Model with added regressors) provides − > > bj = X j M(−j) X j X j U (−j) . (10.1) Further, since a matrix M(−j) is idempotent, we have 2 > X j M(−j) X j = M(−j) X j . > At the same time, M(−j) X j 6= 0n since X j ∈ / M X(−j) , X j 6= 0n . Hence, X j M(−j) X j > 0 and a pseudoinverse in (10.1) can be replaced by an inverse. That is, > bj = βbj = X j M(−j) X j −1 X j> > U (−j) = X j U (−j) > X j M(−j) X j is the LSE of the estimable parameter βj of model M (which is its BLUE). In summary, under the assumptions used to perform derivations above, i.e., while assuming that 0 X = 1n and for chosen j ∈ 1, . . . , k − 1 , the regression coefficient βj is estimable. Consequently, we define a vector of jth partial residuals of model M as follows. 10.2. CORRECT REGRESSION FUNCTION 179 Definition 10.1 Partial residuals. A vector of jth partial residuals1 of model M is a vector U part,j U1 + βbj X1,j .. . = U + βbj X j = . Un + βbj Xn,j Note. We have U part,j = U + βbj X j = Y − Xb − βbj X j = Y − Yb − βbj X j . That is, the jth partial residuals are calculated as (classical) residuals where, however, the fitted values subtract a part that corresponds to the column X j of the model matrix. Theorem 10.2 Property of partial residuals. Let Y Z ∼ Xβ, σ 2 In , rank(Xn×k ) = r ≤ k, X 0 = 1n , β = β0 , . . . , βk−1 . Let j ∈ 1, . . . , k − 1 be such that rank X(−j) = r − 1 and let βbj be the LSE of βj . Let us consider a linear model (regression line with covariates X j ) with • the jth partial residuals U part,j as response; • a matrix 1n , X j as the model matrix; • regression coefficients γ j = γj,0 , γj,1 . The least squares estimators of parameters γj,0 and γj,1 are γ bj,0 = 0, γ bj,1 = βbj . Proof. • U part,j = U + βbj X j . 2 2 part,j j j b • Hence U − γj,0 1n − γj,1 X = U − γj,0 1n + (γj,1 − βj )X = ®. ⊥ • Since 1n ∈ M X , X j ∈ M X , U ∈ M X , we have ® 2 2 2 = U + γj,0 1n + γj,1 − βbj X j ≥ U with equality if and only if γj,0 = 0 & γj,1 = βbj . 1 vektor jtých parciálních reziduí k 10.2. CORRECT REGRESSION FUNCTION 180 Shifted partial residuals Notation (Response, regressor and partial residuals means). Let n Y = n 1X Yi , n j X = i=1 1X Xi,j , n n U part,j i=1 = 1 X part,j Ui . n i=1 If X 0 = 1n (model with intercept), we have 0= n X Ui = i=1 1 n n X n X Uipart,j + βbj Xi,j , i=1 Uipart,j i=1 n 1 X b Xi,j , = βj n i=1 U part,j j = βbj X . Especially for purpose of visualization by plotting the partial residuals against the regressors a shifted partial residuals are sometimes used. Note that this only changes the estimated intercept of the regression line of dependence of partial residuals on the regressor. Definition 10.2 Shifted partial residuals. A vector of jth response-mean partial residuals of model M is a vector j U part,j,Y = U part,j + Y − βbj X 1n . A vector of jth zero-mean partial residuals of model M is a vector j U part,j,0 = U part,j − βbj X 1n . Notes. • A mean of the response-mean partial residuals is the response sample mean Y , i.e., n 1 X part,j,Y Ui =Y. n i=1 • A mean of the zero-mean partial residuals is zero, i.e., n 1 X part,j,0 Ui = 0. n i=1 The zero-mean partial residuals are calculated by the R function residuals with its type argument being set to partial. 10.2. CORRECT REGRESSION FUNCTION 181 Notes (Use of partial residuals). A vector of partial residuals can be interpreted as a response vector from which we removed a possible effect of all remaining regressors. Hence, dependence of U part,j on X j shows • a net effect of the jth regressor on the response; • a partial effect of the jth regressor on the response which is adjusted for the effect of the remaining regressors. The partial residuals are then mainly used twofold: Diagnostic tool. As a (graphical) diagnostic tool, a scatterplot X j , U part,j is used. In case, the jth regressor is correctly included in the original regression model M, i.e., if no transformaj tion of the regressor X is required to achieve E Y Z ∈ M X , points in the scatterplot j part,j X ,U should lie along a line. Visualization. Property that the estimated slope of the regression line in a model U part,j ∼ X j is the same as the jth estimated regression coeffient in the multiple regression model Y ∼ X is also used to visualize dependence of the response of the jth regressor by showing a scatterplot X j , U part,j equipped by a line with zero intercept and slope equal to βbj . 10.2.2 Test for linearity of the effect To examine appropriateness of the linearity of the effect of the jth regressor X j on the response expectation E Y Z by a statistical test, we can use a test on submodel (per se, requires additional assumption of normality). Without loss of generality, assume that the jth regressor X j is the last column of the model matrix X and denote the remaining non-intercept columns of matrix X as X0 . That is, assume that X = 1n , X 0 , X j . Two classical choices of a pair model–submodel being tested in this context are the following. More general parameterization of the jth regressor Submodel is the model M with the model matrix X. The (larger) model is model Mg obtained by replacing column X j in the model matrix X by a matrix V such that Xj ∈ M V , rank V ≥ 2. That is, the model matrices of the submodel and the (larger) model are Submodel M: (Larger) model Mg : 1n , X0 , X j = X; 1n , X 0 , V . Classical choices of the matrix V are such that it corresponds to: (i) polynomial of degree d ≥ 2 based on the regressor X j ; 10.2. CORRECT REGRESSION FUNCTION 182 (ii) regression spline of degree d ≥ 1 based on the regressor X j . In this case, 1n ∈ V and hence for practical calculations, the larger model Mg is usually estimated while using a model matrix X0 , V that does not explicitely include the intercept term which is included implicitely. Categorization of the jth regressor upp Let −∞ < xlow < xupp < ∞ be chosen such that interval xlow j j , xj j X1,j , . . . , Xn,j of the jth regressor. That is, xlow < min Xi,j , j i covers the values max Xi,j < xupp j . i upp Let I1 , . . . , IH be H > 1 subintervals of xlow based on a grid j , xj xlow < λ1 < · · · < λH−1 < xupp j j . Let xh ∈ Ih , h = 1, . . . , H, be chosen representative values for each of the subintervals I1 , . . . , IH (e.g., their midpoints) and let X j,cut = X1j,cut , . . . , Xnj,cut be obtained by categorization of the jth regressor using the division I1 , . . . , IH and representatives x1 , . . . , xH , i.e., (i = 1, . . . , n): Xij,cut = xh ≡ Xij ∈ Ih , h = 1, . . . , H. In this way, we obtained a categorical ordinal regressor X j,cut whose values x1 , . . . , xH , can be considered as collapsed values of the original regressor X j . Consequently, if linearity with respect to the original regressor X j holds then it also does (approximately, depending on chosen division I1 , . . . , IH and the representatives x1 , . . . , xH ) with respect to the ordinal categorical regressor X j,cut if this is viewed as numeric one. Let V be an n × (H − 1) model matrix corresponding to some (pseudo)contrast parameterization of the covariate X j,cut if this is viewed as categorical with H levels. We have X j,cut ∈ M V , and test for linearity of the jth regressor is obtained by considering the following model matrices in the submodel and the (larger) model: Submodel M: (Larger) model Mg : 1n , X0 , X j,cut ; 1n , X0 , V . Drawback of tests for linearity of the effect Remind that hypothesis of linearity of the effect of the jth regressor always forms the null hypothesis of the proposed submodel tests. Hence we are only able to confirm non-linearity of the effect (if the submodel is rejected) but are never able to confirm linearity. 10.3. HOMOSCEDASTICITY 10.3 183 Homoscedasticity We are again assuming a linear model M : Y Z ∼ Xβ, σ 2 In , that written using the error terms is M : Y = Xβ + ε, E ε Z = E ε = 0n , var ε Z = var ε = σ 2 In . The assumption (A2) of homoscedasticity is, in particular, var Y Z = σ 2 In , var ε Z = var ε = σ 2 In , where σ 2 is unknown but most importantly constant. 10.3.1 Tests of homoscedasticity Many tests of homoscedasticity can be found in literature. They mostly consider the following null and alternative hypotheses: H0 : σ 2 = const, H1 : σ 2 = certain function of some factor(s). A particular test is then sensitive (powerful) to detect heteroscedasticity if this expresses itself such that the variance σ 2 is the certain function of the factor(s) as specified by the alternative hypothesis. The test is possibly weak to detect heteroscedasticity (weak to reject the null hypothesis of homoscedasticity) if heteroscedasticity expresses itself in a different way compared to the considered alternative hypothesis. 10.3.2 Score tests of homoscedasticity A wide range of tests of homoscedasticity can be derived by assuming a (full-rank) normal linear model, basing the alternative hypothesis on a particular general linear model and then using an (asymptotic) maximum-likelihood theory to derive a testing procedure. Assumptions. For the rest of this section, we assume that model M (model under the null hypothesis) is normal of full-rank, i.e., M : Y Z ∼ Nn Xβ, σ 2 In , rank Xn×k = k, and an alternative model is a generalization of a general normal linear model Mhetero : Y Z ∼ Nn Xβ, σ 2 W−1 , where W = diag(w1 , . . . , wn ), wi−1 = τ (λ, β, Z i ), i = 1, . . . , n, τ is a known function of λ ∈ Rq , β ∈ Rk (regression coefficients), z ∈ Rp (covariates) such that τ (0, β, z) = 1, for all β ∈ Rk , z ∈ Rp . 10.3. HOMOSCEDASTICITY 184 Model Mhetero is then a model with unknown parameters β, λ, σ 2 which with λ = 0 simplifies into model M. In other words, model M is a nested 2 model of model Mhetero and a test of homoscedasticity corresponds to testing H0 : λ = 0, H1 : λ = 6 0. (10.2) Having assumed normality, both models M and Mhetero are fully parametric models and a standard (asymptotic) maximum-likelihood theory can now be used to derive a test of (10.2). A family of score tests based on specific choices of the weight function τ is derived by Cook and Weisberg (1983). Breusch-Pagan test A particular score test of homoscedasticity was also derived by Breusch and Pagan (1979) who consider the following weight function (x = tX (z) is a transformation of the original covariates that determines the regressors of model M). τ (λ, β, z) = τ (λ, β, x) = exp λ x> β . That is, under the heteroscedastic model, for i = 1, . . . , n, 2 exp λ E Y Z var Yi Z i = var εi Z i = σ 2 exp λ X > β = σ i i , i (10.3) and the test of homoscedasticity is testing H0 : λ = 0, H1 : λ = 6 0. It is seen from the model (10.3) that the Breusch-Pagan test is sensitive (powerful to detect heteroscedasticity) if the residual variance is a monotone function of the response expectation. Note (One-sided tests of homoscedasticity). In practical situations, if it can be assumed that the residual variance is possibly a monotone function of the response expectation then it can mostly be also assumed that it is its increasing function. A more powerful test of homoscedasticity is then obtained by considering the one-sided alternative H1 : λ > 0. Analogously, a test that is sensitive towards alternative of a residual variance which decreases with the response expectation is obtained by considering the alternative H1 : λ < 0. Note (Koenker’s studentized Breusch-Pagan test). The original Breusch-Pagan test is derived using standard maximum-likelihood theory while departing from assumption of a normal linear model. It has been shown in the literature that the test is not robust towards non-normality. For this reason, Koenker (1981) derived a slightly modified version of the Breusch-Pagan test which is robust towards non-normality. It is usually referred to as (Koenker’s) studentized Breusch-Pagan test and its use is preferred to the original test. 2 vnořený 10.3. HOMOSCEDASTICITY 185 Linear dependence on the regressors Let tW : Rp −→ Rq be a given transformation, w := tW (z), W i = tW (Z i ), i = 1, . . . , n. The following choice of the weight function can be considered: τ (β, λ, z) = τ (λ, w) = exp λ> w . That is, under the heteroscedastic model, for i = 1, . . . , n, = var Yi Z i = var εi Z i = var εi On a log-scale: σ 2 exp λ> W i . log var Yi Z i = log(σ 2 ) + λ> W i . | {z } λ0 In other words, the residual variance follows on a log-scale a linear model with regressors given by vectors W i . If tW is a univariate transformation leading to w = tW (z), one-sided alternatives are again possible reflecting assumption that under heteroscedasticity, the residual variance increases/decreases with a value of W = tW (Z). The most common use is then such that tW (z) and related values of W1 = tW (Z 1 ), . . ., Wn = tW (Z n ) correspond to one of the (non-intercept) regressors from either the model matrix X (regressors included in the model), or from the matrix V that contains regressors currently not included in the model. The corresponding score test of homoscedasticity then examines whether the residual variance changes/increases/decreases (depending on chosen alternative) with that regressor. Note (Score tests of homoscedasticity in the R software). In the R software, the score tests of homoscedasticity are provided by functions: (i) ncvTest (abbreviation for a “non-constant variance test”) from package car; (ii) bptest from package lmtest. The Koenker’s studentized variant of the test is only possible with the bptest function. 10.3.3 Some other tests of homoscedasticity Some other tests of homoscedasticity that can be encountered in practice include the following Goldfeld-Quandt test is an adaptation of a classical F-test of equality of the variances of the two independent samples into a regression context proposed by Goldfeld and Quandt (1965). It is applicable in linear models with both numeric and categorical covariates and under the alternative, heteroscedasticity is expressed by a monotone dependence of the residual variance on a prespecified ordering of the observations. G-sample tests of homoscedasticity are tests applicable for linear models with only categorical covariates (ANOVA models). They require repeated observations for each combination of values of the covariates and basically test equality of variances of G independent random samples. The most common tests of this type include: Bartlett test by Bartlett (1937) which, however, is quite sensitive towards non-normality and hence its use is not recommended. It is implemented in the R function bartlett.test; 10.3. HOMOSCEDASTICITY 186 Levene test by Levene (1960), implemented in the R function leveneTest from package car or in the R function levene.test from package lawstat; Brown-Forsythe test by Brown and Forsythe (1974) which is a robustified version of the Levene test and is implemented in the R function levene.test from package lawstat; Fligner-Killeen test by Fligner and Killeen (1976) which is implemented in the R function fligner.test. End of Lecture #18 (03/12/2015) 10.4. NORMALITY 10.4 187 Normality Start of Lecture #19 (03/12/2015) In this section, we are assuming a normal linear model M : Y Z ∼ Nn Xβ, σ 2 In , rank(X) = r, that written using the error terms is i.i.d. M : Yi = X > i β + εi , εi ∼ N (0, σ 2 ), i = 1, . . . , n. (10.4) Our interest now lies in verifying assumption (A4) of normality of the error terms εi , i = 1, . . . , n. Let us remind our standard notation needed in this section: (i) Hat matrix (projection matrix into the regression space M X ): − H = X X> X X> = hi,t i,t=1,...,n ; (ii) Projection matrix into the residual space M X ⊥ : M = In − H = mi,t i,t=1,...,n ; (iii) Residuals: U = Y − Yb = MY = U1 , . . . , Un ; 2 (iv) Residual sum of squares: SSe = U ; (v) Residual mean square: MSe = 1 n−r SSe . (vi) Standardized residuals: U std = U1std , . . . , Unstd , where Ui Uistd = p , MSe mi,i i = 1, . . . , n (if mi,i > 0). Lemma 10.3 Property of normal distribution. Let Z ∼ Nn (0, σ 2 In ). Let T : Rn −→ R be a measurable function satisfying T (cz) = T (z) for all c > 0 and z ∈ Rn . The random variables T (Z) and kZk are then independent. Proof. Proof/calculations were skipped and are not requested for the exam. Proof/calculations below are shown only for those who are interested. • Consider spherical coordinates: Z1 = R cos(φ1 ), Z2 = R sin(φ1 ) cos(φ2 ), Z3 = R sin(φ1 ) sin(φ2 ) cos(φ3 ), .. . Zn−1 = R sin(φ1 ) · · · sin(φn−2 ) cos(φn−1 ), Zn = R sin(φ1 ) · · · sin(φn−2 ) sin(φn−1 ). 10.4. NORMALITY 188 • Distance from origin: R = kZk. > • Direction: φ = φ1 , . . . , φn−1 . • Exercise for the 3rd year bachelor students: If Z ∼ Nn (0, σ 2 In ) then distance R from the origin and direction φ are independent. • R = kZk (distance from origin itself), T (Z) depends on the direction only (since T (Z) = T (cZ) for all c > 0) and hence kZk and T (Z) are independent. k Theorem 10.4 Moments of standardized residuals under normality. Let Y X ∼ Nn Xβ, σ 2 In and let for chosen i ∈ {1, . . . , n}, mi,i > 0. Then E Uistd X = 0, var Uistd X = 1. Proof. Proof/calculations were available on the blackboard in K1. Notes. k If the normal linear model (10.4) holds then Theorems 3.1 and 10.4 provide: (i) For (raw) residuals: U Z ∼ Nn 0n , σ 2 M . That is, the (raw) residuals follow also a normal distribution, nevertheless, the variances of the individual residuals U1 , . . . , Un differ (a diagonal of the projection matrix M is not necessarily constant). On top of that, the residuals are not necessarily independent (the projection matrix M is not necessarily a diagonal matrix). (ii) For standardized residuals (if mi,i > 0 for all i = 1, . . . , n, which is always the case in a full-rank model): E Uistd Z = 0, var Uistd Z = 1, i = 1, . . . , n. That is, the standardized residuals have the same mean and also the variance but are neither necessarily normally distributed nor necessarily independent. In summary, in a normal linear model, neither the raw residuals, nor standardized residuals form a random sample (a set of i.i.d. random variables) from a normal distribution. 10.4.1 Tests of normality There exist formal tests of the null hypothesis on a normality of the error terms: H0 : distribution of ε1 , . . . , εn is normal, (10.5) 10.4. NORMALITY 189 where a distribution of the test statistic is exactly known under the null hypothesis of normality. Nevertheless, those tests have quite a low power and hence are only rarely used in practice. In practice, approximate approaches are used that apply standard tests of normality on either the raw residuals U or the standardized residuals U std (both of them, under the null hypothesis (10.5), do not form a random sample from the normal distribution ). Several empirical studies showed that such approaches maintain quite well a significance level of the test on a requested value. At the same time, they mostly recommend to use the raw residuals U rather than the standardized residuals U std . Classical tests of normality include the following: Shapiro-Wilk test implemented in the R function shapiro.test. Lilliefors test implemented in the R function lillie.test from package nortest. Anderson-Darling test implemented in the R function ad.test from package nortest. 10.5. UNCORRELATED ERRORS 10.5 190 Uncorrelated errors In this section, we are assuming a linear model M: Yi = X > i β + εi , E εi X i = E εi = 0, var εi X i = var εi = σ 2 , i = 1, . . . , n, (10.6) i 6= l. cor(εi , εl ) = 0, Our interest now lies in verifying assumption (A3) of whether the error terms εi , i = 1, . . . , n, are uncorrelated. The fact that errors are uncorrelated often follows from a design of the study/data collection (measurements on independently behaving units, . . . ) and then there is no need to check this assumption. Situation when uncorrelated errors cannot be taken for granted is if the observations are obtained sequentially. Typical examples are (i) time series (time does not have to be a covariate of the model) which may lead to so called serial depedence among the error terms of the linear model; (ii) repeated measurements performed using one measurement unit or on one subject. In the following, we introduce a classical procedure that is used to test a null hypothesis of uncorrelated errors against alternative of serial dependence expressed by the first order autoregressive process. 10.5.1 Durbin-Watson test Assumptions. It is assumed that the ordering of the observations expressed by their indeces 1, . . . , n, has a practical meaning and may induce depedence between the error terms ε1 , . . . , εn of the model. One of the simplest stochastic processes that capture a certain form of serial dependence is the first order autoregressive process AR(1). Assuming this for the error terms ε1 , . . . , εn of the linear model (10.6) leads to a more general model MAR : Yi = X > i β + εi , i = 1, . . . , n, ε1 = η1 , εi = % εi−1 + ηi , i = 2, . . . , n, E ηi X i = E ηi = 0, var ηi X i = var ηi = σ 2 , i = 1, . . . , n, i 6= l, cor(ηi , ηl ) = 0, where −1 < % < 1 is additional unknown parameter of the model. It has been shown in the course Stochastic Processes 2 (NMSA409): • ε1 , . . . , εn is a stacionary process if and only if −1 < % < 1. Notes. • For each m ≥ 0: cor(εi , εi−m ) = %m , i = m + 1, . . . , n. In particular % = cor(εi , εi−1 ), i = 2, . . . , n. (10.7) 10.5. UNCORRELATED ERRORS 191 Test of uncorrelated errors in model M can be now be based on testing H0 : H1 : % = 0, % 6= 0 in model MAR . Since positive autocorrelation (% > 0) is more common in practice, one-sided tests (with H1 : % > 0) are used frequently as well. Let U = U1 , . . . , Un be residuals from model M which corresponds to the null hypothesis. A test statistic proposed by Durbin and Watson (1950, 1951, 1971) takes a form n X (Ui − Ui−1 )2 DW = i=2 n X . Ui2 i=1 A testing procedure is based on observing that a statistic DW is approximately equal to 2 (1 − %b), where %b is an estimator of the autoregression parameter % from model MAR . Calculations. First remember that E(Ui ) := E Ui X = 0, i = 1, . . . , n, and this property is maintained even if the error terms of the model are not uncorrelated (see process of the proof of Theorem 2.3). As residuals can be considered as predictions of the error terms ε1 , . . . , εn , a suitable estimator of their covariance of lag 1 is c εl , εl−1 σ b1,2 = cov n 1 X = Ui Ui−1 . n−1 i=2 Similarly, three possible estimators of the variance σ 2 of the error terms ε1 , . . . , εn are c εl = σ b2 = var n−1 1 X 2 Ui n−1 or i=1 n 1 X 2 Ui n−1 or i=2 n 1 X 2 Ui . n i=1 Then, Pn DW = (Ui − Ui−1 )2 i=2P = n 2 i=1 Ui Pn 2 i=2 Ui + Pn 2 i=2 Ui−1 − Pn 2 i=1 Ui 2 Pn i=2 Ui Ui−1 σ b2 + σ b2 − 2 σ b1,2 σ b1,2 =2 1− 2 ≈ σ b2 σ b = 2 1 − %b . Use of the test statistic DW for tests of H0 : % = 0 is complicated by the fact that distribution of DW under the null hypothesis depends on the model matrix X. It is hence not possible to derive (and tabulate) critical values in full generality. In practice, two approaches are used to calculate approximate critical values and p-values: 10.5. UNCORRELATED ERRORS 192 (i) Numerical algorithm of Farebrother (1980, 1984) which is implemented in the R function dwtest from package lmtest; (ii) General simulation method bootstrap (introduced by Efron, 1979) whose use for the DurbinWatson test is implemented in the R function durbinWatsonTest from package car. 10.6. TRANSFORMATION OF RESPONSE 10.6 193 Transformation of response Especially in situations when homoscedasticity and/or normality does not hold, it is often possible to achieve a linear model where both those assumptions are fulfilled by a suitable (non-linear) transformation t : R −→ R of the response. That is, it is worked with a normal linear model t(Yi ) = X > i β + εi , i = 1, . . . , n, i.i.d. εi ∼ N (0, σ 2 ), (10.8) where it is assumed that both homoscedasticity and normality hold. Disadvantage of a model with transformed response is that the corresponding regression function m(x) = x> β provides a model for expectation of the transformed response and not of the original response, i.e., for x ∈ X (sample space of the regressors): m(x) = E t(Y ) X = x 6= t E Y X = x , unless the transformation t is a linear function. Similarly, regression coefficients have now interpretation of an expected change of the transformed response t(Y ) related to a unity increase of the regressor. 10.6.1 Prediction based on a model with transformed response Nevertheless, the above mentioned interpretational issue is not a problem in a situation when prediction of a new value of the response Ynew , given X new = xnew , is of interest. If this is the case, we can base the prediction on the model (10.8) for the transormed response. In the following, we assume that t is strictly increasing, nevertheless, the procedure can be adjusted for decreasing or even non-monotone t as well: trans,L b trans,U trans and a (1 − α) 100% prediction interval Y bnew for • Construct a prediction Ybnew , Ynew trans Ynew = t(Ynew ) based on the model (10.8). • Trivially, an interval −1 trans,L −1 trans,U L U Ybnew , Ybnew = t Ybnew , t Ybnew (10.9) covers a value of Ynew with a probability of 1 − α. trans lies inside the prediction interval (10.9) and can be considered • A value Ybnew = t−1 Ybnew L , Y bU as a point prediction of Ynew . Only note that the prediction interval Ybnew new is not necessarily centered around a value of Ybnew . 10.6.2 Log-normal model Suitably interpretable model is obtained if the response is logarithmically transformed. Suppose that the following model (normal linear model for log-transformed response) holds: log(Yi ) = X > i β + εi , i = 1, . . . , n, i.i.d. εi ∼ N (0, σ 2 ). We then have Yi = exp X > i β ηi , i.i.d. exp(εi ) = ηi ∼ LN (0, σ 2 ), i = 1, . . . , n, (10.10) 10.6. TRANSFORMATION OF RESPONSE 194 where LN (0, σ 2 ) denotes a log-normal distribution with location parameter 0 and a scale parameter σ. That is, under validity of the model (10.10) for the log-transformed response, errors in a model for the original response are combined multiplicatively with the regression function. We can easily calculate the first two moments of the log-normal distribution which provides (for i = 1, . . . , n), σ2 M := E ηi = exp > 1 (with σ 2 > 0), 2 V := var ηi = exp(σ 2 ) − 1 exp(σ 2 ). Hence, for x ∈ X : E Yi X i = x = M exp x> β , var Yi X i = x = V exp 2 x> β = V · E Yi X i = x 2 . M (10.11) A log-normal model (10.10) is thus suitable in two typical situations that cause non-normality and/or heteroscedasticity of a linear model for the original response Y : (i) a conditional distribution of Y given X = x is skewed. If this is the case, the log-normal distribution which is skewed as well may provide a satisfactory model for this distribution. (ii) a conditional variance var Y X = x increases with a conditional expectation E Y X = x . This feature is captured by (10.11). Indeed, under the by the log-normal model as shown log-normal model, var Y X = x increases with E Y X = x . It is then said that the logarithmic transformation stabilizes the variance. Interpretation of regression coefficients With a log-normal model (10.10), the (non-intercept) regression coefficients have the following interpretation. Let for j ∈ {1, . . . , k − 1}, x = x0 , . . . , xj . . . , xk−1 ∈ X , and xj(+1) := x0 , . . . , xj + 1 . . . , xk−1 ∈ X , and suppose that β = β0 , . . . , βk−1 We then have > E Y X = xj(+1) M exp xj(+1) β = = exp(βj ). E Y X = x M exp x> β Notes. • If ANOVA linear model with log-transformed response is fitted, estimated differences between the group means of the log-response are equal to estimated log-ratios between the group means of the original response. • If a linear model with logarithmically transformed response if fitted, estimated regression coefficients, estimates of estimable parameters etc. and corresponding confidence intervals are often reported back-transformed (exponentiated) due to above interpretation. 10.6. TRANSFORMATION OF RESPONSE 195 Evaluation of impact of the regressors on response Evaluation of impact of the regressors on response requires necessity to perform statistical tests on regression coefficients or estimable parameters of a linear model. Homoscedasticity and for small samples also normality are needed to be able to use standard t- or F-tests. Both homoscedasticity and normality can be achieved by a log transformation of the response. Consequently performed statistical tests still have a reasonable practical interpretation as tests on ratios of two expectations of the (original) response. Chapter 11 Consequences of a Problematic Regression Space As in Chapter 10, that data are represented by n random vectors Yi , Z i , Z i = we assume Zi,1 , . . . , Zi,p ∈ Z ⊆ Rp i = 1, . . . , n. As usual, let Y = Y1 , . . . , Yn and let Zn×p denote a matrix with covariate vectors Z 1 , . . . , Z n in its rows. Finally, let X i , i = 1, . . . , n, where X i = tX (Z i ) for some transformation tX : Rp −→ Rk , be the regressors that give rise to the model matrix X> 1 . 0 .. = X , . . . , X k−1 . Xn×k = > Xn It will be assumed that X 0 = 1, . . . , 1 (almost surely) leading to the model matrix Xn×k = 1n , X 1 , . . . , X k−1 , with explicitely included intercept term. Primarily, to be able to assume that E Y Z = we will assume that the model matrixX is sufficient E Y X = Xβ for some β = β0 , . . . , βk−1 ∈ Rk . That is, we will arrive from assuming Y Z ∼ Xβ, σ 2 In . It will finally be assumed in the whole chapter that the model matrix X is of full rank, i.e., rank X = k < n. 196 11.1. MULTICOLLINEARITY 11.1 197 Multicollinearity A principal assumption of any regression model is correct specification of the function. regression 2 While assuming a linear model Y Z ∼ Xβ, σ In , this means that E Y Z ∈ M X . To guarantee this, it seems to be optimal to choose the regression space M X as rich as possible. In other words, if many covariates are available, it seems optimal to include a high number k of columns in the model matrix X. Nevertheless, as we show in this section, this approach bears certain complications. 11.1.1 Singular value decomposition of a model matrix We are assuming rank Xn×k = k < n. As was shown in the course Fundamentals of Numerical Mathematics (NMNM201), the matrix X can be decomposed as X = U D V> = k−1 X dj uj v > j , D = diag(d0 , . . . , dk−1 ), j=0 where • Un×k = u0 , . . . , uk−1 are the first k orthonormal eigenvectors of the n × n matrix XX> . • Vk×k = v 0 , . . . , v k−1 are (all) orthonormal eigenvectors of the k × k (invertible) matrix X> X. p • dj = λj , j = 0, . . . , k − 1, where λ0 ≥ · · · ≥ λk−1 > 0 are • the first k eigenvalues of the matrix XX> ; • (all) eigenvalues of the matrix X> X, i.e., > X X= k−1 X > λj v j v > j = VΛV , Λ = diag(λ0 , . . . , λk−1 ) j=0 = k−1 X 2 > d2j v j v > j = VD V . j=0 The numbers d0 ≥ · · · ≥ dk−1 > 0 are called singular values1 of the matrix X. We then have k−1 X −1 1 > −2 > X> X = V , 2 vj vj = V D d j j=0 tr n X> X −1 o k−1 X 1 = . d2 j=0 j (11.1) Note (Moore-Penrose pseudoinverse of the matrix X> X). The singular value decomposition of the model matrix X provides also a way to calculate the MoorePenrose pseudoinverse of the matrix X> X if X is of less-than-full rank. If rank Xn×k = r < k, then d0 ≥ · · · ≥ dr−1 > dr = · · · = dk−1 = 0. The Moore-Penrose pseudoinverse of X> X is obtained as r−1 + X 1 > X> X = 2 vj vj . d j=0 j 1 singulární hodnoty 11.1. MULTICOLLINEARITY 11.1.2 198 Multicollinearity and its impact on precision of the LSE It is seen from (11.1) that with dk−1 −→ 0: (i) the matrix X> X tends to a singular matrix, i.e., the columns of the model matrix X tend to being linearly dependent; n −1 o (ii) tr X> X −→ ∞. Situation when the columns of the (full-rank) model matrix X are close to being linearly dependent is referred to as multicollinearity. If a linear model Y Z ∼ Xβ, σ 2 In , rank(Xn×k ) = k is assumed, then we know from GaussMarkov theorem that −1 (i) The fitted values Yb = Yb1 , . . . , Ybn = HY , where H = X X> X X> , is the best linear unbiased estimator (BLUE) of a vector parameter µ = Xβ = E Y Z with var Yb Z = σ 2 H; b = βb0 , . . . , βbk−1 = X> X −1 X> Y is the BLUE of a vector (ii) The least squares estimator β of regression coefficients β with b Z = σ 2 X> X −1 . var β It then follows n X var Ybi Z = tr var Yb Z = tr σ 2 H = σ 2 tr(H) = σ 2 k, i=1 k−1 X n n −1 o −1 o 2 > 2 > b b = σ tr X X . var βj Z = tr var β Z = tr σ X X j=0 This shows that multicollinearity (i) does not have any impact on precision of the LSE of the response expectation µ = Xβ; (ii) may have a serious impact on precision of the LSE of the regression coefficients β. At the same time, since LSE is BLUE, there exist no better linear unbiased estimator of β. If additionally normality is assumed there even exist no better unbiased estimator at all. An impact of multicollinearity can also be expressed by considering a problem of estimating the squared Euclidean norm of µ = Xβ and β, respectively. Asnatural 2 estimators 2 of those squared b norms are the squared norms of the corresponding LSE’s, i.e., Y and b β , respectively. As we show, those estimators are biased, the amount of bias nevertheless, does 2 not depend on a degree 2 b b of multicollinearity in case of Y but depends on it in case of β . 11.1. MULTICOLLINEARITY 199 Lemma 11.1 Bias in estimation of the squared norms. Let Y Z ∼ Xβ, σ 2 In , rank(Xn×k ) = k. The following then holds. h 2 i 2 E Yb − Xβ Z = σ 2 k, h n 2 i −1 o 2 E b β − β Z = σ 2 tr X> X . Proof. In accordance with our convention, condition will be omitted from notation of all expectations and variances. Nevertheless, all are still understood as conditional expectations and variances given the covariate values Z. h 2 i 2 b E Y − Xβ Z • Let us calculate: n n nX 2 2 o X EYb − Xβ = E Ybi − X > β = var Ybi i i=1 i=1 n o = tr σ 2 H = σ 2 tr(H)= σ 2 k. = tr var Yb • At the same time: 2 > EYb − Xβ = E Yb − Xβ Yb − Xβ 2 2 EYb = EYb + EXβ − 2 β > X> |{z} Xβ 2 2 2 2 2 = EYb + Xβ − 2 Xβ = EYb − Xβ . 2 2 • So that, EYb − Xβ 2 EYb = = σ 2 k, 2 Xβ + σ 2 k. h 2 i 2 E b β − β Z • Let us start in a similar way: k−1 k−1 nX 2 2 o X b b E β−β =E βj − βj = var βbj j=0 j=0 n n n o o o b = tr σ 2 X> X −1 = σ 2 tr X> X −1 . = tr var β 11.1. MULTICOLLINEARITY 200 • At the same time: 2 b −β > β b −β Eb β − β = E β 2 2 b β + Eβ − 2 β > Eβ = Eb |{z} β 2 2 2 2 2 = Eb β + β − 2 β = Eb β − β . 2 2 • So that, Eb β − β 2 Eb β = = n −1 o σ 2 tr X> X , n 2 o β + σ 2 tr X> X −1 . {z } | k−1 X var βbj j=0 11.1.3 k Variance inflation factor and tolerance Notation. For a given linear model Y Z ∼ Xβ, σ 2 In , rank(Xn×k ) = k, where Y = Y1 , . . . , Yn , > X = 1n , X 1 , . . . , X k−1 , X j = X1,j , . . . , Xn,j , j = 1, . . . , k − 1, the following (partly standard) notation, will be used: n Response sample mean: Square root of the total sum of squares: 1X Y = Yi ; n i=1 v u n uX 2 TY = t Yi − Y = Y − Y 1n ; i=1 Fitted values: Yb = Yb1 , . . . , Ybn ; Coefficient of determination: Y − Yb 2 Y − Yb 2 R = 1 − . = 1 − Y − Y 1n 2 TY2 Residual mean square: MSe = 2 1 Y − Yb 2 . n−k Further, for each j = 1, . . . , k − 1, consider a linear model Mj , where the vector X j acts as a response and the model matrix is X(−j) = 1n , X 1 , . . . , X j−1 , X j+1 , . . . , X k−1 . The following notation will be used: End of Lecture #19 (03/12/2015) Start of Lecture #20 (10/12/2015) 11.1. MULTICOLLINEARITY 201 n Column sample mean: 1X X = Xi,j ; n j i=1 Square root of the total sum of squares from model Mj : v u n uX j 2 j Tj = t = X j − X 1n ; Xi,j − X i=1 Fitted values from model Mj : cj = X b1,j , . . . , X bn,j ; X Coefficient of determination from model Mj : j j cj 2 cj 2 X − X X − X 2 Rj = 1 − . = 1 − Tj2 X j − X j 1n 2 Notes. (i) If data (response random variables and non-intercept covariates) Yi , Xi,1 , . . . , Xi,k−1 , i = 1, . . . , n are a random sample from a distribution of a generic random vector Y, X1 , . . . , Xk−1 then • The coefficient of determination R2 is also a squared value of a sample coefficient of multiple correlation between Y and X := X1 , . . . , Xk−1 . • For each j = 1, . . . , k − 1, the coefficient of determination Rj2 is also a squared value of a sample coefficient of multiple correlation between Xj and X (−j) := X1 , . . . , Xj−1 , Xj+1 , . . . , Xk−1 . (ii) For given j = 1, . . . , k − 1: • A value of Rj2 close to 1 means that the jth column X j is almost equal to some linear combination of the columns of the matrix X(−j) (remaining columns of the model matrix). We then say that X j is collinear with the remaining columns of the model matrix. • A value of Rj2 = 0 means that • the column X j is orthognal to all remaining non-intercept regressors (non-intercept columns of the matrix X(−j) ); • the jth regressor represented by a random variable Xj is multiply uncorrelated with the remaining regressors represented by the random vector X (−j) . For a given linear model Y Z ∼ Xβ, σ 2 In , rank(Xn×k ) = k, b Z = MSe X> X −1 . c β var −1 The following Theorem shows that diagonal elements of the matrix MSe X> X , i.e., values c βbj Z can also be calculated, for j = 1, . . . , k − 1, using above defined quantities TY , Tj , R2 , var Rj2 . Theorem 11.2 Estimated variances of the LSE of the regression coefficients. For a given linear model Y Z ∼ Xβ, σ 2 In , rank(Xn×k ) = k, diagonal elements of the matrix c βbj Z are, for j = 1, . . . , k − 1, var 2 1 − R2 1 TY c βbj Z = · · . var Tj n−k 1 − Rj2 11.1. MULTICOLLINEARITY Proof. See Zvára (2008, Chapter 11). Proof/calculations were skipped and are not requested for the exam. 202 k Definition 11.1 Variance inflation factor and tolerance. For given j = 1, . . ., k − 1, the variance inflation factor2 and the tolerance3 of the jth regressor of the linear model Y Z ∼ Xβ, σ 2 In , rank(Xn×k ) = k are values VIFj and Tolerj , respectively, defined as 1 1 VIFj = . , Tolerj = 1 − Rj2 = 2 VIFj 1 − Rj Notes. • With Rj = 0 (the jth regressor orthogonal to all remaining regressors, the j regressor multiply uncorrelated with the remaining ones), VIFj = 1. • With Rj −→ 1 (the jth regressor collinear with the remaining regressors, the jth regressor almost perfectly multiply correlated with the remaining ones), VIFj −→ ∞. Interpretation and use of VIF • If we take into account the statement of Theorem 11.2, the VIF of the jth regressor (j = 1, . . . , k − 1) can be interpreted as a factor by which the (estimated) variance of βbj is multiplied (inflated) compared to an optimal situation when the jth regressor is orthogonal to (multiply uncorrelated with) the remaining regressors included in the model. Hence the term variance inflation factor. • Under assumption of normality, the confidence interval for βj with a coverage of 1 − α has the lower and the upper bounds given as q α b c βbj . βj ± tn−k 1 − var 2 Using the statement of Theorem 11.2, the lower and the upper bounds of the confidence interval for βj can also be written as r α TY 1 − R2 p b βj ± tn−k 1 − VIFj . 2 Tj n−k That is, the (square root of) VIF also provides a factor by which the half-length (radius) of the confidence interval is inflated compared to an optimal situation when the jth regressor is orthogonal to (multiply uncorrelated with) the remaining regressors included in the model, namely, Volj 2 VIFj = , (11.2) Vol0,j 2 varianční inflační faktor 3 tolerance 11.1. MULTICOLLINEARITY where Volj = Vol0,j = 203 length (volume) of the confidence interval for βj ; length (volume) of the confidence interval for βj if it was Rj2 = 0. • Regressors with a high VIF are possibly responsible for multicollinearity. Nevertheless, the VIF does not reveal which regressors are mutually collinear. Generalized variance inflation factor Beginning of A generalized variance inflation factor was derived by Fox and Monette (1992) to evaluate a degree skipped part of collinearity between a specified group of regressors and the remaining regressors. Let • J ⊂ 1, . . . , k − 1 , J = m; • β [J ] be a subvector of β having the elements indexed by j ∈ J . Under normality, a confidence ellipsoid for βJ with a coverage 1 − α is n β [J ] ∈ Rm : b β [J ] − β [J ] > MSe V[J ] −1 o b β [J ] − β < m F (1 − α) , m,n−k [J ] −1 V[J ] = (J − J ) block of the matrix X> X . (11.3) Let VolJ : Vol0,J : volume of the confidence ellipsoid (11.3); volume of the confidence ellipsoid (11.3) would all columns of X coresponding to β [J ] be orthogonal to the remaining colums of X. A definition of the generalized variance inflation factor gVIF is motivated by (11.2) as it is given as gVIFJ = VolJ Vol0,J 2 . It is seen that with J = {j} for some j = 1, . . . , k − 1, the generalized VIF simplifies into a standard VIF, i.e., gVIFj = VIFj . Notes. • The generalized VIF is especially useful if J relates to the regression coefficients corresponding to the reparameterizing (pseudo)contrasts of one categorical covariate. It can then be shown that gVIFJ does not depend on a choice of the (pseudo)contrasts. gVIFJ then evaluates the magnitude of the linear dependence of a categorical variable and the remaining regressors. • When comparing gVIF for index sets J , J of different cardinality m, quantities J 1 gVIF 2 m J = VolJ Vol0,J 1 m (11.4) should be compared which all relate to volume units in 1D. • Generalized VIF’s (and standard VIF’s if m = 1) together with (11.4) are calculated by the R function vif from the package car. End of skipped part 11.1. MULTICOLLINEARITY 11.1.4 204 Basic treatment of multicollinearity Especially in situations when inference on the regression coefficients is of interest, i.e., when the primary purpose of the regression modelling is to evaluate which variables influence significantly the response expectation and which not, multicollinearity is a serious problem. Basic treatment of multicollinearity consists of preliminary exploration of mutual relationships between all covariates and then choosing only suitable representatives of each group of mutually multiply correlated covariates. Very basic decision can be based on pairwise correlation coefficients. In some (especially “cook-book”) literature, rules of thumb are applied like “Covariates with a correlation (in absolute value) higher than 0.80 should not be included together in one model.” Nevertheless, such rules should never be applied in an automatic manner (why just 0.80 and not 0.79, . . . ?) Decision on which covariates cause multicollinearity can additionally be based on (generalized) variance inflation factors. Nevertheless, also those should be used comprehensively. In general, if a large set of covariates is available to relate it to the response expectation, a deep (and often timely) analyzis of mutual relationships and their understanding must preceed any regression modelling that is to lead to useful results. 11.2. MISSPECIFIED REGRESSION SPACE 11.2 205 Misspecified regression space We are often in a situation when a large (potentially enormous) number p of candidate regressors is available. The question is then which of them should be included in a linear model. As shown in Section 11.1, inclusion of all possible regressors in the model is not necessarily optimal and may even have seriously negative impact on the statistical inference we would like to draw using the linear model. In this section, we explore some (additional) properties of the least squares estimators and of the related prediction in two situations: (i) Omitted important regressors. (ii) Irrelevant regressors included in a model. 11.2.1 Omitted and irrelevant regressors We will assume that possibly two sets of regressors are available: (i) X i , i = 1, . . . , n, where X i = tX (Z i ) for some transformation tX : Rp −→ Rk . They give rise to the model matrix X> 1 . 0 .. = X , . . . , X k−1 . Xn×k = X> n It will still be assumed that X 0 = 1, . . . , 1 (almost surely) leading to the model matrix Xn×k = 1n , X 1 , . . . , X k−1 , with explicitely included intercept term. (ii) V i , i = 1, . . . , n, where V i = tV (Z i ) for some transformation tV : Rp −→ Rl . They give rise to the model matrix V> 1 . 1 .. = V , . . . , V l . Vn×l = > Vn We will assume that both matrices X and V are of a full column rank and their columns are linearly independent, i.e., we assume rank Xn×k = k, rank Vn×l = l, for Gn×(k+l) := X, V , rank G = k + l < n. The matrices X and G give rise to two nested linear models: Model MX Y Z ∼ Xβ, σ 2 In ; Model MXV Y Z ∼ Xβ + Vγ, σ 2 In . Depending on which of the two models is a correct one and which model is used for inference, we face two situations: 11.2. MISSPECIFIED REGRESSION SPACE 206 Omitted important regressors mean that the larger model MXV is correct (with γ 6= 0m ) but we base inference on model MX . In particular, • β is estimated using model MX ; • σ 2 is estimated using model MX ; • prediction is based on the fitted model MX . Irrelevant regressors included in a model that the smaller model MX is correct but we base inference on model MXV . In particular, • β is estimated (together with γ) using model MXV ; • σ 2 is estimated using model MXV ; • prediction is based on the fitted model MXV . Note that if MX is correct then MXV is correct as well. Nevertheless, it includes redundant parameters γ which are known to be equal to zeros. Notation (Quantities derived under the two models). Quantities derived while assuming model MX will be indicated by subscript X, quantities derived while assuming model MXV will be indicated by subscript XV . Namely, (i) Quantities derived while assuming model MX : • Least squares estimator of β: b = X> X β X −1 X> Y = βbX,0 , . . . , βbX,k−1 ; ⊥ • Projection matrices into the regression space M X and into the residual space M X : −1 HX = X X> X X> , MX = In − HX ; • Fitted values (LSE of a vector Xβ): b = YbX,1 , . . . , YbX,n ; Yb X = HX Y = Xβ X • Residuals U X = MX Y = Y − Yb X = UX,1 , . . . , UX,n ; • Residual sum of squares and residual mean square: 2 SSe,X = U X , MSe,X = SSe,X . n−k (ii) Quantities derived while assuming model MXV : • Least squares estimator of β, γ : b β XV −1 > > b ,γ β G Y, XV b XV = G G b XV = γ γ bXV,1 , . . . , γ bXV,l ; = βbXV,0 , . . . , βbXV,k−1 , ⊥ • Projection matrices into the regression space M G and into the residual space M G : −1 HXV = G G> G G> , MXV = In − HXV ; 11.2. MISSPECIFIED REGRESSION SPACE 207 • Fitted values (LSE of a vector Xβ + Vγ): b b b Yb XV = HXV Y = Xβ XV + Vγ XV = YXV,1 , . . . , YXV,n ; • Residuals U XV = MXV Y = Y − Yb XV = UXV,1 , . . . , UXV,n ; • Residual sum of squares and residual mean square: 2 SSe,XV = U XV , MSe,XV = SSe,XV . n−k−l Consequence of Theorem 10.1: Relationship between the quantities derived while assuming the two models. Quantities derived while assuming models MX and MXV are mutually in the following relationships: −1 Yb XV − Yb X = MX V V> MX V V> U X , b b = X β −β + Vb γ , XV b XV = γ X V> MX V > b b β XV − β X = − X X −1 −1 XV V> U X , X> Vb γ XV , 2 γ XV , SSe,X − SSe,XV = MX Vb HXV = HX + MX V V> MX V −1 V> MX . Proof. Direct use of Lemma 10.1 while taking into account the fact that now, all involved model matrices are of full-rank. −1 > Relationship HXV = HX +MX V V> MX V V MX was shown inside the proof of Lemma 10.1. It easily follows from a general expression of the hat matrix if we realize that M X, V = M X, MX V , and that X> MX V = 0k×l . k Theorem 11.3 Variance of the LSE in the two models. Irrespective of whether MX or MXV holds, the covariance matrices of the fitted values and the LSE of the regression coefficients satisfy the following: 11.2. MISSPECIFIED REGRESSION SPACE 208 var Yb XV Z − var Yb X Z ≥ 0, b b var β XV Z − var β X Z ≥ 0. Notes. • Estimator of the response mean vector µ = E Y Z based on a (smaller) model MX is always (does not matter which model is correct) less or equally variable than the estimator based on the (richer) model MXV . • Estimators of the regression coefficients β based on a (smaller) model MX have always lower (or equal if X> V = 0k×m ) standard errors than the estimator based on the (richer) model MXV . Proof. In accordance with our convention, condition will be omitted from notation of all expectations and variances. Nevertheless, all are still understood as conditional expectations and variances given the covariate values Z. var Yb XV − var Yb X ≥ 0 var Yb X = var HX Y = HX (σ 2 In )HX We have, = σ 2 HX var Yb XV b var β XV (even if MX is not correct). = var HXV Y = σ 2 HXV = σ 2 HX + MX V(V> MX V)−1 V> MX = var Yb X + σ 2 MX V(V> MX V)−1 V> MX . {z } | positive semidefinite matrix b − var β X ≥0 Proof/calculations for this part were skipped and are not requested for the exam. Proof/calculations below are shown only for those who are interested. First, use a formula to calculate an inverse of a matrix divided into blocks (Theorem A.4): −1 n o−1 ! > X X> V > X − X> V V> V −1 V> X b X X β XV = σ2 . var = σ2 > > b XV γ V X V V V V V Further, −1 > −1 −1 > b var β = var X X X Y = X> X X> (σ 2 In )X X> X X −1 = σ 2 X> X (even if MX is not correct). b var β XV n o−1 −1 = σ 2 X> X − X> V V> V V> X . Property of positive definite matrices (“A − B ≥ 0 ⇔ B−1 − A−1 ≥ 0”) finalizes the proof. k 11.2. MISSPECIFIED REGRESSION SPACE 11.2.2 209 Prediction quality of the fitted model To evaluate a prediction quality of the fitted model, we will assume that data Y , Z , Zi = i i Zi,1 , . . . , Zi,p ∈ Z⊆ Rp , i = 1, . . . , n, are a random sample from a distribution of a generic random vector Y, Z , Z = Z1 , . . . , Zp . Let the conditional distribution Y | Z of Y given the covariates Z satisfies E Y Z = m(Z), var Y Z = σ 2 , (11.5) for some (regression) function m and some σ 2 > 0. Replicated response Let z 1 , . . . , z n be the values of the covariate vectors Z 1 , . . . , Z n in the original data that are available to estimate the parameters of the model (11.5). Further, let Yn+i , Z n+i , i = 1, . . . , n, be independent random vectors (new or future data) beingdistributed as a generic random vector Y, Z and being independent of the original data Yi , Z i , i = 1, . . . , n. Suppose that our aim is to predict values of Yn+i , i = 1, . . . , n, under the condition that the new covariate values are equal to the old ones. That is, we want to predict, for i = 1, . . . , n, values of Yn+i given Z n+i = z i . Terminology (Replicated response). A random vector Y new = Yn+1 , . . . , Yn+n , where Yn+i is supposed to come from the conditional distribution Y | Z = z i , i = 1, . . . , n, is called the replicated data. Notes. • The original (old) response vector Y and the replicated response vector Y new are assumed to be independent. • Both Y and Y new are assumed to be generated by the same conditional distribution (given Z), where E Y Z 1 = z1, . . . , Z n = zn = µ = E Y new Z n+1 = z 1 , . . . , Z n+n = z n , var Y Z 1 = z 1 , . . . , Z n = z n = σ 2 In = var Y new Z n+1 = z 1 , . . . , Z n+n = z n , for some σ 2 > 0, and µ = m(z 1 ), . . . , m(z n ) = µ1 , . . . , µn . Prediction of replicated response Let Yb new = Ybn+1 , . . . , Ybn+n be the prediction of a vector Y new based on the regression model (11.5) estimated using the original data Y . Analogously to Section 5.4.3, we shall evaluate a quality of the prediction by the mean squared error of prediction (MSEP). Nevertheless, in contrast to Section 5.4.3, the following issues will be different: 11.2. MISSPECIFIED REGRESSION SPACE 210 (i) A value of a random vector rather than a value of a random variable (as in Section 5.4.3) is predicted now. Now, the MSEP will be given as a sum of the MSEPs of the elements of the random vector being predicted. (ii) Since we are now interested in prediction of new response values given the covariate values being equal to the covariate values in the original data, the MSEP now will be based on a conditional distribution of the responses given Z (given Z i = Z n+i = z i , i = 1, . . . , n). In contrast, variability of the covariates was taken into account in Section 5.4.3. (iii) Variability of the prediction induced by estimation of the model parameters (estimation of the regression function) using the original data Y will also be taken into account now. In contrast, model parameters were assumed to be known when deriving the MSEP in Section 5.4.3. Definition 11.2 Quantification of a prediction quality of the fitted regression model. Prediction quality of the fitted regression model will be evaluated by the mean squared error of prediction (MSEP)4 defined as n h X 2 i MSEP Yb new = E Ybn+i − Yn+i Z , (11.6) i=1 where the expectation is with respect to the (n+n)-dimensional conditional distribution of Y , Y new given Z> Z> 1 n+1 . . . . Z= . = . . Z> Z> n n+n Additionally, we define the averaged mean squared error of prediction (AMSEP)5 as 1 AMSEP Yb new = MSEP Yb new . n Prediction of replicated response in a linear model End of Lecture #20 (10/12/2015) Start of With a linear model, it is assumed that m(z) = x> β for some (known) transformation x = tX (z) Lecture #21 (10/12/2015) and a vector of (unknown) parameters β. Hence, it is assumed that > µ = Xβ = x> 1 β, . . . , xn β = µ1 , . . . , µn , for a model matrix X based on the (transformed) covariate values xi = tX (z i ), i = 1, . . . , n. If we restrict our attention to unbiased and linear predictions of Y new , i.e., to predictions of the form Yb new = AY for some matrix A, a variant of the Gauss-Markov theorem would show that (11.6) is minimized for − Yb new = Yb , Yb = X X> X X> Y , Ybn+i = Ybi , 4 střední čtvercová chyba predikce 5 i = 1, . . . , n. průměrná střední čtvercová chyba predikce 11.2. MISSPECIFIED REGRESSION SPACE 211 That is, for Yb new being equal to the fitted values of the model estimated using the original data. Note also that b, Yb new = Yb =: µ b is the LSE of a vector µ = E Y Z 1 = z 1 , . . . , Z n = z n = E Y new Z n+1 = where µ z 1 , . . . , Z n+n = z n . Lemma 11.4 Mean squared error of prediction in a linear model. In a linear model, the mean squared error of prediction can be expressed as MSEP Yb new 2 = nσ + n X MSE Ybi , i=1 where h 2 i MSE Ybi = E Ybi − µi Z , i = 1, . . . , n, is the mean squared error6 of Ybi if this is viewed as estimator of µi , i = 1, . . . , n. Proof. In accordance with our convention, condition will be omitted from notation of all expectations and variances. Nevertheless, all are still understood as conditional expectations and variances given the covariate values Z. We have for i = 1, . . . , n (remember, Ybn+i = Ybi , i = 1, . . . , n), 2 2 E Ybn+i − Yn+i = E Ybi − Yn+i 2 = E Ybi − µi − (Yn+i − µi ) 2 2 = E Ybi − µi + E Yn+i − µi −2 E(Ybi − µi )(Yn+i − µi ) | {z } b b E(Yi − µi ) E(Yn+i − µi ) = E(Yi − µi ) · 0 = E Ybi − µi 2 + E Yn+i − µi 2 = MSE(Ybi ) + σ 2 . So that MSEP(Yb new ) = n X E Ybn+i − Yn+i i=1 2 = n σ2 + n X MSE Ybi . i=1 Notes. • We can also write n X h 2 i b b MSE Yi = E Y − µ Z . i=1 Hence, MSEP Yb new 6 střední čtvercová chyba h 2 i = n σ 2 + E Yb − µ Z . k 11.2. MISSPECIFIED REGRESSION SPACE 212 • If the assumed linear model is a correct model for data at hand, Gauss-Markov theorem states that Yb is the BLUE of the vector µ in which case h 2 i i = 1, . . . , n. MSE Ybi = E Ybi − µi Z = var Ybi Z , • Nevertheless, if the assumed linear model is not a correct model for data at hand, estimator Yb might be a biased estimator of the vector µ, in which case h 2 i MSE Ybi = E Ybi − µi Z o2 o2 n n , = var Ybi Z + bias Ybi = var Ybi Z + E Ybi − µi Z i = 1, . . . , n. • Expression of the mean squared error of prediction is MSEP(Yb new ) = n σ 2 + n X h 2 i MSE Ybi = n σ 2 + E Yb − µ Z . i=1 By specification of a model for the conditional response i.e., by specification of hexpectation, 2 i a model for µ, we can influence only the second factor E Yb − µ Z . The first factor (n σ 2 ) reflects the true (conditional) variability of the response which does not depend on specification of the model for the expectation. Hence, if evaluating a prediction quality of a linear model with respect to ability to predict replicated data, the only term that matters is n X h 2 i MSE Ybi = E Yb − µ Z , i=1 that relates to the error of the fitted values being considered as an estimator of the vector µ. 11.2.3 Omitted regressors In this section, we will assume that the correct model is model MXV : Y Z ∼ Xβ + Vγ, σ 2 In , with γ 6= 0l . Hence all estimators derived under model MXV are derived under the correct model and hence have usual properties of the LSE, namely, b E β = β, XV Z E Yb XV Z = Xβ + Vγ =: µ, n X i=1 MSE YbXV,i = n X b b var YXV,i Z = tr var Y XV Z = tr σ 2 HXV i=1 = σ 2 (k + l), E MSe,XV Z = σ 2 . (11.7) 11.2. MISSPECIFIED REGRESSION SPACE 213 Nevertheless, all estimators derived under model MX : Y Z ∼ Xβ, σ 2 In are calculated while assuming a misspecified model with omitted important regressors and their properties do not coincide with properties of the LSE calculated under the correct model. Theorem 11.5 Properties of the LSE in a model with omitted regressors. Let MXV : Y Z ∼ Xβ + Vγ, σ 2 In hold, i.e., µ := E Y Z satisfies µ = Xβ + Vγ for some β ∈ Rk , γ ∈ Rl . Then the least squares estimators derived while assuming model MX : Y Z ∼ Xβ, σ 2 In attain the following properties: b Z = β + X> X −1 X> Vγ, E β X E Yb X Z = µ − MX Vγ, n X MSE YbX,i 2 = k σ 2 + MX Vγ , i=1 MX Vγ 2 2 . E MSe,X Z = σ + n−k Proof. In accordance with our convention, condition will be omitted from notation of all expectations and variances. Nevertheless, all are still understood as conditional expectations and variances given the covariate values Z. b Z E β X −1 > > b b By Theorem 10.1: β X Vb γ XV . XV − β X = − X X n o −1 > > b b Hence, E β = E β + X X X Vb γ X XV XV −1 > > = β + X X X Vγ, b bias β X = X> X −1 X> Vγ. E Yb X Z b b γ XV . By Theorem 10.1: Yb XV − Yb X = X β XV − β X + Vb b b Hence, E Yb X = E Yb XV − Xβ γ XV XV + Xβ X − Vb −1 = µ − Xβ + Xβ + X X> X X> Vγ − Vγ n o −1 = µ + X X> X X> − In Vγ = µ − MX Vγ, bias Yb X = − MX Vγ. 11.2. MISSPECIFIED REGRESSION SPACE Pn i=1 MSE YbX,i 214 n > o b b b Let us first calculate MSE Y X = E Y X − µ Y X − µ : MSE Yb X = var Yb X + bias Yb X bias> Yb X = σ 2 HX + MX Vγγ > V> MX . Hence, n X MSE YbX,i = tr MSE Yb X i=1 = tr σ 2 HX + MX Vγγ > V> MX = tr σ 2 HX + tr MX Vγγ > V> MX = σ 2 k + tr γ > V> MX MX Vγ 2 = σ 2 k + MX Vγ . E MSe,X Z Proof/calculations for this part were skipped and are not requested for the exam. Proof/calculations below are shown only for those who are interested. Let us first calculate E SSe,X := E SSe,X Z . To do that, write the linear model MXV using the error terms as Y = Xβ + Vγ + ε, E ε = E ε Z = 0n , var ε = var ε Z = σ 2 In . 2 2 E SSe,X = EMX Y = EMX (Xβ + Vγ + ε) 2 = EMX Vγ + MX ε 2 2 = EMX Vγ + EMX ε + 2 E γ > V> MX MX ε | {z } γ > V> MX Eε=0 2 E ε> M X ε = MX Vγ + | {z } E tr(ε> MX ε) =tr E(MX εε> ) =tr σ 2 MX =σ 2 (n−k) 2 = MX Vγ + σ 2 (n − k). SSe,X Hence, E MSe,X = E n−k MX Vγ 2 = σ2 + , n−k MX Vγ 2 bias MSe,X = . n−k k 11.2. MISSPECIFIED REGRESSION SPACE 215 Least squares estimators −1 > > b b is not Theorem 11.5 shows that bias β X Vγ, nevertheless, the estimator β X = X X X necessarily biased. Let us consider two situations. (i) X> V = 0k×l , which means that each column of X is orthogonal with each column in V. In other words, regressors included in the matrix X are uncorrelated with regressors included in the matrix V. Then b =β b b • β and bias β = 0k . X XV X • Hence β can be estimated using the smaller model MX without any impact on a quality of the estimator. (ii) X> V 6= 0k×l b is a biased estimator of β. • β X Further, for the fitted values Yb X if those are considered as an estimator of the response vector expectation µ = Xβ + Vγ, we have bias Yb X = − MX Vγ. In this case, all elements of the bias vector would be equal to zero if MX V = 0n×l . Nevertheless, this would mean that M V ⊆ M X which is in contradition with our assumption rank X, V = k + l. That is, if the omitted covariates (included in the matrix V) are linearly independent (are not perfectly multiply correlated) with the covariates included in the model matrix X, the fitted values Yb X always provide a biased estimator of the response expectation. Prediction Let us compare predictions Yb new,X = Yb X based on a (misspecified) model MX and predictions Yb new,XV = Yb XV based on a (correct) model MXV . Properties of the fitted values in a correct model (Expressions (11.7)) together with results of Lemma 11.4 and Theorem 11.5 give MSEP Yb new,XV = n σ2 + k σ2 + l σ2, MSEP Yb new,X 2 = n σ 2 + k σ 2 + MX Vγ . That is, the average mean squared errors of prediction are k 2 l σ + σ2, n n 2 k 1 AMSEP Yb new,X = σ 2 + σ 2 + MX Vγ . n n AMSEP Yb new,XV = σ2 + We can now conclude the following. 2 • The term MX Vγ might be huge compared to l σ 2 in which case the prediction using the model with omitted important covariates is (much) worse than the prediction using the (correct) model. • Additionally, σ 2 → 0 with n → ∞ (while increasing the number of predictions). 2 • On the other hand, n1 MX Vγ does not necessarily tend to zero with n → ∞. l n 11.2. MISSPECIFIED REGRESSION SPACE 216 Estimator of the residual variance Theorem 11.5 shows that the mean residual square MSe,X in a misspecified model MX is a biased estimator of the residual variance σ 2 with the bias amounting to MX Vγ 2 bias MSe,X = . n−k Also in this case, bias does not necessarily tend to zero with n → ∞. 11.2.4 Irrelevant regressors In this section, we will assume that the correct model is model MX : Y Z ∼ Xβ, σ 2 In . This means, that also model MXV : Y Z ∼ Xβ + Vγ, σ 2 In holds, nevertheless, γ = 0l and hence the regressors from the matrix V are irrelevant. Since both models MX and MXV hold, estimators derived under both models have usual properties of the LSE, namely, b Z = E β b E β = β, X XV Z E Yb X Z = E Yb XV Z = Xβ =: µ, n X MSE YbX,i = i=1 = n X MSE YbXV,i i=1 = n X = tr σ 2 HX var YbX,i Z = tr var Yb X Z i=1 σ 2 k, n X = tr σ 2 HXV var YbXV,i Z = tr var Yb XV Z i=1 = σ 2 (k + l), E MSe,X Z = E MSe,XV Z = σ 2 . Least squares estimators b and β b Both estimators β X XV are unbiased estimators of a vector β. Nevertheless, as stated in Theorem 11.3, their quality expressed by the mean squared error which in this case coincide with the covariance matrix (may) differ since h h > i > i b b b b b b MSE β XV − MSE β XV = E β XV − β β XV − β Z − E β X − β β X − β Z b b = var β XV Z − var β X Z ≥ 0. In particular, we derived during the proof of Theorem 11.3 that n −1 > o−1 −1 > > > 2 > b b var β XV Z − var β X Z = σ X X−X V V V V X − X X . 11.2. MISSPECIFIED REGRESSION SPACE 217 Let us again consider two situations. (i) X> V = 0k×l , which means that each column of X is orthogonal with each column in V. In other words, regressors included in the matrix X are uncorrelated with regressors included in the matrix V. Then b =β b b b • β X XV and var β X Z = var β XV Z . • Hence β can be estimated using the model MXV with irrelevant covariates included without any impact on a quality of the estimator. (ii) X> V 6= 0k×l b b • The estimator β XV is worse than the estimator β X in terms of its variability. • If we take into account a fact that by including more regressors in the model, we are b increasing a danger of multicollinearity, difference between variability of β XV and that b may become huge. of β X Prediction Let us now compare predictions Yb new,X = Yb X based on a correct model MX and predictions Yb new,XV = Yb XV based on also a correct model MXV , where however, irrelevant covariates were included. Properties of the fitted values in a correct model together with results of Lemma 11.4 give MSEP Yb new,XV = n σ 2 + (k + l) σ 2 , MSEP Yb new,X = n σ2 + k σ2. That is, the average mean squared errors of prediction are k+l 2 σ , n k AMSEP Yb new,X = σ 2 + σ 2 . n AMSEP Yb new,XV = σ2 + The following can now be concluded. • If n → ∞, both AMSEP Yb new,XV and AMSEP Yb new,X tend to σ 2 . Hence on average, if sufficiently large number of predictions is needed, both models provide predictions of practically the same quality. • On the other hand, by using the richer model MXV (which for a finite n provides worse predictions than the smaller model MX ), we are eliminating a possible problem of omitted important covariates that leads to biased predictions with possibly even worse MSEP and AMSEP than that of model MXV . 11.2.5 Summary Interest in estimation of the regression coefficients and inference on them If interest lies in estimation of and inference on the regression coefficients β related to the regressors included in the model matrix X, the following was derived in Sections 11.2.3 and 11.2.4. (i) If we omit important regressors which are (multiply) correlated with the regressors of main interest included in the matrix X, the LSE of the regression coefficients is biased. 11.2. MISSPECIFIED REGRESSION SPACE 218 (ii) If we include irrelevant regressors which are (multiply) correlated with the regressors of main interest in the matrix X, we are facing a danger of multicollinearity and related inflation of the standard errors of the LSE of the regression coefficients. (iii) Regressors which are (multiply) uncorrelated with regressors of main interest influence neither b irrespective of whether they are omitted or irrelevantly included. bias nor variability of β Consequently, if a primary task of the analysis is to evaluate whether and how much the primary regressors included in the model matrix X influence the response expectation, detailed exploration and understanding of mutual relationships among all potential regressors and also between the regressors and the response is needed. In particular, regressors which are (multiply) correlated with the regressors from the model matrix X and at the same time do not have any influence on the response expectation should not be included in the model. On the other hand, regressors which are (multiply) uncorrelated with the regressors of primary interest can, without any harm, be included in the model. In general, it is necessary to find a trade-off between too poor and too rich model. Interest in prediction If prediction is the primary purpose of the regression analysis, results derived in Sections 11.2.3 and 11.2.4 dictate to follow a strategy to include all available covariates in the model. The reasons are the following. (i) If we omit important regressors, the predictions get biased and the averaged mean squared error of prediction is possibly not tending to the optimal value of σ 2 with n → ∞. (ii) If we include irrelevant regressors in the model, this has, especially with n → ∞, a negligible effect on a quality of the prediction. The averaged mean squared error of prediction is still tending to the optimal value of σ 2 . Chapter 12 Simultaneous Inference in a Linear Model In this chapter, we will assume that data are represented by a set of n random vectors Y , X , i i X i = Xi,0 , . . . , Xi,k−1 , i = 1, . . . , n, that satisfy a normal linear model. That is, Y X ∼ Nn Xβ, σ 2 In , rank Xn×k = r ≤ k < n, > where Y = Y1 , . . . , Yn , X is a matrix with vectors X > 1 , . . . , X n in its rows and β = β0 , . . . , βk−1 ∈ Rk and σ 2 > 0 are unknown parameters. Further, we will assume that a matrix Lm×k > (m > 1) with rows l> 1 , . . . , lm is given such that > θ = Lβ = l> 1 β, . . . , lm β = θ1 , . . . , θm is an estimable vector parameter of the linear model. Our interest will lie in a simultaneous inference on elements of the parameter θ. This means, we will be interested in (i) deriving confidence regions for a vector parameter θ; (ii) testing a null hypothesis H0 : θ = θ 0 for given θ 0 ∈ Rm . 219 12.1. BASIC SIMULTANEOUS INFERENCE 12.1 220 Basic simultaneous inference If matrix Lm×k is such that (i) m ≤ r; (ii) its rows, i.e., vectors l1 , . . . , lm ∈ Rk are linearly independent, then we already have a tool for a simultaneous inference on θ = Lβ. It is based on point (x) of Theorem 3.1 (Least squares estimators under the normality). It provides a confidence region for θ with a coverage of 1 − α which is n θ ∈ Rm : b θ−θ o > n − o−1 b < m Fm,n−r (1 − α) , θ−θ MSe L X> X L> (12.1) b = Lb is the LSE of θ. The null hypothesis H0 : θ = θ 0 is tested using the statistic where θ n − > o−1 1 b 0 > > b − θ0 , Q0 = θ−θ MSe L X X L θ m (12.2) which under the null hypothesis follows an Fm,n−r distribution and the critical region of a test on the level of α is C(α) = Fm,n−r (1 − α), ∞ . (12.3) The P-value if Q0 = q0 is then given as p = 1 − CDFF , m, n−r (q0 ). Note that the confidence region (12.1) and the test based on the statistic Q0 and the critical region (12.3) are mutually dual. That is, the null hypothesis is rejected on a level of α if and only if θ 0 is not covered by the confidence region (12.1) with a coverage 1 − α. 12.2. MULTIPLE COMPARISON PROCEDURES 12.2 221 Multiple comparison procedures 12.2.1 Multiple testing 0 ) on a vector parameter θ = θ , . . . , θ The null hypothesis H0 : θ = θ 0 (θ 0 = θ10 , . . . , θm 1 m 0 . can also be written as H0 : θ1 = θ10 & · · · & θm = θm Definition 12.1 Multiple testing problem, elementary null hypotheses, global null hypothesis. A testing problem with the null hypothesis H0 : θ1 = θ10 & ... 0 & θm = θm , (12.4) is called the multiple testing problem1 with the m elementary hypotheses2 0 H1 : θ1 = θ10 , . . . , Hm : θm = θm . Hypothesis H0 is called in this context also as a global null hypothesis. Note. The above definition of the multiple testing problem is a simplified definition of a general multiple testing problem where the elementary null hypotheses are not necessarily simple hypotheses. Further, general multiple testing procedures consider also problems where the null hypothesis H0 is not necessarily given as a conjunction of the elementary hypotheses. Nevertheless, for our purposes in context of this lecture, Definition 12.1 will suffice. Also subsequent theory of multiple comparison procedures will be provided in a simplified way in an extent needed for its use in context of the multiple testing problem according to Definition 12.1 and in context of a linear model. Notation. • When dealing with a multiple testing problem, we will also write H0 or ≡ H0 H1 & H1 , ≡ or H0 = m \ ... ..., & Hm Hm Hj . j=1 • In context of a multiple testing, subscript 1 at H1 will never indicate an alternative hypothesis. A symbol { will rather be used to indicate an alternative hypothesis. • The alternative hypothesis of a multiple testing problem with the null hypothesis (12.4) will always be given by a complement of the parameter space under the global null hypothesis, i.e., H{0 : θ1 6= θ10 ≡ 1 problém vícenásobného testování 2 H{1 OR OR elementární hypotézy ... ... OR OR 0 θm 6= θm , H{m , 12.2. MULTIPLE COMPARISON PROCEDURES 222 where H{j : θj 6= θj0 , j = 1, . . . , m. We will also write H{0 = m [ H{j . j=1 • Different ways of indexing the elementary null hypotheses will also be used (e.g., a double subscript) depending on a problem at hand. Example 12.1 (Multiple testing problem for one-way classified group means). Suppose that a normal linear model Y X ∼ Nn Xβ, σ 2 In is used to model dependence of the response Y on a single categorical covariate Z with a sample space Z = {1, . . . , G}, where the regression space M X of a vector dimension G parameterizes the one-way classified group means m1 := E Y Z = 1 , . . . , mG = E Y Z = G . If we restrict ourselves to full-rank parameterizations (see Section 7.4.4), the regression coefficients vector is β = β0 , β Z , β Z = β1 , . . . , βG−1 and the group means are parameterized as Z mg = β0 + c> gβ , where g = 1, . . . , G, c> 1 . .. C= c> G is a chosen G × (G − 1) (pseudo)contrast matrix. The null hypothesis H0 : m1 = · · · = mG on equality of the G group means can be specified as a multiple testing problem with m = G2 elementary hypotheses (double subscript will be used to index them): H1,2 : m1 = m2 , . . . , HG−1,G : mG−1 = mG . The elementary null hypotheses can now be written in terms of a vector estimable parameter θ = θ1,2 , . . . , θG−1,G , > θg,h = mg − mh = cg − ch β Z , g = 1, . . . , G − 1, h = g + 1, . . . , G as H1,2 : θ1,2 = 0, ..., HG−1,G : θG−1,G = 0, or written directly in term of the group means as H1,2 : m1 − m2 = 0, HG−1,G : mG−1 − mG = 0, The global null hypothesis is H0 : θ = 0, where θ = Lβ. Here, L is an G2 × G matrix 0 . . L= . 0 ..., c1 − c2 .. . > cG−1 − cG > . Since rank C = G − 1, we have rank L = G − 1. We then have 12.2. MULTIPLE COMPARISON PROCEDURES 223 • For G ≥ 4, m = G2 > G. That is, in this case, the number of elementary null hypotheses is higher than the rank of the underlying linear model. • For G ≥ 3, the matrix L has linearly dependent rows. That is, for G ≥ 3, we can (i) neither calculate a simultaneous confidence region (12.1) for θ; (ii) nor use the test statistic (12.2) to test H0 : θ = 0. In this chapter, (i) we develop procedures that allow to test the null hypothesis H0 : Lβ = θ 0 and provide a simultaneous confidence region for θ = Lβ even if the rows of the matrix L are linearly dependent or its rank is higher than the rank of the underlying linear model; (ii) the test procedure will also decide which of the elementary hypotheses is/are responsible (in a certain sense) for rejection of the global null hypothesis; (iii) developed confidence regions will have a more appealing form of a product of intervals. 12.2.2 Simultaneous confidence intervals Suppose that a distribution of the random vector D depends on a (vector) parameter θ = θ1 , . . . , θm ∈ Θ1 × · × Θm = Θ ⊆ Rm . Definition 12.2 Simultaneous confidence intervals. (Random) intervals θjL , θjU , j = 1, . . . , m, where θjL = θjL (D) and θjU = θjU (D), j = 1, . . . , m, are called simultaneous confidence intervals3 for parameter θ with a coverage of 1 − α if for any 0 ∈ Θ, θ 0 = θ10 , . . . , θm L U P θ1L , θ1U × · · · × θm , θm 3 θ 0 ; θ = θ 0 ≥ 1 − α. Notes. • The condition in the definition can also be written as P ∀ j = 1, . . . , m : θjL , θjU 3 θj0 ; θ = θ 0 ≥ 1 − α. • The product of the simultaneous confidence intervals indeed forms a confidence region in a classical sense. Example 12.2 (Bonferroni simultaneous confidence intervals). α Let for each j = 1, . . . , m, θjL , θjU be a classical confidence interval for θj with a coverage of 1− m . That is, α ∀ j = 1, . . . , m, ∀ θj0 ∈ Θj : P θjL , θjU 3 θj0 ; θj = θj0 ≥ 1 − . m We then have α ∀ j = 1, . . . , m, ∀ θj0 ∈ Θj : P θjL , θjU 63 θj0 ; θj = θj0 ≤ . m 3 simultánní intervaly spolehlivosti 12.2. MULTIPLE COMPARISON PROCEDURES 224 Further, using elementary property of a probability (for any θ 0 ∈ Θ) P ∃ j = 1, . . . , m : θjL , θjU 63 θj0 ; θ=θ 0 m X ≤ P θjL , θjU 63 θj0 ; θ = θ 0 j=1 m X α ≤ = α. m j=1 Hence, P ∀ j = 1, . . . , m : θjL , θjU 3 θj0 ; θ = θ 0 ≥ 1 − α. That is, intervals θjL , θjU , j = 1, . . . , m, are simultaneous confidence intervals for parameter θ with a coverage of 1 − α. Simultaneous confidence intervals constructed in this way from univariate confidence intervals are called Bonferroni simultaneous confidence intervals. Their disadvantage is that they are often seriously conservative, i.e., having a coverage (much) higher than requested 1 − α. 12.2.3 Multiple comparison procedure, P-values adjusted for multiple comparison Suppose again that a distribution of the random vector D depends on a (vector) parameter θ = θ1 , . . . , θm ∈ Θ1 × · × Θm = Θ ⊆ Rm . Let for each 0 < α < 1 a procedure be given to construct the simultaneous confidence intervals θjL (α), θjU (α) , j = 1, . . . , m, for parameter θ with a coverage of 1 − α. Let for each j = 1, . . . , m, the procedure creates intervals satisfying a monotonicity condition 1 − α1 < 1 − α2 =⇒ θjL (α1 ), θjU (α1 ) ⊆ θjL (α2 ), θjU (α2 ) . Definition 12.3 Multiple comparison procedure. Multiple comparison procedure (MCP)4 for a multiple testing problem with the elementary null hypotheses Hj : θj = θj0 , j = 1, . . . , m, based on given procedure for construction of simultaneous confidence intervals for parameter θ is the testing procedure that for given 0 < α < 1 (i) rejects the global null hypothesis H0 : θ = θ 0 if and only if L U θ1L (α), θ1U (α) × · · · × θm (α), θm (α) 63 θ 0 ; (ii) for j = 1, . . . , m, rejects the jth elementary hypothesis Hj : θj = θj0 if and only if θjL (α), θjU (α) 3 6 θj0 . Note. Since θ1L (α), θ1U (α) L (α), θ U (α) 63 θ 0 if and only if there exists × · · · × θm m j = 1, . . . , m, such that θjL (α), θjU (α) 63 θj0 , the MCP rejects, for given 0 < α < 1, the global 4 procedura vícenásobného srovnávání 12.2. MULTIPLE COMPARISON PROCEDURES 225 null hypothesis H0 : θ = θ 0 if and only if, it rejects at least one out of m elementary null hypotheses. Note (Control of the type-I error rate). Classical duality between confidence regions and testing procedures provides that for any 0 < α < 1, the multiple comparison procedure defines a statistical test which (i) controls the type-I error rate with respect to the global null hypothesis H0 : θ = θ 0 , i.e., P H0 rejected; θ = θ 0 ≤ α; (ii) at the same time, for each j = 1, . . . , m, controls the type-I error rate with respect to the elementary hypothesis Hj : θj = θj0 , i.e., P Hj rejected; θj = θj0 ≤ α. End of Lecture #21 (10/12/2015) Definition 12.4 P-values adjusted for multiple comparison. Start of P-values adjusted for multiple comparison for a multiple testing problem with the elementary null Lecture #22 hypotheses Hj : θj = θj0 , j = 1, . . . , m, based on given procedure for construction of simultaneous (17/12/2015) adj confidence intervals for parameter θ are values padj 1 , . . . , pm defined as n o L U 0 padj = inf α : θ (α), θ (α) 3 6 θ j = 1, . . . , m. j j j , j Notes. The following is clear from construction: • The multiple comparison procedure rejects for given 0 < α < 1 the jth elementary hypothesis Hj : θj = θj0 (j = 1, . . . , m) if and only if padj j ≤ α. • Since the global null hypothesis H0 : θ = θ 0 is rejected by the MCP if and only if at least one elementary hypothesis is rejected, we have that the global null hypothesis is for given α rejected if and only if adj ≤ α. min padj 1 , . . . , pm That is, adj padj := min padj 1 , . . . , pm is the P-value of a test of the global null hypothesis based on the considered MCP. Example 12.3 (Bonferroni multiple comparison procedure, Bonferroni adjusted P-values). Let for 0 < α < 1, θjL (α), θjU (α) , j = 1, . . . , m, be the confidence intervals for parameters α θ1 , . . . , θm , each with a (univariate) coverage of 1 − m . That is, α ∀ j = 1, . . . , m, ∀ θj0 ∈ Θj : P θjL (α), θjU (α) 3 θj0 ; θj = θj0 ≥ 1 − . m As shown in Example 12.2, θjL (α), θjU (α) , j = 1, . . . , m, are the Bonferroni simultaneous confidence intervals for parameter θ = θ1 , . . . , θm with a coverage of 1 − α. 12.2. MULTIPLE COMPARISON PROCEDURES 226 Let for j = 1, . . . , m, puni be a P-value related to the (single) test of the (jth elementary) hypothesis j 0 Hj : θj = θj being dual to the confidence interval θjL (α), θjU (α) . That is, puni j Hence, α : = inf m θjL (α), n min m puni j , 1 = inf α : θjU (α) 63 θj0 . o θjL (α), θjU (α) 3 6 θj0 . That is, the P-values adjusted for multiple comparison based on the Bonferroni simultaneous confidence intervals are uni pB j = 1, . . . , m. j = min m pj , 1 , The related multiple comparison procedure is called the Bonferroni MCP. Conservativeness of the Bonferroni MCP is seen, for instance, on the fact that the global null hypothesis H0 : θ = θ 0 is rejected for given 0 < α < 1 if and only if, at least one of the elementary hypothesis is rejected by its single test on a significance level of α/m which approaches zero as m, the number of elementary hypotheses, increases. 12.2.4 Bonferroni simultaneous inference in a normal linear model Consider a linear model Y X ∼ Nn Xβ, σ 2 In , Let rank Xn×k = r ≤ k < n. > θ = Lβ = l> 1 β, . . . , lm β = θ1 , . . . , θm be an estimable vector parameter of the linear model. At this point, we shall only require that lj 6= 0k for each j = 1, . . . , m. Nevertheless, we allow for m > r and also for possibly linearly dependent vectors l1 , . . . , lm . b = Lb = l> b, . . . , l> b = θb1 , . . . , θbm be the LSE of the vector θ and let MSe As usual, let θ 1 m be the residual mean square of the model. α It follows from properties of the LSE under normality that for given α, the 1 − 100% m confidence intervals for parameters θ1 , . . . , θm have the lower and the upper bounds given as q − α > > L > θj (α) = lj b − MSe lj X X lj tn−r 1 − , 2m (12.5) q − α > > U > θj (α) = lj b + MSe lj X X lj tn−r 1 − , j = 1, . . . , m. 2m By the Bonferroni principle, intervals θjL (α), θjU (α) , j = 1, . . . , m, are simultaneous confidence intervals for parameter θ with a coverage of 1 − α. For each j = 1, . . . , m, the confidence interval (12.5) is dual to the (single) test of the (jth elementary) hypothesis Hj : θj = θj0 based on the statistic 0 l> j b − θj Tj (θj0 ) = q , >X −l MSe l> X j j 12.2. MULTIPLE COMPARISON PROCEDURES 227 which under the hypothesis Hj follows the Student tn−r distribution. The univariate P-values are then calculated as puni = 2 CDFt, n−r − |tj,0 | , j where tj,0 is the value of the statistic Tj (θj0 ) attained with given data. Hence the Bonferroni adjusted P-values for a multiple testing problem with the elementary null hypotheses Hj : θj = θj0 , j = 1, . . . , m, are n o pB = min 2 m CDF − |t | j = 1, . . . , m. t, n−r j,0 , 1 , j 12.3. TUKEY’S T-PROCEDURE 12.3 228 Tukey’s T-procedure Method presented in this section is due to John Wilder Tukey (1915 – 2000) who published the initial version of the method in 1949 (Tukey, 1949). 12.3.1 Tukey’s pairwise comparisons theorem Lemma 12.1 Studentized range. Let T1 , . . . , Tm be a random sample from N (µ, σ 2 ), σ 2 > 0. Let R = max Tj − j=1,...,m min Tj j=1,...,m be the range of the sample. Let S 2 be the estimator of σ 2 such that S 2 and T = T1 , . . . , Tm are independent and ν S2 ∼ χ2ν for some ν > 0. σ2 Let R Q = . S The distribution of the random variable Q then depends on neither µ, nor σ. Proof. • We can write: R = S o Tj − µ Tj − µ 1n max − min max(Tj − µ) − min(Tj − µ) j j σ σ j j σ = . S S σ σ • Distribution of both the numerator and the denominator depends on neither µ, nor σ since Tj − µ • For all j = 1, . . . , m ∼ N (0, 1). σ S • Distribution of is a transformation of the χ2ν distribution. σ Note. The distribution of the random variable Q = sample size of T ) and ν (degrees of freedom of the S 2 ). χ2 k from Lemma 12.1 still depends on m (the distribution related to the variance estimator R S 12.3. TUKEY’S T-PROCEDURE 229 Definition 12.5 Studentized range. R from Lemma 12.1 will be called studentized range5 of a sample of size S m with ν degrees of freedom and its distribution will be denoted as qm,ν . The random variable Q = Notation. • For 0 < p < 1, the p 100% quantile of the random variable Q with distribution qm,ν will be denoted as qm,ν (p). • The distribution function of the random variable Q with distribution qm,ν will be denoted CDFq,m,ν (·). Theorem 12.2 Tukey’s pairwise comparisons theorem, balanced version. Let T1 , . . . , Tm be independent random variables and let Tj ∼ N (µj , v 2 σ 2 ), j = 1, . . . , m, where v 2 > 0 is a known constant. Let S 2 be the estimator of σ 2 such that S 2 and T = T1 , . . . , Tm are independent and ν S2 ∼ χ2ν for some ν > 0. σ2 Then √ P for all j 6= l: Tj − Tl − (µj − µl ) < qm,ν (1 − α) v 2 S 2 = 1 − α. Proof. • It follows from the assumptions that random variables distribution N (0, σ 2 ). Tj − µj , j = 1, . . . , m, are i.i.d. with the v Tj − µj Tj − µj − min . • Let R = max j j v v R ⇒ ∼ qm, ν . S • Hence for any 0 < α < 1 (qm,ν is a continuous distribution): Tj − µj Tj − µj − min max j j v v 1 − α = P < qm,ν (1 − α) S max(Tj − µj ) − min(Tj − µj ) j = P j vS < qm,ν (1 − α) = P max(Tj − µj ) − min(Tj − µj ) < v S qm,ν (1 − α) j 5 studentizované rozpětí j 12.3. TUKEY’S T-PROCEDURE 230 = P for all j 6= l (Tj − µj ) − (Tl − µl ) < v S qm,ν (1 − α) √ = P for all j 6= l Tj − Tl − (µj − µl ) < qm,ν (1 − α) v 2 S 2 . k Theorem 12.3 Tukey’s pairwise comparisons theorem, general version. Let T1 , . . . , Tm be independent random variables and let Tj ∼ N (µj , vj2 σ 2 ), j = 1, . . . , m, where vj2 > 0, j = 1, . . . , m are known constants. Let S 2 be the estimator of σ 2 such that S 2 and T = T1 , . . . , Tm are independent and ν S2 ∼ χ2ν σ2 for some ν > 0. Then s P for all j 6= l Tj − Tl − (µj − µl ) < qm,ν (1 − α) vj2 + vl2 2 ! S2 ≥ 1 − α. Proof. Proof/calculations were skipped and are not requested for the exam. See Hayter (1984). k Notes. • Tukey suggested that statement of Theorem 12.3 holds already in 1953 (in an unpublished manuscript Tukey, 1953) without proving it. Independently, it was also suggested by Kramer (1956). Consequently, the statement of Theorem 12.3 was called as Tukey–Kramer conjecture. • The proof is not an easy adaptation of the proof of the balanced version. 12.3.2 Tukey’s honest significance differences (HSD) A method of multiple comparison that will now be developed appears under several different names in the literature: Tukey’s method, Tukey–Kramer method, Tukey’s range test, Tukey’s honest significance differences (HSD) test. Assumptions. In the following, we assume that T = T1 , . . . , Tm ∼ Nm (µ, σ 2 V), where 12.3. TUKEY’S T-PROCEDURE 231 • µ = µ1 , . . . , µm ∈ Rm and σ 2 > 0 are unknown parameters; 2 on a diagonal. • V is a known diagonal matrix with v12 , . . . , vm That is, T1 , . . . , Tm are independent and Tj ∼ N (µj , σ 2 vj ), j = 1, . . . , m. Further, we will assume that an estimator S 2 of σ 2 is available which is independent of T and which satisfies ν S 2 /σ 2 ∼ χ2ν for some ν > 0. Multiple comparison problem. A multiple comparison procedure that will be developed aims in testing m? = hypotheses on all pairwise differences between the means µ1 , . . . , µm . Let m 2 elementary θj,l = µj − µl , j = 1, . . . , m − 1, l = j + 1, . . . , m, θ = θ1,2 , θ1,3 , . . . , θm−1,m . The elementary hypotheses of a multiple testing problem that we shall consider are 0 , Hj,l : θj,l (= µj − µl ) = θj,l j = 1, . . . , m − 1, l = j + 1, . . . , m, m? . The global null hypothesis is as usual H : θ = 0 , θ0 , . . . , θ0 for some θ 0 = θ1,2 0 1,3 m−1,m ∈ R θ0 . Note. The most common multiple testing problem in this context is with θ0 = 0m? which corresponds to all pairwise comparisons of the means µ1 , . . . , µm . The global null hypothesis then states that all the means are equal. Some derivations Using either of the Tukey’s pairwise comparison theorems (Theorems 12.2 and 12.3), we have (for chosen 0 < α < 1): s ! vj2 + vl2 P for all j 6= l Tj − Tl − (µj − µl ) < qm,ν (1 − α) S2 ≥ 1 − α, 2 2 . That is, we with equality of the above probability to 1 − α in the balanced case of v12 = · · · = vm have, T − T − (µ − µ ) j j l ql 2 2 P for all j 6= l < qm,ν (1 − α) ≥ 1 − α. vj +vl 2 S 2 0 ∈R Let for j 6= l and for θj,l 0 Tj − Tl − θj,l 0 Tj,l (θj,l ) := s . vj2 + vl2 S2 2 12.3. TUKEY’S T-PROCEDURE 232 That is ! 1 − α ≤ P for all j 6= l Tj,l (θ0 ) < qm,ν (1 − α); θ = θ 0 j,l 0 Tj − Tl − θj,l < qm,ν (1 − α); θ = θ 0 = P for all j 6= l q 2 2 v +v j l 2 S 2 TL TU 0 0 = P for all j 6= l θj,l (α), θj,l (α) 3 θj,l ; θ = θ , (12.6) where T L (α) θj,l T U (α) θj,l q vj2 +vl2 2 S2, q vj2 +vl2 2 S2, = Tj − Tl − qm,ν (1 − α) = Tj − Tl + qm,ν (1 − α) (12.7) (12.8) j < l. Theorem 12.4 Tukey’s honest significance differences. Random intervals given by (12.8) are simultaneous confidence intervals for parameters θj,l = µj − µl , j = 1, . . . , m − 1, l = j + 1, . . . , m with a coverage of 1 − α. 2 , the coverage is exactly equal to 1 − α, i.e., for any θ 0 ∈ Rm In the balanced case of v12 = · · · = vm 0 0 TU TL = 1 − α. P for all j 6= l θj,l (α), θj,l (α) 3 θj,l ; θ = θ ? 0 , θ 0 ∈ R, Related P-values for a multiple testing problem with elementary hypotheses Hj,l : θj,l = θj,l j,l j < l, adjusted for multiple comparison are given by j < l, pTj,l = 1 − CDFq,m,ν t0j,l , 0 )= where t0j,l is a value of Tj,l (θj,l 0 Tj −Tl −θj,l r v 2 +v 2 j l 2 attained with given data. S2 Proof. T L (α), θ T U (α) , j < l, are simultaneous confidence intervals for parameters The fact that θj,l j,l θj,l = µj − µl with a coverage of 1 − α follows from (12.7). The fact that the coverage of the simultaneous confidence intervals is exactly equal to 1 − α in a balanced case follows from the fact that inequality in (12.6) is equality in a balanced case. Calculation of the P-values adjusted for multiple comparison related to the multiple testing problem 0 , j < l, follows from noting the following (for each with the elementary hypotheses Hj,l : θj,l = θj,l j < l): TL TU 0 0 θj,l (α), θj,l (α) 63 θj,l ⇐⇒ Tj,l θj,l ≥ qm, ν (1 − α) It now follows from monotonicity of the quantiles of a continuous Studentized range distribution that T TL TU 0 0 pj,l = inf α : θj,l (α), θj,l (α) 63 θj,l = inf α : Tj,l θj,l ≥ qm, ν (1 − α) 12.3. TUKEY’S T-PROCEDURE 233 is attained for pTj,l satisfying That is, if t0j,l 0 T θ j,l j,l = qm, ν 1 − pTj,l . 0 is a value of the statistic Tj,l θj,l attained with given data, we have pTj,l = 1 − CDFq,m,ν t0j,l . k 12.3.3 Tukey’s HSD in a linear model In context of a normal linear model Y X ∼ Nn Xβ, σ 2 In , rank Xn×k = r ≤ k < n, the Tukey’s honest significance differences are applicable in the following situation. > • Lm×k is a matrix with non-zero rows l> 1 , . . . , lm such that the parameter > η = Lβ = l> 1 β, . . . , lm β = η1 , . . . , ηm is estimable. • Matrix L is such that V := L X> X − L> = vj,l j,l=1,...,m is a diagonal matrix with := vj,j , j = 1, . . . , m. − With b = X> X X> Y and the residual mean square MSe of the fitted linear model, we have (conditionally, given the model matrix X): vj2 > 2 b = l> T := η 1 b, . . . , lm b = Lb ∼ Nm η, σ V , (n − r)MSe ∼ χ2n−r , σ2 b and MSe independent. η Hence the Tukey’s T-procedure can be used for a multiple comparison problem on (also estimable) parameters > θj,l = ηj − ηl = lj − ll β, j < l. The Tukey’s simultaneous confidence intervals for parameters θj,l , j < l, with a coverage of 1 − α have then the lower and the upper bound given as q 2 2 vj +vl T L MSe , θj,l (α) = ηbj − ηbl − qm,n−r (1 − α) 2 q 2 2 vj +vl T U (α) = η θj,l bj − ηbl + qm,n−r (1 − α) MSe , j < l. 2 Calculation of the P-values adjusted for multiple comparison related to the multiple testing problem with elementary hypotheses 0 Hj,l : θj,l = θj,l , j < l, 0 ∈ R, is based on statistics for chosen θj,l 0 ηbj − ηbl − θj,l 0 Tj,l (θj,l )= s , 2 2 vj + vl MSe 2 j < l. 12.3. TUKEY’S T-PROCEDURE 234 The above procedure is in particular applicable if all involved covariates are categorical and the model corresponds to one-way, two-way or higher-way classification. If normal and homoscedastic errors in the underlying linear model are assumed, the Tukey’s HSD method can then be used to develop a multiple comparison procedure for differences between the group means or between the means of the group means. One-way classification P Let Y = Y1,1 , . . . , YG,nG , n = G g=1 ng , and Yg,j = mg + εg,j , g = 1, . . . , G, j = 1, . . . , nG , i.i.d. εg,j ∼ N (0, σ 2 ). We then have (see Theorem 9.1, with random covariates conditionally given the covariate values) 1 0 Y1 m1 n1 . . . . .. .. . .. ∼ NG ... , σ 2 ... T := . . 0 . . . n1G YG mG Moreover, the mean square error MSe of the underlying one-way ANOVA linear model satisfies, with νe = n − G, νe MSe ∼ χ2νe , MSe and T independent σ2 (due to the fact that T is the LSE of the vector of group means m = m1 , . . . , mG ). Hence the Tukey’s simultaneous confidence intervals for θg,h = mg −mh , g = 1, . . . , G−1, h = g +1, . . . , G with a coverage of 1 − α, have then the lower and upper bounds given as s 1 1 1 Y g − Y h ± qG, n−G (1 − α) + MSe , g < h. 2 ng nh In case of a balanced data (n1 = · · · = nG ), the coverage of those intervals is even exactly equal to 1 − α, otherwise, the intervals are conservative (having a coverage greater than 1 − α). Calculation of the P-values adjusted for multiple comparison related to the multiple testing problem with elementary hypotheses 0 Hg,h : θg,h = θg,h , g < h, 0 ∈ R, is based on statistics for chosen θg,h 0 Tg,h (θg,h ) 0 Y g − Y h − θg,h =s , 1 1 1 + MSe 2 ng nh g < h. Note. The R function TukeyHSD applied to objects obtained using the function aov (performs LSE based inference for linear models involving only categorical covariates) provides a software implementation of the Tukey’s T multiple comparison described here. 12.3. TUKEY’S T-PROCEDURE 235 Two-way classification PH P Let Y = Y1,1,1 , . . . , YG,H,nG,H , n = G h=1 ng,h , and g=1 Yg,h,j = mg,h + εg,h,j , g = 1, . . . , G, h = 1, . . . , H, j = 1, . . . , ng,h , i.i.d. εg,h,j ∼ N (0, σ 2 ). Let, as usual, ng• = H X ng,h , Y g• H ng,h 1 XX Yg,h,j , = ng• h=1 j=1 h=1 mg• H 1 X = mg,h , H h=1 mwt g• H 1 X = ng,h mg,h , ng• g = 1, . . . , G. h=1 Balanced data In case of balanced data (ng,h = J for all g, h), we have ng• = J H, mwt g = mg . Further, 1 0 Y 1• m1• JH ... . . .. .. 2 .. . . T := . . . ∼ NG . , σ . , Y G• mG• 0 . . . J 1H see Consequence of Theorem 9.2. Further, let MSZW and MSZ+W be the residual mean squares e e from the interaction model and the additive model, respectively, νeZW = n − G H, and νeZ+W = n − G − H + 1 degrees of freedom, respectively. We have shown in the proof of Consequence of Theorem 9.2 that for both the interaction model and the additive model, the sample means Y 1• , . . . , Y G• are LSE’s of estimable parameters m1• , . . . , mG• and hence, for both models, vector T is independent of the corresponding residual mean square. Further, depending on whether the interaction model or the additive model is assumed, we have νe? MS?e ∼ χ2νe? , σ2 where MS?e is the residual mean square of the model that is assumed (MSZW or MSZ+W ) and νe? e e ZW Z+W the corresponding degrees of freedom (νe or νe ). Hence the Tukey’s simultaneous confidence intervals for θg1 ,g2 = mg1 • − mg2 • , g1 = 1, . . . , G − 1, g2 = g1 + 1, . . . , G have then the lower and upper bounds given as r 1 Y g1 • − Y g2 • ± qG, νe? (1 − α) MS?e , JH and the coverage of those intervals is even exactly equal to 1 − α. Calculation of the P-values adjusted for multiple comparison related to the multiple testing problem with elementary hypotheses End of Lecture #22 Hg1 ,g2 : θg1 ,g2 = θg01 ,g2 , g1 < g2 , (17/12/2015) 0 for chosen θg1 ,g2 ∈ R, is based on statistics Tg1 ,g2 (θg01 ,g2 ) = Y g1 • − Y g2 • − θg01 ,g2 r , 1 ? MSe JH g1 < g2 . 12.3. TUKEY’S T-PROCEDURE 236 Unbalanced data With unbalanced data, direct calculation shows that 1 Y 1• mwt 1• n1• . . 2 .. . . T := . ∼ NG . , σ . 0 Y G• mwt G• ... .. . ... 0 .. . . 1 nG• wt Further, the sample means Y 1• , . . . , Y G• are LSE’s of the estimable parameters mwt 1• , . . ., mG• in both the interaction and the additive model. This is obvious for the interaction model since there we know the fitted values (≡ LSE’s of the group means mg,h ). Those are Ybg,h,j = Y g,h• , g = 1, . . . , G, h = 1, . . . , H, j = 1, . . . , ng,h (Theorem 9.2). Hence the sample means Y 1• , . . ., Y G• , which are their linear combinations, are LSE’s of the corresponding linear combinations of wt the group means mg,h . Those are the weighted means of the means mwt 1• , . . ., mG• . To show that wt the sample means Y 1• , . . ., Y G• are the LSE’s for the estimable parameters mwt 1• , . . ., mG• in the additive model would, nevertheless, require additional derivations. For the rest, we can proceed in the same way as in the balanced case. That is, let MS?e and νe? denote the residual mean square and the residual degrees of freedom of the model that can be assumed (interaction or additive). Owing to the fact that T is a vector of the LSE’s of the estimable parameters for both models, it is independent of MS?e . The Tukey’s T multiple comparison procedure is now applicable for inference on parameters wt θgwt = mwt g1 • − mg2 • , 1 ,g2 g1 = 1, . . . , G − 1, g2 = g1 + 1, . . . , G. wt The Tukey’s simultaneous confidence intervals for θgwt = mwt g1 • − mg2 • , g1 = 1, . . . , G − 1, g2 = 1 ,g2 g1 + 1, . . . , G, with a coverage of 1 − α, have the lower and upper bounds given as s 1 1 1 ? Y g1 • − Y g2 • ± qG, νe? (1 − α) + MSe . 2 ng1 • ng2 • Calculation of the P-values adjusted for multiple comparison related to the multiple testing problem with elementary hypotheses Hg1 ,g2 : θgwt = θgwt,0 , 1 ,g2 1 ,g2 g1 < g2 , for chosen θgwt,0 1 ,g2 ∈ R, is based on statistics Y g1 • − Y g2 • − θgwt,0 1 ,g2 s , Tg1 ,g2 (θgwt,0 ) = 1 ,g2 1 1 1 + MS?e 2 ng1 • ng2 • g1 < g2 . Notes. • Analogous procedure applies for the inference on the means of the means G m•h = 1 X mg,h , G g=1 mwt •h = G 1 X ng,h mg,h , n•h h = 1, . . . , H, g=1 by the second factor of the two-way classification. wt • The weighted means of the means mwt g• or m•h have a reasonable interpretation only in certain special situations. If this is not the case, the Tukey’s multiple comparison with unbalanced data does not make much sense. Beginning of skipped part 12.3. TUKEY’S T-PROCEDURE 237 • Even with unbalanced data, we can, of course, calculate the LSE’s of the (unweighted) means of the means mg• or m•h . Nevertheless, those LSE’s are correlated with unbalanced data and hence we cannot apply the Tukey’s procedure. Note (Tukey’s HSD in the R software). The R function TukeyHSD provides the Tukey’s T-procedure also for the two-way classification (for both the additive and the interaction model). For balanced data, it performs a simultaneous inference on parameters θg1 ,g2 = mg1 • − mg2 • (and analogous parameters with respect to the second factor) in a way described here. For unbalanced data, it performs a simultaneous inference wt on parameters θgwt = mwt g1 • − mg2 • as described here, nevertheless, only for the first factor 1 ,g2 mentioned in the model formula. Inference on different parameters is provided with respect to the second factor in the model formula. That is, with unbalanced data, output from the R function TukeyHSD and interpretation of the results depend on the order of the factors in the model formula. TukeyHSD with two-way classification for the second factor uses “new” observations that adjust for ? , given as the effect of the first factor. That is, it is worked with “new” observations Yg,h,j ? Yg,h,j = Yg,h,j − Y g• + Y , g = 1, . . . , G, h = 1, . . . , H, j = 1, . . . , ng,h . The Tukey’s T procedure is then applied to the sample means ? Y •h = Y •h − G 1 X ng,h Y g• + Y , n•h h = 1, . . . , H, g=1 whose expectations are mwt •h G G H 1 X 1XX wt ng,h2 mg,h2 , − ng,h mg• + n•h n g=1 h = 1, . . . , H, g=1 h2 =1 which, with unbalanced data, are not equal to mwt •h . End of skipped part 12.4. HOTHORN-BRETZ-WESTFALL PROCEDURE 12.4 238 Hothorn-Bretz-Westfall procedure Start of The multiple comparison procedure presented in this section is applicable for any parametric model Lecture #23 where the parameters estimators follow either exactly (as in the case of a normal linear model) or (17/12/2015) at least asymptotically a (multivariate) normal or t-distribution. In full generality, it was published only rather recently (Hothorn et al., 2008, 2011), nevertheless, the principal ideas behind the method are some decades older. 12.4.1 Max-abs-t distribution Definition 12.6 Max-abs-t-distribution. Let T = T1 , . . . , Tm ∼ mvtm,ν Σ , where Σ is a positive semidefinite matrix. The distribution of a random variable H = max |Tj | j=1,...,m will be called the max-abs-t-distribution of dimension m with ν degrees of freedom and a scale matrix Σ and will be denoted as hm,ν (Σ). Notation. • For 0 < p < 1, the p 100% quantile of the distribution hm,ν (Σ) will be denoted as hm,ν (p; Σ). That is, hm,ν (p; Σ) is the number satisfying P max |Tj | ≤ hm,ν (p; Σ) = p. j=1,...,m • The distribution function of the random variable with distribution hm,ν (Σ) will be denoted CDFh,m,ν (·; Σ). Notes. • If the scale matrix Σ is positive definite (invertible), the random vector T ∼ mvtm,ν Σ has a density w.r.t. Lebesgue measure ν+m − 1 Γ ν+m t> Σ−1 t − 2 2 2 fT (t) = 1+ , t ∈ Rm . m m Σ ν Γ ν2 ν 2 π 2 • The distribution function CDFh,m,ν (·; Σ) of a random variable H = maxj=1,...,m |Tj | is then (for h > 0): CDFh,m,ν (h; Σ) = P max |Tj | ≤ h = P ∀j = 1, . . . , m |Tj | ≤ h j=1,...,m Z h Z h ··· = −h fT (t1 , . . . , tm ) dt1 · · · dtm . −h • That is, when calculating the CDF of the random variable H having the max-abs-t distribution, it is necessary to calculate integrals from a density of a multivariate t-distribution. • Computationally efficient methods not available until 90’s of the 20th century. • Nowadays, see, e.g., Genz and Bretz (2009) and the R packages mvtnorm or mnormt. • Calculation of CDFh,m,ν (·; Σ) is also possible with a singular scale matrix Σ. 12.4. HOTHORN-BRETZ-WESTFALL PROCEDURE 12.4.2 239 General multiple comparison procedure for a linear model Assumptions. In the following, we consider a normal linear model Y X ∼ Nn Xβ, σ 2 In , rank(Xn×k ) = r ≤ k. Further, let l> 1 . . = . l> m Lm×k be a matrix such that > θ = Lβ = l> 1 β, . . . , lm β = θ1 , . . . , θm is an estimable vector parameter with l1 6= 0k , . . . , lm 6= 0k . Notes. • The number m of the estimable parameters of interest may be arbitrary, i.e., even greater than r or k. • The rows of the matrix L may be linearly dependent vectors. Multiple comparison problem. A multiple comparison procedure that will be developed aims in providing a simultaneous inference on m estimable parameters θ1 , . . . , θm with the multiple testing problem composed of m elementary hypotheses Hj : θj = θj0 , j = 1, . . . , m, 0 ∈ Rm . The global null hypothesis is as usual H : θ = θ 0 . for some θ 0 = θ10 , . . . , θm 0 Notation. In the following, the following (standard) notation will be used: − • b = X> X X> Y (any solution to normal equations X> Xb = X> Y ); b = Lb = l> b, . . . , l> b = θb1 , . . . , θbm : LSE of θ; • θ 1 m − • V = L X> X L> = vj,l j,l=1,...,m (which does not depend on a choice of a pseudoinverse − X> X ); 1 1 • D = diag √ , ..., √ ; v1,1 vm,m • MSe : the residual mean square of the model with νe = n − r degrees of freedom. 12.4. HOTHORN-BRETZ-WESTFALL PROCEDURE 240 Reminders from Chapter 3 • For j = 1, . . . , m, (both conditionally given X and unconditionally as well): θbj − θj Zj := p σ 2 vj,j ∼ N (0, 1), θbj − θj Tj := p MSe vj,j ∼ tn−r . • Further (conditionally given X): 1 b − θ ∼ Nm 0m , DVD , Z = Z1 , . . . , Zm = √ D θ σ2 1 b − θ ∼ mvtm, n−r DVD . D θ T = T1 , . . . , Tm = √ MSe Notes. • Matrices V and DVD are not necessarily invertible. • If rank L = m ≤ r then both matrices V and DVD are invertible and Theorem 3.1 further provides (both conditionally given X and unconditionally as well) that under H0 : θ = θ 0 : > −1 1 b b − θ 0 = 1 T > DVD −1 T ∼ Fm, n−r . Q0 = θ − θ0 MSe V θ m m This was used to test the global null hypothesis H0 : θ = θ 0 and to derive the elliptical confidence sets for θ. • It can also be shown that if m0 = rank L then under H0 : θ = θ 0 : > + 1 b b − θ 0 = 1 T > DVD + T ∼ Fm , n−r θ − θ0 MSe V θ 0 m m (both conditionally given X and unconditionally), where symbol + denotes the Moore-Penrose pseudoinverse. Q0 = Some derivations Let for θj0 ∈ R, j = 1, . . . , m, θbj − θj0 Tj (θj0 ) = p , MSe vj,j j = 1, . . . , m. Then, under H0 : θ = θ 0 : 0 T θ 0 := T1 (θ10 ), . . . , Tm (θm ) ∼ mvtm, n−r (DVD). We then have, for 0 < α < 1: 1 − α = P max Tj (θj0 ) < hm, n−r (1 − α; DVD); θ = θ 0 j=1,...,m = P for all j = 1, . . . , m Tj (θj0 ) < hm, n−r (1 − α; DVD); θ = θ 0 ! b θj − θj0 < hm, n−r (1 − α; DVD); θ = θ 0 = P for all j = 1, . . . , m p MSe vj,j = P for all j = 1, . . . , m θjHL (α), θjHU (α) 3 θj0 ; θ = θ 0 , (12.9) 12.4. HOTHORN-BRETZ-WESTFALL PROCEDURE where 241 θjHL (α) = θbj − hm, n−r (1 − α; DVD) p MSe vj,j , p θjHU (α) = θbj + hm, n−r (1 − α; DVD) MSe vj,j , j = 1, . . . , m. (12.10) Theorem 12.5 Hothorn-Bretz-Westfall MCP for linear hypotheses in a normal linear model. Random intervals given by (12.10) are simultaneous confidence intervals for parameters θj = l> β, 0 0 0 m j = 1, . . . , m, with an exact coverage of 1 − α, i.e., for any θ = θ1 , . . . , θm ∈ R HL HU 0 0 P for all j = 1, . . . , m θj (α), θj (α) 3 θj ; θ = θ = 1 − α. Related P-values for a multiple testing problem with elementary hypotheses Hj : θj = θj0 , θj0 ∈ R, j = 1, . . . , m, adjusted for multiple comparison are given by 0 pH j = 1, . . . , m, j = 1 − CDFh,m,n−r tj ; DVD , θbj −θj0 where t0j is a value of Tj (θj0 ) = √ MSe vj,j attained with given data. Proof. The fact that θjHL (α), θjHU (α) , j = 1, . . . , m, are simultaneous confidence intervals for pa- rameters θj = l> j β with an exact coverage of 1 − α follows from (12.9). Calculation of the P-values adjusted for multiple comparison related to the multiple testing problem with the elementary hypotheses Hj : θj = θj0 , j = 1, . . . , m, follows from noting the following (for each j = 1, . . . , m): HL HU 0 0 θj (α), θj (α) 63 θj ⇐⇒ Tj θj ≥ hm, n−r (1 − α; DVD). It now follows from monotonicity of the quantiles of a continuous max-abs-t-distribution that H HL HU 0 0 pj = inf α : θj (α), θj (α) 63 θj = inf α : Tj θj ≥ hm, n−r (1 − α; DVD) is attained for pH j satisfying Tj θj0 = hm, n−r (1 − pH j ; DVD). That is, if t0j is a value of the statistic Tj θj0 attained with given data, we have 0 pH = 1 − CDF t ; DVD . h,m,n−r j j k 12.4. HOTHORN-BRETZ-WESTFALL PROCEDURE 242 Note (Hothorn-Bretz-Westfall MCP in the R software). In the R software, the Hothorn-Bretz-Westfall MCP for linear hypotheses on parameters of (generalized) linear models is implemented in package multcomp. After fitting a model (by the function lm), it is necessary to call sequentially the following functions: (i) glht. One of its arguments specifies the linear hypothesis of interest (specification of the L matrix). Note that for some common hypotheses, certain keywords can be used. For example, pairwise comparison of all group means in context of the ANOVA models is achieved by specifying the keyword “Tukey”. Nevertheless, note that invoked MCP is still that of Hothorn-Bretz-Westfall and it is not based on the Tukey’s procedure. The “Tukey” keyword only specifies what should be compared and not how it should be compared. (ii) summary (applied on an object of class glht) provides P-values adjusted for multiple comparison. (iii) confint (applied on an object of class glht) provides simultaneous confidence intervals which, among other things, requires calculation of a critical value hm, n−r (1 − α), that is also available in the output. Note that both calculation of the P-values adjusted for multiple comparison and calculation of the critical value hm, n−r (1 − α) needed for the simultaneous confidence intervals requires calculation of a multivariate t integral. This is calculated by a Monte Carlo integration (i.e., based on a certain stochastic simulation) and hence the results slightly differ if repeatedly calculated at different occasions. Setting a seed of the random number generator (set.seed()) is hence recommended for full reproducibility of the results. 12.5. CONFIDENCE BAND FOR THE REGRESSION FUNCTION 12.5 243 Confidence band for the regression function In this section, we shall assume that data are represented by i.i.d. random vectors Yi , Z i , i = 1, . . . , n, being sampled from a distribution of a generic random vector Y, Z ∈ R1+p . It is further assumed that for some known transformation t : Rp −→ Rk , a normal linear model with regressors X i = t(Z i ), i = 1, . . . , n, holds. That is, it is assumed that Yi = X > i β + εi , i.i.d. εi ∼ N (0, σ 2 ), for some β ∈ Rk , σ 2 > 0. The corresponding regression function is E Y X = t(z) = E Y Z = z = m(z) = t> (z)β, z ∈ Rp . It will further be assumed that the corresponding model matrix > (Z ) X> t 1 1 . . .. = .. X= > > Xn t (Z n ) b will be the LSE of a vector of is of full-rank (almost surely), i.e., rank Xn×k = k. As it is usual, β β and MSe the residual mean square. Reminder from Section 3.3 Let z ∈ Rp be given. Theorem 3.2 then states that a random interval with the lower and upper bounds given as q −1 b ± tn−k 1 − α MSe t> (z) X> X t(z), t> (z)β 2 is the confidence interval for m(z) = t> (z)β with a coverage of 1 − α. That is, for given z ∈ Rp , for any β 0 ∈ Rk , q −1 α > b P t (z)β ± tn−k 1 − MSe t> (z) X> X t(z) 3 t> (z)β 0 ; β = β 0 = 1 − α. 2 Theorem 12.6 Confidence band for the regression function. Let Yi , Z i , i = 1, . . . , n, Z i ∈ Rp , be such that Yi = X > i β + εi , i.i.d. εi ∼ N (0, σ 2 ), where X i = t(Z i ), i = 1, . . . , n, for a known transformation t : Rp −→ Rk , and β ∈ Rk and σ 2 > 0 are unknown parameters. Let for all z ∈ Rp t(z) 6= 0k and let rank Xn×k = k, where X is a matrix with vectors X 1 , . . . , X n in rows. Then for any β 0 ∈ Rk P for all z ∈ Rp b ± t> (z)β q −1 k Fk, n−k (1 − α) MSe t> (z) X> X t(z) 3 t> (z)β 0 ; β = β0 = 1 − α. 12.5. CONFIDENCE BAND FOR THE REGRESSION FUNCTION 244 Note. Requirement t(z) 6= 0k for all z ∈ Rp is not too restrictive from a practical point of view as it is satisfied, e.g., for all linear models with intercept. Proof. Let (for 0 < α < 1) n K = β ∈ Rk : b β−β > X> X o b ≤ k MSe Fk,n−k (1 − α) . β−β Section 3.2: K is a confidence ellipsoid for β with a coverage of 1 − α, that is, for any β 0 ∈ Rk P K 3 β 0 ; β = β 0 = 1 − α. K is an ellipsoid in Rk , that is, bounded, convex and with our definition also closed subset of Rk . Let for z ∈ Rp : L(z) = inf t> (z)β, β∈K U (z) = sup t> (z)β. β∈K From construction: β ∈ K ⇒ ∀z ∈ Rp L(z) ≤ t> (z)β ≤ U (z). Due to the fact that K is bounded, convex and closed, we also have ∀z ∈ Rp That is, L(z) ≤ t> (z)β ≤ U (z) ⇒ β ∈ K. β ∈ K ⇔ ∀z ∈ Rp L(z) ≤ t> (z)β ≤ U (z). and hence, for any β 0 ∈ Rk , 1 − α = P K 3 β 0 ; β = β 0 = P for all z ∈ Rp L(z) ≤ t> (z)β 0 ≤ U (z); β = β 0 . (12.11) Further, since t> (z)β is a linear function (in β) and K is bounded, convex and closed, we have L(z) = inf t> (z)β= min t> (z)β, β∈K β∈K U (z) = sup t> (z)β= max t> (z)β, β∈K β∈K and both extremes must lie on a boundary of K, that is, both extremes are reached for β satisfying b > X> X β − β b = k MSe Fk,n−k (1 − α). β−β Method of Lagrange multipliers: o 1 n b > X> X β − β b − k MSe Fk,n−k (1 − α) ϕ(β, λ) = t> (z)β + λ β − β 2 ( 12 is only included to simplify subsequent expressions). 12.5. CONFIDENCE BAND FOR THE REGRESSION FUNCTION 245 Derivatives of ϕ: ∂ϕ b , (β, λ) = t(z) + λ X> X β − β ∂β o ∂ϕ 1n b > X> X β − β b − k MSe Fk,n−k (1 − α) . (β, λ) = β−β ∂λ 2 With given λ, the first set of equations is solved (with respect to β) for b − 1 X> X −1 t(z). β(λ) = β λ Use β(λ) in the second equation: −1 −1 1 > t (z) X> X X> X X> X t(z) = k MSe Fk,n−k (1 − α), 2 λ s −1 t> (z) X> X t(z) . λ=± k MSe Fk,n−k (1 − α) Hence, β which minimizes/maximizes t> (z)β subject to b > X> X β − β b = k MSe Fk,n−k (1 − α) β−β is given as s −1 k MSe Fk,n−k (1 − α) > X X t(z), −1 t> (z) X> X t(z) s −1 k MSe Fk,n−k (1 − α) X> X t(z). −1 t> (z) X> X t(z) b− β min = β β max b+ =β −1 Note that with our assumptions of t(z) 6= 0, we never divide by zero since X> X is a positive definite matrix. That is, L(z) = t> (z)β min > b − = t (z)β q MSe t> (z)(X> X)−1 t(z) k Fk,n−k (1 − α), U (z) = t> (z)β max b + = t> (z)β q MSe t> (z)(X> X)−1 t(z) k Fk,n−k (1 − α). The proof is finalized by looking back at expression (12.11) and realizing that, due to continuity, 1 − α = P for all z ∈ Rp L(z) ≤ t> (z)β 0 ≤ U (z); β = β 0 = P for all z ∈ Rp L(z) < t> (z)β 0 < U (z); β = β 0 = P for all z ∈ Rp b ± t> (z)β q −1 k Fk, n−k (1 − α) MSe t> (z) X> X t(z) 3 t> (z)β 0 ; β = β 0 . 12.5. CONFIDENCE BAND FOR THE REGRESSION FUNCTION 246 k Terminology (Confidence band for the regression function). If the covariates Z1 , . . . , Zn ∈ R, confidence intervals according to Theorem (12.6) are often calculated for an (equidistant) sequence of values z1 , . . . , zN ∈ R and then plotted together with b z ∈ R. A band that is obtained in this way is called the fitted regression function m(z) b = t> (z)β, the confidence band for the regression function 6 as it covers jointly all true values of the regression function with a given probability of 1 − α. Note (Confidence band for and around the regression function). For given z ∈ R: Half width of the confidence band FOR the regression function (overall coverage) is q k Fk,n−k (1 − α) MSe t> (z)(X> X)−1 t(z). Half width of the confidence band AROUND the regression function (pointwise coverage) is q α tn−k 1 − MSe t> (z)(X> X)−1 t(z) 2 q = F1,n−k (1 − α) MSe t> (z)(X> X)−1 t(z), α since for any ν > 0, t2ν 1 − = F1,ν (1 − α). 2 For k ≥ 2, and any ν > 0, k Fk,ν (1 − α) > F1,ν (1 − α) and hence the confidence band for the regression function is indeed wider than the confidence band around the regression function. Their width is the same only if k = 1. 6 pás spolehlivosti pro regresní funkci Chapter 13 Asymptotic Properties of the LSE and Sandwich Estimator 13.1 Assumptions and setup Assumption (A0). (i) Let Y1 , X 1 , Y2 , X 2 , . . . be a sequence of (1 + k)-dimensional independent and identi cally distributed (i.i.d.) random random vector Y, X , vectors being distributed as a generic (X = X0 , X1 , . . . , Xk−1 , X i = Xi,0 , Xi,1 , . . . , Xi,k−1 , i = 1, 2, . . .); (ii) Let β = β0 , . . . , βk−1 be an unknown k-dimensional real parameter; (iii) Let E Y X = X > β. Notation (Error terms). We denote ε = Y − X > β, εi = Yi − X > i β, i = 1, 2, . . .. Convention. In this chapter, all unconditional expectations are understood as expectations with respect to the joint distribution of a random vector (Y, X) (which depends on the vector β). Note. From assumption (A0), the error terms ε1 , ε2 , . . . are i.i.d. with a distribution of a generic error term ε. The following can be concluded for their first two (conditional) moments: E ε X = E Y − X >β X = 0, var ε X = var Y − X > β X = var Y X =: σ 2 (X), 247 13.1. ASSUMPTIONS AND SETUP 248 E ε = E E εX = E 0 = 0, var ε = var E ε X + E var ε X = var 0 + E σ 2 (X) = E σ 2 (X) . Assumption (A1). Let the covariate random vector X = X0 , . . . , Xk−1 satisfy (i) EXj Xl < ∞, j, l = 0, . . . , k − 1; (ii) E XX > = W, where W is a positive definite matrix. Notation (Covariates second and first mixed moments). Let W = wj,l j,l=0,...,k−1 . We have, wj2 := wj,j = E Xj2 , j = 0, . . . , k − 1, wj,l = E Xj Xl , Let V := W−1 = vj,l j 6= l. j,l=0,...,k−1 . Notation (Data of size n). For n ≥ 1: Yn Y1 . . := . , Yn X> 1 . . Xn := . , X> n Wn := X> n Xn = n X X iX > i , i=1 Vn := X> n Xn −1 (if it exists). Lemma 13.1 Consistent estimator of the second and first mixed moments of the covariates. Let assumpions (A0) and (A1) hold. Then 1 a.s. Wn −→ W n a.s. n Vn −→ V as n → ∞, as n → ∞. 13.1. ASSUMPTIONS AND SETUP 249 Proof. The statement of Lemma follows from applying, for each j = 0, . . . , k − 1 and l = 0, . . . , k−1, the strong law of large numbers for i.i.d. random variables (Theorem C.2) to a sequence Zi,j,l = Xi,j Xi,l , i = 1, 2, . . . . k End of Lecture #23 (17/12/2015) LSE based on data of size n Since 1 n Start of Lecture #24 (07/01/2016) a.s. X> n Xn −→ W > 0 then P there exists n0 > k such that for all n ≥ n0 rank Xn = k = 1 and we define (for n ≥ n0 ) b = β n X> n Xn −1 X> nY n = n X X iX > i n −1 X i=1 MSe,n = X i Yi , i=1 1 1 b 2 = Y n − Xn β n n−k n−k n X b 2 (Yi − X > i βn ) , i=1 which are the LSE of β and the residual mean square based on the assumed linear model for data of size n. Mn : Y n Xn ∼ Xn β, σ 2 In . Further, for n ≥ n0 any non-trivial linear combination of regression coefficients is estimable parameter of model Mn . • For a given real vector l = l0 , l1 , . . . , lk−1 6= 0k we denote θ = l> β, b . θbn = l> β n > > > • For a given m × k matrix L with rows l> 1 6= 0k , . . . , lm 6= 0k we denote ξ = Lβ, b . b ξ n = Lβ n It will be assumed that m ≤ k and that the rows of L are lineary independent. Interest in asymptotics (as n → ∞) behavior of b ; (i) β n (ii) MSe,n ; b for given l 6= 0k ; (iii) θbn = l> β n b for given m × k matrix L with linearly independent rows; (iv) b ξ n = Lβ n under two different scenarios (two different truths) (i) homoscedastic errors (i.e., model Mn : Y n Xn ∼ Xn β, σ 2 In is correct); 13.1. ASSUMPTIONS AND SETUP 250 (ii) heteroscedastic errors where var ε X is not necessarily constant and perhaps depends on the covariate values X (i.e., model Mn is not necessarily fully correct). Normality of the errors will not be assumed. Assumption (A2 homoscedastic). Let the conditional variance of the response satisfy σ 2 (X) := var Y X = σ 2 , where ∞ > σ 2 > 0 is an unknown parameter. Assumption (A2 heteroscedastic). Let σ 2 (X) := var Y X satisfy, for each j, l = 0, . . . , k − 1, the condition E σ 2 (X)Xj Xl < ∞. Notes. • Condition (A2 heteroscedastic) states that the matrix WF := E σ 2 (X) XX > is a real matrix (with all elements being finite). • If (A0) and (A1) are assumed then (A2 homoscedastic) =⇒ (A2 heteroscedastic). Hence everything that will be proved under (A2 heteroscedastic) holds also under (A2 homoscedastic). • Under assumptions (A0) and (A2 homoscedastic), we have E Yi X i = X > var Yi X i = var εi X i = σ 2 , i β, i = 1, 2, . . . , and for each n > 1, Y1 , . . . , Yn are, given Xn , independent and satisfying a linear model Y n Xn ∼ Xn β, σ 2 In . • Under assumptions (A0) and (A2 heteroscedastic), we have E Yi X i = X > var Yi X i = var εi X i = σ 2 (X i ), i β, i = 1, 2, . . . , and for each n > 1, Y1 , . . . , Yn are, given Xn , independent with σ 2 (X 1 ) . . . 0 .. .. .. . E Y n Xn = Xn β, var Y n Xn = . . . 2 0 . . . σ (X n ) 13.2. CONSISTENCY OF LSE 13.2 251 Consistency of LSE We shall show in this section: b , θbn , b (i) Strong consistency of β ξ n (LSE’s regression coefficients or their linear combinations). n • No need of normality; • No need of homoscedasticity. (ii) Strong consistency of MSe,n (unbiased estinator of the residual variance). • No need of normality. Theorem 13.2 Strong consistency of LSE. Let assumptions (A0), (A1) and (A2 heteroscedastic) hold. Then a.s. b −→ β β n a.s. as n → ∞, b = θbn −→ θ l> β n = l> β as n → ∞, a.s. b = b ξ n −→ ξ Lβ n = Lβ as n → ∞. Proof. a.s. b −→ It is sufficient to show that β β. The remaining two statements follow from properties of n convergence almost surely. We have −1 > X> Xn Y n n Xn −1 1 > 1 > X Xn X Yn , = n n n n | {z }| {z } An Bn b = β n where An = 1 > X Xn n n −1 a.s. −→ W−1 by Lemma 13.1. Further Bn = n 1 X 1 > > Xn Y n = X i Yi − X > i β + Xi β n n i=1 = (a) C n = n n 1 X 1 X X i εi + X iX > i β. n n | i=1{z } | i=1 {z } Cn Dn n 1 X a.s. X i εi −→ 0k due to the SLLN (i.i.d., Theorem C.2). This is justified as follows. n i=1 n n 1 X 1 X • The jth (j = 0, . . . , k − 1) element of the vector X i εi is Xi,j εi . n n i=1 i=1 13.2. CONSISTENCY OF LSE 252 • The random variables Xi,j εi , i = 1, 2, . . . are i.i.d. by (A0). • E Xi,j εi = E E Xi,j εi X i = E Xi,j E εi X i = E Xi,j 0 = • var Xi,j εi 0. = + var E Xi,j εi X i E var Xi,j εi X i 2 var ε X E Xi,j + var Xi,j 0 i i = 2 σ 2 (X ) E Xi,j i < ∞ by (A2 heteroscedastic) = =⇒ EXi,j εi < ∞. n 1 1 X a.s. (b) D n = Wn β −→ Wβ by Lemma 13.1. X iX > i β = n n i=1 a.s. b = An C n + D n , where An −→ In summary: β W−1 , n a.s. C n −→ 0k , a.s. D n −→ W β. Hence a.s. b −→ β W−1 W β = β. n Theorem 13.3 Strong consistency of the mean squared error. Let assumptions (A0), (A1), (A2 homoscedastic) hold. Then a.s. MSe,n −→ σ 2 as n → ∞. Proof. We have MSe,n = n 1 n 1 X b 2 Yi − X > SSe,n = i β . n−k n−k n i=1 k 13.2. CONSISTENCY OF LSE Since lim n n→∞ n−k 253 = 1, it is sufficient to show that n a.s. 2 1 X b 2 −→ Yi − X > σ i βn n as n → ∞. i=1 We have n n X 1 X > >b 2 b 2 = 1 Yi − X > β Yi − X > i n i β + X i β − X i βn n n i=1 i=1 1 = n | n X Yi − n n o2 > 1 X > 2 X b b + X i β − βn + Yi − X > i β X i β − βn . n n } | i=1 {z } | i=1 {z } Bn Cn 2 X> i β i=1 {z An n n 2 1 X 2 a.s. 1 X > Yi − X i β = εi −→ σ 2 due to the SLLN (i.i.d., Theorem C.2). (a) An = n n i=1 i=1 This is justified by noting the following. • The random variables ε2i , i = 1, 2, . . . are i.i.d. by (A0). • E εi = 0 =⇒ E ε2i = var εi = E σ 2 (X i ) = E σ 2 = σ 2 by assumption (A2 homoscedastic). • Eε2i = E ε2i = σ 2 < ∞ by assumption (A2 homoscedastic). (b) B n n o2 a.s. 1 X > b Xi β − β −→ 0, which is seen as follows. = n n i=1 Bn n o2 1 X > b Xi β − β = n n i=1 = n 1 X b > X iX > β − β b β −β n n i n i=1 n > 1 X b X iX > β −β n i n = b β −β n = b . b > 1 X> Xn β − β β −β n n n n i=1 Now a.s. b −→ β −β 0k due to Theorem 13.2. n 1 > a.s. X Xn −→ W due to Lemma 13.1. n n Hence a.s. B n −→ 0> k W 0k = 0. 13.2. CONSISTENCY OF LSE 254 n > a.s. 2 X b (c) C n = Yi − X > i β X i β − β n −→ 0, which is justified by the following. n i=1 Cn n > 2 X b Yi − X > = i β X i β − βn n i=1 = 2 Now n 1 X n εi X > i b . β −β n i=1 n 1 X a.s. > εi X > i −→ 0k as was shown in the proof of Theorem 13.2. n i=1 a.s. b −→ β −β 0k due to Theorem 13.2. n Hence In summary: MSe,n = a.s. C n −→ 0> k 0k = 0. n An + B n + C n , where n−k n n−k → 1, a.s. An −→ σ 2 , a.s. B n −→ 0, a.s. C n −→ 0. Hence a.s. MSe,n −→ 1 (σ 2 + 0 + 0) = σ 2 . k 13.3. ASYMPTOTIC NORMALITY OF LSE UNDER HOMOSCEDASTICITY 13.3 255 Asymptotic normality of LSE under homoscedasticity b , θbn , b We shall show in this section: asymptotic normality of β ξ n (LSE’s regression coefficients or n their linear combinations) when homoscedasticity of the errors is assumed but not their normality. n Reminder. V = E XX > o−1 . Theorem 13.4 Asymptotic normality of LSE in homoscedastic case. Let assumptions (A0), (A1), (A2 homoscedastic) hold. Then √ D b −β n β −→ Nk (0k , σ 2 V) n √ D n θbn − θ −→ N1 (0, σ 2 l> V l) √ D n b ξn − ξ −→ Nm (0m , σ 2 L V L> ) as n → ∞, as n → ∞, as n → ∞. Proof. Will be done jointly with Theorem 13.5. 13.3.1 k Asymptotic validity of the classical inference under homoscedasticity but non-normality For given n ≥ n0 > k, the following statistics are used to infer on estimable parameters of the linear model Mn based on the response vector Y n and the model matrix Xn (see Chapter 3): θbn − θ Tn := q −1 , MSe,n l> X> X l n n Qn := 1 m b ξn − ξ > n −1 > o−1 b L X> L ξn − ξ n Xn MSe,n Reminder. • Vn = X> n Xn −1 . a.s. • Under assumptions (A0) and (A1): n Vn −→ V as n → ∞. (13.1) . (13.2) 13.3. ASYMPTOTIC NORMALITY OF LSE UNDER HOMOSCEDASTICITY 256 Consequence of Theorem 13.4: Asymptotic distribution of t- and F-statistics. Under assumptions of Theorem 13.4: Tn m Qn D as n → ∞, D as n → ∞. −→ N1 (0, 1) −→ χ2m Proof. It follows directly from Lemma 13.1, Theorem 13.4 and Cramér-Slutsky theorem (Theorem C.7) as follows. √ >b n l β n − l> β p −1 = σ 2{zl> Vl X> X l n n | } b − l> β l> β n Tn = q MSe,n l> D −→ N (0, 1) m Qm = b − Lβ Lβ n D σ 2 l> Vl n −1 o . MSe,n l> n X> l n Xn | {z } P −→ 1 > n −1 > o−1 b − Lβ Lβ MSe,n L X> X L n n n √ = v u u t n −1 > o−1 MSe,n L n X> X L n {zn } | b − Lβ > n Lβ n | {z } −→ Nm 0m , σ 2 LVL> P −→ σ 2 LVL> D b − Lβ Lβ n | {z √ n } . −→ Nm 0m , σ 2 LVL> Convergence to χ2m in distribution follows from a property of (multivariate) normal distribution concerning the distribution of a quadratic form. k If additionaly normality is assumed, i.e., if it is assumed Y n Xn Theorem 3.1 (LSE under the normality) provides ∼ Nn Xn β, σ 2 In then Tn ∼ tn−k , Qn ∼ Fm, n−k . This is then used for inference (derivation of confidence intervals and regions, construction of tests) on the estimable parameters of a linear model under assumption of normality. The following holds in general: D Tν ∼ tν then Tν −→ N (0, 1) as ν → ∞, Qν ∼ Fm, ν then m Qν −→ χ2m D as ν → ∞. (13.3) This, together with Consequence of Theorem 13.4 then justify asymptotic validity of a classical inference based on statistics Tn (Eq. 13.1) and Qn (Eq. 13.2), respectively and a Student t and Fdistribution, respectively, even if normality of the error terms of the linear model does not hold. The only requirements are assumptions of Theorem 13.4. That is, for example, both intervals 13.3. ASYMPTOTIC NORMALITY OF LSE UNDER HOMOSCEDASTICITY (i) InN := θbn − u(1−α/2) q > −1 X> l, n Xn MSe,n l q −1 (ii) Int := θbn − tn−k (1−α/2) MSe,n l> X> l, n Xn 257 q −1 MSe,n l> X> l ; n Xn q −1 θbn + tn−k (1−α/2) MSe,n l> X> X l , n n θbn + u(1−α/2) satisfy, for any θ0 ∈ R (even without normality of the error terms) P InN 3 θ0 ; θ = θ0 −→ 1 − α as n → ∞, P Int 3 θ0 ; θ = θ0 −→ 1 − α as n → ∞. Analogously, due to a general asymptotic property of the F-distribution (Eq. 13.3), asymptotically valid inference on the estimable vector parameter ξ = Lβ of a linear model can be based either on the statistic m Qn and the χ2m distribution or on the statistic Qn and the Fm, n−k distribution. For example, for both ellipsoids n o > n −1 > o−1 (i) Knχ := ξ ∈ Rm : ξ − b ξ MSe,n L X> L ξ−b ξ < χ2m (1 − α) ; n Xn o n −1 > o−1 > n ξ−b ξ < m Fm,n−k (1 − α) , MSe,n L X> L (ii) KnF := ξ ∈ Rm : ξ − b ξ n Xn we have for any ξ 0 ∈ Rm (under assumptions of Theorems 13.4): P Knχ 3 ξ 0 ; ξ = ξ 0 −→ 1 − α as n → ∞, P KnF 3 ξ 0 ; ξ = ξ 0 −→ 1 − α as n → ∞. 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 13.4 258 Asymptotic normality of LSE under heteroscedasticity b , θbn , b We shall show in this section: asymptotic normality of β ξ n (LSE’s regression coefficients or n their linear combinations) when even homoscedasticity of the errors is not assumed. Reminder. n o−1 . • V = E XX > • WF = E σ 2 (X) XX > . Theorem 13.5 Asymptotic normality of LSE in heteroscedastic case. Let assumptions (A0), (A1), (A2 heteroscedastic) hold. Then √ D b −β n β −→ Nk (0k , VWF V) n √ D n θbn − θ −→ N1 (0, l> VWF V l) √ n b ξn − ξ as n → ∞, as n → ∞, D −→ Nm (0m , L VWF V L> ) as n → ∞. Proof. We will jointly prove also Theorem 13.4. We have b = β n X> Xn | n {z −1 X> nY n } Vn = Vn n X X i Yi i=1 = Vn n X Xi X> i β + εi i=1 = Vn X n X iX > i β + Vn {z } V−1 n = β + Vn X i εi i=1 i=1 | n X n X X i εi . i=1 That is, b − β = Vn β n n X X i εi = n V n i=1 n 1 X X i εi . n (13.4) i=1 a.s. By Lemma 13.1, n Vn −→ V which implies P n Vn −→ V as n → ∞. (13.5) 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 259 Pn In the following, let us explore asymptotic behavior of the term n1 i=1 X i εi . 1 Pn From assumption (A0), the term n i=1 X i εi is a sample mean of i.i.d. random vector X i εi , i = 1, . . . , n. The mean and the covariance matrix of the distribution of those random vectors are E Xε = 0k (was shown in the proof of Theorem 13.2), + var E Xε X var Xε = E var Xε X = E X var ε X X > + var X E ε X | {z } | {z } 0 σ 2 (X) = E σ 2 (X)XX > . Depending, on whether (A2 homoscedastic) or (A2 heteroscedastic) is assumed, we have σ 2 E XX > = σ 2 W, (A2 homoscedastic), var Xε = E σ 2 (X)XX > = WF , (A2 heteroscedastic). (13.6) Under both (A2 homoscedastic) and (A2 heteroscedastic) all elements of the covariance matrix var Xε are finite. Hence by Theorem C.5 (multivariate CLT for i.i.d. random vectors): n n √ 1 X 1 X D X i εi = √ n X i εi −→ Nk 0k , E σ 2 (X)XX > n n i=1 as n → ∞. i=1 From (13.4) and (13.5), we now have, b − β β n n 1 X √ X i εi n i=1 {z } | = n Vn | {z } P −→V D 1 √ . n −→Nk 0k , E σ 2 (X)XX > That is, √ b − β n β n = n Vn | {z } P −→V D n 1 X √ X i εi n i=1 | {z } −→Nk 0k , E σ 2 (X)XX > Finally, by applying Theorem C.7 (Cramér–Slutsky): D √ b − β −→ n β Nk 0k , V E σ 2 (X)XX > V> n . as n → ∞. By using (13.6) and realizing that V> = V, we get Under (A2 homoscedastic) V E σ 2 (X)XX > V> = V σ 2 W V = σ 2 V V−1 V = σ 2 V and hence √ b − β n β n D −→ Nk 0k , σ 2 V as n → ∞. 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 260 Under (A2 heteroscedastic) V E σ 2 (X)XX > V> = V WF V and hence D √ F b − β −→ n β N 0 , V W V k k n as n → ∞. b and of ξ = Lβ b follows now from Theorem C.6 (Cramér– Asymptotic normality of θbn = l> β n n n Wold). k Notation (Residuals and related quantities based on a model for data of size n). For n ≥ n0 > k, the following notation will be used for quantities based on the model Mn : Y n Xn ∼ Xn β, σ 2 In . −1 • Hat matrix: Hn = Xn X> n Xn • Residual projection matrix: Mn = In − Hn ; • Diagonal elements of matrix Hn : hn,1 , . . . , hn,n ; • Diagonal elements of matrix Mn : mn,1 = 1 − hn,1 , . . . , mn,n = 1 − hn,n ; U n = Mn Y n = Un,1 , . . . , Un,n . • Residuals: X> n; Reminder. • Vn = n X X iX i> −1 = X> n Xn −1 . i=1 a.s. • Under assumptions (A0) and (A1): n Vn −→ V as n → ∞. Theorem 13.6 Sandwich estimator of the covariance matrix. Let assumptions (A0), (A1), (A2 heteroscedastic) hold. Let additionally, for each s, t, j, l = 0, . . . , k−1 Eε2 Xj Xl < ∞, Then Eε Xs Xj Xl < ∞, a.s. F n Vn WF n Vn −→ V W V EXs Xt Xj Xl < ∞. as n → ∞, where for n = 1, 2, . . ., WF n = n X 2 > Un,i X iX > i = Xn Ωn Xn , i=1 Ωn = diag ωn,1 , . . . , ωn,n , 2 ωn,i = Un,i , i = 1, . . . , n. End of Lecture #24 (07/01/2016) Start of Lecture #26 (14/01/2016) 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 261 Proof. First, remind that n n o−1 o−1 , E σ 2 (X) XX > E XX > E XX > V WF V = and we know from Lemma 13.1 that −1 a.s. n o−1 > =V n Vn = n X> X −→ E XX n n as n → ∞. Hence, if we show that n 1 X 2 1 F a.s. Wn = Un,i X i X > −→ E σ 2 (X) XX > = WF i n n as n → ∞, i=1 the statement of Theorem will be proven. Remember, σ 2 (X) = var ε X = E ε2 X . From here, for each j, l = 0, . . . , k − 1 E ε2 Xj Xl = E E ε2 Xj Xl X = E Xj Xl E ε2 X = E σ 2 (X) Xj Xl . For each j, l = 0, . . . , k − 1, Eε2 Xj Xl < ∞ by assumptions of Theorem. By assumption (A0), εi Xi,j Xi,l , i = 1, 2, . . ., is a sequence of i.i.d. random variables. Hence by Theorem C.2 (SLLN, i.i.d.), n 1 X 2 a.s. εi Xi,j Xi,l −→ E σ 2 (X) Xj Xl n as n → ∞. i=1 That is, in a matrix form, n 1 X 2 a.s. εi X i X > −→ E σ 2 (X) XX > = WF i n as n → ∞. (13.7) i=1 In the following, we show that (unobservable) squared error terms ε2i in (13.7) can be replaced by 2 = Y − X >β b 2 while keeping the same limitting matrix WF as in (13.7). squared residuals Un,i i i n We have n n 1 X 2 1 X b 2 Xi X> Un,i X i X > = Yi − X > i i βn i n n i=1 i=1 | {z } WF n 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 262 n 1 X > >b 2 Xi X> = Yi − X > i i β + X i β − X i βn | {z } n i=1 εi = n n 1 X 2 1 X b > X iX > β − β b X iX > + εi X i X > β−β n n i i i n n i=1 i=1 {z } {z } | | An Bn + (a) An = n 2 X b > X i εi X i X > . β−β n i n i=1 | {z } Cn n 1 X 2 a.s. 2 > εi X i X > = WF due to (13.7). i −→ E σ (X) XX n i=1 n 1 X b > X iX > β − β b X i X > , we can realize that β − β−β n n i i n i=1 b is a scalar quantity. Hence β−β n (b) To work with Bn = b β n > Xi = X> i Bn n 1 X b > X i X iX > X > β − β b β−β = n n i i n i=1 and the (j, l)th element of matrix Bn (j, l = 0, . . . , k − 1) is Bn (j, l) = n 1 X b > X i (Xi,j Xi,l ) X > β − β b β−β n n i n i=1 = b β−β n > n 1 X > b . (Xi,j Xi,l ) X i X i β−β n n i=1 a.s. b • From Theorem 13.2: β − β n −→ 0k as n → ∞. • Due to assumption (A0) and assumption EXs Xt Xj Xl < ∞ for any s, t, j, l = 0, . . . , k − 1, by Theorem C.2 (SLLN, i.i.d.), for any j, l = 0, . . . , k − 1: n 1 X a.s. > (Xi,j Xi,l ) X i X > . i −→ E Xj Xl XX n i=1 a.s. > • Hence, for any j, l = 0, . . . , k − 1, Bn (j, l) −→ 0> 0k = 0 and finally, k E Xj Xl XX a.s. as n → ∞. Bn −→ 0k×k n 2 X b > X i εi X i X > and the (j, l)th element of matrix Cn (j, l = (c) Cn = β−β n i n i=1 0, . . . , k − 1) is Cn (j, l) = n 2 X b > X i εi Xi,j Xi,l β−β n n i=1 b = 2 β−β n > n 1 X X i εi Xi,j Xi,l . n i=1 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 263 a.s. b • From Theorem 13.2: β − β n −→ 0k as n → ∞. • Due to assumption (A0) and assumption Eε Xs Xj Xl < ∞ for any s, j, l = 0, . . . , k −1, by Theorem C.2 (SLLN, i.i.d.), for any j, l = 0, . . . , k − 1: n 1 X a.s. X i εi Xi,j Xi,l −→ E Xε Xj Xl . n i=1 a.s. • Hence, for any j, l = 0, . . . , k − 1, Cn (j, l) −→ 2 0> k E Xε Xj Xl = 0 and finally, a.s. Cn −→ 0k×k as n → ∞. In summary: n Vn WF n Vn = n Vn 1 n WF n Vn n = n Vn An + Bn + Cn n Vn , a.s. where n Vn −→ V, a.s. An −→ WF , a.s. Bn −→ 0k×k , a.s. Cn −→ 0k×k . Hence a.s. F n Vn WF + 0k×k + 0k×k V = VWF V n Vn −→ V W as n → ∞. k Terminology (Heteroscedasticity consistent (sandwich) estimator of the covariance matrix). Matrix > Vn W F n Vn = Xn Xn −1 > X> n Ωn Xn Xn Xn −1 (13.8) b of is called the heteroscedasticity consistent (HC) estimator of the covariance matrix of the LSE β n the regression coefficients. Due to its form, the matrix (13.8) is also called as the sandwich estimator −1 > composed of a bread X> Xn and a meat Ωn . n Xn Notes (Alternative sorts of meat for the sandwich). • It is directly seen that the meat matrix Ωn can, for a chosen sequence νn , such that n → ∞, be replaced by a matrix n Ωn , νn n νn → 1 as and the statement of Theorem 13.6 remains valid. A value νn is then called degrees of freedom of the sandwich. 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 264 • It can also be shown (see references below) that the meat matrix Ωn can, for a chosen sequence νn , such that νnn → 1 as n → ∞ and a suitable sequence δ n = δn,1 , . . . , δn,n , n = 1, 2, . . ., be replaced by a matrix ΩHC := diag ωn,1 , . . . , ωn,n , n ωn,i = 2 n Un,i , νn mδn,i n,i i = 1, . . . , n. • The following choices of sequences νn and δ n have appeared in the literature (n = 1, 2, . . ., i = 1, . . . , n): HC0: νn = n, δn,i = 0, that is, 2 ωn,i = Un,i . This is the choice due to White (1980) who was the first who proposed the sandwich estimator of the covariance matrix. This choice was also used in Theorem 13.6. HC1: νn = n − k, δn,i = 0, that is, n U2 . n − k n,i This choice was suggested by MacKinnon and White (1985). ωn,i = HC2: νn = n, δn,i = 1, that is, ωn,i = 2 Un,i . mn,i This is the second proposal of MacKinnon and White (1985). HC3: νn = n, δn,i = 2, that is, ωn,i = 2 Un,i m2n,i . This is the third proposal of MacKinnon and White (1985). HC4: νn = n, δn,i = min 4, n hn,i /k , that is, ωn,i = 2 Un,i δ n,i mn,i . This was proposed relatively recently by Cribari-Neto (2004). Note that k = hence n n h o 1 X n,i δn,i = min 4, , hn = hn,i . n hn i=1 Pn i=1 hn,i , and • An extensive study towards small sample behavior of different sandwich estimators was carried out by Long and Ervin (2000) who recommended usage of the HC3 estimator. Even better small sample behavior, especially in presence of influential observations was later concluded by Cribari-Neto (2004) for the HC4 estimator. • Labels HC0, HC1, HC2, HC3, HC4 for the above sandwich estimators are used by the R package sandwich (Zeileis, 2004) that enables for their easy calculation based on the fitted linear model. 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 13.4.1 265 Heteroscedasticity consistent asymptotic inference Let for given sequences νn and δ n , n = 1, 2, . . ., ΩHC be a sequence of the meat matrices that n b . Let for lead to the heteroscedasticity consistent estimator of the covariance matrix of the LSE β n given n ≥ n0 > k, −1 > HC −1 VHC := X> Xn Ωn Xn X> . n n Xn n Xn Finally, let the statistics TnHC and QHC be defined as n θbn − θ , TnHC := q l> VHC l n QHC := n > 1 b > −1 b ξn − ξ LVHC ξn − ξ . n L m Note that the statistics TnHC and QHC are the usual statistics Tn (Eq. 13.1) and Qn n , respectively, −1 > (13.2), respectively, in which the term MSe,n Xn Xn is replaced by the sandwich estimator VHC . n Consequence of Theorems 13.5 and 13.6: Heteroscedasticity consistent asymptotic inference. Under assumptions of Theorem 13.5 and 13.6: TnHC m QHC n D as n → ∞, D as n → ∞. −→ N1 (0, 1) −→ χ2m Proof. Proof/calculations were available on the blackboard in K1. k Due to a general asymptotic property of the Student t-distribution (Eq. 13.3), asymptotically valid inference on the estimable parameter θ = l> β of a linear model where neither normality, nor homoscedasticity is necessarily satisfied, can be based on the statistic TnHC and either a Student tn−k or a standard normal distribution. Under assumptions of Theorems 13.5 and 13.6, both intervals q q bn + u(1 − α/2) l> VHC l ; l, θ (i) InN := θbn − u(1 − α/2) l> VHC n n q q bn + tn−k (1 − α/2) l> VHC l , (ii) Int := θbn − tn−k (1 − α/2) l> VHC l, θ n n satisfy, for any θ0 ∈ R: P InN 3 θ0 ; θ = θ0 −→ 1 − α P Int 3 θ0 ; θ = θ0 −→ 1 − α as n → ∞, as n → ∞. 13.4. ASYMPTOTIC NORMALITY OF LSE UNDER HETEROSCEDASTICITY 266 Analogously, due to a general asymptotic property of the F-distribution (Eq. 13.3), asymptotically valid inference on the estimable vector parameter ξ = Lβ of a linear model can be based either on the statistic m QHC and the χ2m distribution or on the statistic QHC and the Fm, n−k distribution. n n For example, for both ellipsoids n o > m HC > −1 2 b b (i) := ξ ∈ R : ξ − ξ L Vn L ξ − ξ < χm (1 − α) ; n o > > −1 b (ii) KnF := ξ ∈ Rm : ξ − b ξ L VHC L ξ − ξ < m F (1 − α) , m,n−k n Knχ we have for any ξ 0 ∈ Rm (under assumptions of Theorems 13.5 and 13.6): P Knχ 3 ξ 0 ; ξ = ξ 0 −→ 1 − α as n → ∞, P KnF 3 ξ 0 ; ξ = ξ 0 −→ 1 − α as n → ∞. End of Lecture #26 (14/01/2016) Chapter 14 Unusual Observations In the whole chapter, we assume a linear model M : Y X ∼ Xβ, σ 2 In , Start of Lecture #25 (07/01/2016) rank(Xn×k ) = r ≤ k, where standard notation is considered. That is, − • b = X> X X> Y = b0 , . . . , bk−1 : any solution to normal equations; − • H = X X> X X> = hi,t i,t=1,...,n : the hat matrix; • M = In − H = mi,t i,t=1,...,n : the residual projection matrix; • Yb = HY = Xb = Yb1 , . . . , Ybn : the vector of fitted values; • U = MY = Y − Yb = U1 , . . . , Un : the residuals; 2 • SSe = U : the residual sum of squares; SSe is the residual mean square; = U1std , . . . , Unstd : vector of standardized residuals, • MSe = • U std 1 n−r Uistd = √ Ui , MSe mi,i i = 1, . . . , n. The whole chapter will deal with idntification of “unusual” observations in a particular dataset. Any probabilistic statements will hence be conditioned by the realized covariate values X 1 = x1 , . . . , X n = xn . The same symbol X will be used for (in general random) model matrix and its realized counterpart, i.e., X> x> 1 1 . . . . X= . = . . > Xn x> n 267 14.1. LEAVE-ONE-OUT AND OUTLIER MODEL 14.1 Leave-one-out and outlier model Notation. For chosen t ∈ 1, . . . , n , we will use the following notation. • Y (−t) : vector Y without the tth element; • xt : the tth row (understood as a column vector) of the matrix X; • X(−t) : matrix X without the tth row; • j t : vector 0, . . . , 0, 1, 0, . . . , 0 of length n with 1 on the tth place. Definition 14.1 Leave-one-out model. The tth leave-one-out model1 is a linear model M(−t) : Y (−t) X(−t) ∼ X(−t) β, σ 2 In−1 . Definition 14.2 Outlier model. The tth outlier model2 is a linear model out 2 Mout t : Y X ∼ Xβ + j t γt , σ In . Notation (Quantities related to the leave-one-out and outlier models). • Quantities related to model M(−t) will be recognized by subscript (−t), i.e., b(−t) , Yb (−t) , SSe,(−t) , MSe,(−t) , . . . • Quantities related to model Mout will be recognized by subscript t and superscript out, i.e., t out out out b bout t , Y t , SSe,t , MSe,t , . . . • Solutions to normal equations in model Mout will be denoted as t out bout . t , ct • If γtout is an estimable parameter of model Mout then its LSE will be denoted as γ btout . t Theorem 14.1 Four equivalent statements. The following four statements are equivalent: (i) rank(X) = rank X(−t) , i.e., xt ∈ M X> (−t) ; 1 model vynechaného ttého pozorovńí 2 model ttého odlehlého pozorovńí 268 14.1. LEAVE-ONE-OUT AND OUTLIER MODEL 269 (ii) mt,t > 0; (iii) γtout is an estimable parameter of model Mout t ; (iv) µt := E Yt X t = xt = x> t β is an estimable parameter of model M(−t) . Proof. (ii) ⇔ (i) • We will show this by showing non(i) ⇔ non(ii). > . • non(i) means that xt ∈ / M X> (−t) ⊂ M X > M X> and M X(−t) 6= M X> . (−t) ⊂ M X ⊥ ⊥ ⊥ ⊥ ⇔ M X> ⊂ M X> and M X> 6= M X> . (−t) (−t) ⊥ ⊥ > such that a ∈ / M X> . • That is, ⇔ ∃a ∈ M X(−t) > ⇔ ∃a ∈ Rk such that a> X> & a> X> 6= 0> . (−t) = 0 ⇔ ∃a ∈ Rk such that X(−t) a = 0 & Xa 6= 0. It must be Xa = 0, . . ., 0, c, 0, . . ., 0 > = c jt for some c 6= 0. ⇔ ∃a ∈ Rk such that Xa = cj t , c 6= 0. ⇔ jt ∈ M X . ⇔ Mj t |{z} = 0. tth column of M ⇔ mt = 0. 2 ⇔ mt = mt,t = 0. ⇔ non(ii). mt denotes the tth row of M (and also its t column since M is symmetric). (iii) ⇔ (i) > • γtout = 0> ,1 | k {z } l> ! β . γtout 14.1. LEAVE-ONE-OUT AND OUTLIER MODEL 270 > • γtout is estimable parameter of Mout ⇔ l ∈ M X, j . t t ⇔ ∃a ∈ Rn such that 0> , 1 = a> X, j t . ⇔ ∃a ∈ Rn such that 0> = a> X & 1 = a> j t . ⇔ ∃a ∈ Rn such that 0> = a> X & at = 1. > ⇔ ∃a ∈ Rn such that x> t = − a(−t) X(−t) . ⇔ xt ∈ M X> (−t) . ⇔ (i). (iv) ⇔ (i) • Follows directly from Theorem 2.7. k Theorem 14.2 Equivalence of the outlier model and the leave-one-out model. are the same, i.e., 1. The residual sums of squares in models M(−t) and Mout t SSe,(−t) = SSout e,t . out 2. Vector b(−t) solves the normal equations of model M(−t) if and only if a vector bout t , ct out solves the normal equations of model Mt , where bout = b(−t) , t cout = Yt − xt > b(−t) . t Proof. Solution to normal equations minimizes the corresponding sum of squares. The sum of squares to be minimized w.r.t. β and γtout in the outlier model Mout is t 2 SSout β, γtout = Y − Xβ − j t γtout separate the tth element of the sum t 2 out 2 = Y (−t) − X(−t) β + Yt − x> t β − γt out 2 = SS(−t) (β) + Yt − x> , t β − γt where SS(−t) (β) is the sum of squares to be minimized w.r.t. β in the leave-one-out model M(−t) . out The term Yt − x> t β − γt 2 can for any β ∈ Rk be equal to zero if we, for given β ∈ Rk , take γtout = Yt − x> t β. 14.1. LEAVE-ONE-OUT AND OUTLIER MODEL 271 That is out (i) min SSout t (β, γt ) = min SS(−t) (β); β β, γtout | {z } | {z } out SS e,(−t) SSe,t (ii) A vector b(−t) ∈ Rk minimizes SS(−t) (β) if and only if a vector k+1 b(−t) , Yt − x> t b(−t) ∈ R {z } | {z } | bout cout t t out minimizes SSout t (β, γt ). k Notation (Leave-one-out least squares estimators of the response expectations). If mt,t > 0 for all t = 1, . . . , n, we will use the following notation: Yb[t] := x> t b(−t) , t = 1, . . . , n, which is the LSE of the parameter µt = E Yt X t = xt = x> t β based on the leave-one-out model M(−t) ; Yb [•] := Yb[1] , . . . , Yb[n] , which is an estimator of the parameter µ = µ1 , . . . , µn = E Y X , where each element is estimated using the linear model based on data with the corresponding observation being left out. Calculation of quantities of the outlier and the leave-one-out models Model Mout is a model with added regressor for model M. Suppose that mt,t > 0 for given t t = 1, . . . , n. By applying Lemma 10.1, we can express the LSE of the parameter γtout as γ btout = j> t Mj t − − −1 j> t U = (mt,t ) Ut = (mt,t ) Ut = Ut . mt,t Analogously, other quantities of the outlier model can be expressed using the quantities of model M. Namely, − Ut bout = b− X> X x t , t mt,t out Ut Yb t = Yb + mt , mt,t 2 Ut2 SSe − SSout = = MSe Utstd , e,t mt,t where mt denotes the tth column (and row as well) of the residual projection matrix M. 14.1. LEAVE-ONE-OUT AND OUTLIER MODEL 272 Lemma 14.3 Quantities of the outlier and leave-one-out model expressed using quantities of the original model. Suppose that for given t ∈ {1, . . . , n}, mt,t > 0. The following quantities of the outlier model Mout t and the leave-one-out model M(−t) are expressable using the quantities of the original model M as follows. Ut b γ btout = Yt − x> , t b(−t) = Yt − Y[t] = mt,t − Ut b(−t) = bout = b− X> X x t , t mt,t (14.1) Ut2 std 2 out = SSe − MSe Ut SSe,(−t) = SSe,t = SSe − , mt,t 2 MSout MSe,(−t) n − r − Utstd e,t = = . MSe MSe n−r−1 Proof. Equality between the quantities of the outlier and the leave-one-out model follows from Theorem 14.2. Remaining expressions follow from previously conducted calculations. To see the last equality in (14.1), remember that the residual degrees of freedom of both the outlier and the leave-one-out models are equal to n − r − 1. That is, whereas in model M, MSe = SSe , n−r in the outlier and the leave-one-out model, MSe,(−t) = SSout SSe,(−t) e,t = = MSout e,t . n−r−1 n−r−1 k Notes. • Expressions in Lemma 14.3 quantify the influence of the tth observation on (i) the LSE of a vector β of the regression coefficients (in case they are estimable); (ii) the estimate of the residual variance. • Lemma 14.3 also shows that it is not necessary to fit n leave-one-out (or outlier models) to calculate their LSE-related quantities. All important quantities can be calculated directly from the LSE-related quantities of the original model M. 14.1. LEAVE-ONE-OUT AND OUTLIER MODEL 273 Definition 14.3 Deleted residual. If mt,t > 0, then the quantity γ btout = Yt − Yb[t] = is called the tth deleted residual of the model M. Ut mt,t 14.2. OUTLIERS 14.2 274 Outliers By outliers3 of the model M, we shall understand observations for which the response expectation does not follow the assumed model, i.e., the tth observation (t ∈ {1, . . . , n}) is an outlier if E Yt X t = xt 6= x> t β, in which case we can write out E Yt X t = xt = x> t β + γt . As such, an outlier can be characterized as an observation with unusual response (y) value. If mt,t > 0, γtout is an estimable parameter of the tth outlier model Mout (for which the model M t is a submodel) and decision on whether the tth observation is an outlier can be transferred into a problem of testing H0 : γtout = 0 in the tth outlier model Mout t . Note that the above null hypothesis also expresses the fact that the submodel M of the model Mout holds. t If normality is assumed, this null hypothesis can be tested using a classical t-test on a value of the estimable parameter. The corresponding t-statistic has a standard form γ bout Tt = q t c γ var btout and under the null hypothesis follows the Student t distribution with n − r − 1 degrees of freedom (residual degrees of freedom of the outlier model). From Section 14.1, we have γ btout = Ut = Yt − Yb[t] . mt,t Hence (the variance is conditional given the covariate values), (?) 1 2 Ut 1 σ2 out var γ bt = var = var U = σ m = . t t,t mt,t mt,t m2t,t m2t,t (?) The equality = holds irrespective of whether γtout = 0 (and model M holds) or γtout 6= 0 (and model Mout holds). t The estimator γ btout is the LSE of a parameter of the outlier model and hence MSout e,t c γ var btout = , mt,t and finally, γ bout Tt = r t . MSout e,t mt,t Two useful expressions of the statistic Tt are obtained by remembering from Section 14.1 (a) t MSout btout = Yt − Yb[t] = γ btout = mUt,t . This leads e,t = MSe,(−t) and (b) two expressions of γ to Yt − Yb[t] √ Ut Tt = p mt,t = p . MSe,(−t) MSe,(−t) mt,t 3 odlehlá pozorování 14.2. OUTLIERS 275 Definition 14.4 Studentized residual. If mt,t > 0, then the quantity Yt − Yb[t] √ Ut Tt = p mt,t = p MSe,(−t) MSe,(−t) mt,t is called the tth studentized residual4 of the model M. Notes. • Using the last equality in (14.1), we can derive one more expression of the studentized residual using the standardized residual Ut Utstd = p . MSe mt,t Namely, s Tt = n−r−1 n−r− 2 Utstd Utstd . This directly shows that it is not necessary to fit the leave-one-out or the outlier model to calculate the studentized residual of the initial model M. Theorem 14.4 On studentized residuals. 2 Let Y X ∼ Nn Xβ, σ In , where rank Xn×k = r ≤ k < n. Let further n > r + 1. Let for given t ∈ 1, . . . , n mt,t > 0. Then 1. The tth studentized residual Tt follows the Student t-distribution with n − r − 1 degrees of freedom. 2. If additionally n > r + 2 then E Tt = 0. n−r−1 3. If additionally n > r + 3 then var Tt = . n−r−3 Proof. Point (i) follows from preceeding derivations, points (ii) and (iii) follow from properties of the Student t distribution. Test for outliers The studentized residual Tt of the model M is the test statistic (with tn−r−1 distribution under the null hypothesis) of the test 4 studentizované reziduum 14.2. OUTLIERS H0 : γtout = 0, H1 : γtout 6= 0 276 out 2 in the tth outlier model Mout t : Y X ∼ Nn Xβ + j t γt , σ In . The above testing problem can also be interpreted as a test of H0 : tth observations is not outlier of model M, H1 : tth observations is outlier of model M, where “outlier” means outlier with respect to model M: Y X ∼ Nn Xβ, σ 2 In : • The expected value of the tth observation is different from that given by model M; • The observed value of Yt is unusual under model M. When performing the test for outliers for all observations in the dataset, we are in fact facing a multiple testing problem and hence adjustment of the P-values resulted from comparison of the values of the studentized residuals with the quantiles of the Student tn−r−1 distribution are needed to keep the rate of falsely identified outliers under the requested level of α. For example, Bonferroni adjustment can be used. Notes. • Two or more outliers next to each other can hide each other. • A notion of outlier is always relative to considered model (also in other areas of statistics). Observation which is outlier with respect to one model is not necessarily an outlier with respect to some other model. • Especially in large datasets, few outliers are not a problem provided they are not at the same time also influential for statistical inference (see next section). • In a context of a normal linear model, presence of outliers may indicate that the error distribution is some distribution with heavier tails than the normal distribution. • Outlier can also suggest that a particular observation is a data-error. • If some observation is indicated to be an outlier, it should always be explored: • Is it a data-error? If yes, try to correct it, if this is impossible, no problem (under certain assumptions) • • • • to exclude it from the data. Is the assumed model correct and it is possible to find a physical/practical explanation for occurrence of such unusual observation? If an explanation is found, are we interested in capturing such artefacts by our model or not? Do the outlier(s) show a serious deviation from the model that cannot be ignored (for the purposes of a particular modelling)? .. . • NEVER, NEVER, NEVER exclude “outliers” from the analysis in an automatic manner. • Often, identification of outliers with respect to some model is of primary interest: • Example: model for amount of credit card transactions over a certain period of time depending on some factors (age, gender, income, . . . ). • Model found to be correct for a “standard” population (of clients). • Outlier with respect to such model ≡ potentially a fraudulent use of the credit card. • If the closer analysis of “outliers” suggest that the assumed model is not satisfactory capturing the reality we want to capture (it is not useful), some other model (maybe not linear, maybe not normal) must be looked for. 14.3. LEVERAGE POINTS 14.3 277 Leverage points By leverage points5 of the model M, we shall understand observations with, in a certain sense, unusual regressor (x) values. As will be shown, the fact whether the regressor values of a certain observation are unusual is closely related to the diagonal elements h1,1 , . . . , hn,n of the hat matrix − H = X X> X X> of the model. Terminology (Leverage). A diagonal element ht,t (t = 1, . . . , n) of the hat matrix H is called the leverage of the tth observation. Interpretation of the leverage To show that the leverage expresses how unusual the regressor values of the tth observations are, let us consider a linear model with intercept, i.e., the realized model matrix is X = 1n , x1 , . . . , xk−1 , where x1,1 . . x1 = . , xn,1 Let x1,k−1 . . = . . xn,k−1 ..., xk−1 ..., xk−1 = n x1 = 1X xi,1 , n n i=1 1X xi,k−1 n i=1 be the means of the non-intercept columns of the model matrix. That is, a vector x = x1 , . . . , xk−1 provides the mean values of the non-intercept regressors included in the model matrix X and as such is a gravity centre of the rows of the model matrix X (with excluded intercept). e be the non-intercept part of the model matrix X with all columns being centered, i.e., Further, let X x1,1 − x1 . . . x1,k−1 − xk−1 .. .. .. e = x1 − x1 1n , . . . , xk−1 − xk−1 1n = . X . . . 1 k−1 xn,1 − x . . . xn,k−1 − x e . Hence the hat matrix H = X X> X − X> can also be calculated Clearly, M X = M 1n , X e , where we can use additional property 1> X e = 0> : using the matrix 1, X n k−1 n > o− > e e e e H = 1n , X 1n , X 1n , X 1n , X > e − 1> n 1n 1n X | {z } | {z } > n 1n 0> k−1 e = 1n , X e> e> X e >X e X 1n X | {z } 0k−1 5 vzdálená pozorování 14.3. LEVERAGE POINTS 278 1 e n = 1n , X 0k−1 = 0> k−1 e> X e X − 1> n e> X 1 e e> e − X e >. 1n 1> n + X X X n That is, the tth leverage equals ht,t = > − > 1 e X e xt,1 − x1 , . . . , xt,k−1 − xk−1 . + xt,1 − x1 , . . . , xt,k−1 − xk−1 X n The second term is then a square of the generalized distance between the non-intercept regressors xt,1 , . . . , xt,k−1 of the tth observation and the vector of mean regressors x. Hence the observations with a high value of the leverage ht,t are observations with the regressor values being far from the mean regressor values and in this sense have unusual regressor (x) values. High value of a leverage To evaluate which values of the leverage are high enough to call a particular observation as a leverage point, let us remind an expression of the hat matrix using the orthonormal basis Q of the regression space M X , which is a vector space of dimension r = rankX. We know that H = QQ> and hence n X hi,i = tr(H) = tr QQ> = tr Q> Q = tr(Ir ) = r. i=1 That is, n h= 1X r hi,i = . n n (14.2) i=1 Several rules of thumbs can be found in the literature and software implementations concerning a lower bound for the leverage to call a particular observation as a leverage point. Owing to (14.2), a reasonable bound is a value higher than nr . For example, the R function influence.measures marks the tth observation as a leverage point if ht,t > 3r . n Influence of leverage points The fact that the leverage points may constitute a problem for the least squares based statistical inference in a linear model comes from remembering an expression for the variance (conditional given the covariate values) of the residuals of a linear model: var(Ut ) = σ 2 mt,t = σ 2 (1 − ht,t ), t = 1, . . . , n. Remind that Ut = Yt − Ybt and hence also var Yt − Ybt = σ 2 (1 − ht,t ), t = 1, . . . , n. That is, var(Ut ) = var Yt − Ybt is low for observations with a high leverage. In other words, the fitted values of high leverage observations are forced to be closer to the observed response values than those of low leverage observations. In this way, the high leverage observations have a higher impact on the fitted regression function than the low leverage observations. End of Lecture #25 (07/01/2016) 14.4. INFLUENTIAL DIAGNOSTICS 14.4 279 Influential diagnostics Start of Both outliers and leverage points do not necessarily constitute a problem. This occurs if they too Lecture #26 much influence statistical inference of primary interest. Also other observations (neither outliers nor (14/01/2016) leverage points) may harmfully influence the statistical inference. In this section, several methods of quantifying the influence of a particular, tth (t = 1, . . . , n) observation on statistical inference will be introduced. In all cases, we will compare a quantity of primary interest based on the model at hand, i.e., M : Y X ∼ Xβ, σ 2 In , rank Xn×k = r, and the quantity based on the leave-one-out model M(−t) : Y (−t) X(−t) ∼ X(−t) β, σ 2 In−1 . It will overally be assumed, that mt,t > 0 which implies (see Theorem 14.1) rank X(−t) rank X = r. 14.4.1 = DFBETAS Let r = k, i.e., both models M and M(−t) are full-rank models. The LSE’s of the vector of regression coefficients based on the two models are −1 > b = βb0 , . . . , βbk−1 > M: β = X> X X Y, M(−t) : b β (−t) = βb(−t),0 , . . . , βb(−t),k−1 Using (14.1): b − β b β (−t) = > = X(−t) > X(−t) −1 X(−t) > Y (−t) . −1 Ut X> X xt , mt,t (14.3) which quantifies influence of the tth observation on the LSE of the regression coefficients. In the following, let v 0 = v0,0 , . . . , v0,k−1 , . . . , v k−1 = vk−1,0 , . . . , vk−1,k−1 be the rows of the −1 matrix X> X , i.e., v> v . . . v 0,0 0,k−1 0 −1 . . .. .. > . . . X X = . = . . . v> v . . . v k−1,0 k−1,k−1 k−1 Expression (14.3) written elementwise lead to a quantities called DFBETA: DFBETAt,j := βbj − βb(−t),j = Ut > v xt , mt,t t t = 1, . . . , n, j = 0, . . . , k − 1. Note that DFBETAt,j has a scale of the jth regressor. To get a dimensionless quantity, we can divide it by the standard error of either βbj or βb(−t),j . We have S.E. βbj = p MSe vj,j , S.E. βb(−t),j = q MSe,(−t) v(−t),j,j , −1 where v(−t),j,j is p the jth diagonal element of matrix X(−t) > X(−t) . In practice, a combined quantity, namely MSe,(−t) vj,j is used leading to so called DFBETAS (the last “S” stands for 14.4. INFLUENTIAL DIAGNOSTICS 280 “scaled”): βbj − βb(−t),j DFBETASt,j := p MSe,(−t) vj,j = mt,t p Ut v> xt , MSe,(−t) vj,j t t = 1, . . . , n, j = 0, . . . , k − 1. p The reason for using MSe,(−t) vj,j as a scale factor is that MSe,(−t) is a safer estimator of the residual variance σ 2 not being based on the observation whose influence is examined but at the same time, it can still be calculated from quantities of the full model M (see Eq. 14.1). On the other hand, a value of v(−t),j,j (that fits with the leave-one-out residual mean square MSe,(−t) ) cannot, in general, be calculated from quantities of the full model M and hence (a close) value of vj,j is used. Consequently, all values of DFBETAS can be calculated from quantities of the full model M and there is no need to fit n leave-one-out models. Note (Rule-of-thumb used by R). The R function influence.measures marks the tth observation as being influential with respect to the LSE of the jth regression coefficient if DFBETASt,j > 1. 14.4.2 DFFITS We are assuming mt,t > 0 and hence by Theorem 14.1, parameter µt := E Yt X t = xt = x> t β − > > is estimable in both models M and M(−t) . Let as usual, b = X X X Y be any solution to normal equations in model M (which is now not necessarily of a full-rank) and let b(−t) = − X(−t) > X(−t) X(−t) > Y (−t) be any solution to normal equations in the leave-one-out model M(−t) . The LSE’s of µt in the two models are M: M(−t) : Ybt = x> t b, Yb[t] = x> t b(−t) . Using (14.1): n − o ht,t Ut Ut > > − > Yb[t] = x> b − X X xt = Ybt − xt X X xt = Ybt − Ut . t mt,t mt,t mt,t Difference between Ybt and Yb[t] is called DFFIT and quantifies influence of the tth observation on the LSE of its own expectation: ht,t , DFFITt := Ybt − Yb[t] = Ut mt,t t = 1, . . . , n. Analogously to DFBETAS, also DFFIT is scaled by a quantity that resembles the standard error of p either Ybt or Yb[t] (remember, S.E. Ybt = MSe ht,t ) leading to a quantity called DFFITS: Ybt − Yb[t] DFFITSt := p MSe,(−t) ht,t ht,t Ut p = = mt,t MSe,(−t) ht,t s ht,t Ut p = mt,t MSe,(−t) mt,t s ht,t Tt , mt,t t = 1, . . . , n, 14.4. INFLUENTIAL DIAGNOSTICS 281 where Tt is the tth studentized residual of the model M. Again, all values of DFFITS can be calculated from quantities of the full model M and there is no need to fit n leave-one-out models. Note (Rule-of-thumb used by R). The R function influence.measures marks the tth observation as excessively influencing the LSE of its expectation if r r DFFITSt > 3 . n−r 14.4.3 Cook distance In this Section, we concentrate on evaluation of the influence of the tth observation − on the LSE of a vector parameter µ := E Y X = Xβ. As in Section 14.4.2, let b = X> X X> Y be any − solution to normal equations in model M and let b(−t) = X(−t) > X(−t) X(−t) > Y (−t) be any solution to normal equations in the leave-one-out model M(−t) . The LSE’s of µ in the two models are M: M(−t) : Yb = Xb = HY , Yb (−t•) := Xb(−t) . Note. Remind that Yb (−t•) , Yb [•] and Yb (−t) are three different quantities. Namely, Yb (−t•) = Xb(−t) x> 1 b(−t) .. , = . > xn b(−t) Yb [•] Yb[1] x> 1 b(−1) . .. . . = . . = > b Y[n] xn b(−n) Finally, Yb (−t) = X(−t) b(−t) is a subvector of length n − 1 of a vector Yb (−t•) of length n. Possible quantification of influence of the tth observation on the LSE of a vector parameter µ is obtained by considering a quantity Yb − Yb (−t•) 2 . Let us remind from Lemma 14.3: b − b(−t) = − Ut X> X xt . mt,t Hence, Yb − Yb (−t•) = X b − b(−t) = − Ut X X> X x t . mt,t Then 2 Yb − Yb (−t•) 2 = Ut X X> X − xt mt,t = − Ut2 > > − > xt X X X X X> X xt 2 mt,t = Ut2 ht,t . m2t,t The equality (14.4) follows from noting that (14.4) 14.4. INFLUENTIAL DIAGNOSTICS 282 − > − − − > (a) x> X X X> X xt is the tth diagonal element of matrix X X> X X> X X> X X> ; t X X − − − (b) X X> X X> X X> X X> = X X> X X> = H by the five matrices rule (Theorem A.2). The so called Cook distance of the tth observation is (14.4) modified to get a unit-free quantity. Namely, the Cook distance is defined as Dt := 1 Yb − Yb (−t•) 2 . r MSe Expression (14.4) shows that it is again not necessary to fit the leave-one-out model to calculate the Cook distance. Moreover, we can express it as follows Dt = 2 1 ht,t Ut2 1 ht,t = Utstd . r mt,t MSe mt,t r mt,t Notes. • We are assuming mt,t > 0. Hence ht,t = 1 − mt,t ∈ (0, 1) and the term ht,t /mt,t increases with the leverage ht,t (having a limit of ∞ with ht,t → 1). The “ht,t /mt,t ” part of the Cook distance thus quantifies how much is the tth observation the leverage point. • The “Utstd ” part of the Cook distance increases with the distance between the observed and fitted value which is high for outliers. • The Cook distance is thus a combined measure being high for observations which are either leverage points or outliers or both. Cook distance in a full-rank model If r = k and both M and M(−t) are of full-rank, we have b = b = β b b(−t) = β (−t) = X> X −1 X> Y , X(−t) > X(−t) −1 X(−t) > Y (−t) . Then, directly from definition, 2 b − Xβ b Yb − Yb (−t•) 2 = Xβ = (−t) b b β (−t) − β > b b X> X β (−t) − β . The Cook distance is then Dt = b b β (−t) − β > b b X> X β (−t) − β , k MSe b and β b which is a distance between β (−t) in a certain metric. Remember now that under normality, the confidence region for parameter β with a coverage of 1 − α, derived while assuming model M is C(α) = β : b β − β That is b β (−t) ∈ C(α) > b X> X β − β if and only if < k MSe Fk,n−k (1 − α) . Dt < Fk,n−k (1 − α). (14.5) 14.4. INFLUENTIAL DIAGNOSTICS 283 This motivates the following rule-of-thumb. Note (Rule-of-thumb used by R). The R function influence.measures marks the tth observation as excessively influencing the LSE of the full response expectation µ if Dt > Fr,n−r (0.50). 14.4.4 COVRATIO In this Section, we will again assume full-rank models (r = k) and explore influence of the tth observation on precision of the LSE of the vector of regression coefficients. The LSE’s of the vector of regression coefficients based on the two models are M: b = β M(−t) : b β (−t) = X> X −1 X> Y , X(−t) > X(−t) −1 X(−t) > Y (−t) . b and β b The estimated covariance matrices of β (−t) , respectively, are b c β var −1 = MSe X> X , b c β var (−t) = MSe,(−t) X> (−t) X(−t) −1 . Influence of the tth observation on the precision of the LSE of the vector of regression coefficients is quantified by so called COVRATIO being defined as n o b c β det var (−t) n t = 1, . . . , n. COVRATIOt = o , b c β det var After some calculation (see below), it can be shown that COVRATIOt 1 = mt,t n − k − Utstd n−k−1 2 k , t = 1, . . . , n. That is, it is again not necessary to fit n leave-one-out models to calculate the COVARTIO values for all observations in the dataset. Note (Rule-of-thumb used by R). The R function influence.measures marks the tth observation as excessively influencing precision of the estimation of the regression coefficients if 1 − COVRATIOt > 3 k . n−k 14.4. INFLUENTIAL DIAGNOSTICS 284 Calculation towards COVRATIO First, remind a matrix identity (e.g., Anděl, 2007, Theorem A.4): If A and D are square invertible matrices then A B = A · D − CA−1 B = D · A − BD−1 C. C D Use twice the above identity: X> X x −1 t > xt = X> X mt,t , = X> X · 1 − x> t X X > xt | {z } 1 1 − ht,t = mt,t = X> X(−t) . = |1| · X> X − xt x> t (−t) So that, mt,t X> X = X> (−t) X(−t) . Then, n o −1 > X b c β det var MS X e,(−t) (−t) (−t) (−t) o n = −1 b c β det var MSe X> X = MSe,(−t) MSe k > −1 X MSe,(−t) k 1 (−t) X(−t) . · = · X> X−1 MSe mt,t Expression (14.1): MSe,(−t) n − k − Utstd = MSe n−k−1 2 Hence, n o 2 k b c β det var (−t) n − k − Utstd 1 n . o = m n−k−1 b t,t c β det var 14.4.5 Final remarks . • All presented influence measures should be used sensibly. • Depending on what is the purpose of the modelling, different types of influence are differently harmful. • There is certainly no need to panic if some observations are marked as “influential”! End of Lecture #26 (14/01/2016) Appendix A Matrices A.1 Pseudoinverse of a matrix Definition A.1 Pseudoinverse of a matrix. The pseudoinverse of a real matrix An×k is such a matrix A− of dimension k × n that satisfies AA− A = A. Notes. • The pseudoinverse always exists. Nevertheless, it is not necessarily unique. • If A is invertible then A− = A−1 is the only pseudoinverse. Definition A.2 Moore-Penrose pseudoinverse of a matrix. The Moore-Penrose pseudoinverse of a real matrix An×k is such a matrix A+ of dimension k × n that satisfies the following conditions: (i) AA+ A = A; (ii) A+ AA+ = A+ ; > (iii) AA+ = AA+ ; > (iv) A+ A = A+ A. Notes. • The Moore-Penrose pseudoinverse always exists and it is unique. • The Moore-Penrose pseudoinverse can be calculated from the singular value decomposition (SVD) of the matrix A. 285 Start of Lecture #2 (08/10/2015) A.1. PSEUDOINVERSE OF A MATRIX 286 Theorem A.1 Pseudoinverse of a matrix and a solution of a linear system. Let An×k be a real matrix and let cn×1 be a real vector. Let there exist a solution of a linear system Ax = c, i.e., the linear system Ax = c is consistent. Let A− be the pseudoinverse of A. A vector xk×1 solves the linear system Ax = c if and only if x = A− c. Proof. See Anděl (2007, Appendix A.4). k Theorem A.2 Five matrices rule. For a real matrix An×k , it holds That is, a matrix A> A − − A A> A A> A = A. A> is a pseudoinverse of a matrix A. Proof. See Anděl (2007, Theorem A.19). k End of Lecture #2 (08/10/2015) A.2. KRONECKER PRODUCT A.2 287 Kronecker product Start of Lecture #7 (22/10/2015) Definition A.3 Kronecker product. Let Am×n and Cp×q be real matrices. Their Kronecker product A ⊗ C is a matrix Dm·p×n·q such that a1,1 C . . . a1,s C . .. .. . D=A⊗C= . . . = ai,j C i=1,...,m,j=1,...,n . ar,1 C . . . ar,s C Note. For a ∈ Rm , b ∈ Rp , we can write a b> = a ⊗ b> . Theorem A.3 Properties of a Kronecker product. It holds for the Kronecker product: (i) 0 ⊗ A = 0, A ⊗ 0 = 0. (ii) (A1 + A2 ) ⊗ C = (A1 ⊗ C) + (A2 ⊗ C). (iii) A ⊗ (C1 + C2 ) = (A ⊗ C1 ) + (A ⊗ C2 ). (iv) aA ⊗ cC = a c (A ⊗ C). (v) A1 A2 ⊗ C1 C2 = (A1 ⊗ C1 ) (A2 ⊗ C2 ). −1 (vi) A ⊗ C = A−1 ⊗ C−1 , if the inversions exist. − (vii) A ⊗ C = A− ⊗ C− , for arbitrary pseudoinversions. > (viii) A ⊗ C = A> ⊗ C> . (ix) A, C ⊗ D = A ⊗ D, C ⊗ D . (x) Upon a suitable reordering of the columns, matrices A ⊗ C, A ⊗ D and A ⊗ C, D are the same. (xi) rank A ⊗ C = rank A rank C . Proof. See Rao (1973, Section 1b.8). k End of Lecture #7 (22/10/2015) A.2. KRONECKER PRODUCT 288 Start of Lecture #11 (05/11/2015) Definition A.4 Elementwise product of two vectors. Let a = a1 , . . ., ap ∈ Rp , c = b1 , . . . , bp ∈ Rp . Their elementwise product1 is a vector a1 c1 , . . . , ap cp that will be denoted as a : c. That is, a1 c1 . . a:c= . . ap cp Definition A.5 Columnwise product of two matrices. Let An×p = a1 , . . . , ap and Cn×q = c1 , . . . , cq be real matrices. Their columnwise product2 A : C is a matrix Dn×p·q such that D = A : C = a1 : c1 , . . . , ap : c1 , . . . , a1 : cq , . . . , ap : cq . Notes. • If we write a> 1 . . A= . , > an c> 1 . . C= . , > cn the columnwise product of two matrices can also be written as a matrix rows of which are obtained as Kronecker products of the rows of the two matrices: > c> ⊗ a 1 1 .. . A:C= (A.1) . > > cn ⊗ an • It perhaps looks more logical to define the columnwise product of two matrices as > a> ⊗ c 1 1 .. = a1 : c1 , . . . , a1 : cq , . . . , ap : c1 , . . . , ap : cq , A:C= . > a> n ⊗ cn which only differs by ordering of the columns of the resulting matrix. Our definition (A.1) is motivated by the way in which an operator : acts in the R software. 1 součin po složkách 2 součin po sloupcích End of Lecture #11 (05/11/2015) A.3. ADDITIONAL THEOREMS ON MATRICES A.3 289 Additional theorems on matrices Theorem A.4 Inverse of a matrix divided into blocks. Let M= A B B> D be a positive definite matrix divided in blocks A, B, D. Then the following holds: (i) Matrix Q = A − BD−1 B> is positive definite. (ii) Matrix P = D − B> A−1 B is positive definite. (iii) The inverse to M is M−1 = = Q−1 − D−1 B> Q−1 − Q−1 BD−1 D−1 A−1 + A−1 BP−1 B> A−1 − P−1 B> A−1 Proof. See Anděl (2007, Theorem A.10 in Appendix A.2). + D−1 B> Q−1 BD−1 − A−1 BP−1 P−1 . k Appendix B Distributions B.1 Non-central univariate distributions Definition B.1 Non-central Student t-distribution. Let U ∼ N (0, 1), let V ∼ χ2ν for some ν > 0 and let U and V be independent. Let λ ∈ R. Then we say that a random variable U +λ T = r V ν follows a non-central Student t-distribution1 with ν degrees of freedom2 and a non-centrality parameter 3 λ. We shall write T ∼ tν (λ). Notes. • Non-central t-distribution is different from simply a shifted (central) t-distribution. • Directly seen from definition: tν (0) ≡ tν . • Moments of a non-central Student t-distribution: r Γ ν−1 ν 2 λ , if ν > 1, 2 Γ ν2 E(T ) = does not exist, if ν ≤ 1. 2 2) 2 Γ ν−1 ν(1 + λ νλ 2 − , if ν > 2, ν−2 2 Γ ν2 var(T ) = does not exist, if ν ≤ 2. 1 necentrální Studentovo t-rozdělení 2 stupně volnosti 3 parametr necentrality 290 Start of Lecture #6 (22/10/2015) B.1. NON-CENTRAL UNIVARIATE DISTRIBUTIONS 291 Definition B.2 Non-central χ2 distribution. Let U1 , . . . , Uk be independent random variables. Let further Ui ∼ N (µi , 1), i = 1, .. . , k, for some µ1 , . . . , µk ∈ R. That is U = U1 , . . . , Uk ∼ Nk µ, Ik , where µ = µ1 , . . . , µk . Then we say that a random variable k X 2 X= Ui2 = U i=1 follows a non-central chi-squared distribution4 with k degrees of freedom and a non-centrality parameter k X λ= µ2i = kµk2 . i=1 We shall write X ∼ χ2k (λ). Notes. • It can easily be proved that the of the random variable X from Definition B.2 indeed Pdistribution k 2 depends only on k and λ = i=1 µi and not on the particular values of µ1 , . . . , µk . • As an exercise for the use of a convolution theorem, we can derive a density of the χ2k (λ) distribution which is ∞ − x+λ k−2 X λ j xj k−1 1 e 2 x 2 B , +j , x > 0, k 1 (2j)! 2 2 f (x) = Γ 2 2 Γ k−1 j=0 2 2 0, x ≤ 0. • The non-central χ2 distribution with general degrees of freedom ν ∈ (0, ∞) is defined as a distribution with the density given by the above expression with k replaced by ν. • χ2ν (0) ≡ χ2ν . • Moments of a non-central χ2 distribution: E(X) = ν + λ, var(X) = 2 (ν + 2λ). 4 necentrální chí-kvadrát rozdělení B.1. NON-CENTRAL UNIVARIATE DISTRIBUTIONS 292 Definition B.3 Non-central F-distribution. Let X ∼ χ2ν1 (λ), where ν1 , λ > 0. Let Y ∼ χ2ν2 , where ν2 > 0. Let further X and Y be independent. Then we say that a random variable X ν1 Q= Y ν2 follows a non-central F-distribution5 with ν1 and ν2 degrees of freedom and a noncentrality parameter λ. We shall write Q ∼ Fν1 ,ν2 (λ). Notes. • Directly seen from definition: Fν1 ,ν2 (0) ≡ Fν1 ,ν2 . • Moments of a non-central F-distribution: ν (ν + λ) 2 1 , if ν2 > 2, ν1 (ν2 − 2) E(Q) = does not exist, if ν2 ≤ 2. 2 2 + (ν + 2λ) (ν − 2) (ν + λ) ν2 1 1 2 2 , if ν2 > 4, 2 (ν2 − 2) (ν2 − 4) ν1 var(Q) = does not exist, if ν2 ≤ 4. End of Lecture #6 (22/10/2015) 5 necentrální F-rozdělení B.2. MULTIVARIATE DISTRIBUTIONS B.2 293 Multivariate distributions Start of Lecture #4 (15/10/2015) Definition B.4 Multivariate Student t-distribution. Let U ∼ Nn (0n , Σ), where Σn×n is a positive semidefinite matrix. Let further V ∼ χ2ν for some ν > 0 and let U and V be independent. Then we say that a random vector r ν T =U V follows an n-dimensional multivariate Student t-distribution6 with ν degrees of freedom and a scale matrix7 Σ. We shall write T ∼ mvtn,ν (Σ). Notes. • Directly seen from definition: mvt1,ν (1) ≡ tν . • If Σ is a regular (positive definite) matrix, then the density of the mvtn,ν (Σ) distribution is f (t) = ν+n 2 Γ Γ ν 2 n 2 ν π n 2 − 1 Σ 2 t> Σ−1 t 1+ ν − ν+n 2 , t ∈ Rn . • Expectation and a covariance matrix of T ∼ mvtn,ν (Σ) are E(T ) = var(T ) = 0n , if ν > 1, does not exist, if ν ≤ 1. ν Σ, ν−2 if ν > 2, does not exist, if ν ≤ 2. Lemma B.1 Marginals of the multivariate Student t-distribution. > Let T = T1 , . . . , Tn ∼ mvtn,ν (Σ), where the scale matrix Σ has positive diagonal elements σ12 > 0, . . . , σn2 > 0. Then Tj ∼ tν , j = 1, . . . , n. σj r Proof. ν • From definition of the multivariate t-distribution, T can be written as T = U , where V > U = U1 , . . . , Un ∼ Nn (0n , Σ) and V ∼ χ2ν are independent. 6 vícerozměrné Studentovo t-rozdělení 7 měřítková matice B.2. MULTIVARIATE DISTRIBUTIONS • Then for all j = 1, . . . , n: Tj Uj = σj σj 294 r Zj ν =q , V V ν where Zj ∼ N (0, 1) is independent of V ∼ χ2ν . k End of Lecture #4 (15/10/2015) Appendix C Asymptotic Theorems Start of Lecture #22 Theorem C.1 Strong law of large numbers (SLLN) for i.n.n.i.d. random variables. (17/12/2015) Let Z1 , Z2 , . . . be a sequence of independent not necessarily identically distributed (i.n.n.i.d.) random variables. Let E(Zi ) = µi , var(Zi ) = σi2 , i = 1, 2, . . .. Let ∞ X σ2 i i=1 Then i2 < ∞. n 1 X a.s. (Zi − µi ) −→ 0 n as n → ∞. i=1 Proof. See Probability and Mathematical Statistics (NMSA202) lecture (2nd year of the Bc. study programme). k Theorem C.2 Strong law of large numbers (SLLN) for i.i.d. random variables. Let Z1 , Z2 , . . . be a sequence of independent identically distributed (i.i.d.) random variables. Then n 1 X a.s. Zi −→ µ n as n → ∞ i=1 for some µ ∈ R if and only if EZ1 < ∞, in which case µ = E Z1 . 295 296 Proof. See Probability and Mathematical Statistics (NMSA202) lecture (2nd year of the Bc. study programme). k Theorem C.3 Central limit theorem (CLT), Lyapunov. Let Z1 , Z2 , . . . be a sequence of i.n.n.i.d. random variables with E Zi = µi , ∞ > var Zi = σi2 > 0, Let for some δ > 0 Zi − µi 2+δ E i=1 −→ 0 P 2+δ 2 n 2 i=1 σi Pn Then i = 1, 2, . . . as n → ∞. Pn D i=1 (Zi − µi ) q −→ N (0, 1) as n → ∞. Pn 2 i=1 σi Proof. See Probability Theory 1 (NMSA333) lecture (3rd year of the Bc. study programme). k Theorem C.4 Central limit theorem (CLT), i.i.d.. Let Z1 , Z2 , . . . be a sequence of i.i.d. random variables with E Zi = µ, ∞ > var Zi = σ 2 > 0, P Let Z n = n1 ni=1 Zi . i = 1, 2, . . . . Then n 1 X Zi − µ D √ −→ N (0, 1) σ n as n → ∞, i=1 √ n Zn − µ D −→ N (0, σ 2 ) as n → ∞. Proof. See Probability Theory 1 (NMSA333) lecture (3rd year of the Bc. study programme). k 297 Theorem C.5 Central limit theorem (CLT), i.i.d. multivariate. Let Z 1 , Z 2 , . . . be a sequence of i.i.d. p-dimensional random vectors with E Z i = µ, var Z i = Σ, i = 1, 2, . . . , P where Σ is a real positive semidefinite matrix. Let Z n = n1 ni=1 Z i . Then √ n Zn − µ D −→ Np (0p , Σ). If Σ is positive definite then also n D 1 X −1/2 √ Σ Z i − µ −→ Np (0p , Ip ). n i=1 Proof. See Probability Theory 1 (NMSA333) lecture (3rd year of the Bc. study programme). k Theorem C.6 Cramér-Wold. Let Z 1 , Z 2 , . . . be a sequence of p-dimensional random vectors. Let Z be a p-dimensional random vector. D Z n −→ Z as n → ∞ if and only if for all l ∈ Rp D l> Z n −→ l> Z as n → ∞. Proof. See Probability Theory 1 (NMSA333) lecture (3rd year of the Bc. study programme). k 298 Theorem C.7 Cramér-Slutsky. Let Z 1 , Z 2 , . . . be a sequence of random vectors such that D Z n −→ Z as n → ∞, where Z be a random vector. Let S1 , S2 , . . . be a sequence of random variables such that P Sn −→ S as n → ∞, where S ∈ R is a real constant. Then D (i) Sn Z n −→ S Z (ii) 1 D 1 Z n −→ Z Sn S as n → ∞. as n → ∞, if S 6= 0. Proof. See Probability Theory 1 (NMSA333) lecture (3rd year of the Bc. study programme). See also Shao (2003, Theorem 1.11 in Section 1.5). k End of Lecture #22 (17/12/2015) Bibliography Anděl, J. (2007). Základy matematické statistiky. Matfyzpress, Praha. ISBN 80-7378-001-1. Bartlett, M. S. (1937). Properties of sufficiency and statistical tests. Proceedings of the Royal Society of London, Series A, Mathematical and Physical Sciences, 160(901), 268–282. doi: 10.1098/rspa.1937. 0109. Breusch, T. S. and Pagan, A. R. (1979). A simple test for heteroscedasticity and random coefficient variation. Econometrica, 47(5), 1287–1294. doi: 10.2307/1911963. Brown, M. B. and Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346), 364–367. doi: 10.1080/01621459.1974.10482955. Cipra, T. (2008). Finanční ekonometrie. Ekopress, Praha. ISBN 978-80-86929-43-9. Cook, R. D. and Weisberg, S. (1983). Diagnostics for heteroscedasticity in regression. Biometrika, 70 (1), 1–10. doi: 10.1093/biomet/70.1.1. Cribari-Neto, F. (2004). Asymptotic inference under heteroskedasticity of unknown form. Computational Statistics and Data Analysis, 45(2), 215–233. doi: 10.1016/S0167-9473(02)00366-3. de Boor, C. (1978). A Practical Guide to Splines. Springer, New York. ISBN 0-387-90356-9. de Boor, C. (2001). A Practical Guide to Splines. Springer-Verlag, New York, Revised edition. ISBN 0-387-95366-3. Dierckx, P. (1993). Curve and Surface Fitting with Splines. Clarendon, Oxford. ISBN 0-19-853440-X. Draper, N. R. and Smith, H. (1998). Applied Regression Analysis. John Wiley & Sons, New York, Third edition. ISBN 0-471-17082-8. Durbin, J. and Watson, G. S. (1950). Testing for serial correlation in least squares regression I. Biometrika, 37, 409–428. Durbin, J. and Watson, G. S. (1951). Testing for serial correlation in least squares regression II. Biometrika, 38(1/2), 159–177. doi: 10.2307/2332325. Durbin, J. and Watson, G. S. (1971). Testing for serial correlation in least squares regression III. Biometrika, 58(1), 1–19. doi: 10.2307/2334313. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. doi: 10.1214/aos/1176344552. 299 BIBLIOGRAPHY 300 Eilers, P. H. C. and Marx, B. D. (1996). Flexible smoothing with B-splines and penalties (with Discussion). Statistical Science, 11(1), 89–121. doi: 10.1214/ss/1038425655. Farebrother, R. W. (1980). Algorithm AS 153: Pan’s procedure for the tail probabilities of the Durbin-Watson statistics. Applied Statistics, 29(2), 224–227. Farebrother, R. W. (1984). Remark AS R53: A remark on algorithm AS 106, AS 153, AS 155: The distribution of a linear combination of χ2 random variables. Applied Statistics, 33, 366–369. Fligner, M. A. and Killeen, T. J. (1976). Distribution-free two-sample tests for scale. Journal of the American Statistical Association, 71(353), 210–213. doi: 10.2307/2285771. Fox, J. and Monette, G. (1992). Generalized collinearity diagnostics. Journal of the American Statistical Association, 87(417), 178–183. doi: 10.1080/01621459.1992.10475190. Genz, A. and Bretz, F. (2009). Computation of Multivariate Normal and t Probabilities. Springer-Verlag, New York. ISBN 978-3-642-01688-2. Goldfeld, S. M. and Quandt, R. E. (1965). Some tests for homoscedasticity. Journal of the American Statistical Association, 60(310), 539–547. doi: 10.1080/01621459.1965.10480811. Hayter, A. J. (1984). A proof of the conjecture that the Tukey-Kramer multiple comparisons procedure is conservative. The Annals of Statistics, 12(1), 61–75. doi: 10.1214/aos/1176346392. Hothorn, T., Bretz, F., and Westfall, P. (2008). Simultaneous inference in general parametric models. Biometrical Journal, 50(3), 346–363. doi: 10.1002/bimj.200810425. Hothorn, T., Bretz, F., and Westfall, P. (2011). Multiple Comparisons Using R. Chapman & Hall/CRC, Boca Raton. ISBN 978-1-5848-8574-0. Khuri, A. I. (2010). Linear Model Methodology. Chapman & Hall/CRC, Boca Raton. ISBN 978-1-58488481-1. Koenker, R. (1981). A note on studentizing a test for heteroscedasticity. Journal of Econometrics, 17 (1), 107–112. doi: 10.1016/0304-4076(81)90062-2. Kramer, C. Y. (1956). Extension of multiple range tests to group means with unequal numbers of replications. Biometrics, 12(3), 307–310. doi: 10.2307/3001469. Levene, H. (1960). Robust tests for equality of variances. In Olkin, I., Ghurye, S. G., Hoeffding, W., Madow, W. G., and Mann, H. B., editors, Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling, pages 278–292. Stanford University Press, Standord. Long, J. S. and Ervin, L. H. (2000). Using heteroscedasticity consistent standard errors in the linear regression model. The American Statistician, 54(3), 217–224. doi: 10.2307/2685594. MacKinnon, J. G. and White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. Journal of Econometrics, 29(3), 305–325. doi: 10.1016/0304-4076(85)90158-7. R Core Team (2015). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org/. Rao, C. R. (1973). Linear Statistical Inference and its Applications. John Wiley & Sons, New York, Second edition. ISBN 0-471-21875-8. BIBLIOGRAPHY 301 Searle, S. R. (1987). Linear Models for Unbalanced Data. John Wiley & Sons, New York. ISBN 0-471-84096-3. Seber, G. A. F. and Lee, A. J. (2003). Linear Regression Analysis. John Wiley & Sons, New York, Second edition. ISBN 978-0-47141-540-4. Shao, J. (2003). Mathematical Statistics. Springer Science+Business Media, New York, Second edition. ISBN 0-387-95382-5. Sun, J. (2003). Mathematical Statistics. Springer Science+Business Media, New York, Second edition. ISBN 0-387-95382-5. Tukey, J. W. (1949). Comparing individual means in the Analysis of variance. Biometrics, 5(2), 99–114. doi: 10.2307/3001913. Tukey, J. W. (1953). The problem of multiple comparisons (originally unpublished manuscript). In Braun, H. I., editor, The Collected Works of John W. Tukey, volume 8, 1994. Chapman & Hall, New York. Weisberg, S. (2005). Applied Linear Regression. John Wiley & Sons, Hoboken, Third edition. ISBN 0-471-66379-4. White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48(4), 817–838. doi: 10.2307/1912934. Zeileis, A. (2004). Econometric computing with HC and HAC covariance matrix estimators. Journal of Statistical Software, 11(10), 1–17. URL http://www.jstatsoft.org/v11/i10/. Zvára, K. (1989). Regresní analýza. Academia, Praha. ISBN 80-200-0125-5. Zvára, K. (2008). Regrese. Matfyzpress, Praha. ISBN 978-80-7378-041-8.

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement