Tufte Without Tears Flexible tools for visual exploration and presentation of statistical models Christopher Adolph Department of Political Science and Center for Statistics and the Social Sciences University of Washington, Seattle Chris Adolph (UW) Tufte without tears 1 / 106 Edward Tufte & Scientific Visuals In several beautifully illustrated books Edward Tufte gives the following advice: 1 Assume the reader’s interest and intelligence Chris Adolph (UW) Tufte without tears 2 / 106 Edward Tufte & Scientific Visuals In several beautifully illustrated books Edward Tufte gives the following advice: 1 Assume the reader’s interest and intelligence 2 Maximize information, minimize ink & space: the data-ink ratio Chris Adolph (UW) Tufte without tears 2 / 106 Edward Tufte & Scientific Visuals In several beautifully illustrated books Edward Tufte gives the following advice: 1 Assume the reader’s interest and intelligence 2 Maximize information, minimize ink & space: the data-ink ratio 3 Show the underlying data and facilitate comparisons Chris Adolph (UW) Tufte without tears 2 / 106 Edward Tufte & Scientific Visuals In several beautifully illustrated books Edward Tufte gives the following advice: 1 Assume the reader’s interest and intelligence 2 Maximize information, minimize ink & space: the data-ink ratio 3 Show the underlying data and facilitate comparisons 4 Use small multiples: repetitions of a basic design Tufte’s recommended medical chart illustrates many of these ideas: Chris Adolph (UW) Tufte without tears 2 / 106 This chart is annotated for pedagogical purposes Lots of information; little distracting scaffolding A model that can be repeated once learned. . . Chris Adolph (UW) Tufte without tears 3 / 106 A complete layout using small multiples to convey lots of info Elegant, information-rich. . . and hard to make Chris Adolph (UW) Tufte without tears 4 / 106 What most discussions of statistical graphics leave out Tufte’s books have had a huge impact on information visualization However, they have two important limits: Modeling Most examples are either exploratory or very simple models; Social scientists want cutting edge applications Tools Need to translate aesthetic guidelines into software Social scientists are unlikely to do this on their own— and shouldn’t have to! Chris Adolph (UW) Tufte without tears 5 / 106 Key problem Ready-to-use techniques to visually present model results: for many variables Chris Adolph (UW) Tufte without tears 6 / 106 Key problem Ready-to-use techniques to visually present model results: for many variables for many robustness checks Chris Adolph (UW) Tufte without tears 6 / 106 Key problem Ready-to-use techniques to visually present model results: for many variables for many robustness checks showing uncertainty Chris Adolph (UW) Tufte without tears 6 / 106 Key problem Ready-to-use techniques to visually present model results: for many variables for many robustness checks showing uncertainty without accidental extrapolation Chris Adolph (UW) Tufte without tears 6 / 106 Key problem Ready-to-use techniques to visually present model results: for many variables for many robustness checks showing uncertainty without accidental extrapolation for an audience without deep statistical knowledge Not covered here The theory behind effective visual display of data Visual displays for data (not model) exploration For these and other topics, and a reading list, see my course at faculty.washington.edu/cadolph/vis Lots of examples. . . Too many if we need to discuss methods in detail Chris Adolph (UW) Tufte without tears 6 / 106 Who votes in American elections? Source: King, Tomz, and Wittenberg Method: Logistic regression What do alligators eat? Source: Agresti Method: Multinomial logit How do Chinese leaders gain power? Source: Shih, Adolph, and Liu Method: Bayesian model of partially observed ranks When do governments choose liberal or conservative central bankers? Source: Adolph Method: Zero-inflated compositional data model What explains the tier of European governments controlling health policies? Source: Adolph, Greer, and Fonseca Method: Multilevel multinomial logit Chris Adolph (UW) Tufte without tears 7 / 106 Presenting Estimated Models in Social Science Most empirical work in social science is regression model-driven, with a focus on conditional expectation Our regression models are full of covariates often non-linear usually involve interactions and transformations If there is anything we need to visualize well, it is our models Yet we often just print off tables of parameter estimates Limits readers’ and analysts’ understanding of the results Chris Adolph (UW) Tufte without tears 8 / 106 Coefficients are not enough Some limits of typical presentations of statistical results: Everything written in terms of arcane intermediate quantites (for most people, this includes logit coefficients) Little effort to transform results to the scale of the quantities of interest → really want the conditional expectation, E(y |x) Little effort to make informative statements about estimation uncertainty → really want to know how uncertain is E(y |x) Little visualization at all, or graphs with low data-ink ratios Chris Adolph (UW) Tufte without tears 9 / 106 Voting Example (Logit Model) We will explore a simple dataset using a simple model of voting People either vote (Votei = 1), or they don’t (Votei = 0) Many factors could influence turn-out; we focus on age and education Data from National Election Survey in 2000. “Did you vote in 2000 election?” [1,] [2,] [3,] [4,] [5,] [6,] [7,] [8,] [9,] [10,] ... vote00 age hsdeg coldeg 1 49 1 0 0 35 1 0 1 57 1 0 1 63 1 0 1 40 1 0 1 77 0 0 0 43 1 0 1 47 1 1 1 26 1 1 1 48 1 0 Chris Adolph (UW) Tufte without tears 10 / 106 Logit of Decision to Vote, 2000 Presidential NES est. Age Age2 High School Grad College Grad Constant 0.074 −0.0004 1.168 1.085 −3.05 s.e. 0.017 0.0002 0.178 0.131 0.418 p-value 0.000 0.009 0.000 0.000 0.000 Age enters as a quadratic to allow the probability of voting to first rise and eventually fall over the life course Results look sensible, but what do they mean? Which has the bigger effect, age or education? What is the probability a specific person will vote? Chris Adolph (UW) Tufte without tears 11 / 106 An alternative to printing eye-glazing tables 1 Run your model as normal. Treat the output as an intermediate step. Chris Adolph (UW) Tufte without tears 12 / 106 An alternative to printing eye-glazing tables 1 Run your model as normal. Treat the output as an intermediate step. 2 Translate your model results back into the scale of the response variable I Modeling war? Show the change in probability of war associated with X I Modeling counts of crimes committed? Show how those counts vary with X I Unemployment rate time series? Show how a change in X shifts the unemployment rate over the following t years Chris Adolph (UW) Tufte without tears 12 / 106 An alternative to printing eye-glazing tables 1 Run your model as normal. Treat the output as an intermediate step. 2 Translate your model results back into the scale of the response variable 3 I Modeling war? Show the change in probability of war associated with X I Modeling counts of crimes committed? Show how those counts vary with X I Unemployment rate time series? Show how a change in X shifts the unemployment rate over the following t years Calculate or simulate the uncertainty in these final quantities of interest Chris Adolph (UW) Tufte without tears 12 / 106 An alternative to printing eye-glazing tables 1 Run your model as normal. Treat the output as an intermediate step. 2 Translate your model results back into the scale of the response variable I Modeling war? Show the change in probability of war associated with X I Modeling counts of crimes committed? Show how those counts vary with X I Unemployment rate time series? Show how a change in X shifts the unemployment rate over the following t years 3 Calculate or simulate the uncertainty in these final quantities of interest 4 Present visually as many scenarios calculated from the model as needed Chris Adolph (UW) Tufte without tears 12 / 106 A bit more formally. . . We want to know the behavior of E(y |x) as we vary x. In non-linear models with multiple regressors, this gets tricky. The effect of x1 depends on all the other x’s and β̂’s Generally, we will need to make a set of “counterfactual” assumptions: x1 = a, x2 = b, x3 = c, . . . Choose a, b, c, . . . to match a particular counterfactual case of interest or Hold all but one of the x’s at their mean values (or other reference baseline), then systematically vary the remaining x. The same trick works if we are after differences in y related to changes in x, such as E(yscen1 − yscen2 |xscen1 , xscen2 ) Chris Adolph (UW) Tufte without tears 13 / 106 Calculating quantities of interest Our goal to obtain “quantities of interest,” like Expected Values: E(Y |Xc ) Differences: E(Y |Xc2 ) − E(Y |Xc1 ) Risk Ratios: E(Y |Xc2 )/E(Y |Xc1 ) or any other function of the above for some counterfactual Xc ’s. For our Voting example, that’s easy—just plug Xc into E(Y |Xc ) = Chris Adolph (UW) 1 1 + exp(−Xc β) Tufte without tears 14 / 106 Getting confidence intervals is harder, but there are several options: For maximum likelihood models, simulate the response conditional on the regressors See King, Tomz, and Wittenberg, 2000, American Journal of Political Science, and the Zelig package for R or Clarify for Stata. These simulations can easily be summarized as CIs: sort them and take percentiles For Bayesian models, usual model output is a set of posterior draws See Andrew Gelman and Jennifer Hill, 2006, Data Analysis Using Hierarchical/ Multilevel Models, Cambridge UP. Once we have the quantities of interest and confidence intervals, we’re ready to make some graphs. . . but how? Chris Adolph (UW) Tufte without tears 15 / 106 Here is the graph that King, Tomz, and Wittenberg created for this model How would we make this? Chris Adolph (UW) Tufte without tears 16 / 106 We could use the default graphics in Zelig or Clarify (limiting, not as nice as the above) Or we could do it by hand (hard) Chris Adolph (UW) Tufte without tears 17 / 106 Wanted: an easy-to-use R package that 1 takes as input the output of estimated statistical models 2 makes a variety of plots for model interpretation 3 plots “triples” (lower, estimate, upper) from estimated models well 4 lays out these plot in a tiled arrangement (small multiples) 5 takes care of axes, titles, and other fussy details Chris Adolph (UW) Tufte without tears 18 / 106 Wanted: an easy-to-use R package that 1 takes as input the output of estimated statistical models 2 makes a variety of plots for model interpretation 3 plots “triples” (lower, estimate, upper) from estimated models well 4 lays out these plot in a tiled arrangement (small multiples) 5 takes care of axes, titles, and other fussy details With considerable work, one could coerce R’s basic graphics to do this badly or get lattice to do this fairly well for a specific case But an easy-to-use, general solution is lacking Chris Adolph (UW) Tufte without tears 18 / 106 The tile package My answer is the tile package, written using R’s grid graphics Some basic tile graphic types: scatter lineplot ropeladder Scatterplots with fits, CIs, and extrapolation checking Line plots with fits, CIs, and extrapolation checking Dot plots with CIs and extrapolation checking Each can take as input draws from the posterior of a regression model A call to a tile function makes a multiplot layout: ideal for small multiples of model parameters Chris Adolph (UW) Tufte without tears 19 / 106 Column Title 1 Column Title 2 Column Title 3 Plot Title 1 Plot Title 2 Plot Title 3 Top Axis 2 1 5 10 20 35 1 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 1 0 5 10 20 35 X Axis 1 Under Title 1 Under Title 2 Plot Title 4 Top Axis 4 1 5 10 20 Plot Title 5 35 1 0.4 0.2 0 Under Title 4 Chris Adolph (UW) 0.4 0.2 0 1 5 10 20 35 X Axis 3 Under Title 3 Plot Title 6 35 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0 Right Axis 6 Row Title 2 Right Axis 4 0.6 0.6 Top Axis 6 1 5 10 20 1 0.8 1 0.8 Right Axis 2 Row Title 1 Y Axis 1 1 0.8 0.2 1 0 5 10 20 35 X Axis 5 Under Title 5 Tufte without tears Under Title 6 20 / 106 Plot simulations of QoI Generally, we want to plot triples: lower, estimate, upper We could do this for specific discrete scenarios, e.g. Pr(Voting) given five distinct sets of x’s Recommended plot: Dotplot with confidence interval lines Chris Adolph (UW) Tufte without tears 21 / 106 Plot simulations of QoI Generally, we want to plot triples: lower, estimate, upper We could do this for specific discrete scenarios, e.g. Pr(Voting) given five distinct sets of x’s Recommended plot: Dotplot with confidence interval lines Or for a continuous stream of scenarios, e.g., Hold all but Age constant, then calculate Pr(Voting) at every level of Age Recommended plot: Lineplot with shaded confidence intervals Chris Adolph (UW) Tufte without tears 21 / 106 This example is obviously superior to the table of logit coefficients But is there anything wrong or missing here? Chris Adolph (UW) Tufte without tears 22 / 106 1 College Probability of Voting 0.8 0.6 High School 0.4 Less than HS 0.2 Logit estimates: 95% confidence interval is shaded 0 20 30 40 50 60 70 Age of Respondent 80 90 18 year old college grads?! And what about high school dropouts? Chris Adolph (UW) Tufte without tears 23 / 106 1 College Probability of Voting 0.8 0.6 High School 0.4 Less than HS 0.2 Logit estimates: 95% confidence interval is shaded 0 20 30 40 50 60 70 Age of Respondent 80 90 tile helps us systematize plotting model results, and helps avoid unwanted extrapolation by limiting results to the convex hull Chris Adolph (UW) Tufte without tears 24 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. Chris Adolph (UW) Tufte without tears 25 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. I Could be a set of points, or text labels, or lines, or a polygon Chris Adolph (UW) Tufte without tears 25 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. I Could be a set of points, or text labels, or lines, or a polygon I Could be a set of points and symbols, colors, labels, fit line, CIs, and/or extrapolation limits Chris Adolph (UW) Tufte without tears 25 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. I Could be a set of points, or text labels, or lines, or a polygon I Could be a set of points and symbols, colors, labels, fit line, CIs, and/or extrapolation limits I Could be the data for a dotchart, with labels for each line Chris Adolph (UW) Tufte without tears 25 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. I Could be a set of points, or text labels, or lines, or a polygon I Could be a set of points and symbols, colors, labels, fit line, CIs, and/or extrapolation limits I Could be the data for a dotchart, with labels for each line I Could be the marginal data for a rug I All annotation must happen in this step Chris Adolph (UW) Tufte without tears 25 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. I Could be a set of points, or text labels, or lines, or a polygon I Could be a set of points and symbols, colors, labels, fit line, CIs, and/or extrapolation limits I Could be the data for a dotchart, with labels for each line I Could be the marginal data for a rug I All annotation must happen in this step I Basic traces: linesTile(), pointsile(), polygonTile(), polylinesTile(), and textTile() I Complex traces: lineplot(), scatter(), ropeladder(), and rugTile() Chris Adolph (UW) Tufte without tears 25 / 106 Trace functions in tile Primitive trace functions: linesTile Plot a set of connected line segments pointsTile Plot a set of points polygonTile Plot a shaded region polylinesTile Plot a set of unconnected line segments textTile Plot text labels Complex traces for model or data exploration: lineplot ropeladder rugTile scatter Chris Adolph (UW) Plot lines with confidence intervals, extrapolation warnings Plot dotplots with confidence intervals, extrapolation warnings, and shaded ranges Plot marginal data rugs to axes of plots Plot scatterplots with text and symbol markers, fit lines, and confidence intervals Tufte without tears 26 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. 2 Plot the data traces. Using the tile() function, simultaneously plot all traces to all plots. Chris Adolph (UW) Tufte without tears 27 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. 2 Plot the data traces. Using the tile() function, simultaneously plot all traces to all plots. I This is the step where the scaffolding gets made: axes and titles I Set up the rows and columns of plots I Titles of plots, axes, rows of plots, columns of plots, etc. I Set up axis limits, ticks, tick labels, logging of axes Chris Adolph (UW) Tufte without tears 27 / 106 Three steps to make tile plots 1 Create data traces. Each trace contains the data and graphical parameters needed to plot a single set of graphical elements to one or more plots. 2 Plot the data traces. Using the tile() function, simultaneously plot all traces to all plots. 3 Examine output and revise. Look at the graph made in step 2, and tweak the input parameters for steps 1 and 2 to make a better graph. Chris Adolph (UW) Tufte without tears 28 / 106 R Syntax for Lineplot of Voting Logit # Using simcf and tile to explore an estimated logistic regression # Voting example using 2000 NES data after King, Tomz, and Wittenberg # Chris Adolph # Load libraries library(RColorBrewer) library(MASS) library(simcf) library(tile) # Download simcf and tile packages # from faculty.washington.edu/cadolph # Load data (available from faculty.washington.edu/cadolph) file <- "nes00.csv" data <- read.csv(file, header=TRUE) Chris Adolph (UW) Tufte without tears 29 / 106 R Syntax for Lineplot of Voting Logit # Set up model formula and model specific data frame model <- vote00 ~ age + I(age^2) + hsdeg + coldeg mdata <- extractdata(model, data, na.rm=TRUE) # Run logit & extract results logit.result <- glm(model, family=binomial, data=mdata) pe <- logit.result$coefficients # point estimates vc <- vcov(logit.result) # var-cov matrix # Simulate parameter distributions sims <- 10000 simbetas <- mvrnorm(sims, pe, vc) Chris Adolph (UW) Tufte without tears 30 / 106 R Syntax for Lineplot of Voting Logit # Set up counterfactuals: all ages, each of three educations xhyp <- seq(18,97,1) nscen <- length(xhyp) nohsScen <- hsScen <- collScen <- cfMake(model, mdata, nscen) for (i in 1:nscen) { # No High school scenarios (loop over each age) nohsScen <- cfChange(nohsScen, "age", x = xhyp[i], scen = i) nohsScen <- cfChange(nohsScen, "hsdeg", x = 0, scen = i) nohsScen <- cfChange(nohsScen, "coldeg", x = 0, scen = i) # HS grad hsScen <hsScen <hsScen <- scenarios (loop over each age) cfChange(hsScen, "age", x = xhyp[i], scen = i) cfChange(hsScen, "hsdeg", x = 1, scen = i) cfChange(hsScen, "coldeg", x = 0, scen = i) # College grad scenarios (loop collScen <- cfChange(collScen, collScen <- cfChange(collScen, collScen <- cfChange(collScen, over each age) "age", x = xhyp[i], scen = i) "hsdeg", x = 1, scen = i) "coldeg", x = 1, scen = i) } Chris Adolph (UW) Tufte without tears 31 / 106 R Syntax for Lineplot of Voting Logit # Simulate expected probabilities for all scenarios nohsSims <- logitsimev(nohsScen, simbetas, ci=0.95) hsSims <- logitsimev(hsScen, simbetas, ci=0.95) collSims <- logitsimev(collScen, simbetas, ci=0.95) # Get 3 nice colors for traces col <- brewer.pal(3,"Dark2") # Set up lineplot traces of expected probabilities nohsTrace <- lineplot(x=xhyp, y=nohsSims$pe, lower=nohsSims$lower, upper=nohsSims$upper, col=col[1], extrapolate=list(data=mdata[,2:ncol(mdata)], cfact=nohsScen$x[,2:ncol(hsScen$x)], omit.extrapolated=TRUE), plot=1) Chris Adolph (UW) Tufte without tears 32 / 106 R Syntax for Lineplot of Voting Logit hsTrace <- lineplot(x=xhyp, y=hsSims$pe, lower=hsSims$lower, upper=hsSims$upper, col=col[2], extrapolate=list(data=mdata[,2:ncol(mdata)], cfact=hsScen$x[,2:ncol(hsScen$x)], omit.extrapolated=TRUE), plot=1) collTrace <- lineplot(x=xhyp, y=collSims$pe, lower=collSims$lower, upper=collSims$upper, col=col[3], extrapolate=list(data=mdata[,2:ncol(mdata)], cfact=collScen$x[,2:ncol(hsScen$x)], omit.extrapolated=TRUE), plot=1) Chris Adolph (UW) Tufte without tears 33 / 106 R Syntax for Lineplot of Voting Logit # Set up traces with labels and legend labelTrace <- textTile(labels=c("Less than HS", "High School", "College"), x=c( 55, 49, 30), y=c( 0.26, 0.56, 0.87), col=col, plot=1) legendTrace <- textTile(labels=c("Logit estimates:", "95% confidence", "interval is shaded"), x=c(82, 82, 82), y=c(0.2, 0.16, 0.12), plot=1) Chris Adolph (UW) Tufte without tears 34 / 106 R Syntax for Lineplot of Voting Logit # Plot traces using tile tile(nohsTrace, hsTrace, collTrace, labelTrace, legendTrace, width=list(null=5), limits=c(18,94,0,1), xaxis=list(at=c(20,30,40,50,60,70,80,90)), xaxistitle=list(labels="Age of Respondent"), yaxistitle=list(labels="Probability of Voting"), frame=TRUE ) Chris Adolph (UW) Tufte without tears 35 / 106 Chomp! Coefficient tables can hide the punchline Agresti offers the following example of a multinominal data analysis Alligators in a certain Florida lake were studied, and the following data collected: Principal Food 1 = Invertebrates, 2 = Fish, 3= “Other” (!!! Floridians?) Size of alligator Sex of alligator in meters male or female The question is how alligator size influences food choice. We fit the model in R using multinomial logit and get . . . Chris Adolph (UW) Tufte without tears 36 / 106 Chomp! Coefficient tables can hide the punchline Multinomial Logit of Alligator’s Primary Food Source Intercept Size Female ln(πInvertebrates /πFish ) ln(πOther /πFish ) 4.90 (1.71) −2.53 (0.85) −0.79 (0.71) −1.95 (1.53) 0.13 (0.52) 0.38 (0.91) Direct interpretation? Chris Adolph (UW) Tufte without tears 37 / 106 Chomp! Coefficient tables can hide the punchline Multinomial Logit of Alligator’s Primary Food Source Intercept Size Female ln(πInvertebrates /πFish ) ln(πOther /πFish ) 4.90 (1.71) −2.53 (0.85) −0.79 (0.71) −1.95 (1.53) 0.13 (0.52) 0.38 (0.91) Direct interpretation? We could do it, using odds ratios and a calculator. Invertebrates vs Fish: A 1 meter increase in length makes the odds that an alligator will eat invertebrates rather than fish exp(−2.53 − 0) = 0.08 times smaller. Chris Adolph (UW) Tufte without tears 37 / 106 Chomp! Coefficient tables can hide the punchline Multinomial Logit of Alligator’s Primary Food Source Intercept Size Female ln(πInvertebrates /πFish ) ln(πOther /πFish ) 4.90 (1.71) −2.53 (0.85) −0.79 (0.71) −1.95 (1.53) 0.13 (0.52) 0.38 (0.91) Invertebrates vs Other: A 1 meter increase in length makes the odds that an alligator will eat invertebrates rather than “other” food exp(−2.53 − 0.13) = 0.07 times smaller. Chris Adolph (UW) Tufte without tears 38 / 106 Chomp! Coefficient tables can hide the punchline Multinomial Logit of Alligator’s Primary Food Source Intercept Size Female ln(πInvertebrates /πFish ) ln(πOther /πFish ) 4.90 (1.71) −2.53 (0.85) −0.79 (0.71) −1.95 (1.53) 0.13 (0.52) 0.38 (0.91) Invertebrates vs Other: A 1 meter increase in length makes the odds that an alligator will eat invertebrates rather than “other” food exp(−2.53 − 0.13) = 0.07 times smaller. No reader is going to do this Chris Adolph (UW) Tufte without tears 38 / 106 Chomp! Coefficient tables can hide the punchline Multinomial Logit of Alligator’s Primary Food Source Intercept Size Female ln(πInvertebrates /πFish ) ln(πOther /πFish ) 4.90 (1.71) −2.53 (0.85) −0.79 (0.71) −1.95 (1.53) 0.13 (0.52) 0.38 (0.91) Fish vs Other: A 1 meter increase in length makes the odds that an alligator will eat fish rather than “other” food exp(0 − 0.13) = 0.87 times smaller. There has to be a better way Chris Adolph (UW) Tufte without tears 39 / 106 Food choice by alligator size and sex 1 1 Pr(Food preference) Invertebrates 0.8 0.8 Fish 0.6 Fish 0.6 0.4 0.4 Other 0.2 0 Invertebrates 1 2 Other 0.2 3 4 Size of alligator 0 Male 1 2 3 Size of alligator 4 Female A simple lineplot made with tile shows the whole covariate space Much more dramatic than the table Chris Adolph (UW) Tufte without tears 40 / 106 Food choice by alligator size and sex Pr(Food preference) 1 Invertebrates 1 Fish 0.8 0.8 0.6 0.6 0.4 0.4 Other 0.2 0.2 0 0 1 2 3 Invertebrates 4 Fish Other 1 2 3 Size of alligator Size of alligator Male Female 4 We can also add easy to interpret measures of uncertainty Above are standard error regions Chris Adolph (UW) Tufte without tears 41 / 106 Food choice by alligator size and sex Pr(Food preference) 1 Invertebrates 1 Fish 0.8 0.8 0.6 0.6 0.4 0.4 Other 0.2 0.2 0 0 1 2 3 Invertebrates 4 Fish Other 1 2 3 Size of alligator Size of alligator Male Female 4 Easy to highlight where the model is interpolating into the observed data . . . and where it’s extrapolating into Hollywood territory Chris Adolph (UW) Tufte without tears 42 / 106 When simulation is the only option: Chinese leadership Shih, Adolph, and Liu investigate the advancement of elite Chinese leaders in the Reform Period (1982–2002) Explain (partially observed) ranks of the top 300 to 500 Chinese Communist Party leaders as a function of: Demographics Education Performance Faction age, sex, ethnicity level of degree provincial growth, revenue birth, school, career, and family ties to top leaders Bayesian model of partially observed ranks of CCP officials Model parameters difficult to interpret: on a latent scale and individual effects are conditioned on all other ranked members Only solution: Simulate ranks of hypothetical officials as if placed in the observed hierarchy Chris Adolph (UW) Tufte without tears 43 / 106 0 < High School Female High School Han Non−Princeling Average Member Graduate School College Jiang Zemin Faction Male Relative GDP Growth +1 sd Princeling Party Exp +1 sd Relative Fiscal Growth +1 sd Minority Hu Jintao Faction Deng Faction Age +1 sd Expected percentile 40 60 80 100 Black circles show expected ranks for otherwise average Chinese officials with the characteristic listed at left ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 356 Chris Adolph (UW) 20 285 214 142 Expected rank Tufte without tears 71 0 44 / 106 0 < High School Female High School Han Non−Princeling Average Member Graduate School College Jiang Zemin Faction Male Relative GDP Growth +1 sd Princeling Party Exp +1 sd Relative Fiscal Growth +1 sd Minority Hu Jintao Faction Deng Faction Age +1 sd Expected percentile 40 60 80 100 ● ● ● ● ● Thick black horizontal lines are 1 std error bars, and thin lines are 95% CIs ● ● ● ● ● ● ● ● ● ● ● ● ● 356 Chris Adolph (UW) 20 285 214 142 Expected rank Tufte without tears 71 0 45 / 106 0 < High School Female High School Han Non−Princeling Average Member Graduate School College Jiang Zemin Faction Male Relative GDP Growth +1 sd Princeling Party Exp +1 sd Relative Fiscal Growth +1 sd Minority Hu Jintao Faction Deng Faction Age +1 sd Expected percentile 40 60 80 100 ● ● ● Blue triangles are officials with random effects at ±1 sd; how much unmeasured factors matter ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 356 Chris Adolph (UW) 20 285 214 142 Expected rank Tufte without tears 71 0 46 / 106 0 < High School Female High School Han Non−Princeling Average Member Graduate School College Jiang Zemin Faction Male Relative GDP Growth +1 sd Princeling Party Exp +1 sd Relative Fiscal Growth +1 sd Minority Hu Jintao Faction Deng Faction Age +1 sd Expected percentile 40 60 80 100 ● ● ● ● ● Helps to sort rows of the plot from smallest to largest effect ● ● ● ● ● ● ● ● ● ● ● ● ● 356 Chris Adolph (UW) 20 285 214 142 Expected rank Tufte without tears 71 0 47 / 106 Change in Rank Percentile 40 GDP Growth +1 sd 0 −20 College Degree '82 Change in Rank Percentile Fiscal Revenue +1 sd 20 '87 '92 '97 '02 '82 '87 Ethnic Minority 40 '92 '97 '02 '82 '87 '92 '97 '02 Female 20 0 −20 Graduate Degree '82 '87 '92 '97 '02 '82 '87 '92 '97 '02 '82 '87 '92 '97 '02 Shih, Adolph, and Liu re-estimate the model for each year, leading to a large number of results A complex lineplot helps organize them and facilitate comparisons Chris Adolph (UW) Tufte without tears 48 / 106 Change in Rank Percentile 40 GDP Growth +1 sd 20 0 −20 College Degree '82 Change in Rank Percentile Fiscal Revenue +1 sd '87 '92 '97 '02 '82 '87 Ethnic Minority 40 '92 '97 '02 '82 '97 '02 '82 '87 '92 '97 '02 Female 20 0 −20 Graduate Degree '82 '87 '92 '97 '02 '82 '87 '92 '87 '92 '97 '02 Note that these results are now first differences: the expected percentile change in rank for an otherwise average official who gains the characteristic noted Chris Adolph (UW) Tufte without tears 49 / 106 Change in Rank Percentile 40 Mao Faction 20 0 −20 Long Marcher 2nd Field Army Vet Change in Rank Percentile '82 40 '87 '82 Zhao Ziyang Faction '87 '82 Jiang Zemin Faction '87 Deng Faction '82 '87 '92 '97 Hu Jintao Faction Hu Yaobang Fact'n '02 '82 '87 Princeling 20 0 −20 '87 '92 '92 '97 '02 '97 '02 '82 '87 '92 '97 '02 Over time, officials’ economic performance never matters, but factions often do Runs counter to the conventional wisdom that meritocratic selection of officials lies behind Chinese economic success Chris Adolph (UW) Tufte without tears 50 / 106 Robustness Checks So far, we’ve presenting conditional expectations & differences from regressions But are we confident that these were the “right” estimates? The language of inference usually assumes we correctly specified our model correctly measured our variables chose the right probability model don’t have influential outliers, etc. Chris Adolph (UW) Tufte without tears 51 / 106 Robustness Checks So far, we’ve presenting conditional expectations & differences from regressions But are we confident that these were the “right” estimates? The language of inference usually assumes we correctly specified our model correctly measured our variables chose the right probability model don’t have influential outliers, etc. We’re never completely sure these assumptions hold. Most people present one model, and argue it was the best choice Sometimes, a few alternatives are displayed Chris Adolph (UW) Tufte without tears 51 / 106 The race of the variables Model 1 Model 2 Model 3 Model 4 My variable of interest, X1 X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) A control I "need" X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) A control I "need" X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) A candidate control A candidate control X.XX (X.XX) X.XX (X.XX) X.XX (X.XX) Alternate measure of X1 Chris Adolph (UW) Model 5 X.XX (X.XX) X.XX (X.XX) Tufte without tears 52 / 106 Robustness Checks Problems with the approach above? 1 Lots of space to show a few permutations of the model Most space wasted or devoted to ancillary info Chris Adolph (UW) Tufte without tears 53 / 106 Robustness Checks Problems with the approach above? 1 Lots of space to show a few permutations of the model Most space wasted or devoted to ancillary info 2 What if we’re really interested in E(Y |X ), not β̂? E.g., because of nonlinearities, interactions, scale differences, etc. Chris Adolph (UW) Tufte without tears 53 / 106 Robustness Checks Problems with the approach above? 1 Lots of space to show a few permutations of the model Most space wasted or devoted to ancillary info 2 What if we’re really interested in E(Y |X ), not β̂? E.g., because of nonlinearities, interactions, scale differences, etc. 3 The selection of permutations is ad hoc. Chris Adolph (UW) Tufte without tears 53 / 106 Robustness Checks Problems with the approach above? 1 Lots of space to show a few permutations of the model Most space wasted or devoted to ancillary info 2 What if we’re really interested in E(Y |X ), not β̂? E.g., because of nonlinearities, interactions, scale differences, etc. 3 The selection of permutations is ad hoc. We’ll try to fix 1 & 2. Objection 3 is harder, but worth thinking about. Chris Adolph (UW) Tufte without tears 53 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y 2 Choose: I a measure of X , denoted X , Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y 2 Choose: I a measure of X , denoted X , I a measure of Y, denoted Y , Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y 2 Choose: I a measure of X , denoted X , I a measure of Y, denoted Y , I a set of confounders, Z , Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y 2 Choose: I a measure of X , denoted X , I a measure of Y, denoted Y , I a set of confounders, Z , I a functional form, g(·) Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y 2 Choose: 3 I a measure of X , denoted X , I a measure of Y, denoted Y , I a set of confounders, Z , I a functional form, g(·) I a probability model of Y , f (·) Estimate the probability model Y ∼ f (µ, α), µ = g(vec(X , Z ), β). Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y 2 Choose: I a measure of X , denoted X , I a measure of Y, denoted Y , I a set of confounders, Z , I a functional form, g(·) I a probability model of Y , f (·) 3 Estimate the probability model Y ∼ f (µ, α), µ = g(vec(X , Z ), β). 4 Simulate the quantity of interest, e.g., E(Y |X ) or E(Y2 − Y1 |X2 , X1 ), to obtain a point estimate and confidence interval. Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y 2 Choose: I a measure of X , denoted X , I a measure of Y, denoted Y , I a set of confounders, Z , I a functional form, g(·) I a probability model of Y , f (·) 3 Estimate the probability model Y ∼ f (µ, α), µ = g(vec(X , Z ), β). 4 Simulate the quantity of interest, e.g., E(Y |X ) or E(Y2 − Y1 |X2 , X1 ), to obtain a point estimate and confidence interval. 5 Repeat 2–4, changing at each iteration one of the choices in step 2. Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: An algorithm 1 Identify a relation of interest between a concept X and a concept Y 2 Choose: I a measure of X , denoted X , I a measure of Y, denoted Y , I a set of confounders, Z , I a functional form, g(·) I a probability model of Y , f (·) 3 Estimate the probability model Y ∼ f (µ, α), µ = g(vec(X , Z ), β). 4 Simulate the quantity of interest, e.g., E(Y |X ) or E(Y2 − Y1 |X2 , X1 ), to obtain a point estimate and confidence interval. 5 Repeat 2–4, changing at each iteration one of the choices in step 2. 6 Compile the results in a variant of the dot plot called a ropeladder. Chris Adolph (UW) Tufte without tears 54 / 106 Robustness Checks: A central banks example In The Dilemma of Discretion, I argue central bankers’ career backgrounds explain their monetary policy choices Central bankers with financial sector backgrounds choose more conservative policies, leading to lower inflation but potentially higher unemployment I argue more conservative governments should prefer to appoint more conservative central bankers, e.g., those with financial sector backgrounds Chris Adolph (UW) Tufte without tears 55 / 106 Robustness Checks: A central banks example I test this claim using data on central bankers setting monetary policy over 20 countries and 30 years Conservatism of central bankers is measured by percentage share in “conservative” careers, and summarized by an index CBCC Partisanship of government is measured by PCoG: higher values = more conservative Partisan“Center of Gravity” I estimate a zero-inflated compositional regression on career shares and find the expected relationship: conservative goverments pick conservative central bankers Chris Adolph (UW) Tufte without tears 56 / 106 Robustness Checks: A central banks example Conservative goverments pick conservative central bankers Is this finding robust to specification assumptions? In my case, the statistical model is so demanding it’s hard to include many regressors at once. So try one at a time, and show a long “ropeladder” plot. . . Chris Adolph (UW) Tufte without tears 57 / 106 Robustness Ropeladder: Partisan central banker appointment Estimated increase in Central Bank Conservatism (CBCC) resulting from . . . Control added Shifting control from Shifting PCoG from low (µ-1.5 sd) From left gov (µ-1.5 sd) to high (µ+1.5) to right gov (µ+1.5 sd) [None] Office appointed to CBI (3 index avg.) CBI (Cukierman) Lagged inflation Lagged unemployment Trade Openness Endebtedness Financial Sector Employment Financial Sector Score Time Trend Central Bank staff size -0.4 -0.2 0.0 0.2 0.4 0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 Chris8.4: AdolphPartisanship (UW) Tufte without tears Figure of central banker appointment: Robustness, part 2. Each row presents a58 / 106 Anatomy of a ropeladder plot I call this a ropeladder plot. The column of dots shows the relationship between Y and a specific X under different model assumptions Each entry corresponds to a different assumption about the specification, or the measures, or the estimation method, etc. Chris Adolph (UW) Tufte without tears 59 / 106 Anatomy of a ropeladder plot I call this a ropeladder plot. The column of dots shows the relationship between Y and a specific X under different model assumptions Each entry corresponds to a different assumption about the specification, or the measures, or the estimation method, etc. If all the dots line up, with narrow, similar CIs, we say the finding is robust, and reflects the data under a range of reasonable assumptions If the ropeladder is “blowing in the wind”, we may be skeptical of the finding. It depends on model assumptions that may be controversial Chris Adolph (UW) Tufte without tears 59 / 106 Anatomy of a ropeladder plot I call this a ropeladder plot. The column of dots shows the relationship between Y and a specific X under different model assumptions Each entry corresponds to a different assumption about the specification, or the measures, or the estimation method, etc. If all the dots line up, with narrow, similar CIs, we say the finding is robust, and reflects the data under a range of reasonable assumptions If the ropeladder is “blowing in the wind”, we may be skeptical of the finding. It depends on model assumptions that may be controversial The shaded gray box shows the full range of the point estimates for the QoI. Narrow is better. Chris Adolph (UW) Tufte without tears 59 / 106 Why ropeladders? 1 Anticipate objections on model assumptions, and have concrete answers. Avoid: “I ran it that other way, and it came out the ‘same’.” Instead: “I ran it that other way, and look —it made no substantive or statistical difference worth speaking of.” Or: “. . . it makes this much difference.” Chris Adolph (UW) Tufte without tears 60 / 106 Why ropeladders? 1 Anticipate objections on model assumptions, and have concrete answers. Avoid: “I ran it that other way, and it came out the ‘same’.” Instead: “I ran it that other way, and look —it made no substantive or statistical difference worth speaking of.” Or: “. . . it makes this much difference.” 2 Investigate robustness more thoroughly. Traditional tabular presentation would have run to 7 pages, making comparison hard and discouraging a thorough search Chris Adolph (UW) Tufte without tears 60 / 106 Why ropeladders? 1 Anticipate objections on model assumptions, and have concrete answers. Avoid: “I ran it that other way, and it came out the ‘same’.” Instead: “I ran it that other way, and look —it made no substantive or statistical difference worth speaking of.” Or: “. . . it makes this much difference.” 2 Investigate robustness more thoroughly. Traditional tabular presentation would have run to 7 pages, making comparison hard and discouraging a thorough search 3 Find patterns of model sensitivity. Two seemingly unrelated changes in specification had the same effect. (Unemployment and Financial Sector Size) Turned out to be a missing third covariate. (Time trend) Chris Adolph (UW) Tufte without tears 60 / 106 81 Robustness for several QoIs at once Change in inflation, five years after +1 s.d. in . . . FinExp Specification FMExp CBExp GovExp CBCC Baseline Robust Estimation Add π world Use Cukierman CBI Omit Imports/GDP Add Exchange Regime Add % Left-Appointees Add Partisan CoG of Aptees Add % with Econ PhDs −3 −1 1 −3 −1 1 −1 1 3 −1 1 3 −3 −1 1 Figure 3.7: The career-inflation link under alternative specifications. Each plot shows the five-year first difference in inflation resulting from a one standard deviation increase in a career variable (FinExp, Each ropeladder, or column, GovExp, etc.), given the specification noted at left. See text for a description of the baseline model and Circles anddifferent squares indicate point estimates of the first difference, and horizontal lines 90 shows alternatives. the effect of a variable on the response That percent confidence intervals. The results marked with a square are the same as those used in Figure 3.6. The shaded areas highlight the range of point estimates across all alternatives. Effects of controls not (of thoseacross considered,shows only π worldthe had results a significantfrom effect on is,shown reading a inflation). single model Reading down shows the results for a single question across different models A similar pattern of robustness emerges if we keep the baseline model in place and instead vary the construction of the key explanatory variables. Most central banks have Chris Adolph (UW) Tufte without tears 61 / 106 Chinese Officials Redux Earlier, I showed the relationship between many different covariates and the expected political rank of Chinese officials I also showed how these relationships changed over time, making for a complex plot But our findings were controversial: countered the widely accepted belief that Chinese officials are rewarded for economic performance Critics asked for lots of alternative specifications to probe our results I can use tile to show how exactly what difference these robustness checks made using overlapping lineplots Chris Adolph (UW) Tufte without tears 62 / 106 Change in Rank Percentile 40 GDP Growth +1 sd 0 −20 College Degree '82 Change in Rank Percentile Fiscal Revenue +1 sd 20 '87 '92 '97 '02 '82 '87 Ethnic Minority 40 '92 '97 '02 '82 '87 '92 '97 '02 Female 20 0 −20 Graduate Degree '82 '87 '92 '97 '02 Original Model '82 '87 '92 '97 '02 '82 '87 '92 '97 '02 Factional Tie Requires Job Overlap Some critics worried that our measures of faction were too sensitive, so we considered a more specific alternative This didn’t salvage the conventional wisdom on growth. . . Chris Adolph (UW) Tufte without tears 63 / 106 Change in Rank Percentile 40 Mao Faction 20 0 −20 Long Marcher 2nd Field Army Vet Change in Rank Percentile '82 40 '87 '82 Zhao Ziyang Faction '87 '82 Jiang Zemin Faction Deng Faction '87 '82 '87 '92 '97 Hu Jintao Faction Hu Yaobang Fact'n '02 '82 '87 Princeling 20 0 −20 '87 '92 '92 Original Model '97 '02 '97 '02 '82 '87 '92 '97 '02 Factional Tie Requires Job Overlap But did (unsuprisingly) strengthen our factional results (Specific measures pick up the strongest ties) Chris Adolph (UW) Tufte without tears 64 / 106 Change in Rank Percentile 40 Mao Faction 20 0 −20 Long Marcher 2nd Field Army Vet Change in Rank Percentile '82 40 '87 '82 Zhao Ziyang Faction '87 '82 Jiang Zemin Faction '87 Deng Faction '82 '87 '92 '97 Hu Jintao Faction Hu Yaobang Fact'n '02 '82 '87 Princeling 20 0 −20 '87 '92 '92 '97 '02 '97 Original Model '02 '82 '87 '92 '97 '02 Surprise Performance, ARMA(p,q) Other critics worried about endogeneity or selection effects flowing from political power to economic performance We used measures of unexpected growth to zero in on an official’s own performance in office—which still nets zero political benefit Chris Adolph (UW) Tufte without tears 65 / 106 Change in Rank Percentile 40 GDP Growth +1 sd 0 −20 College Degree '82 Change in Rank Percentile Fiscal Revenue +1 sd 20 '87 '92 '97 '02 '82 '87 Ethnic Minority 40 '92 '97 '02 '82 '97 '02 '82 '87 '92 '97 '02 Female 20 0 −20 Graduate Degree '82 '87 '92 '97 '02 '82 '87 '92 Original Model '87 '92 '97 '02 Surprise Performance, ARMA(p,q) But is there a more efficient way to show that our results stay essentially the same? Chris Adolph (UW) Tufte without tears 66 / 106 Change in Rank Percentile GDP Growth +1 sd 40 0 −20 College Degree '82 Change in Rank Percentile Fiscal Revenue +1 sd 20 '87 '92 '97 '02 '82 '87 Ethnic Minority 40 '92 '97 '02 '82 '97 '02 '82 '87 '92 '97 '02 Female 20 0 −20 Graduate Degree '82 '87 Original Model '92 '97 '02 '82 '87 '92 Ignore exact ACC ranks '87 '92 '97 '02 Job Overlap Only 7 models controlling Surprise Performance, Five Factor Perform. & Schools In our printed article, we show only this plot, which overlaps the full array of robustness checks Chris Adolph (UW) Tufte without tears 67 / 106 Change in Rank Percentile 40 Mao Faction 20 0 −20 Long Marcher 2nd Field Army Vet Change in Rank Percentile '82 40 '87 '82 Zhao Ziyang Faction '87 '82 Jiang Zemin Faction '87 Deng Faction '82 '87 '92 '97 Hu Jintao Faction Hu Yaobang Fact'n '02 '82 '87 Princeling 20 0 −20 '87 Original Model '92 '92 '97 '02 '97 '02 Ignore exact ACC ranks '82 '87 '92 '97 '02 Job Overlap Only 7 models controlling Surprise Performance, Five Factor Perform. & Schools Conveys hundreds of separate findings in a compact, readable form No knowledge of Bayesian methods or partial rank coefficients required! Chris Adolph (UW) Tufte without tears 68 / 106 Bonus Example: Allocation of Authority for Health Policy Adolph, Greer, and Fonseca consider the problem of explaining whether local, regional, or national European government have power over specific health policy areas and instruments Areas: Pharamceuticals, Secondary/Tertiary, Primary Care, Public Health Instruments: Frameworks, Finance, Implementation, Provision Each combination for each country is a case Chris Adolph (UW) Tufte without tears 69 / 106 Bonus Example: Allocation of Authority for Health Policy Adolph, Greer, and Fonseca consider the problem of explaining whether local, regional, or national European government have power over specific health policy areas and instruments Areas: Pharamceuticals, Secondary/Tertiary, Primary Care, Public Health Instruments: Frameworks, Finance, Implementation, Provision Each combination for each country is a case Fiscal federalism suggests lower levels for information-intensive policies and higher levels for policies with spillovers or public goods Also control for country characteristics and country random effects With 3 nominal outcomes for each case, need a multilevel multinomial logit Chris Adolph (UW) Tufte without tears 69 / 106 Bonus Example: Allocation of Authority for Health Policy Covariates: Policy area Policy instrument Regions old or new Country size Number of regions Mountains Ethnic heterogeneity Nominal Nominal Binary Continuous Continuous Countinuous Continuous Tricky part to the model: some cases have structural zeros for regions (when they don’t exist!) Chris Adolph (UW) Tufte without tears 70 / 106 How to set up counterfactuals? We could set all but one covariate to the mean, then predict the probability of each level of authority given varied levels of the remaining covariate We should do this separately for countries with and without regional governments Let’s fix everthing but policy instrument to the mean values, then simulate the probability of authority at each level for each instrument We show the results using a “nested” dot plot, made using ropeladder() in the tile package Chris Adolph (UW) Tufte without tears 71 / 106 Two−tier countries 0 Probability of allocation 0.2 0.4 0.6 0.8 Three−tier countries 1 Probability of allocation 0.2 0.4 0.6 0.8 0 state state region Provision ● local local state state region Implementation ● local local state state region Finance ● local local state state Framework region ● local 0 Chris Adolph (UW) 1 local 0.2 0.4 0.6 0.8 Probability of allocation 1 Tufte without tears 0 0.2 0.4 0.6 0.8 Probability of allocation 1 72 / 106 Special plots for compositional data Probabilities have a special property: they sum to one Variables that sum to a constraint are compositional We can plot a two-part composition on a line, and a three-part composition on a triangular plot This makes it easier to show more complex counterfactuals, such as every combination of policy area and instrument But we also need to work harder to explain these plots Chris Adolph (UW) Tufte without tears 73 / 106 Lo ca l Probability of allocation of authority by policy type Pro vis ion Local ● • Provision • 0.6 0.4 0.2 0.8 Public Health Primary Care Secondary/Tertiary Pharmacueticals Finance 0.6 • 0.4 me nta t Fin a ion nc e ● ● Implementation Framework 0.2 0.4 0.6 0.8 State • 0.8 Im ple • 0.2 (Ph ceu arma tica ls) ew ork Fra m Sta te ● Region Above holds country characteristics at their means Chris Adolph (UW) Tufte without tears 74 / 106 Lo ca l Probability of allocation of authority by country type 0.2 Local 0.8 Mountainous Ethnically Diverse ● Post-1973 Regions 0.4 Many Small Regions Pre-1973 Regions Typical Three Tier • Few Large Regions Ethnically Diverse Mountainous Country Large Country Sta te Large Country 0.2 Ethnically Homogeneous 0.8 ● Flat Country Typical Two-Tiered • Flat Country 0.6 Ethnically Homogeneous 0.6 0.4 Small Country Small Country 0.2 0.4 0.6 0.8 State Region Above holds policy area and instrument “at their means” Chris Adolph (UW) Tufte without tears 75 / 106 Lo ca l Residual country effects Local ● 0.2 Finland Lithuania ● 0.4 0.8 0.6 • Czech Republic Latvia ● Greece ● Portugal ● Ireland Slovenia ● Cyprus ● Luxembourg ● Netherlands ● ● Slovakia ● • France ● Bulgaria ● Romania ● •• Switzerland • Italy Austria Belgium Germany •• • Denmark ● United Kingdom Sweden 0.2 0.4 0.6 State 0.8 Sta •• Spain Poland te Malta 0.2 • 0.4 0.8 ● 0.6 Hungary Estonia Region Looking at the country random effects might suggest omitted variables Chris Adolph (UW) Tufte without tears 76 / 106 Lo ca l Residual country effects Local ● 0.2 Finland Lithuania ● 0.4 0.8 0.6 • Czech Republic Latvia ● Greece ● Portugal ● Ireland Slovenia ● Cyprus ● Luxembourg ● Netherlands ● ● Slovakia ● • France ● Bulgaria ● Romania ● •• Switzerland • Italy Austria Belgium Germany •• • Denmark ● United Kingdom Sweden 0.2 0.4 0.6 State 0.8 Sta •• Spain Poland te Malta 0.2 • 0.4 0.8 ● 0.6 Hungary Estonia Region On a previous iteration, mountainous countries clustered as high Pr(Regions) Chris Adolph (UW) Tufte without tears 77 / 106 Lessons for practice of data analysis Simulation + Graphics can summarize complex models for a broad audience You might even find something you missed as an analyst And even for fancy or complex models, we can and should show uncertainty Payoff to programming: this is hard the first few times, but gets easier Code is re-usable, and encourages more ambitious modeling Chris Adolph (UW) Tufte without tears 78 / 106 Teaching with tile tile helps clarify data and models in research Also helps in teaching statistical models I incorporate this software throughout our graduate statistics sequence Greatly aids intuitive understanding of models Find out much more, and download the software, from: faculty.washington.edu/cadolph Chris Adolph (UW) Tufte without tears 79 / 106 Building a scatterplot: Supplementary exploratory example In my graphics class, I have students build a scatterplot “from scratch” This helps us see the many choices to make, and implications for: 1 perception of the data 2 exploration of relationships 3 assessment of fit A good warm up for tile before the main event (application to models) See how tile helps follow Tufte recommendations Chris Adolph (UW) Tufte without tears 80 / 106 Building a scatterplot: Redistribution example Data on political party systems and redistributive effort from various industrial countries Source of data & basic plot: Torben Iversen & David Soskice, 2002, “Why do some democracies redistribute more than others?” Harvard University. Chris Adolph (UW) Tufte without tears 81 / 106 Building a scatterplot: Redistribution example Concepts for this example (electoral systems and the welfare state): Effective number of parties: Chris Adolph (UW) Tufte without tears 82 / 106 Building a scatterplot: Redistribution example Concepts for this example (electoral systems and the welfare state): Effective number of parties: # of parties varies across countries electoral rules determine # I Winner take all (US) →∼ 2 parties. I Proportional representation → more parties To see this, need to discount trivial parties. Chris Adolph (UW) Tufte without tears 82 / 106 Building a scatterplot: Redistribution example Concepts for this example (electoral systems and the welfare state): Effective number of parties: # of parties varies across countries electoral rules determine # I Winner take all (US) →∼ 2 parties. I Proportional representation → more parties To see this, need to discount trivial parties. Poverty reduction: Chris Adolph (UW) Tufte without tears 82 / 106 Building a scatterplot: Redistribution example Concepts for this example (electoral systems and the welfare state): Effective number of parties: # of parties varies across countries electoral rules determine # I Winner take all (US) →∼ 2 parties. I Proportional representation → more parties To see this, need to discount trivial parties. Poverty reduction: Percent lifted out of poverty by taxes and transfers. Poverty = an income below 50% of mean income. Chris Adolph (UW) Tufte without tears 82 / 106 100 90 80 povredct 70 60 50 40 30 20 10 0 0 2 4 6 8 enpart 10 12 14 Initial plotting area is often oddly shaped (I’ve exaggerated) Plotting area hiding relationship here. Sometimes can even exclude data! Filled circles: okay for a little data; open is better when data overlap Chris Adolph (UW) Tufte without tears 83 / 106 70 povred 60 50 40 30 20 2 3 4 5 6 7 enpart Sensible, data based plot limits Appears to be a curvilinear relationship. Can bring that out with. . . Chris Adolph (UW) Tufte without tears 84 / 106 70 povredct 60 50 40 30 20 0.3 0.4 0.5 0.6 logpart 0.7 0.8 Log scaling. But what logarthmic base? And why print the exponents? Chris Adolph (UW) Tufte without tears 85 / 106 70 povredct 60 50 40 30 20 2 3 4 5 6 7 8 9 10 enpart Combine a log scale with linear labels. Now everyone can read. . . Next: Axis labels people can understand Chris Adolph (UW) Tufte without tears 86 / 106 Percent lifted out of poverty by taxes and transfers 70 60 50 40 30 20 2 3 4 5 6 7 Effective number of political parties (log10 scale) 8 9 10 So what’s going on the data? What are those outliers? Chris Adolph (UW) Tufte without tears 87 / 106 Percent lifted out of poverty by taxes and transfers Belgium 70 Norway Netherlands Sweden 60 50 France United Kingdom Germany Italy Australia 40 30 Denmark Finland Canada 20 Switzerland United States 2 3 4 5 6 7 8 Effective number of political parties (log10 scale) 9 10 With little data & big outliers, show the name of each case Now we can try to figure out what makes the US and Switzerland so different Chris Adolph (UW) Tufte without tears 88 / 106 Percent lifted out of poverty by taxes and transfers 80 Belgium Denmark 70 Netherlands Norway Finland Sweden 60 France 50 United Kingdom 40 30 Germany Italy Australia Electoral system Majoritarian Proportional Unanimity Canada 20 10 Switzerland United States 2 3 4 5 6 Effective number of political parties (log10 scale) Chris Adolph (UW) Tufte without tears 7 8 9 10 89 / 106 % lifted from poverty by taxes & transfers 80 Majoritarian Proportional Unanimity Belgium Denmark Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 Plot redone using scatter (tile package in R) Chris Adolph (UW) Tufte without tears 90 / 106 % lifted from poverty by taxes & transfers 80 Majoritarian Proportional Unanimity Belgium Denmark Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 Scatterplots relate two distributions. Why not make those marginal distributions explicit? Chris Adolph (UW) Tufte without tears 91 / 106 % lifted from poverty by taxes & transfers 80 Majoritarian Proportional Unanimity Belgium Denmark Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 Rugs accomplish this by replacing the axis lines with the plots We could choose any plotting style: from the histogram-like dots. . . Chris Adolph (UW) Tufte without tears 92 / 106 % lifted from poverty by taxes & transfers 80 Majoritarian Proportional Unanimity Belgium Denmark Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 . . . to a strip of jittered data. . . Chris Adolph (UW) Tufte without tears 93 / 106 % lifted from poverty by taxes & transfers 80 Majoritarian Proportional Unanimity Belgium Denmark Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 . . . to a set of very thin lines marking each observation Because we have so few cases, thin lines work best for this example Chris Adolph (UW) Tufte without tears 94 / 106 % lifted from poverty by taxes & transfers 80 Majoritarian Proportional Unanimity Belgium Denmark Norway Netherlands Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 Let’s add a parametric model of the data: a least squares fit line tile can do this for us Chris Adolph (UW) Tufte without tears 95 / 106 % lifted from poverty by taxes & transfers 80 Belgium Majoritarian Proportional Unanimity Denmark Norway Netherlands Finland 60 Sweden France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 But we don’t have to be parametric A local smoother, like loess, often helps show non-linear relationships Chris Adolph (UW) Tufte without tears 96 / 106 % lifted from poverty by taxes & transfers 80 Belgium Majoritarian Proportional Unanimity Denmark Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 M-estimators weight observations by an influence function to minimize the influence of outliers Chris Adolph (UW) Tufte without tears 97 / 106 % lifted from poverty by taxes & transfers 80 Belgium Majoritarian Proportional Unanimity Denmark Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 Even with an M-estimator, every outlier has some influence Thus any one distant outlier can bias the result Chris Adolph (UW) Tufte without tears 98 / 106 % lifted from poverty by taxes & transfers 80 Majoritarian Proportional Unanimity Belgium Denmark Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 A robust and resistant MM-estimator, shown above, largely avoids this problem Only a (non-outlying) fraction of the data influence this fit. rlm(method="MM") Chris Adolph (UW) Tufte without tears 99 / 106 % lifted from poverty by taxes & transfers 80 Majoritarian Proportional Unanimity Denmark Belgium Netherlands Norway Finland Sweden 60 France United Kingdom 40 Germany Italy Australia Canada 20 Switzerland United States 0 2 3 4 Effective number of parties 5 6 7 In our final plot, we add 95 percent confidence intervals for the MM-estimator A measure of uncertainty is essential to reader confidence in the result Chris Adolph (UW) Tufte without tears 100 / 106 Syntax for redistribution scatter library(tile) # Load data data <- read.csv("iver.csv", header=TRUE) attach(data) Chris Adolph (UW) Tufte without tears 101 / 106 Syntax for redistribution scatter # First, collect all the data inputs into a series of "traces" # The actual scattered points trace1 <- scatter(x = enp, # X coordinate of the data y = povred, # Y coordinate of the data labels = cty, # Labels for each point # Plot symbol for each point pch = recode(system,"1=17;2=15;3=16"), # Color for each point col = recode(system,"1=’blue’;2=’darkgreen’;3=’red’"), # Offset text labels labelyoffset = -0.03, # on npc scale # Fontsize fontsize = 9, Chris Adolph (UW) Tufte without tears 102 / 106 Syntax for redistribution scatter # Marker size size = 1, # could be vector for bubble plot # Add a robust fit line and CI fit = list(method = "mmest", ci = 0.95), # Which plot(s) to plot to plot = 1 ) # The rugs with marginal distributions rugX1 <- rugTile(x=enp, type="lines", plot = 1) rugY1 <- rugTile(y=povred, type="lines", plot = 1) Chris Adolph (UW) Tufte without tears 103 / 106 Syntax for redistribution scatter # A legend legendSymbols1 <- pointsTile(x= c(1.8, y= c(78, pch=c(17, col=c("blue", fontsize = 9, size=1, plot=1 ) 1.8, 1.8), 74, 70), 15, 16), "darkgreen", "red"), legendLabels1 <- textTile(labels=c("Majoritarian", "Proportional", "Unanimity"), x= c(2.05, 2.05, 2.05), y= c(78, 74, 70), pch=c(17, 15, 16), col=c("blue", "darkgreen", "red"), fontsize = 9, plot=1 ) Chris Adolph (UW) Tufte without tears 104 / 106 Syntax for redistribution scatter # Now, send that trace to be plotted with tile tile(trace1, # Could list as many rugX1, # traces here as we want rugY1, # in any order legendSymbols1, legendLabels1, # Some generic options for tile RxC = c(1,1), #frame = TRUE, #output = list(file="iverson1", width=5, type="pdf"), height = list(plot="golden"), # Limits of plotting region limits=c(1.6, 7.5, 0, 82), Chris Adolph (UW) Tufte without tears 105 / 106 Syntax for redistribution scatter # x-axis controls xaxis=list(log = TRUE, at = c(2,3,4,5,6,7) ), # x-axis title controls xaxistitle=list(labels=c("Effective number of parties")), # y-axis title controls yaxistitle=list(labels=c("% lifted from poverty by taxes & transfers" # Plot titles plottitle=list(labels=("Party Systems and Redistribution")) ) Chris Adolph (UW) Tufte without tears 106 / 106

Download PDF

advertisement