Financial Aid and College Persistence Do Student Loans Help or Hurt? A Selection Control Study Version: May 2015* Draft: Do not cite without permission Serge Herzog, PhD Director, Institutional Analysis Consultant, CRDA StatLab University of Nevada, Reno Reno, NV 89557 [email protected] *Prepared for presentation at annual forum of the Association for Institutional Research (AIR), Denver, CO, May 25-29, 2015 FINANCIAL AID AND COLLEGE PERSISTENCE ABSTRACT Using data from two freshmen cohorts at a public research university (N = 3,730), this study examines the relationship between loan aid and second-year enrollment persistence. Applying a counterfactual analytical framework that relies on propensity score (PS) weighing and matching to address selection bias associated with treatment status, the study estimates that loan aid exerts a significant negative effect on persistence for students from low-income background (i.e., Pell eligible) and those taking up high amounts of loans in order to meet total cost of attendance. However, no significant incremental effect associated with unsubsidized loan aid, net of subsidized loan aid, could be detected. The estimated effect of loan aid on persistence controls for first-year academic experience and takes into account 25 factors related to loan selection and persistence in order to match students with loan aid to a counterfactual case in covariate adjusted regression. Comparison with results from single-stage standard regression suggests selection bias masks the negative effect of loans detected with matched-sample estimation. Validity of covariates determining the loan selection process and criteria for acceptable balance in the matched data are discussed, and implications for future research are addressed. Keywords: student persistence, financial aid, loans, low income, causal inference estimation, propensity score 1|Page FINANCIAL AID AND COLLEGE PERSISTENCE Introduction A recent analysis of nationally representative enrollment data of college students in the United States fails to identify entry barriers for prepared low-income students and instead stresses the lack of enrollment persistence following initial college matriculation among such students. Lack of continued college participation of low-income students is attributed primarily to inadequate academic preparation and student self-reported reasons other than financial aid (Adelman, 2007). Pascarella and Terenzini (2005) arrived at a similar finding after reviewing the cumulative research since the 1980s, which shows that on average students with financial aid are no more likely to persist and graduate than those without aid. However, they did conclude that the effect varies by aid type, with merit-based scholarship aid having the greatest positive impact, while need-based grants having a modest positive effect on persistence and graduation. In contrast, the impact of work-study is more ambiguous, and the effect of student loans is “mixed” at best (Pascarella and Terenzini, 2005, p. 411). More recent reviews on the impact of aid largely corroborate previous findings and highlight that incongruities in results stem from differences in methodology, data sources, scope of the analysis, type of aid covered, and how aid is measured (Welbeck et al., 2014; Chen and Zerquera, 2011; Hossler et al., 2009). In fact, the influence of aid on student enrollment persistence is mediated by a complex web of interrelated factors: including the timing, type, and amount of aid and how they correlate with persistence in the presence of other student attributes (Pascarella and Terenzini, 2005). In spite of voluminous research on student success, enrollment retention models yield limited insight to evaluate the impact of financial aid. Descriptive studies fail to control for the multiple influences that govern student behavior. Equally problematic are studies that employ inferential statistics without subject randomization or mechanisms to reduce selection bias associated with students that receive aid. The importance of moving beyond correlational analysis and estimating causal relationships in education research has been stressed for some time (Murnane & Willett, 2010; Morgan & Winship, 2007; U.S. Department of Education, 2003; Angrist, 2003). Since randomized control trials (RCTs) are costly, raise ethical issues, and are fraught with operational difficulties, they are typically not available to the analyst. Instead, one may rely on techniques that attempt to mimic random assignment in order to reduce selection bias with observational data. One of these techniques is centered on the propensity score (PS), a measure of the chance a person will select an experience or ‘treatment’, in this case whether or not a student selects loan aid to finance college. Thus, this study uses the PS approach to examine the influence of student loans on enrollment persistence. Loan Selection The focus of this study is limited to loan aid, since the selection process associated with loan aid is governed far more by discretionary decision compared to scholarship aid or need-based grant aid. In a review of the determinants of loan selection Cho, Xu, and Kiss (2015) highlight the importance of family and parental contributions to cover student educational expenses. Brown et al. (2012) found that the degree of unmet expected family support is associated with demand for financial aid, while Elliott and Nam (2013) reported that students with college savings accounts are less likely to seek financial aid. Data from the 1999-2000 and 2003-2004 waves of the 2|Page FINANCIAL AID AND COLLEGE PERSISTENCE National Postsecondary Student Aid Study (NPSAS) confirm that students with the highest unmet need are most likely to take up loans (Cadena and Keys, 2013). Student proclivity to rely on loans is also associated with past behavior, consideration of parental views, gender, and support for future goals in life (Chudry, Foxall, and Pallister, 2011). In contrast, access to financial aid information does not appear to affect the likelihood of loan take-up based on findings from a randomized field experiment (Booij, Leuven, and Oosterbeek, 2012). However, assistance with filling out and processing the Free Application for Federal Student Aid (FAFSA), which covers subsidized and unsubsidized student loans, does increase the rate of aid application (Bettinger, E. P. et al., 2009). Thus, cognitive constraints may influence whether or not a student applies for loan aid. Loans are typically a ‘last resort’ source of aid for students after merit-based scholarships and need-based grants are exhausted (Ziskin et al., 2014). Similarly, students who qualify for subsidized loans may select to rely on unsubsidized loans if the former are considered insufficient. In contrast to scholarship and grant aid that are available (and thus selected) based on academic merit and income background, selection of loan aid is more directly tied to perceived need, which is a function of the actual remaining need after taking into account all other aid received (Cadena and Keys, 2013; Brown et al., 2012). Unlike scholarship and grant aid, which typically offer little discretion for the student to select and accept or not—amounts are usually set and eligibility clear cut—students face much greater discretion whether or not to take out a loan and for what amount. To receive a loan, students are required to sign off prior to receipt, unlike grant and scholarship aid which is pushed to students after completion of the FAFSA. Thus, the presence of selection bias is decidedly more plausible with loans compared to scholarships and grants, a point highlighted in Hossler et al. (2009, pp. 401, 413) in their review of the research: “Because of the complex interplay of privilege, opportunity, and conditioned action in society, the question of endogeneity bias becomes an inherent controversy in the study of financial aid and persistence in college. [Since] loans have to be repaid, the possibility of selfselection bias influencing any analysis of the impact of student loans is great.” To address this challenge in the research, this study estimates the influence of loans on firstyear student persistence with a series of regression models that weigh and match students based on their PS. The next section briefly explains the PS methodology and analytical framework, followed by a description of the study and statistical results, and a discussion of the findings and its implication for future research. The Propensity Score and Counterfactual Inference Model Like regression discontinuity and instrumental variable (IV) methods (see Bowman and Herzog, 2014; Murnane and Willett, 2010), the PS method offers a way to address selection bias and to establish control groups as benchmarks to gauge the effect of student loans on enrollment persistence. Use of the PS in observational studies was popularized by Rosenbaum and Rubin (1983, 1984) and is closely linked to the counterfactual analytical framework that emerged in statistics and econometrics (Holland, 1986; Rosenbaum, 2002; Rubin, 2006; Heckman, 2000). The counterfactual model seeks an answer to the “what if” question. In the context of this study, what is the potential enrollment outcome for a student who took up loan aid to pay for college had that student not chosen to rely on loan aid? In other words, a student has potentially two outcomes, but only one can be observed. In order to measure this unobserved outcome, one has to account for the likelihood (or predisposition) of a student selecting loan aid when measuring 3|Page FINANCIAL AID AND COLLEGE PERSISTENCE the relationship between loan aid and enrollment persistence. To render the correlation insignificant between a student’s chance to select loan aid (i.e. the ‘treatment’) and the outcome of interest (i.e., enrollment persistence), one has to have a comparator case, a student with equal or similar chance to select the treatment but did not, and who can be equally observed like the ‘treated’ student with respect to the outcome. The ability to create a counterfactual case—the comparable untreated student that substitutes for the unobserved outcome of the treated student—is at the core of causal inference estimation after controlling for treatment selection (i.e., selection bias). Rosenbaum and Rubin (1983) showed that matching a treated case with an untreated case on a single probability (propensity) score that adequately captures the linear combination of factors that predict treatment selection offers a way to reduce selection bias. Formally, the propensity score pˆ(Xi) = Pr(Di = 1|Xi) measures the probability (Pr) of selecting into treatment D conditional on observable predictors (covariates X) for each case (i). Assuming the treatment can be measured dichotomously (D = 1 or 0), the average treatment effect (ATE) conditional on X is formalized as E(∆i|Xi = x) = E[(Yi| Di = 1)-(Yi| Di = 0)]| Xi = x, where E(∆i|Xi = x) is the expected difference in the outcome Yi between the treated (Di = 1) and untreated (Di = 0), controlling for observable factors (Xi = x) that predict treatment (D) selection. Since financial aid evaluation in education is focused typically on intervention or support for those students who need the aid, the counterfactual centers on the student with a loan—i.e. had that student not selected loan aid—and thus the average treatment effect on the treated (ATT) is defined as E(∆i| Di = 1) = E[(Yi| Di = 1)-(Yi| Di = 0)]| Di = 1, or the difference in outcome under treatment and non-treatment for those students who actually did select loan aid. Since only the treated outcome (Yi| Di = 1) is observed for the treated, the untreated outcome (Yi| Di = 0) is derived from untreated cases (i.e., those actually without loan aid) with similar or identical chance of treatment selection as the treated based on observable characteristics (Xs). The effectiveness of the PS to yield unbiased effect estimates hinges on the selection of observables—known as the conditional independence assumption, strongly ignorable treatment assignment, or unconfoundedness—to render treatment selection independent of the potential outcomes for the treated and untreated. That is to say, likelihood of treatment selection does not affect the outcome of interest. Accordingly, the chance of selecting loan aid does not influence the student’s enrollment persistence. This independence cannot be established empirically, but must be anchored in covariate selection that is guided by explicit theory and an understanding of the factors that prompt students to seek loan aid to finance their education. Thus, only observable factors (predictors) that temporally precede the selection of loan aid can be used in the PS estimation. The PS must also show sufficient overlap in distribution between students with loans and those without loans in order to produce a comparator case; the individual covariates used in estimating the PS should be closely balanced between the two groups to rule 4|Page FINANCIAL AID AND COLLEGE PERSISTENCE out bias associated with observable characteristics; and enrollment persistence should not be influenced by the loan status of other students (known as the Stable Unit Treatment Value Assumption; see Rubin, 1980). While the first two conditions can be ascertained empirically ex post facto, the last condition is best addressed with a study design that minimizes, or at least does not promote, interaction between students with loans and those without loans. After calculating the PS with a logit or probit regression model for a binary treatment (i.e., selecting loan aid or not), the effect of loan aid on enrollment persistence can be estimated by matching students with loans to those without loans on their PS, by stratifying students into subclasses (ranges) of the PS, by inverse probability of treatment weighting (IPTW) using the PS, or by including the PS and treatment indicator in covariate adjusted regression (Austin, 2011; Murnane & Willett, 2010). Unlike standard regression, which uses parametric specification to account for covariate effects, application of the PS relaxes the linearity assumption through preprocessing of the data for either matching on a single metric to establish a control group (untreated) in order to directly measure the difference in outcome to the treated group (nonparametric), or through pre-selecting or weighting cases in semi-parametric covariate regression. This study uses the PS in both IPTW and covariate adjusted regression and contrasts results with standard regression in order to identify possible selection bias. Estimating the Effect of Student Loans on Enrollment Persistence Data source and Measures. Data are drawn from a moderately selective university (average SAT of 1080, ACT of 23) and capture the profile and academic experience of new first-year undergraduate students that enrolled in the fall semester of 2011 and 2012. The effective sample of 3,730 freshmen excludes students with less than 12 attempted semester credits, students with complete credit withdrawal after registration, students without entry survey data, those who did not complete the Free Application for Federal Student Aid (FAFSA), and statistical outlier cases identified in a regression model (using Cook’s and Mahalanobis distance) that included all the covariates used in the subsequent analysis. The effective sample amounts to 70 percent of all freshmen in the two cohort years. Selection of the covariates to estimate the PS is governed by several criteria. First, they are all pretreatment baseline characteristics that exist prior to student selection of loan aid and thus are not conditioned by treatment assignment. Second, all of the covariates may be related to both selection of loan aid and enrollment persistence. While the most important covariates are those related to selection of loan aid, both theoretical and empirical research show that PS estimation with a large set of covariates (at least ten or more), including covariates indirectly related to the outcome, yields better PS balance between treatment and control groups (Stuart and Rubin, 2008). Third, most of the covariates selected here have been used extensively in prior research to understand student academic success and enrollment persistence (Pascarella & Terenzini, 2005; Astin, 1993). Twenty-five covariates were used to estimate the PS with logistic regression using loan status as the dependent variable. Remaining need ($ amount), estimated family/student contribution (EFC, $ amount), and whether or not a student had scholarship aid or a Pell Grant are included in the PS estimation. These variables in combination with on-campus living status, out-of-state residency status, participation in a living-and-learning community, and student work plans (taken from entry survey) emerged as the strongest predictors of student loan selection (Wald > 30). Not surprisingly, most of these variables determine a student’s cost of attendance and are 5|Page FINANCIAL AID AND COLLEGE PERSISTENCE sourced directly from the student’s FAFSA information (Cadena and Keys, 2013). The inclusion of a continuous metric for remaining need—the amount outstanding after accounting for EFC and all types of aid received—allows testing of the impact of loan aid across different need levels. Similarly, incorporation of EFC offers an opportunity to examine the influence of loan aid as a function of the ability to pay for college, rather than merely income level, which fails to reflect the estimated expense burden facing a student. The remaining covariates capture a student’s socio-demographic profile (being age 19 or older, male, non-Asian ethnic minority, father has 4-year college degree, mother has 4-year college degree, in-state residency outside of local area), academic preparation (high school GPA, ACT/SAT test scores, high school class rank percentile, advanced standing at entry), and academic motivation and educational goal (delayed college entry by 6 months or more, ACT/SAT test date, institution is first choice, undeclared major, math-intensive major, plans to attend graduate school). Except for the precollege preparation index (composed of high school GPA and test scores), the high school percentile rank, and the test date, all covariates are dummy coded (no = 0, yes = 1). The test date is a continuous metric that measures the months elapsed between the first time the student took the ACT/SAT test and the start of college. Advanced standing indicates if a student earned college credit in high school, while math-intensive major identifies students who chose an academic program that requires advanced math in the first year of study. Test date and delayed entry are proxy indicators for student commitment and motivation, following a previous study that used admission date for that purpose (Pike, Hansen, & Lin, 2011). Identifying students with a math-intensive major takes into account that math courses dominate the list of courses undergraduate students are most likely to fail or drop without completion (Adelman, 2004). All data originated with the institution’s matriculation system, except for several student selfreported data elements that were recorded as part of the mandatory start-of-semester orientation survey. They include measures of parent level of education, intent to attend graduate school, plans to work full-time or plans to not work while in college (plans to work half-time being the reference category), and having selected the institution as first choice. The PS for loan status (0 = no, 1 = yes) is estimated first for all types of loans—subsidized and unsubsidized—and subsequently for only unsubsidized loans. This offers the opportunity to gauge the incremental effect of unsubsidized loans after factoring in subsidized loan aid. This distinction is important, since students that are eligible for subsidized loans are less likely to take out unsubsidized loans compared to those ineligible for subsidized loans (Ziskin et al., 2014). PS methods. The effect of student loans on fall-to-fall enrollment persistence (coded 0 = no; 1 = yes) is estimated with a series of two-stage covariate adjusted regression models that separate students by EFC level and remaining need level, all using data that match and weigh students on the PS associated with loan status. Results of each PS method are estimates of the ATT—i.e., the effect on students with loans had they not taken out loan aid—and are compared to standard regression with the same set of covariates. Data were matched using the nearest neighbor and full matching functions in the MatchIt R program (Ho et al., 2011), and genetic matching using the Matching program in R by Sekhon (2011). These algorithms yielded the best covariate balance among several others in a PS evaluation study (Herzog, 2014) and thus are relied upon here. With the focus set on the ATT, the common support area is defined by the PS range of students with loan aid that are matched with students without loan aid based on their PS. The nearest neighbor with replacement (NNR:1) match chooses only the control case with the closest 6|Page FINANCIAL AID AND COLLEGE PERSISTENCE PS to a treated case, and thus the same control case may be matched to multiple treated cases to minimize the average difference (distance) in the PS. This type of matching is suitable for PS distributions with limited overlap between treated and control cases, and where the latter are fewer in number. An extension of the more commonly used matching by PS stratification is the method of full matching (Full SubClass), algorithmically automated to minimize the PS difference by partitioning the data into an optimal number of subclasses that contain the best bi-directional ratio of treated and control cases. The last matching method relies on a genetic search algorithm designed to maximize the PS balance across all covariates. Balance is determined with paired ttests for dichotomous variables and the Kolmogorov-Smirnov test for continuous variables at every iteration. This lets the evolutionary algorithm automatically adjust the number of cases used in the random trail to meet operator restrictions in order to optimize the solution. Assuming good covariate and PS balance between students with loan aid and those without loan aid, the full sub-classification method has the greatest potential to minimize both variance and bias in the data, for it retains more control cases than one-on-one matching (Hansen, 2004). Therefore, genetic matching is used as a backup when sub-classification matching fails to produce good balance. Results from PS-matched data are compared with standard regression and with IPTW. With IPTW, treated cases receive a weight of 1, while all control cases are weighted on (𝑃i(𝑋))/(1−𝑃𝑖(𝑋)), where 𝑃i(𝑋) is the PS of the control case, thereby magnifying untreated cases that resemble more closely the treated cases (for more on IPTW, see Reynolds and DesJardins, 2010). Given the significant influence of the first-year academic experience on student persistence in documented research (Pascarella & Terenzini, 2005; Astin, 1993), the impact of loan aid is estimated with and without a set of first-year experience variables. Accordingly, the covariate adjusted model that regresses the outcome (persistence) on the PS and loan status also controls for whether or not a student took math or English, or enrolled in an online distance education course; used any of the on-campus tutoring centers; visited the on-campus student diversity center; worked on campus (all coded as dummy variables); and academic success. The latter is measured with a 100-point momentum index composed equally of first-semester and first-year cumulative grades (GPA) and number of course credits completed. The use of academic momentum indices to gauge student success goes back to the work of Adelman (2006), and academic momentum based on grades and credits of first-year students is highly predictive of degree completion beyond pre-college preparation and socio-demographic background (Attewell, Heil, & Reisel, 2012). These variables likely mitigate bias in the estimated outcome, as found in other PS-based studies that included post-treatment measures (Austin, 2011; Titus, 2007). This method assumes that the covariance between the PS and the outcome is correctly modeled, and that variables exhibit tolerable collinearity. The variance inflation factor (VIF) for the combined set of preand post-treatment variables remained well below 2.5, the recommended upper limit (Allison, 2012). Descriptive Data and Balance Verification. Table 1 shows that students with an EFC level between $5,200 and $20,000 are most likely to rely on loans, both subsidized and unsubsidized. These students are in the bottom half of those that do not qualify for the federal Pell Grant. The 7|Page FINANCIAL AID AND COLLEGE PERSISTENCE upper half, those with an EFC above $20,000, are less likely to rely on loans, especially subsidized loans for which few would qualify. Students with a low EFC level—i.e., those eligible for the Pell Grant—are slightly less likely to rely on loans compared to mid-level EFC students, but more likely to have loans compared to high-level EFC students. Looking at the academic momentum of these students, Table 2 suggests that students on unsubsidized loans barely differ in their academic success from those with only subsidized loans, regardless of the level of EFC. Since unsubsidized loans incur an immediate payback obligation in contrast to subsidized loans, one may expect students carrying unsubsidized loans to be at an academic disadvantage. Tables 3 and 4 indicate that students with no remaining need as well as students with high remaining need are mostly from high EFC background. Expectedly, students with high remaining need are less likely to have a low EFC level, which allows students to qualify for more need-based aid such as the Pell Grant. Still, over a fifth of students with high remaining need (i.e., over $ 10,000) are Pell-eligible. These students make up 27 percent of all Pell-eligible students (Table 5). In contrast, students with a high EFC level (i.e., over $ 20,000) are twice as likely to have high remaining need compared to Pell-eligible students (57 percent, Table 6). While these distributions convey some sense of the financial burden faced by freshmen, they do not reveal how reliance on loans may impact enrollment persistence given a student’s level of ability to pay for college. Before proceeding with an interpretation of the results from the statistical regression, one must verify that the resampled data from PS matching resulted in suitable control groups to establish a counterfactual analytical framework. Figure 1 depicts the PS distribution for the treated (loan aid) and control (no loan aid) groups before and after matching (using full subclassification). The histograms before matching (left side) are distinctly different in shape, showing a sharp decline in the number of students with no loans (raw control) that exhibit a high propensity for loan selection; conversely, students with loans (raw treated) are mostly clustered at the high end of the propensity scale, as one would expect. In contrast, the histograms after matching (right side) look very similar, confirming that no-loan students in the control group largely resemble students with loan aid across the PS distribution. Figure 2 shows the standardized difference in mean for each covariate used in the PS estimation before (left) and after (right) matching. With the exception of two covariates (dark lines), the standardized difference for all covariates dropped after matching (right), leaving all covariates well within +/0.25, a threshold range recommended for reliable regression adjustment (Stuart, 2010; Rubin, 2001). Figure 3 illustrates a great improvement in the PS distribution between students with unsubsidized loans and those without such loans, rendering both groups highly comparable based on the covariates included. Similar improvements in the PS and covariate balance occurred after applying nearest neighbor matching (Figures 4 and 5). Table 7 summarizes the results from the PS and covariate balance checks. Both overall balance as well as balance for covariates that are strongly related to the loan selection process (EFC, remaining need, aid received, out of state tuition, and living on campus) are well within the recommended threshold range. Results. Descriptive data in Table 8 indicate that students with loans are less likely to persist than students without loans. The average seven percentage point difference grows to 10 percentage points when the comparison is limited to students with high remaining need, suggesting that high remaining need in conjunction with loan aid may put those students at an 8|Page FINANCIAL AID AND COLLEGE PERSISTENCE added disadvantage. Without considering other factors, however, one may scarcely conclude that loans are a key determinant in student persistence, much less a cause that explains the lower rate for students with loans. The influence of loan aid in the presence of all the factors used to estimate the PS are listed in Table 9 under the ‘Null Model’ for all students and broken out by EFC and remaining need level. Estimation of the influence of loan aid after controlling for first-year experience are listed under the ‘Post-treatment Model’. The estimated impact of unsubsidized loan aid, net of subsidized loan aid, is furnished in Table 10. Standard errors for nearest neighbor matched samples (NNR:1) are based on Lechner’s (1999) variance approximation: ∑ 𝑖{𝐷 = 0}(𝑊𝑖 )2 1 𝑉𝑎𝑟(𝑌(1)|𝐷 = 1) + ∗ 𝑉𝑎𝑟(𝑌(0)|𝐷 = 0) 𝑁1 𝑁1 where N1 is the number of matched treated cases and Wi is the number of repeat uses of a control case when matching with replacement in order to account for potential bias in bootstrapped standard errors introduced with multiple use of the same case (Caliendo and Kopeinig, 2008). Results from the logit coefficients show that, on average, loan aid exerts a negative influence on persistence both before and after accounting for first-year experience (Table 9). However, that effect reaches statistical significance only with PS-weighted (IPTW) and matched sample data (NNR:1). Standard regression fails to detect a significant effect after accounting for firstyear experience. Separating the analysis by EFC and remaining need level, the only significant effects net of first-year experience emerge from data using full sub-classification matching. Accordingly, students with a low EFC level (i.e., Pell eligible) and students with no remaining need are estimated to be negatively affected by loan aid. In either case, standard regression using unmatched data fails to show a significant effect. Fewer significant effects are associated with unsubsidized loans, which seem to matter only when ignoring what students experience during the first year (Table 10). Students with high remaining need and those with a high EFC appear to be negatively impacted by unsubsidized loans, but only prior to factoring in their first-year experience. Using the significant coefficients from matched data based on full sub-classification, Table 11 translates the estimated effect on enrollment persistence as the percentage change associated with loan aid and unsubsidized loan aid vis-à-vis the baseline reference category, students without loan aid or unsubsidized loan aid, applying Cruce’s (2009) corrected Delta-p statistic. A comparison of results between standard regression and covariate adjusted regression using PS matching suggests the former would fail to detect the negative effect of loan aid on persistence for students with low EFC and those with no remaining need. For students with high need, the effect size of unsubsidized loan aid prior to factoring in first-year experience is estimated to be lower with matched data compared to standard regression. Data for the PS-matched samples that show significant loan effects are well balanced (i.e., within the threshold range of +/- 0.25 standardized mean difference) between students with loan aid and those without loan aid across all key covariates that strongly predict loan selection, as listed in Table 12. 9|Page FINANCIAL AID AND COLLEGE PERSISTENCE Discussion of Findings Results from the series of regression models that separate students by EFC and remaining need levels produce several key findings. Loans exert a negative influence on enrollment persistence of Pell-eligible students, namely those with an EFC no larger than $5,200, as determined by the FAFSA data. Loans also seem to negatively affect persistence of students with no remaining financial need after factoring in their EFC and all aid received, inclusive of loans. In contrast, there is no significant incremental effect associated with unsubsidized loans after factoring in whether a student took out subsidized loans. These estimates take into account all the factors used in the calculation of a student’s propensity to select loan aid as well as their first-year college experience, including their GPA and number of completed course credits. Moreover, the results control for a range of socio-demographic, pre-college, academic motivation, and campus integration factors deemed important in college persistence research (Pascarella & Terenzini, 2005; Astin, 1993). Gauging the effect of loans both before and after controlling for the first-year experience reveals that the impact of loan aid is significantly diminished by the college experience for students with high remaining need (Table 9), even if they carry unsubsidized loans (Table 10). The same finding applies to high-EFC students with unsubsidized loans. Controlling for receipt of subsidized loans, unsubsidized loans fail to exert a negative influence on students with low EFC (Pell eligible) and those with no remaining need, regardless of a student’s first-year experience (Table 10). These differences in results suggest that average estimated effects mask both the magnitude and nature of the influence of loan aid on students from different income background and with varied levels of financial need. This supports the conclusion by Welbeck et al. (2014, pp. 16, 18) in their review of the research on aid that level of remaining need and student ability to afford cost of attendance are key factors in gauging the influence of aid on student success. In their review of the research since 1991, Hossler et al. (2009, p. 394) identified 32 “high quality” studies on the impact of financial aid, but few (if any) of these examined aid taking into account both EFC and remaining need. Similarly, only a few of the covered studies offer evidence of how campus social integration, student motivation, and academic success interact with financial aid. Lack of statistical control over these factors also limits the finding in a more recent study linking receipt of need-based aid (including subsidized loans) to higher student dropout (Gross, Hossler, Ziskin, and Berry, 2015). In addition to the finding that the effect of loans varies with ability to pay (EFC) and remaining need, the results here add to the already substantial evidence in the cumulative research that academic performance is the key predictor of enrollment persistence for most students (Pascarella & Terenzini, 2005). Moreover, the findings here corroborate those from a previous study on students at the same institution (Herzog, 2008), showing that academic success in the first year exerts a greater effect on persistence of low-income students compared to highincome students net of all other factors (including financial aid, socio-demographics, motivation, and educational goal). As Table 13 shows, first-year academic momentum is a far more significant covariate for Pell-eligible students compared to students with a high EFC, a difference consistent across all tested regression models. The analytical framework in this study also addresses the challenge of selection bias associated with observational data and tests the counterfactual hypothesis, what would be the rate of persistence of students with loan aid had they not selected loans to finance college? As listed in Table 9, there is a significant effect associated with loans net of first-year experience for Pell10 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE eligible students with low EFC and students whose total aid package covers the cost of college. Had these students not relied on loan aid—most of it subsidized—their estimated persistence rate would have been 4 percentage points and 7 percentage points higher, respectively, as shown in Table 11. These negative effects would not have been detected with single-stage standard regression, nor would it yield a result that gauges the effect size associated with the counterfactual for students that actually relied on loan aid. Being able to answer the ‘what if’ question in the counterfactual framework yields an estimate of what would have happened to students with loan aid had they not chosen that type of financing option. Moreover, results show that, net of subsidized loans and first-year experience, unsubsidized loans exert no significant influence on enrollment persistence, regardless of the student’s level of EFC or remaining need. Therefore, the incremental effect associated with unsubsidized loans, beyond that which may occur with subsidized loan aid, has a negligible impact on student persistence. Whether the difference to standard regression can be attributed to reduced selection bias with the PS-matched data depends on how well the observable characteristics (covariates) included in the PS estimator capture the process of loan selection by students. Recent research confirms that student loan selection is largely governed by both perceived and actual credit constraints, which derive from the combination of income background, cost of attendance, and willingness (or aversion) to take on debt (Ziskin et al., 2014; Cadena and Keys, 2013; Avery and Turner, 2012). Ziskin et al. (2014) document that loan aid is indeed a ‘last resort’ financing option that students typically rely on after maximizing all other aid options. The sequence of offered, accepted, and self-selected aid is well captured through the FAFSA data. Results from the PS estimation show that the FAFSA-derived metrics for EFC and remaining need in combination with indicators for variation in cost of attendance (i.e., residency status, on-campus living cost) together emerged as the most significant predictors of loan aid (Wald > 30) . The balance of these covariates and the PS in the matched group data is well within the threshold range for reliable regression adjustment (Stuart, 2010; see Table 12). Thus, the selection of covariates for PS estimation and the degree of balance in the matched data together render the conditional independence assumption (i.e., a strongly ignorable treatment assignment) plausible, if not highly likely, assuming the absence of significant omitted variables that are related to loan selection and persistence, and that are uncorrelated with those included here. Under that assumption, balance in the chosen covariates to estimate the PS are sufficient to render the likelihood of loan selection uncorrelated with enrollment persistence for students with no loan aid. Therefore, one may assume that results from the corresponding single-stage standard regression with unmatched data likely mask the significant negative effect associated with loan aid for Pell-eligible students and students with no remaining need detected by the two-stage PS-adjusted regression. Also, these estimated effects draw on counterfactuals that are matched with the entire sample of students with loans. None of the PS matching methods forced deletion of treated cases (i.e., students with actual loan aid) from the analysis due to lack of common support. As a result, the findings are more likely to hold in general for firstyear students with loans. Do Student Loans Help or Hurt? The answer to that question is it depends. For the majority of students in the two selected cohorts (fall 2011 and fall 2012), loan aid—either in subsidized or unsubsidized form—appears to exert no significant influence on enrollment persistence. This finding holds even after a 11 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE student’s academic performance in terms of grades and course credits completed is factored in. However, when testing the counterfactual hypothesis with matched data that compare students with no loan aid and highly similar disposition for selecting such aid to students that actually carry loan aid, the findings indicate that Pell-eligible students and students with no remaining need (i.e., whose aid package covers the cost of college) would persist at a higher rate without loan aid than with loan aid. Hossler et al. (2009), in their review of 32 highly relevant, high-quality studies produced since 1991, concluded that loans are more likely to have a negative effect on persistence if the metric is based on a dichotomized indicator for loan aid. Examining studies that used national databases (e.g., NPSAS), Hossler et al. (2009) report no effect associated with loan aid if measured on a continuous scale (i.e., amount received) once cost of attendance and living are included in the estimation. However, the studies from which these findings are drawn did not attempt to control for selection bias associated with receipt of loan aid. The one study covered in the review by Hossler et al. that did address selection bias (with an instrumental variable probit model) found a negative effect of loan aid on a student’s chance of graduation (Alon, 2005). The expansion of state-based and other large-scale grant programs to promote college enrollment of low-income high school graduates has prompted a growing number of evaluation studies that incorporate mechanisms to address selection bias, but none of these studies isolate the effect of loan aid (Angrist, Autor, Hudson, and Pallais, 2014; Welbeck et al., 2014). The finding that the negative impact of loan aid is limited to low-income students and students who receive enough aid to end up with no remaining need offers evidence of the importance to apply an analytical framework in the study of aid that differentiates by ability to pay, as reflected in the EFC, and by the amount of unmet need net of all aid received. The evidence here partly corroborates results from a previous study on students at the same institution that shows a negative influence of loan aid on first-year student persistence (Herzog, 2005). However, that study did not gauge the effect of loans separately by EFC level, nor did it explore the interaction effect between loan status and a student’s ability to pay (as reflected in the EFC amount), nor did it address selection bias in the statistical estimator. Though a direct comparison between results in that study and this study is of limited value, given the difference in covariate controls, it is reasonable to assume that an analytical approach focused on selection bias and testing of the counterfactual hypothesis likely yields a more accurate estimate of the causal effect of loan aid. Like the negative effect of loan aid on graduation that Alon (2005) found after separating the effect of aid eligibility from the impact of aid received, the negative effect of loan aid detected here emerged only after addressing selection bias with PS-matched data. Accordingly, had Pelleligible students with loan aid not selected such aid, their persistence would have been 4 percentage points higher; the persistence rate of those with no remaining need would have been 7 percentage points higher (Table 11). Given the proportion of Pell-eligible students in the cohorts examined, the overall freshmen persistence rate at the institution would increase by one percentage point had the Pell-eligible not relied on loans; a similar increase in overall freshmen persistence would result if students with no remaining need would have refrained from loans. Such estimates assume that the financial need covered by loans could be addressed with other sources of aid (or cost discounts) that do not exert a negative effect on persistence. Johnson (2013) found that for academically average students augmenting tuition subsidies (e.g. with grants or discounts) would mitigate the dropout risk and increase degree completion. Both Gross et al. (2015) and DesJardins, Ahlburg, and McCall (2002) concluded that greater reliance on 12 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE loans as part of the total aid received elevated the dropout risk for students. Consistent with the finding in this study, Kim (2007), using national data from the Beginning Postsecondary Student (BPS) survey, produced evidence that loan aid during the first-year lowered degree completion odds for low-income students. In contrast, Marx and Turner (2015) observed mixed effects on degree attainment after replacing loans with Pell Grant aid. Since the results here are limited to first-year students, the observed negative effect of loan aid may not endure as students in the examined cohorts academically progress. Those advancing are on average academically more successful and may feel more comfortable with rising loan debt. As Table 14 shows, loans constitute about the same proportion of total aid received for students with no remaining need as it does for those with remaining need of up to $10,000. However, the former receive on average loans that are 60 percent larger than the latter group, or more than double the size of loans taken up by students with high remaining need (i.e. over $10,000). The higher loan amounts for students with no remaining need is likely associated with the higher cost of on-campus living—on average they are 35 percent more likely to live on campus—and echoes the finding in Cadena and Keys (2013) that on-campus students are more likely to take up loans. Thus, these students are more likely to end up with no outstanding need as loan aid is directly applied to on-campus room and board expenses. This pattern is consistent across the different student affordability levels (EFC ranges) and shows that the confluence of low affordability (Pell eligible) and loan amount taxes enrollment persistence. Low-EFC students with no remaining need take up loans more than twice as large as Low-EFC students with high remaining need (over $10,000, see Table 14). The substantial difference in average loan amount across remaining need levels combined with the finding that unsubsidized loans do not elevate the dropout risk (net of subsidized loans) suggests that the level of debt may exert a negative psychological effect separate from the economic effect associated with an immediate financial payback obligation and accumulating interest payments. If so, it may corroborate results in Cadena and Keys (2013) that self-control among some students leads to debt-aversion and thus lower loan take-up rates, at least for students with alternate funding sources or those seeking to avoid excessive consumption while in college. The data also show that Low-EFC students who are academically more challenged are least likely to live on campus and hence incur the lowest loan debt in the first year (see Table 14). However, the low academic success and persistence of these students suggests that other factors are at work not captured here. While the estimation model controls for on-campus employment and takes into account student plans to work while in college, data on actual offcampus work hours was not available. Since low-income students with credit constraints are more likely to work longer hours while in college (Soria, Weiner, and Lu, 2014), which may heighten their dropout risk (Mendoza, 2012), it is plausible that Low-EFC students with high remaining need eschew larger loan debt in favor of off-campus employment while in college, thus engaging in a trade-off that seemingly compromises their academic success and progress just the same. Conclusion The estimated effect of loan aid on student persistence varies with ability to pay for college as well as with the amount of loan aid and associated remaining financial need. Accordingly, Pell Grant-eligible students with loans and students who take-up higher amounts of loan aid to meet college costs are at an elevated risk of departure after the first year of college. Unsubsidized 13 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE loans exert no significant impact on enrollment persistence beyond the estimated effect of interest-free subsidized government loans. The observed significant effects emerged only after addressing selection bias associated with loan status. Thus, this study confirms the importance of taking into account student discretion in selecting loan aid when gauging the impact of this type of financing on academic outcomes. Having chosen a counterfactual analytical framework to estimate the effect of student loans, this study seeks to respond to the call for more rigorous research to understand the role of financial aid in college success (Hossler et al., 2009). Though the findings here are drawn from data at one institution, with inference necessarily limited to that campus, the insight gained by Gross et al. (2015, p. 244) from the large-scale NPSAS data shows that “student departure is strongly influenced by the unique experiences of students on individual campuses,” prompting the authors to encourage more research in this area at individual institutions. At the same time, the results here are based on the experience of firstyear students and may not be extrapolated to students progressing to the second year and beyond. Moreover, the estimated effects may still be biased to the extent that unobserved factors governing student selection of loan aid are uncorrelated with those included in the selection estimation, as well as possibly significant omitted factors related to first-year student persistence. The finding that loan aid for some students reduces the institution’s overall freshmen retention rate by at least one percent may offer the institution the opportunity to tweak its student aid policy in order to move the needle on a ‘key performance indicator’ widely used in assessment, program review, accreditation, and college survey rankings. Results indicate that estimation of the effect of loan aid on student outcomes can tangibly support policy decisions at the senior management level if the analysis accounts for student ability to pay (EFC) and the remaining financial need a student faces after considering all aid received. Since the observed negative effect of loan aid is limited to students least likely to afford college (Pell-eligible) and those relying on large amounts of loans to meet college expenses, the institution should explore alternate funding or discount options for these students. Such effort should also consider enhanced academic support (e.g. tutoring and counseling), given that academic performance has a greater impact on the persistence of low-income students compared to students less challenged to pay for college. 14 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE References Adelman, C. (2004). Principal Indicators of Student Academic Histories in Postsecondary Education, 1972-2000. Washington, DC: U.S. Department of Education. Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through college. Washington, DC: U.S. Department of Education. Adelman, C. (2007). Do we really have a college access problem? Change (July-August): 48-51. Allison, P. (2012). When can you safely ignore multicollinearity? Statistical Horizons. Retrieved August 8, 2014, at http://www.statisticalhorizons.com/multicollinearity Alon, S. (2005). Model mis-specification in assessing the impact of financial aid on academic outcomes. Research in Higher Education, 46 (1), 109-125. Angrist, J. D. (2003). Randomized trials and quasi-experiments in education research. NBER Reporter, No.4. National Bureau of Economic Research. Angrist, J., Autor, D., Hudson, S., & Pallais, A. (2014, December). Leveling up: Early results from a randomized evaluation of post-secondary aid. (NBER Working Paper 20800). Cambridge, MA: National Bureau of Economic Research. Astin, A. W. (1993). What Matters in College? Four Critical Years Revisited. San Francisco: Jossey-Bass. Attewell, P., Heil, S., & Reisel, L. (2012). What is academic momentum? And does it matter? Educational Evaluation and Policy Analysis, 34, 27-44 Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46, 399-424. Avery, C., & Turner, S. (2012). Student loans: Do college students borrow too much—or not enough? The Journal of Economic Perspectives, 26 (1), 165-192. Bettinger, E. P., Long, B. T., Oreopoulos, P., & Sanbonmatsu, L. (2009, September). The role of simplification and information in college decisions: Results from the H&R Block FAFSA experiment. (NBER Working Paper 15361). Cambridge, MA: National Bureau of Economic Research. Booij, A. S., Leuven, E. & Oosterbeek. (2011). The role of information in the take-up of student loans. Economics of Education Review, 31, 33-44. Bowman, N. A., & Herzog, S. (eds.) (2014). Methodological Advances and Issues in Studying College Impact. New Directions for Institutional Research. no. 161, San Francisco: Jossey-Bass. Brown, M., Haughwout, A., Lee, D., Scally, J., & van der Klaaw, W. (2014, April). Measuring student debt and its performance (Staff Report No. 668). New York, NY: Federal Reserve Bank of New York. Cadena, B. C., & Keys, B. J. (2013). Can self-control explain avoiding free money? Evidence from interest-free student loans. The Review of Economics and Statistics, 95 (4), 1117-1129. Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22, 31-72. 15 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Chen, J., & Zerquera, D. (2011). “A methodological review of studies on effects of financial aid on college student success. “ Paper prepared for the 36th annual conference of the Association for Education Finance and Policy. Cho, S. H., Xu, Y., & Kiss, D. E. (2015). Understanding student loan decisions: A literature review. Family and Consumer Sciences Research Journal, 43, 229-243. Chudry, F., Foxall, G., & Pallister, J. (2011). Exploring attitudes and predicting intentions: Profiling student debtors using an extended theory of planned behavior. Journal of Applied Social Psychology, 41, 119-149. Cruce, T. M. (2009). A note on the calculation and interpretation of the Delta-p statistic for categorical independent variables. Research in Higher Education, 50, 608-622. DesJardins, S. L., Ahlburg, D. A., & McCall, B. P. (2002). Simulating the longitudinal effects of changes in financial aid on student departure from college. Journal of Human Resources, 37, 653-679. Elliot, W., & Nam, I. (2013). Reducing student loan debt through parents’ college savings (Vol. 49, pp. 265-286). St. Louis, MO, Center for Social Development at Washington University. Gross, J. P., Hossler, D., Ziskin, M., & Berry, M. S. (2015). Institutional merit-based aid and student departure: A longitudinal analysis. The Review of Higher Education, 38, 221-250. Hansen, B. B. (2004). Full matching in an observational study of coaching for the SAT. Journal of the American Statistical Association, 99, 609-618. Heckman, J. J. (2000). Causal parameters and policy analysis in economics: A twentieth century retrospective. The Quarterly Journal of Economics, 115, 45-97. Herzog, S. (2005). Measuring determinants of student return vs. dropout/stopout vs. transfer: A first-to-second year analysis of new freshmen. Research in Higher Education, 46 (8), 883-928. Herzog, S. (2008, January). Estimating the influence of financial aid on student retention (Education Working Paper Archive). Fayetteville, AR: University of Arkansas, Department of Education Reform. Herzog, S. (2014). The Propensity Score Analytical Framework: An Overview and Institutional Research Example. In. N.A. Bowman and S. Herzog (eds.) Methodological Advances and Issues in Studying College Impact. New Directions for Institutional Research. no. 161, San Francisco: Jossey-Bass. Ho, D. E., Imai, K., King, G., & Stuart, E. (2011). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42 (8), 1-28. Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945-960. Hossler, D., Ziskin, M., Gross, J., Kim S., & Cekic, O. (2009). Student aid and its role in encouraging persistence. In J.C. Smart (ed.) Higher Education: Handbook of Theory and Research. (pp. 389-325). New York: Springer. Johnson, M. T. (2013). Borrowing constraints, college enrollment, and delayed entry. Journal of Labor Economics, 31, 669-725. Kim, D. (2007). The effect of loans on students’ degree attainment: Differences by student and institutional characteristics. Harvard Educational Review, 77, 64-100. Lechner, M. (1999, December). Identification and estimation of causal effects of multiple treatments under the conditional independence assumption (Discussion Paper No. 91). Bonn, Germany: Institute for the Study of Labor (IZA). 16 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Marx, B. M., & Turner, L. J. (2015, January). Borrowing trouble? Student loans, the cost of borrowing, and implications for the effectiveness of need-based grant aid. (NBER Working Paper 20850). Cambridge, MA: National Bureau of Economic Research. Mendoza, P. (2012). Should I work or should I borrow? A counterfactual analysis on the effect of working while in enrolled and debt on baccalaureate completion. Journal of Student Financial Aid, 42, 25-59. Morgan, S. L., & Winship, C. (2007). Counterfactuals and causal inference: Methods and principles for social research. New York: Cambridge University Press. Murnane, R. J., & Willett, J. B. (2010). Methods matter: Improving causal inference in educational and social science research. Oxford University Press. Pascarella, E. T., & Terenzini, P. T. (2005). How College Affects Students: A Third Decade of Research. San Francisco, CA: Jossey-Bass. Pike, G. R., Hansen, M. J., & Lin, C. (2011). Using instrument variables to account for selection effects in research on first-year programs. Research in Higher Education, 52, 194-214. Reynolds, C. L., & DesJardins, S. L. (2010). The use of matching methods in higher education research. In J. C. Smart (Ed.), Higher Education: Handbook of Theory and Research, Volume XXIV (pp. 47-104). New York: Springer. Rosenbaum, P. R. (2002). Observational Studies. New York: Springer. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55. Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516-524. Rubin, D. B. (1980). Bias reduction using Mahalanobis metric matching. Biometrics, 36, 293-298. Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169-188. Rubin, D. B. (2006). Matched Sampling for Causal Inference. New York: Cambridge University Press. Sekhon, J. S. (2011). Multivariate and propensity score matching software with automated balance optimization: The Matching package for R. Journal of Statistical Software, 42 (7), 1-22. Soria, K. M., Weiner, B., & Lu, E. C. (2014). Financial decisions among undergraduate students from low-income and working-class social backgrounds. Journal of Student Financial Aid, 44, 2-23. Stuart, E. A. (2010). Matching methods for causal inference: A review and look forward. Statistical Science, 25, 121. Stuart, E. A., & Rubin, D. B. (2008). Best practices in quasi-experimental designs. In J. W. Osborne (Ed.), Best Practices in Quantitative Methods (pp. 155-176). Thousand Oaks, CA: SAGE. Titus, M. A. (2007). Detecting selection bias, using propensity score matching, and estimating treatment effects. Research in Higher Education, 48, 487-521. U.S. Department of Education. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. Washington, DC: Institute of Education Sciences. 17 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Welbeck, R., Diamond, J., Mayer, A., & Richburg-Hayes, L. (2014). Piecing together the college affordability puzzle, MDRC Report, New York Ziskin, M., Fischer, M. A., Torres, V., Pellicciotti, B., & Player-Sanders, J. (2014). Working students’ perceptions of paying for college: Understanding the connection between financial aid and work. The Review of Higher Education, 37 (4), 429-467. 18 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE TABLES AND FIGURES Table 1: Students with Loans by EFC Level Any Loans Subsidized Unsubsidized Low (Pell eligible), N=1386 Mid ($5.2K to $20K), N=1171 High (>$20K), N=1173 51% 56% 43% 50% 50% 6% 37% 44% 40% Total, N=3730 50% 37% 40% Table 2: Academic Momentum (GPA-Credits) of Students Low (Pell eligible), N=1386 Mid ($5.2K to $20K), N=1171 High (>$20K), N=1173 Any Loans Subsidized Unsubsidized 77.3 78.7 81.3 77.4 79.0 79.8 77.2 78.3 81.0 78.2 78.8 Total, N=3730 81.2 Acad Momentum: Mean = 81.2; Std Dev = 20.7 Table 3: EFC Level of Students with No Remaining Need low (Pell eligible) mid ($5.2K to $20K) high (>$20K) Total N 228 221 304 751 Percent 30.1 29.4 40.5 100 Table 4: EFC Level of Students with High Remaining Need low (Pell eligible) mid ($5.2K to $20K) high (>$20K) Total N 374 624 672 1670 Percent 22.4 37.4 40.2 100 19 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Table 5: Remaining Need of Students with Low EFC (Pell eligible) none < $10K > $10K Total N 228 784 374 1386 Percent 16.4 56.6 27 100 Table 6: Remaining Need of Students with High EFC (> $20K) none < $10K > $10K Total N 304 197 672 1173 Percent 25.9 16.8 57.3 100 Figure 1: Propensity Score Balance with Full Sub-Classification (Loans) 20 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Figure 2: Covariate Balance with Full Sub-Classification (Loans) Figure 3: Propensity Score Balance with Full Sub-Classification (Unsub) 21 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Figure 4: Covariate Balance with Nearest Neighbor (Loans) Figure 5: Covariate Balance with Nearest Neighbor (Unsub) 22 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Table 7: Covariate and Propensity Score Balance for Matched Data NNR:1 Std. Difference in Mean* for: All covariates Balance Improvement % Out of state, living in dorms Balance Improvement % EFC, Need, Pell, Merit aid, Sub loans^ Balance Improvement % Work plans, Parent education Balance Improvement % Propensity score Balance Improvement % Key Covariates with SD > 0.25 Key Covariates with SD > 0.1 All Loans Full SubCl -0.017 49.2 -0.057 79.3 0.007 86.4 -0.022 70.2 0.000 100.0 0 0 -0.017 53.0 -0.080 70.6 -0.014 81.2 -0.041 66.1 0.000 100.0 0 1 Unsubsidized Loans NNR:1 Full SubCl 0.006 51.5 0.036 86.4 -0.019 12.2 0.031 54.4 0.000 100.0 0 3 0.002 47.0 0.026 83.9 -0.021 6.4 0.019 47.8 0.000 100.0 0 2 *Formula denominator uses standard deviation of treated, not pooled, unlike Cohen's d ^Unsubsidized loan models only Table 8: Fall-to-Fall Persistence and Naïve Estimator All students EFC Low (<$5.2K, Pell eligible) (N= 704, 682) EFC High (>$20K) (N= 507, 666) Remaining Need Zero ($ 0) (N= 416, 335) Remaining Need High (> $10K) (N= 666, 1004) E(Y1) Loans E(Y0) No Loans (N=1870) 78% (N=1860) 85% 75% 82% -7% 82% 88% -6% 79% 85% -6% 74% 84% -10% E(Y1) - E(Y0) -7% 23 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Table 9: Estimation of Loan Effect on Second-Year Persistence Treated (N = all students) Untreated (N = all students) All students: Null Model Logit Regr (N=1870) (N=1860) -.216** IPTW (N=1870) (N=1860) -.206** NNR:1 (N=1870) (N=800) -.237** Full Sub-Class (N=1870) (N=1860) -.100 .099 .0844 .1081 .0808 -.135 -.155* -.250** -.126 .106 .0894 .1166 .0862 -.355** -.261* -.224 -0.199 .162 .1373 .1694 .1247 EFC Low: Post-Treat Model -.214 -.137 -.261 -.255** .176 .1453 .1894 .1413 EFC High: Null Model -.047 -.050 .082 .033 .197 .1779 .2023 .1547 -.024 -.072 .078 .033 .211 .1886 .2237 .1665 Zero Need: Null Model -.290 -.276* -.453* -.569*** .221 .1688 .2467 .2024 Zero Need: Post-Treat Model -.293 -.269 -.367 -.479** .231 .1743 .2587 .2122 -.485*** -.335** -.147 -.193* All students: Post-Treat Model EFC Low: Null Model EFC High: Post-Treat Model High Need: Null Model High Need: Post-Treat Model .156 .1514 .1671 .1171 -.274 -.099 -.178 -.174 .172 .1654 .1879 .1267 Alpha significance: * = 0.10, ** = 0.05, *** = 0.01. Standard errors in italics, bootstrapped for IPTW and Full Sub-Class 24 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Table 10: Estimation of Unsubsidized Loan Effect on Second-Year Persistence Treated (N = all students) Untreated (N = all students) All students: Null Model Logit Regr (N=1497) (N=2233) .048 IPTW (N=1497) (N=2233) -.028 NNR:1 (N=1497) (N=589) .108 Full Sub-Class (N=1497) (N=2233) -0.079 .116 .0918 .1161 .082 .030 .069 .209* .028 .124 .0987 .1263 .0905 EFC Low: Null Model -.048 .080 -.116 -.083 .214 .1744 .2307 .1284 EFC Low: Post-Treat Model -.038 .131 -.213 -.168 .233 .1907 .2536 .1410 EFC High: Null Model -.140 -.313 -.410* -.280* .196 .2088 .2238 .1616 EFC High: Post-Treat Model -.070 -.260 -.316 -.235 .210 .2261 .2410 .1742 .235 .081 .0 -.065 .265 .2000 .2616 .1963 .375 .296 .262 .220 All students: Post-Treat Model Zero Need: Null Model Zero Need: Post-Treat Model High Need: Null Model High Need: Post-Treat Model .277 .2138 .2776 .2111 -.409** -.349** ~~ -.306*^ .175 .1589 -.259 -.162 .190 .1768 .1825 ~~ -.227 .2043 Alpha significance: * = 0.10, ** = 0.05, *** = 0.01. ^ Genetic algorithm. ~~ Algorithm failed to reach acceptable data balance Standard errors in italics, bootstrapped for IPTW and Full Sub-Class Table 11: Effect of Loan and Unsubsidized Loan Aid on Enrollment Persistence All Loans Steps (Cruce formula) Formula (FullEFClow Ex) 1) Parameter Est of baseline (log) Unsubsidized Only EFCHi NeedHi Null Null FullEFClow ZeroNeed Ln(0.78 / 1 - 0.78) 1.27 1.52 1.73 1.15 2) Parameter Est of cent ref gr (log) 1.27+(-.255 * -.51) 1.40 1.78 1.84 1.35 3) Parameter Est of cent tar gr (log) 1.27+(-.255 * .49) 1.15 1.30 1.56 1.04 4) Probability (%) of reference group Exp(1.40) / (1 + Exp 1.40) 0.80 0.86 0.86 0.79 5) Probability (%) of target group Exp(1.15) / (1 + Exp 1.15) 0.76 0.79 0.83 0.74 6) Change (%) due to target group Step 5 minus step 4 -4% -7% n/s -7% -6% n/s -4% -6% n/s -5% -9% -7% Naïve Estimator E(Y1) - E(Y0) Standard Logit Regression 25 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Table 12: Covariate and PS Balance for Models with Significant Loan Coefficient Std. Difference in Mean* for: All covariates Out of state, living in dorms EFC, Need, Pell, Merit aid, Sub loans^ Work plans, Parent education Propensity score Key Covariates with SD > 0.25 Key Covariates with SD > 0.1 All Loans Full SubCl Full SubCl EFC Low Zero Need -0.013 -0.002 -0.020 0.050 -0.017 -0.011 -0.029 0.024 0.000 0.001 0 0 2 1 Unsubsidized Loans Full SubCl Genetic EFC High High Need -0.006 0.022 0.090 0.149 -0.050 -0.004 -0.004 0.101 0.009 0.377 0 0 2 6 *Formula denominator uses standard deviation of treated, not pooled, unlike Cohen's d ^Unsubsidized loan models only Table 13: Statistical Significance of Academic Momentum (Wald Statistic) EFC High (> $20K) EFC Low (Pell Eligible) Unmatched Unmatched Unsub IPTW IPTW Unsub Full Sub Class Full Sub Class Unsub Near R:1 Near R:1 Unsub 63 62 50 47 81 73 63 51 127 77 90 76 172* 116 126 64 * Loan coefficient is significant 26 | P a g e FINANCIAL AID AND COLLEGE PERSISTENCE Table 14: Selected Statistics for EFC Level by Remaining Need Level Aid from Loans Average Loan Live On Campus Low EFC (Pell eligible) 21.6% $5,740 78.0% 74.6 78.9 78.8 93.0% 71.0% Mid EFC ($5.2K to $20K) 27.0% $6,827 80.0% 75.8 84.5 82.3 97.0% 81.0% High EFC (>$20K) 33.6% $8,827 78.0% 76.0 83.2 81.0 94.0% 84.0% Total (N = 416) 27.7% $7,208 79.0% 75.5 82.3 80.7 95.0% 79.0% Low EFC (Pell eligible) 20.1% $3,457 59.0% 74.6 79.0 79.8 95.0% 79.0% Mid EFC ($5.2K to $20K) 35.4% $5,501 58.0% 75.0 79.4 75.6 91.0% 83.0% High EFC (>$20K) 38.3% $6,665 65.0% 75.0 82.8 81.7 97.0% 87.0% Total (N = 785) 27.2% $4,536 59.0% 74.8 79.7 79.0 94.0% 81.0% Low EFC (Pell eligible) 25.6% $2,746 55.0% 70.8 69.5 63.6 78.0% 63.0% Mid EFC ($5.2K to $20K) 38.4% $2,973 58.0% 75.2 75.6 73.8 89.0% 75.0% High EFC (>$20K) 43.1% $3,442 57.0% 74.9 79.2 76.1 91.0% 79.0% Total (N = 666) 37.5% $3,090 57.0% 74.3 75.7 72.6 88.0% 74.0% No Remaining Need Academic Fall Spring Spring Second-Year Preparation Momentum Momentum Persistence Peristence Remaining Need up to $10K Remaining Need over $10K 27 | P a g e

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement