Financial Aid and College Persistence: Do Student Loans Help or Hurt?

Financial Aid and College Persistence: Do Student Loans Help or Hurt?
Financial Aid and College
Do Student Loans Help or Hurt?
A Selection Control Study
Version: May 2015*
Draft: Do not cite without permission
Serge Herzog, PhD
Director, Institutional Analysis
Consultant, CRDA StatLab
University of Nevada, Reno
Reno, NV 89557
[email protected]
*Prepared for presentation at annual forum of the Association for Institutional Research
(AIR), Denver, CO, May 25-29, 2015
Using data from two freshmen cohorts at a public research university (N = 3,730), this study
examines the relationship between loan aid and second-year enrollment persistence. Applying a
counterfactual analytical framework that relies on propensity score (PS) weighing and matching
to address selection bias associated with treatment status, the study estimates that loan aid exerts
a significant negative effect on persistence for students from low-income background (i.e., Pell
eligible) and those taking up high amounts of loans in order to meet total cost of attendance.
However, no significant incremental effect associated with unsubsidized loan aid, net of subsidized
loan aid, could be detected. The estimated effect of loan aid on persistence controls for first-year
academic experience and takes into account 25 factors related to loan selection and persistence in
order to match students with loan aid to a counterfactual case in covariate adjusted regression.
Comparison with results from single-stage standard regression suggests selection bias masks the
negative effect of loans detected with matched-sample estimation. Validity of covariates
determining the loan selection process and criteria for acceptable balance in the matched data are
discussed, and implications for future research are addressed.
Keywords: student persistence, financial aid, loans, low income, causal inference estimation,
propensity score
A recent analysis of nationally representative enrollment data of college students in the United
States fails to identify entry barriers for prepared low-income students and instead stresses the
lack of enrollment persistence following initial college matriculation among such students. Lack
of continued college participation of low-income students is attributed primarily to inadequate
academic preparation and student self-reported reasons other than financial aid (Adelman, 2007).
Pascarella and Terenzini (2005) arrived at a similar finding after reviewing the cumulative
research since the 1980s, which shows that on average students with financial aid are no more
likely to persist and graduate than those without aid. However, they did conclude that the effect
varies by aid type, with merit-based scholarship aid having the greatest positive impact, while
need-based grants having a modest positive effect on persistence and graduation. In contrast, the
impact of work-study is more ambiguous, and the effect of student loans is “mixed” at best
(Pascarella and Terenzini, 2005, p. 411).
More recent reviews on the impact of aid largely corroborate previous findings and highlight
that incongruities in results stem from differences in methodology, data sources, scope of the
analysis, type of aid covered, and how aid is measured (Welbeck et al., 2014; Chen and
Zerquera, 2011; Hossler et al., 2009). In fact, the influence of aid on student enrollment
persistence is mediated by a complex web of interrelated factors: including the timing, type, and
amount of aid and how they correlate with persistence in the presence of other student attributes
(Pascarella and Terenzini, 2005). In spite of voluminous research on student success, enrollment
retention models yield limited insight to evaluate the impact of financial aid. Descriptive studies
fail to control for the multiple influences that govern student behavior. Equally problematic are
studies that employ inferential statistics without subject randomization or mechanisms to reduce
selection bias associated with students that receive aid.
The importance of moving beyond correlational analysis and estimating causal relationships
in education research has been stressed for some time (Murnane & Willett, 2010; Morgan &
Winship, 2007; U.S. Department of Education, 2003; Angrist, 2003). Since randomized control
trials (RCTs) are costly, raise ethical issues, and are fraught with operational difficulties, they are
typically not available to the analyst. Instead, one may rely on techniques that attempt to mimic
random assignment in order to reduce selection bias with observational data. One of these
techniques is centered on the propensity score (PS), a measure of the chance a person will select
an experience or ‘treatment’, in this case whether or not a student selects loan aid to finance
college. Thus, this study uses the PS approach to examine the influence of student loans on
enrollment persistence.
Loan Selection
The focus of this study is limited to loan aid, since the selection process associated with loan aid
is governed far more by discretionary decision compared to scholarship aid or need-based grant
aid. In a review of the determinants of loan selection Cho, Xu, and Kiss (2015) highlight the
importance of family and parental contributions to cover student educational expenses. Brown et
al. (2012) found that the degree of unmet expected family support is associated with demand for
financial aid, while Elliott and Nam (2013) reported that students with college savings accounts
are less likely to seek financial aid. Data from the 1999-2000 and 2003-2004 waves of the
National Postsecondary Student Aid Study (NPSAS) confirm that students with the highest
unmet need are most likely to take up loans (Cadena and Keys, 2013). Student proclivity to rely
on loans is also associated with past behavior, consideration of parental views, gender, and
support for future goals in life (Chudry, Foxall, and Pallister, 2011). In contrast, access to
financial aid information does not appear to affect the likelihood of loan take-up based on
findings from a randomized field experiment (Booij, Leuven, and Oosterbeek, 2012). However,
assistance with filling out and processing the Free Application for Federal Student Aid (FAFSA),
which covers subsidized and unsubsidized student loans, does increase the rate of aid application
(Bettinger, E. P. et al., 2009). Thus, cognitive constraints may influence whether or not a student
applies for loan aid.
Loans are typically a ‘last resort’ source of aid for students after merit-based scholarships and
need-based grants are exhausted (Ziskin et al., 2014). Similarly, students who qualify for
subsidized loans may select to rely on unsubsidized loans if the former are considered
insufficient. In contrast to scholarship and grant aid that are available (and thus selected) based
on academic merit and income background, selection of loan aid is more directly tied to
perceived need, which is a function of the actual remaining need after taking into account all
other aid received (Cadena and Keys, 2013; Brown et al., 2012). Unlike scholarship and grant
aid, which typically offer little discretion for the student to select and accept or not—amounts are
usually set and eligibility clear cut—students face much greater discretion whether or not to take
out a loan and for what amount. To receive a loan, students are required to sign off prior to
receipt, unlike grant and scholarship aid which is pushed to students after completion of the
FAFSA. Thus, the presence of selection bias is decidedly more plausible with loans compared to
scholarships and grants, a point highlighted in Hossler et al. (2009, pp. 401, 413) in their review
of the research: “Because of the complex interplay of privilege, opportunity, and conditioned
action in society, the question of endogeneity bias becomes an inherent controversy in the study
of financial aid and persistence in college. [Since] loans have to be repaid, the possibility of selfselection bias influencing any analysis of the impact of student loans is great.”
To address this challenge in the research, this study estimates the influence of loans on firstyear student persistence with a series of regression models that weigh and match students based
on their PS. The next section briefly explains the PS methodology and analytical framework,
followed by a description of the study and statistical results, and a discussion of the findings and
its implication for future research.
The Propensity Score and Counterfactual Inference Model
Like regression discontinuity and instrumental variable (IV) methods (see Bowman and Herzog,
2014; Murnane and Willett, 2010), the PS method offers a way to address selection bias and to
establish control groups as benchmarks to gauge the effect of student loans on enrollment
persistence. Use of the PS in observational studies was popularized by Rosenbaum and Rubin
(1983, 1984) and is closely linked to the counterfactual analytical framework that emerged in
statistics and econometrics (Holland, 1986; Rosenbaum, 2002; Rubin, 2006; Heckman, 2000).
The counterfactual model seeks an answer to the “what if” question. In the context of this study,
what is the potential enrollment outcome for a student who took up loan aid to pay for college
had that student not chosen to rely on loan aid? In other words, a student has potentially two
outcomes, but only one can be observed. In order to measure this unobserved outcome, one has
to account for the likelihood (or predisposition) of a student selecting loan aid when measuring
the relationship between loan aid and enrollment persistence. To render the correlation
insignificant between a student’s chance to select loan aid (i.e. the ‘treatment’) and the outcome
of interest (i.e., enrollment persistence), one has to have a comparator case, a student with equal
or similar chance to select the treatment but did not, and who can be equally observed like the
‘treated’ student with respect to the outcome. The ability to create a counterfactual case—the
comparable untreated student that substitutes for the unobserved outcome of the treated
student—is at the core of causal inference estimation after controlling for treatment selection
(i.e., selection bias).
Rosenbaum and Rubin (1983) showed that matching a treated case with an untreated case on a
single probability (propensity) score that adequately captures the linear combination of factors
that predict treatment selection offers a way to reduce selection bias. Formally, the propensity
pˆ(Xi) = Pr(Di = 1|Xi)
measures the probability (Pr) of selecting into treatment D conditional on observable predictors
(covariates X) for each case (i). Assuming the treatment can be measured dichotomously (D = 1
or 0), the average treatment effect (ATE) conditional on X is formalized as
E(∆i|Xi = x) = E[(Yi| Di = 1)-(Yi| Di = 0)]| Xi = x,
where E(∆i|Xi = x) is the expected difference in the outcome Yi between the treated (Di = 1) and
untreated (Di = 0), controlling for observable factors (Xi = x) that predict treatment (D) selection.
Since financial aid evaluation in education is focused typically on intervention or support for
those students who need the aid, the counterfactual centers on the student with a loan—i.e. had
that student not selected loan aid—and thus the average treatment effect on the treated (ATT) is
defined as
E(∆i| Di = 1) = E[(Yi| Di = 1)-(Yi| Di = 0)]| Di = 1,
or the difference in outcome under treatment and non-treatment for those students who actually
did select loan aid. Since only the treated outcome (Yi| Di = 1) is observed for the treated, the
untreated outcome (Yi| Di = 0) is derived from untreated cases (i.e., those actually without loan
aid) with similar or identical chance of treatment selection as the treated based on observable
characteristics (Xs).
The effectiveness of the PS to yield unbiased effect estimates hinges on the selection of
observables—known as the conditional independence assumption, strongly ignorable treatment
assignment, or unconfoundedness—to render treatment selection independent of the potential
outcomes for the treated and untreated. That is to say, likelihood of treatment selection does not
affect the outcome of interest. Accordingly, the chance of selecting loan aid does not influence
the student’s enrollment persistence. This independence cannot be established empirically, but
must be anchored in covariate selection that is guided by explicit theory and an understanding of
the factors that prompt students to seek loan aid to finance their education. Thus, only
observable factors (predictors) that temporally precede the selection of loan aid can be used in
the PS estimation. The PS must also show sufficient overlap in distribution between students
with loans and those without loans in order to produce a comparator case; the individual
covariates used in estimating the PS should be closely balanced between the two groups to rule
out bias associated with observable characteristics; and enrollment persistence should not be
influenced by the loan status of other students (known as the Stable Unit Treatment Value
Assumption; see Rubin, 1980). While the first two conditions can be ascertained empirically ex
post facto, the last condition is best addressed with a study design that minimizes, or at least does
not promote, interaction between students with loans and those without loans.
After calculating the PS with a logit or probit regression model for a binary treatment (i.e.,
selecting loan aid or not), the effect of loan aid on enrollment persistence can be estimated by
matching students with loans to those without loans on their PS, by stratifying students into
subclasses (ranges) of the PS, by inverse probability of treatment weighting (IPTW) using the
PS, or by including the PS and treatment indicator in covariate adjusted regression (Austin, 2011;
Murnane & Willett, 2010). Unlike standard regression, which uses parametric specification to
account for covariate effects, application of the PS relaxes the linearity assumption through
preprocessing of the data for either matching on a single metric to establish a control group
(untreated) in order to directly measure the difference in outcome to the treated group
(nonparametric), or through pre-selecting or weighting cases in semi-parametric covariate
regression. This study uses the PS in both IPTW and covariate adjusted regression and contrasts
results with standard regression in order to identify possible selection bias.
Estimating the Effect of Student Loans on Enrollment Persistence
Data source and Measures. Data are drawn from a moderately selective university (average
SAT of 1080, ACT of 23) and capture the profile and academic experience of new first-year
undergraduate students that enrolled in the fall semester of 2011 and 2012. The effective sample
of 3,730 freshmen excludes students with less than 12 attempted semester credits, students with
complete credit withdrawal after registration, students without entry survey data, those who did
not complete the Free Application for Federal Student Aid (FAFSA), and statistical outlier cases
identified in a regression model (using Cook’s and Mahalanobis distance) that included all the
covariates used in the subsequent analysis. The effective sample amounts to 70 percent of all
freshmen in the two cohort years.
Selection of the covariates to estimate the PS is governed by several criteria. First, they are
all pretreatment baseline characteristics that exist prior to student selection of loan aid and thus
are not conditioned by treatment assignment. Second, all of the covariates may be related to both
selection of loan aid and enrollment persistence. While the most important covariates are those
related to selection of loan aid, both theoretical and empirical research show that PS estimation
with a large set of covariates (at least ten or more), including covariates indirectly related to the
outcome, yields better PS balance between treatment and control groups (Stuart and Rubin,
2008). Third, most of the covariates selected here have been used extensively in prior research
to understand student academic success and enrollment persistence (Pascarella & Terenzini,
2005; Astin, 1993).
Twenty-five covariates were used to estimate the PS with logistic regression using loan status
as the dependent variable. Remaining need ($ amount), estimated family/student contribution
(EFC, $ amount), and whether or not a student had scholarship aid or a Pell Grant are included in
the PS estimation. These variables in combination with on-campus living status, out-of-state
residency status, participation in a living-and-learning community, and student work plans (taken
from entry survey) emerged as the strongest predictors of student loan selection (Wald > 30).
Not surprisingly, most of these variables determine a student’s cost of attendance and are
sourced directly from the student’s FAFSA information (Cadena and Keys, 2013). The inclusion
of a continuous metric for remaining need—the amount outstanding after accounting for EFC
and all types of aid received—allows testing of the impact of loan aid across different need
levels. Similarly, incorporation of EFC offers an opportunity to examine the influence of loan
aid as a function of the ability to pay for college, rather than merely income level, which fails to
reflect the estimated expense burden facing a student.
The remaining covariates capture a student’s socio-demographic profile (being age 19 or
older, male, non-Asian ethnic minority, father has 4-year college degree, mother has 4-year
college degree, in-state residency outside of local area), academic preparation (high school
GPA, ACT/SAT test scores, high school class rank percentile, advanced standing at entry), and
academic motivation and educational goal (delayed college entry by 6 months or more,
ACT/SAT test date, institution is first choice, undeclared major, math-intensive major, plans to
attend graduate school). Except for the precollege preparation index (composed of high school
GPA and test scores), the high school percentile rank, and the test date, all covariates are dummy
coded (no = 0, yes = 1). The test date is a continuous metric that measures the months elapsed
between the first time the student took the ACT/SAT test and the start of college. Advanced
standing indicates if a student earned college credit in high school, while math-intensive major
identifies students who chose an academic program that requires advanced math in the first year
of study. Test date and delayed entry are proxy indicators for student commitment and
motivation, following a previous study that used admission date for that purpose (Pike, Hansen,
& Lin, 2011). Identifying students with a math-intensive major takes into account that math
courses dominate the list of courses undergraduate students are most likely to fail or drop
without completion (Adelman, 2004).
All data originated with the institution’s matriculation system, except for several student selfreported data elements that were recorded as part of the mandatory start-of-semester orientation
survey. They include measures of parent level of education, intent to attend graduate school,
plans to work full-time or plans to not work while in college (plans to work half-time being the
reference category), and having selected the institution as first choice. The PS for loan status (0 =
no, 1 = yes) is estimated first for all types of loans—subsidized and unsubsidized—and
subsequently for only unsubsidized loans. This offers the opportunity to gauge the incremental
effect of unsubsidized loans after factoring in subsidized loan aid. This distinction is important,
since students that are eligible for subsidized loans are less likely to take out unsubsidized loans
compared to those ineligible for subsidized loans (Ziskin et al., 2014).
PS methods. The effect of student loans on fall-to-fall enrollment persistence (coded 0 = no; 1
= yes) is estimated with a series of two-stage covariate adjusted regression models that separate
students by EFC level and remaining need level, all using data that match and weigh students on
the PS associated with loan status. Results of each PS method are estimates of the ATT—i.e.,
the effect on students with loans had they not taken out loan aid—and are compared to standard
regression with the same set of covariates. Data were matched using the nearest neighbor and
full matching functions in the MatchIt R program (Ho et al., 2011), and genetic matching using
the Matching program in R by Sekhon (2011). These algorithms yielded the best covariate
balance among several others in a PS evaluation study (Herzog, 2014) and thus are relied upon
here. With the focus set on the ATT, the common support area is defined by the PS range of
students with loan aid that are matched with students without loan aid based on their PS. The
nearest neighbor with replacement (NNR:1) match chooses only the control case with the closest
PS to a treated case, and thus the same control case may be matched to multiple treated cases to
minimize the average difference (distance) in the PS. This type of matching is suitable for PS
distributions with limited overlap between treated and control cases, and where the latter are
fewer in number.
An extension of the more commonly used matching by PS stratification is the method of full
matching (Full SubClass), algorithmically automated to minimize the PS difference by
partitioning the data into an optimal number of subclasses that contain the best bi-directional
ratio of treated and control cases. The last matching method relies on a genetic search algorithm
designed to maximize the PS balance across all covariates. Balance is determined with paired ttests for dichotomous variables and the Kolmogorov-Smirnov test for continuous variables at
every iteration. This lets the evolutionary algorithm automatically adjust the number of cases
used in the random trail to meet operator restrictions in order to optimize the solution. Assuming
good covariate and PS balance between students with loan aid and those without loan aid, the
full sub-classification method has the greatest potential to minimize both variance and bias in the
data, for it retains more control cases than one-on-one matching (Hansen, 2004). Therefore,
genetic matching is used as a backup when sub-classification matching fails to produce good
Results from PS-matched data are compared with standard regression and with IPTW. With
IPTW, treated cases receive a weight of 1, while all control cases are weighted on
where 𝑃i(𝑋) is the PS of the control case, thereby magnifying untreated cases that resemble more
closely the treated cases (for more on IPTW, see Reynolds and DesJardins, 2010). Given the
significant influence of the first-year academic experience on student persistence in documented
research (Pascarella & Terenzini, 2005; Astin, 1993), the impact of loan aid is estimated with
and without a set of first-year experience variables. Accordingly, the covariate adjusted model
that regresses the outcome (persistence) on the PS and loan status also controls for whether or
not a student took math or English, or enrolled in an online distance education course; used any
of the on-campus tutoring centers; visited the on-campus student diversity center; worked on
campus (all coded as dummy variables); and academic success. The latter is measured with a
100-point momentum index composed equally of first-semester and first-year cumulative grades
(GPA) and number of course credits completed. The use of academic momentum indices to
gauge student success goes back to the work of Adelman (2006), and academic momentum
based on grades and credits of first-year students is highly predictive of degree completion
beyond pre-college preparation and socio-demographic background (Attewell, Heil, & Reisel,
2012). These variables likely mitigate bias in the estimated outcome, as found in other PS-based
studies that included post-treatment measures (Austin, 2011; Titus, 2007). This method assumes
that the covariance between the PS and the outcome is correctly modeled, and that variables
exhibit tolerable collinearity. The variance inflation factor (VIF) for the combined set of preand post-treatment variables remained well below 2.5, the recommended upper limit (Allison,
Descriptive Data and Balance Verification. Table 1 shows that students with an EFC level
between $5,200 and $20,000 are most likely to rely on loans, both subsidized and unsubsidized.
These students are in the bottom half of those that do not qualify for the federal Pell Grant. The
upper half, those with an EFC above $20,000, are less likely to rely on loans, especially
subsidized loans for which few would qualify. Students with a low EFC level—i.e., those
eligible for the Pell Grant—are slightly less likely to rely on loans compared to mid-level EFC
students, but more likely to have loans compared to high-level EFC students. Looking at the
academic momentum of these students, Table 2 suggests that students on unsubsidized loans
barely differ in their academic success from those with only subsidized loans, regardless of the
level of EFC. Since unsubsidized loans incur an immediate payback obligation in contrast to
subsidized loans, one may expect students carrying unsubsidized loans to be at an academic
Tables 3 and 4 indicate that students with no remaining need as well as students with high
remaining need are mostly from high EFC background. Expectedly, students with high
remaining need are less likely to have a low EFC level, which allows students to qualify for more
need-based aid such as the Pell Grant. Still, over a fifth of students with high remaining need
(i.e., over $ 10,000) are Pell-eligible. These students make up 27 percent of all Pell-eligible
students (Table 5). In contrast, students with a high EFC level (i.e., over $ 20,000) are twice as
likely to have high remaining need compared to Pell-eligible students (57 percent, Table 6).
While these distributions convey some sense of the financial burden faced by freshmen, they do
not reveal how reliance on loans may impact enrollment persistence given a student’s level of
ability to pay for college.
Before proceeding with an interpretation of the results from the statistical regression, one
must verify that the resampled data from PS matching resulted in suitable control groups to
establish a counterfactual analytical framework. Figure 1 depicts the PS distribution for the
treated (loan aid) and control (no loan aid) groups before and after matching (using full subclassification). The histograms before matching (left side) are distinctly different in shape,
showing a sharp decline in the number of students with no loans (raw control) that exhibit a high
propensity for loan selection; conversely, students with loans (raw treated) are mostly clustered
at the high end of the propensity scale, as one would expect. In contrast, the histograms after
matching (right side) look very similar, confirming that no-loan students in the control group
largely resemble students with loan aid across the PS distribution. Figure 2 shows the
standardized difference in mean for each covariate used in the PS estimation before (left) and
after (right) matching. With the exception of two covariates (dark lines), the standardized
difference for all covariates dropped after matching (right), leaving all covariates well within +/0.25, a threshold range recommended for reliable regression adjustment (Stuart, 2010; Rubin,
Figure 3 illustrates a great improvement in the PS distribution between students with
unsubsidized loans and those without such loans, rendering both groups highly comparable based
on the covariates included. Similar improvements in the PS and covariate balance occurred after
applying nearest neighbor matching (Figures 4 and 5). Table 7 summarizes the results from the
PS and covariate balance checks. Both overall balance as well as balance for covariates that are
strongly related to the loan selection process (EFC, remaining need, aid received, out of state
tuition, and living on campus) are well within the recommended threshold range.
Results. Descriptive data in Table 8 indicate that students with loans are less likely to persist
than students without loans. The average seven percentage point difference grows to 10
percentage points when the comparison is limited to students with high remaining need,
suggesting that high remaining need in conjunction with loan aid may put those students at an
added disadvantage. Without considering other factors, however, one may scarcely conclude
that loans are a key determinant in student persistence, much less a cause that explains the lower
rate for students with loans.
The influence of loan aid in the presence of all the factors used to estimate the PS are listed in
Table 9 under the ‘Null Model’ for all students and broken out by EFC and remaining need level.
Estimation of the influence of loan aid after controlling for first-year experience are listed under
the ‘Post-treatment Model’. The estimated impact of unsubsidized loan aid, net of subsidized
loan aid, is furnished in Table 10. Standard errors for nearest neighbor matched samples
(NNR:1) are based on Lechner’s (1999) variance approximation:
∑ 𝑖{𝐷 = 0}(𝑊𝑖 )2
𝑉𝑎𝑟(𝑌(1)|𝐷 = 1) +
∗ 𝑉𝑎𝑟(𝑌(0)|𝐷 = 0)
where N1 is the number of matched treated cases and Wi is the number of repeat uses of a control
case when matching with replacement in order to account for potential bias in bootstrapped
standard errors introduced with multiple use of the same case (Caliendo and Kopeinig, 2008).
Results from the logit coefficients show that, on average, loan aid exerts a negative influence
on persistence both before and after accounting for first-year experience (Table 9). However,
that effect reaches statistical significance only with PS-weighted (IPTW) and matched sample
data (NNR:1). Standard regression fails to detect a significant effect after accounting for firstyear experience. Separating the analysis by EFC and remaining need level, the only significant
effects net of first-year experience emerge from data using full sub-classification matching.
Accordingly, students with a low EFC level (i.e., Pell eligible) and students with no remaining
need are estimated to be negatively affected by loan aid. In either case, standard regression using
unmatched data fails to show a significant effect. Fewer significant effects are associated with
unsubsidized loans, which seem to matter only when ignoring what students experience during
the first year (Table 10). Students with high remaining need and those with a high EFC appear
to be negatively impacted by unsubsidized loans, but only prior to factoring in their first-year
Using the significant coefficients from matched data based on full sub-classification, Table 11
translates the estimated effect on enrollment persistence as the percentage change associated with
loan aid and unsubsidized loan aid vis-à-vis the baseline reference category, students without
loan aid or unsubsidized loan aid, applying Cruce’s (2009) corrected Delta-p statistic. A
comparison of results between standard regression and covariate adjusted regression using PS
matching suggests the former would fail to detect the negative effect of loan aid on persistence
for students with low EFC and those with no remaining need. For students with high need, the
effect size of unsubsidized loan aid prior to factoring in first-year experience is estimated to be
lower with matched data compared to standard regression. Data for the PS-matched samples that
show significant loan effects are well balanced (i.e., within the threshold range of +/- 0.25
standardized mean difference) between students with loan aid and those without loan aid across
all key covariates that strongly predict loan selection, as listed in Table 12.
Discussion of Findings
Results from the series of regression models that separate students by EFC and remaining need
levels produce several key findings. Loans exert a negative influence on enrollment persistence
of Pell-eligible students, namely those with an EFC no larger than $5,200, as determined by the
FAFSA data. Loans also seem to negatively affect persistence of students with no remaining
financial need after factoring in their EFC and all aid received, inclusive of loans. In contrast,
there is no significant incremental effect associated with unsubsidized loans after factoring in
whether a student took out subsidized loans. These estimates take into account all the factors
used in the calculation of a student’s propensity to select loan aid as well as their first-year
college experience, including their GPA and number of completed course credits. Moreover, the
results control for a range of socio-demographic, pre-college, academic motivation, and campus
integration factors deemed important in college persistence research (Pascarella & Terenzini,
2005; Astin, 1993).
Gauging the effect of loans both before and after controlling for the first-year experience
reveals that the impact of loan aid is significantly diminished by the college experience for
students with high remaining need (Table 9), even if they carry unsubsidized loans (Table 10).
The same finding applies to high-EFC students with unsubsidized loans. Controlling for receipt
of subsidized loans, unsubsidized loans fail to exert a negative influence on students with low
EFC (Pell eligible) and those with no remaining need, regardless of a student’s first-year
experience (Table 10). These differences in results suggest that average estimated effects mask
both the magnitude and nature of the influence of loan aid on students from different income
background and with varied levels of financial need. This supports the conclusion by Welbeck et
al. (2014, pp. 16, 18) in their review of the research on aid that level of remaining need and
student ability to afford cost of attendance are key factors in gauging the influence of aid on
student success.
In their review of the research since 1991, Hossler et al. (2009, p. 394) identified 32 “high
quality” studies on the impact of financial aid, but few (if any) of these examined aid taking into
account both EFC and remaining need. Similarly, only a few of the covered studies offer
evidence of how campus social integration, student motivation, and academic success interact
with financial aid. Lack of statistical control over these factors also limits the finding in a more
recent study linking receipt of need-based aid (including subsidized loans) to higher student
dropout (Gross, Hossler, Ziskin, and Berry, 2015).
In addition to the finding that the effect of loans varies with ability to pay (EFC) and
remaining need, the results here add to the already substantial evidence in the cumulative
research that academic performance is the key predictor of enrollment persistence for most
students (Pascarella & Terenzini, 2005). Moreover, the findings here corroborate those from a
previous study on students at the same institution (Herzog, 2008), showing that academic success
in the first year exerts a greater effect on persistence of low-income students compared to highincome students net of all other factors (including financial aid, socio-demographics, motivation,
and educational goal). As Table 13 shows, first-year academic momentum is a far more
significant covariate for Pell-eligible students compared to students with a high EFC, a
difference consistent across all tested regression models.
The analytical framework in this study also addresses the challenge of selection bias
associated with observational data and tests the counterfactual hypothesis, what would be the rate
of persistence of students with loan aid had they not selected loans to finance college? As listed
in Table 9, there is a significant effect associated with loans net of first-year experience for Pell10 | P a g e
eligible students with low EFC and students whose total aid package covers the cost of college.
Had these students not relied on loan aid—most of it subsidized—their estimated persistence rate
would have been 4 percentage points and 7 percentage points higher, respectively, as shown in
Table 11. These negative effects would not have been detected with single-stage standard
regression, nor would it yield a result that gauges the effect size associated with the
counterfactual for students that actually relied on loan aid. Being able to answer the ‘what if’
question in the counterfactual framework yields an estimate of what would have happened to
students with loan aid had they not chosen that type of financing option. Moreover, results show
that, net of subsidized loans and first-year experience, unsubsidized loans exert no significant
influence on enrollment persistence, regardless of the student’s level of EFC or remaining need.
Therefore, the incremental effect associated with unsubsidized loans, beyond that which may
occur with subsidized loan aid, has a negligible impact on student persistence.
Whether the difference to standard regression can be attributed to reduced selection bias with
the PS-matched data depends on how well the observable characteristics (covariates) included in
the PS estimator capture the process of loan selection by students. Recent research confirms
that student loan selection is largely governed by both perceived and actual credit constraints,
which derive from the combination of income background, cost of attendance, and willingness
(or aversion) to take on debt (Ziskin et al., 2014; Cadena and Keys, 2013; Avery and Turner,
2012). Ziskin et al. (2014) document that loan aid is indeed a ‘last resort’ financing option that
students typically rely on after maximizing all other aid options. The sequence of offered,
accepted, and self-selected aid is well captured through the FAFSA data. Results from the PS
estimation show that the FAFSA-derived metrics for EFC and remaining need in combination
with indicators for variation in cost of attendance (i.e., residency status, on-campus living cost)
together emerged as the most significant predictors of loan aid (Wald > 30) . The balance of
these covariates and the PS in the matched group data is well within the threshold range for
reliable regression adjustment (Stuart, 2010; see Table 12).
Thus, the selection of covariates for PS estimation and the degree of balance in the matched
data together render the conditional independence assumption (i.e., a strongly ignorable
treatment assignment) plausible, if not highly likely, assuming the absence of significant omitted
variables that are related to loan selection and persistence, and that are uncorrelated with those
included here. Under that assumption, balance in the chosen covariates to estimate the PS are
sufficient to render the likelihood of loan selection uncorrelated with enrollment persistence for
students with no loan aid. Therefore, one may assume that results from the corresponding
single-stage standard regression with unmatched data likely mask the significant negative effect
associated with loan aid for Pell-eligible students and students with no remaining need detected
by the two-stage PS-adjusted regression. Also, these estimated effects draw on counterfactuals
that are matched with the entire sample of students with loans. None of the PS matching
methods forced deletion of treated cases (i.e., students with actual loan aid) from the analysis due
to lack of common support. As a result, the findings are more likely to hold in general for firstyear students with loans.
Do Student Loans Help or Hurt?
The answer to that question is it depends. For the majority of students in the two selected
cohorts (fall 2011 and fall 2012), loan aid—either in subsidized or unsubsidized form—appears
to exert no significant influence on enrollment persistence. This finding holds even after a
11 | P a g e
student’s academic performance in terms of grades and course credits completed is factored in.
However, when testing the counterfactual hypothesis with matched data that compare students
with no loan aid and highly similar disposition for selecting such aid to students that actually
carry loan aid, the findings indicate that Pell-eligible students and students with no remaining
need (i.e., whose aid package covers the cost of college) would persist at a higher rate without
loan aid than with loan aid.
Hossler et al. (2009), in their review of 32 highly relevant, high-quality studies produced since
1991, concluded that loans are more likely to have a negative effect on persistence if the metric
is based on a dichotomized indicator for loan aid. Examining studies that used national
databases (e.g., NPSAS), Hossler et al. (2009) report no effect associated with loan aid if
measured on a continuous scale (i.e., amount received) once cost of attendance and living are
included in the estimation. However, the studies from which these findings are drawn did not
attempt to control for selection bias associated with receipt of loan aid. The one study covered in
the review by Hossler et al. that did address selection bias (with an instrumental variable probit
model) found a negative effect of loan aid on a student’s chance of graduation (Alon, 2005). The
expansion of state-based and other large-scale grant programs to promote college enrollment of
low-income high school graduates has prompted a growing number of evaluation studies that
incorporate mechanisms to address selection bias, but none of these studies isolate the effect of
loan aid (Angrist, Autor, Hudson, and Pallais, 2014; Welbeck et al., 2014).
The finding that the negative impact of loan aid is limited to low-income students and
students who receive enough aid to end up with no remaining need offers evidence of the
importance to apply an analytical framework in the study of aid that differentiates by ability to
pay, as reflected in the EFC, and by the amount of unmet need net of all aid received. The
evidence here partly corroborates results from a previous study on students at the same
institution that shows a negative influence of loan aid on first-year student persistence (Herzog,
2005). However, that study did not gauge the effect of loans separately by EFC level, nor did it
explore the interaction effect between loan status and a student’s ability to pay (as reflected in
the EFC amount), nor did it address selection bias in the statistical estimator. Though a direct
comparison between results in that study and this study is of limited value, given the difference
in covariate controls, it is reasonable to assume that an analytical approach focused on selection
bias and testing of the counterfactual hypothesis likely yields a more accurate estimate of the
causal effect of loan aid.
Like the negative effect of loan aid on graduation that Alon (2005) found after separating the
effect of aid eligibility from the impact of aid received, the negative effect of loan aid detected
here emerged only after addressing selection bias with PS-matched data. Accordingly, had Pelleligible students with loan aid not selected such aid, their persistence would have been 4
percentage points higher; the persistence rate of those with no remaining need would have been 7
percentage points higher (Table 11). Given the proportion of Pell-eligible students in the cohorts
examined, the overall freshmen persistence rate at the institution would increase by one
percentage point had the Pell-eligible not relied on loans; a similar increase in overall freshmen
persistence would result if students with no remaining need would have refrained from loans.
Such estimates assume that the financial need covered by loans could be addressed with other
sources of aid (or cost discounts) that do not exert a negative effect on persistence. Johnson
(2013) found that for academically average students augmenting tuition subsidies (e.g. with
grants or discounts) would mitigate the dropout risk and increase degree completion. Both Gross
et al. (2015) and DesJardins, Ahlburg, and McCall (2002) concluded that greater reliance on
12 | P a g e
loans as part of the total aid received elevated the dropout risk for students. Consistent with the
finding in this study, Kim (2007), using national data from the Beginning Postsecondary Student
(BPS) survey, produced evidence that loan aid during the first-year lowered degree completion
odds for low-income students. In contrast, Marx and Turner (2015) observed mixed effects on
degree attainment after replacing loans with Pell Grant aid. Since the results here are limited to
first-year students, the observed negative effect of loan aid may not endure as students in the
examined cohorts academically progress. Those advancing are on average academically more
successful and may feel more comfortable with rising loan debt.
As Table 14 shows, loans constitute about the same proportion of total aid received for
students with no remaining need as it does for those with remaining need of up to $10,000.
However, the former receive on average loans that are 60 percent larger than the latter group, or
more than double the size of loans taken up by students with high remaining need (i.e. over
$10,000). The higher loan amounts for students with no remaining need is likely associated with
the higher cost of on-campus living—on average they are 35 percent more likely to live on
campus—and echoes the finding in Cadena and Keys (2013) that on-campus students are more
likely to take up loans. Thus, these students are more likely to end up with no outstanding need
as loan aid is directly applied to on-campus room and board expenses. This pattern is consistent
across the different student affordability levels (EFC ranges) and shows that the confluence of
low affordability (Pell eligible) and loan amount taxes enrollment persistence. Low-EFC
students with no remaining need take up loans more than twice as large as Low-EFC students
with high remaining need (over $10,000, see Table 14).
The substantial difference in average loan amount across remaining need levels combined
with the finding that unsubsidized loans do not elevate the dropout risk (net of subsidized loans)
suggests that the level of debt may exert a negative psychological effect separate from the
economic effect associated with an immediate financial payback obligation and accumulating
interest payments. If so, it may corroborate results in Cadena and Keys (2013) that self-control
among some students leads to debt-aversion and thus lower loan take-up rates, at least for
students with alternate funding sources or those seeking to avoid excessive consumption while in
college. The data also show that Low-EFC students who are academically more challenged are
least likely to live on campus and hence incur the lowest loan debt in the first year (see Table
14). However, the low academic success and persistence of these students suggests that other
factors are at work not captured here. While the estimation model controls for on-campus
employment and takes into account student plans to work while in college, data on actual offcampus work hours was not available. Since low-income students with credit constraints are
more likely to work longer hours while in college (Soria, Weiner, and Lu, 2014), which may
heighten their dropout risk (Mendoza, 2012), it is plausible that Low-EFC students with high
remaining need eschew larger loan debt in favor of off-campus employment while in college,
thus engaging in a trade-off that seemingly compromises their academic success and progress
just the same.
The estimated effect of loan aid on student persistence varies with ability to pay for college as
well as with the amount of loan aid and associated remaining financial need. Accordingly, Pell
Grant-eligible students with loans and students who take-up higher amounts of loan aid to meet
college costs are at an elevated risk of departure after the first year of college. Unsubsidized
13 | P a g e
loans exert no significant impact on enrollment persistence beyond the estimated effect of
interest-free subsidized government loans. The observed significant effects emerged only after
addressing selection bias associated with loan status. Thus, this study confirms the importance
of taking into account student discretion in selecting loan aid when gauging the impact of this
type of financing on academic outcomes. Having chosen a counterfactual analytical framework
to estimate the effect of student loans, this study seeks to respond to the call for more rigorous
research to understand the role of financial aid in college success (Hossler et al., 2009).
Though the findings here are drawn from data at one institution, with inference necessarily
limited to that campus, the insight gained by Gross et al. (2015, p. 244) from the large-scale
NPSAS data shows that “student departure is strongly influenced by the unique experiences of
students on individual campuses,” prompting the authors to encourage more research in this area
at individual institutions. At the same time, the results here are based on the experience of firstyear students and may not be extrapolated to students progressing to the second year and beyond.
Moreover, the estimated effects may still be biased to the extent that unobserved factors
governing student selection of loan aid are uncorrelated with those included in the selection
estimation, as well as possibly significant omitted factors related to first-year student persistence.
The finding that loan aid for some students reduces the institution’s overall freshmen retention
rate by at least one percent may offer the institution the opportunity to tweak its student aid
policy in order to move the needle on a ‘key performance indicator’ widely used in assessment,
program review, accreditation, and college survey rankings. Results indicate that estimation of
the effect of loan aid on student outcomes can tangibly support policy decisions at the senior
management level if the analysis accounts for student ability to pay (EFC) and the remaining
financial need a student faces after considering all aid received. Since the observed negative
effect of loan aid is limited to students least likely to afford college (Pell-eligible) and those
relying on large amounts of loans to meet college expenses, the institution should explore
alternate funding or discount options for these students. Such effort should also consider
enhanced academic support (e.g. tutoring and counseling), given that academic performance has
a greater impact on the persistence of low-income students compared to students less challenged
to pay for college.
14 | P a g e
Adelman, C. (2004). Principal Indicators of Student Academic Histories in Postsecondary Education, 1972-2000.
Washington, DC: U.S. Department of Education.
Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through college.
Washington, DC: U.S. Department of Education.
Adelman, C. (2007). Do we really have a college access problem? Change (July-August): 48-51.
Allison, P. (2012). When can you safely ignore multicollinearity? Statistical Horizons. Retrieved August 8, 2014, at
Alon, S. (2005). Model mis-specification in assessing the impact of financial aid on academic outcomes. Research
in Higher Education, 46 (1), 109-125.
Angrist, J. D. (2003). Randomized trials and quasi-experiments in education research. NBER Reporter, No.4.
National Bureau of Economic Research.
Angrist, J., Autor, D., Hudson, S., & Pallais, A. (2014, December). Leveling up: Early results from a randomized
evaluation of post-secondary aid. (NBER Working Paper 20800). Cambridge, MA: National Bureau of Economic
Astin, A. W. (1993). What Matters in College? Four Critical Years Revisited. San Francisco: Jossey-Bass.
Attewell, P., Heil, S., & Reisel, L. (2012). What is academic momentum? And does it matter? Educational
Evaluation and Policy Analysis, 34, 27-44
Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in
observational studies. Multivariate Behavioral Research, 46, 399-424.
Avery, C., & Turner, S. (2012). Student loans: Do college students borrow too much—or not enough? The Journal
of Economic Perspectives, 26 (1), 165-192.
Bettinger, E. P., Long, B. T., Oreopoulos, P., & Sanbonmatsu, L. (2009, September). The role of simplification and
information in college decisions: Results from the H&R Block FAFSA experiment. (NBER Working Paper
15361). Cambridge, MA: National Bureau of Economic Research.
Booij, A. S., Leuven, E. & Oosterbeek. (2011). The role of information in the take-up of student loans. Economics of
Education Review, 31, 33-44.
Bowman, N. A., & Herzog, S. (eds.) (2014). Methodological Advances and Issues in Studying College Impact. New
Directions for Institutional Research. no. 161, San Francisco: Jossey-Bass.
Brown, M., Haughwout, A., Lee, D., Scally, J., & van der Klaaw, W. (2014, April). Measuring student debt and its
performance (Staff Report No. 668). New York, NY: Federal Reserve Bank of New York.
Cadena, B. C., & Keys, B. J. (2013). Can self-control explain avoiding free money? Evidence from interest-free
student loans. The Review of Economics and Statistics, 95 (4), 1117-1129.
Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score matching.
Journal of Economic Surveys, 22, 31-72.
15 | P a g e
Chen, J., & Zerquera, D. (2011). “A methodological review of studies on effects of financial aid on college student
success. “ Paper prepared for the 36th annual conference of the Association for Education Finance and Policy.
Cho, S. H., Xu, Y., & Kiss, D. E. (2015). Understanding student loan decisions: A literature review. Family and
Consumer Sciences Research Journal, 43, 229-243.
Chudry, F., Foxall, G., & Pallister, J. (2011). Exploring attitudes and predicting intentions: Profiling student debtors
using an extended theory of planned behavior. Journal of Applied Social Psychology, 41, 119-149.
Cruce, T. M. (2009). A note on the calculation and interpretation of the Delta-p statistic for categorical independent
variables. Research in Higher Education, 50, 608-622.
DesJardins, S. L., Ahlburg, D. A., & McCall, B. P. (2002). Simulating the longitudinal effects of changes in
financial aid on student departure from college. Journal of Human Resources, 37, 653-679.
Elliot, W., & Nam, I. (2013). Reducing student loan debt through parents’ college savings (Vol. 49, pp. 265-286).
St. Louis, MO, Center for Social Development at Washington University.
Gross, J. P., Hossler, D., Ziskin, M., & Berry, M. S. (2015). Institutional merit-based aid and student departure: A
longitudinal analysis. The Review of Higher Education, 38, 221-250.
Hansen, B. B. (2004). Full matching in an observational study of coaching for the SAT. Journal of the American
Statistical Association, 99, 609-618.
Heckman, J. J. (2000). Causal parameters and policy analysis in economics: A twentieth century retrospective. The
Quarterly Journal of Economics, 115, 45-97.
Herzog, S. (2005). Measuring determinants of student return vs. dropout/stopout vs. transfer: A first-to-second year
analysis of new freshmen. Research in Higher Education, 46 (8), 883-928.
Herzog, S. (2008, January). Estimating the influence of financial aid on student retention (Education Working Paper
Archive). Fayetteville, AR: University of Arkansas, Department of Education Reform.
Herzog, S. (2014). The Propensity Score Analytical Framework: An Overview and Institutional Research Example.
In. N.A. Bowman and S. Herzog (eds.) Methodological Advances and Issues in Studying College Impact. New
Directions for Institutional Research. no. 161, San Francisco: Jossey-Bass.
Ho, D. E., Imai, K., King, G., & Stuart, E. (2011). MatchIt: Nonparametric preprocessing for parametric causal
inference. Journal of Statistical Software, 42 (8), 1-28.
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945-960.
Hossler, D., Ziskin, M., Gross, J., Kim S., & Cekic, O. (2009). Student aid and its role in encouraging persistence. In
J.C. Smart (ed.) Higher Education: Handbook of Theory and Research. (pp. 389-325). New York: Springer.
Johnson, M. T. (2013). Borrowing constraints, college enrollment, and delayed entry. Journal of Labor Economics,
31, 669-725.
Kim, D. (2007). The effect of loans on students’ degree attainment: Differences by student and institutional
characteristics. Harvard Educational Review, 77, 64-100.
Lechner, M. (1999, December). Identification and estimation of causal effects of multiple treatments under the
conditional independence assumption (Discussion Paper No. 91). Bonn, Germany: Institute for the Study of Labor
16 | P a g e
Marx, B. M., & Turner, L. J. (2015, January). Borrowing trouble? Student loans, the cost of borrowing, and
implications for the effectiveness of need-based grant aid. (NBER Working Paper 20850). Cambridge, MA:
National Bureau of Economic Research.
Mendoza, P. (2012). Should I work or should I borrow? A counterfactual analysis on the effect of working while in
enrolled and debt on baccalaureate completion. Journal of Student Financial Aid, 42, 25-59.
Morgan, S. L., & Winship, C. (2007). Counterfactuals and causal inference: Methods and principles for social
research. New York: Cambridge University Press.
Murnane, R. J., & Willett, J. B. (2010). Methods matter: Improving causal inference in educational and social
science research. Oxford University Press.
Pascarella, E. T., & Terenzini, P. T. (2005). How College Affects Students: A Third Decade of Research. San
Francisco, CA: Jossey-Bass.
Pike, G. R., Hansen, M. J., & Lin, C. (2011). Using instrument variables to account for selection effects in research
on first-year programs. Research in Higher Education, 52, 194-214.
Reynolds, C. L., & DesJardins, S. L. (2010). The use of matching methods in higher education research. In J. C.
Smart (Ed.), Higher Education: Handbook of Theory and Research, Volume XXIV (pp. 47-104). New York:
Rosenbaum, P. R. (2002). Observational Studies. New York: Springer.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal
effects. Biometrika, 70, 41-55.
Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the
propensity score. Journal of the American Statistical Association, 79, 516-524.
Rubin, D. B. (1980). Bias reduction using Mahalanobis metric matching. Biometrics, 36, 293-298.
Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco
litigation. Health Services & Outcomes Research Methodology, 2, 169-188.
Rubin, D. B. (2006). Matched Sampling for Causal Inference. New York: Cambridge University Press.
Sekhon, J. S. (2011). Multivariate and propensity score matching software with automated balance optimization:
The Matching package for R. Journal of Statistical Software, 42 (7), 1-22.
Soria, K. M., Weiner, B., & Lu, E. C. (2014). Financial decisions among undergraduate students from low-income
and working-class social backgrounds. Journal of Student Financial Aid, 44, 2-23.
Stuart, E. A. (2010). Matching methods for causal inference: A review and look forward. Statistical Science, 25, 121.
Stuart, E. A., & Rubin, D. B. (2008). Best practices in quasi-experimental designs. In J. W. Osborne (Ed.), Best
Practices in Quantitative Methods (pp. 155-176). Thousand Oaks, CA: SAGE.
Titus, M. A. (2007). Detecting selection bias, using propensity score matching, and estimating treatment effects.
Research in Higher Education, 48, 487-521.
U.S. Department of Education. (2003). Identifying and implementing educational practices supported by rigorous
evidence: A user friendly guide. Washington, DC: Institute of Education Sciences.
17 | P a g e
Welbeck, R., Diamond, J., Mayer, A., & Richburg-Hayes, L. (2014). Piecing together the college affordability
puzzle, MDRC Report, New York
Ziskin, M., Fischer, M. A., Torres, V., Pellicciotti, B., & Player-Sanders, J. (2014). Working students’ perceptions
of paying for college: Understanding the connection between financial aid and work. The Review of Higher
Education, 37 (4), 429-467.
18 | P a g e
Table 1: Students with Loans by EFC Level
Any Loans
Low (Pell eligible), N=1386
Mid ($5.2K to $20K), N=1171
High (>$20K), N=1173
Total, N=3730
Table 2: Academic Momentum (GPA-Credits) of Students
Low (Pell eligible), N=1386
Mid ($5.2K to $20K), N=1171
High (>$20K), N=1173
Any Loans
Total, N=3730
Acad Momentum: Mean = 81.2; Std Dev = 20.7
Table 3: EFC Level of Students with No Remaining Need
low (Pell eligible)
mid ($5.2K to $20K)
high (>$20K)
Table 4: EFC Level of Students with High Remaining Need
low (Pell eligible)
mid ($5.2K to $20K)
high (>$20K)
19 | P a g e
Table 5: Remaining Need of Students with Low EFC (Pell eligible)
< $10K
> $10K
Table 6: Remaining Need of Students with High EFC (> $20K)
< $10K
> $10K
Figure 1: Propensity Score Balance with Full Sub-Classification (Loans)
20 | P a g e
Figure 2: Covariate Balance with Full Sub-Classification (Loans)
Figure 3: Propensity Score Balance with Full Sub-Classification (Unsub)
21 | P a g e
Figure 4: Covariate Balance with Nearest Neighbor (Loans)
Figure 5: Covariate Balance with Nearest Neighbor (Unsub)
22 | P a g e
Table 7: Covariate and Propensity Score Balance for Matched Data
Std. Difference in Mean* for:
All covariates
Balance Improvement %
Out of state, living in dorms
Balance Improvement %
EFC, Need, Pell, Merit aid, Sub loans^
Balance Improvement %
Work plans, Parent education
Balance Improvement %
Propensity score
Balance Improvement %
Key Covariates with SD > 0.25
Key Covariates with SD > 0.1
All Loans
Full SubCl
Unsubsidized Loans
Full SubCl
*Formula denominator uses standard deviation of treated, not pooled, unlike Cohen's d
^Unsubsidized loan models only
Table 8: Fall-to-Fall Persistence and Naïve Estimator
All students
EFC Low (<$5.2K, Pell eligible)
(N= 704, 682)
EFC High (>$20K)
(N= 507, 666)
Remaining Need Zero ($ 0)
(N= 416, 335)
Remaining Need High (> $10K)
(N= 666, 1004)
E(Y1) Loans
E(Y0) No Loans
E(Y1) - E(Y0)
23 | P a g e
Table 9: Estimation of Loan Effect on Second-Year Persistence
Treated (N = all students)
Untreated (N = all students)
All students: Null Model
Logit Regr
Full Sub-Class
EFC Low: Post-Treat Model
EFC High: Null Model
Zero Need: Null Model
Zero Need: Post-Treat Model
All students: Post-Treat Model
EFC Low: Null Model
EFC High: Post-Treat Model
High Need: Null Model
High Need: Post-Treat Model
Alpha significance: * = 0.10, ** = 0.05, *** = 0.01. Standard errors in italics, bootstrapped for IPTW and Full Sub-Class
24 | P a g e
Table 10: Estimation of Unsubsidized Loan Effect on Second-Year Persistence
Treated (N = all students)
Untreated (N = all students)
All students: Null Model
Logit Regr
Full Sub-Class
EFC Low: Null Model
EFC Low: Post-Treat Model
EFC High: Null Model
EFC High: Post-Treat Model
All students: Post-Treat Model
Zero Need: Null Model
Zero Need: Post-Treat Model
High Need: Null Model
High Need: Post-Treat Model
Alpha significance: * = 0.10, ** = 0.05, *** = 0.01. ^ Genetic algorithm. ~~ Algorithm failed to reach acceptable data balance
Standard errors in italics, bootstrapped for IPTW and Full Sub-Class
Table 11: Effect of Loan and Unsubsidized Loan Aid on Enrollment Persistence
All Loans
Steps (Cruce formula)
Formula (FullEFClow Ex)
1) Parameter Est of baseline (log)
Unsubsidized Only
Ln(0.78 / 1 - 0.78)
2) Parameter Est of cent ref gr (log)
1.27+(-.255 * -.51)
3) Parameter Est of cent tar gr (log)
1.27+(-.255 * .49)
4) Probability (%) of reference group
Exp(1.40) / (1 + Exp 1.40)
5) Probability (%) of target group
Exp(1.15) / (1 + Exp 1.15)
6) Change (%) due to target group
Step 5 minus step 4
Naïve Estimator E(Y1) - E(Y0)
Standard Logit Regression
25 | P a g e
Table 12: Covariate and PS Balance for Models with Significant Loan Coefficient
Std. Difference in Mean* for:
All covariates
Out of state, living in dorms
EFC, Need, Pell, Merit aid, Sub loans^
Work plans, Parent education
Propensity score
Key Covariates with SD > 0.25
Key Covariates with SD > 0.1
All Loans
Full SubCl
Full SubCl
Zero Need
Unsubsidized Loans
Full SubCl
EFC High
High Need
*Formula denominator uses standard deviation of treated, not pooled, unlike Cohen's d
^Unsubsidized loan models only
Table 13: Statistical Significance of Academic Momentum (Wald Statistic)
EFC High (>
EFC Low (Pell
Full Sub
* Loan coefficient is significant
26 | P a g e
Table 14: Selected Statistics for EFC Level by Remaining Need Level
Aid from
Live On
Low EFC (Pell eligible)
Mid EFC ($5.2K to $20K)
High EFC (>$20K)
Total (N = 416)
Low EFC (Pell eligible)
Mid EFC ($5.2K to $20K)
High EFC (>$20K)
Total (N = 785)
Low EFC (Pell eligible)
Mid EFC ($5.2K to $20K)
High EFC (>$20K)
Total (N = 666)
No Remaining Need
Preparation Momentum Momentum Persistence Peristence
Remaining Need up to $10K
Remaining Need over $10K
27 | P a g e
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF