Authors: Jessa L. Valentine & Derek V. Price, DVP-PRAXIS LTD
This brief provides final evaluation results from the Carolina Works initiative, which aimed to improve retention and completion outcomes for students at 10 North Carolina community colleges through the delivery of proactive, technology-mediated student success coaching. Key findings include:
Almost 6 million students enroll in community colleges each Fall, and only 40% earn a postsecondary credential within six years. Technology-mediated success coaching is one strategy being implemented by colleges across the country in an effort to more effectively support their students and boost completion rates. By providing services that are personalized, proactive, and informed by real-time data on students’ academic and non-academic progress and challenges, success coaching has the potential to greatly improve postsecondary outcomes.
This brief presents final evaluation results from the Carolina Works initiative, which aimed to improve retention and completion outcomes for students at 10 North Carolina community colleges through the provision of proactive, technology-mediated student success coaching. Findings from a mixed-methods, independent evaluation of the initiative led by DVP-PRAXIS LTD point to the promise of success coaching for improving community college student outcomes and highlight key implementation factors that can increase the effectiveness of this student success strategy.
Success coaching is part of a larger movement by colleges to take a more holistic, personalized, and proactive approach to supporting the academic and non-academic needs of students. College students overall, and especially at community colleges, are increasingly enrolled part-time; often face challenges of poverty, food insecurity, or homelessness; and balance their education with work, parenting, and other family obligations.
Success coaches develop personal relationships with students, serving as a main point of contact as well as a connector to other key supports and resources at the college and beyond. In addition to providing direct support to students, coaches refer students to other college personnel and resources and follow up with students to help with any next steps. At each of the 10 Carolina Works community colleges, success coaches use a predictive analytics and case management software called Aviso Retention to monitor student grades, attendance, and other important information provided in real-time to help them target proactive outreach to students when they need it most. First-hand feedback from students in the Carolina Works initiative underscores the importance of a relationship of trust developed between students and coaches, which helps students engage in difficult conversations about academic or personal challenges and which serves as a foundation for other services that coaches can provide. Success coaching is part of a larger movement by colleges to take a more holistic, personalized, and proactive approach to supporting the academic and non-academic needs of students. College students overall, and especially at community colleges, are increasingly enrolled part-time; often face challenges of poverty, food insecurity, or homelessness; and balance their education with work, parenting, and other family obligations. Success coaches develop personal relationships with students, serving as a main point of contact as well as a connector to other key supports and resources at the college and beyond. In addition to providing direct support to students, coaches refer students to other college personnel and resources and follow up with students to help with any next steps. At each of the 10 Carolina Works community colleges, success coaches use a predictive analytics and case management software called Aviso Retention to monitor student grades, attendance, and other important information provided in real-time to help them target proactive outreach to students when they need it most. First-hand feedback from students in the Carolina Works initiative underscores the importance of a relationship of trust developed between students and coaches, which helps students engage in difficult conversations about academic or personal challenges and which serves as a foundation for other services that coaches can provide.
Success Coaching Boosts Students’ Longer-Term Retention, with Larger Benefits for Male Students and Black Students
Study results indicate that success coaching is a promising strategy for improving community college students’ longer-term outcomes. Figure 1 presents retention and completion rates for new students entering the sample in the Fall 2016, Fall 2017, and Fall 2018 terms, whose outcomes are tracked for at least two full academic years. Although there are no detectable effects of coaching on students’ short-term retention, these effects grow notably larger for longer-term retention outcomes (Fall-Fall and Fall-to-second-Spring) — students with a coach are 1.7 percentage points more likely to be enrolled at the end of two academic years compared to students without a coach (44.2% v. 42.5%), representing a 4% increase in retention over the control group average. These results suggest that the benefits of coaching emerge over time, as coaches develop deeper relationships with students across several semesters. Students with a coach are more likely to have earned a credential during the study period, however this finding is small and not statistically significant.
The Carolina Works Initiative
The Carolina Works initiative, led by Central Carolina Community College, was one of two validation grants awarded in 2015 by the U.S. Department of Education’s First in the World program to test interventions for student success that are supported by previous evidence. Carolina Works aimed to increase student retention and credential completion across 10 North Carolina community colleges through the provision of proactive success coaching informed by Aviso Retention, a web-based early alert and advising system using predictive analytics.
Evaluation Design
In collaboration with project partners, DVP-PRAXIS LTD conducted an independent evaluation of the Carolina Works initiative comprised of two components: (1) a Randomized Controlled Trial (RCT) to assess causal impacts of the success coaching “treatment” on students’ retention and completion outcomes, and (2) an implementation study based on in-depth site visits to participating colleges to assess implementation fidelity related to the coaching model as well as institutional support for the success coaching intervention.
Evaluation Design
In collaboration with project partners, DVP-PRAXIS LTD conducted an independent evaluation of the Carolina Works initiative comprised of two components: (1) a Randomized Controlled Trial (RCT) to assess causal impacts of the success coaching “treatment” on students’ retention and completion outcomes, and (2) an implementation study based on in-depth site visits to participating colleges to assess implementation fidelity related to the coaching model as well as institutional support for the success coaching intervention.
Analytic Sample
The Carolina Works study included approximately 11,000 first-time fall students across 10 North Carolina community colleges. In this brief, we present results for students randomly assigned to the treatment or control group across three fall terms (fall 2016, fall 2017, fall 2018), with retention and completion outcomes tracked through Spring 2020, which is at least two academic years for all students in the sample. Approximately one-half of students were assigned a success coach (i.e. treatment group), and the other half were assigned to their college’s business-as-usual condition (i.e. control group). The random assignment process resulted in treatment and control groups of students that were statistically identical on key baseline student characteristics.
In addition to assessing impacts of coaching for all students, the evaluation examined outcomes for different groups of students to explore variation in impacts with potential equity implications. As shown in Figure 2, which displays the percentage point difference in outcomes between students with a coach and students without a coach, male students and Black students are especially likely to benefit from success coaching. These findings carry equity implications given that male students and Black students – both in this study sample and nationwide — have less favorable postsecondary outcomes on average.
Male first-time fall students assigned a success coach were 3.6 percentage points more likely to be enrolled in the subsequent fall compared to male students without a coach (50.8% v. 47.2%), representing an 8% increase in Fall-Fall retention over the control group average; male students with a coach were also significantly more likely to be enrolled after two academic years. In addition, male students with a coach were 2.0 percentage points more likely to complete a credential (28.2% v. 26.2%), representing an 8% increase in completion over the control group average. Greater impacts of coaching among male students is consistent with prior studies of coaching.
Among Black students, those assigned a success coach experienced a 3.0 percentage point boost in Fall-Fall retention (38.0% v. 35.0%), representing an 8% increase over the Black student control group average. In addition, Black students with a coach experienced a nearly 5 percentage point increase in Fall-to-second-Spring retention (32.8% v. 27.9%), representing an 18% increase over the control group average.
In addition to examining differences in outcomes by select student characteristics, the evaluation assessed the extent to which impacts of coaching vary based on key implementation factors theorized to impact effectiveness of the intervention. First, impacts of success coaching were assessed for students within 3 colleges that were flagged as high-performers according to a series of implementation fidelity metrics. Based on interviews and focus groups with college staff, faculty, and administrators, colleges were scored on six dimensions of model fidelity including three institutional ‘implementation drivers’ (support from campus leadership; campus-wide communication and engagement; technology adoption and use) and three ‘implementation core components’ (high-fidelity coaching practices; faculty and staff support of coaches; commitment to and awareness of the RCT design, i.e. restriction of coaching to treatment students only). The 3 colleges implementing the intervention with high fidelity demonstrated strong leadership support for success coaching, successfully folding coaches into their existing staff and building institution-wide buy-in through campus-wide communication and engagement. Second, impacts of success coaching were assessed within the subset of 6 colleges that had coaches in place for the duration of the five-year study. Many colleges experienced multiple instances of success coach turnover, theorized to have a negative impact on student-coach relationships, especially since these relationships can take time to develop.
Results for these subsamples of institutions suggests that implementation matters. The impact of coaching on student retention and completion is notably larger within institutions that implemented the intervention with high fidelity. As shown in Figure 3, success coaching within high-fidelity colleges resulted in a 4 percentage point increase in students’ Fall-to-second-Spring retention (48.3% v. 44.3%), representing a 9% increase over the control group average in these colleges. Students assigned to a coach also increased their credential completion by 2.8 percentage points (31.8% v. 29.0%), representing a 9% increase in completion over the control group average in these 3 colleges.
A similar pattern emerges when examining variation in the effects of success coaching according to coach tenure: namely, students benefit more from success coaching when their coaches don’t change. Within the 6 institutions with coaches in place for the entire study period, students assigned to these coaches experienced more than a 3 percentage point increase in Fall-Fall retention (52.0% v. 48.9%), Fall-to-second-Spring retention (45.8% v. 42.4%), and credential completion (29.8% v. 26.5%). Compared to the control group average in these colleges, this represents a 6% increase in Fall-Fall retention, an 8% increase in Fall-to-second-Spring retention, and – most notably – a 12% increase in credential completion.
All students should be supported to attend – and finish – college. Findings from the evaluation of Carolina Works suggest that success coaching is an effective strategy to help more students stay on their pathways and earn college credentials. Findings also point to the following key lessons for other institutions looking to adopt or scale success coaching within their institutions:
See how our tools are helping clients right now, get in-depth information on topics that matter, and stay up-to-date on trends in higher ed.
