This evaluation brief provides an update on the status of the Carolina Works initiative, which aims to improve retention and credential completion outcomes for students at 10 North Carolina community colleges through the provision of proactive, data-informed student success coaching. Preliminary results from Carolina Works indicate that:
Technology-mediated advising or coaching is an increasingly popular strategy being implemented by colleges and universities across the country in an effort to more effectively support their students. By providing services that are personalized, proactive, and informed by real-time data on students’ academic and non-academic progress and challenges, success coaching has the potential to greatly improve postsecondary attainment rates
This evaluation brief provides an update on the status of the Carolina Works initiative, which aims to improve retention and credential completion outcomes for students at 10 North Carolina community colleges through the provision of proactive, data-informed student success coaching. Early results from Carolina Works indicate that success coaching has a positive impact on students’ longer-term retention, and that key coaching practices may be particularly effective in improving these student outcomes.
The Carolina Works study- led by Central Carolina Community College- is one of two validation grants awarded in 2015 by the U.S. Department of Education’s First in the World program to test interventions for student success that are supported by previous evidence. The goal of the Carolina Works project is to increase student retention and credential completion across 10 North Carolina community colleges through the provision of proactive success coaching informed by Aviso Retention, a web-based early alert and advising system using predictive analytics.
The study design for the Carolina Works project is a Randomized Controlled Trial, or RCT. An RCT is an experimental study to assess causality between a “treatment” (some sort of intervention) and an outcome and is often referred to as the “gold standard” of research evidence. In an RCT, students in the study sample are randomly assigned to either receive the treatment (the “treatment group”) or not (the “control group). Proactive student success coaching is the treatment being tested at the 10 colleges participating in the Carolina Works project.
The Carolina Works study sample includes more than 14,000 students who were randomly assigned to the treatment or control group across five academic terms-fall 2016, spring 2017, fall 2017, spring 2018, and fall 2018. Approximately 7,000 students were assigned a success coach (i.e. treatment group) and 7 ,000 students were assigned to their college’s business-as-usual condition (i.e. control group). Baseline characteristics for the study sample show the random assignment process resulted in treatment and control groups of students that are statistically identical on key baseline characteristics. Students’ academic progress will be tracked through the summer of 2020.
Preliminary retention outcomes for the first cohort of students-those entering the sample in Fall 2016-are promising. At the time of writing, outcome data are available through the Spring 2018 academic term, and thus it is possible to examine 6-month (Fall 2016 to Spring 2017), 1-year (Fall 2016 to Fall 2017), and 18-month (Fall 2016 to Spring 2018) retention for this initial student cohort.
As shown in the figure below-which displays retention rates for treatment and control students who entered college in Fall 2016-the benefits of coaching are more prominent when assessing longer-term outcomes. Although one-semester and one-year retention rates are slightly higher for treatment versus control students, these differences are small and are not statistically significant.
More substantial benefits of coaching begin to emerge at 18 months- 39% of treatment students are enrolled 18 months later, compared to 36% of control students, a three-percentage point difference. While preliminary, these results suggest that the benefits of coaching emerge over time-with stronger effects on longer-term versus shorter-term retention outcomes. One reason the benefits of coaching may grow over time is that coaches develop deeper relationships with students across several semesters. The outcomes for the full sample will be reported at the conclusion of the grant in September 2020.
Randolph Community College was identified as a high-performing campus based on the evaluation team’s 2016 and 2017 site visits focused on implementation readiness and implementation fidelity, respectively, as well as on feedback from grant leadership; early retention outcomes for this college also outperform the other nine colleges in the consortium. Data collected from the March 2018 in-depth site visit suggest four aspects of the coaching model that may be especially important for improving student progress towards completion.
Interacting face-to-face with students. Success Coaches spend a majority of their time on any given day outside of their offices, tracking down students in classrooms and other common areas across campus. In doing so, their goal is to meet with all students on their caseloads face-to-face before issues arise, acknowledging that many students will not initially seek them out in response to emails, phone calls, or texts. Once the initial face-to-face connection occurs, the student is more likely to be responsive to less intrusive coach outreach.
Synthesizing data to build a holistic understanding of each student. Using data available in the Aviso system, coaches build a holistic picture of each student that helps them organize and plan their strategy for engagement. Coaches demonstrate strong analytic ability to take in information on prior and current student performance, course difficulty level, personal circumstances, and early alerts sent by faculty, using these data points to build a comprehensive understanding of students and their context before reaching out.
Advocating to empower. Success Coaches work to simplify the college for students, while also modeling and teaching processes and behaviors that the student should eventually be able to carry out themselves. While coaches aggressively reach out to their students initially, helping them address issues that affect performance (which may include advocating for them with faculty and staff), the goal is to teach them how to be successful students, which includes making good decisions and following the necessary processes to resolve issues like dropping courses, registering, getting off probation, and dealing with financial aid.
Leveraging faculty leadership. Coaches report to a leader within Academic Affairs-and the strong leadership, communication, and advocacy on coaches’ behalf within the academic division has facilitated strong faculty buy-in and support. Coaches are also extremely visible and truly integrated with the campus community, in large part given their proactive, out-of-office outreach. Faculty offering their perspective on the coaches describe them as a critical asset for the college, and especially for students.
The evaluation team is documenting the coaching model across Carolina Works institutions to better understand best practices that may improve success rates for more students. As part of this examination, the evaluation team collected on-site qualitative data in April 2018 from a single, high-performing campus, Randolph Community College noted in the call-out box on page 3. During this site visit, the evaluation team conducted in-depth interviews with success coaches and spent a half-day shadowing the coaches as they went about their daily tasks. The call-out box provides a high-level summary of key aspects of the coaching model being implemented on this campus.
One notable aspect of the coaching model at Randolph Community College is coaches’ tendency to spend large portions of their day engaging with students outside of their offices, tracking down students near classrooms or at other campus locations like the library or cafeteria. As a preliminary exploration of the importance of this coaching best practice, retention results were examined for the five colleges in the Carolina Works study in which coaches self-reported spending at least 30% of their time outside their offices engaging with students (self-reports were from a survey administered to all success coaches in Summer 2018).
As noted in the figure below, for these five colleges, retention rates are higher for treatment students than for the sample as a whole — and the difference in retention rates between treatment and control students are larger. Moreover, the data show a significant impact of coaching on student retention at one-year -six months earlier than for the overall sample. For this subset of colleges, retention rates for treatment students are more than five percentage points higher at both the 1-year and the 18-month marks.
These exploratory analyses suggest that specific features of the coaching model-in this case, coaches engaging with students face-to-face outside of their offices-may be particularly effective for building relationships with students and improving their academic outcomes.
Central Carolina Community College began its Success Coaching initiative through a Department of Education Title III grant in 2012. After initial successes with the coaching model, CCCC was awarded a second Department of Education grant under the First in the World initiative. This latest grant is one of two awarded to validate the effectiveness of Success Coaching. Central Carolina leads the 10 North Carolina community college consortium in this effort. Since implementing Success Coaching in 2012, students at Central Carolina have seen an increase of 17% in retention and completion rates have soared by over 10%. This First in the World project is part of a federal grant portfolio at Central Carolina joining other student success initiatives including six TRiO programs, a National Science Foundation STEM scholars program, and a new Department of Labor Youth Build Program.
Aviso Retention identifies the students where your institution can make the greatest impact, right now. Our solution combines best practices, advanced technology, millions of recorded engagements and measured results into higher education’s first Student Success Engagement Engine that learns where risk lies and how your Advisors, Success Coaches, Faculty and Staff can optimize the student experience. Please visit http://avisoretention.com to learn more.
Jobs for the Future is a national nonprofit that accelerates the alignment, transformation, and reimagination of the American workforce and education systems. JFF designs innovative solutions, scales proven programs, and influences policy to drive lasting impact. As a partner in the Carolina Works consortium, JFF provides overall strategic guidance with a focus on sustaining data-informed student success coaching across the ten consortium colleges and scaling coaching to more colleges within North Carolina and nationwide.
DVP-PRAXIS LTD leads the evaluation for Carolina Works. We are an action-oriented consulting firm focused on higher education and the workforce. We specialize in mixed-method formative and summative evaluation services to inform implementation and measure impact.
DISCLAIMER: Content developed under grant P11GX150010 from the U.S. Department of Education but does not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government.
8888 Keystone Crossing, Suite 1300 Indianapolis, Indiana 46240 • www.dvp-praxis.org
See how our tools are helping clients right now, get in-depth information on topics that matter, and stay up-to-date on trends in higher ed.
