Minority Male Success Initiative (MMSI) Performance Progress Report: Spring 2022

In 2018, Watermark worked independently with three institutions within the North Carolina Community College System (NCCCS) to quantitatively analyze student outcomes measuring course completion and persistence for minority male students. The goal was to best determine specific training and outreach messages using Watermark Early Alerts and predictive models identifying at-risk students. Given the success of this initiative, further research was desired and a much larger consortium of institutions was developed through the NCCCS Minority Male Success Initiative (MMSI).

The purpose of the Minority Male Success Initiative (MMSI) is to increase the progression and completion rate of minority male students and to increase the utilization of campus resources and services through deliberate and intentional interactions between students and the campus community.

As part of the latest round of funding for the initiative, Institutions were invited to apply for and to participate in one of three projects. This report offers insights into the progress for Institutions participating in the Early Alert project to illustrate an alignment of people, process, and technology geared towards helping at-risk and minority male students.

Supported by Watermark’s student success platform incorporating predictive analytics to prioritize engagement with ‘At-Risk’ students, the objectives of the project are to:

• Identify and reduce access barriers for all prospective students, particularly among underserved populations.
• Integrate targeted support services that promote student success and reduce achievement gaps for underserved students.
• Increase the progression, retention, and completion rates of minority male students through proactive success coaching facilitated by an early alert / advising system.
• Provide integrated and targeted supports and interventions when they are most effective.

This Early Alert project was modeled after a Randomized Controlled Trial (RCT) project that was completed in late 2020 as part of a First in the World research grant project. As part of this project, ten North Carolina community Institutions collaborated to determine if technology-enabled success coaching could be an effective tool for strategic enrollment management. Participants were: Central Carolina CC, Caldwell CCTI, Carteret CC (also part of the MMSI project), Cleveland CC, College of the Albemarle, Isothermal CC, Pamlico CC, Randolph CC, Roanoke-Chowan CC (also part of the MMSI project) and Southwestern CC.

Of the several published outcomes, it was found that Black students assigned to a success coach were eight percent more likely to stay enrolled in the college for one year and 18 percent more likely to stay enrolled for two academic years (Valentine and Price 2020).

Building upon this research, the MMSI Early Alert project was designed to further explore the impact of technology-enabled success coaching on Minority Male students and to determine if certain covariates can inform changes to program and interventions serving Minority Male students on campus. It should be noted that this project was built to be less rigorous and to have more institutional control, unlike that of a randomized controlled trial study, however the goal is still to define cohorts of first-time, new, postsecondary, Minority Male students and to assign a success coach to conduct proactive outreach throughout the term by utilizing the support technology and predictive analytics to prioritize outreach.

MMSI Cohort

• Student assigned a Success Coach
• Success Coach conducts proactive outreach first week of class and at other key points throughout term
• Success Coach receives Watermark automated alerts for grades, attendance, registration, etc. and follows up with student
• Success Coach receives faculty and staff manual early alerts, and follows up with MMSI cohort students
• Success Coach uses Watermark predictive risk indicators to prioritize outreach to students

All Students

• All students receive services already in operation at college (e.g., student services, counseling, faculty advisor, writing center, welcome center)
• Faculty, staff, and students receive Watermark automated alerts for grades, attendance, registration, etc.
• Faculty and staff manual alerts can be submitted via Watermark and will be followed up by the appropriate staff
• Faculty, staff, and students can receive Watermark automated achievements

Definition of “New Student”:

• New first-time, post-secondary; full or part-time; Minority Male students where new is defined as not having attempted prior post secondary curriculum credit at the institution

This analysis sets out to identify student characteristics correlated with student outcomes measuring course completion and student persistence for Minority Male students and to determine how those correlating factors compare with other student populations at the institution.

To best support the chosen project design, a longitudinal cohort analysis was leveraged to analyze and observe changes in student success outcomes. A longitudinal cohort analysis can be a helpful tool allowing institutions to track progress over time as part of the terms or cohorts they participate in.

Additionally a longitudinal cohort analysis can make use of disaggregated data to better understand covariates that may have differing impacts on students within a cohort. For this research, we focus primarily on New and Returning student cohorts and how Minority Male and White Male student success measures compare over time. Additionally we view these student cohorts in relationship to covariates that may or may not have varying levels of impact on outcomes. These covariates include: Academic Session, Primary Course Modality, Incoming Transfer Status, Enrollment Status, Academic Pathway, Student Risk level and Institutional Engagement level.

Further, to demonstrate an impact of a certain student success metrics over time, we compared aggregate outcome measures historically to what we call ‘Impact’ terms, or terms where advisors and success coaches are fully able to utilize Watermark for engaging new Minority Male postsecondary students on campus. Detailed observations are compiled below in the Longitudinal Analytics section while relative measures for each covariate are summarized in the Consortium level findings.

• Compare and contrast outcome measures demonstrated by Minority Male students and White Male students.
• Identify underlying relationships between student risk factors and outcome measures that might provide deeper insights into the course completion and persistence behaviors demonstrated by Minority Male students
• Demonstrate impact and effectiveness of various intervention strategies on Minority Male outcomes applied during the grant period.

Data for this report was extracted from the Watermark data warehouse. Used primarily to build and support the various risk models assigning and displaying student risk profiles within Watermark Student Success & Engagement, this data warehouse contains many of the data points of interest when exploring long term impact of student success initiatives on campus. This data warehouse is updated daily by the Watermark Student Success & Engagement product with the primary purpose to extract, transform and load institutional data from each of the primary systems on campus (SIS, LMS, Financial Aid, etc.).

Additional geographic information, including students’ home addresses, was gathered from the U.S. Census, American Community Survey, and U.S. Bureau of Labor Statistics.

The end of the 2019-20 school year was a stark contrast to the beginning. With the COVID-19 pandemic restricting in-person college courses across the country and around the world, many colleges and universities had to scramble to still provide a quality education amid a public health and economic crisis. Most faculty taught courses online for the first time, while advisors and success coaches had to quickly adapt to support students struggling with the academic and mental health obstacles thrown their way. This disruption to the educational process impacted Course Completion and Persistence during the first year of this study.

Student Outcomes

These institutional outcomes are derived from the set of all course records having positive attempted credit and displaying a course registration status of “Registered” or “Withdrawn” for the range of Historical and Impact terms at the institution. The following outcome measures are evaluated for all population subgroups and their courses.

1. Course Completion
   The course completion measure is defined as the successful completion of a course with a satisfactory grade. A satisfactory grade is defined as any final grade resulting in an A, B, C, or Passing/ Satisfactory for all postsecondary and high school students.
2. Student Persistence
   Persistence is measured by identifying which students within each cohort attempt coursework in a session starting within six months following the cohort end date OR have completed their degree requirements for graduation. Persistence was not measured for the high school population.

Race/Ethnicity & Gender Subgroups

To begin comparing outcome measures across student populations, unique student subpopulations were defined and created using a combination of Gender, Ethnic Code, and Race codes generated within and pulled from each college’s SIS. A detailed distribution of each subpopulation is provided below.

• Minority Male – Any student having a gender code of ‘M’ and an Ethnicity Code of ‘Hispanic’ OR a Race Code equal to ‘AN’, ‘AS’, ’BL’, or ’HP’.
• White Male – Any student having a gender code of ‘M’ and a Race Code of ‘WH’.

Student Types

To begin comparing outcome measures across student types, students were categorized as either new or returning:

• New – Defined as any student NOT attempting credit previously at the institution.
• Returning – Defined as any student previously attempting credit at the institution.

Historical and Impact Terms

Historical Terms are cohorts of students represented in the study prior to the implementation of the MMSI Early Alert project on campus, while Impact Terms are cohorts occurring post commencement of the Early Alert projects. These Impact terms include Spring 2020, Fall 2020, Spring 2021, and Fall 2021 enrollment cohorts.

Course Modality

Changes in Course modalities since the start of the pandemic have certainly presented a potential impact on student success outcomes and is something that needs to be understood to determine if learning learning outcomes are impacted and if this change in covariate is something that should be a focus as we continue to work towards narrowing the equity gap. To explore this, we categorized students as follows:

1. Some Onlinea. No Time – No Online Coursework.

   b. Some Time – Less than 50% Online Coursework.

2. Mostly Onlinea. Most Time – Greater than 50% Online Coursework.

   b. Full Time – 100% Online Coursework.

While the shift in instruction is noticeable, we will also see later in the reports that course modality could in fact have a significant impact on success outcomes and even more so that technology enabled success coaching can reduce this negative impact.

Institutional Engagement

To provide a comparative measure of institutional engagement relative to peer institutions, Auto Alerting, Early Alerting, Messaging, and Note Taking were measured and indexed across institutions to arrive at a High vs. Low engagement measurement for each institution. Variable weights were assigned to each engagement type to determine if this overall fidelity grouping has an impact on success outcomes. It should be noted that Early Alerting and Messaging were weighted higher than Note Taking and Note Taking was weighted higher than Auto Alerting in the creation of this index. Looking across activity at each institution with respect to these engagement metrics, the top six institutions were graded as having High Engagement and the bottom five as Low.

The Consortium level findings are used to best understand impact across all Institutions in a pooled manner, allowing for increased confidence in the measured impact. As the study progressed with additional cohorts collected and with cohorts outcome measures maturing, observations were made to monitor the progress of the study along the way.

This progress report is the first to include all intended Study cohorts (Spring 2020 through Fall 2021) and focuses specifically on the impact of success outcomes for New and Returning Postsecondary Minority Male students across the following Covariates: Academic Session, Primary Course Modality, Incoming Transfer Status, Enrollment Status, Academic Pathway, Student Risk level, and Institutional Engagement level.

While final analysis is pending, preliminary observations collected through the completion of the Fall 2021 academic term suggest that success coaching, predictive analytics, and the use of automated early alerts can assist in closing the equity gap in higher education and improve student outcomes for underserved student populations.

Persistence gap between Minority and White Male students was eliminated within the first year.

Course Completion

• New Student Minority Male postsecondary course completion rates increased 1.0% during Spring cohorts, while Returning Student Minority Male postsecondary course completion rates increased 1.9% during Spring cohorts.
• New Student Minority Male postsecondary course completion rates for students taking Mostly Online courses increased 8.6%, while Returning Student Minority Male postsecondary course completion rates for students taking Mostly Online courses increased 3.0%. Both New and Returning Minority Male postsecondary students taking Mostly Online courses experienced larger increases in course completion than White Males.
• New and Returning postsecondary Male students having medium or high Risk Levels experienced increases in course completion. More specifically, high Risk New Minority Male students demonstrate a 19.4% increase in course completion, while Returning students demonstrate a 16.3% increase. Increases in course completion were also realized for medium Risk students.

• New Student Minority Male postsecondary persistence rates increased 4.8% overall. This was .07 percentage points higher than White Males during the student period. Returning Student Minority Male postsecondary persistence rates increased 1%.
• Spring term enrollments appear to benefit the most from technology enabled success coaching, as New Student Minority Male postsecondary persistence rates increased 3.7% during spring terms.
• New Student Minority Male postsecondary persistence rates for students taking Mostly Online courses increased 16.8%, while Returning Student Minority Male postsecondary persistence rates for students taking Mostly Online courses increased 5.2%.
• New Student Minority Male postsecondary persistence rates for students possessing Transfer credit increased 10.6%.
• New and Returning postsecondary Male students having medium or high Risk Levels experienced increases in persistence. Specifically, High Risk New Minority Male students demonstrate a 15.8% increase in course completion, while Returning students demonstrate a 18% increase.
• New Student Minority Male postsecondary persistence rates for students attending Institutions with higher levels of Institutional Engagement increased 6.4%.

As we continue to follow these student cohorts over the next several years, our research will be expanded to not just focus on outcome impact, but to develop more insight into coaching relationships, academic pathways, how engaged students are with their support network at each institution, and specific interventions that had the greatest impact.

While the initial results are encouraging and suggest that progress is being made, a better understanding of engagement levels, barriers, and interventions will lead to strategic recommendations on how to best serve Minority Male students within the North Carolina Community College system.