This post was originally published by Vinod Bakthavachalam on Coursera’s Medium page.
Among committed enrollees on Coursera, male learners are more likely than their female counterparts to complete coursework in the most career-relevant content. This is especially true in the field of Data Science.
One driver is that female learners’ persistence is more adversely affected by a low score on an early assignment. That is, a low score on an early assignment appears to be more likely to deter a female learner from continuing in the course than a male learner in the same position. Claudia Goldin of Harvard identified a similar phenomenon at a prestigious college: undergraduate women’s persistence in economics was more sensitive to their grade in the first course than was the persistence of their male classmates.
In digging into the behavior of struggling learners, we noticed another difference by gender: after failing an assignment, female learners are much more likely to review course material before re-attempting the assignment; male learners, meanwhile, are more likely to simply re-attempt. Perhaps unsurprisingly, learners of both genders who review some content before re-attempting the assessment are more likely to end up passing that assessment.
Given these insights, we decided to test nudging struggling learners with helpful review material. Of course that requires knowing the best material to review, and knowing it across thousands of courses, tens of thousands of assessments, and hundreds of thousands of videos. Lucky for us, we’re emboldened with granular learning data across more than 100 million historical enrollments. This allows us to programmatically generate review material recommendations based on which item past learners struggling with the same assessment reviewed before a successful reattempt.
To summarize, here was our process…
- We began with simple insights: Female learner persistence is more adversely affected by failing an assignment than is persistence of their male counterparts; female learners are also more likely to try to review material after struggling on an assessment; learners who review are more likely to go on to be successful in the course.
- From there we formulated a hypothesis: Learners across Coursera could benefit from receiving targeted and personalized review material just after failing an assignment; and if female learners engage with the recommended material more than male learners it might help narrow the gender gap in course completion rates.
- We defined and built an intervention: We coded up a model to infer for each assignment which lecture was most important to review, based primarily on what learners who struggled with the same concepts reviewed before going on to pass successfully.
- We tested our intervention (and hypothesis!): We served the intervention to a random set of all struggling learners in the course and observed the treatment effect overall and by learner gender.
Here’s what we found: providing targeted review material to struggling learners significantly boosts continuation rates for female learners and has no real effect on average performance across male learners. The mechanism underlying the difference? Female learners receiving the intervention are 20% more likely than their male counterparts to click through and actually review the recommended material.
Given the positive impact on overall learner persistence, we’re keeping this intervention live across learner types. We are also implementing a feedback loop so that the review material recommendations improve each day based on the observed success of all past learners receiving the intervention.