PREDICTIVEANALYTICSOn the Journey
Nudge ExampleFun Theory
One of the largest public, two-year institutions in SC Primarily serving residents of Greenville CountyFall 2019 Headcount 11,12759% Female41% Male59% Part-Time41% Full-TimeAverage Age 24 years old16 to 1 Student to Faculty Ratio 74% of students receiving financial aid
100 Curricular offerings include:Certificates, diplomas, and associate degreesUniversity transfer courses and associate degreesApplied Baccalaureate in Advanced Manufacturing (Fall 2019) Academic program offerings include:Business, computer technology, health sciences,engineering technologies, advanced manufacturingtechnologies, and public service
Civitas Learning& Predictive Analytics
INFORMING ACTION ON DAY ONE OF A TERMPREDICTIVEMODEL82% 2.0 GPATRIGGER21%
What is Predictive Analytics?Predictive analytics is the useof historical data to determine thelikelihood of future events in order totake action to affect the outcome.
What is Persistence Probability?Persistence Probability is thelikelihood of a selected student toenroll in your institution in afuture term and stay enrolled pastyour institution’s add/drop period.
Collecting DataColleagueGRADUATEDDROPPED OUTPERSISTEDPERSISTEDLMS LoginsFinancial AidGPATRANSFERREDDemographicsSections AttemptedCIVITAS LEARNING COPYRIGHT 2017BlackboardLMS GradesDROPPED OUTDiscussion BoardContent ActivityGRADUATED
Power in cross functional teamsDIAL Working GroupPresident’s Cabinet Sponsors VP for Student Services VP for Learning & Workforce DevelopmentMembers Business Process Analyst Director Institutional Research Dean, School of Academic Advancement (IFA) Assistant Dean, School of Academic Advancement (IFA) CIO Dean of Enrollment Services (IFA) To serve as a champion for Civitas encouraging useand adoption across the institution. To serve as a leadership team that providescommunication, accountability, and initial direction forCivitas use. To serve as a source of information about the impactof Civitas to the President’s Cabinet. To inform student success initiatives to help leadershipprioritize resources. Establish a continuous improvement process withimplementation milestones and student successtargets that monitors, evaluates, and reports results. To work with the vendor to stay current with systemupgrades and to ensure our practices are in keepingwith best in class Civitas users. Director of Business Office & Auxiliary Services (Beta Test SiteLead)Director of Information Services (Campaigns)Director of Center for Professional Excellence (Training,adoptions, and use rates)Dean, School of Education and Professional Studies (IFF)Dean, School of Arts and Sciences (Courses)The working group will function as an implementationteam, focused on maximizing Civitas’ impact insupport of Strategic Imperatives and Outcomes,Institutional Success Measures, and InstitutionalObjectives. The group will develop a structure andprocesses that will identify student needs or otherareas of interest and communicate these findingswith various offices, academic advisors, andcommittees.10
Illume Students Powerful Predictor InsightUNIFIED DATA LAYERFall 2019: Learning from historical data 47% of our non-persisting students registered within 33 days before thestart of the term64% of our persisting students registered 33 days or MORE before the startof the term
African American Male ScholarsInitiative President’s Persistence and Retention Commission foundthat African American Males were at the bottom of everysuccess indicator reviewed by the team AAMSI is designed to provided advising services, lunchand learn sessions, and textbook vouchers to programparticipants Weekly updates of how likely their students are to persiston a group and individual basis paired with actions takenfor in-process adjustments Estimate of the Treatment Effects using a regressiondiscontinuity design and Illume Impact
Understand the efficacy of your student success initiativesso you can allocate resources toward initiatives that work,and connect the right students to the right programs.Key Features Measure intervention efficacy with statistical rigor See which interventions are more impactful for which types ofstudents Get results within a few hours of data submission
PPSM (Prediction-based Propensity Score Matching)Persistence PredictionPropensity ScoreMatching a student who participated in aninitiative with a similar student who didnot participate in the same initiative. Thismatching occurs on two dimensions:1.Similar persistence probabilities of the students(persistence prediction) and2.Similar probability of taking part in the initiativein question (propensity score)
2017 Academic Coaching In 2017 Academic Coaching was offered to (all/referred) students freeof charge All students are eligible(?) Many students were referred to Academic Coaching by Advisors orInstructorsResearch Question Impact is Measuring: Does providing scheduledand walk-in coaching improve persistence rates?
Analysis Results The analysis shows an 8.5 % Lift in Persistence Results met the .05 threshold for statistical significance with a p-value of 0.01 (golden) There were 1,030 analyzed participants over the 3 terms Lift in persistence is measured by comparing participating student persistence rates tothose of comparable students, matched on prediction and propensity scores Participants include all students that met with an Academic Coach one or more times in2017
Impact by Student Group Overall: 8.5% Lift Completed Terms 0 terms: 10.43% 1-3: 13.05% FT v PT FT: 7.87% PT: 8.71% Gender Female: 9.15% Male: 7.21% PredictionQuartilesNote: Results shown in blue meet the .05 p-value threshold for statisticalsignificance. Bottom: 15.11% Second: 8.73%
Greenville Technical CollegeCIVITAS LEARNING COPYRIGHT 2017
Inspire for Advisor
Inspire for Advisor
Inspire for Advisor Day 1 insights into students persistence likelihood Enables personal outreach to similar students at the same timeAllows Advisors to be more Effective and EfficientRecords outreachServes Students BETTERGetting overwhelmingly positive feedback fromfaculty/advisors on ease of use, time savings, and studentresponse
Sample Feedback - Students “I haven’t heard word like that since I came here I would like to justthank you for the words of encouragement. We need more instructorslike you.” “Hello Michelle! Thank you so much! This spring semester has beentough. I had to deal with death, sicknesses, and alot of personalissues. This encouragement gave me a second wind. I really doappreciate it.” “Thank you, for the words of encouragement. I did not do it alone, thepeople at Greenville Tech made this journey a great one. I cannotexpress to you how much they had a part in helping me along theway. I wish there was a way that I can show them how much thembeing here means to me. My professors have been some of the mostwonderful people, that I have had the pleasure of meeting. My thanksgoes to you for making Greenville Tech the college that it is.”
Sample Feedback – Faculty/Staff “First time I have ever emailed all of my Advisees andreceived a response from every one of them.” “Inspire for Advisor is very intuitive to use. It makes it soeasy to reach out to my students.” “Inspire is my favorite tool for communicating with mystudents. It makes it easy to personalize my outreach.”
CIVITAS LEARNING COPYRIGHT 2017
Inspire for Faculty
Inspire for Faculty
CIVITAS LEARNING, INC. -- CONFIDENTIAL INFORMATION
What are nudges?Nudges are small pushes inthe right direction that do notrequire prescribed actions, butencourage certainbehaviors.- Kuh, George D., et al. Student success incollege: Creating conditions that matter. JohnWiley & Sons, 2011.
What is a nudge campaign?A communication strategy(built upon a data-informed insight)that is designed to nudge atargeted group of students to achieve aspecific, measurable outcome.
Mindset“If a student believes challenges cannot beovercome, or his abilities are fixed, he is less likelyto push through the challenges he encountersalong the way. Telling students that thechallenges they experience can be overcome, andthat with effort, their abilities will grow, changestudents’ mindsets about challenges and, in turn,build resiliency.”- David Yeager & Carol Dweck
Mattering“An important part of being in a community isfeeling the sense that you matter to the people inthe community. Telling students that theirsuccess matters to people at your college oruniversity can positively influence their sense ofbelonging at your institution and encouragebehavior leading to successful outcomes.”– Nancy Schlossberg
Nudging Best PracticesSHORT – Just a few short sentences with one main ideaPERSONAL & AUTHENTIC – Conversational tone,include your ”voice” or personalityPOSITIVE & ENCOURAGING – Use a combination ofurgency, empathy or concern, hope, and encouragementto promote a growth mindsetTIMELY & RELEVANT – The right information at the righttime for the right students (align intended result withappropriate timing)ACTION ORIENTED – Include one, specific call to action
Subject Lines Matter!PERSONAL – Use of “you”FROM A FAMILIAR SENDER – A recognizable nameOPEN ENDED or URGENT – Time sensitive, Ask aQuestionSHORT – 40 characters or less, 50 maximumEYE CATCHING – Create urgency or inspire curiositySPECIFIC – Reflect the contentBRANDED – Reminder of the community
Example EmailSubject Line: School Name is Proud of You!Hi Student First Name ,You’ve worked hard and done so well here. I wanted to send you anote to see how your semester is going. College can be challenging this timeof year and even the most successful students can struggle. Commonchallenges for students include job commitments, family responsibilities,financial pressures, transportation issues, or a combination of these andother issues. Do any of these sound familiar to you?If you are experiencing any of these issues, or other things, I am here tohelp!I am proud of your progress and excited to support your continuedsuccess.Sincerely, Advisor Name
Nudge Campaign ResultsTargetPopulationHigh GPA Low PersistencePrediction (bottom half)Spring 2018Fall 2018 5.8%Spring 2019 5.0% 6.2%( /- 0.5%)*Online only Students 3.0GPA low PersistencePrediction (bottom half)Continuing Students(completed 4 or moreterms) - OverallBusiness & Technology-0.0% 1.5% 0.9%NA 4.8% 2.2%NA 5.0% 6.3%Health & WellnessNA 8.6% 10.3%Public Service, Arts &ScienceNew transfer (app typetransfer, 0 termscompleted), no GPA, noonline onlyNA 2.9%-4.0%-1.2% 4.8% 1.8%African American Male 3.9%N/A 9.7%OVERALL 3.6% 4.03 3.0%**
Setting up a nudge campaign Purpose Why are we doing this campaign? Goals What are we striving to achieve this with campaign? Communication Strategy & Logistical Considerations How are we planning to communicate this nudge? What logistical considerations do we need to account for? Targeted Population What is the targeted population of students we are going to focus on? List of student population Describe the steps for how to get the list of the student population. Data & Insights Supported by data, why are we selecting this student population to focus on? Execution Plan Draft the message(s) to be sent out, how often, by who, how, etc. Learning A conclusion statement regarding the results of the campaign.Adapted from Civitas Learning
Now you try!What targeted population of students could/should you focus on?Why are you going to focus on this population of students?What do we want to achieve with this nudge campaign?How do you plan on communicating this nudge campaign?Practice drafting the first communication to this population.
Lessons Learned Senior leader commitment REQUIRED Use of recommended Nudge Techniques High open rates Fear of overwhelming email response Training better by department Lacking dedicated resources to retention andpersistence efforts Frequent college-wide updates are important
What’s Next? Keep momentum going Continue Training push Increasing communication across the college Retention department with resources dedicated toutilizing Civitas insights weekly Creating area specific ownership Incorporate data insights into InstitutionalObjectives & key performance indicators Brainstorming ways to use Inspire for Advisor toincrease student survey response rates
engineering technologies, advanced manufacturing technologies, and public service. Civitas Learning & Predictive Analytics. INFORMING ACTION ON DAY ONE OF A TERM PREDICTIVE MODEL 82% 2.0 GPA TRIGGER 21%. Predictive analytics is the use of historical data to determine the likelihood o