Transcription

Virtual Classrooms: How Online College Courses Affect StudentSuccessBy ERIC P. BETTINGER, LINDSAY FOX, SUSANNA LOEB, AND ERIC S. TAYLOR*Online college courses are a rapidly expanding feature of highereducation, yet little research identifies their effects relative totraditional in-person classes. Using an instrumental variablesapproach, we find that taking a course online, instead of in-person,reduces student success and progress in college. Grades are lowerboth for the course taken online and in future courses. Students areless likely to remain enrolled at the university. These estimates arelocal average treatment effects for students with access to both onlineand in-person options; for other students online classes may be theonly option for accessing college-level courses.* Bettinger and Loeb: Stanford University, Center for Education Policy Analysis, 520 Galvez Mall, Stanford, CA 94305(emails: [email protected], [email protected]); Fox: Mathematica Policy Research, 505 14th Street, Suite 800,Oakland, CA 94612 (email: [email protected]); Taylor: Harvard University, Gutman Library 469, 6 AppianWay, Cambridge, MA 02138 (email: eric [email protected]). We greatly appreciate the support of the university whosedata we study in this paper. We also thank Tom Dee and seminar participants at UC Berkeley, Brigham Young University,CESifo, IZA, Mathematica Policy Research, University of Michigan, Stanford University, University of Stavanger,University of Texas Austin, Teachers College Columbia University, Texas A&M University, University of Uppsala, andUniversity of Virginia for helpful discussions and comments. Financial support was provided by the Institute of EducationSciences, U.S. Department of Education, through Grant R305B090016 to Stanford University. The views expressed andany mistakes are those of the authors. A previous version of this paper was circulated with the title “Changingdistributions: How online college courses alter student and professor performance.” The authors declare that they have norelevant or material financial interests that relate to the research described in this paper.Online college courses are a rapidly growing feature of higher education. One outof three students now takes at least one course online during their college career,and that share has increased threefold over the past decade (Allen and Seaman

2013). The promise of cost savings, partly through economies of scale, fuelsongoing investments in online education by both public and private institutions(Deming et al. 2015). Non-selective and for-profit institutions, in particular, haveaggressively used online courses.In this paper we estimate the effects of taking a college course online, instead ofin a traditional in-person classroom setting, on student achievement and progressin college. We examine both mean effects and how online courses change thedistribution of student outcomes. While online course-taking is both prevalent andgrowing, there remains relatively little evidence about how taking a course online,instead of in-person, affects student success in college. Evidence on this questionfrom the for-profit sector is particularly scarce.Our empirical setting has three advantageous features: the substantial scale of alarge for-profit college, an intuitive counterfactual for each online course, and aninstrument which combines two plausibly-exogenous sources of variation inwhether students take a course online. The combination of these three features—and the resulting contributions to identification and generalizability—has not beenpossible in prior work.We study students at one large for-profit university with an undergraduateenrollment of more than 100,000 students, 80 percent of whom are seeking abachelor’s degree. The university’s average student takes two-thirds of her coursesonline. The remaining one-third of courses meet in conventional in-person classesheld at one of the university’s 102 physical campuses. The data for this paper covermore than four years of operations, including over 230,000 students enrolled in168,000 sections of more than 750 different courses.11This paragraph describes the university during the period we study—2009 to 2013. In recent years student enrollmenthas declined substantially, and many physical campuses have closed.

The university’s approach to online education creates an intuitive counterfactual.Each course is offered both online and in-person, and each student enrolls in eitheran online section or an in-person section. Online and in-person sections are identicalin most ways: both follow the same syllabus and use the same textbook; class sizesare approximately the same; both use the same assignments, quizzes, tests, andgrading rubrics. The contrast between online and in-person sections is primarily themode of communication. In online sections, all interaction—lecturing, classdiscussion, group projects—occurs in online discussion boards, and much of theprofessor’s “lecturing” role is replaced with standardized videos. In online sections,participation is often asynchronous while in-person sections meet on campus atscheduled times. In short, the university’s online classes attempt to replicate itstraditional in-person classes, except that student-student and student-professorinteractions are virtual and asynchronous.The contrast between online and in-person classes at the university we study is,we think, consistent with intuitive definitions of “online” and “in-person” classes.We use these two labels throughout the paper as shorthand for this specificapproach. Many other quite-different approaches to education are also commonlycalled “online education” or “online classes” (McPherson and Bacow 2015 providea review), for example, massively open online courses (MOOCs). Our shorthand“online” should not read as broadly representative of all online education. However,the form of online education used by the university we study is widely used in boththe public and private sector.To estimate the effects of taking a course online, instead of in-person, we use aninstrumental variables approach. Our strategy makes use of two key influences onstudents’ course-taking behavior: (i) changes from term to term in which coursesare offered in-person at each student’s local campus, and (ii) the distance eachstudent must travel to attend an in-person course at that local campus. Either of thetwo might be used as an instrument on its own. Distance has in fact often been used

in studies of education, but with reservations (Card 2001, Xu and Jaggars 2013).Instead of using either alone, our instrument is the interaction of these twovariables.2 With the interaction serving as the excluded instrument, we control forthe main effects of both variables in the first and second stages, following a strategyfirst proposed by Card (1995).3A causal interpretation of our estimates still involves an exclusion restriction, butthat assumption is more plausible than it would be if we used either distance orcourse offerings alone as the instrument. For example, if we used distance alone asthe instrument, the exclusion restriction would require that student distance fromcampus can only affect course grades by changing the probability that students takea course online instead of in-person. Distance from campus is a function of studentchoices about where to live (and university choices about campus locations) andthus may be related to unobservable characteristics. By contrast, as we explainbelow, the interaction design exclusion restriction permits “other mechanisms” andonly requires that (a) any other mechanism through which student distance fromcampus affects course grades is constant across terms with and without an in-personclass option; and (b) any other mechanism causing grades to differ between termswith and without an in-person class option affects students homogeneously withrespect to their distance from campus.Our estimates provide evidence that online courses do less to promote studentacademic success and progression than do in-person courses. Taking a courseonline reduces student achievement, as measured by grades, in that course by aboutone-third of a standard deviation. Taking a course online also reduces studentgrades in future courses by one-eighth of a standard deviation, and reduces the2The interaction of (i) an indicator 1 if student 𝑖’s home campus 𝑏 offered course 𝑐 on campus in a traditional inperson class setting during term 𝑡, and (ii) the distance between student 𝑖’s residence and her home campus 𝑏. Results usingeither (i) or (ii) as the instrument are similar and available from the authors upon request.3We further limit variation to within-course, with-home-campus, and within-major; control flexibly for secular trends;and control for prior achievement and other student observables.

probability of remaining enrolled a year later by over ten percentage points (over abase of 69 percent). Additionally, we find that student achievement outcomes aremore variable in online classes, driven in part by a greater negative effect of onlinecourse-taking on students with lower prior GPA. While the data and setting westudy allow us to say, with some confidence, that taking a class online has negativeeffects on student success, we cannot address empirically how these negativeeffects arise. The data and setting do not lend themselves to a comprehensive studyof the underlying mechanisms.Our research contributes to two strands of literature. First, it provides substantialnew evidence of the impact of online college classes— in particular, the impact forstudents in broad-access higher education institutions. Several prior studiesrandomly assign students to an online or in-person section of one course and findnegative effects on student test scores (Figlio, Rush, and Yin 2013, Alpert, Couch,and Harmon 2014, Joyce et al. 2015) or, at best, null results (Bowen et al. 2014).4These studies are well-identified but each examines only a single course ineconomics or statistics, and their focus is on college students at relatively-selectivepublic four-year colleges. We examine more than 700 courses, and students at anon-slective for-profit college, a population of particular interest for policy. At suchcolleges, online courses have grown most rapidly and are central to the institutions’teaching strategy. Several other quasi-experimental studies examine two-yearcommunity colleges and students taking a broad set of courses; the estimated effectsof online course-taking are again negative.5 Xu and Jaggers (2013, 2014) andStreich (2014b) use instrumental variables designs: distance from home to campusand availability of seats in in-person classes, respectively. A research design using4Using non-experimental methods, Brown and Liedholm (2002) and Coates et al. (2004) also find negative effectsstudying microeconomics principles courses.5For comparison, one in three for-profit students takes all of her courses online, compared to one in ten communitycollege students (McPherson and Bacow 2015).

either of these two instruments, on its own, requires relatively strong identifyingassumptions for making causal claims. Our design substantially weakensidentifying assumptions by combining two instruments.6Second, our paper adds to the new and growing literature on private for-profitcolleges and universities. Research on for-profit institutions—the university westudy and its peers—is increasingly important to a complete understanding ofAmerican higher education. The for-profit share of college enrollment and degreesis large: nearly 2.4 million undergraduate students (full-time equivalent) enrolledat for-profit institutions during the 2011-12 academic year, and the sector grantedapproximately 18 percent of all associate degrees. For-profit colleges serve manynon-traditional college students, who are often the focus of policy. Deming, Goldin,and Katz (2012) provide an overview of the for-profit sector.Our study is the first, of which we are aware, to estimate the effects of onlinecourses among students at large for-profit colleges and universities. Our estimatescomplement a growing literature on labor market outcomes for for-profit collegestudents. Deming et al. (2016) report that graduates from mostly-online for-profitcolleges are less likely to receive a callback for a job interview compared toobservably similarly graduates from non-selective public colleges. For example,job applicants with a business degree were 22 percent less likely to be called back.Using a similar resume-audit design, Darolia et al. (2015) find for-profit graduatesare no more likely to get a callback than are applicants without a college degree.These differences in hiring may or may not translate into differences in earnings(Turner 2012, Lang and Weinstein 2013, Cellini and Chaudhary 2014, Cellini and6Additionally, Hart, Friedmann, and Hill (forthcoming), makes a case for causal identification by focusing on withincourse and within-student variation in whether a course is taken online or in-person. Streich (2014a) finds some evidence ofpositive effects on employment, though in years when the student is likely still enrolled in college. Finally, the widely-citedconclusion of a US Department of Education (2009) meta-analysis is that outcomes in online courses are better thantraditional in-person courses; that meta-analysis, however, includes a wide variety of “online” courses. When the analysis islimited to studies of semester-length, fully-online courses there is no difference in outcomes (Jaggers and Bailey 2010).

Turner 2016), but for-profit graduates would need substantially greater earningsthan students from other institutions in order to offset the higher costs of attendinga for-profit school (Cellini 2012). Poorer labor market outcomes for for-profitstudents may be in part because employers have learned what this study findseconometrically: students taking online courses have poorer academic achievementon average. However, a for-profit degree likely creates or signals other potentialdifferences beyond a history of online course-taking, not the least of which is thedifferential selection of students with different abilities into the for-profit sector.I. Setting and DataWe study undergraduate, degree-seeking students taking courses at one large forprofit university. While the institution began primarily as a technical school, today80 percent of the university’s undergraduate students are seeking a bachelor’sdegree, and most students major in business management, technology, health, orsome combination. Each course at the university is offered through both onlineclasses and traditional in-person classes, though the availability of in-person classesvaries over time and from campus to campus. For the average course, two-thirds ofundergraduate classes occur online, and the other third occur at one of over 100physical campuses throughout the United States. Students at the university we studyare older and more often African-American or Latino(a) than students at public andnon-profit colleges, though the university’s students are quite similar to students atother for-profit institutions (Appendix Table 1).The university provided us with data linking students to their courses for allonline and in-person sections of all undergraduate courses from Spring 2009through Fall 2013. These data include information on over 230,000 students inmore than 168,000 sections of 750 different courses. About one-third of thestudents in our data took courses both online and in-person. Table 1 describes the

sample. Just under half of the students are female and average approximately 31years of age, though there is substantial variability in age. Students in online coursesare more likely to be female (55 percent vs. 35 percent) and older (33 years vs. 28years).7[ Insert Table 1 Approximately Here ]The focus of this paper is on the level and distribution of student outcomes.Ideally, we would like to know how much students learn in each course they take(whether online or in-person), but we have no direct measure of learning. Insteadwe examine the several different observed outcomes which are imperfect correlatesof learning, most notably grades in the current course and future courses. There arereasons to be cautious about over-interpreting course grades. In many highereducation institutions, course grades are subject to professor discretion, andprofessors may exercise that discretion differently in online and in-person classes.That discretion is a consideration in this paper, but university’s grading processpermits less discretion than the typical selective college or university. For eachcourse, professors are asked to follow a common rubric for evaluating individualassignments and assigning grades. In many cases quizzes and tests are standardizedacross sections, whether online or in-person. Additionally, alongside course gradesboth for the target course and in future courses, we present results for persistence—a consequential outcome for students seeking a degree and one not influenced byprofessor subjectivity.As shown in Table 1, the average grade was 2.8 (approximately B-), on thetraditional zero (F) to four (A) scale. Grades vary substantially: the standarddeviation is 1.3 (more than a full letter grade). Over 88 percent of students were7The observations are student-by-course-by-term. Of those observations 7.6 percent are students retaking a course theypreviously took. Our effect estimates are robust to focusing only on first try enrollments by excluding repeater observations.

still enrolled at the university in the following semester or had completed theirdegree. The average student GPA in that following semester was 2.8. Over 69percent were enrolled one year later or had completed their degree. These outcomemeans are consistently higher in the in-person classes than in the online setting. Thedifferences could result from systematically different students enrolling in inperson classes. In the next section we discuss our strategies for overcoming theselection bias.II. Instrumental Variables StrategyOur objective is to estimate the effect of taking a course online, instead of in atraditional in-person classroom, on student success in the course, success in futurecourses, and persistence in college. The decision to take a course in an onlinesection or traditional section is likely endogenous—driven by unobservableinformation that could also influence each student’s chances of success in the onlineversus traditional options. Our identification strategy is to use only variation inonline versus in-person course-taking that arises because of two key influences onstudents’ course-taking: (i) changes from term to term in which courses are offeredin-person at each student’s local campus, and (ii) the distance each student musttravel to attend an in-person course at that local campus. Our instrument is theinteraction of these two influences.Our first student outcome is the grade, 𝑦𝑖𝑐𝑡 , received by student i in coursec during term t. Each professor assigns traditional A-F letter grades, which weconvert to the standard 0-4 point equivalents.8 To estimate δ—the mean differencein course grades between students in online and traditional in-person classes—wefirst specify the following statistical model:8An A is 4 points, A- is 3.7, B is 3.3, B is 3, etc.

(1) 𝑦𝑖𝑐𝑡 𝛿𝑂𝑛𝑙𝑖𝑛𝑒𝑖𝑐𝑡 𝒚(𝑖,𝜏 𝑡) 𝛼 𝑿𝑖𝑡 𝛽 𝜋𝑐 𝜙𝑡 𝜓𝑏(𝑖𝑡) 𝜌𝑝(𝑖𝑡) 𝜀𝑖𝑐𝑡 ,where the indicator variable 𝑂𝑛𝑙𝑖𝑛𝑒𝑖𝑐𝑡 1 if the course was taken online, and 0if it was taken in-person. Specification 1 includes several additional controls. First,we control for student i’s prior grades, 𝒚(𝑖,𝜏 𝑡) , in all terms prior to term t. Thevector 𝒚(𝑖,𝜏 𝑡) includes two primary variables: (i) student i’s prior grade pointaverage (GPA) in all courses taken online, and separately (ii) her GPA in all coursestaken in-person.9 We also include observable student characteristics, 𝑿𝑖𝑡 (genderand age); course fixed effects, 𝜋𝑐 , for each of the 750 courses; major fixed effects,𝜌𝑝(𝑖𝑡) , for each of the 22 different degree programs; and a non-parametric timetrend, 𝜙𝑡 , over the 27 terms in our data (spanning 4.5 years at 8 weeks per term).Finally, Specification 1 includes fixed effects for student i’s “home campus”represented by 𝜓𝑏(𝑖𝑡) . During the period of time we study, the university operated102 local campuses throughout the United States. We assign each student to onehome campus, b, based on the physical distance between the student’s homeaddress and the local campus addresses, selecting as “home” the campus with theminimum distance.10 Throughout the paper standard errors allow for clusteringwithin campuses b.In response to selection bias concerns, we propose an instrumental variablesstrategy in which we instrument for 𝑂𝑛𝑙𝑖𝑛𝑒𝑖𝑐𝑡 in Specification 1 with theinteraction of two variables: (i) 𝑂𝑓𝑓𝑒𝑟𝑒𝑑𝑏(𝑖)𝑐𝑡 , an indicator 1 if student 𝑖’s homecampus 𝑏 offered course 𝑐 on campus in an in-person class setting during term 𝑡,9Each GPA is simply the weighted mean of prior course grades, where the weights are course credits. Because not allstudents have taken both online and in-person courses, 𝒚𝑖,𝜏 𝑡 also includes indicators for having previously taken any coursesonline, and any courses in-person. When online (in-person) GPA is missing we set it equal to the mean online (in-person)GPA.10This distance is the straight-line distance. In addition to excluding students with missing address data, we exclude allstudents with international addresses and students whose addresses are not within the continental United States. The resultingsample contains 78 percent of the universe of undergraduate students at the university over the time frame.

and (ii) 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑖𝑡 , the distance in miles between student 𝑖’s residence and herhome campus 𝑏. With the interaction serving as the excluded instrument, weinclude main effects for 𝑂𝑓𝑓𝑒𝑟𝑒𝑑𝑏(𝑖)𝑐𝑡 and 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑖𝑡 in both the first and secondstage. Card (1995) first proposed this interaction-instrument strategy. We use thesame specification and interaction instrument strategy to estimate 𝛿 for otherstudent outcomes, including the grades a student receives in subsequent classes andstudent persistence at the university.11Using this interaction instrument requires weaker identifying assumptions thanwould using either 𝑂𝑓𝑓𝑒𝑟𝑒𝑑𝑏(𝑖)𝑐𝑡 or 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑖𝑡 alone. Consider the relationshipbetween an instrument and the outcome measure; in other words, the instrument’scoefficient in the reduced-form equation. The exclusion restriction requires that thisrelationship be caused by only one mechanism: in our setting, the instrumentinduced some students to take a class online, instead of in-person, and the onlineclass hurt (improved) their academic outcomes. If we were to use 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑖𝑡 aloneas the instrument, that reduced-form slope would capture any effect of the onlineclass mechanism, but also reflect the effects of other plausible mechanisms. Forexample, higher achieving students may choose to live nearer to campus, or theuniversity may locate campuses based on students’ potential outcomes.12 Bycontrast, in the interaction instrument design, the reduced-form coefficient onlymeasures how the slope, between distance and grade, changes when students areoffered an in-person class option. The main effect of distance (included in both thefirst and second stages) nets out any “other plausible mechanisms” which are11The university’s academic calendar divides the year into six terms, with two consecutive terms equivalent to a semesterin a more traditional calendar. We define “enrollment the next semester” as enrollment during either term 𝑡 1 or 𝑡 2 orboth, and “enrollment one year later” as enrollment during either term 𝑡 5 or 𝑡 6 or both.12The university may not be consciously or intentionally correlating location and potential outcomes. The universitymay, for example, primarily locates campuses in suburbs where students’ prior educational inputs are likely greater, onaverage, than in urban and rural areas. This example would still violate the exclusion restriction.

constant across terms with and without an in-person option. Parallel reasoning canbe constructed for the 𝑂𝑓𝑓𝑒𝑟𝑒𝑑𝑏(𝑖)𝑐𝑡 component of the instrument.Thus we can state the exclusion restriction for the interaction instrument asfollows: With one exception stated below, (a) any mechanism through whichstudents’ distance from campus affects course grades (persistence) is constantacross terms with and without an in-person class option; and (b) any mechanismcausing grades (persistence) to differ between terms with and without an in-personclass option affects students homogeneously with respect to their distance fromcampus. The one important exception is that in terms when students have a choicebetween an online and in-person class, the distance a student lives from campusmay affect the probability of choosing to take the class online, instead of in-person,and that taking the course online could harm (improve) the student’s academicoutcomes.What would violate this exclusion restriction? For one example, students mightchange their residence from term to term in response to in-person class offerings.We show below that student moves are relatively rare and unrelated to studentobservables, and our estimates are robust to excluding movers. There are essentiallyno campus openings, closures, or moves in our data. For another example, theuniversity might decide when to offer in-person classes based on students’ potentialoutcomes, but to violate the exclusion restriction the decision rule would need togive systematic differential weight based on student distance from campus. Wediscuss what we know about university’s decision-making process on in-personofferings in the next section.We show some limited empirical evidence consistent with these assumptions inAppendix Table 2. That table reports a series of covariate tests where we estimateSpecification 1 replacing the outcome variable with one student characteristicmoved from the right hand side: gender, age, prior GPA in online and in-personcourses. We also examine other pre-treatment measures: the number of prior

courses online and in-person, whether the student moved in the prior term, whetherthe student is repeating the course, and how many terms since the student’s lastcourse. In all cases the “effect” of taking a course online on these pre-treatmentoutcomes is not statistically different from zero.13Even if the exclusion restriction holds, our estimates could be biased by a weakinstrument. A strong first stage is plausible. Assume the availability of an in-personclass does reduce the probability of taking the course online; that effect should beheterogeneous with the reducing effect becoming weaker the further away fromcampus a student lives. We see this pattern empirically in the first-stage resultspresented in Table 2: availability of an in-person class reduces the probability oftaking the course online by 22 percentage points for a student living next to thecampus, but that reduction is smaller the further away from campus the studentlives, about a 1.4 percentage point reduction for every 10 miles.[ Insert Table 2 Approximately Here ]The interaction instrument has distinct advantages over using or 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑖𝑡 alone as an instrument, but the interactionnevertheless uses the underlying variation in 𝑂𝑓𝑓𝑒𝑟𝑒𝑑𝑏(𝑖)𝑐𝑡 and 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑖𝑡 . In thenext two subsections we describe what gives rise to that variation.A. Courses Offered In-personWhether an in-person class option at the local campus is available to a studenttaking a particular course varies meaningfully. By contrast, an online class optionis available every term for almost all courses. The university divides its academicyear into six eight-week terms. The typical course is, on average, offered in-person13Appendix Table 2 also shows these tests omitting all right hand side control variables and fixed effects. In those teststwo of nine covariates show statistically significant differences: female and repeating course.

one out of every four terms at the typical campus (conditional on the campus everoffering the course in-person). The interquartile range is one out of thirteen termsto one out of three terms. About half of the total variation in in-person offerings iswithin course and campus—variation between academic terms within course-bycampus cells.14 In our analysis we limit identifying variation to this within-courseand within-campus variation. The remainder of this section focuses on that residualvariation and its causes.Decisions about offering in-person classes are left largely to administrators ateach local campus, especially decisions from term to term for a given course.15 Theempirical data suggest that those decisions are not driven by current or past studentdemand. First, variation in in-person offerings is not explained, notably, by priorenrollment in the course at the campus. Enrollment in in-person sections during theprior year (six terms) explains just 0.1-0.6 percent of the variation.16 Totalenrollment—combining in-person and online enrollments for students assigned tothe campus—explains similarly little of the variation. Moreover, the variation is notexplained by the observable characteristics of students who enrolled in the coursein prior terms, characteristics like GPA and whether they had taken an in-personcourse previously. Second, while more difficult to test, in-person offering decisionsdo not appear to be a function of demand in the current term 𝑡. Our partial test relieson a (supposed) university norm that in-person classes should be cancelled if fewerthan five students enroll. If in-person offering decisions do respond to currentdemand we should see no classes with fewer than five students, or at least adiscontinuous jump in the density

called "online education" or "online classes" (McPherson and Bacow 2015 provide a review), for example, massively open online courses (MOOCs). Our shorthand "online" should not read as broadly representative of all online education. However, the form of online education used by the university we study is widely used in both