Maria Paula Cacault, Christian Hildebrand, Jeremy Laurent-Lucchetti, Michele Pellizzari 23 June 2019
In the past decades, almost all sectors of the economy have experienced enormous changes in production technologies. Education has been slow at joining this trend but the so-called educational technology industry is now booming with an ever-evolving offer of e-learning platforms, distance learning tools, and massive open online courses or MOOCs. Given the enormous increase in the demand for higher education around the world, distance learning technologies are attracting special attention in this area. However, credible evidence about their effects on students’ outcomes is scarce (Escueta et al. 2017, US Department of Education 2010).
The few existing studies on this topic are either based on small-scale experiments that are unlikely to generalise to broader contexts (Alpert et al. 2016, Bowen et al. 2014, Coates et al. 2004, Figlio et al. 2013, Joyce et al. 2015) or peculiar higher education institutions – such as community colleges or private for-profit colleges – whose student populations are rather different from those of traditional universities (Bettinger et al. 2017, Xu and Jaggars 2013). Not surprisingly, results are very mixed.
In this column, we describe the results of an experiment that we conducted at the University of Geneva in which first-year bachelor students were offered access to an online platform showing lectures of many compulsory classes in live-stream format (Cacault et al. 2019). This is an institutional context that is representative of many higher education institutions around the world. We also cover a broad set of subjects, from economics to management or human resources or mathematics and statistics. Hence, we believe that our results are likely to generalise easily to a large number of other interesting settings.
We randomised access to the platform both across students and over weeks of the term so that the same student could attend classes online in some weeks but not others. Being compulsory introductory courses, the syllabuses were very standardised and the final exams were all designed as sets of multiple-choice questions. Thus, we were able to map all questions in the final exams to the weeks of the term in which the material was covered in class, allowing us to exploit variation both across and within students and weeks. Classroom attendance was always available to all students, allowing us to also study students’ choices of attendance mode.
We find that students use live streaming only occasionally, apparently when attending in class is too costly (e.g. due to shocks to their health or commuting conditions). Furthermore, attending lectures online has a negative impact on achievement for low-ability students, measured by high-school grades, and a positive one for high-ability ones. Having access to the streaming platform reduces the probability of correctly answering an exam question by two percentage points for students in the lowest quintile of the ability distribution, compared to an average probability of 55%. The effect at the opposite end of the ability distribution (i.e. students in the top 20%) is positive and equal to 2.5 percentage points (see Figure 1).
Figure 1 Change in probability of correctly answering exam questions by student ability (with access to streaming platform)
Additionally, we find that offering access to the live-streaming platform has only a modest effect on class size: on average, only eight out of a 100 students with access to live streaming do not show up in class.
The low usage of the streaming service is consistent with the idea that students have a general preference for classroom attendance because the professors’ explanations make it easier to understand the material. Only when random events (such as disease or bad weather) make it too costly to attend do they use the streaming platform. In accordance with this interpretation, we document that students are more likely to use the streaming service on days of adverse weather conditions and during an outbreak of seasonal influenza.
However, when streaming is not available, students of different abilities make different choices. Those of high ability are more likely to stay at home and study the material on their own. They are good students and they can understand the material without the lectures by putting in only a little bit more effort. The less able students, however, find it harder to understand the material without the explanations given in class; hence, they tend to go to class even when those random adverse events occur. In other words, the counterfactual to streaming is no attendance for the good students and regular in-class attendance for the least-able, rationalising the heterogeneous returns that we document in our empirical analysis. Streaming has positive effects on learning outcomes if it is used as a substitute for no attendance and negative effects if it substitutes in-class attendance.
These results are of immediate policy importance. The heterogeneous effects on grades that we document in our experiment indicate that online learning can potentially exacerbate education inequalities. If universities care about the academic performance of their students, they might use our results to motivate and implement policies offering access to distance-learning tools based on merit, so as to limit the negative effects on the least-able students.
Moreover, our evidence suggests that students have a general preference for classroom attendance. Hence, the use of distance-learning technologies is unlikely to solve problems of physical overcrowding. Of course, it is hard to say whether other technologies would have different effects but, at a minimum, our results suggest caution with the idea that distant learning tools can reduce class size.
Our analysis is specific to a very common learning environment, namely large introductory classes with many students and rather standardised content. Hence, despite being a single case study, it easily generalises to many similar settings.
However, it is also fair to say that it is unlikely to generalise to very different contexts, especially those where the complementarity between the teachers’ and the students’ abilities are more important. In small classes, for example, student-teacher interactions may benefit the good students the most, thus reversing the implications of our theory. Further studies in this direction would be important to explore the implications of distance learning technologies for higher education and ultimately for the design of education policies.
Alpert, W T, K A Couch and O R Harmon (2016). “A randomized assessment of online learning”, American Economic Review: Papers & Proceedings 106(5):378–382.
Bettinger, E P, L Fox, S Loeb and E S Taylor (2017). “Virtual classrooms: How online college courses affect student success’, American Economic Review 107(9): 2855-2875.
Bowen, W G, M M Chingos, K A Lack and T I Nygren (2014). “Interactive learning online at public universities: Evidence from a six-campus randomized trial”, Journal of Policy Analysis and Management 33(1):94–111.
Cacault, M P, C Hildebrand, J Laurent-Lucchetti and M Pellizzari (2019). “Distance Learning in Higher Education: Evidence from a Randomized Experiment,” CEPR Discussion Paper 13666.
Coates, D, B R Humphreys, J Kane and M A Vachris (2004). “’No significant distance’ between face-to-face and online instruction: Evidence from principles of economics”, Economics of Education Review 23:533–546.
Escueta, M, V Quan, A J Nickow and P Oreopoulos (2017). “Education technology: An evidence-based review”, NBER Working Paper Series 23744.
Figlio, D, M Rush and L Yin (2013). “Is it live or is it internet? Experimental estimates of the effects of online instruction on student learning”, Journal of Labor Economics 31(4).
Joyce, T, S Crockett, D A Jaeger, O Altindag and S D O’Connell (2015). “Does classroom time matter?”, Economics of Education Review 46:64–77.
US Department of Education (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies, Technical Report.
Xu, D, and S S Jaggars (2013). “The impact of online learning on students’ course outcomes: Evidence from a large community and technical college system”, Economics of Education Review 37(C):46–57.