Inequality in learning is a major concern after school closures

April 21, 2021
118 (20) e2105243118
Research Article
Learning loss due to school closures during the COVID-19 pandemic
Per Engzell, Arun Frey, Mark D. Verhagen
The COVID-19 pandemic has come with many nonpharmaceutical interventions to curb the spread of the virus. One of the most significant measures has been to close schools for in-school education, at the primary, secondary, and tertiary levels of schooling. While such interventions have been demonstrated to be effective in reducing the spread of the virus (1), they come at a cost: an intermittence of the learning process that is not easily repaired with online instruction. The well-designed study by Engzell et al. (2) shows, convincingly, that the “learning loss” due to school closures is severe (although learning delay may be a more appropriate term). Using data from primary schools in The Netherlands and analyzing test scores on externally standardized tests, the study reports a learning loss of 3.16 percentiles on a composite index of math, reading, and spelling, an effect that varies in size by socioeconomic background and school composition.
At least as concerning as the overall learning delay is the reported inequality by socioeconomic background and school composition. Using school records of parental educational attainment (classifications used to determine weighted student funding for schools with children from disadvantaged backgrounds), the study shows that children of very low-educated parents (i.e., none of the parents have more than lower-secondary education; in total, 8% of the families) suffer more from school closure than children from more-educated backgrounds. Using a more fine-grained measure of school-level disadvantage, supplementary analyses show that the learning delay is much stronger in schools with a higher share of disadvantaged children (which could be simply compositional, as the school-level indicators were not available at the individual student level). Other European studies on the impact of COVID-19−related school closures show similar patterns (3, 4), with stronger delays among children with disadvantaged backgrounds or schools with higher concentrations of disadvantage.
Possible explanations for a learning delay gap are related to parental involvement with education (5), socioeconomic differences in information and communications technology (ICT) access and skills among students and the schools they attend (6, 7), and parents’ ability to help with homework during the school closures (8). More studies are needed to understand these mechanisms. Following the “digital divide” literature, providing students with technological devices alone is unlikely to solve the problem, as inequality in digital skills and usage are additional sources of the digital gap between socioeconomic groups (9).
In interpreting effect sizes, Engzell et al. (2) estimate that the average learning progress has been close to zero during the closures. The overall learning delay of 3.16 percentiles equals 0.08 SDs. Comparing this to the estimated SD growth from American studies and the World Bank of 0.40 SD learning growth per year under normal circumstances, the study concludes that the loss equals 8 wk of lost progress, similar to the 8-wk school closure (close to one-fifth of a school year). If, instead of using American or global estimates on school progress under normal circumstances, we use Dutch estimates on the spelling test that is analyzed [available for spelling growth between Dutch grades 6 and 7, roughly from age 10 y to 11 y (10)], we can further benchmark the estimated 3.03 percentile learning loss in spelling for this grade, in SI Appendix, table S11 of ref. 2. The national SD growth equals 0.64 SD growth on the midyear test and 0.45 SD at the end-of-year tests between grades 6 and 7, averaging 0.55 SD (calculations based on table 2.1 of ref. 10). Assuming 3.03 percentile loss equals (3.03/3.16*0.08 =) 0.077 SD, the learning delay would then be roughly 0.077/0.55 = 0.14 of a school year, equaling 5.6 wk of learning delay over the 8-wk closure (assuming a 40-wk school year). Following these rough estimates, the closure weeks then worked at a 30% efficiency. The study by Engzell et al. (2) goes a long way in interpreting the effect sizes, and they too present some estimates that suggest there may be some, but strongly reduced, efficiency of the online learning process that replaced in-school instruction.
Two control variables have been inserted in the difference-in-differences models of ref. 2: an overall trend over years and the number of days between the midyear and end-of-year tests. Without these control variables, the negative impact of school closures is less strong. In defense of their inclusion of the trend parameter, the placebo analysis had more expected null effects than without a trend control. The trend parameter is positive, suggesting that, across the years 2017−2020, school performance growth has increased, while other student assessment data on students in Dutch grade 6 show a stable or declining level of literacy (11) and mathematics (12). This could be due to the difference between levels and growth (i.e., lower growth at higher levels). Unfortunately, the study does not show trend parameters in the comparison years only (the placebo analysis is only reported graphically, and not in table form with all coefficients).
The control for number of days between tests requires somewhat more thought. Under normal circumstances, a larger number of days between tests comes with higher performance due to prolonged learning, but, in the treatment (pandemic) year, the second test was taken later, while performance was lower. This means that the number of days between tests functions as a suppressor variable, as the treatment year is positively associated with number of days between tests, and negatively associated with student performance, while the relationship between number of days and student performance is positive. As the leverage of the (positive) slope originates from the comparison years, a linear control for number of days for all years together is likely not a functional form that would fit the data well, but the model does not allow differential effects by treatment status. The robustness checks in ref. 2 report, indeed, that the negative effect of school closures is reduced by 12% if the control variable is omitted.
The study of Engzell et al. (2) demonstrates how much can be learned about student progression during the pandemic if the data infrastructure is up to the task. It uses well-known externally validated and standardized tests, made available to the researchers through an alliance with a provider of analytics services to schools. Recently, other data on the same and other student assessments have become available for researchers. Building upon new secure register-based school career data of the National Cohort Study on Education (Nationaal Cohortonderzoek Onderwijs) (13), student monitoring data can be connected to further school careers, and student socioeconomic and migration background, in more detail than in ref. 2. Recent reports on the effects of the school closure using these data show comparable results (14): a decline of between 14% (spelling) and 25% (reading) relative to prepandemic years, and strong socioeconomic gaps in the decline also in more fine-grained measures. Gaps by migration background (net of socioeconomic background) are smaller but found for reading.
Importantly, the rich Dutch data infrastructure also enables researchers to closely monitor student progression in the years to come. To help students get back on track, schools also can be informed regularly about their students’ performance in comparison to schools that have a wide range of similar characteristics. Moreover, researchers can study school careers beyond primary and secondary education. This is relevant, as the effects of school closures may be different across the school career. From a sociological perspective, primary education is mostly concentrated on the foundation of basic skills, secondary education is mostly concentrated on differentiation in learning trajectories, and tertiary education is mostly focused on credentialization: acquiring qualifications that provide access to the labor market (Fig. 1). Impacts of school closures on inequalities in learning (the dotted arrows) may be most prominent in primary education, because the foundation is laid for future learning progression. Closures can furthermore impact socioeconomic inequalities in differentiation in secondary education, especially when the transition from primary to secondary education happens after periods of closure, or when standardized tests have been cancelled during school closures (15). Swiss research showed that online instruction during school closures was equally as effective as in-school teaching at the secondary level, while primary education was more negatively affected by online instruction (16). Inequalities in tertiary education may also be less strongly affected by closures, as the credentialization process continues. A study on a 1968 French protest, which led students to be given their upper-secondary baccalauréat diploma easily, showed that these students have not suffered from it. More of them graduated from university, and they received higher earnings compared to a counterfactual when the diploma had been as selectively awarded as before (17).
Fig. 1.
Possible school closure effects for the educational career.
Governments and schools face a heavy task: How can the delay in learning be remedied? Some governments are willing to spend substantial funds to this challenge. The Dutch government has, for instance, allocated EUR 8.5 billion over 2.5 y, or 3.4 billion per year, which equals around 8% of the yearly budget for the Ministry of Education, Culture and Sciences. A high-quality data infrastructure can help the field to understand the effectiveness of promising (inequality-reducing) interventions such as offering after-school tutoring (18), enrolling students in summer programs (19), and employing additional teachers and teacher assistants (20), provided that such interventions are focused and intensive as these are known conditions for success. Countries across the globe can benefit from the lessons learned from such interventions, provided that a solid research infrastructure exists to monitor their effects.

Acknowledgments

H.G.v.d.W.’s research is supported by a Vici Grant by the Netherlands Organisation for Scientific Research, Grant 453-14-017.

References

1
N. Haug et al., Ranking the effectiveness of worldwide COVID-19 government interventions. Nat. Hum. Behav. 4, 1303–1312 (2020).
2
P. Engzell, A. Frey, M. D. Verhagen, Learning loss due to school closures during the COVID-19 pandemic. Proc. Natl. Acad. Sci. U.S.A. 118 e2022376118 (2021).
3
J. E. Maldonado, K. De Witte, “The effect of school closures on standardised student test outcomes” (Discussion Paper DPS20.17, KU Leuven Department of Economics, 2020).
4
S. Rose et al., “Impact of school closures and subsequent support strategies on attainment and socio-emotional wellbeing in Key Stage 1” (Interim Pap. 1, National Foundation for Educational Research, 2021).
5
M. Castro et al., Parental involvement on student academic achievement: A meta-analysis. Educ. Res. Rev. 14, 33–46 (2015).
6
U. Samuelsson, T. Olsson, “Digital inequality in primary and secondary education: Findings from a systematic literature review” in Media and Education in the Digital Age: Concepts, Assessments, Subversions, M. Stocchetti, Ed. (Peter Lang, 2014), pp. 41–62.
7
H. van de Werfhorst, E. Kessenich, S. Geven, The digital divide in online education. Inequality in digital preparedness of students and schools before the start of the COVID-19 pandemic. SocArXiv [Preprint] (2020). https://osf.io/preprints/socarxiv/58d6p/ (Accessed 30 March 2021).
8
T. Bol, Inequality in homeschooling during the Corona crisis in the Netherlands. First results from the LISS Panel. SocArXiv [Preprint] (2020). https://doi.org/1031235/osf.io/hf32q (Accessed 30 March 2021).
9
A. Scheerder, A. van Deursen, J. van Dijk, Determinants of Internet skills, uses and outcomes. A systematic review of the second- and third-level digital divide. Telemat. Inform. 34, 1607–1624 (2017).
10
R. Engelen, P. Roumans, M.-A. Keizer, I. Jongen, Wetenschappelijke verantwoording taalverzorging groep 6 tot en met 8 [Scientific Justification Language Assessment Grades 6 to 8] (Cito, 2016).
11
J. Gubbels, A. Netten, L. Verhoeven, Vijftien jaar leesprestaties in Nederland [Fifteen Years of Reading Performance in the Netherlands] (Radboud University, 2017).
12
M. Meelissen, E. Hamhuis, L. Weijn, Leerlingprestaties in de exacte vakken in groep 6 van het basisonderwijs: Resultaten TIMSS-2019 [Student Performance in Science and Mathematics in Grade 6 of Primary Education: Results TIMSS-2019] (Universiteit Twente, 2021).
13
C. Haelermans et al., Using data to advance educational research, policy, and practice: Design, content, and research potential of the Netherlands Cohort Study on Education. Eur. Sociol. Rev. 36, 643–662 (2020).
14
C. Haelermans, M. Jacobs, L. Van Vugt, “Factsheets eerste schoolsluiting n.a.v. COVID-19 crisis. Technische toelichting [Factsheets first school closure in the COVID-19 crisis. Technical supplement]” (Netherlands Initiative for Educational Research, 2021). https://www.nationaalcohortonderzoek.nl/factsheets. Accessed 30 March 2021.
15
S. Geven, Track recommendation procedures and the cancellation of the school leavers’ test during the COVID-19 pandemic in the Netherlands. SocArXiv [Preprint] (2020). https://doi.org/1031235/osf.io/rdw3m (Accessed 30 March 2021).
16
M. J. Tomasik, L. A. Helbling, U. Moser, Educational gains of in-person vs. distance learning in primary and secondary schools: A natural experiment during the COVID-19 pandemic school closures in Switzerland. Int. J. Psychol., (2020).
17
E. Maurin, S. McNally, “Vive la révolution! Long term returns of 1968 to the angry students” (IZA Discuss. Pap. 1504, Institute of Labor Economics, 2005).
18
J. Dietrichson, M. Bøg, T. Filges, A.-M. Klint Jørgensen, Academic interventions for elementary and middle school students With low socioeconomic status: A systematic review and meta-analysis. Rev. Educ. Res. 87, 243–282 (2017).
19
J. S. Kim, D. M. Quinn, The effects of summer reading on low-income children’s literacy achievement from kindergarten to grade 8: A meta-analysis of classroom and home interventions. Rev. Educ. Res. 83, 386–431 (2013).
20
S. W. Hemelt, H. F. Ladd, C. R. Clifton, Do teacher assistants improve student outcomes? Evidence from school funding cutbacks in North Carolina. Educ. Eval. Policy Anal., (2021).

Information & Authors

Information

Published in

Go to Proceedings of the National Academy of Sciences
Go to Proceedings of the National Academy of Sciences
Proceedings of the National Academy of Sciences
Vol. 118 | No. 20
May 18, 2021
PubMed: 33883271

Classifications

Submission history

Published online: April 21, 2021
Published in issue: May 18, 2021

Acknowledgments

H.G.v.d.W.’s research is supported by a Vici Grant by the Netherlands Organisation for Scientific Research, Grant 453-14-017.

Notes

See companion article, “Learning loss due to school closures during the COVID-19 pandemic,” https://doi.org/10.1073/pnas.2022376118.

Authors

Affiliations

Department of Sociology, University of Amsterdam, Amsterdam 1001 NA, The Netherlands;
Amsterdam Centre for Inequality Studies (AMCIS), University of Amsterdam, Amsterdam 1001 NA, The Netherlands

Notes

Author contributions: H.G.v.d.W. wrote the paper.

Competing Interests

Competing interest statement: H.G.v.d.W. is chair of the Programme Committee of the National Cohort Study on Education (Nationaal Cohortonderzoek Onderwijs) that is mentioned in the paper.

Metrics & Citations

Metrics

Note: The article usage is presented with a three- to four-day delay and will update daily once available. Due to ths delay, usage data will not appear immediately following publication. Citation information is sourced from Crossref Cited-by service.


Citation statements




Altmetrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited by

    Loading...

    View Options

    View options

    PDF format

    Download this article as a PDF file

    DOWNLOAD PDF

    Get Access

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Personal login Institutional Login

    Recommend to a librarian

    Recommend PNAS to a Librarian

    Purchase options

    Purchase this article to get full access to it.

    Single Article Purchase

    Inequality in learning is a major concern after school closures
    Proceedings of the National Academy of Sciences
    • Vol. 118
    • No. 20

    Media

    Figures

    Tables

    Other

    Share

    Share

    Share article link

    Share on social media