What failure to predict life outcomes can teach us

Prediction is not a typical goal in the social sciences despite recent arguments that it should be (6). Social scientists focus on inference: that is, understanding how an outcome is related to some input. The researcher selects a few inputs, specifies a parametric (often linear) model to connect inputs to the outcome, and estimates the parameters from data. The result is a simple and interpretable model that performs well in the sample at hand. In SML, by contrast, the researcher includes many inputs, considers flexible (often nonparametric) models linking inputs to the outcome, and picks the model that best predicts the outcome in new data. The result is a complex model that might perform well out of sample but often offers little insight into the mechanism linking inputs to the outcome.

Recent work connects these two cultures in different ways (7). First, researchers identify prediction tasks within the classical statistical framework and use SML to improve inference (8, 9). Second, scholars use predictions as a starting point to develop new theory (10). As SML becomes more mainstream in the social sciences and more students are drawn to seeking training in it, a crucial question remains unanswered. Apart from its use in well-delineated tasks, how helpful is SML in predicting human behaviors and outcomes?

Salganik et al. (5) offer a leap forward in evaluating the limits and promise of SML in the social sciences. The authors use the common task framework (11), where research teams compete on the same task, and launch a mass collaboration called the Fragile Families Challenge that is one of the first of its kind in the social sciences (12). The common task, in this case, is to predict six life outcomes in the Fragile Families study, a rich data source tracking over 4,000 families since the year 2000. The 160 research teams have full access to the first five waves of the study, which contain background variables, and partial access to the sixth wave, which measures outcomes to be predicted, such as children’s grade point average or households’ eviction experience. The teams use these data to train predictive models with any (SML or other) technique of their choice and are judged by a common metric (mean squared error) evaluating the accuracy of their predictions in the unseen part of the sixth wave that is available only to the challenge organizers.

The results are striking. The submitted models, regardless of the method used, all perform rather poorly in predicting the outcomes in the held-out data. Indeed, even the best predictions are not remarkably better than those offered by a simple benchmark model that employs standard regression models with a few expert-chosen inputs. The results produced by 160 independent teams using myriad strategies are clearly not an artifact of any one method and suggest that SML tools offer little improvement over standard methods in social science data.

This exercise has many lessons to teach us. First, if research teams went on their own to predict life outcomes with SML, their results (or nonresults) would probably never see the light of day. The common task method, in this instance, provides a solution to the well-known file drawer problem, whereby only positive results appear in scientific publications. Social scientists can use this method and follow the example set by the Fragile Families Challenge to ask fundamental questions of interest and to produce collective answers that are not bound by publication bias.

Second, the predictive framework shows us the importance of out-of-sample testing in social sciences, even when our goal is not prediction per se. When we estimate and evaluate a model on the same sample, we run the risk of overfitting: that is, capturing the noise as well as the signal in data. The Fragile Families Challenge illustrates how humbling it can be to evaluate our models in new data. Whether our goal is prediction or not, we can use out-of-sample testing (or cross-validation procedures to efficiently partition data) to minimize overfitting (or p hacking) (13) and to assess the true performance of our models as well as the underlying theories (6). The recent push toward independent replication in social sciences represents a move in a similar direction (14, 15).

Third, the results from the challenge suggest what might be worth exploring further. There is a curious pattern in the data that requires more scrutiny. The prediction error from the competing models is strongly correlated with the family being predicted and only weakly related to the method being used. To put it differently, the predictions are accurate for most people in the data, regardless of the method used. However, the predictions are consistently off for a subset of families across all methods. This pattern begs the question: are we missing things essential to some families in our surveys? Is the problem, in other words, with our data as much as our methods?

Fourth, and the final point is also related to the data. The Fragile Families survey is a good representation of the kind of data typically available to social scientists in terms of its size and breadth. However, the data might be too limited for SML to truly shine. Future work needs to continue to evaluate these tools for the social sciences using larger datasets.

To experts of SML, the unpredictability of life outcomes might come as a surprise, one that, for now, can be attributed to limitations of the Fragile Families data. However, what else could this observation mean? Salganik et al. (5) offer several useful insights. Our failure to predict life outcomes might mean that our survey measures (driven by our current theories) fall short of capturing relevant dimensions of people’s lives, or it might mean that life outcomes are too idiosyncratic and subject to a predictability ceiling.

The fact that we cannot predict life outcomes, however, does not mean that we have little understanding of them. Our data and models might not allow us to forecast outcomes for each and every individual, but they can still help us produce aggregate descriptions [for example, of racial differences in school performance (16)] or identify causal relationships [such as the effect of education on earnings (17)]. Indeed, even if we were able to predict life outcomes with high accuracy, we would still want to isolate the mechanisms linking inputs to these outcomes, to ask counterfactual questions, and to design potential interventions. The predictive framework of the Fragile Families Challenge, in other words, does not replace or invalidate our standard inferential approach but rather, complements it.

Overall, the Fragile Families Challenge is a breakthrough in the social sciences for setting an example for mass collaboration and for evaluating the predictability of life outcomes in a high-quality longitudinal survey. The failure to predict individual and family outcomes, in this case, is anything but disappointing. It teaches us the value of collectively attacking a core question and subjecting our models to rigorous out-of-sample testing. It also reveals what our data (and the ideas on which they are based) might be missing and thus, charts a fruitful direction for future work.