Deep models of superficial face judgments

To explore the structure of attribute inferences, we first computed the correlation between the mean face ratings for each pair of attributes (Fig. 1). Many attributes were highly correlated, including happy–outgoing (r = 0.93) and dominant–trustworthy ($r=-0.81$), while others were largely unrelated, including smart–attractive (r = 0.01), smart–trustworthy (r = 0.02), liberal/conservative–believes in god (r = 0.08), and electable–attractive (r = 0.05).

Although some of these correlations are consistent with previous findings (8), others are not. First, although past work has found that judgments of trustworthiness and dominance are often negatively correlated, the correlation is generally small (on the order of –0.2), whereas the correlation observed here (–0.81) was much stronger (4, 28). Second, judgments of smartness or competence have been found to be highly positively correlated with judgments of attractiveness and trustworthiness (with values as high as $\approx 0.8$), whereas we found only marginal correlations between those attribute inferences (8). One explanation for these discrepancies is that the face stimuli used here are more diverse than in the comparison studies, especially with respect to age; the present study includes (simulated) childrens’ faces. This explanation is plausible given that the correlational structure of judgments of children’s faces is different from the structure of judgments of adult faces (29). To probe this hypothesis, we recomputed interattribute correlations on subsets of the data with restricted age ranges (SI Appendix, Fig. S9). We found that the inclusion of the children’s faces partially explains some discrepancies (e.g., smart–attractive) and does not explain others (trustworthy–dominant). Third, memorable faces were more attractive, as observed in the positive correlation between the respective ratings (Fig. 1). This finding is inconsistent with work showing that actual memorability of faces is negatively correlated with attractiveness to the extent that predictions of memorability are veridical (30). Last, familiar faces were seen as more attractive and average-looking, consistent with the finding that average faces tend to be perceived as more attractive (ref. 31, but see ref. 32).

The attribute outdoors (whether the photo appeared to be taken outdoors or indoors) was included to assess potential confounds in using naturalistic face photos. It was found to be the least correlated with other attributes, having the lowest per-attribute maximum absolute correlation (outdoors–electable, r = 0.20). In comparison, the attribute with the next-lowest maximum was skinny/fat (skinny/fat–attractive, r = 0.43), which despite having twice the magnitude was one of the easier attributes to predict (Fig. 2). Furthermore, that outdoors had the lowest mean absolute correlation with all other attributes (r = 0.08) indicates minimal contribution of contextual effects due to naturalistic backgrounds and lighting.

To model an attribute, we start with the high-dimensional representation vectors ${\mathbf{z}}_i=\{z_1,\dots \hspace{0.17em}{z}_d\}$ assigned to each synthetic face i in our stimulus set by a pretrained state-of-the-art GAN (21, 22, 33). The GAN has learned a mapping from each such vector to an image through extensive training on a large database of real, nonsynthetic face photographs (Methods and Materials). We then model each psychological attribute, measured via average ratings y_i, as a linear combination of features: $y_i=w_0+w_1{z}_1+\dots +w_d{z}_d$. The vector of weights ${\mathbf{w}}_k=\{w_1,\dots \hspace{0.17em}{w}_d\}$ represents the attribute as a linear dimension cross-cutting the representational space and is fit using cross-validated, L2-regularized linear regression. A diagram summarizing the modeling pipeline for predicting attribute inferences is provided in SI Appendix, Fig. S1.

Average cross-validated (i.e., out-of-sample) model performance for each attribute is reported in Fig. 2. Prediction for most attributes was reasonably successful, with most R² values ranging from above 0.5 to almost 0.8, with attributes typical, familiar, and gay being the exceptions.

Because participants partly disagree in their appraisals (34), perfect prediction is impossible. To better understand the prediction ceiling imposed by limited interrater reliability, we computed the split-half reliability for each attribute, averaging the squared correlations between the averages of 100 random splits of the ratings for each image. These prediction ceilings vary across attributes and are plotted in Fig. 2 alongside the corresponding model shortfalls they imply. (See Factors Influencing Prediction Performance for a detailed characterization.)

Interestingly, the models of familiar and looks like you showed the smallest gaps between performance and reliability, indicating that their unpredictability is not due to poor model quality or lack of useful input features. Rather, it seems likely that familiar more so than other attributes is based on both a shared concept or experience and a much larger personal concept or experience; only the former can be predicted for participants in aggregate. This is corroborated by a similar effect for the attribute looks like you, which can be predicted only at the aggregate level to the extent that our participant pool has a shared representation of their respective facial features, which may be a byproduct of the participant pool having less diversity in appearance than does the stimulus set.

Attributes corresponding to some racial or ethnic social categories, such as “Black,” exhibited a larger gap between reliability and model performance than did other attributes. One possible reason for this gap is a sampling bias in the stimulus generator. Indeed, rating-distribution violin plots provide some indication that, for example, Black faces were undersampled (SI Appendix, Fig. S3). A second possible reason for the gap is that the degree of undersampling of participants who report membership in a racially or ethnically minoritized group might correlate with the content of the aggregate attribute inferences. However, we observe no such correlation: the second most common participant self-identifier was Black, which had one of the largest gaps between reliability and model performance, whereas attributes corresponding to even less commonly self-reported racial and ethnic social categories had a smaller gap. A third possible reason is that the predominantly White participant pool might have similar stereotypes of all racially or ethnically minoritized groups, leading to comparable predictability across the relevant attributes. However, this explanation is inconsistent with the lack of strong correlation observed across those attributes (Fig. 1). Taken together, this suggests that the stimulus selection contributes more to the observed gap than does the composition of the participant pool. Even so, no firm conclusion can be drawn because we cannot rule out limitations in the representational capacity of the neural network features.

To characterize the factors influencing prediction performance, we first investigated the effect of the number of faces rated on predictive performance (Fig. 3, Top). Performance curves were generated by fitting models for each of 30 random samples of images with sizes ranging from 100 to 1,000. Most attributes benefit from increases in the number of faces rated, with significant variation across them with respect to how performance scales with the number of unique faces that were rated. Interestingly, fewer images were needed to saturate model performance for the attribute feminine/masculine than for most other attributes. For all other attributes, adding additional images improved performance through the full range.

Next, we investigated the relationship between the number of ratings by unique participants obtained for each face stimulus and predictive performance (Fig. 3, Middle). Performance curves were generated by fitting models on down-sampled datasets with sizes ranging from 5 to 30 unique ratings per image, with 30 datasets sampled per size. Aside from attributes feminine/masculine and age, which elicit less disagreement, performance increases considerably as the number of ratings increases for all attributes. Gains due to the number of ratings diminish with increases in the number of unique ratings but at a slower rate than gains due to the number of faces (Fig. 3, Top). It remains to be seen the extent to which scaling the number of rated faces and the number of ratings per face beyond the range explored here accounts for the gap between model performance and the ceiling imposed by interrater reliability.

Finally, we investigated the relationship between the number of image features (512 total) and predictive performance (Fig. 3, Bottom). Performance curves were generated by fitting models using reduced feature sets obtained via principal components analysis, varying the dimensionality between 10 and 512. In all cases, performance saturates quickly but is improved marginally with a greater number of dimensions in some cases. The various profiles of saturation indicate that as few as 10 dimensions of this latent feature space may be enough to account for the bulk of variance in attribute inferences, with a subset of attributes benefiting considerably from higher-dimensional feature representations.

It is possible that the quality of the learned representation from the particular deep neural network we employed was a limiting factor in predictive performance. Factors beyond predictive performance (specifically, the ability to generate images in addition to representing them) guided network selection for the present study. Other architectures with other forms of supervision may offer improvement in predictive performance. For example, there is evidence that identity-supervised models provide representations that are highly predictive of diverse attribute information (26, 35, 36).

Because the learned attribute vectors correspond to linear dimensions, we can manipulate an arbitrary face represented by features ${\mathbf{z}}_i$ with respect to attribute k using vector arithmetic: ${\mathbf{z}}_i+\beta {\mathbf{w}}_k$, where β is a scalar that controls the positive or negative modulation of the attributes. We apply a symmetric range of β around 0 to each attribute vector to manipulate a series of base face representations in both the negative and positive directions and decode the results for visualization using the same decoder/generator component of the neural network that was used to derive representations (see SI Appendix for more details).

The results of these transformations for six sample attribute inferences are shown in Fig. 4. The manipulations are strikingly smooth and effective along each attribute dimension. For example, modulating trustworthiness increases features associated with perceptions of trustworthiness, such as eye gaze, degree of smiling, face shape, and facial femininity (4, 37). Manipulations of attribute inferences may affect more than one dimension of appearance. For example, increasing smartness may add glasses or change the facial expression. Increasing outgoingness may increase smiling, as expected, but also give glasses a more rounded and cartoonish appearance. Other dimensions allow for greater levels of extrapolation. For example, faces can be made considerably skinnier or fatter than any examples in the dataset, yet still maintain a realistic appearance. Faces with strongly manipulated happiness also resemble convincing caricatures.

It is possible to manipulate one dimension s (e.g., smartness) while controlling for another t (e.g., trustworthiness) by creating an orthogonal vector, subtracting the projected component of the dimension to be controlled for:

An example of such transformations controlling for trustworthiness on a given exemplar face can be seen in SI Appendix, Fig. S11.

Note that the attribute-inference manipulations can affect both internal facial features and external features. When only internal face features are altered, it is not because the GAN manipulates only internal features but because the external features are orthogonal or irrelevant to that attribute inference in the region of the manipulated face.

Do the attribute models generated above reliably change participants’ impressions of faces transformed with them? To answer this question, we ran a series of 20 preregistered experiments with over 1,000 participants to verify that our models can indeed manipulate attribute impressions in observers. Each of the experiments paired one of two face image types (artificial vs. real) with one of 10 different attribute dimensions, chosen to represent a wide range of different model performances and levels of objectivity/subjectivity (age, feminine/masculine, skinny/fat, trustworthy, attractive, dominant, smart, outgoing, memorable, and familiar). Like in the attribute-modeling experiments, for the artificial face experiments we generated 50 unique synthetic faces at random using StyleGAN2 (21, 22), a state-of-the-art GAN architecture (SG2). However, for the real-face experiments, we encoded into our model 50 unique face photograph, chosen from a commonly used database of real faces from the psychological literature (38). On each trial, participants were shown a single face and asked to rate it. Critically, each face image was transformed by one (experimentally assigned) perceived attribute model to evoke one of three levels of that impression; faces could be set to the mean observed value of the perceived attribute or at ± 0.5 SD from that mean. Every face was shown at every level of the assigned attribute (50 identities × 3 transformation levels $=150$ unique faces), and once again, 20% of trials were repeated to measure test–retest reliability (in order to exclude subjects with negative such reliability) for a total of 180 trials. If the attribute model transformations indeed change participants’ impressions of the faces, then we should observe that participants’ ratings of the faces increase with increasing levels of the manipulation. We found just that: repeated measures ANOVAs revealed that all of the manipulations yielded highly significant results [all $F(2,98)\text{s}>14.13$, all values of P < 0.000005, all values of ${\eta }^2>$ 0.223]. Critically, all of the experiments’ data showed a strongly significant positive linear trend [all values of t(98) > 2.69, all values of P $\le$ 0.008, with the exception of real faces manipulated along the familiar attribute dimension, t(98) $=-1.60$, P $=0.112$; see SI Appendix, Fig. S12 for a swarmplot of all the data].*

Importantly, while the primary goal of this work is to support scientific modeling, the framework developed here adds significantly to the ethical concerns that already enshroud image manipulation software. In contrast to traditional photo editing, which may be limited in effectiveness by the intuitions of a particular artist, the current method may be more accurate and at the least is faster and more efficient through its automation. Further, in contrast to other methods making use of deep neural networks such as DeepFakes (40), which can affect the social perception of an individual by placing them in an unwanted or compromising context (e.g., superimposed on a body in an arbitrary target image), our model can induce (perceived) changes within the individual’s face itself and may be difficult to detect when applied subtly enough. We argue that such methods (as well as their implementations and supporting data) should be made transparent from the start, such that the community can develop robust detection and defense protocols to accompany the technology, as they have done, for example, in developing highly accurate image forensics techniques to detect synthetic faces generated by SG2 (41, 42). More generally, to the extent that improper use of the image manipulation techniques described here is not covered by existing defamation law (43, 44), it is appropriate to consider ways to limit use of these technologies through regulatory frameworks proposed in the broader context of face-recognition technologies (45, 46).

There is also potential for our data and models to perpetuate the biases they measure, which are first impressions of the population under study and have no necessary correspondence to the actual identities, attitudes, or competencies of people whom the images resemble or depict. While the bias in our sample of raters comes from the same population as most crowdsourced studies, it may be particularly important to understand in the context of social attribute perception, given that it consists of primarily White participants from the United States. Further, we found that the generative model that synthesized our stimuli, while highly diverse, nonetheless undersamples faces of Black people and other minoritized groups. Applications of the model will thereby produce face images that are more closely aligned with this bias than are the original inputs. We quantify part of this bias beyond participant demographics primarily using the White, familiar, and looks like you attributes, all of which are moderately correlated (Fig. 1).

Modern data-driven methods from machine learning provide new tools for representing and manipulating complex, naturalistic stimuli but are not explicitly designed to model or explain human mental representations. However, applying the same “big data” philosophy to behavioral experiments allows us to align these powerful models with human perception. The deep models of superficial face judgments that we explore in this paper can in turn be used to broaden the range of behavioral data we can collect because they define an infinite set of realistic and psychologically controlled stimuli for a new generation of behavioral experiments.

Our experiments make use of 1,004 synthetic yet photorealistic images of faces generated using SG2. The generator network component of SG2 models the distribution of face images conditioned on a 512-dimensional, unit-variance, multivariate normal latent variable. When a vector is sampled from this distribution and passed through the network, it is mapped to a second, intermediate 512-dimensional representation (for which the distribution is unknown), which is in turn fed through multiple layers and ultimately mapped to an output image resembling those from the dataset on which the model was trained. Thus, either of the two 512-dimensional representations can be used for our modeling applications, each associating one fully descriptive (latent) feature vector with each face. We used the latter representation throughout because it yielded superior results in all analyses. Specifically, we use these representations from a pretrained model that was trained on the Flickr-Faces-HQ Dataset (21), containing 70,000 high-quality images at a resolution of 1,024 × 1,024 pixels. Images generated by this model are rendered at the same resolution.

The synthetic faces generated by SG2 are diverse and convincingly realistic in most cases but can occasionally contain visual artifacts that appear odd or even jarring. We minimized these artifacts in our dataset using two strategies. First, SG2 employs a parameter ψ for posttraining image generation that bounds the norm of each multivariate input sample and, as a result, trades off between sample diversity and sample quality. We set ψ to 0.75, which by inspection appeared to jointly maximize the criteria for our purposes. Second, we manually inspected and filtered the generated images, removing all instances that contained obviously distorted faces, multiple faces, hands, localized blotches of color, implausible headdress, or any particularly notable visual artifact. Specifically, we sampled ∼10,000 512-dimensional normal vectors, fed them through the generator network of SG2 to obtain 10,000 candidate face stimuli for our dataset, and took the first $\approx 1,000$ that met the criteria for quality. Random examples from the stimulus set are provided in SI Appendix.

For the model-validation studies, 50 real face identities were generated by encoding 50 faces from the Chicago Face Database (CFD) (38). The CFD faces used in these real-face experiments were roughly balanced in terms of the four races and two genders available in the main stimulus set. The final set included 12 East Asian, 14 Black, 12 Latin American, and 12 White faces (with equal numbers of male and female faces in each racial group). The 50 faces used in the artificial face experiments were chosen via the same procedure as in the previous attribute modeling studies. Each unique face identity was then transformed along one of 10 perceived attribute dimensions (age, feminine/masculine, skinny/fat, trustworthy, attractive, dominant, smart, outgoing, memorable, and familiar) at three levels of the attribute (–0.5 SD, 0 SD, and +0.5 SD from the mean ratings observed in the attribute model studies). This yielded 150 unique images per model-validation study and therefore 3,000 unique images in total.

For the attribute model studies, we used Amazon Mechanical Turk to recruit a total of 4,157 participants across 10,974 sessions, of which 10,633 ($\approx 97\%$) met our criteria for inclusion (SI Appendix, Data Quality). Participants identified their gender as female (2,065) or male (2,053), preferred not to say (21), or did not have their gender listed as an option (18). The mean age was ∼39 y old. Participants identified their race/ethnicity as either White (2,935), Black/African American (458), Latinx/a/o or Hispanic (158), East Asian (174), Southeast Asian (71), South Asian (70), Native American/American Indian (31), Middle Eastern (12), Native Hawaiian or Other Pacific Islander (3), or some combination of two or more races/ethnicities (215). The remaining participants either preferred not to say (22) or did not have their race/ethnicity listed as an option (8).

For the model-validation studies, we recruited a total of 1,022 workers from Amazon Mechanical Turk via CloudResearch (47), of which 1,000 (∼98%) met our criteria for inclusion. Of those, 18 participants were excluded for low test–retest reliability, one was excluded for participating in the experiment twice, and three were overrecruited beyond our target sample size of 1,000. Participants identified their gender as female (530) or male (484), preferred not to say (3), or did not have their gender listed as an option (5). The mean age was ∼42 y old. Participants identified their race/ethnicity as either White (781), Black/African American (77), Latinx/a/o or Hispanic (37), East Asian (37), Southeast Asian (11), South Asian (12), Native Hawaiian or Other Pacific Islander (3), or some combination of two or more races/ethnicities (57). The remaining participants either preferred not to say (3) or did not have their race/ethnicity listed as an option (4).

The Institutional Review Board at Princeton University approved both sets of studies. Participants provided informed consent before beginning the study.

For the attribute model studies, we used a between-subjects design where participants evaluated faces with respect to each attribute. Participants first consented. Then they completed a preinstruction agreement to answer open-ended questions at the end of the study. In the instructions, participants were given 25 examples of face images in order to provide a sense of the diversity they would encounter during the experiment. Participants were instructed to rate a series of faces on a continuous slider scale where extremes were bipolar descriptors such as “trustworthy” and “not trustworthy.” We did not supply definitions of each attribute to participants and instead relied on participants’ intuitive notions of each.

Each participant then completed 120 trials with the single attribute to which they were assigned. One hundred of these trials displayed images randomly selected (without replacement) from the full set; the remaining 20 trials were repeats of earlier trials, selected randomly from the 100 unique trials, which we used to assess intrarater reliability. Each stimulus in the full set was judged by at least 30 unique participants.

At the end of the experiment, participants were given a survey that queried what participants believed we were assessing and asked for a self-assessment of their performance and feedback on any potential points of confusion, as well as demographic information such as age, race, and gender. Participants were given 30 min to complete the entire experiment, but most completed it in under 20 min. Each participant was paid $1.50.

The model-validation studies followed an identical procedure to that of the attribute modeling studies above, except where noted below.

Each participant completed one of 20 preregistered experiments involving a pair of one of two different face image types (artificial vs. real) with manipulations along one of 10 different attribute dimensions (age, feminine/masculine, skinny/fat, trustworthy, attractive, dominant, smart, outgoing, memorable, and familiar). In each experiment, observers rated 150 unique images (with 30 repeats) along the assigned perceived attribute dimension. Each participant was paid $2.00 due to the longer experiment duration.

To broadly capture human face attribute perception, a model should accurately reproduce human judgments about the attributes of natural faces. More formally, we seek a function $\varphi {(·)}_{PE}$ (what we call a “psychological encoder”) that maps from any possible face stimulus ${\mathbf{x}}_i=\{x_1,\dots \hspace{0.17em}{x}_m\}$ (i.e., an m-dimensional vector of raw pixel intensities) to a given psychological attribute inference (average judgment for face ${\mathbf{x}}_i$):

We further define $\varphi {(·)}_{PE}$ as a decomposition of functions:where $\varphi {({\mathbf{x}}_i)}_F={\mathbf{z}}_i=\{z_1,\dots \hspace{0.17em}{z}_d\}$ is a rich feature representation of face stimulus ${\mathbf{x}}_i$ and $\varphi {(·)}_S$ maps these features to psychological dimensions of interest. This formulation allows us to leverage state-of-the-art neural networks to featurize arbitrary, complex face images.

We then relate these features ${\mathbf{z}}_i$ to psychological features by assuming that $\varphi {(·)}_S$ is a linear function and thereby implying that each attribute is a 512-dimensional (potentially sparse) vector in the overall feature space. The function $\varphi {(·)}_S$ is learned from human attribute judgment data. In particular, given continuous-scale attribute judgments (i.e., degree of trustworthiness on a scale from 1 to 100), we use linear regression to map 512-dimensional feature vectors ${\mathbf{z}}_i$ to average attribute ratings y_i:

In both cases, weight vector ${\mathbf{w}}_k=\{w_1,\dots \hspace{0.17em}{w}_d\}$ represents a single attribute k as a linear factor. Therefore, at the heart of our model is a matrix $W\in I\text{}{R}^{k\times d}$, a set of d-dimensional linear factors for each of the k attributes, each obtained by fitting a separate linear model.

The above components of the model enable predictions of attributes to be made for arbitrary face stimuli. We further desire the flexibility to manipulate these attributes for a given face. Because we represent each attribute as a vector ${\mathbf{w}}_k$ in the feature representation space, we can manipulate each face in this space (i.e., represented by ${\mathbf{z}}_i$) using vector addition:where ${\mathbf{z}}_i^{\prime }$ is the new transformed face and β is a scalar parameter that controls the strength of the transformation, which can be positive or negative. When $\beta =0,\hspace{0.17em}{\mathbf{z}}_i^{\prime }={\mathbf{z}}_i$, and no transformation takes place. In other words, β scales the attribute vector that is added to the given face representation. Finally, in order to generate a new stimulus corresponding to our transformation, the inverse featurizer (i.e., decoder/generator network of SG2) ${\varphi }^{-1}{(·)}_F$ is employed to map from features ${\mathbf{z}}_i$ back to a face stimulus ${\mathbf{x}}_i$, such that manipulation of face images can be fully described bywhere ${\mathbf{x}}_i^{\prime }$ is the attribute-transformed version of input face ${\mathbf{x}}_i$.

The success of the above formulation (i.e., good prediction of human attribute judgments for arbitrary faces) is highly dependent on the choice of the feature encoder $\varphi {(·)}_F$, which abstracts over raw pixels and provides the basis for modeling attributes. If the features are not sufficiently expressive, the model will fail to make good predictions of human attribute judgments. Likewise, the ability of the inverse function ${\varphi }^{-1}{(·)}_F$ to generate face stimuli given their feature representations determines whether attribute-transformed face stimuli will successfully avoid the uncanny valley effect. There are many modern neural networks that could make for a good choice of featurizer $\varphi {(·)}_F$. For example, convolutional neural networks, which learn hierarchies of translation-invariant features, can be trained to classify faces to a high level of accuracy, and their hidden representations can be taken as a feature representation z. However, this method does not yield an inverse function from features back to stimuli and attempts to invert models after the fact often introduce artifacts (48).

Instead, we selected a model primarily aimed at solving the inverse problem alone. GANs are a form of deep latent variable model that learn to model a distribution of images using two components: a generator network that generates images by mapping Gaussian noise to (synthetic) images and a discriminator network that discriminates between real and generated data. When properly trained in a way that balances the two components, the discriminator network forces the generator to produce realistic images, and the discriminator can no longer distinguish between real and generated images. SG2, described earlier, is one of the most successful applications of this model structure and training paradigm; it includes several key improvements that yield highly convincing results (see example faces in SI Appendix).

SG2 yields only the inverse function $\varphi {({\mathbf{x}}_i)}_F^{-1}$, a learned convolutional generator or decoder function which maps from features to images. In order to apply our model to arbitrary face images outside of our set of 1,004, inverting this function is required. While the authors of SG2 supply their own solution to this problem, we find that it is not accurate enough for our purposes. Instead, we define an encoder function and featurizer $\varphi {({\mathbf{x}}_i)}_F$ as an optimization process that searches via gradient descent for the vector input to SG2 that produces an output image with a good likeness to the one we wish to featurize. This likeness is defined as Euclidean distance in the feature space of another external convolutional network pretrained to recognize faces (20). Additionally, because this process is slow, we initialize the image-encoding vector using a first-pass approximation from yet another convolutional neural network that we trained to regress thousands of SG2 image samples to the output vectors that generated them. This encoder is much less accurate, but much faster, and drastically speeds convergence of the slower and more accurate decoding process outlined above.

A summary diagram of our modeling pipeline is provided in SI Appendix, Fig. S1.

All linear regression models were fit using the least squares algorithm. Because image feature representations (i.e., vectors of predictors in the design matrix) are high-dimensional, there is a significant risk of overfitting, which could potentially result in suboptimal or meaningless model solutions. To address this, we use ridge regression, which penalizes solutions ${\mathbf{w}}_k$ that have a large euclidean distance from the 0 vector. The strength of this penalty and its influence on the resulting solution is controlled by a free parameter λ. We search for the optimal value of this parameter based on the generalization performance of the model, specifically using 10-fold cross-validation. All reported model scores are averages over those for each of the 10 folds, such that we never report performance on data that was used to fit our models.