New Research In
Physical Sciences
Social Sciences
Featured Portals
Articles by Topic
Biological Sciences
Featured Portals
Articles by Topic
 Agricultural Sciences
 Anthropology
 Applied Biological Sciences
 Biochemistry
 Biophysics and Computational Biology
 Cell Biology
 Developmental Biology
 Ecology
 Environmental Sciences
 Evolution
 Genetics
 Immunology and Inflammation
 Medical Sciences
 Microbiology
 Neuroscience
 Pharmacology
 Physiology
 Plant Biology
 Population Biology
 Psychological and Cognitive Sciences
 Sustainability Science
 Systems Biology
Symmetry, probability, and recognition in face space

Communicated by Mitchell J. Feigenbaum, The Rockefeller University, New York, NY, December 12, 2008 (received for review June 9, 2008)
Abstract
The essential midline symmetry of human faces is shown to play a key role in facial coding and recognition. This also has deep and important connections with recent explorations of the organization of primate cortex, as well as human psychophysical experiments. Evidence is presented that the dimension of face recognition space for human faces is dramatically lower than previous estimates. One result of the present development is the construction of a probability distribution in face space that produces an interesting and realistic range of (synthetic) faces. Another is a recognition algorithm that by reasonable criteria is nearly 100% accurate.
Visual space may be conveniently regarded as a tableau of gray levels assigned to a square of O(10^{5}) pixels. Human faces belong to a subspace of this highdimensional space, and the representation/identification of faces termed the Rogue's Gallery problem (1, 2) can be carried out in the reduced space, defined by the masks of Fig. 1. Dimension might be halved by exploiting the latent symmetry of faces and associating each face with its midline symmetrized counterpart (see Fig. 1A). Recent laboratory experiments strongly imply that the primate visual system adopts this right/left symmetry, and, moreover, does so for a full range of poses (3); in fact, right and left poses equally angled from the frontal position give rise to equal neuronal response. As will be demonstrated, this thread leads to an exceptional dimension reduction. After a correction for the illumination artifact, a probabilistic exploration of face space leads to a Gaussian probability description in terms of appropriately transformed variables. One result of this is that synthetic faces, i.e., faces drawn from the probability distribution, result in realistic human faces that pass the test of visual inspection. This is illustrated (see Fig. 5) by a small range of reasonable faces, drawn from the probability distribution and are, hence, faces of people we will never encounter.
Mounting evidence indicates that in the primate brain, appreciable resources involving many cortical areas are dedicated to face perception (3–7). As emphasized in the last reference, (see also ref. 8), this provides compelling evidence for the modular character of face recognition in support of the single “domainspecific” side in the cognitive science debate on whether mechanisms are specific or general. As a consequence, as has been observed in many studies faces are our biologically most significant stimuli. For this reason, the role of the Rogue's Gallery problem takes on an importance well beyond its practical applications. Another consequence of these deliberations impacts on the facerecognition debate on whether a face should be parsed or viewed holistically (9). In early physiological studies on macaque, it was found that elements of incomplete faces elicited little or no response, in contrast to the robust response produced by a complete face (10). This is supported by a recent human psychophysical investigation (11) showing dramatic responses to fullface stimuli and virtually no effect arising from partial faces. These studies provide evidence that the human visual system chooses the holistic approach taken here, although for purposes of machine face recognition, parsing may yet prove more efficient.
On a practical note, this investigation leads to a recognition algorithm which, by reasonable criteria, gives nearly perfect frontal identification even under naturally diverse lighting conditions.
Background
Facerecognition investigations are largely based on derivatives of pixel image representations in the form of an empirical eigenfunction decomposition. Specifically, if {f_{n}(x)} denotes an ensemble of faces in terms of gray level f at pixel locations x, then a face can generally be represented by some orthonormal function set {ϕ_{n}(x)}, so that where the coefficients for the jth face are given by the projections If the set {ϕ_{n}} are the eigenfunctions of the covariance operator of the ensemble, these are termed eigenfaces, and it can be shown that for any fixed N, Eq. 1 produces the minimal average Euclidean error. These considerations, which were first applied to the Rogue's Gallery problem (1, 2), have become standard (see, e.g., cited references and www.facerec.org/newpapers/).^{†} Based on a tolerable average error of ≈3%, one can fix N and then regard this as a dimension estimate. For the limited early ensemble (1, 2) of ≈125 faces, this dimension estimate was estimated to be ≈100. A subsequent study with a larger ensemble suggested O(500) (12). Another approach with origins in signal filtering considers the eigenvalue spectrum of the covariance operator as being divided into signal and noise, with each approximately obeying a different power law (13). Applying this in the present case leads to a dimension of N ≈ 175. Such deliberations are based on Euclidean measures and, as such, lie within the framework of a machine recognition perspective. A recent psychophysical study furnished a cortical estimate of face space dimension and found this to be O(100) (13), which as we show, can be further reduced. An interesting feature of that study was the subjective variation in dimension. Perhaps not surprisingly, some subjects appeared to be exceptional in recognizing a face from significantly less information, thus leading to a significantly smaller individual dimension estimate.
Informally, it is reasonable to estimate a tolerable standard deviation in determining a single dimension as σ = 0.1. On estimating this by 1/
Results
A typical face from the population is shown in Fig. 1A. In Fig. 1B, background was removed with a mask; for comparison, the mean face of the ensemble is shown in Fig. 1C; and when the mean is removed from Fig. 1B, the caricature, Fig. 1D, results. Representative eigenfunctions are shown in Fig. 2. The 2 highindex examples show the noisy quality at the tail of the spectrum. The actual population contains 329 individuals, which exploiting the latent symmetry of faces (2), is doubled by including the midline mirror image faces. Recent remarkable experiments (3) strongly suggests that the cortex also makes use of this latent symmetry (also see ref. 14). For sound mathematical reasons, the use of symmetry renders all eigenfaces as even or odd, in the midline (2). This dichotomy is evident in Fig. 2, where odd eigenfaces are indicated by white background numbers.
The corresponding eigenvalue spectrum, in log–log form, is displayed in Fig. 3, and if one is not too exacting, the spectrum can approximately be separated into 2 power law regimes. Low indices (low spatial frequency) are characteristic of signal, and high indices (high spatial frequency) are characterized by noise. The eigenfaces are sorted by the decreasing size of eigenvalues and are found to be wellcorrelated with increasing spatial frequency. The crossing point at N ≈ 175, furnishes an estimate of face space dimension. According to our earlier deliberations, we might need >17,000 exemplars to confidently deal with this dimensionality. Thus, with the available ensembles, we are only sparsely sampling face space. Therefore we explore leveraging the limited data by means of a probabilistic description.
Probability Considerations
The goal is to obtain a representative probability distribution function (pdf) that will reasonably describe face space, so that a face selection based on the pdf results in a corresponding (synthetic) face that is reasonable by the standard of visual inspection. To pursue this thread, we adopt the following null hypothesis: that gray levels of pixels are independent identically distributed random variables.
In Eq. 1, the coefficients are determined by the inner product (Eq. 2), and hence from the form of the eigenfaces, as illustrated in Fig. 2, each coefficient is a sum over many pixels, and therefore under the null hypotheses and the central limit theorem (15) we infer that coefficients {a_{n}^{j}}, (n fixed), should obey a Gaussian pdf.
a_{n}^{j} is calculated as the nth coefficient of the expansion (Eq. 1) of the jth face in the population (Eq. 2), and we can estimate the pdf of a_{n} by As pointed out in ref. 16, this is the basis of a maximumlikelihood estimate, a zerobias estimate and a bootstrap procedure. For our purposes Eq. 3 is ideally suited to dealing with the difficulties of relatively small amounts of data. We denote the cumulative pdf of Eq. 3 by C*, and use the zerobias estimate to construct the cumulative Gaussian pdf denoted by C_{G}. The following variant of the Kolmogorov–Smirnov (KS) statistic, is taken as the criterion statistic for normality of the pdf. For the ensemble under consideration the results are depicted in Fig. 4. In Fig. 4B, the KS statistics of the coefficients (Eq. 2) are presented, with even (red dots) and odd coefficients (blue dots) distinguished.
Fig. 4 contains diverse statistical information. For example in Fig. 4B the KS statistic (Eq. 4) is displayed for even (red) and odd (blue) coefficients. Clearly for KS small normality of the pdf is implied, and the question of a threshold arises. In this connection, we mention that the appearance of odd eigenfaces at low indices was greeted with surprise in the investigation leading to ref. 1 until it was realized that this was due to the daily variation of sunlight as images were acquired.
It is our contention that if face images could be acquired under ideal conditions of totally diffuse light, then odd eigenfaces would be relegated to the high indices especially if hair is removed, as we have with the mask shown in Fig. 1. Therefore the presence of odd eigenfunctions at low indices is an indicator of the lighting artifact. It is clear from Fig. 4B that after the first red point, the ratio of cumulative blue points to cumulative red points climbs to a sharp maximum as KS is decreased. After this the ratio falls rapidly into basic parity with the red dots. At this point KS = 0.046, and this is taken as the criterion point for normality of the pdf. This is indicated by the horizontal line shown in Fig. 4B. For the relevant range of n < 175 just 2 even eigenface coefficients are nonGaussian. Thus the null hypothesis is refuted for a large portion of the odd eigenfaces and only for 2 even eigenfaces.
Some insight into this result is provided by Fig. 4A which furnishes a pixel map as measured by the KS statistic. Essential to the central limit theorem is that the summed quantity be independent and random. Application of the theorem in this instance depends on correlation lengths, random light sources, pixel size, and ensemble size. Highly correlated regions such as hair and eyes are clearly nonrandom, as indicated in Fig. 4A. Other sources that lack randomness are the mask boundaries, where the vector normal to the original face surface can be nearly perpendicular to incident light.
As has been pointed out in (17) illumination variation very likely exceeds actual face variations, and from our investigation illumination appears to be the greatest source of nonrandom behavior. In most instances lighting for capture of a facial image is frontal, and as Fig. 4A indicates the forehead is a large area of highly nonrandom pixels. The forehead behaves almost as a mirror which, along with the hair artifact, is largely accounted for by eigenfaces 3 and 5 of Fig. 2 (second and fourth even eigenfaces), the 2 nonGaussian cases.
Synthetic Faces
In keeping with the contention that odd eigenfaces are largely the result of the lighting artifact, we explore dropping these from consideration. In addition we drop the 2 special even eigenfaces. The remaining coefficients are regarded as normally distributed with zero mean, and variance proportional to the eigenvalue of the covariance as given in Fig. 3. We can then construct synthetic faces out of just the even eigenfaces by drawing even coefficients {a_{n}^{e}} from the specified appropriate normal probability distribution, thus generating the form
Four exemplars of this construction are exhibited in Fig. 5A–D, where for viewing purposes, the mean has been added to each representative as well as (
We note in passing that the idea of synthetic faces appeared earlier (18), where less realistic “pseudofaces” are used to explore cortical organization of faces.
If the full probability distribution, comprising all odd and even eigenfaces is used in a Gaussian fit, then grotesque images can emerge. To illustrate this we have taken the images of Fig. 5 B and D and added to them the contribution of terms excluded from Eq. 5, using an appropriate Gaussian distribution for the coefficients. These are shown in Fig. 5 F and H, and they are clearly distorted and unnatural. The mottled appearance of the lower row of Fig. 5 can be traced to the odd eigenfaces, which carry localized variations in lighting.
Rogue's Gallery Problem
The even rendering of a face results in a visually recognizable face. This transformation was applied to Fig. 1A clearly without loss of identity. Thus, ignoring odd components removes the lighting artifact without cost to identification. To follow up on these observations, we next explore the face recognition problem. In anticipation of this, additional male images were acquired from the 3 face databases (19–21). Each of these collections contained 3 images of each individual: 1 frontal image with center lighting and 2 frontal images that are marred by variable lighting from the right or left. All centerlit images, 47 in number, were included as inpopulation faces of the male population.^{‡} The corresponding 94 male images with pronounced lighting variations were reserved for testing recognition and were not included in the population.
Recognition was based on the L_{2} norm of the 175 coefficients that were deemed to carry the signal. Table 1 contains the results leading to what we deem to be an optimal procedure. Less optimal results are included for their informational content. If all coefficients are considered, 82 (of 94) errors are made, an 87% error rate. If only the subspace of even coefficients are considered, the error rate drops to 22%, and if the subspace is further reduced by dropping a_{2}^{e} and a_{4}^{e}, the error rate drops to 4%. The 4 errors in this case are depicted in Fig. 6, and in each instance, one sees that contrary to the instructions from the “photographer,” the subject changed his pose for the marred image.
For the procedure thus far, coefficients carry their natural weighting, which because 〈a_{k}^{2}〉 = λ_{k} says only that a coefficient's weight diminishes with index. Under the ideal of an ensemble of unlimited number, each eigenface would be perfectly resolved, which would then argue for weighting the a_{k} term in the L_{2} norm by the reciprocal variance, λ_{k}^{−1}, the socalled whitened norm. In our case, this leads to negligible improvement, 1 less error. We adopt a geometric compromise and weight each a_{k} term in the L_{2} norm by 1/
Discussion
The data for frontal poses marred by lighting artifacts is somewhat limited but has been acquired from 3 different databases (19–21). The sole error of recognition occurred because the subject did not follow instructions and closed his eyes. If on this basis, we exclude this case, recognition becomes 100%.
Face recognition was accomplished by using the even subspace minus the 2 nonGaussian eigenfaces of the estimated 175dimensional face space, and is therefore of dimension ≈110. This form of recognition takes place using an L_{2} norm and therefore lies in the framework of machine recognition. An earlier psychophysical study (13), involving a merged population of male and female faces, demonstrated that human observers require on average only dimension ≈ 100 (of N ≈ 200) for recognition. However, the reduction to even eigenfaces was not applied in that study. With the estimate that >30% of the relevant eigenfaces are odd, we can revise the earlier estimate (13) to <70 dimensions for the human recognition system. This framework is consistent with the physiological studies (3, 6) discussed in the introduction.
Acknowledgments
We thank Bruce Knight, Ehud Kaplan, and Yu Zhang for reading, commenting on, and making other helpful contributions to the manuscript. We also thank the two extremely professional referees for their criticisms and suggestions. This work was supported in part by National Institutes of Health Grants NIH/NIGMS 1 P50 GM071558 and NIH/NEI EY16224.
Footnotes
 ^{1}To whom correspondence should be addressed. Email: lawrence.sirovich{at}mssm.edu

Author contributions: L.S. designed research; M.M. performed research; L.S. analyzed data; and L.S. wrote the paper.

The authors declare no conflict of interest.

↵† In the following, faces have been normalized so that eye separation is standardized and so that the total reflectivity is the same for each face. Hence, each face distantly viewed is the same gray blur. Unless said otherwise, f_{j} has the mean face subtracted.

↵‡ This was not essential, and we were able to achieve the same degree of recognition accuracy without including these in the base population.
 © 2009 by The National Academy of Sciences of the USA
References
 ↵
 ↵
 ↵
 Freiwald W,
 Tsao D
 ↵
 Tanaka K,
 Saito H,
 Fukada Y,
 Moriya M
 ↵
 Kanwisher N,
 McDermott J,
 Chun M
 ↵
 Tsao D,
 Freiwald W,
 Tootell R,
 Livingstone M
 ↵
 Kanwisher N,
 Yovel G
 ↵
 Kanwisher N
 ↵
 Wechsler H
 ↵
 Kobatake E,
 Tanaka K
 ↵
 ↵
 Penev P,
 Sirovich L
 ↵
 ↵
 Everson R,
 et al.
 ↵
 Papoulis A
 ↵
 Lupton R
 ↵
 Eklundh JO
 Moses Y,
 Adini Y,
 Ullman S
 ↵
 ↵
 ↵
 ↵
 Martinez A,
 Benavente R
Citation Manager Formats
Sign up for Article Alerts
Article Classifications
 Physical Sciences
 Applied Mathematics