Dissociation of face-selective cortical responses by attention

fMRI results (Fig. 2) identified regions in ventral temporal cortex with significantly different responses to faces and houses, in the same locations as shown in previous studies (3–9, 23). In the face-responsive region in the lateral fusiform gyrus, the difference between responses to faces and houses on scrambled backgrounds (F_SCR–H_SCR) did not differ quantitatively from the difference between responses when subjects attended to faces and houses in stimuli that contained intact images of both object categories (F_ATTN versus H_ATTN; P > 0.1). Similarly, in the house-responsive regions in medial fusiform and inferior temporal cortex, response differences in the attention conditions did not differ quantitatively from response differences in the scrambled-background conditions (P > 0.1). In the face-responsive fusiform region, the response to the unattended face in the H_ATTN condition was essentially identical to the response when a face was not present in the H_SCR condition (0.25 ± 0.35% versus 0.25 ± 0.27%, mean ± SD, for H_ATTN versus H_SCR, respectively). We also identified face-responsive regions in inferior occipital and superior temporal sulcal cortex (2) and found no trend toward a greater response in either region in the H_ATTN condition as compared with the H_SCR condition (P > 0.1). These results show that attention strongly suppressed the spatially defined, category-related hemodynamic responses to the unattended faces and houses. Table 1, which is published as supporting information on the PNAS web site, shows the magnitudes of the fMRI responses during each of the experimental conditions.

Fig. 3A shows an example of the averaged MEG responses from one sensor for a single subject. Sensors that showed responses with similar features were found in all eight subjects. To summarize the data across subjects, we identified sensors in each subject that showed significant differences between responses to faces and houses for all stimulus conditions (see Methods for selection criteria and Fig. 3B Inset for the locations of these sensors). The strongest responses to faces were observed at 142 ± 18 ms (mean ± SD) for single-exposure faces and at 141 ± 19 ms for faces on scrambled houses. These responses correspond to the M170. The strongest responses to houses peaked 20–33 ms earlier (109 ± 14 ms for single-exposure houses, P < 0.02; 121 ± 22 ms for houses on scrambled faces, P = 0.12).

Fig. 3B shows the time courses, averaged across subjects, for the differences between face and house responses in each stimulus condition. Note that the large, early differences between responses to faces and houses that are seen in the single-exposure and scrambled-background conditions are absent in the attention conditions. The face–house differences started statistically significantly later (at 194 ± 28 ms) in the attention conditions than in the single-exposure (103 ± 15 ms; P < 0.0001) and scrambled-background (116 ± 9 ms; P < 0.0001) conditions. Note that the face–house differences started significantly earlier for single-exposure pictures than for pictures on scrambled backgrounds (P < 0.02). The earlier face-selective response with single-exposure pictures may, therefore, be due to the different spatial frequency spectra of face and house pictures, which were controlled in the pictures of intact faces or houses on scrambled backgrounds.

Fig. 3B shows that responses were different when subjects attended to faces versus houses, but it does not show the similarities between these responses and the responses to single-exposure and scrambled-background images. Therefore, in Fig. 3C, group average responses are replotted to show all similarities among responses. Multidimensional scaling was used to project the response to each condition onto a single dimension defined at each time point by the difference between responses to single-exposure faces and houses (see Methods). Note that the plot in Fig. 3C agrees well with the single-subject data shown in Fig. 3A but does not illustrate the early nonselective response (<100 ms). Responses during selective attention to faces and houses were indistinguishable from each other until 194 ms after stimulus onset, and both displayed a strong face-selective M170. By contrast, late responses (>235 ms) during selective attention to faces and houses differed markedly from each other and were indistinguishable from responses to faces and houses, respectively, in the single-exposure and scrambled-background conditions. Early responses in the attention conditions, however, were not identical to the response to faces on a scrambled background, showing a significant deflection toward the early house response peaking at 120 ms (P < 0.005, two-tailed, for both comparisons of F_ATTN and H_ATTN to F_SCR), indicating some early processing of the house that also was not affected by attention but was then obscured by the slightly later and stronger, face-selective M170. Analysis of MEG data restricted to the four subjects in whom we also studied hemodynamic responses with fMRI showed the same effects that we found in the full group (see Fig. 4, which is published as supporting information on the PNAS web site).

Fig. 3D shows the distribution of face–house differences over the full sensor array. Consistent with the data shown in Fig. 3 B and C, based on only a small set of sensors in each subject, no sensors showed any early face–house differences in the attention conditions (F_ATTN versus H_ATTN). The early face–house differences in the nonattention conditions were more restricted to bilateral posterior temporal sites, whereas the late face–house differences were more widely distributed, including more anterior temporal sites on the right and a dorsal occipital site.

The results demonstrate that the face-selective cortical response that is detected by hemodynamic imaging with fMRI reflects different neural activity than the early face-selective cortical response that is measured by electrophysiological recording with MEG. Attention strongly modulated the face-selective pattern of the fMRI hemodynamic responses in fusiform cortex, as well as in inferior occipital and superior temporal sulcal cortex, but had no effect on the early face-selective MEG response (11). Others have shown that an early face-specific response (the N200 in intracranial recordings and the M170 in MEG) is generated by neural activity in the fusiform cortex (9, 10, 12, 14). Intracranial recordings have found additional sites that generate a face-selective N200 in more posterior inferior occipital cortex and in lateral temporal cortex over the middle temporal gyrus and superior temporal sulcus (14, 15). Our results show that this response contributes little to the hemodynamic response observed in essentially the same locations with fMRI (2). MEG also revealed later category-related responses that were strongly modulated by attention, suggesting that category-related hemodynamic responses in fusiform cortex are due primarily to these late responses.

The preliminary processing that was unaltered by attention in our task may reflect a feed-forward phase of perceptual processing that precedes activity that is modulated by a more distributed neural system (2, 29, 30). Later responses show stronger modulation by effects that suggest top-down control, such as attention, priming, and familiarity (13, 15, 17, 19, 20). The early response, which reflects a feed-forward phase driven by sensory input, and the late response, which reflects modulation by interregional interactions, may involve activity in the same neurons. Single-unit recordings from inferior temporal (31), middle temporal (32), and earlier visual cortices (V2 and V1) have shown that the activity in individual cells carries different information in early and late responses (33).

Under some circumstances, preliminary processing of the unattended object may play a role in selecting the attended object. In the attention-to-houses condition, the undiminished face-specific M170 response indicated unaltered early processing of the unattended face. Similarly, in the attention-to-faces condition, features of the early house-specific response were seen, suggesting that the unattended house also was processed before the onset of attentional modulation. In our task, the segregation of the attended and unattended objects required processing at the level of object-recognition operations. Thus, processing of the unattended object may have been necessary to identify object features in the stimulus that needed to be suppressed. Other studies suggest that attention can have a small but significant effect on the face-specific N170/M170 when task demands allow the attended nonface stimulus to be selected without processing the unattended face (1, 18, 21). In one study, the specific attended object was primed by presenting it in advance of the double-exposure stimulus (1). In another study, the attended stimulus was a word presented in opaque lettering; consequently, the contours of the face did not interfere with identification of the edges that define the letters (18). In a third study, the unattended faces were presented in unattended, peripheral locations (21). These other results suggest that preliminary, feed-forward processing can be biased to process attended information preferentially if the unattended information is not required for stimulus selection. These effects of attention on early responses, however, are much weaker (a small reduction in amplitude or delay in latency) than the effect of attention on later responses, where the response to the unattended face was essentially eliminated (Fig. 3).

The task that subjects performed, one-back repetition detection with different images of the same face or house, was chosen to manipulate attention. This task does involve holding one item in working memory (WM) at all times. Although WM load was the same for both the single- and double-exposure tasks, the change in WM content could cause some change in the prestimulus MEG field patterns. Within the face and house conditions, however, this effect of WM on prestimulus patterns should be equivalent, and after the stimulus offset (occurring 500 ms after stimulus onset) the responses returned rapidly toward baseline level. Repetitions involved different images of the same face or house to ensure that subjects were not responding to simple image matches.

fMRI measures hemodynamic changes that reflect the metabolic demand generated by neural activity, whereas electric and magnetic evoked responses are affected by the synchrony of neural activity (34). The M170 response, therefore, may be generated by a small set of neurons that fire in synchrony but generate a minimal metabolic demand. We cannot rule out the possibility that other MEG responses, such as rhythmical brain activity that is time-locked but not phase-locked to stimulus presentation, might show an earlier effect of attention.

The fraction of the cortical response to faces that is accounted for by the M170 response appears to be too small to detect in the average hemodynamic response. The possibility remains that a small focus or a brief, initial deflection of hemodynamic activity exists in the fusiform gyrus that is related to the M170 response to faces. Our fMRI methods did not detect any such evidence, but an event-related experimental design that allows a more detailed analysis of the earliest phases of the hemodynamic response might. Combined MEG–fMRI studies that systematically vary the strength of early and late face-selective responses also may detect a hemodynamic correlate of the early response.

We have shown previously that faces and several object categories evoke distinct, overlapping patterns of response in ventral temporal cortex, as measured with fMRI (23). The results of our current study indicate that these patterns reflect only late responses that can be modulated by attention. Early face-specific and other category-related responses, on the other hand, reflect a rapid phase of face and object processing that is under minimal control by attention.

Task and Stimuli. Subjects performed a repetition-detection task. Photographic stimuli were presented sequentially, and subjects determined whether each contained a picture of the same face or house that was shown in the immediately preceding picture. Eight subjects (five male and three female) participated in the MEG experiment, four of whom (three male and one female) also were studied subsequently with fMRI. All subjects gave written, informed consent to participate in these studies. Pictures were presented for 500 ms, followed by a 2.5-s interstimulus interval. Images subtended 5.6° × 6.2° of visual angle for both experiments. Fig. 1 shows sample stimuli. Pictures with different views of the same person or house were presented, so that subjects could not base their responses on a simple pattern match. Images superimposed on a scrambled background (F_SCR and H_SCR) had luminance, contrast, and spatial frequency spectra equivalent to those of the double-exposure pictures. Double-exposure pictures of superimposed faces and houses were used for both the attention-to-faces (F_ATTN) and attention-to-houses (H_ATTN) tasks. For the fMRI study, only the intact images on scrambled backgrounds and the double-exposure pictures were used. Because the MEG results showed that the phase-scrambled backgrounds had a negligible effect on evoked fields after 100 ms, we decided that including the single-exposure stimulus conditions in the fMRI experiment was not necessary. In addition, a control condition, using phase-scrambled double-exposure pictures, was added to the fMRI study to identify object-responsive cortex. Subjects were instructed to maintain fixation and to respond only to a match. Matches occurred on only 8% of trials and were not used in the analysis of evoked fields. Each subject performed one run of each task as practice before the MEG experiment.

For the MEG experiment, stimuli were presented on a rear projection screen, with a video projector (VistaPro, Electrohome) that was outside of the magnetically shielded room. Subjects were informed of the stimulus condition with a cue at the beginning of each run. Subjects responded by lifting their right index fingers. During each run of 64 trials, a single stimulus condition was used; there were two runs per condition. For the fMRI experiment, stimuli were projected onto a rear-projection screen, and subjects responded using a hand-held response button. Each run consisted of five 48-s blocks of stimuli, one for each condition, separated by 12-s rest periods. Sixteen stimuli were presented in each block.

fMRI. Gradient-echo, echo-planar imaging (repetition time, 2.5-s; echo time, 40 ms) was used to measure the blood oxygen level-dependent response using a General Electric 3-tesla scanner. In each of 10 runs, 122 whole-brain volumes, comprised of 40 contiguous, 3.5-mm, sagittal slices, were obtained, resulting in a total of 1,220 volumes. fMRI data were obtained at the National Institutes of Health subsequent to the MEG experiment at the Helsinki University of Technology.

MEG. Evoked magnetic fields were measured with the Vector-view system (Neuromag, Helsinki), a whole-head device with 306 sensors, 3 at each of 102 locations. Each sensor unit hosts two orthogonal planar gradiometers and a magnetometer. Only planar gradiometer signals were included in the analyses; the planar gradiometers give the largest signals just above a locally activated brain area, and thereby the locations with largest signals can be readily used as first guesses of the activated brain areas (Fig. 3D) (24), which largely facilitates the preliminary analysis. MEG signals were sampled at 600 Hz and bandpass-filtered at 0.1–172 Hz. Signals were averaged online over a time interval starting 0.2 s before and ending 1.0 s after the onset of the stimulus. All trials contaminated by excessive eye movements or abrupt changes in the magnetic field were excluded from the on-line response averaging. Before the analysis, the averaged signals were digitally low-pass-filtered at 40 Hz, and a 200-ms prestimulus baseline was applied for amplitude measurements. Identical runs were averaged together subsequently.

fMRI Time Series Data Analysis. The fMRI time series data were analyzed on a voxel-by-voxel basis by using multiple regression (35, 36). Selected contrasts between responses to different task conditions were calculated as effects of interest: responses to F_SCR versus responses to H_SCR, responses during attention to faces (F_ATTN) versus attention to houses (H_ATTN). Regions of interest were defined as areas showing significant responses to faces or houses, relative to the scrambled image control condition (Z > 5.6, and P = 10^–8) with a minimum volume of seven contiguous voxels. These regions were divided into face-responsive and house-responsive subregions based on the average responses during scrambled-background and attention conditions. For each subject, mean time series were obtained for face-responsive and house-responsive subregions of fusiform cortex as well as for face-responsive subregions of inferior occipital cortex and for house-responsive regions of inferior and dorsal occipital cortex. The mean strength of response to each stimulus condition, expressed as percent changes in signal, was calculated for the face-responsive and house-responsive subregions of each subject. Strength of response was calculated as the mean across 13 images in each block, beginning 15 s after the onset of the first stimulus, thus factoring out responses to the cue and the first two items because these responses may reflect activity before the subject has established an attentional focus. The statistical significance of differences between responses was tested using a random-effects repeated-measures ANOVA with planned comparisons.

MEG Data Analysis. Sensors that detected a difference between responses to faces and houses were identified in each individual. Responses to the first two trials were not included in the analysis, similar to the analysis of fMRI data (see above). Noise level was determined separately in each subject for each stimulus condition and each sensor by calculating the standard deviation of differences between face and house recognition trials for all time points during the prestimulus period. These calculations were performed on the mean evoked field averaged across all trials. The poststimulus evoked fields were then examined to identify the sensors in which the differences between the evoked fields in face and house recognition trials exceeded baseline variability by at least 8 SDs in all of the stimulus conditions at one time point 0–500 ms after stimulus onset. This set of significant sensors was then used to analyze responses for all stimulus conditions. The number of significant sensors ranged from 6 to 15 (mean = 11, and SD = 3) in different subjects.

At each time point, the response to each condition (R_F, R_H, R_Fscr, R_Hscr, R_Fattn, and R_Hattn) can be represented as a point in a multidimensional space, with each dimension corresponding to one significant sensor. The origin of this multidimensional space is the prestimulus baseline. The coordinate on each dimension is the response amplitude (relative to the prestimulus baseline) in the corresponding sensor, given in units of femtotesla/cm. In this multidimensional (6–15 dimensions, depending on the subject) space, the difference between responses to faces and houses at each time point can be represented as a Euclidean distance. Dividing this distance by the square root of the number of dimensions (sensors) allows this distance to be expressed in units equivalent to the signal in one sensor.

Data were resampled in time to align the time of the early peak difference between face and house responses for each individual to the group mean values of 137 ms for single-exposure and 143 ms for scrambled-background stimuli. Differences between responses during face and house conditions were calculated for single-exposure stimuli (F versus H), for intact stimuli on scrambled backgrounds (F_SCR versus H_SCR), and for the attention conditions (F_ATTN versus H_ATTN) and plotted in Fig. 3B. The time at which face and house responses were considered to first be significantly different was defined in each individual subject as the first difference that was >3 SDs of the variability of the prestimulus baseline.

A second display of the difference between responses to faces and houses was produced to illustrate all similarities among responses (Fig. 3C). This plot represents a form of multidimensional scaling in which each response is projected onto a single dimension at each time point. This dimension is defined by the line connecting the responses to single-exposure pictures of faces and houses (R_F and R_H). This dimension does not pass through the origin. The zero point on this dimension was defined by the ratio of the F and H responses relative to their prestimulus baselines with a positive value assigned to the face response: PR_F = + D_{F_vs_H} × [D_F/(D_F + D_H)] and PR_H = – D_{F_vs_H} × [D_H/(D_F + D_H)], in which PR_F and PR_H are the projected responses to single-exposure face and house images. D_{F_vs_H} is the Euclidean distance between R_F and R_H, corrected for the number of sensors. D_F and D_H are the Euclidean distances from the origin to R_F and R_H, again corrected for the number of sensors.

The responses on face and house trials in the scrambled-background and attention conditions (R_Fscr, R_Hscr, R_Fattn, and R_Hattn) were then projected onto the dimension defined by the line connecting R_F and R_H at each time point. For example, the orthogonal projection of the point defined by R_Fscr onto the dimension defined by the two points, R_F and R_H, was calculated by treating these three points as defining a triangle with D_{F_vs_H}, D_{F_vs_Fscr}, and D_{H_vs_Fscr} as the lengths of the sides of this triangle.

Fig. 3C shows the results of this projection plot of responses to the six stimulus conditions, first calculated for each subject individually and then averaged across subjects. Note that this plot explicitly shows the similarity between responses during the attention conditions as compared with responses during the single-exposure and scrambled-background conditions. The plot also captures the different strengths of the face and house responses, relative to the prestimulus baseline, as well as the different timing of early response peaks to faces and houses.

To display the distribution of differences between face and house responses over sensor locations for early (100–180 ms) and late (250–500 ms) time windows, time points with the maximal response difference were identified in each subject in early and late time windows. Mean latencies for the early and late maxima were 140 ± 11 ms and 313 ± 45 ms, respectively. The magnitude of the face–house difference at each location was calculated as the square root of the sum of the squares of the two gradiometer responses. In addition to plotting the distribution of face–house differences across all sensors, source localization was attempted by using both current dipole modeling and minimum current estimates (37) with and without weighting by fMRI data. Because of the complexity of the evoked fields, neither method produced a consistent, coherent estimate of the activated cortical areas. Although it was possible to model the M170 response to faces in most cases, we failed to reliably define to what extent the early and late responses to faces and houses were generated by the same/different neuronal sources. Consequently, we do not report the results of these analyses here.

Conflict of interest statement: No conflicts declared.

Abbreviations: fMRI, functional MRI; MEG, magnetoencephalography; EEG, electroencephalography.