Neuronal integration in visual cortex elevates face category tuning to conscious face perception

Johannes J. Fahrenfort; Tineke M. Snijders; Klaartje Heinen; Simon van Gaal; H. Steven Scholte; Victor A. F. Lamme

doi:10.1073/pnas.1207414110

Edited by Charles Gross, Princeton University, Princeton, NJ, and approved November 12, 2012 (received for review May 10, 2012)

Abstract

The human brain has the extraordinary capability to transform cluttered sensory input into distinct object representations. For example, it is able to rapidly and seemingly without effort detect object categories in complex natural scenes. Surprisingly, category tuning is not sufficient to achieve conscious recognition of objects. What neural process beyond category extraction might elevate neural representations to the level where objects are consciously perceived? Here we show that visible and invisible faces produce similar category-selective responses in the ventral visual cortex. The pattern of neural activity evoked by visible faces could be used to decode the presence of invisible faces and vice versa. However, only visible faces caused extensive response enhancements and changes in neural oscillatory synchronization, as well as increased functional connectivity between higher and lower visual areas. We conclude that conscious face perception is more tightly linked to neural processes of sustained information integration and binding than to processes accommodating face category tuning.

Image processing and image perception are not the same thing. Faces, for example, are processed even when they are invisible to the observer (1), and face processing has been shown to continue in anesthetized macaques (2). Apparently, even highly complex neuronal tuning responses are not sufficient to achieve perception. So what neural operations turn complex tuning responses into conscious representations? A window into these operations might be obtained by investigating perceptual organization. Perceptual organization is an umbrella term for cortical functions that organize sensory input into coherent and interpretable perceptual structures. It is thought to encompass processes such as figure–ground segregation, object detection, and object categorization (3). Theoretical accounts of human vision going back to Rubin (1915) have suggested that perception requires objects to first be segregated from their background (3 ⇓–5) and that object recognition follows figure–ground segregation. This notion has been challenged by behavioral studies showing that object recognition influences figure–ground assignment and might even precede it (6). Given their putative role in perception, resolving the relationship between category tuning and figure–ground segregation may tell us how sensory input is transformed into perceptual representations.

To investigate this relationship, we measured signals of category tuning and figure–background processing while subjects viewed visible and invisible objects. Given the speed of object category extraction (7, 8), we hypothesized that basic-level category tuning results from simple-to-complex feedforward computations (9) and is unrelated to object perception (1). We therefore predicted that it should be possible for both visible and invisible objects to elicit category tuning. Figure–ground processing, on the other hand, requires incremental grouping mechanisms in which tuning information from neurons with different receptive field sizes is integrated across visual areas over longer periods of time (10, 11). It has been suggested that mechanisms of neuronal integration are crucial to feature binding (12) and conscious perception (13). These claims are motivated by key properties of everyday perception that can be uniquely explained by mechanisms of information integration: differentiation, the availability of a seemingly infinite number of conscious experiences, and integration, the perceptual unity of each of these experiences (14). Therefore, our second prediction was that only consciously segregated objects show markers of sustained neuronal integration, directly linking neuronal integration to conscious experience.

Using a dichoptic fusion paradigm (1, 11), we presented objects that were either visible or not visible (Fig. 1A). Faces, houses, nonsense objects, and homogenous screens (Fig. 1B) were constructed using orientation-defined textures of Gabor elements. Monocularly, objects were created using different orientations for object and background (Fig. 1A, left and right eye). When properly viewing stimuli with both eyes, textures in the left and right eye fuse together. The fused textures could either be different for object and background (visible, Fig. 1A, Upper) or the same for object and background (invisible, Fig. 1A, Lower). Despite having very different perceptual properties when viewed with both eyes, average monocular stimulation was the same for visible and invisible conditions, allowing us to investigate how signals of category tuning and figure–ground processing are impacted when viewing visible and invisible objects. Neural correlates of category tuning were obtained by contrasting objects of different categories among each other. Comparing these objects with homogenous textures made it possible to look at the processes that are active when segregating objects from their background (see Figs. S1 and S2 for examples of these contrasts and SI Methods for details). To be sure, we should note that the figure–ground contrast not only highlights figure–ground processes but also category extraction, as individual object stimuli contain a category, whereas homogenous textures do not. Because functional MRI (fMRI) suffers from spatial blood-oxygen level–dependent (BOLD) summation of the underlying neuronal responses (15), the generic figure–ground profile in category-selective regions could also reflect averaged individual category-selective responses, even though we refer to this contrast as the figure–ground contrast for ease of reference.

Fig. 1.

Experimental setup. (A) Illustrations of visible (Upper) and invisible (Lower) dichoptic fusion using oriented line elements. Objects were defined using a 45° orientation difference between figure and background elements. (B) Illustrations of the four stimulus types: faces, houses, nonsense objects, and homogenous screens. Black lines illustrate orientation discontinuities. (C) 3D depiction of a subject performing the in front/behind distractor task. Both in front and behind are shown for illustrative purposes. (D) Stimulus classification during the postscanning control task (visible: upper panels; invisible: lower panels). Graphs show the mean response percentage (±SEM) for each response to that stimulus type. Paired t test of the hits against false alarms determined whether a category was identified above chance level (^N.S.P > 0.05; ***P < 10⁻⁸).

Results

To minimize attentional and postperceptual differences between visible and invisible conditions (16), subjects performed a distractor task instead of stimulus categorization. On each trial they had to indicate whether a stereoscopically presented ellipse appeared to be hovering in front or behind the stimulus screen due to a slight offset in left and right eye (Fig. 1C; SI Methods). The task could only be performed reliably when stereoscopically focusing on the fixation dot, requiring subjects to fuse left and right eye images. In a behavioral control task performed directly after scanning, but still inside the scanner, subjects combined ellipse localization with a stimulus categorization response (face, house, nonsense object, or homogenous; Fig. 1D). Task performance on ellipse localization during the control task was the same as during scanning (mean hit rate control = 0.91 vs. scanning = 0.90, F_1,15 = 0.445, P = 0.515), confirming that stereoscopic viewing conditions between the experimental and control runs were comparable. Visible categories were classified with high accuracy (mean d′ = 3.47, F_1,15 = 548.65, P < 10⁻¹², all individual P < 10⁻⁹), whereas none of the invisible categories were identified above chance level (mean d′ = 0.11, F_1,15 = 1.58, P = 0.228, all individual P > 0.18).

To identify signals related to category tuning, we used fMRI to isolate classical face- and place-selective areas (17, 18). The face-selective contrast (faces > houses and nonsense objects, visible and invisible combined) activated well-defined group-level clusters in the fusiform cortex (Fig. 2A) known as the fusiform face area (FFA). A localizer using photographs of faces and other objects was used to verify that these voxels corresponded to the classical FFA (Fig. S3). The contrast houses > faces and nonsense objects did not reveal place-selective clusters in the parahippocampal place area (PPA). Consequently, the focus of this paper is on face selectivity, although we test for other category-selective responses in all analyses.

Fig. 2.

Univariate category-selective activation to visible and invisible faces. (A) The fusiform face area (FFA) was defined as those voxels within ventral fusiform cortex responding more strongly to face stimuli than to houses and other objects. Shown is the averaged response from two datasets, each obtained through a split half procedure (see Methods and Fig. S3 for details). MNI coordinates: x = 40, y = −43, z = −27. (B) Mean percent signal change obtained from the split half procedure (error bars, ±SEM) for the faces > houses and objects contrast, separately for visible and invisible stimuli in the left and right FFA. (C) Deconvolution of the hemodynamic response to each stimulus category in the FFA (Upper) and the face-selective part of the response to visible and invisible stimuli in the FFA (Lower).

We examined BOLD signals in the left and right FFA of functional data that were independent from the data used for FFA selection (Methods; Fig. S3). A three-way ANOVA showed that these responses were face selective (faces vs. houses and nonsense objects: F_1,15 = 11.26, P = 0.004), larger to visible than to invisible stimuli (F_1,15 = 42.68, P < 10⁻⁵), and that responses were stronger in the right than in the left hemisphere (F_1,15 = 11.70, P = 0.004). Strikingly, face selectivity did not interact with visibility (F_1,15 = 1.13, P = 0.304), indicating that category-selective responses to visible and invisible faces were comparable in strength (Fig. 2 B and C). The absence of an interaction between face selectivity and visibility was confirmed by three ANOVAs that used various combinations of the face category and other categories. Although all ANOVAs showed main effects of visibility and category, none showed an interaction between visibility and category (faces, houses, nonsense objects: F_2,14 = 1.55, P = 0.247; faces, houses: F_1,15 = 0.404, P = 0.534; faces, nonsense objects: F_1,15 = 3.11, P = 0.098). Paired t tests further confirmed the presence of face-selective responses in the left and right FFA for both visible and invisible stimuli (invisible left: t₁₅ = 2.06, P = 0.029; visible left: t₁₅ = 2.85, P = 0.006; invisible right: t₁₅ = 2.59, P = 0.010; visible right: t₁₅ = 2.20, P = 0.022; one-tailed), showing that the FFA contains voxels from which face-category information can be extracted, irrespective of whether a face is perceived or not.

To determine the degree to which this is reflected in the time course of the response, we deconvolved the hemodynamic response function (HRF) in the FFA for each of the conditions (19). Overall response amplitudes to visible categories were much larger than to invisible categories (Fig. 2C, Upper), consistent with observations that perceived items lead to stronger responses than items that are not perceived (1, 20). Notably however, FFA responses to visible faces were strongest among all visible conditions, and responses to invisible faces were strongest among all invisible conditions. Isolation of the face-selective part of the time course demonstrates that visible and invisible face-selective HRFs are actually quite similar (Fig. 2C, Lower), seemingly at odds with many studies that do show a relationship between category representations and stimulus visibility (21). This apparent contradiction is treated in more detail in SI Discussion.

Although these findings confirm that visible and invisible faces are able to produce face-selective responses (1), it leaves unresolved to what degree the distributed cortical representation of visible and invisible category information is similar. Multivoxel response patterns are known to be more sensitive to category information than univariate responses and are thought to reflect distributed neural representations (22). We scanned all volumes of each subject using a spherical searchlight kernel (23). Correlations were computed between spatial activation patterns across voxels in the kernel, at each location in the brain, for each possible combination of visible and invisible categories. Next we performed a permutation test at each location to assess whether the pattern correlation around that location carried visibility invariant category information. Specifically, we tested whether within-category visible-invisible correlations were higher than between-category visible-invisible correlations (SI Methods). This analysis was performed separately for faces, houses, and nonsense objects. Visible-invisible face correlations (large green dot) were significantly higher than all between-category correlations (small green dots) in a right-lateralized cluster extending across the right occipitotemporal fusiform cortex, confirming the presence of a high-level representation of face-category information that is invariant to stimulus visibility (Fig. 3A; familywise error rate controlled at P < 0.01). The same cluster emerged when testing whether visible faces were uniquely correlated with invisible faces (large red dot over small red dots, Fig. 3B) and when testing whether invisible faces were uniquely correlated with visible faces (large blue dot over small blue dots, Fig. 3C). House and nonsense categories did not result in visibility invariant category extraction anywhere. Some may wonder how monocularly presented faces are able to penetrate into high-level visual cortex when their constituent elements seem to be fused at lower levels. This topic is covered in more detail in SI Discussion.

Fig. 3.

Analysis of multivoxel pattern overlap between visible and invisible face-selective responses. (A) Whole brain analysis of searchlight locations for which the pattern correlations between visible and invisible faces in the kernel were significantly higher than the pattern correlations between nonmatching categories. (B) Searchlight locations where pattern responses to visible faces were uniquely correlated with invisible faces over other invisible object categories. (C) Searchlight locations where pattern responses to invisible faces were uniquely correlated with visible faces over other visible object categories. P values were corrected for multiple comparisons by controlling the familywise error rate. Brains are shown at MNI coordinates: x = 41, y = −46, z = −25. (D) Pattern classification of multivoxel responses to visible categories using responses to invisible categories and vice versa (error bars, ±SEM) in VOT and BA17.

To further confirm that visibility invariant face-category extraction was specific to high-level visual cortex and not driven by low level visual information, we attempted to classify the multivoxel patterns of the visible object categories using the invisible category patterns and vice versa in two regions of interest (ROIs): (i) left and right ventral occipitotemporal cortex (VOT) (24) and (ii) Brodmann area 17 (BA17) (Fig. S4; SI Methods). VOT comprises a set of object-, face- (17), and place-selective (18) regions bounded by the fusiform gyrus and the parahippocampal gyrus. BA17 (early visual cortex) is known to be sensitive to low-level image features but insensitive to category information (24). Fig. 3D shows that only the right VOT contained multivoxel activation patterns that allowed categorization of invisible faces using visible faces (permutation test: P = 0.0004) and vice versa (P = 0.0012), confirming that the visible-invisible pattern overlap is specific to high-level (right-lateralized) object-selective cortex and specific to faces.

These results show that visible and invisible faces produce and share univariate and multivariate category-selective responses. Central to our research question, we wanted to determine what is required in addition to category tuning to achieve perception. We speculated that object perception critically depends on sustained spatiotemporal integration during figure–ground processing. To isolate regions putatively involved in figure–ground processing, we contrasted all stimuli (visible and invisible combined) containing a texture-defined surface with textures that do not contain a surface (houses, faces, and nonsense objects > homogenous; Figs. S1C and S2A). This contrast activated a large cluster of areas in the ventral and dorsal visual pathways corresponding to regions that have been implicated in object (24) and figure–ground processing (11) (Fig. 4A; cluster creation threshold, Z > 2.3; cluster-corrected probability, P < 0.05). We divided this cluster into four atlas-defined ROIs, based on the functional significance of these regions during object processing (Methods): (i) activation in the superior lateral occipital cortex (dorsal foci), (ii) activation in the inferior lateral occipital cortex (LOC), (iii) VOT, and (iv) BA17.

Fig. 4.

Figure–ground responses. (A) Figure–ground modulation (faces, houses, and nonsense objects > homogenous textures), from which VOT, dorsal foci, LOC, and BA17 were selected. Shown is the averaged response from two datasets, each obtained from a split half procedure (Methods). MNI coordinates: x = 33, y = −81, z = 3. (B) Figure–ground modulation and (C) face-ground modulation (error bars, ±SEM) during the first functional runs. (D) The evoked figure–ground EEG response to visible and invisible stimuli at electrode Oz, low-pass filtered at 20 Hz for illustration purposes. Thick lines indicate cluster-corrected significance. (E) Topographic distributions of the figure–ground signal evoked by invisible (Upper) and visible (Lower) figures. Thick electrodes indicate cluster-corrected significance. (F) Induced EEG responses to figure–ground modulation. (Left) Time frequency representations of the figure–ground signal at electrodes P3, PO3, PO7, P4, PO4, and P08 of bilateral occipitotemporal cortex. (Right) Topographic distribution of significant clusters, indicated by the dashed lines in the time-frequency distribution on the left (only present in the visible condition).

To investigate whether visible and invisible objects contributed equally to the figure–ground signal, we estimated percent BOLD signal changes in functional data that were not used during ROI selection (Methods). A three-way ANOVA revealed main effects of figure–ground processing (faces, houses, and nonsense objects vs. homogenous textures: F_1,15 = 15.02, P = 0.001), larger responses to visible than to invisible stimuli (F_1,15 = 54.78, P < 10⁻⁵), and response strength differentiation across all regions of interest (FFA, VOT, LOC, dorsal foci, BA17), with the largest responses in early visual cortex (BA17) weakening along the posterior-anterior dimension (F_4,12 = 23.00, P < 10⁻⁴, see Fig. S5 for responses to all conditions in all ROIs). Strikingly, although we showed that face-selective responses in our FFA ROI are invariant to stimulus visibility, there was a strong interaction effect between stimulus visibility and the figure–ground response (F_1,15 = 43.96, P < 10⁻⁵), with figure–ground responses occurring only for visible objects. Fig. 4B shows figure–ground responses to visible and invisible objects in the four ROIs and in the FFA. Two-tailed paired t tests confirmed that visible objects strongly modulated figure–ground signals in each of the ROIs: VOT (t₁₅ = 5.99, P < 10⁻⁴), LOC (t₁₅ = 5.78, P < 10⁻⁴), dorsal foci (t₁₅ = 5.23, P < 0.001), and BA17 (t₁₅ = 4.33, P = 0.001), as well as in the FFA (t₁₅ = 5.84, P < 10⁻⁴), whereas invisible stimuli produced no significant figure–ground response in any of these areas in either direction (all P > 0.1). Note that BA17 seems to show a negative invisible figure–ground and face-ground responses (Fig. 4 B and C), but these effects are not significant (figure–ground: t₁₅ = −1.75, P = 0.1; face-ground: t₁₅ = −1.51, P = 0.153). A whole-brain analysis using all runs from the dataset showed no figure–ground responses to invisible objects anywhere (cluster creation threshold, Z > 1.6; cluster-corrected probability, P < 0.05).

The fact that these figure–ground signals occur only when viewing visible objects might be taken as a first indication that face perception is more strongly related to figure–ground processing than to face-category tuning. When inspecting the face-specific figure–ground contrast (faces > homogenous screens; Fig. 4C), the result is nearly identical, although invisible faces did activate the FFA more than homogenous textures (FFA: t₁₅ = 2.73, P = 0.032). This finding is unsurprising, given that the face-ground contrast not only captures a figure–ground relationship but also a face-selective response. This invisible face-ground response is not significantly different from the face-selective response in the FFA (t₁₅ = −0.309, P = 0.762; Fig. 2B), suggesting that it reflects a face-selective rather than a figure–ground response. Interestingly, the visible face-ground response is about five times larger than the visible face-selective response, showing the large contribution of the object-ground relationship to the overall response. Separate face-ground, object-ground, and house-ground ROIs and their corresponding figure–ground profiles can be found in Fig. S6.

To further investigate the presence of invisible face-ground signals, we performed a whole-brain conjunction analysis of visible and invisible face-ground responses, looking for voxels that were jointly activated by visible face-ground and invisible face-ground contrasts (P < 0.01 in both contrasts, uncorrected). This analysis turned up a small bilateral cluster in the FFA/VOT only (peak activations—left: x = −34, y = −60, z = −26; right: x = 36, y = −40, z = −30), but no activations anywhere else. Together, these results show that visible objects cause widespread figure–ground modulation across the visual cortex, whereas invisible objects do not, suggesting that faces that do not elicit a figure–ground response across the visual cortex still evoke category-selective responses in the FFA. See SI Discussion for a more detailed account of how this might work.

Crucially, we wanted to know whether consciously segregated objects uniquely display markers of neuronal integration (14). Although information integration is difficult to quantify in practice (25), educated guesses can be made with respect to the types of neural activity that could reflect it. For example, information integration across spatially separated sets of neurons might be coded through neural synchrony (12) or concurrent response enhancement (10). Such mechanisms would manifest as power changes in the time-frequency domain of the EEG or through increased functional connectivity between areas as measured with fMRI (26). To test this hypothesis, we collected EEGs using the same experimental protocol that was used during the fMRI experiment. t tests showed that subjects from both experiments performed the same across all behavioral measures (see Fig. S7 for categorization responses during the EEG experiment). To increase spatial resolution and to filter out deep sources, we generated a scalp current source density (CSD) estimate of the EEG using spherical splines (27). Evoked responses (ERPs) were obtained by averaging the CSD waveforms from stimulus onset onward. We isolated figure–ground responses by subtracting the ERP to homogenous textures from ERPs to stimuli containing a texture-defined surface (faces, houses, and nonsense objects minus homogenous textures), separately for the visible and the invisible conditions (Fig. 4D). Fig. 4E shows the topographic distribution of these figure–ground responses over time. Cluster-based permutation testing (Methods) revealed a negative cluster over midline occipital electrodes during the 80- to 125-ms interval for both visible (P = 0.004) and invisible conditions (P = 0.002). Beyond 125 ms, only visible figures evoked a sustained negative response compared with homogenous textures (P < 0.001), accompanied by a positive response spreading into the temporoparietal cortex (left cluster: P = 0.004; right cluster P = 0.22). Together, these data suggest that visible figure–ground responses are widespread across visual cortex (Fig. 4B) and persistent in time, whereas the invisible figure–ground signal is focal in space (Fig. 4C) and transient in time (Fig. 4 D and E). We also tested for category-selective responses in the EEG signal, but neither visible nor invisible faces evoked N170 or other face-selective responses. See SI Discussion for further discussion of this finding.

To investigate the presence of neural synchrony in visible and invisible figure–ground responses, we performed a power analysis of the time-frequency spectrum (Fig. 4F). Changes in the EEG power spectrum that are not phase-locked to stimulus onset (induced) are more likely to reflect changes in neural synchrony, whereas changes that are phase-locked to stimulus onset (evoked) are more likely to reflect transient afferent activity (28). Therefore, we further isolated the induced response by subtracting the average ERP to each condition from its respective single-trial responses before performing a power spectrum analysis (SI Methods). A cluster-based permutation test did not reveal power changes in the time-frequency signal for invisible textures containing a figure compared with homogenous textures, suggesting that the invisible figure–ground response was indeed driven by the evoked portion of the signal. Visible figures, however, exhibited a strong power decrease in the low beta band (12–20 Hz) between 250 and 1,000 ms (cluster-level P < 0.001), as well as power increases in the high gamma band (50–80 Hz) at 400–1,000 ms (cluster-level P < 0.012) and in the theta band (4–6 Hz) between 150 and 700 ms (cluster-level P < 0.001), with a topographic distribution over the bilateral occipitotemporal cortex (Fig. 4F). These changes in the time-frequency spectrum show that the processing of visible, but not invisible, figures results in sustained oscillatory (de)synchronization of cell assemblies (28), providing a mechanism by which neurons may label shared surface and boundary elements of the figure–ground display (29). The face-ground response profile was virtually identical, although the face-ground gamma responses were weaker and did not reach statistical significance (Fig. S8). For a more in-depth treatment of functions that may be associated with this spectral fingerprint, see SI Discussion.

To probe whether the induced changes in the time-frequency spectrum of the visible figure–background response coincide with increased functional connectivity across the visual cortex, we took the bilateral FFA as a seed region and performed a psychophysiological interaction (PPI) analysis on the fMRI data (30) (Methods). Responses to visible faces in the FFA were associated with stronger functional connectivity with early visual areas than responses to invisible faces (Fig. 5), an effect that cannot be attributed to differences in response amplitude between visible and invisible faces (SI Discussion). This finding suggests that the processing of visible faces results in stronger intercortical integration than the processing of invisible faces and that this integrative process takes place between higher and lower visual areas on a multisecond timescale. Separate PPI analyses of visible and invisible face-ground processing showed that this effect was not an additive effect for visible over invisible faces, but was a qualitative effect. FFA activity to visible faces had increased functional connectivity with the same cortical areas compared with homogenous textures, but invisible faces showed no increase in functional connectivity with these areas compared with homogenous textures. These results provide topographically specific evidence for the locus of information integration across visual areas during the processing of visible faces.

Fig. 5.

Increased PPI with FFA activity when viewing visible compared with invisible faces. Ventral and medial renderings of the regions in the cortex that show increased functional connectivity with the FFA when subjects are viewing visible faces compared with when they are viewing invisible faces (Z > 1.6, corrected P < 0.05). The regions showing increased functional connectivity are indicated in purple. In the right hemisphere, this cluster runs ventrally from the right FFA through large parts of extrastriate cortex to BA17. In the left hemisphere, the cluster is confined to BA17 and BA18.

Discussion

Summarizing, just like visible faces, invisible faces generate category-specific responses but do not generate responses that contain the neural markers of cortico-cortical integration that manifest in the putative figure–ground signals of visible objects. From this, we conclude that traditional theories on human perception may be mistaken in their notion that figure–ground processing is a necessary precursor to object categorization (3 ⇓–5). We suggest that face category extraction can be achieved through feedforward computations, whereas conscious representations require large-scale neuronal integration through recurrent interactions in visual cortex. Importantly, we combined visible and invisible conditions in all contrasts that were used for ROI selection. This procedure preferentially targets voxels that are selective in both the visible and invisible contrast, if they exist. We do not claim that all voxels that are face selective for visible faces should also be face selective for invisible faces or the other way around. However, because only visible objects generated markers of neuronal integration, these seem to be a better indicator of perceptual organization than category extraction itself. Further note that we only observed category-selective responses to faces. A lack of task relevance to the objects during the task may have resulted in a lack of category-selective responses to other categories (31). Indeed, others have observed category-selective responses to invisible task-relevant categories other than faces (1). Alternatively, invisible category-selective responses do not depend on task relevance but on domain specificity of the regions involved (17, 18). If this is the case, our result might not generalize to other stimulus categories, even under conditions of task relevance. SI Discussion contains a more in-depth treatment of this issue.

Finally, it might be argued that the markers of neuronal integration we observed are partially caused by selective attention. We observed markers of neuronal integration despite the fact that our subjects performed a distractor task during scanning in which attentional resources were directed elsewhere. Indeed, evidence suggests that perceptual organization does not depend on attention, but rather provides the structure on which it operates (32). However, even if the visible objects grabbed attention involuntarily, our conclusions remain unchanged. We do not claim that the effects of neuronal integration we observed must be caused by some specific process, but rather that (i) none of the effects of neuronal integration that we observed seems to be required for face-category tuning, and (ii) correlates of neuronal integration are a better marker for perceptual organization and conscious perception than face-category tuning. We suggest that going from face-category tuning to neuronal integration across the visual cortex marks the transition from unconscious to conscious face representations.

Methods

Subjects.

Thirty-eight subjects (18 fMRI, 3 males; 20 EEG, 4 males) viewed texture stimuli in a single-session experiment. Four subjects (two EEG subjects) were excluded because of artifacts or hardware. All subjects were healthy adults with normal vision. All provided written informed consent. The research was approved by the ethical committee of the Psychology Department of the University of Amsterdam.

Equipment.

Stimuli were presented at 800 × 600 resolution (16.9° × 12.7° visual angle) at a rate of 60 Hz. Images for the left and the right eye were sent through differently polarized filters while subjects viewed the screen using correspondingly polarized glasses.

Experimental Procedure.

All subjects participated in a short training session. Subsequently, subjects performed the experimental task, during which fMRI or EEG imaging data were acquired (see SI Methods for detailed acquisition parameters). Directly after fMRI/EEG acquisition, subjects performed the control task to establish stimulus visibility.

fMRI Analysis.

Using the Oxford Centre for Functional MRI of the Brain (FMRIB) Software Library (FSL), functional data were motion corrected, slice-time aligned, temporally filtered with a high-pass filter (35 s), and spatially filtered using a Gaussian envelope (univariate analyses: 5 mm; multivoxel analyses: 2 mm). Data were spatially normalized to MNI space using FMRIB’s Nonlinear Image Registration Tool (FNIRT). All ROI analyses were performed using a split half procedure. Each trial was pseudorandomly assigned to one of two datasets, each containing exactly half of the data. Subsequently, one dataset was used to draw ROIs, exporting responses in those ROIs from the other dataset, and the same procedure was repeated after switching the datasets used for ROI selection and export. Finally, the exported data were averaged and used for statistical testing (Fig. S3). Whole-brain searchlight analyses were performed on the entire dataset using custom code in Matlab (Mathworks). For each subject and each run, a general linear model was created. A predictor convolved with a standard HRF modeled each condition. Single subject parameter estimates were generated by combining runs using a fixed-effects higher-level analysis. Individual subject statistics were combined using FMRIB's Local Analysis of Mixed Effects (FLAME1) for univariate and PPI analyses, correcting for multiple comparisons using cluster thresholding. Multivoxel analyses were combined across subjects using FMRIB’s randomize function for nonparametric permutation testing (33), correcting for multiple comparisons using threshold-free cluster enhancement (TCFE) (34).

Functional Connectivity Analysis.

A nine-column design matrix was constructed: (i) a regressor for the contrast of interest, e.g., visible vs. invisible faces, convolved with the HRF; (ii) a time series regressor containing the time course of FFA activation; (iii) an interaction regressor containing the interaction term between 1 and 2, and (iv–ix) HRF-convolved regressors for the other six stimulus types: visible and invisible houses, nonsense objects, and homogenous screens. The regression coefficient of the interaction term gives a measure of increased functional connectivity with the FFA for visible compared with invisible faces (30).

EEG Analysis.

The data were preprocessed using the Brain Vision Analyzer with a high-pass filter (0.5 Hz), a notch filter (50 Hz), and segmented into (−300, 1,300) millisecond periods. Segments containing transients exceeding ±200 µV were removed before ocular artifact correction. Finally, segments containing transients exceeding ±50 µV were removed, and a CSD transformation was applied to obtain reference-free data. All subsequent analyses were performed in Matlab with the FieldTrip toolbox (35) (Radboud University). Differences between conditions were assessed using cluster-based permutation testing (36).

ERPs.

We visualized three time bins based on previously established properties of the ERP during the processing of figure–ground displays (37): 0–80 ms (does not differentiate between figure and no-figure textures), 80–125 ms (first figure–ground response but does not correlate with perception), and 125–250 ms (first figure–ground response to correlate with perception). We performed a permutation test over each averaged bin to establish clusters of electrodes exhibiting an evoked response in that time window.

Acknowledgments

We thank Tomas Knapen and two anonymous reviewers for helpful comments on earlier drafts of this manuscript. T.M.S. is supported by Vici Grant 453-07-004 from the Netherlands Organisation for Scientific Research and V.A.F.L. is supported by Advanced Investigator Grant 230355 from the European Research Council.

Footnotes

↵¹To whom correspondence should be addressed. E-mail: fahrenfort.work{at}gmail.com.

Author contributions: J.J.F., K.H., S.v.G., H.S.S., and V.A.F.L. designed research; J.J.F. performed research; J.J.F. and T.M.S. analyzed data; and J.J.F. and T.M.S. wrote the paper.
The authors declare no conflict of interest.
This article is a PNAS Direct Submission.
This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1207414110/-/DCSupplemental.

References

↵
1. Moutoussis K,
2. Zeki S
(2002) The relationship between cortical activation and perception investigated with invisible stimuli. Proc Natl Acad Sci USA 99(14):9527–9532.
OpenUrl Abstract/FREE Full Text
↵
1. Ku SP,
2. Tolias AS,
3. Logothetis NK,
4. Goense J
(2011) fMRI of the face-processing network in the ventral temporal lobe of awake and anesthetized macaques. Neuron 70(2):352–362.
OpenUrl CrossRef PubMed
↵
1. Marr D
(1982) Vision: A Computational Investigation Into the Human Representation and Processing of Visual Information (W. H. Freeman, San Francisco).
↵
1. Rubin E
(1958) Figure and Ground (Van Nostrand, New York), pp 194–203.
↵
1. Osherson DN,
2. Gleitman LR,
3. Kosslyn SM
1. Nakayama K,
2. Zijiang JH,
3. Shinsuke S
(1995) in An Invitation to Cognitive Science, Visual surface representation: A critical link between lower-level and higher-level vision, eds Osherson DN, Gleitman LR, Kosslyn SM (MIT Press, Cambridge), pp 1–70.
↵
1. Peterson MA,
2. Gibson BS
(1994) Must figure-ground organization precede object recognition? An assumption in peril. Psychol Sci 5(5):253–259.
OpenUrl Abstract/FREE Full Text
↵
1. Liu H,
2. Agam Y,
3. Madsen JR,
4. Kreiman G
(2009) Timing, timing, timing: Fast decoding of object information from intracranial field potentials in human visual cortex. Neuron 62(2):281–290.
OpenUrl CrossRef PubMed
↵
1. Thorpe S,
2. Fize D,
3. Marlot C
(1996) Speed of processing in the human visual system. Nature 381(6582):520–522.
OpenUrl CrossRef PubMed
↵
1. Serre T,
2. Oliva A,
3. Poggio T
(2007) A feedforward architecture accounts for rapid categorization. Proc Natl Acad Sci USA 104(15):6424–6429.
OpenUrl Abstract/FREE Full Text
↵
1. Roelfsema PR
(2006) Cortical algorithms for perceptual grouping. Annu Rev Neurosci 29:203–227.
OpenUrl CrossRef PubMed
↵
1. Zipser K,
2. Lamme VAF,
3. Schiller PH
(1996) Contextual modulation in primary visual cortex. J Neurosci 16(22):7376–7389.
OpenUrl Abstract/FREE Full Text
↵
1. Singer W,
2. Gray CM
(1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586.
OpenUrl CrossRef PubMed
↵
1. Tononi G,
2. Edelman GM
(1998) Consciousness and complexity. Science 282(5395):1846–1851.
OpenUrl Abstract/FREE Full Text
↵
1. Tononi G
(2004) An information integration theory of consciousness. BMC Neurosci 5:42.
OpenUrl CrossRef PubMed
↵
1. Avidan G,
2. Hasson U,
3. Hendler T,
4. Zohary E,
5. Malach R
(2002) Analysis of the neuronal selectivity underlying low fMRI signals. Curr Biol 12(12):964–972.
OpenUrl CrossRef PubMed
↵
1. Koch C,
2. Tsuchiya N
(2007) Attention and consciousness: Two distinct brain processes. Trends Cogn Sci 11(1):16–22.
OpenUrl CrossRef PubMed
↵
1. Kanwisher N,
2. McDermott J,
3. Chun MM
(1997) The fusiform face area: A module in human extrastriate cortex specialized for face perception. J Neurosci 17(11):4302–4311.
OpenUrl Abstract/FREE Full Text
↵
1. Epstein R,
2. Kanwisher N
(1998) A cortical representation of the local visual environment. Nature 392(6676):598–601.
OpenUrl CrossRef PubMed
↵
1. Glover GH
(1999) Deconvolution of impulse response in event-related BOLD fMRI. Neuroimage 9(4):416–429.
OpenUrl CrossRef PubMed
↵
1. Grill-Spector K,
2. Kushnir T,
3. Hendler T,
4. Malach R
(2000) The dynamics of object-selective activation correlate with recognition performance in humans. Nat Neurosci 3(8):837–843.
OpenUrl CrossRef PubMed
↵
1. Tong F,
2. Nakayama K,
3. Vaughan JT,
4. Kanwisher N
(1998) Binocular rivalry and visual awareness in human extrastriate cortex. Neuron 21(4):753–759.
OpenUrl CrossRef PubMed
↵
1. Haxby JV,
2. et al.
(2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293(5539):2425–2430.
OpenUrl Abstract/FREE Full Text
↵
1. Kriegeskorte N,
2. Goebel R,
3. Bandettini P
(2006) Information-based functional brain mapping. Proc Natl Acad Sci USA 103(10):3863–3868.
OpenUrl Abstract/FREE Full Text
↵
1. Grill-Spector K,
2. Malach R
(2004) The human visual cortex. Annu Rev Neurosci 27:649–677.
OpenUrl CrossRef PubMed
↵
1. Tononi G,
2. Sporns O
(2003) Measuring information integration. BMC Neurosci 4:31.
OpenUrl CrossRef PubMed
↵
1. Varela F,
2. Lachaux JP,
3. Rodriguez E,
4. Martinerie J
(2001) The brainweb: Phase synchronization and large-scale integration. Nat Rev Neurosci 2(4):229–239.
OpenUrl CrossRef PubMed
↵
1. Perrin F,
2. Pernier J,
3. Bertrand O,
4. Echallier JF
(1989) Spherical splines for scalp potential and current density mapping. Electroencephalogr Clin Neurophysiol 72(2):184–187.
OpenUrl CrossRef PubMed
↵
1. Pfurtscheller G,
2. Lopes da Silva FH
(1999) Event-related EEG/MEG synchronization and desynchronization: Basic principles. Clin Neurophysiol 110(11):1842–1857.
OpenUrl CrossRef PubMed
↵
1. Castelo-Branco M,
2. Goebel R,
3. Neuenschwander S,
4. Singer W
(2000) Neural synchrony correlates with surface segregation rules. Nature 405(6787):685–689.
OpenUrl CrossRef PubMed
↵
1. Friston KJ,
2. et al.
(1997) Psychophysiological and modulatory interactions in neuroimaging. Neuroimage 6(3):218–229.
OpenUrl CrossRef PubMed
↵
1. Lavie N,
2. Ro T,
3. Russell C
(2003) The role of perceptual load in processing distractor faces. Psychol Sci 14(5):510–515.
OpenUrl Abstract/FREE Full Text
↵
1. Qiu FTT,
2. Sugihara T,
3. von der Heydt R
(2007) Figure-ground mechanisms provide structure for selective attention. Nat Neurosci 10(11):1492–1499.
OpenUrl CrossRef PubMed
↵
1. Nichols TE,
2. Holmes AP
(2002) Nonparametric permutation tests for functional neuroimaging: A primer with examples. Hum Brain Mapp 15(1):1–25.
OpenUrl CrossRef PubMed
↵
1. Smith SM,
2. Nichols TE
(2009) Threshold-free cluster enhancement: Addressing problems of smoothing, threshold dependence and localisation in cluster inference. Neuroimage 44(1):83–98.
OpenUrl CrossRef PubMed
↵
1. Oostenveld R,
2. Fries P,
3. Maris E,
4. Schoffelen JM
(2011) FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intell Neurosci 2011:156869.
OpenUrl PubMed
↵
1. Maris E,
2. Oostenveld R
(2007) Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods 164(1):177–190.
OpenUrl CrossRef PubMed
↵
1. Fahrenfort JJ,
2. Scholte HS,
3. Lamme VAF
(2007) Masking disrupts reentrant processing in human visual cortex. J Cogn Neurosci 19(9):1488–1497.
OpenUrl CrossRef PubMed

Article Alerts

Email Article

Citation Tools

Request Permissions

Article Classifications

Biological Sciences
Neuroscience

Proceedings of the National Academy of Sciences: 109 (52)

Table of Contents

Submit

[1] ↵
Moutoussis K,
Zeki S
(2002) The relationship between cortical activation and perception investigated with invisible stimuli. Proc Natl Acad Sci USA 99(14):9527–9532.
OpenUrl Abstract/FREE Full Text

[2] Moutoussis K,

[3] Zeki S

[4] ↵
Ku SP,
Tolias AS,
Logothetis NK,
Goense J
(2011) fMRI of the face-processing network in the ventral temporal lobe of awake and anesthetized macaques. Neuron 70(2):352–362.
OpenUrl CrossRef PubMed

[5] Ku SP,

[6] Tolias AS,

[7] Logothetis NK,

[8] Goense J

[9] ↵
Marr D
(1982) Vision: A Computational Investigation Into the Human Representation and Processing of Visual Information (W. H. Freeman, San Francisco).

[10] Marr D

[11] ↵
Rubin E
(1958) Figure and Ground (Van Nostrand, New York), pp 194–203.

[12] Rubin E

[13] ↵
Osherson DN,
Gleitman LR,
Kosslyn SM
Nakayama K,
Zijiang JH,
Shinsuke S
(1995) in An Invitation to Cognitive Science, Visual surface representation: A critical link between lower-level and higher-level vision, eds Osherson DN, Gleitman LR, Kosslyn SM (MIT Press, Cambridge), pp 1–70.

[14] Osherson DN,

[15] Gleitman LR,

[16] Kosslyn SM

[17] Nakayama K,

[18] Zijiang JH,

[19] Shinsuke S

[20] ↵
Peterson MA,
Gibson BS
(1994) Must figure-ground organization precede object recognition? An assumption in peril. Psychol Sci 5(5):253–259.
OpenUrl Abstract/FREE Full Text

[21] Peterson MA,

[22] Gibson BS

[23] ↵
Liu H,
Agam Y,
Madsen JR,
Kreiman G
(2009) Timing, timing, timing: Fast decoding of object information from intracranial field potentials in human visual cortex. Neuron 62(2):281–290.
OpenUrl CrossRef PubMed

[24] Liu H,

[25] Agam Y,

[26] Madsen JR,

[27] Kreiman G

[28] ↵
Thorpe S,
Fize D,
Marlot C
(1996) Speed of processing in the human visual system. Nature 381(6582):520–522.
OpenUrl CrossRef PubMed

[29] Thorpe S,

[30] Fize D,

[31] Marlot C

[32] ↵
Serre T,
Oliva A,
Poggio T
(2007) A feedforward architecture accounts for rapid categorization. Proc Natl Acad Sci USA 104(15):6424–6429.
OpenUrl Abstract/FREE Full Text

[33] Serre T,

[34] Oliva A,

[35] Poggio T

[36] ↵
Roelfsema PR
(2006) Cortical algorithms for perceptual grouping. Annu Rev Neurosci 29:203–227.
OpenUrl CrossRef PubMed

[37] Roelfsema PR

[38] ↵
Zipser K,
Lamme VAF,
Schiller PH
(1996) Contextual modulation in primary visual cortex. J Neurosci 16(22):7376–7389.
OpenUrl Abstract/FREE Full Text

[39] Zipser K,

[40] Lamme VAF,

[41] Schiller PH

[42] ↵
Singer W,
Gray CM
(1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586.
OpenUrl CrossRef PubMed

[43] Singer W,

[44] Gray CM

[45] ↵
Tononi G,
Edelman GM
(1998) Consciousness and complexity. Science 282(5395):1846–1851.
OpenUrl Abstract/FREE Full Text

[46] Tononi G,

[47] Edelman GM

[48] ↵
Tononi G
(2004) An information integration theory of consciousness. BMC Neurosci 5:42.
OpenUrl CrossRef PubMed

[49] Tononi G

[50] ↵
Avidan G,
Hasson U,
Hendler T,
Zohary E,
Malach R
(2002) Analysis of the neuronal selectivity underlying low fMRI signals. Curr Biol 12(12):964–972.
OpenUrl CrossRef PubMed

[51] Avidan G,

[52] Hasson U,

[53] Hendler T,

[54] Zohary E,

[55] Malach R

[56] ↵
Koch C,
Tsuchiya N
(2007) Attention and consciousness: Two distinct brain processes. Trends Cogn Sci 11(1):16–22.
OpenUrl CrossRef PubMed

[57] Koch C,

[58] Tsuchiya N

[59] ↵
Kanwisher N,
McDermott J,
Chun MM
(1997) The fusiform face area: A module in human extrastriate cortex specialized for face perception. J Neurosci 17(11):4302–4311.
OpenUrl Abstract/FREE Full Text

[60] Kanwisher N,

[61] McDermott J,

[62] Chun MM

[63] ↵
Epstein R,
Kanwisher N
(1998) A cortical representation of the local visual environment. Nature 392(6676):598–601.
OpenUrl CrossRef PubMed

[64] Epstein R,

[65] Kanwisher N

[66] ↵
Glover GH
(1999) Deconvolution of impulse response in event-related BOLD fMRI. Neuroimage 9(4):416–429.
OpenUrl CrossRef PubMed

[67] Glover GH

[68] ↵
Grill-Spector K,
Kushnir T,
Hendler T,
Malach R
(2000) The dynamics of object-selective activation correlate with recognition performance in humans. Nat Neurosci 3(8):837–843.
OpenUrl CrossRef PubMed

[69] Grill-Spector K,

[70] Kushnir T,

[71] Hendler T,

[72] Malach R

[73] ↵
Tong F,
Nakayama K,
Vaughan JT,
Kanwisher N
(1998) Binocular rivalry and visual awareness in human extrastriate cortex. Neuron 21(4):753–759.
OpenUrl CrossRef PubMed

[74] Tong F,

[75] Nakayama K,

[76] Vaughan JT,

[77] Kanwisher N

[78] ↵
Haxby JV,
et al.
(2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293(5539):2425–2430.
OpenUrl Abstract/FREE Full Text

[79] Haxby JV,

[80] et al.

[81] ↵
Kriegeskorte N,
Goebel R,
Bandettini P
(2006) Information-based functional brain mapping. Proc Natl Acad Sci USA 103(10):3863–3868.
OpenUrl Abstract/FREE Full Text

[82] Kriegeskorte N,

[83] Goebel R,

[84] Bandettini P

[85] ↵
Grill-Spector K,
Malach R
(2004) The human visual cortex. Annu Rev Neurosci 27:649–677.
OpenUrl CrossRef PubMed

[86] Grill-Spector K,

[87] Malach R

[88] ↵
Tononi G,
Sporns O
(2003) Measuring information integration. BMC Neurosci 4:31.
OpenUrl CrossRef PubMed

[89] Tononi G,

[90] Sporns O

[91] ↵
Varela F,
Lachaux JP,
Rodriguez E,
Martinerie J
(2001) The brainweb: Phase synchronization and large-scale integration. Nat Rev Neurosci 2(4):229–239.
OpenUrl CrossRef PubMed

[92] Varela F,

[93] Lachaux JP,

[94] Rodriguez E,

[95] Martinerie J

[96] ↵
Perrin F,
Pernier J,
Bertrand O,
Echallier JF
(1989) Spherical splines for scalp potential and current density mapping. Electroencephalogr Clin Neurophysiol 72(2):184–187.
OpenUrl CrossRef PubMed

[97] Perrin F,

[98] Pernier J,

[99] Bertrand O,

[100] Echallier JF

[101] ↵
Pfurtscheller G,
Lopes da Silva FH
(1999) Event-related EEG/MEG synchronization and desynchronization: Basic principles. Clin Neurophysiol 110(11):1842–1857.
OpenUrl CrossRef PubMed

[102] Pfurtscheller G,

[103] Lopes da Silva FH

[104] ↵
Castelo-Branco M,
Goebel R,
Neuenschwander S,
Singer W
(2000) Neural synchrony correlates with surface segregation rules. Nature 405(6787):685–689.
OpenUrl CrossRef PubMed

[105] Castelo-Branco M,

[106] Goebel R,

[107] Neuenschwander S,

[108] Singer W

[109] ↵
Friston KJ,
et al.
(1997) Psychophysiological and modulatory interactions in neuroimaging. Neuroimage 6(3):218–229.
OpenUrl CrossRef PubMed

[110] Friston KJ,

[111] et al.

[112] ↵
Lavie N,
Ro T,
Russell C
(2003) The role of perceptual load in processing distractor faces. Psychol Sci 14(5):510–515.
OpenUrl Abstract/FREE Full Text

[113] Lavie N,

[114] Ro T,

[115] Russell C

[116] ↵
Qiu FTT,
Sugihara T,
von der Heydt R
(2007) Figure-ground mechanisms provide structure for selective attention. Nat Neurosci 10(11):1492–1499.
OpenUrl CrossRef PubMed

[117] Qiu FTT,

[118] Sugihara T,

[119] von der Heydt R

[120] ↵
Nichols TE,
Holmes AP
(2002) Nonparametric permutation tests for functional neuroimaging: A primer with examples. Hum Brain Mapp 15(1):1–25.
OpenUrl CrossRef PubMed

[121] Nichols TE,

[122] Holmes AP

[123] ↵
Smith SM,
Nichols TE
(2009) Threshold-free cluster enhancement: Addressing problems of smoothing, threshold dependence and localisation in cluster inference. Neuroimage 44(1):83–98.
OpenUrl CrossRef PubMed

[124] Smith SM,

[125] Nichols TE

[126] ↵
Oostenveld R,
Fries P,
Maris E,
Schoffelen JM
(2011) FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intell Neurosci 2011:156869.
OpenUrl PubMed

[127] Oostenveld R,

[128] Fries P,

[129] Maris E,

[130] Schoffelen JM

[131] ↵
Maris E,
Oostenveld R
(2007) Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods 164(1):177–190.
OpenUrl CrossRef PubMed

[132] Maris E,

[133] Oostenveld R

[134] ↵
Fahrenfort JJ,
Scholte HS,
Lamme VAF
(2007) Masking disrupts reentrant processing in human visual cortex. J Cogn Neurosci 19(9):1488–1497.
OpenUrl CrossRef PubMed

[135] Fahrenfort JJ,

[136] Scholte HS,

[137] Lamme VAF

Main menu

User menu

Search

Neuronal integration in visual cortex elevates face category tuning to conscious face perception

Abstract

Results

Discussion