Temporally flexible feedback signal to foveal cortex for peripheral object recognition

Although the above-cited studies (3, 4) demonstrated that foveal stimulation affected peripheral object discrimination, it was unclear whether the foveal information was effective over a narrow or broad time window. In the current experiment, we investigated whether and when foveal noise could disrupt discrimination of objects presented in the periphery. Using stimuli similar to those in previous studies (1–3), two objects either the same or slightly different were presented for 100 ms in diagonal quadrants on two sides of the fixation point (either in upper right and lower left or in lower right and upper left). Subjects were asked to make a same or different decision based on the detailed feature of objects. A dynamic noise patch was presented for 83 ms in the foveal region at stimulus onset asynchronies (SOAs) of 50, 150, 250, 350, and 450 ms after the onset of the object images (Fig. 1).

As shown in Fig. 2, subjects’ performance (represented as d′) clearly depended on the SOA of the foveal noise relative to the peripheral objects. One-way repeated ANOVA with Bonferroni multiple comparison correction revealed a significant main effect of SOA (F_5,50 = 7.193, P < 0.0001; Fig. 2). More specifically, subjects’ performance decreased when noise appeared at the SOA of 50 ms (d′ = 1.155, P < 0.001) and 250 ms (d′ = 1.284, P = 0.005), compared with a control condition without foveal noise (dashed line in Fig. 2). The current result showed that foveal processing is disrupted at 250 ms following the onset of the peripheral object images, consistent with the engagement of foveal cortical regions. For the performance drop at 50-ms SOA, the foveal noise overlapped with the peripheral objects in time, and presumably attracted attention away from the object images.

We then conducted three control experiments to rule out alternative explanations for the delayed interference effect. First, the size of the noise patch was reduced to 4 × 4°. We found that the performance drops at 250-ms SOA remained robust (Fig. S1A). Thus, a noise patch more restricted to the foveal region was as effective in disrupting the peripheral object discrimination performance as the larger one, suggesting the involvement of foveal cortical areas rather than an area closer to the peripheral objects. Second, noise in the periphery failed to impact the peripheral object discrimination performance (Fig. S1B and SI Methods). Third, foveal noise was still effective in inducing the performance drop at 250-ms SOA, even when the two peripheral objects were both presented above the fixation and foveal noise was not in between the two peripheral objects (Fig. S1C).

In summary, we have firmly established that, within a narrow (FWHM:132 ms ±11 SEM, fit to a Gaussian function after normalization) and delayed temporal window, foveal interference produces an impairment in a peripheral object discrimination task, strongly implying that the foveal cortex is engaged for peripheral object discrimination at a specific time.

Given the specializations of foveal vision, we hypothesized that the foveal cortex may not be automatically engaged for a peripheral task, but rather it depends on both the task and stimuli—specifically on object discrimination tasks that involve spatial details.

In the experiments described above, subjects had to analyze the fine details of the objects to complete the task. In this next experiment, we investigated whether foveal noise would still be effective if (i) the stimulus contains only low spatial frequency information (Fig. 3A) or (ii) the task is based on motion signals (Fig. 3C). In the first condition, subjects still performed the object discrimination task, but had to make judgments according to the general shapes of the objects which were spatially low-pass filtered. In the second condition the task was to discriminate the speeds of motion of two patches of random dots. Results show that, whereas a performance drop at 50-ms SOA was clearly observed [filtered objects: P = 0.022 (F_5,50 = 3.154, P = 0.015); moving dots: P = 0.002 (F_5,35 = 4.063, P = 0.005)], as would be predicted by the general attentional distraction of the noise onset when the peripheral stimuli were still on the screen, there was no selective reduction in performance at later time points, including at the 250-ms SOA in either the blurred image condition (d′ = 0.954, P > 0.999; Fig. 3B), or in the motion speed discrimination condition (d′ = 1.364, P = 0.796; Fig. 3D). Thus, the results show that the temporally specific noise-induced performance reduction at 250 ms occurred when subjects performed an object discrimination task where fine spatial details of the peripheral objects were available.

One possible reason for this foveal engagement is that there is a tendency to foveate peripheral objects, and thus the visual system may have built-in mechanisms to engage foveal cortical processing in anticipation of receiving object information at the fovea in the next moments, before the actual saccade. The following two-part experiment was designed to investigate the role of saccade planning in the observed effect of delayed foveal noise. The question was whether the delayed foveal noise could disrupt the peripheral object discrimination if a saccade was required away from the peripheral object, to the opposite side of the fixation. In the first part, subjects were asked to discriminate a single object in the periphery. At the beginning of the experiment, subjects learned the shape features of target objects; then during the experiment, their task was to report whether the object presented was a target based on the learned shape features. We found that performance was impaired by foveal noise presented at 150-ms SOA (Fig. 4A), which is consistent with the results of Yu and Shim (3), but goes beyond in showing that the foveal interference effect is temporally specific with a rather narrow window (FWHM: 86 ms).

After establishing the effect of foveal noise in the single peripheral object condition, we next performed a dual-task experiment in which an object was presented in the upper-right visual quadrant for 100 ms, whereas a small disk was presented in the lower-left quadrant (Fig. 4B). The primary task was to report whether the object was a target, and the secondary task was to make a saccade to the small disk and report whether it was missing a wedge or was intact. Given the dynamics of the eye position, noise patches were presented at both the original fixation point as well as at the destination of the saccade. Results showed that the object discrimination performance was no longer affected by the foveal noise (Fig. 4B), not at 150 ms or any other SOA. Eye tracking data show that subjects typically started their saccades away from the object between 200 and 300 ms (Fig. 4C). In other words, subjects were still fixating at the original central fixation point when noise patches were presented at SOA of 150 ms. This result suggests that saccade programming, and specifically the direction of saccade, plays an important role in the interaction between peripheral object stimulus and the foveal noise.

The experiments above indicate that the engagement of the foveal cortex is task and stimulus dependent. However, in terms of the timing of this engagement, it is unclear whether the foveal retinotopic cortical processing is automatically engaged once the feedforward processing propagates to certain high level, or whether it is engaged at a flexible time point when fine detail information is needed. In this experiment, to test whether the timing of the foveal engagement is fixed or task dependent, we presented the two peripheral objects in different relative orientations where one of them was in the horizontal orientation, and the other one was rotated at a certain angle with respect to the horizontal. Subjects were specifically instructed to mentally rotate the tilted objects clockwise to horizontal and then compare with see whether two objects were same or different, as in the classical studies by Shepard and co-workers (5, 6).

We first examined the effects of mental rotation on reaction time in our paradigm, which was expected to increase as rotation angle become bigger (6, 7). Two conditions, mental rotation (rotation angles 30, 40, or 50°) and no mental rotation, were involved in this experiment. Consistent with previous studies, averaged reaction time was longer with mental rotation (1,070 ms) than without (854 ms), and the reaction time increased with increasing rotation angle.

We then performed the main experiment to see whether the time window of effective foveal noise would shift with the addition of the mental rotation task and, additionally, whether the degree of temporal shift would be related to the degree of the rotation angle. The angular differences between the two objects were 0, 40, or 80°. The 0° condition replicated the earlier test without any mental rotation. Results are shown in Fig. 5. Clearly, d′ varied as a function of SOA, presumably due to the early effect of attention distraction and the delayed interference effect from foveal noise. These two components are schematically depicted in Fig. 5 as purple and yellow dashed lines. The no mental rotation condition replicated the robust finding of the performance drop at 250 ms. For the mental rotation conditions, one-way repeated ANOVA showed statistically significant differences between different time points (40° rotation: F_6,48 = 2.385, P = 0.043; 80° rotation: F_7,91 = 2.479, P = 0.022). Introducing mental rotation in the task slowed down the recovery from the initial attentional distraction, but more remarkably, shifted the later interference effect further back in time. The typical performance reduction at the SOA of 250 ms was no longer observed. Instead, noise which appeared at the SOA of 450 ms (for the 40° rotation condition) and 550 ms (for the 80° rotation condition) reduced the performance significantly (40° rotation: P = 0.021; 80° rotation: P = 0.014; Fig. 5). Although the results shown in Fig. 5 were obtained from separate groups of subjects, the key aspect of the results—namely, the dependency of the critical time window on the rotation angle—was replicated in a group of subjects using a mixed-trial design. Apparently, the addition of the mental rotation task delayed the engagement of the foveal retinotopic processing. The delayed time window for engagement could not be explained by increased task difficulty, because the task difficulty levels for the 40 and 80° conditions were nearly the same (d′ = 1.17, d′ = 1.13; P = 0.721).

Thus, as already hinted by results shown in Fig. 4, the results from the mental rotation task show that foveal cortical processing is not automatically engaged, rather the timing of the engagement is flexible and can change depending on the processes required at higher cortical levels.

The psychophysical experiments, together with previous fMRI and TMS experiments, suggested that the foveal cortex was engaged during peripheral object recognition, in particular for object discrimination tasks that benefit from spatial details. Whereas previous fMRI experiments showed that object category information could be decoded in the foveal cortex, what is represented in the foveal cortex remains an open question. Is it primarily object category based (i.e., representing the distinct object category features) or image based (i.e., representing the image properties of the objects, including their orientation)? To investigate this question, we generated two categories of stimuli [cubies and smoothies as in Williams et al. (1)] and elongated them so that they could be presented either horizontally or vertically (Fig. 6A). Subjects performed the object discrimination task in the scanner, and fMRI data were analyzed with multivoxel pattern analysis in three independently identified ROIs [foveal cortex, peripheral cortex mapping the objects location, and lateral occipital complex (LOC)] (8–10). For each ROI, a straightforward measure of pattern similarity, spatial correlation, was computed for the same and different stimulus types.

In the peripheral retinotopic cortex where the stimuli were projected to, spatial patterns of activation should be image based (i.e., sensitive to the orientation of the object images), reflecting the basic retinotopic property. Results showed a clear orientation sensitivity: correlation for same orientation was higher than correlation for different orientation under both same (t₅ = 8.883, P = 0.001) and different (t₅ = 4.586, P = 0.012) category conditions (Fig. 6B), and there was an interaction between orientation and object category (F_1,5 = 9.234, P = 0.029). However, in the LOC, an area known for its object level sensitivity (11), the correlation for same orientation was higher than correlation for different orientation only under same category condition (t₅ = 4.133, P = 0.018; Fig. 6C), and there was interaction between orientation and category (F_1,5 = 7.209, P = 0.044). LOC carries both orientation and category information of objects (1, 11), but in contrast to early visual cortex, orientation information could not be read out when the category was different.

For the current study, the critical ROI is the foveal ROI. Similar to the LOC, the foveal ROI is sensitive to both orientation (F_1,5 = 11.61, P = 0.019) and object category (F_1,5 = 16.28, P = 0.01); however, in contrast to LOC, these two kinds of information were independent, in that there was no interaction between category and orientation (F_1,5 = 2.365, P = 0.185; Fig. 6D). Thus, the spatial pattern of activity in the foveal cortex contains not only feature information that distinguishes the object category (cubie vs. smoothie), but also information about the overall configuration (orientation) of the object. As a control, we selected an ROI in between the original two ROIs in the visual cortex and performed the same decoding analysis. Results show that there is no decodable information about the stimulus in this control ROI. Overall, the results showed that the neural representation in the foveal cortex may be more image-like, containing information about the general shape (here, overall orientation) as well as the spatial details (here, features that define object categories). Moreover, the relatively independent representation of orientation and category information is interesting, revealing a novel way for the foveal cortex to jointly represent different types and scales of information.

A total of 88 subjects (age range, 18–26; 44 female) participated in these experiments, with some subjects participating in more than one experiment; all had normal or corrected-to-normal vision. Different sets of 11 subjects participated in the objects discrimination experiments (7 × 7° noise, 4 × 4° noise, peripheral noise, single object with saccade, and low spatial frequency object). Eight subjects participated in the motion speed comparison experiment, objects on the same side experiment, and single-object experiment. A total of 14, 9, and 14 volunteers took part in the main part of the mental rotation experiment (rotation angle of 0, 40, and 80°). Six subjects were recruited for fMRI experiment.

Recent studies have pointed to the possibility that there is a feedback signal from high-level cortex to foveal retinotopic cortex when discriminating objects presented in the periphery (1–4). Consistent with this hypothesis, we found that foveal noise presented 250 ms after the appearance of peripheral objects selectively disrupted discrimination performance in the periphery. Further, we showed that this temporally delayed engagement of the foveal cortex was specific for tasks that use spatial details (Fig. 3), which provided psychophysical evidence that the high-resolution specialization of the foveal cortex can be used for discriminating fine spatial details of peripheral objects. These results support the hypothesis of temporally flexible feedback signal from higher-level cortex to lower-level foveal cortex.

Remarkably, we found that the temporal window for the foveal engagement is flexible rather than fixed. When there was less or more processing of the peripheral objects, the timing of the noise-induced performance drop was either advanced or delayed; this can be seen in the comparison of the experiments that involve one vs. two objects (Fig. 2 vs. Fig. 4) and the mental rotation experiment (Fig. 5). The mental rotation experiment is highly suggestive of a feedback signal to early foveal cortex that is not automatically triggered by a peripheral stimulus, but instead is only engaged when the high level process is ready. In the case of mental rotation, the right superior posterior parietal lobe is believed to play a key role (12–14). It is possible that the delay in feedback signal to early visual cortex was due to the additional processing and interaction between, for example, LOC (11) and posterior parietal cortex.

Notice that there was a performance drop at 50-ms SOA in the object discrimination task, motion speed comparison task, and the low-spatial frequency object discrimination task (Figs. 2 and 3). At 50-ms SOA, the foveal noise overlapped with the peripheral objects in time, so it is likely that the observed performance drop in the peripheral tasks was due to attention distraction caused by the onset of the foveal noise. Indeed, when both peripheral objects were presented above the fixation, the foveal noise presented at 50-ms SOA was no longer disruptive, presumably because subjects would shift their attention to the upper two quadrants in advance because the peripheral objects consistently appeared in the upper field (15); this voided the attentional distraction effect of the foveal noise at 50 ms (16, 17). For the other experiments, subjects’ attention remained at the fixation point at the beginning of each trial because the two peripheral objects would appear either in the upper right/lower left or in the lower right/upper left locations, leading to a significant attention distraction effect at 50-ms SOA (18). This effect was also observed in the mental rotation experiments, though was not statistically significant.

The previous Chambers et al. (2) study using a similar behavioral task showed that disrupting the foveal cortex with TMS at 350–400 ms after stimuli onset impaired the object discrimination performance. Our psychophysical study revealed that foveal visual noise was effective when presented ∼250 ms following the onset of the peripheral objects. These estimates of the effective temporal window for the engagement of foveal retinotopic cortex are in fact consistent, considering that the TMS was applied to the posterior calcarine site directly, whereas the fovea visual noise takes 50 ms or more to arrive at the primary visual cortex (19–21).

Results from our fMRI experiment show that orientation information of peripheral objects was represented in the foveal cortex, as well as the object category information. This pattern of results (Fig. 6) suggests that the signal representation in the foveal cortex is more image-like than that in LOC, but more sensitive to object category information than that in the peripheral retinotopic cortex. In the peripheral retinotopic cortex, orientation information could be read out regardless of the object category; however, in the foveal cortex, orientation information could only be read out for objects in the same category. Moreover, the pattern correlation for orientation and object category information in the foveal cortex showed no interaction, unlike the LOC or peripheral retinotopic cortex. A possible scenario is that the foveal cortex represents the object category and orientation information at different spatial scales, with the object category and orientation information primarily supported at fine and coarse scales, respectively. Using multielectrode recordings from passively fixating macaques, Hong et al. (22) recently showed that “category-orthogonal” features of an object, such as orientation, could be linearly decoded from high-level object areas in the ventral stream, and that this information was likely available on a feedforward pass. At one level, our imaging results are consistent in that we found that LOC was sensitive to the category-orthogonal feature of image orientation; however, this was only for stimuli that were in the same category. Our results raise the possibility that feedback may be required for tasks that not only benefit from fine-grain comparisons, but also for comparisons of category-orthogonal features across categories.

The delayed effect of foveal noise, as much as a few hundred milliseconds after the object onset, combined with the stimulus feature selectivity, makes it difficult to explain the observed effect in terms of attentional interruption (1). The large distance between the peripheral object and foveal noise makes it very unlikely that the effect was due to direct interaction between foveal noise and peripheral object through lateral horizontal connections within early cortical areas. Instead, two previous theoretical proposals are more relevant to our results. In 1998, Lee et al. (23) proposed a high-resolution spatial buffer hypothesis in which the primary visual cortex is used by the higher-level visual cortex for computations that requiring high-resolution detail (24). Likewise, in the reverse hierarchy theory proposed by Ahissar and Hochstein (25), initial feedforward processing provides a gist description of the input to the high-level cortex; then, at a later point in time, fine-detailed information is queried through feedback interactions with lower areas (25–27). However, our study shows that in the case of the analysis of fine detail in the periphery, the important feedback interactions occurred between the high-level cortex and the foveal cortex, rather than the region occupied by stimuli in lower areas. A possible reason for this arrangement is that the fovea is normally used for processing fine details due to its high-resolution capacity, and normally we move our eyes to bring the image details into the fovea. In our experiment, eye movement was not allowed, yet the system may have built-in mechanisms to engage the foveal cortex in the feedback stage, maybe in anticipation that the input image would end up in the fovea, as is typically the case. Anticipatory activation before saccades has also been found in other cases; for instance, neurons in some brain areas become responsive to the visual stimuli that will be brought into their receptive field by an imminent saccade, described as predictive remapping (28–31). It is possible that the natural tendency to make a saccade when objects were presented in the periphery in our paradigm initiated the predictive remapping mechanism. The result that foveal noise was not effective when subjects planned and executed a saccade away from the peripheral object supports this possibility (Fig. 4B). However, our results show that this process was delayed with the added mental rotation operation, which would either suggest that the saccade planning itself was also delayed, or that a mechanism for predictive remapping also has a flexible and task-dependent timing. Thus, our results, on the one hand, provide further support and refinement for the high-resolution buffer hypothesis and reverse hierarchy theory, but on the other hand they also provide important constraints that theories of object recognition need to satisfy.

A total of 88 subjects (age range 18–26; 44 female) participated in these experiments, with some subjects participating in more than one experiment (see SI Methods for details). All subjects provided written informed consent before the experiments, and protocols were approved by the institutional review board of the Institute of Biophysics, Chinese Academy of Sciences.