Pupil dilation deconvolution reveals the dynamics of attention at high temporal resolution
Edited by John R. Anderson, Carnegie Mellon University, Pittsburgh, PA, and approved April 18, 2012 (received for review February 1, 2012)
Abstract
The size of the human pupil increases as a function of mental effort. However, this response is slow, and therefore its use is thought to be limited to measurements of slow tasks or tasks in which meaningful events are temporally well separated. Here we show that high-temporal-resolution tracking of attention and cognitive processes can be obtained from the slow pupillary response. Using automated dilation deconvolution, we isolated and tracked the dynamics of attention in a fast-paced temporal attention task, allowing us to uncover the amount of mental activity that is critical for conscious perception of relevant stimuli. We thus found evidence for specific temporal expectancy effects in attention that have eluded detection using neuroimaging methods such as EEG. Combining this approach with other neuroimaging techniques can open many research opportunities to study the temporal dynamics of the mind’s inner eye in great detail.
Sign up for PNAS alerts.
Get alerts for new articles, or get an alert when an article is cited.
The size of the human eye pupil often is used as a measure of mental effort because it is assumed that the pupil size is related to the amount of cognitive control (1), attention (2), and cognitive processing (3) required by a given task. However, because the pupillary response is slow—pupil size increases slowly in response to a relevant event and peaks after approximately 1 s—measuring effort by assessing pupil dilation traditionally was reserved for slow tasks or tasks in which meaningful events are well separated in time.
Here we show that high-temporal-resolution (∼10 Hz) tracking of attention and cognitive processes can be obtained from the slow pupillary response (∼1 Hz). Using automated dilation deconvolution, based on the quantitative analysis of the pupillary response (4), we isolated and tracked the dynamics of attention in a fast-paced temporal attention task, allowing us to uncover the amount of mental activity that is critical for conscious perception of relevant stimuli.
We modeled the pupillary response as a function of a series of cognitive events, extending the approach of Hoeks and Levelt (4). In their model, each cognitive event is associated with an attentional pulse, which is assumed to trigger a dilation of the pupil as a function of that attentional pulse’s strength. The number of pulses, the temporal location of pulses, and the strength of each pulse that add up to a dilation of the pupil can be set at specific values or can be free to vary. Given the additive nature of the pupillary response (4), a prediction for the pupillary response pattern evoked by a task can be derived by convolving the attentional pulses with a pupillary response function, similar to the convolution process in functional MRI (fMRI) analyses. This pupillary response function is described as an Erlang gamma function, and its constants have been determined empirically (4). Apart from predicting a pupillary response, this method also can be used to derive a pattern of pulses that underlies an observed pupillary response by means of a deconvolution process. However, the method described by Hoeks and Levelt (4) is limited to an isolated dilation during a relatively short interval. Our extensions allow deconvolving longer intervals in which multiple independent cognitive events might take place. An optimization algorithm takes a vector of the temporal locations of the attentional pulses and determines the strength associated with each pulse by minimizing the mismatch between the normalized observed and the predicted pupillary response. Concurrent with the estimation of the strength of the pulses, the slope accounting for linear drifts in the data is estimated. Normalizing the pupil dilation and accounting for the slow drift allows the analysis of longer intervals of pupillary data, up to complete experimental trials with multiple cognitive events.
Because the temporal resolution is determined solely by the temporal locations defined in the attentional pulses vector, this deconvolution method allows the use of pupil dilation to assess the involvement of the cognitive system at a much higher temporal resolution than the slow pupillary response seemingly would allow. To demonstrate the benefits of extracting information with high temporal resolution from pupillary response patterns, we recorded pupil size during an attentional blink (AB) task (5). The AB is a deficit in reporting the second of two targets presented in close temporal succession (∼150–500 ms) within a stream of nontargets (i.e., distractors), reflecting temporal limitations of attention. To assess the involvement of the cognitive system at all stages of the AB task, the onset of all stimuli, both distractors and targets, are represented in the attentional pulses vector.
The AB paradigm is particularly well suited to test our method for the following three reasons. First, it has been shown that a single target presented in a rapid stream of nontargets elicits a pupillary response (6). Second, because of the fast-paced nature of the task, with stimuli typically presented every 100 ms, the pupillary responses to two closely succeeding targets are bound to overlap. Thus, the paradigm allowed us to test whether our method can identify the cognitive involvement of targets presented at a frequency higher than a single pupillary response. Third, the AB is one of the most intensively studied phenomena in attention research in the past two decades and has been investigated with various neuroimaging techniques, including EEG and magnetoencephalography (MEG), both of which excel in the temporal domain (7). Given this extensive literature, we can compare and validate our results.
Results
Behavioral Results.
An example of a trial and the behavioral results of the AB task are shown in Fig. 1. Participants were required to detect and identify zero, one, or two unspecified target letters that were presented within a sequential stream of nontarget digits (i.e., distractors). In trials in which the stream contained two targets, the second target was presented either in the AB critical period at 300 ms after the first target or outside the critical period. Trials were presented in a random order. In the remainder of this paper, we focus on no-target trials, one-target trials, and dual-target trials in which the second target was presented 300 ms after the first target.
Fig. 1.
Isolating Temporally Proximal Stimuli.
To test whether it was possible to isolate the attentional pulses corresponding to two distinct but temporally proximal stimuli, we compared the one-target trials (in which only a single meaningful stimulus was presented and correctly identified) with the no-blink trials (the condition in which both targets were correctly identified). The orange line in Fig. 2A represents one-target trials; as expected, only a single pupillary response is visible. In contrast, the green line shows that the dilation response to the first and second target is not clearly separable in the no-blink trials, because the observed normalized dilation consists of one complex dilation pattern. However, dilation deconvolution clearly identified two distinct groups of attentional pulses, as is shown in Fig. 2B. Similar activity has been reported in EEG studies (8, 9), but rarely are the effects in EEG studies as clearly separable as shown in Fig. 2B.
Fig. 2.
Cognitive Involvement of the First Target.
To examine the relationship between the cognitive involvement associated with the first target and the occurrence of a subsequent AB, we contrasted the no-blink trials vs. the blink trials (trials in which a second target was presented but not identified correctly). The pupil size corresponding to these two types of trials, which feature identical stimuli and instructions but have different behavioral outcomes, is shown in Fig. 2C. As expected, the strengths of the attentional pulses associated with the second target were lower during blink trials (Fig. 2D). This finding is in line with EEG studies in which target-specific activity time-locked to the second target typically is absent (8–11). Interestingly, the estimated strength for the first target was higher for blink trials than for no-blink trials. Thus, the current study replicates the finding that processing demands of the first target are critical for the occurrence of the AB (12–15). However, whereas previous studies involved experimental manipulations such as changing the difficulty of the first target, this study demonstrates this phenomenon without any experimental manipulation, mirroring the results of an MEG study (16). This result suggests that our method is highly sensitive and can account for subtle fluctuations of attention that have remained elusive in fMRI or EEG measurements.
Expectancy Effects in Attention.
Because of the structure of the task, participants might have come to expect a second target if a first target was perceived. To test whether we could find any expectancy effects in temporal attention, we compared one-target trials with no-target trials. Fig. 2E shows the normalized pupil dilation for both types of trials and shows a fairly stable pupil dilation over the whole trial for the no-target trials, whereas the presentation of a target in the one-target trials results in an increased dilation of the pupil that peaks about 1,000 ms after presentation. Fig. 2F depicts the attentional pulses derived from the dilation deconvolution method. Up to 500 ms, the plot closely resembles what might be extracted from the normalized pupil dilation plot, because the increased strength for the initial pulses corresponds with the processing of the first stimulus. Although the normalized pupil dilation in Fig. 2E does not show any salient effects after the initial dilation peak, the strengths of the attentional pulses show a second increase for the one-target trials observed at about 1,000 ms. Because this time frame is associated with the effect of a second target in dual-target trials, this increase in strength for the attentional pulses most likely reflects an expectancy effect. Indeed, more general effects of expectancy have been found in attentional processing (17). However, the expectancy effect in the AB reported here might easily have eluded discovery because of its fairly small effect size in comparison with the effects observed with the normal processing of target stimuli. For example, the much stronger effects of the processing of the first target might obscure the expectancy effect in EEG studies, as it did in the raw pupillary patterns shown in Fig. 2E.
Discussion
As shown empirically in this study, dilation deconvolution can provide valuable information regarding the occurrence and timing of attentional processes that underlie human cognition. The results of the experiment show that dilation deconvolution can track and isolate attentional processing of multiple events at close temporal proximity, thus revealing the temporal dynamics of the mind’s eye at a surprisingly high resolution.
Although the dilation deconvolution presented here is based on the pupillary response function of Hoeks and Levelt (4), other models of the pupillary response could be implemented as well. For example, the more complex bimodal pupillary response function as described by O’Neill and Zimmerman (18) could be implemented in our pupil deconvolution method. Although in theory distinguishing between sympathetic and parasympathetic influences on pupil dilation might provide additional information, implementing this response function would have doubled the number of parameters (or even tripled them, depending on whether the distance between sympathetic and parasympathetic pulses is fixed). We therefore opted for the more parsimonious pupillary response function of Hoeks and Levelt.
The reliability of the present pupil deconvolution method is strengthened by the consistency of our results with those in previous neuroimaging studies. Our results also are consistent with AB theories (7, 19). In addition, evidence of subtle expectancy effects in temporal attention was obtained, as was direct evidence for the crucial role of the processing demands of the first target. This pattern of results supports the notion that both pupil dilation and the AB might be linked closely to the activity of the noradrenergic system (20–22). Finally, it is worth noting that a practical advantage of using our pupil dilation approach is that, in comparison with other neuroimaging techniques, it is a relatively cost-effective and accessible method to measure cognitive workload and attention.
In conclusion, our method allows the analysis of the processing of stimuli that are presented in close temporal succession, both within a trial (as demonstrated here) and between trials of tasks in which an induced pupil dilation on a given trial overlaps with the pupil dilation of a subsequent trial (23). Combining this method with other behavioral and neuroimaging methods can open a wide range of interesting research opportunities.
Methods
The study was approved by the Psychology Ethical Committee of the University of Groningen. In total, 20 students of the University of Groningen participated in the study. The students received course credits in return for their participation. Informed consent was obtained from each participant. Two participants were excluded from the analyses because of artifacts and poor performance. After exclusion, nine male and nine female participants with normal vision, age 18–28 y, remained.
AB Task.
Participants performed a standard AB task, requiring unspeeded identification of zero to two target letters (uppercase consonants excluding Q, V, and Y) presented in a sequential stream of nontarget digits (excluding 0 and 1). Each item was presented in black on a white background in a bold 18-point Courier New font in the middle of a 17-inch computer screen set at a resolution of 1,024 × 786 pixels with a refresh rate of 100 Hz. Viewing distance was ∼50 cm. The stream consisted of 50 items presented at a rate of 100 ms per item. The experiment contained one practice block of 32 trials and two experimental blocks of 160 trials each, with a short break between the two blocks. Each trial contained zero, one, or two targets. The first target (T1) always was the tenth item presented in the stream. The second target (T2) was the 1st, 3rd, 8th, 15th, or 30th item after T1 (i.e., lag 1, lag 3, lag 8, lag 15, or lag 30, respectively). Each block contained 30 trials for lag 1, lag 3, and lag 15, because these were the lags of interest, and five trials for each of the two filler lags (i.e., lag 8 and 30). Within a block, these dual-target trials were intermixed randomly with 30 no-target and 30 one-target trials. Each trial started with the presentation of a fixation-cross in the middle of the screen. Participants were instructed to press the spacebar to initiate the trial. The fixation-cross disappeared 900 ms after the space bar was pressed, and the first item appeared on the screen. After all targets and distractors in a trial had been presented, a comma or a dot was shown on the screen for 100 ms.
Before reporting the targets, participants had to indicate whether the last character was a comma or a dot. This comma/dot task was included to encourage the participants to remain fixated to the center of the screen throughout stimulus presentation. Following the comma/dot task, participants were prompted to report which letters were presented by pressing the corresponding letters on the keyboard in the order in which the letters had appeared or to press the spacebar if no target was observed. The accuracy data were analyzed using binominal mixed-effects models available in the lme4 package of the statistical software package R (www.r-project.org). In the analysis of T2 accuracy, only trials in which T1 was reported correctly were analyzed.
Preprocessing of Pupil Size Data.
Pupil size was measured using the EyeLink 1000 eye-tracker (www.sr-research.com). Data were sampled at 250 Hz and down-sampled to 50 Hz. Because we were interested primarily in the pupil dilation in response to the first target and the second target presented at lag 3 and wanted to avoid artifacts caused by the eye blinks that frequently occur during the last phase of the trial, the data were segmented in segments of 4,300 ms and time locked to 200 ms before stream onset. Segments containing eye blinks were excluded from the analysis. The average pupil size of the interval from −200 ms to 0 ms to stream onset was used as a baseline. Data were normalized by calculating the percentage increase of the pupil size compared with the baseline for each data point 
Pupillary Response Model.
The deconvolution of the dilation patterns is driven by an optimization algorithm. By calculating the misfit between an observed pupil dilation pattern and a pupil dilation pattern derived from a sequence of attentional pulses that is slightly changed per iteration, the optimization algorithm effectively performs the deconvolution by repeated convolutions. In total, 34 attentional pulses were modeled corresponding to the first 34 items in the AB stream. We limited our analyses to these pulses, because the development of the pupillary response from pulse onset to maximum peak dilation falls within the 4,300-ms segment for these pulses. The remaining 16 items were not modeled, because a large part of the pupillary response to these items exceeds the segment and would thus not be estimated properly by the model.
The predicted pupil dilation pattern was calculated by convolving the attentional input i = {w1,w2,w3,…,w32,w33,w34} with the Erlang gamma function 
; where wl is the strength of the attentional pulses, l is the position of each pulse in vector i, n is the number of layers set at 10.1, tmax = 930 is the position of the maximum response, and s = 1/1027 is a constant to scale the pupillary response function. Hoeks and Levelt (4) empirically determined the constants n and tmax. The temporal location of each pulse was fixed at the onset of each stimulus (i.e., 100 ms between each pulse). The predicted pupil dilation then was obtained by x = l ⋅ b + (i * h); where b is the slope parameter that accounts for drifts in the data. The strengths wl were fit by minimizing the mean square error 
using a linearly constrained optimization algorithm, where xp is the predicted pupil dilation and xm is the measured data. The initial values of the strengths for the optimization algorithm were chosen randomly from a range of 0.2 to 0.5 for each pulse. Strengths were constrained so that values below zero were not allowed. Because this optimization method could result in local minima, this process was repeated 200 times per combination of participant and condition to obtain stable strength averages of the attentional pulses. An implementation of the pupil dilation deconvolution method as described above and with the preprocessed data of the experiment can be found in SI Text.
; where wl is the strength of the attentional pulses, l is the position of each pulse in vector i, n is the number of layers set at 10.1, tmax = 930 is the position of the maximum response, and s = 1/1027 is a constant to scale the pupillary response function. Hoeks and Levelt (4) empirically determined the constants n and tmax. The temporal location of each pulse was fixed at the onset of each stimulus (i.e., 100 ms between each pulse). The predicted pupil dilation then was obtained by x = l ⋅ b + (i * h); where b is the slope parameter that accounts for drifts in the data. The strengths wl were fit by minimizing the mean square error using a linearly constrained optimization algorithm, where xp is the predicted pupil dilation and xm is the measured data. The initial values of the strengths for the optimization algorithm were chosen randomly from a range of 0.2 to 0.5 for each pulse. Strengths were constrained so that values below zero were not allowed. Because this optimization method could result in local minima, this process was repeated 200 times per combination of participant and condition to obtain stable strength averages of the attentional pulses. An implementation of the pupil dilation deconvolution method as described above and with the preprocessed data of the experiment can be found in SI Text.Analysis of Attentional Pulses.
The attentional pulses generated by the model were tested using nonparametric permutation tests that are used commonly in fMRI studies. Permutation tests have the advantage over parametric tests of being more conservative and relying on fewer assumptions (24, 25). In the analyses, the first three and last three pulses were excluded. In each test, two conditions were compared. The permutation distribution was obtained by randomly labeling the conditions for each pulse per participant 10,000 times. The null hypothesis stated that there would be no difference between condition A and condition B; thus the difference score (i.e., A − B) would equal zero for a pulse if there were no effect. The statistic used in the test was the standard t-statistic. To account for multiple comparisons within the permutation test, a single threshold test was performed per permutation test. Thus, each pulse with a t-value exceeding the single threshold would reject the null hypothesis. In total, three permutation tests were done. In the first test, the one-target trials were compared with the trials in which T2 was presented at lag 3 and correctly identified (i.e., no-blink trials). In the second test, lag-3 trials with correct T2 responses were compared with lag-3 trials with incorrect T2 responses (i.e., blink trials). Finally, no-target trials were compared with one-target trials.
Acknowledgments
We thank the Laboratory of Experimental Ophthalmology of the University Medical Center, Groningen, for the use of the eye-tracking equipment. This research was supported by a grant of the Behavioural and Cognitive Neurosciences Research School of the University Medical Center Groningen, University of Groningen.
Supporting Information
Supporting Information (PDF)
Supporting Information
- Download
- 24.51 KB
sd01.txt
- Download
- 1.52 MB
sd02.txt
- Download
- 5.08 KB
sd03.txt
- Download
- 3.91 KB
References
1
M Jepma, S Nieuwenhuis, Pupil diameter predicts changes in the exploration-exploitation trade-off: Evidence for the adaptive gain theory. J Cogn Neurosci 23, 1587–1596 (2011).
2
SP Verney, E Granholm, SP Marshall, Pupillary responses on the visual backward masking task reflect general cognitive ability. Int J Psychophysiol 52, 23–36 (2004).
3
S Moresi, et al., Pupil dilation in response preparation. Int J Psychophysiol 67, 124–130 (2008).
4
B Hoeks, W Levelt, Pupillary dilation as a measure of attention: A quantitative system analysis. Behav Res Methods 25, 16–26 (1993).
5
JE Raymond, KL Shapiro, KM Arnell, Temporary suppression of visual processing in an RSVP task: An attentional blink? J Exp Psychol Hum Percept Perform 18, 849–860 (1992).
6
CM Privitera, LW Renninger, T Carney, S Klein, M Aguilar, Pupil dilation during visual target detection. J Vis 10, 3 (2010).
7
S Martens, B Wyble, The attentional blink: Past, present, and future of a blind spot in perceptual awareness. Neurosci Biobehav Rev 34, 947–957 (2010).
8
S Martens, J Munneke, H Smid, A Johnson, Quick minds don’t blink: Electrophysiological correlates of individual differences in attentional selection. J Cogn Neurosci 18, 1423–1438 (2006).
9
P Craston, B Wyble, S Chennu, H Bowman, The attentional blink reveals serial working memory encoding: Evidence from virtual and human event-related potentials. J Cogn Neurosci 21, 550–566 (2009).
10
EK Vogel, SJ Luck, KL Shapiro, Electrophysiological evidence for a postperceptual locus of suppression during the attentional blink. J Exp Psychol Hum Percept Perform 24, 1656–1674 (1998).
11
C Kranczioch, S Debener, AK Engel, Event-related potential correlates of the attentional blink phenomenon. Brain Res Cogn Brain Res 17, 177–187 (2003).
12
SM Wierda, H van Rijn, NA Taatgen, S Martens, Distracting the mind improves performance: An ERP study. PLoS ONE 5, e15024 (2010).
13
EG Akyürek, M Leszczyński, A Schubö, The temporal locus of the interaction between working memory consolidation and the attentional blink. Psychophysiology 47, 1134–1141 (2010).
14
C Ouimet, P Jolicoeur, Beyond task 1 difficulty: The duration of T1 encoding modulates the attentional blink. Vis Cogn 15, 290–304 (2007).
15
S Martens, K Elmallah, R London, A Johnson, Cuing and stimulus probability effects on the P3 and the AB. Acta Psychol (Amst) 123, 204–218 (2006).
16
K Shapiro, F Schmitz, S Martens, B Hommel, A Schnitzler, Resource sharing in the attentional blink. Neuroreport 17, 163–166 (2006).
17
Á Correa, J Lupiáñez, E Madrid, P Tudela, Temporal attention enhances early visual processing: A review and new evidence from event-related potentials. Brain Res 1076, 116–128 (2006).
18
WD O’Neill, S Zimmerman, Neurologial interpretations and the information in the cognitive pupillary response. Methods Inf Med 39, 122–124 (2000).
19
PE Dux, R Marois, The attentional blink: A review of data and theory. Atten Percept Psychophys 71, 1683–1700 (2009).
20
G Aston-Jones, JD Cohen, An integrative theory of locus coeruleus-norepinephrine function: Adaptive gain and optimal performance. Annu Rev Neurosci 28, 403–450 (2005).
21
S Nieuwenhuis, EJ De Geus, G Aston-Jones, The anatomical and functional relationship between the P3 and autonomic components of the orienting response. Psychophysiology 48, 162–175 (2010).
22
S Nieuwenhuis, MS Gilzenrat, BD Holmes, JD Cohen, The role of the locus coeruleus in mediating the attentional blink: A neurocomputational theory. J Exp Psychol Gen 134, 291–307 (2005).
23
MS Gilzenrat, S Nieuwenhuis, M Jepma, JD Cohen, Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cogn Affect Behav Neurosci 10, 252–269 (2010).
24
TE Nichols, AP Holmes, Nonparametric permutation tests for functional neuroimaging: A primer with examples. Hum Brain Mapp 15, 1–25 (2002).
25
AP Holmes, RC Blair, JD Watson, I Ford, Nonparametric analysis of statistic images from functional mapping experiments. J Cereb Blood Flow Metab 16, 7–22 (1996).
Information & Authors
Information
Published in
Classifications
Submission history
Published online: May 14, 2012
Published in issue: May 29, 2012
Keywords
Acknowledgments
We thank the Laboratory of Experimental Ophthalmology of the University Medical Center, Groningen, for the use of the eye-tracking equipment. This research was supported by a grant of the Behavioural and Cognitive Neurosciences Research School of the University Medical Center Groningen, University of Groningen.
Notes
This article is a PNAS Direct Submission.
Authors
Competing Interests
The authors declare no conflict of interest.
Metrics & Citations
Metrics
Altmetrics
Citations
Cite this article
Pupil dilation deconvolution reveals the dynamics of attention at high temporal resolution, Proc. Natl. Acad. Sci. U.S.A.
109 (22) 8456-8460,
https://doi.org/10.1073/pnas.1201858109
(2012).
Copied!
Copying failed.
Export the article citation data by selecting a format from the list below and clicking Export.
Cited by
Loading...
View Options
View options
PDF format
Download this article as a PDF file
DOWNLOAD PDFLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Personal login Institutional LoginRecommend to a librarian
Recommend PNAS to a LibrarianPurchase options
Purchase this article to access the full text.