Better explicating the strengths and shortcomings of these models will help refine projections and improve transparency in the years ahead.
Image credit: Witsawat.S.
Edited by Charles R. Gallistel, Rutgers University, Piscataway, NJ, and approved October 1, 2015 (received for review August 26, 2015)
Significance
The Rescorla–Wagner model of associative learning has guided research in behavioral and neural sciences for several decades. Although phenomena associated with the model have previously been linked to nucleo-olivary inhibition, many questions regarding the neural mechanisms underlying the model still remain. In this paper, we present evidence from our eyeblink conditioning setup, indicating that the variables used in Rescorla and Wagner's model have physiological correlates.
Abstract
A central tenet of Rescorla and Wagner’s model of associative learning is that the reinforcement value of a paired trial diminishes as the associative strength between the presented stimuli increases. Despite its fundamental importance to behavioral sciences, the neural mechanisms underlying the model have not been fully explored. Here, we present findings that, taken together, can explain why a stronger association leads to a reduced reinforcement value, within the context of eyeblink conditioning. Specifically, we show that learned pause responses in Purkinje cells, which trigger adaptively timed conditioned eyeblinks, suppress the unconditional stimulus (US) signal in a graded manner. Furthermore, by examining how Purkinje cells respond to two distinct conditional stimuli and to a compound stimulus, we provide evidence that could potentially help explain the somewhat counterintuitive overexpectation phenomenon, which was derived from the Rescorla–Wagner model.
The Rescorla–Wagner model of associative learning is arguably the most influential theory of associative learning in recent history. The model successfully predicted several behavioral phenomena (1). Moreover, in contrast to the Hebbian model, it is a prime example of an error correction process in which behavioral changes result from violation of expectations (2). A central tenet of the model is that the reinforcement value of a paired trial depends on the existing associative strength between the presented stimuli (3, 4). Neural mechanisms for several phenomena related to the Rescorla–Wagner model have already been proposed (5⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓–16). In this paper, we present evidence from our eyeblink setup that builds on and advances prior thinking regarding the physiological basis of the Rescorla–Wagner model.
In eyeblink conditioning, repeated presentations of a neutral conditional stimulus (CS), such as a tone or a light stimulus, followed by a blink-eliciting unconditional stimulus (US), such as an air puff to the cornea or electrical stimulation of the periorbital skin, results in the acquisition of a conditioned blink response (CR). Previous studies have shown that eyeblink conditioning depends on the cerebellum (17, 18) and that the cerebellar cortex plays a crucial role (19). Specifically, conditioned blink responses appear to be triggered by pause responses in GABAergic Purkinje cells (20). These pause responses, which are acquired gradually during conditioning (21), disinhibit cells in the cerebellar nuclei (22), thus allowing them to activate muscles controlling the eyelid (23, 24). Disinhibition of the cerebellar nuclei also activates GABAergic cells that project to the inferior olive (15). Because the US signal enters the cerebellum via the inferior olive (18), we hypothesized that the nucleo-olivary inhibition that arises during learning, suppresses the US signal and therefore reduces the reinforcement value of a paired trial, as postulated in the Rescorla–Wagner model.
A surprising consequence of the model is that if a subject has learned to respond to two different CSs, the reinforcement value of a trial where both CSs and the US are presented will be negative and will cause extinction. This counterintuitive prediction has been confirmed in several learning paradigms including fear conditioning (25, 26), appetitive conditioning (27, 28), and eyeblink conditioning (10). The phenomenon is known as “overexpectation” because the subject may be said to “expect” a US that is more intense than the one it actually receives. We hypothesized that, within the context of eyeblink conditioning, overexpectation occurs because the compound CS results in stronger simple spike suppression in the Purkinje cells and, consequently, a stronger nucleo-olivary inhibition of the US signal, to the extent that the reinforcement value becomes negative.
Results
To test our hypotheses, we recorded simple and complex spikes from 30 Purkinje cells in the C3 zone of the cerebellar cortex in 23 decerebrated ferrets (Fig. 1). The animals had been trained in a classical delay-conditioning paradigm where we alternated between presenting two different CSs, paired with a US. CSA consisted of superior colliculus stimulation (12 cells), stimulation of ipsilateral whiskers (14 cells), or pontine stimulation (2 cells). CSB was always forelimb stimulation. As US, we used periorbital electrical stimulation (17 cells) or climbing-fiber stimulation (13 cells). When we found a cell in a trained animal, we examined how the cell responded to each CS as well as to simultaneous presentation of both CSs. In a set of 10 cells, we also examined how the presentation of a CS influences the probability that a subsequent periorbital US elicits a complex spike. To test this, we compared the probability that a periorbital US would elicit a complex spike if it was preceded by CSA, CSB, CSA + CSB, or no CS.
Fig. 1.
Experimental setup and field potential recordings. (A) Illustration of the experimental setup including relevant afferent and efferent cerebellar pathways. As CS, we used stimulation of the forelimb, superior colliculus (SC) pontine nuclei, or whiskers. As US, we used stimulation of climbing fibers (CF) or the periorbital skin (eye). CN, cerebellar nuclei; GC, granule cells; NO, nucleo-olivary pathway; PC, Purkinje cell. (B) Examples of field potentials elicited on the cerebellar cortex following stimulation of cerebellar afferents. (C) A Purkinje cell recording showing simple spikes, complex spikes, and a learned pause response or Purkinje cell CR.
The fact that a subject can acquire CRs to more than one CS raises the question of whether an individual Purkinje cell can acquire pause responses to more than one CS, or if separate stimuli require separate Purkinje cells. Of 26 Purkinje cells, 23 exhibited a reduced firing rate (<100%, relative to background), in response to each of the two CSs, and despite limited training we found 10 cells with distinctive pause responses to each CS (Fig. 2). These results are important because we can say, with certainty, that individual Purkinje cells can acquire conditioned pause responses to at least two CSs that belong to different sensory modalities.
Fig. 2.
Single Purkinje cells can acquire pause responses to two different CSs. (A) Raster plots showing the responses of four Purkinje cells to 20 trials of CSA (superior colliculus, n = 3, or whisker air puff, n = 1, red) and CSB (forelimb, n = 4, green). (B) Composite raster diagram with spikes from 10 Purkinje cells showing pause responses to CSA (superior colliculus, n = 6, or whisker air puff, n = 4) and CSB (forelimb, n = 10). (C) Line diagram showing average response profiles to each of the two CSs in the 10 cells.
Having established that Purkinje cells can acquire pause responses to two different CSs, we proceeded to analyze the effect of presenting both stimuli simultaneously. Presenting this compound CS resulted in a stronger suppression of Purkinje cell activity compared with when either of the two CSs was presented individually (Fig. 3). Of the 26 cells, 18 exhibited a lower firing rate in response to the compound CS. A repeated-measures ANOVA showed that the activity during the CS depended on the type of stimulation used [F(2,25) = 14.74, P < 0.0001***]. The firing rate on trials with the compound CS was lower than on trials where either CSA [F(1,25) = 35.09, P < 0.00001***] or CSB [F(1,25) = 8.1, P = 0.0087**] was presented alone. As illustrated in Fig. 3D, the effect was most pronounced in the latter part of the CS, which is consistent with earlier reports (21) and with the known time course of nucleo-olivary inhibition (14, 29).
Fig. 3.
Presentation of two CSs simultaneously results in a stronger pause response than any of the two CSs elicits individually. (A) Sample recordings from one cell showing five trials for each CS (red and green) as well as five trials in which both CSs were presented as a compound (blue). (B) Raster plots showing how two cells respond to two different CSs (red and green), and to both CSs combined (blue). (C) Boxplot showing the distribution of Purkinje cell activity during CSA (red), CSB (green), and CSA ± CSB (blue). (D) Line diagram illustrating the average profile of the Purkinje cell pause response (mean ± SEM), for CSA (red), CSB (green), and CSA ± CSB (blue).
To test whether a suppression of Purkinje cell activity can reduce the probability that a subsequent periorbital US elicits a complex spike, we trained 10 Purkinje cells until they appeared to exhibit reduced firing rates in response to both CSA (superior colliculus stimulation, n = 6, or whisker air puff, n = 4) and CSB (forelimb stimulation). We then examined whether the probability that the periorbital US elicited a complex spike was dependent on whether a CS preceded the stimulus. The analysis showed that the periorbital US, presented alone, elicited complex spikes on 79 ± 5% of trials. The corresponding values when the periorbital stimulation was preceded by CSA and CSB were 49 ± 11% and 38 ± 12%, respectively. Combining CSA and CSB reduced the percentage to 37 ± 12%. A repeated-measures ANOVA revealed a significant difference in the percentage of trials in which the periorbital US elicited a complex spike, depending on the type of CS that preceded the stimulus [F(3,9) = 13.41, P < 0.0001***] (Fig. 4A). However, the compound CS did not reduce the complex spike probability more than presenting the individual CSs (Fig. 4A). This apparent anomaly may be due to greater variability in the effectiveness of the different CSs in suppressing simple spike activity. To examine this more closely, we analyzed cells in which each CS induced at least a 15% reduction in Purkinje cell activity. In the six cells meeting this criterion, the average probability that a periorbital US would elicit a complex spike was 26 ± 9% following CSA, 19 ± 13% following CSB, and 13 ± 12% following CSA + CSB. This indicates that a compound CS does indeed induce a stronger suppression of the inferior olive than the constituent CSs presented individually, as long as each CS induces some suppression of Purkinje cell activity.
Fig. 4.
The probability that a periorbital US elicits a complex spike is reduced if it is preceded by a CS-induced suppression of Purkinje cell activity. (A) The probability that a periorbital stimulus elicits a complex spike depends on whether it is preceded by a CS. (B) Simple spike activity during the presentation of different CS combinations or without a CS, relative to the background-firing rate in the same cell. (C) Correlation between the suppression of Purkinje cell activity induced by a CS and the probability that a subsequent periorbital stimulus elicits a complex spike in the same Purkinje cell. Each point in the scatterplot corresponds to the average value for each CS type, for each cell (1 cell = 4 points).
The Rescorla–Wagner model predicts that the reinforcement value should decrease gradually as the learned association between the stimuli gets stronger. In the context of eyeblink conditioning, this should mean that a CS that elicits a strong suppression of simple spike activity also induces a strong suppression of the inferior olive and is therefore associated with a lower complex spike probability following a periorbital US. To test this, we calculated the correlation between the Purkinje cell firing rate during the presentation of a given CS and the probability that a periorbital US, presented at the CS offset, elicited a complex spike. As shown in Fig. 4C, there was a strong correlation between the Purkinje cell suppression induced by a certain CS and the probability that a subsequent periorbital US elicited a complex spike (r = 0.58). Excluding cases in which the simple spike activity during the CS exceeded 200% resulted in an even stronger correlation (r = 0.67). This suggests that the strength of the Purkinje cell suppression, which reflects the amount of learning that has taken place, regulates the probability that a periorbital US will be transmitted to the cerebellar cortex, and that this occurs in a graded manner.
Discussion
Animals can acquire conditioned blink responses to a wide range of stimuli, including tones and light stimuli (30), forelimb stimulation, whisker stimulation (31), lateral geniculate nucleus stimulation, superior colliculus stimulation (32), and mossy fiber stimulation (33). Consistent with their purported role in eyeblink conditioning, Purkinje cells can also acquire conditioned pause responses to different CSs, including stimuli applied to the forelimb, mossy fibers (21), superior colliculus (34), and parallel fibers (35). The present results show that individual Purkinje cells can also acquire pause responses to two CSs of different modalities. This is consistent with a recent study in which it was shown that Purkinje cells can acquire pause responses to two tones with different frequencies (36). Next, we examined the effect of presenting two CSs simultaneously, as a compound CS. At the behavioral level, this will give rise to a stronger blink response (10), and we predicted that the same would hold true for CS-induced suppression of Purkinje cell activity. This prediction was confirmed. Thus, cells that had been trained to two different CSs produced a stronger pause response when these were combined compared with when either CS was presented alone.
A Purkinje cell initiates behavior by disinhibiting the cerebellar nuclei, allowing them to initiate motor activity (37). However, disinhibiting the cerebellar nuclei also leads to inhibition of the inferior olive via the GABAergic nucleo-olivary pathway (6, 7, 16, 38, 39). This can be seen by recording field potentials on the cerebellar surface (40, 41), or the frequency of spontaneous complex spikes in Purkinje cells, during conditioning (14). Because the inhibition is associated with an unusually long delay (42), it will coincide with the arrival of the US (13), which suggests that the nucleo-olivary pathway is part of a negative-feedback system that controls cerebellar learning (5, 16).
Accordingly, the present results show that presentation of a CS that has previously been paired with the US reduces the probability that a periorbital US elicit complex spikes. This means that a suppression of Purkinje cell activity is sufficiently strong to suppress the signal generated by a periorbital US. However, these results do not contradict the idea that the nucleo-olivary inhibition can change the olivary signal in a graded manner, such as by reducing the number of spikes in the climbing-fiber signal (15, 43⇓–45). We also hypothesized that, when each CS induced a suppression of the simple spike activity, presentation of a compound CS would result in an even lower probability of a complex spike following the periorbital US. When we examined only those cells in which each CS suppressed Purkinje cell activity by at least 15%, there was indeed a more pronounced inhibition of complex spikes when both stimuli were presented simultaneously. However, due to the small sample size (n = 6), we could not test these differences statistically.
To assess more accurately how a learned pause response affects the excitability of the inferior olive, we calculated the correlation between the suppression of Purkinje cell activity induced by a CS and the probability that a periorbital US would elicit a complex spike. This analysis showed that a CS that elicits a strong suppression of Purkinje cell activity also induces a strong suppression of periorbitally elicited complex spikes. This is in perfect agreement with the observation that Purkinje cells control cells in the inferior olive from which they receive their climbing-fiber afferents (29, 46).
According to the Rescorla–Wagner model of classical conditioning, the reinforcing value of a paired trial decreases as the US becomes more predictable, or in other words, the extent to which a US is surprising (4). In mathematical terms the change in associative strength (ΔV) on a given trial is given by the following (simplified) version of the original Rescorla–Wagner equation:
In the context of eyeblink conditioning, our results suggest plausible neural correlates of the variables on which the Rescorla–Wagner model is based. As previously mentioned, conditioning leads to a suppression of Purkinje cell activity that also suppresses the inferior olive, and thus the US signal, until an equilibrium is reached. If we assume that λ is determined by the intensity of the US (climbing-fiber signal) and V corresponds to the degree of simple spike suppression induced by the CS, we can begin to explain several behavioral phenomena that have been derived from the Rescorla–Wagner model. For example, extinction occurs when a CS that has a high associative strength (V) is presented without a US (λ = 0), so that (λ − V), and consequently (ΔV), is negative leading to a reduction in associative strength, which is extinction. Consistent with this idea, it has been suggested that the nucleo-olivary pathway is crucial for extinction to occur (9). Another phenomenon, blocking, refers to the fact that conditioning will not occur in response to a CS if it is only presented together with another CS, to which the subject has already been conditioned (47). The absence of learning to the second CS is presumably due to the inhibition of the US induced by the other CS, and without the US, learning will not occur (8). In mathematical terms, (λ − V) will be near zero because the associative strength (V) of the first CS is close to λ and adding a second CS will not change that.
Overexpectation is another consequence of the Rescorla–Wagner model. To account for situations where a compound of two CSs is used, each with its own associative strength, the basic formula needs to be modified so that ΔV becomes proportional to λ − (V1 + V2). If two CSs are already maximally associated with the US (λ = V), the current associative strength of the compound will be V1 + V2, meaning that λ − (V1 + V2), and consequently ΔV, will be negative. The model therefore predicts that presenting two CSs simultaneously, each of which already produces a CR, will result in a partial extinction of the CRs, even if the US is still presented.
Our results are consistent with the idea that overexpectation occurs because the compound CS causes a stronger suppression of the simple spike firing than the individual CSs. The increased suppression could push the excitability of the inferior olive below the equilibrium level, and thus weaken the US signal. However, because many of our cells were trained with a climbing-fiber US, which is not affected by nucleo-olivary inhibition, we cannot prove that the stronger pause response induced by the compound CS causes extinction. To properly test the idea that overexpectation is a result of aggregated nucleo-olivary inhibition, one would have to train animals with a periorbital US and then see that the pause response, and preferably also the overt CR, is extinguished.
Materials and Methods
Subjects and Surgery.
Subjects were 23 decerebrated male ferrets in which 30 extracellular Purkinje cell recordings were made. The ferrets were initially anesthetized in a mixture of O2 and air with 1.5–2% (vol/vol) isoflurane (Baxter Medical), which was later replaced by propofol (10 mg/mL Diprivan; AstraZeneca) administered intravenously. When deep anesthesia had been achieved, a tracheotomy was performed and gas was channeled directly into a tracheal tube. The end-expiratory CO2 concentration, arterial blood pressure, and rectal temperature were monitored continuously and kept within physiological limits throughout the experiment. The animal’s head was fixed in a stereotaxic frame. The skull was then opened on the left side, and the caudal half of the left cerebral hemisphere, together with a substantial part of the thalamus on the left side, were removed by aspiration. The aspiration exposed the cerebellum and the superior and inferior colliculi. The animals were decerebrated by sectioning the brainstem with a blunt spatula 1–2 mm rostral to the superior colliculus. After decerebration, anesthesia was discontinued. With the cerebellum and colliculi exposed, a pool was constructed of cotton-reinforced agar and filled with warm high-density perfluoro carbon liquid (FC-40). To ensure mechanical stability in the tissue, the animals were curarized, artificially ventilated, and were kept hanging by the spine, with the head fixed in the stereotaxic frame. The dura covering the cerebellum was removed, and the cerebellar surface was covered with agarose gel (18–20 mg/mL) to provide recording stability and prevent edema near the site of recording. This study has been reviewed and approved by the Malmö–Lund animal experimentation ethics committee.
Stimulation of Cerebellar Afferents.
Forelimb and periorbital stimulation consisted of electrical pulses (1–3 mA, 1 ms), passed through insulated insect needles. For whisker stimulation, we gave three to six air puffs directed at the ipsilateral whiskers, with a pressure of 5–20 psi. The pressure was reduced if the air puff elicited eye muscle activity. For stimulation of cerebellar afferents, we used insulated tungsten electrodes with a tip of 30 µm. Climbing fibers were stimulated by lowering an electrode 4.0–5.0 mm below the posterior cerebellar surface, at an angle of 45°, 4 mm lateral to the midline and 4 mm rostral to the caudal border of the cerebellar vermis. The superficial layer of the ipsilateral superior colliculus was stimulated by placing an electrode 1 mm medial of the lateral border of the superior colliculus at a depth of ∼100 µm. The pontine nuclei were stimulated by lowering an electrode, at an angle of 90°, down through the middle of the superior colliculus to a depth of 8 mm. When stimulating cerebellar afferents, we used intensities ranging from 50 to 750 µA, with pulse duration of 0.1 ms. To find the optimal stimulation site, single electrical pulses (100 µA) were applied and the evoked field potentials from the cerebellar surface were recorded. Field potentials were used to verify that climbing-fiber and periorbital stimulation were effective and to identify the eyeblink area in the C3 zone of the hemispheral lobule VI. Stimulation sites and stimulus intensities were verified again and adjusted if necessary, when recording the activity of single Purkinje cells.
Training Protocol.
CSA consisted of superior colliculus stimulation (12 cells), stimulation of ipsilateral whiskers (16 cells), or pontine stimulation (2 cells). In all 30 cells, CSB consisted of forelimb stimulation. Forelimb, superior colliculus, and pontine CSs consisted of a 50-Hz, 300-ms stimulation train of electrical pulses. The stimulation intensity ranged between 50 and 300 μA for central stimulation (superior colliculus and pontine stimulation), and 1 and 3 mA for forelimb stimulation. The air puff CS consisted of six 20-ms air puffs, repeated at 20 Hz, with a pressure ranging between 0.3 and 1.3 bar. Before training, we verified that the air puff did not elicit alpha responses. The US consisted of either climbing-fiber stimulation (13 cells) or periorbital stimulation (17 cells). The climbing-fiber US consisted of two 10-ms stimulus trains delivered with a 20-ms interval, each consisting of five impulses at 500 Hz, with an intensity ranging between 100 and 500 μA, applied to the ipsilateral climbing fibers. The periorbital US consisted of three pulses delivered bilaterally to the periorbital skin, repeated at 50 Hz, with a duration of 1 ms and with an intensity ranging between 2 and 3 mA. When examining the effect of CS-induced suppression of Purkinje cell activity on the probability that a periorbital US elicits a complex spike, we used a single periorbital stimuli to avoid interference from stimulus artifacts. We consistently used an intertrial interval of 15 ± 1 s and an interstimulus interval of 300 ms.
Training consisted of alternating blocks of 20 trials of CSA + US, and CSB + US pairings. This training protocol was used until a change in Purkinje cell activity was detected in response to one or both CSs. If only one CS appeared to result in a pause response, we proceeded to train only with the CS that did not elicit a response. If both CSs induced a Purkinje cell CR, we stopped the training and proceeded to test the effect of CSA and CSB individually, as well as the effect of the compound stimulus (CSA + CSB) in three independent blocks of 20 CS-alone trials. To test the effect of the Purkinje cell pause response on the probability that a subsequent periorbital stimulation pulse (eye) would elicit a complex spike, we repeatedly presented, in succession: (i) eye only, (ii) CSA + eye, (iii) CSB + eye, and (iv) CSA + CSB + eye.
Purkinje Cell Recordings.
Extracellular recordings of Purkinje cells, identified by the presence of complex spikes, were performed using 30- to 40-µm metal core diameter, quartz glass-coated platinum–tungsten fiber microelectrodes with an impedance ranging from 5 to 10 MΩ (Thomas Recording). The signal from the microelectrode was fed through a preamplifier and filter module from Digitimer, before entering a Power 1401 AD converter (CED), which passed the signal on to a PC running Spike2, version 7, software. On-line and off-line spike sorting was done in Spike2, and subsequent data analysis was done in Matlab. The following recording strategy was used for obtaining single-cell records: (i) find a Purkinje cell; (ii) verify that periorbital stimulation elicits a complex spike with a latency consistent with the properties of the C3 zone (24, 48); (iii) determine the stimulation threshold for the US; (iv) start paired CS–US presentations and record cell activity; (v) if a cell shows a conditioned response to one CS but not the other, then proceed to train only with the CS that does not elicit a response; (vi) if both CSs elicit Purkinje cell CRs, discontinue the training protocol and test the effects of the CSs individually, as well as the effect of the compound CS on CS-alone trials; if a cell is lost, go back to step 1.
Acknowledgments
This work was supported by Swedish Research Council Grants 2015-00276 (to A.R.), 349-2007-8695 (to The Linnaeus Centre for Cognition, Communication and Learning), and 09899 (to G.H.), and grants from the Söderberg, Krapperup, and Åhlen Foundations.
Footnotes
- ↵1To whom correspondence should be addressed. Email: anders.rasmussen{at}med.lu.se.
Author contributions: A.R., D.-A.J., and G.H. designed research; A.R., R.Z., and F.J. performed research; A.R. and R.Z. analyzed data; and A.R. and G.H. wrote the paper.
The authors declare no conflict of interest.
This article is a PNAS Direct Submission.
Freely available online through the PNAS open access option.
References
Neural basis of the Rescorla–Wagner model
Article Classifications
- Biological Sciences
- Neuroscience
- Social Sciences
- Psychological and Cognitive Sciences
Sign up for Article Alerts
Jump to section
You May Also be Interested in
Did the Caribbean sweep into the western Amazon millions of years ago, shaping the region’s rich biodiversity?
Image credit: Tacio Cordeiro Bicudo (University of São Paulo, São Paulo, Brazil), Victor Sacek (University of São Paulo, São Paulo, Brazil), and Lucy Reading-Ikkanda (artist).
Settlements 4,200 years ago may have suffered from overpopulation before drought and lower temperatures ultimately made them unsustainable.
Image credit: Andrea Ricci.
Mara Reed and Michael Manga explore why Yellowstone's Steamboat Geyser resumed erupting in 2018.
Some songbird parents might improve their own fitness by manipulating their offspring into leaving the nest early, at the cost of fledgling survival, a study finds.
Image credit: Gil Eckrich (photographer).