Musical reward prediction errors engage the nucleus accumbens and motivate learning

Benjamin P. Gold, Ernest Mas-Herrero, Yashar Zeighami, Mitchel Benovoy, Alain Dagher, and Robert J. Zatorre
PNAS February 19, 2019 116 (8) 3310-3315; first published February 6, 2019 https://doi.org/10.1073/pnas.1809855116

  1. Edited by Dale Purves, Duke University, Durham, NC, and approved January 3, 2019 (received for review June 8, 2018)

This article has a Letter. Please see:

See related content:

Significance

Prediction errors are crucial for perception, learning, and adaptability. Can they also explain the abstract pleasures we derive from seemingly nonadaptive behaviors? We present evidence of musically elicited reward prediction errors (RPEs), illustrating that an abstract stimulus without apparent biological value can engage the reward system simply by manipulating expectations. Our results demonstrate that musical events can elicit formally modeled RPEs like those observed for concrete rewards, such as food or money, and that these signals support learning. This extension of the RPE model to music implies that predictive processing might play a much wider role in reward and pleasure than previously realized, and inspires new perspectives on aesthetics as well as potential therapeutic and educational applications.

Abstract

Enjoying music reliably ranks among life’s greatest pleasures. Like many hedonic experiences, it engages several reward-related brain areas, with activity in the nucleus accumbens (NAc) most consistently reflecting the listener’s subjective response. Converging evidence suggests that this activity arises from musical “reward prediction errors” (RPEs) that signal the difference between expected and perceived musical events, but this hypothesis has not been directly tested. In the present fMRI experiment, we assessed whether music could elicit formally modeled RPEs in the NAc by applying a well-established decision-making protocol designed and validated for studying RPEs. In the scanner, participants chose between arbitrary cues that probabilistically led to dissonant or consonant music, and learned to make choices associated with the consonance, which they preferred. We modeled regressors of trial-by-trial RPEs, finding that NAc activity tracked musically elicited RPEs, to an extent that explained variance in the individual learning rates. These results demonstrate that music can act as a reward, driving learning and eliciting RPEs in the NAc, a hub of reward- and music enjoyment-related activity.

Music is one of life’s greatest pleasures (1). Like many other pleasures (2), enjoying music reliably engages critical components of the reward system, including the nucleus accumbens (NAc), caudate, orbitofrontal cortex, anterior cingulate, insula, and amygdala (3⇓–5). Of these, the activity of the NAc most strongly correlates with ratings of music liking and wanting, as does its functional connectivity to the auditory cortices, orbitofrontal cortex, anterior cingulate, and amygdala (3, 4, 6).

A growing body of research suggests that abstract pleasures may become rewarding by manipulating expectations (7⇓–9). As music unfurls over time, often across many highly regular structures, it is particularly well suited to manipulate expectations, as proposed by musicological and psychological models (10⇓⇓–13). Although other mechanisms likely also contribute to musical pleasure, such as social interaction (14), associations (15), or beauty (16), the few empirical studies of musical expectancy and affective responses all suggest that expectations account for some of music’s strongest emotional and pleasurable effects (17⇓–19).

Prediction is foundational to perception, cognition, and behavior, and is crucial for adaptive fitness in a dynamic world. Correct predictions allow us to anticipate our needs and environments, while incorrect ones help us adapt as these change. Predictions also facilitate information processing: Fully predicted events validate preexisting models, while surprises indicate the extent of error in the prediction (20⇓–22), prompting refinements in synaptic weights and neural circuits to update predictions and/or behaviors (23). Considering that neurons express prediction errors across multiple levels of processing, from primary perception to complex decision making (23, 24), the “predictive coding” theory proposes that minimizing these errors might be the brain’s central organizing principle (20).

When a prediction concerns some expected value, its update depends on both the magnitude and direction of the error (i.e., precisely how good or bad the outcome was relative to the expectation) (22). Bidirectional reward prediction errors (RPEs) are thus a special case of prediction errors that motivate reinforcement learning, or the maximization of rewards and the minimization of punishments (22). RPE signals in human and nonhuman animals arrive in the NAc from dopaminergic midbrain neurons (25⇓–27), coinciding in at least some cases with emotional and psychophysiological arousal (28). The NAc also receives inputs from the ventral prefrontal cortex, amygdala, hippocampus, and thalamus, and is therefore well positioned to integrate cognitive and emotional information for action selection (29). Although RPEs can also be observed in other brain regions, human neuroimaging studies consistently find their greatest functional correlate in the NAc (30, 31).

Dopamine transmission also seems to play a role in music enjoyment, increasing in the NAc during moments of intense musical pleasure and in the caudate during the anticipation of these events (3). Integrating this expectation-related activity of NAc dopamine with the pleasurable significance of manipulating expectations and the NAc’s central role in pleasure, current models posit that this network might encode dopaminergic RPEs during pleasurable music listening (12, 13), and that these signals could help to explain the strong emotional, psychophysiological, and pleasurable impact of musical surprises (17⇓–19).

RPE computation is thus a promising and widely invoked explanatory mechanism for abstract pleasures like music listening, but there is currently no direct evidence that it occurs during this or any other abstract hedonic experience. Here, we tested whether music could elicit formally modeled RPEs in the NAc using fMRI and a reinforcement-learning protocol validated for studying RPEs (32, 33). We adapted this protocol to a musical context: In each of 60 trials, participants chose between two sets of arbitrary cues (first colors and then directions) that were associated with an ongoing musical piece ending either consonantly [which they preferred: two-tailed paired samples t(19) = 5.72, P < 0.01; SI Appendix, Fig. S1] or dissonantly (Fig. 1; sample stimuli are shown in SI Appendix). Although music is typically consonant, the probabilistic nature of this paradigm meant that our participants could never fully predict how a trial would end, ensuring measurable prediction errors for each trial based on the interaction of expectancy and pleasantness. Prediction errors were therefore positive for consonant outcomes, negative for dissonant ones, and greater the more the outcome was unexpected based on the listener’s task experience. The basis of our RPE modeling was thus the salient feature of consonance/dissonance as an exemplar of general musical prediction errors.

Fig. 1.
Fig. 1.

Probabilistic musical decision-making task. Participants first chose between two arbitrary colors to initiate the playing of a Bach chorale: The color choice determined its timbre. As the chorale reached halfway, a cue prompted participants to choose between two arbitrary directions: This choice determined whether the chorale ended consonantly or dissonantly. Each choice was associated with a specific outcome, but these outcomes were probabilistic to elicit prediction errors. Each color led to its associated timbre 70% of the time. Within one timbre context, each direction choice led to its ending 85% of the time; in the other timbre, it was 70%. The associations were randomized across participants, who were not made explicitly aware of the contingencies but were simply told to try to make optimal choices to hear the music they wanted. The optimal choices for the case shown in the figure were thus yellow to select its associated the harp timbre and then left if the chorale played in that timbre or right if not. The task had 60 trials in two runs of two equal blocks each.

We used a temporal difference [TD(λ)] “Q-learning” algorithm to model these RPEs according to each participant’s choices, outcomes, and outcome preferences (22, 32⇓–34). Our first hypothesis was that the RPE model would substantially explain participants’ choices, suggesting that they used these signals to learn during the experiment. We then tested whether the RPEs manifested in the blood oxygen level-dependent (BOLD) activity of the NAc, as predicted from the neuroimaging literature, and whether this activity would represent RPEs rather than some other correlated signal (30). Finally, since computing RPEs should facilitate the learning of rewarding behavior (22, 23), we investigated the relationship of the RPE-related neural signals to individual learning outcomes. These analyses revealed strong behavioral and neural evidence of musical RPEs.

Results

Reinforcement-Learning Behavior.

Participants successfully navigated the musical reinforcement-learning task. They were more likely than chance to make choices leading to consonant endings, as measured by one-sample t tests of the color and direction choices separately [overall accuracy mean ± SD = 57.93 ± 14.44%, t(18) = 2.40, P = 0.03; Fig. 2A] or together [two-step accuracy mean ± SD = 37.01 ± 19.63%, t(18) = 2.67, P = 0.02; Fig. 2B]. We evaluated learning throughout the experiment as the change in accuracy across sliding 10-trial windows: Permutation tests of these learning slopes showed that overall accuracy improved for the group as a whole (β^ = 0.02, P = 0.04; Fig. 2C) and for 12 of the 18 participants evaluated (all β^s > 0.02, P < 0.05), while two-step accuracy did not significantly improve for the whole group (β^ = 0.01, P = 0.23; Fig. 2D) but did for eight of the 18 participants evaluated (all β^s > 0.02, P < 0.05). The variability in performance reflects the difficulty of the task, and allows us to measure the relationship between learning and brain activity.

Fig. 2.
Fig. 2.

Task behavior and model fit. (A) Overall accuracy. Considering color and direction choices independently, participants made optimal choices significantly more often than chance [t(18) = 2.40, P = 0.03]. (B) Two-step accuracy. Participants also made both optimal decisions within a trial significantly more often than chance, suggesting learning of the task’s two-step pathway [t(18) = 2.67, P = 0.02]. (C) Overall accuracy improved throughout the task for the group as a whole (β^ = 0.02, P = 0.04) and for 12 of the 18 participants tested (all β^s > 0.02, P < 0.05). (D) Two-step accuracy did not improve throughout the task for the group as a whole (β^ = 0.01, P = 0.23), but it did for eight of the 18 participants tested (all β^s > 0.02, P < 0.05). (E) Reinforcement-learning models fit the group’s choices significantly better than corresponding null models [t(19) = 3.93, P < 0.01]. Red lines indicate means, boxes indicate 1 SD from the mean, and blue lines show the chance level references of the statistical tests.

To determine how well the reinforcement-learning models described the group’s behavior, we compared the negative log likelihood of each model, given the participant’s choices, with that of a null model that had no information about the choices. The reinforcement-learning models described the group’s choices significantly better than the null models [mean ± SD pseudo-R2 = 12.85 ± 14.64%, t(19) = 3.93, P < 0.01; Fig. 2E], enabling us to explore the neural correlates of these modeled RPEs. The best-fitting parameters for the models are shown in SI Appendix, Fig. S2.

Neural Evidence of RPEs.

To assess the specific prediction that the NAc reflected RPE signaling in this musical reinforcement-learning context, we convolved the computationally modeled RPEs with a standard hemodynamic response function and evaluated their correspondence to the BOLD activity within the bilateral NAc region of interest. We defined this area a priori with the Harvard-Oxford anatomical subcortical atlas (35). The modeled RPEs significantly correlated with the BOLD activity of the right NAc after family-wise error (FWE) correction (peak voxel: 12, 8, −8; z = 3.53; PFWE = 0.01; Fig. 3A), and this was the strongest correlation throughout the whole brain, illustrating that this area exhibits RPEs even from an abstract stimulus like music, when the values of predicted and obtained outcomes are less clear than with a concrete stimulus like money. To validate that this activity reflected canonical RPEs rather than reward value or surprise, we performed a conjunction analysis of distinct RPE components in the same a priori-defined region. RPEs are characterized as greater for rewards (consonant endings) than punishments (dissonant endings), larger for greater or less expected rewards, and smaller for greater or less expected punishments (30). We found a conjunction of these features in the same area of the right NAc as indicated by the modeled RPEs (peak voxel = 12, 8, −10; z = 2.38; P < 0.01; Fig. 3B). A post hoc examination of this conjunction throughout the rest of the brain verified that the strongest conjunction was in the NAc. This result corroborates our original NAc finding by exhibiting axiomatic RPE properties in the same previously identified structure.

Fig. 3.
Fig. 3.

Musically elicited RPEs reflected in NAc activity. (A) Within the bilateral NAc region of interest (purple), computationally modeled RPEs significantly correlated with activity in a right-hemisphere cluster after family-wise error correction (peak voxel: 12, 8, −8; z = 3.53; PFWE = 0.01; orange scale). (B) Conjunction of three analyses supports the original finding. Blue indicates consonant endings > dissonant endings (outcome contrast), red indicates RPEs for consonant endings (parametric effect of rewards), green indicates RPEs for dissonant endings (parametric effect of punishments), and yellow indicates conjunction. All images are thresholded at z ≥ 2. In coronal slices, y = 8. In sagittal slices, x = 12. L, left; R, right.

Learning Correlates.

Finally, we explored whether the RPE-related activity of the right NAc facilitated reinforcement learning by testing whether it could significantly explain task performance or model fitness. All behavioral learning measures (overall accuracy, overall learning slope, two-step accuracy, and two-step learning slope) were highly correlated with each other and with the measure of model fitness (pseudo-R2), so we used stepwise linear regression to identify the one(s) with the strongest explanatory power. This approach settled on the overall learning slope, which accounted for ∼31% of the variance in the RPE parameter estimates [F(1,16) = 7.18, P = 0.02, R2 = 0.31, adjusted (Adj.) R2 = 0.27]. We validated this effect with a robust regression to reduce the influence of outliers [F(1,16) = 6.03, P = 0.03, R2 = 0.27, Adj. R2 = 0.23; Fig. 4A]. This relationship was positive, such that stronger RPE signaling in the right NAc corresponded to more overall learning in the reinforcement-learning task. Noting that the participant with the weakest RPE signaling in the right NAc and the worst overall learning also scored the lowest on the Barcelona Music Reward Questionnaire (BMRQ) (14), we further identified a significant positive relationship between BMRQ scores and RPE signaling in the right NAc [F(1,14) = 5.04, P = 0.04, R2 = 0.27, Adj. R2 = 0.21; Fig. 4B]. The participants who derived less reward from musical stimuli generally reflected musical RPEs less reliably (a model combining BMRQ scores, RPE signals, and learning is shown in SI Appendix, Fig. S3). Together, these analyses implicate the RPE-related neural activity we observed in the right NAc in the behavioral effect of learning.

Fig. 4.
Fig. 4.

Correlates of RPE-related activity in the right NAc. (A) Parameter estimates (param. estim.) of RPE signaling in the right NAc (shown in pink at x = 12) significantly correlate with the standardized learning slopes of overall accuracy [F(1,16) = 6.03, P = 0.03, R2 = 0.27, Adj. R2 = 0.23]. (B) BMRQ scores also correlate with RPE-like activity in the right NAc. *P < 0.05.

Discussion

Although many authors have proposed that the intense emotions and pleasures of music result from expectancies, predictions, and their outcomes (conceptualized here as RPEs) (10⇓⇓–13), direct evidence for this proposition has been lacking. Here, using a music-based probabilistic task adapted from reinforcement-learning studies, computational modeling, and formal validation of RPEs, we report direct evidence of musically elicited RPEs. With only musical feedback, participants learned to find their preferred musical endings more often as the task progressed and generated RPEs as they did. As is typical in other domains (22, 23), RPEs supported learning: While most participants learned during the task, those whose NAc activity reflected RPEs more reliably (perhaps better encoding these signals) tended to improve the most. In addition to supporting the widely hypothesized ability of music to elicit RPEs, these findings establish music as a neurobiological reward capable of motivating learning in a complex environment, illustrating how an abstract stimulus can engage the brain’s reward system to potentially pleasurable effect and implying that RPEs could play a broader role in pleasure than previously known.

Music as Reward.

The adaptive value of music is not readily apparent. One possibility is that music might be rewarding by facilitating emotions (36), which can be pleasurable even when the emotions are negative (5). However, musical emotions, like pleasure, depend at least partly on the manipulation of expectations (10⇓–12), just as RPEs sometimes coincide with strong emotional arousal and pleasure (28). The value of music might therefore be directly related to the predictions it engenders and the quick feedback it provides across multiple evolving structures, facilitating our “fundamental imperative” to predict and learn from our environment (ref. 37, p. 43). The human brain exhibits predictive processing of musical sequences even from birth (38). This processing, which likely occurs within frontotemporal cortical circuits (13), could represent refinements of the listener’s model, which may rely on dopaminergic prediction errors (21, reviewed in ref. 39). Unlike the danger of incorrectly predicting some other auditory signals, such as those of predators or weather, being wrong about an abstract stimulus like music has minimal biological consequences, and might even be pleasurable after cognitive appraisal (11). This interpretation could apply to several other abstract pleasures, which might engage the NAc via RPEs related to success (i.e., a better-than-expected outcome or world model) or knowledge acquisition/uncertainty reduction (i.e., better-than-expected information for world-model refinement) (7⇓–9). By this explanation, the most pleasurable music would be that which helps to hone our predictions, with extremely predictable or familiar music being too simplistic to contribute and extremely unpredictable or novel music being too complex to integrate (12).

In the present study, the difference between consonance and dissonance was the principal motivator of behavior. Listeners of Western tonal music typically exhibit a strong and reliable preference for consonant music (reviewed in ref. 40), as we observed in our sample (SI Appendix, Fig. S1). Likewise, our participants’ RPEs represent the deviations from their expected choice outcomes, positive if consonant and negative if dissonant, since the participants could never fully predict either ending. Although the sources of prediction errors likely differ in typical music listening experiences, during which dissonance may be more surprising than consonance and predictions also pertain to other musical features not tested here (cf. ref. 18), the present results illustrate that musical RPEs are sufficient to guide decisions and learning (Fig. 2 A and B). Learning occurred despite the absence of any concrete outcome or reward, and without any explicit instruction or understanding of the probabilistic task structure.

Learning was evident in the participants’ choices, as they made significantly more choices associated with consonant endings throughout the experiment (Fig. 2C). Likewise, about half of the sample became more likely to choose both the color and the direction most associated with consonant endings within a trial (Fig. 2D). That these effects manifested within 60 trials, whereas most reinforcment-learning studies use many more trials and more concrete rewards, such as money (e.g., refs. 28, 32, 33), highlights music’s potency as a reward.

Although the participants exhibited significant learning as a group, our behavioral measures also illustrate considerable variability, with some individuals performing no better than chance (Fig. 2 A–D). We designed the task to be difficult to encourage exploration, and the only instructions were to “learn about which choices are most likely to lead to the music you want to hear, and then make these choices.” Some participants reported after the experiment that they had made decisions based on musical characteristics other than consonance/dissonance, such as major/minor or ascending/descending; however, these features would not have aided their decisions, because the stimuli were identical except for their consonant/dissonant endings. These strategies, and individual differences in decision-making strategies and biases, musical backgrounds, memory capacities, and reward sensitivities, could explain some of the learning differences.

Modeling and Validating Musical RPEs.

We used a well-validated computational model of reinforcement learning (22, 34) to contextualize our findings within the reinforcement-learning literature. This model described 18 of the 20 participants’ choices better than the corresponding null models, with an average improvement of 14.28% for those 18 participants (Fig. 2E). Other studies using similar models with more concrete feedback typically report values between about 25% and 40% (32, 33), but our modeling nonetheless allowed us to explore the neural correlates of the RPE parameter. The best-fitting free parameters were akin to those in other studies (32, 33) (SI Appendix, Fig. S2).

As we hypothesized, the BOLD activity of the NAc significantly reflected formally modeled RPEs (Fig. 3A). Such correlations might arise from related processes, such as valence tracking (30), so we supported this finding by conjoining distinct RPE axioms (cf. ref. 30). The observed NAc activity was indeed greater for rewards (consonance) than punishments (dissonance), for more preferred or less expected rewards (i.e., larger positive RPEs), and for more preferred or more expected punishments (i.e., smaller negative RPEs), expressing the bidirectional computation that rigorously defines RPEs (30).

Incorporating information from cortical, limbic, and midbrain sources, dopamine neurons in the NAc have been identified as the main purveyor of RPEs and goal-oriented action selection in other domains (30, 31, reviewed in ref. 29). The NAc is also the brain region most strongly associated with the enjoyment of music (3, 4, 6), as well as with several other pleasures (reviewed in ref. 2). Although pleasures, including music, often engage the NAc bilaterally (2⇓–4, 6), our right-hemisphere finding is consistent with others showing a similar asymmetry (3, 4), and with a meta-analysis identifying the right, but not left, NAc as a reliable indicator of musically evoked emotions (41). This lateralization could be related to the cortical circuitry in the right temporal and frontal cortices that has been implicated in tonal perceptual processing (42), including responses to unexpected events (17, 19), and to the NAc’s predominantly ipsilateral connections (e.g., ref. 43). Correspondingly, disruption of right auditory cortex–NAc functional interactions is associated with a loss of musical pleasure (6).

A large body of research cautions that NAc dopamine transmission, including RPEs, is more strongly associated with motivational arousal and approach than with hedonic pleasure per se (reviewed in refs. 2, 29). The RPE-related activity that we observe here might thus reflect motivation to hear consonant over dissonant endings rather than pleasure. However, musically elicited RPEs could be both motivational and pleasurable, as predictive success often is (44). Musical motivation and pleasure, although distinguishable, are highly correlated and colocated in frontostriatal circuits (4, 45), and they likely coincide, especially if, as discussed above, positive RPEs arise from musical outcomes that reduce uncertainty (i.e., are more predictable than expected) and negative RPEs arise from those that increase it. Future experiments should consider the relationship between RPEs and pleasure in more naturalistic music listening (i.e., beyond a reinforcement-learning paradigm) to explore the musical structures that give rise to musical RPEs without the influence of action–outcome associations.

Linking Musical RPEs to Behavioral Learning.

In computing the deviation between reward predictions and outcomes, RPEs are essentially teaching signals that update future predictions and behaviors (22, 23). Here, we find that the strength of the RPE signaling in the right NAc explained ∼31% of the variance in how much the participants learned (Fig. 4A). Participants who were more sensitive to music reward in general (14) better reflected RPE-like activity, but it was ultimately the extent of RPE signaling in the NAc that explained learning during the experiment. Those who represented RPEs more faithfully presumably benefited from the updates these computations generated (e.g., through changes in synaptic and/or effective connectivity) (23). This effect validates our discovery of musically elicited RPEs in the NAc via their functional significance.

Conclusion

These results support the widely hypothesized role of RPEs in musical pleasure, illustrating that an abstract stimulus can engage the same mechanism we use to learn about concrete, evolutionarily advantageous rewards like food, money, or sex. By exploiting our “fundamental imperative” to predict and learn (ref. 37, p. 43) and our diffuse mesocorticolimbic connections (reviewed in ref. 29), musical RPEs seem to integrate cognitive and emotional processing in the NAc; these mechanisms might help explain why so many people find music engaging and pleasurable (1). Although music is especially well suited to manipulating expectations (10⇓⇓–13), our findings suggest that RPEs could also be important for other abstract rewards and/or pleasures, such as learning (cf. ref. 7), poetry (cf. ref. 8), or humor (cf. ref. 9). Understanding these abstract pleasures would have important implications for therapeutic interventions and for happiness itself, just as understanding subjective processes of prediction can elucidate much of how we perceive, think, and behave.

Materials and Methods

Participants.

Twenty-three healthy volunteers with normal hearing participated in this experiment after providing informed consent. Two did not explore the task choices, rendering the computational model unable to accurately devise their decision values and RPEs; they were thus excluded from all analyses. Technical issues caused the loss of all of the data for one participant and the second half of the data for another, so we excluded the former from all analyses and the latter from analyses of learning over time. The final sample (n = 20, 12 females) was aged between 18 and 27 y old (mean age ± SD = 21.60 ± 2.89 y), with an average of 5.24 ± 5.20 y of formal or informal musical training and an average BMRQ score (14) of 78.83 ± 9.37 out of 100. This study was approved by the Research Ethics Board of the Montreal Neurological Institute.

Stimuli.

The stimuli were based on 12 four-part Bach chorales, chosen because they follow widespread rules of Western tonal music and have consistent structures. Each chorale contained four musical phrases of eight beats each and was 25.60 s long at 75 beats per minute. Six were in major keys, six were in minor keys, and all were in duple meter. We generated a consonant (original) ending and a dissonant ending for each chorale in two timbres using MuseScore software (2016 MuseScore BVBA). The dissonant endings had each note alternatingly transposed half a step higher or lower, with the soprano and tenor parts moving up first and the alto and bass parts moving down first; the consonant and dissonant versions of the stimuli were otherwise identical due to the use of Musical Instrument Digital Interface (MIDI) instruments. Thus, the “consonant” endings were altogether much less dissonant than the “dissonant” ones, even though they could also contain some elements of dissonance. We chose the timbres according to an Internet survey with a separate group of 22 volunteers, settling on those rated most consistently and similarly pleasant on a seven-point Likert scale: mandolin (mean pleasantness rating ± SD = 5.63 ± 2.13, Cronbach’s α = 0.89) and harp (mean pleasantness rating ± SD = 5.56 ± 1.92, Cronbach’s α = 0.89). Participants listened to and rated their enjoyment of each chorale in a MIDI piano timbre before the experimental task. Although one person preferred the chorales that ended dissonantly, the overall group significantly preferred the consonant stimuli [two-tailed paired samples t(19) = 5.72, P < 0.01; SI Appendix, Fig. S1]. Participants also rated two short chord sequences in both timbres, ensuring no a priori preference for either timbre [two-tailed paired-samples t(19) = −1.76, P = 0.09; SI Appendix, Fig. S1].

Experimental Task.

The decision-making task was based on “two-step tasks” used in other investigations of human RPEs (32, 33). The chief difference was that, instead of money or points, the feedback in this task was only musical (Fig. 1). Participants began each trial with a button press, after which two colors appeared on the screen (blue on the left and yellow on the right). These colors probabilistically determined the timbre of the trial, so choosing a color with the left or right button of the response box initiated the (unaltered) beginning of a randomly selected chorale in that timbre. The color choice was displayed on the screen for the first half of the chorale, after which a cue prompted a similar choice between two directions (left and right) that probabilistically determined whether the chorale ended consonantly or dissonantly. If the participants failed to respond before the final phrase of the chorale (i.e., within 2.25 s), the trial aborted and the next one began. If not, the direction choice was displayed on the screen during the ending. The color choice probability was always 70%, but the direction choice probability was 85% in one timbre context and 70% in the other. To maximize the likelihood of hearing consonant endings, a participant’s best course of action was therefore to choose the color that would most likely lead to the timbre in which the direction choice was more influential and then the direction most associated with the consonant ending. The only instructions were to “learn about which choices are most likely to lead to the music you want to hear, and then make these choices”; the participants were blinded to the task probabilities. The task had 60 trials, evenly divided into four blocks with a break between each, lasting about 40 min. We adapted the task timings and probabilities through pilot testing with a separate group of volunteers, selecting parameters that allowed participants to perform above chance and improve during the task without learning so well that they stopped sampling task choices. The choice–outcome associations and the more influential timbre were randomly assigned to each participant at the beginning of the experiment to control for any associative biases.

Computational Reinforcement-Learning Model.

We modeled RPEs with a TD(λ) Q-learning algorithm (22, 34). In each state s (i.e., when faced with the color choice or the direction choice in either timbre context), this algorithm computes the value of each action a as Q(s(t),a(t))←Q[s(t),a(t)]+λαδ(t), where t is the time step (1 in the first state, 2 in the second state, and 3 for feedback), Q(s(t),a(t)) is the expected value of all future outcomes for choice a in state s, λ is the memory trace attributing value to the first step in the two-step sequence [set to 1 when s(t)=1], α is the model’s learning rate, and δ(t) is the RPE for that time step. The RPE updates by δ(t)=r(t)+maxaϵAQ[s(t+1),a(t+1)]−Q[s(t),a(t)], where r(t) is the feedback at step t (the participant’s pretask rating of the same chorale in the piano timbre, rescaled from −3 to 3, or 0 in the case of no feedback) and A is the set of all possible actions in the state. Starting with all Q values at 0, this model thus approaches the true expected future value of each choice by computing the RPE as the difference between each outcome and the (discounted) best expected outcome, at a rate depending on the learning rate α. We fit these parameters to participants’ choices with a “softmax” function to model the probability of choosing each action (e.g., X or Y) in each state: P[a(t)=X|s(t)=s]=eβQ[s,X]eβQ[s,X]+eβQ[s,Y], where β is a temperature parameter that indicates a learner’s propensity to explore the environment. We fit the model’s free parameters (α,λ, and β) by exhaustive search, choosing the set that minimized the negative log likelihood of the model making the same choices as the participants (31⇓–33). This process yielded RPE values for each outcome of each participant, which became regressors of interest in the fMRI design matrices.

fMRI Data Acquisition and Preprocessing.

We acquired MRI data with a Siemens TIM Trio 3-T scanner and a 32-channel head coil at the McConnell Brain Imaging Centre of the Montreal Neurological Institute. We used a T2*-weighted multiband echo planar imaging sequence to collect whole-brain functional BOLD images at high temporal resolution [52 slices, echo time (TE) = 30 ms, repetition time (TR) = 885 ms, multiband acceleration factor = 4, flip angle = 90°, matrix size = 195 × 195 × 130, voxel size = 2.5 mm isotropic]. When whole-brain coverage was not possible with these parameters, we prioritized ventral temporal and frontal regions at the cost of dorsal parietal ones. The experimental task occurred over two functional runs of 30 trials, which lasted about 18 min each, with a short break in between for rest. Participants experienced and responded to the task via Presentation software (Neurobehavioral Systems, Inc.) using an angled mirror on top of the head coil, MR-compatible S14 Insert Earphones (Sensimetrics Corporation), and an MR-compatible two-button box in the right hand. We collected a high-resolution T1-weighted image for each participant [magnetization prepared rapid gradient echo (MPRAGE): TE = 2.98 ms, TR = 2,300 ms, matrix size = 256 × 256 × 192, voxel size = 1 mm isotropic] for anatomical registration. We preprocessed the functional images with FSL FEAT Version 6 (FMRIB), including motion correction, brain extraction, spatial smoothing with a Gaussian kernel of 5-mm FWHM, grand-mean intensity normalization, high-pass temporal filtering with Gaussian-weighted least-squares straight-line fitting sigma = 50.0 s, and linear registration to the T1-weighted images and then to the MNI152 2-mm standard brain.

Behavioral Analysis.

Behavioral analyses were performed with custom MATLAB (MathWorks) scripts. Given preferences for consonance, we measured “overall” accuracy as the proportion of choices most likely to lead to the more favorable timbre context or the consonant ending and “two-step” accuracy as the proportion of trials on which a participant made both the color and direction choice most likely to lead to a consonant ending. We compared each with chance performance (50% for overall accuracy and 25% for two-step accuracy) with two-tailed, one-sample t tests, excluding the participant who reported a preference for dissonance. To measure learning throughout the task, we divided these accuracy measures into sliding 10-trial windows (i.e., window 1 was trials 1–10, window 2 was trials 2–11, etc.) and obtained learning slopes with linear regressions of these values against the window number. We evaluated the group learning slopes with one-tailed permutation tests against regression slopes from 20,000 series of accuracy bins randomly sampled without replacement, excluding the one participant who reported a preference for dissonance and the participant with incomplete data. We characterized the fit of the reinforcement-learning model for each participant as the pseudo-R2 value (i.e., the percent improvement of the model’s negative log likelihood compared with that of a null model treating each decision as random) (32, 33), and compared the model fitness for the group to 0% with a two-tailed, one-sample t test of the pseudo-R2 values.

fMRI Analysis.

We conducted two fMRI analyses with three general linear models (GLMs). The first GLM, to identify whether computationally modeled RPEs correlated with BOLD activity in the bilateral NAc, included a parametric regressor of the modeled and standardized RPEs and three nuisance regressors for the times between the color cues and choices (“state 1”), the direction cues and choices (“state 2”), and the direction choices and their outcomes (“anticipation”). We designed the second and third GLMs to analyze the axioms of RPE signals and corroborate the result of the first GLM. These included the same nuisance regressors as the first GLM, but the second model had separate regressors for the positive and negative RPEs and the third had binary regressors for the consonant and dissonant outcomes instead. We convolved these regressors with a canonical hemodynamic response function and their resulting temporal derivatives to create the regressors of interest. Each first-level model also included 24 motion regressors to account for movement-related variance: one for each directional axis, one for the derivative of each axis, and the squares of these 12 values. Finally, we scrubbed volumes with framewise displacement above 0.9 with additional nuisance regressors (46). We subjected each model to a second-level fixed-effects analysis to average the contrast estimates of the two runs for each participant (the contrasts were unchanged for the participant with one run of data) and to a third-level mixed-effects analysis to find group effects. Reported coordinates are in Montreal Neurological Institute space.

We tested the hypothesis of NAc activity reflecting RPEs with a bilateral anatomical mask from the Harvard-Oxford atlas in FSL (35), thresholding at P < 0.05 after one-tailed FWE correction. To test whether this region reflected RPEs or some correlated signal like reward magnitude or valence, we analyzed a conjunction of axiomatic RPE properties in the same mask: that they should be greater for rewards than punishments and increase for both larger/less expected rewards and smaller/more expected punishments (30). To do so, we conjoined parametric regressors of positive and negative RPEs from the second GLM with a contrast of consonant > dissonant endings from the third GLM. We analyzed the conjunction as the minimum z statistic of the three contrasts and thresholded the conjunction map using clusters determined by voxel z > 2.3 and cluster P < 0.01.

We investigated the behavioral relevance of the NAc RPE-like activity with stepwise linear regression of the averaged beta estimates in the right NAc (as defined independently by the Harvard-Oxford atlas) against the pseudo-R2 values of the reinforcement-learning model fit and the measures of decision-making task performance (overall accuracy, two-step accuracy, overall learning slope, and two-step learning slope). This analysis, using MATLAB’s stepwiselm function with default settings, adds or removes variables based on F tests of the change in the sum of squared error. We validated this approach with a robust-fit regression on the same variables, reducing the influence of outliers with weightings, using the “RobustOpts” option with default settings in MATLAB’s fitlm function. We followed this analysis with a simple linear regression of right NAc RPE-like activity and BMRQ scores, and by adding BMRQ scores as a regressor in the significant model of overall learning slopes and right NAc RPE-like activity (for the 15 participants who completed the BMRQ). Data are available at https://neurovault.org/collections/4778/ (47).

Acknowledgments

We thank Vincent K. M. Cheung for helping with stimulus design and pilot testing and Karl A. Neumann for assistance with recruiting participants, pilot testing, and data collection. This work was supported by a Fulbright Canada Science, Technology, Engineering, and Mathematics (STEM) Graduate Award (to B.P.G.), a Natural Sciences and Engineering Research Council of Canada Collaborative Research and Training Experience Graduate Award (to B.P.G.), and by a Foundation Grant (to R.J.Z.) from the Canadian Institutes of Health Research. R.J.Z. is also a senior fellow of the Canadian Institute for Advanced Research.

Footnotes

  • ↵1To whom correspondence should be addressed. Email: benjamin.gold{at}mail.mcgill.ca.

Published under the PNAS license.

References

  1. ↵
    1. Dubé L,
    2. Le Bel J
    (2003) The content and structure of laypeople’s concept of pleasure. Cogn Emotion 17:263–295.
    OpenUrl
  2. ↵
    1. Berridge KC,
    2. Kringelbach ML
    (2015) Pleasure systems in the brain. Neuron 86:646–664.
    OpenUrlCrossRefPubMed
  3. ↵
    1. Salimpoor VN,
    2. Benovoy M,
    3. Larcher K,
    4. Dagher A,
    5. Zatorre RJ
    (2011) Anatomically distinct dopamine release during anticipation and experience of peak emotion to music. Nat Neurosci 14:257–262.
    OpenUrlCrossRefPubMed
  4. ↵
    1. Salimpoor VN, et al.
    (2013) Interactions between the nucleus accumbens and auditory cortices predict music reward value. Science 340:216–219.
    OpenUrlAbstract/FREE Full Text
  5. ↵
    1. Brattico E, et al.
    (2016) It’s sad but I like it: The neural dissociation between musical emotions and liking in experts and laypersons. Front Hum Neurosci 9:676.
    OpenUrl
  6. ↵
    1. Martínez-Molina N,
    2. Mas-Herrero E,
    3. Rodríguez-Fornells A,
    4. Zatorre RJ,
    5. Marco-Pallarés J
    (2016) Neural correlates of specific musical anhedonia. Proc Natl Acad Sci USA 113:E7337–E7345.
    OpenUrlAbstract/FREE Full Text
  7. ↵
    1. Jepma M,
    2. Verdonschot RG,
    3. van Steenbergen H,
    4. Rombouts SA,
    5. Nieuwenhuis S
    (2012) Neural mechanisms underlying the induction and relief of perceptual curiosity. Front Behav Neurosci 6:5.
    OpenUrlCrossRefPubMed
  8. ↵
    1. Wassiliwizky E,
    2. Koelsch S,
    3. Wagner V,
    4. Jacobsen T,
    5. Menninghaus W
    (2017) The emotional power of poetry: Neural circuitry, psychophysiology and compositional principles. Soc Cogn Affect Neurosci 12:1229–1240.
    OpenUrl
  9. ↵
    1. Franklin RG Jr,
    2. Adams RB Jr
    (2011) The reward of a good joke: Neural correlates of viewing dynamic displays of stand-up comedy. Cogn Affect Behav Neurosci 11:508–515.
    OpenUrlCrossRefPubMed
  10. ↵
    1. Meyer LB
    (1956) Emotion and Meaning in Music (Chicago Univ Press, Chicago).
  11. ↵
    1. Huron D
    (2006) Sweet Anticipation: Music and the Psychology of Expectation (MIT Press, Cambridge, MA).
  12. ↵
    1. Gebauer L,
    2. Kringelbach ML,
    3. Vuust P
    (2012) Ever-changing cycles of musical pleasure: The role of dopamine and anticipation. Psychomusicology 22:152–167.
    OpenUrl
  13. ↵
    1. Salimpoor VN,
    2. Zald DH,
    3. Zatorre RJ,
    4. Dagher A,
    5. McIntosh AR
    (2015) Predictions and the brain: How musical sounds become rewarding. Trends Cogn Sci 19:86–91.
    OpenUrlCrossRefPubMed
  14. ↵
    1. Mas-Herrero E,
    2. Marco-Pallarés J,
    3. Lorenzo-Seva U,
    4. Zatorre RJ,
    5. Rodriguez-Fornells A
    (2013) Individual differences in music reward experiences. Music Percept 31:118–138.
    OpenUrlAbstract/FREE Full Text
  15. ↵
    1. Juslin PN,
    2. Västfjäll D
    (2008) Emotional responses to music: The need to consider underlying mechanisms. Behav Brain Sci 31:559–575, discussion 575–621.
    OpenUrlCrossRefPubMed
  16. ↵
    1. Brattico E,
    2. Pearce M
    (2013) The neuroaesthetics of music. Psychol Aesthet Creat Arts 7:48–61.
    OpenUrlCrossRef
  17. ↵
    1. Steinbeis N,
    2. Koelsch S,
    3. Sloboda JA
    (2006) The role of harmonic expectancy violations in musical emotions: Evidence from subjective, physiological, and neural responses. J Cogn Neurosci 18:1380–1393.
    OpenUrlCrossRefPubMed
  18. ↵
    1. Egermann H,
    2. Pearce MT,
    3. Wiggins GA,
    4. McAdams S
    (2013) Probabilistic models of expectation violation predict psychophysiological emotional responses to live concert music. Cogn Affect Behav Neurosci 13:533–553.
    OpenUrlCrossRefPubMed
  19. ↵
    1. Koelsch S,
    2. Kilches S,
    3. Steinbeis N,
    4. Schelinski S
    (2008) Effects of unexpected chords and of performer’s expression on brain responses and electrodermal activity. PLoS One 3:e2631.
    OpenUrlCrossRefPubMed
  20. ↵
    1. Friston K
    (2010) The free-energy principle: A unified brain theory? Nat Rev Neurosci 11:127–138.
    OpenUrlCrossRefPubMed
  21. ↵
    1. Sharpe MJ, et al.
    (2017) Dopamine transients are sufficient and necessary for acquisition of model-based associations. Nat Neurosci 20:735–742.
    OpenUrlCrossRefPubMed
  22. ↵
    1. Sutton RS,
    2. Barto AG
    (1998) Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA).
  23. ↵
    1. den Ouden HE,
    2. Daunizeau J,
    3. Roiser J,
    4. Friston KJ,
    5. Stephan KE
    (2010) Striatal prediction error modulates cortical coupling. J Neurosci 30:3210–3219.
    OpenUrlAbstract/FREE Full Text
  24. ↵
    1. Ylinen S, et al.
    (2016) Predictive coding of phonological rules in auditory cortex: A mismatch negativity study. Brain Lang 162:72–80.
    OpenUrlCrossRefPubMed
  25. ↵
    1. Schultz W,
    2. Dayan P,
    3. Montague PR
    (1997) A neural substrate of prediction and reward. Science 275:1593–1599.
    OpenUrlAbstract/FREE Full Text
  26. ↵
    1. Hart AS,
    2. Rutledge RB,
    3. Glimcher PW,
    4. Phillips PE
    (2014) Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. J Neurosci 34:698–704.
    OpenUrlAbstract/FREE Full Text
  27. ↵
    1. Zhang Y,
    2. Larcher KM,
    3. Misic B,
    4. Dagher A
    (2017) Anatomical and functional organization of the human substantia nigra and its connections. eLife 6:e26653.
    OpenUrlCrossRef
  28. ↵
    1. Seymour B, et al.
    (2005) Opponent appetitive-aversive neural processes underlie predictive learning of pain relief. Nat Neurosci 8:1234–1240.
    OpenUrlCrossRefPubMed
  29. ↵
    1. Floresco SB
    (2015) The nucleus accumbens: An interface between cognition, emotion, and action. Annu Rev Psychol 66:25–52.
    OpenUrlCrossRefPubMed
  30. ↵
    1. Rutledge RB,
    2. Dean M,
    3. Caplin A,
    4. Glimcher PW
    (2010) Testing the reward prediction error hypothesis with an axiomatic model. J Neurosci 30:13525–13536.
    OpenUrlAbstract/FREE Full Text
  31. ↵
    1. Chase HW,
    2. Kumar P,
    3. Eickhoff SB,
    4. Dombrovski AY
    (2015) Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis. Cogn Affect Behav Neurosci 15:435–459.
    OpenUrlCrossRefPubMed
  32. ↵
    1. Gläscher J,
    2. Daw N,
    3. Dayan P,
    4. O’Doherty JP
    (2010) States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66:585–595.
    OpenUrlCrossRefPubMed
  33. ↵
    1. Daw ND,
    2. Gershman SJ,
    3. Seymour B,
    4. Dayan P,
    5. Dolan RJ
    (2011) Model-based influences on humans’ choices and striatal prediction errors. Neuron 69:1204–1215.
    OpenUrlCrossRefPubMed
  34. ↵
    1. Watkins CJCH
    (1989) Learning from delayed rewards. PhD dissertation (King’s College, Cambridge, UK).
  35. ↵
    1. Frazier JA, et al.
    (2005) Structural brain magnetic resonance imaging of limbic and thalamic volumes in pediatric bipolar disorder. Am J Psychiatry 162:1256–1265.
    OpenUrlCrossRefPubMed
  36. ↵
    1. Salimpoor VN,
    2. Benovoy M,
    3. Longo G,
    4. Cooperstock JR,
    5. Zatorre RJ
    (2009) The rewarding aspects of music listening are related to degree of emotional arousal. PLoS One 4:e7487.
    OpenUrlCrossRefPubMed
  37. ↵
    1. Friston KJ,
    2. Friston DA
    (2013) A free energy formulation of music generation and perception: Helmholtz revisited. Sound–Perception–Performance, Current Research in Systematic Musicology, ed Bader R (Springer, Heidelberg), pp 43–69.
  38. ↵
    1. Virtala P,
    2. Huotilainen M,
    3. Partanen E,
    4. Fellman V,
    5. Tervaniemi M
    (2013) Newborn infants’ auditory system is sensitive to Western music chord categories. Front Psychol 4:492.
    OpenUrl
  39. ↵
    1. Keiflin R,
    2. Janak PH
    (2017) Error-driven learning: Dopamine signals more than value-based errors. Curr Biol 27:R1321–R1324.
    OpenUrl
  40. ↵
    1. Virtala P,
    2. Tervaniemi M
    (2017) Neurocognition of major-minor and consonance-dissonance. Music Percept 34:387–404.
    OpenUrlAbstract/FREE Full Text
  41. ↵
    1. Koelsch S
    (2014) Brain correlates of music-evoked emotions. Nat Rev Neurosci 15:170–180.
    OpenUrlCrossRefPubMed
  42. ↵
    1. Zatorre RJ,
    2. Belin P,
    3. Penhune VB
    (2002) Structure and function of auditory cortex: Music and speech. Trends Cogn Sci 6:37–46.
    OpenUrlCrossRefPubMed
  43. ↵
    1. Groenewegen HJ,
    2. Room P,
    3. Witter MP,
    4. Lohman AHM
    (1982) Cortical afferents of the nucleus accumbens in the cat, studied with anterograde and retrograde transport techniques. Neuroscience 7:977–996.
    OpenUrlCrossRefPubMed
  44. ↵
    1. Mandler G
    (1975) Mind and Emotion (Wiley, New York).
  45. ↵
    1. Mas-Herrero E,
    2. Dagher A,
    3. Zatorre RJ
    (2017) Modulating musical reward sensitivity up and down with transcranial magnetic stimulation. Nat Hum Behav 2:27–32.
    OpenUrl
  46. ↵
    1. Siegel JS, et al.
    (2014) Statistical improvements in functional magnetic resonance imaging analyses produced by censoring high-motion data points. Hum Brain Mapp 35:1981–1996.
    OpenUrlCrossRefPubMed
  47. ↵
    1. Gold BP, et al.
    (2019) Data from “Musical reward prediction errors engage the nucleus accumbens and motivate learning.” NeuroVault. Available at https://neurovault.org/collections/4778/. Deposited January 28, 2019.
View Abstract
Back to top
Article Alerts
Email Article
Citation Tools
Request Permissions
Musical reward prediction errors engage the nucleus accumbens and motivate learning
Benjamin P. Gold, Ernest Mas-Herrero, Yashar Zeighami, Mitchel Benovoy, Alain Dagher, Robert J. Zatorre
Proceedings of the National Academy of Sciences Feb 2019, 116 (8) 3310-3315; DOI: 10.1073/pnas.1809855116
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
Proceedings of the National Academy of Sciences: 116 (45)
Current Issue

Submit

Article Classifications

Jump to section

You May Also be Interested in

Reconstructing the first day of the Cenozoic Era, following the asteroid impact 66 million years ago. Image courtesy of The University of Texas at Austin Jackson School of Geosciences.
Melt-rich rock from Chicxulub crater
Reconstructing the first day of the Cenozoic Era, following the asteroid impact 66 million years ago.
Image courtesy of The University of Texas at Austin Jackson School of Geosciences.
QnAs with NAS member and parasitologist Patricia Johnson. Image courtesy of Joy Ahn (University of California, Los Angeles, CA).
Featured QnAs
QnAs with NAS member and parasitologist Patricia Johnson
Image courtesy of Joy Ahn (University of California, Los Angeles, CA).

Similar Articles