Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Image credit: Dave Cutler (artist).
New Research In
Physical Sciences
Featured Portals
Articles by Topic
Biological Sciences
Featured Portals
Articles by Topic
-
Edited by Dale Purves, Duke University Medical Center, Durham, NC, and approved April 13, 2007 (received for review October 6, 2006)
Article Figures & SI
Figures
- Fig. 1.
Schematic of the idea of a fictive error and task design. (A) At time t, an agent in state St transitions to a new state St +1 by taking action at and observes a reward rt . However, the agent also observes other rewards r′ t that could have been received had alternative actions a′ t been chosen. (B) The figure under time t − 1 shows the state of the task immediately after a snippet of market has been revealed. At time t the subject makes a new allocation between cash and stock (in this case increasing the bet). When the market goes up, bigger investments are immediately revealed as better choices, generating the fictive error “best choice − actual choice.” Likewise for market drops, smaller investments would have been better.
- Fig. 2.
Experiment screen and time line. (A) Screen like that seen by subject (the background was dark in the scanner). The subject has just lost 23.92% (right box), has a portfolio worth $139 (left box), has 50% invested in the market (middle bar), and has nine choices remaining (from examining the screen). (B) Time-line of experiment. After the market outcome is revealed, the middle bar (which indicates the bet size) is grayed out, and a new bet cannot be submitted. The bar is illuminated 750 ms later, and the subject has a free response period to submit a new bet. After the new bet is submitted, the next snippet of market is revealed 750 ms later. The major regressors (including the fictive error) used in the fMRI analysis are time-locked to this event.
- Fig. 3.
Influence of fictive error signal on behavior. Barplot of the average normalized change in next investment versus the level of the fictive error. Changes in investment were converted into z-scores within each subject. The fictive error signal was binned into three levels [(0.00, 0.04), (0.04, 0.08), (0.08, ∞)] for the figure (see SI Fig. 8 for a scatterplot). Error bars are standard errors.
- Fig. 4.
Brain responses to fictive error signal. (Upper) SPM t-statistic map for the fictive error regressor (ft + ) showing activation in motor strip (a), inferior frontal gyrus (b), caudate and putamen (c), and PPC (d). Threshold: p < 1 × 10−5 (uncorrected); cluster size ≥5. Slices defined by y = 8 and y = −72. Random effects over subjects, n = 54. (Lower) SPM t-statistic map for the positive market return (r NL +) regressor in the “Not Live” condition showing no activation in the striatum but strong activation in the visual cortex (e). Threshold: p < 1 × 10−5 (uncorrected); cluster size ≥5. Slices defined by y = 8 and y = −72. Random effects over subjects, n = 54.
- Fig. 5.
Basic TD regressor and fictive error signal. SPM t-statistic maps of the basic TD regressor (Upper) and the fictive error signal (Lower) showing activation in the striatum associated with each. The fictive error regressor is orthogonalized with respect to the TD regressor. Threshold: p < 1 × 10−5 (uncorrected); cluster size ≥5. Random effects over subjects, n = 54. (Insets) Separate colored-coded activations for fictive error only, TD error only, and the overlap region of the two. These activations are shown at three levels of significance and suggest that activations to fictive error only may be segregated to the ventral caudate.
- Fig. 6.
Q-learning TD regressor and fictive error signal. SPM2 t-maps of the Q-learning TD regressor (Upper) and the fictive error signal (Lower) again showing activation in the ventral striatum associated with the TD error, and in the ventral caudate for the fictive error. The fictive error regressor is orthogonalized with respect to the TD regressor. Threshold: p < 1 × 10−5 (uncorrected); cluster size ≥5. Random effects over subjects, n = 54. (Insets) Separate colored-coded activations for fictive error only, TD error only, and the overlap region of the two. The area of overlap is larger for the Q-learning model and fictive error than for the TD regressor and fictive error (see Fig. 5).
Tables
Coefficient Estimate SE t value p value c −0.026 0.013 −2.11 0.023 b̃t 0.582 0.031 18.9 0.000 rt + 5.56 0.651 8.54 0.000 rt − −3.76 0.529 −7.09 0.000 bt · rt + −2.91 1.16 −2.51 0.006 bt · rt − −1.55 1.23 −1.26 0.105 -
Results of linear multiple regression of b̃ t+1, normalized next bet, on indicated variables: b̃ t is normalized previous bet, rt + = max(rt , 0), where rt is the previous market return, rt − = max(−r, 0), and bt is the unnormalized previous bet. bt · rt + is the actual investor return for the positive market case, and similarly for bt · rt − . Random effects over subjects, n = 54.
-
Data supplements
Lohrenz et al. 10.1073/pnas.0608842104.
Supporting Information
Files in this Data Supplement:
SI Figure 7
SI Dataset
SI Table 2
SI Appendix
SI Figure 8
SI Figure 9
SI Text
SI Figure 7Fig. 7. Graphs of markets. Shown are separate graphs of each market used in the task. Names used for the markets match those used in SI Data Set.
SI Figure 8Fig. 8. Scatterplot version of Fig. 3. x axis is fictive error signal, y axis is normalized (within-subject z-score) change in bet.
SI Figure 9SI Fig. 9. SPM t-statistic map of activation correlated with a negative market return.
SPM t-statistic map of the NegMkt regressor. The NegMkt regressor was constructed by modulating (multiplying point-wise in time) the RevealL regressor by
. The basic TD regressor and orthogonalized fictive error were also in the model. Threshold: 
, cluster size 
. Random effects over subjects, n = 54. Slices defined by 
. Activation is right inferior frontal gyrus (IFG) and right superior temporal gyrus.Table 2. Alternative behavioral regression
A potential concern about our behavioral analysis is that since
, the interaction term 
in the multiple regression reported in Table 1 may reflect the fact that for larger allocations subjects simply cannot make much larger allocations. In fact, this worry conforms to our intuition about a fictive error signal: the asset allocation is a finite resource; as such, there is always a boundary, and an organism should stop receiving critic signals when there is no possible further allocation of resources. From a mathematical point of view, if one posits that the influence of 
should wane as the allocation gets closer to one, then a reasonable assumption is that the coefficient of should be a function of 
that vanishes at 
. The simplest non-constant function that accomplishes this is
, making the term in the regression 
, which is the fictive error term. In order to more fully investigate this question we performed an ordinal logistic regression (function polr in R) with dependent variable the next allocation, categorized as "Small," "Neutral," and "Big," with independent variables including , and the previous allocation 
, normalized within subject. The results of this regression show again a significant interaction between and normalized allocation. Since the dependent variable in this regression is categorical the results are not muddied by possible "edge effects" from regressing a bounded variable on unbounded regressors.SI Text
Instructions Read to Subjects
Experiment Instructions
This experiment is designed to investigate economic decision making. During this experiment you will see a sequence of price histories from actual markets. There are two alternating conditions during the experiment. During the "Live" condition, you see an initial segment of price history. You will then use a slider bar activated by one of your two button boxes (which side will be chosen at random before the experiment) to allocate an endowment between cash and a risky asset whose value is given by the prices on the screen. Your allocation is represented by the percentage of value of your portfolio due to the risky asset. When you have decided your allocation push either button on the other button box to submit your allocation. The next segment of price history will then appear, your new portfolio value will be shown, and the percentage gain or loss of your portfolio will also be shown. Then you will have the opportunity to reallocate your assets, and the process will repeat. Each trading round will be 20 price-segments long. Your initial endowment will be $100. During the "Not Live" condition the events you see will almost be the same, but instead of making an asset allocation decision, you will decide whether the current price is higher or lower than the price from two price-segments earlier. Again, make your choice with the slider bar: if you think the current price is higher or equal to the two-segment-ago price, toggle the slider bar (using the button box) above 50%; if you think it is lower, move the slider below 50%. Once you have decided, submit the choice using the other button box as before. At the conclusion of the experiment, you will be paid in cash the final value of your portfolio, plus $10 if your choices during the "Not Live" condition were at least 80% correct.
Neural signature of fictive learning signals in a sequential investment task
Sign up for Article Alerts
Jump to section
You May Also be Interested in
Several large or long-lived animals seem strangely resistant to developing cancer. Elucidating the reasons why could lead to promising cancer-fighting strategies in humans.
Image credit: Shutterstock.com/ronnybas frimages.
PNAS Profile of NAS member and biochemist Hao Wu
A study suggests a putative gene-expression hallmark common to monogamous male vertebrates of some species, namely cichlid fishes, dendrobatid frogs, passeroid songbirds, common voles, and deer mice, and identifies 24 candidate genes potentially associated with monogamy.
Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Physical and social well-being in old age are linked to self-assessments of life worth, and a spectrum of behavioral, economic, health, and social variables may influence whether aging individuals believe they are leading meaningful lives.
Image courtesy of Pixabay/MabelAmber.
More Articles of This Classification
Economic Sciences
Biological Sciences
Related Content
Cited by...
- Converging structural and functional connectivity of orbitofrontal, dorsolateral prefrontal, and posterior parietal cortex in the human striatum
- Contrasting temporal difference and opportunity cost reinforcement learning in an empirical money-emergence paradigm
- Interaction of Instrumental and Goal-Directed Learning Modulates Prediction Error Representations in the Ventral Striatum
- Neurocomputational mechanisms of prosocial learning and links to empathy
- Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward
- Dopamine: Context and counterfactuals
- The Good, the Bad, and the Irrelevant: Neural Mechanisms of Learning Real and Hypothetical Rewards and Effort
- Converging Structural and Functional Connectivity of Orbitofrontal, Dorsolateral Prefrontal, and Posterior Parietal Cortex in the Human Striatum
- Belief about nicotine selectively modulates value and reward prediction error signals in smokers
- Differential Modulation of Reinforcement Learning by D2 Dopamine and NMDA Glutamate Receptor Antagonism
- Irrational exuberance and neural crash warning signals during endogenous experimental market bubbles
- Exploiting and exploring the options
- Reward-Related Activity in Ventral Striatum Is Action Contingent and Modulated by Behavioral Relevance
- Hierarchical Learning Induces Two Simultaneous, But Separable, Prediction Errors in Human Basal Ganglia
- Don't Look Back in Anger! Responsiveness to Missed Chances in Successful and Nonsuccessful Aging
- Adaptive coding of reward prediction errors is gated by striatal coupling
- Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning
- Medial prefrontal cortex and striatum mediate the influence of social comparison on the decision process
- Neural Correlates of Forward Planning in a Spatial Decision Task in Humans
- Signals in Human Striatum Are Appropriate for Policy Update Rather than Value Prediction
- A Behavioral and Neural Evaluation of Prospective Decision-Making under Risk
- Neural mechanisms of observational learning
- Brain Hemispheres Selectively Track the Expected Value of Contralateral Options
- Neural Correlates of Value, Risk, and Risk Aversion Contributing to Decision Making under Risk
- Fictive Reward Signals in the Anterior Cingulate Cortex
- Neural correlates of mentalizing-related computations during strategic interactions in humans
- Reinforcement Learning Signals in the Human Striatum Distinguish Learners from Nonlearners during Reward-Based Decision Making