Abstract
The best rewards are often distant and can only be achieved by planning and decision-making over several steps. We designed a multi-step choice task in which monkeys followed internal plans to save rewards toward self-defined goals. During this self-controlled behavior, amygdala neurons showed future-oriented activity that reflected the animal's plan to obtain specific rewards several trials ahead. This prospective activity encoded crucial components of the animal's plan, including value and length of the planned choice sequence. It began on initial trials when a plan would be formed, reappeared step by step until reward receipt, and readily updated with a new sequence. It predicted performance, including errors, and typically disappeared during instructed behavior. Such prospective activity could underlie the formation and pursuit of internal plans characteristic of goal-directed behavior. The existence of neuronal planning activity in the amygdala suggests that this structure is important in guiding behavior toward internally generated, distant goals.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnn.3925/MediaObjects/41593_2015_Article_BFnn3925_Fig1_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnn.3925/MediaObjects/41593_2015_Article_BFnn3925_Fig2_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnn.3925/MediaObjects/41593_2015_Article_BFnn3925_Fig3_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnn.3925/MediaObjects/41593_2015_Article_BFnn3925_Fig4_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnn.3925/MediaObjects/41593_2015_Article_BFnn3925_Fig5_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnn.3925/MediaObjects/41593_2015_Article_BFnn3925_Fig6_HTML.jpg)
Similar content being viewed by others
References
Benhabib, J. & Bisin, A. Modeling internal commitment mechanisms and self-control: A neuroeconomics approach to consumption-saving decisions. Games Econ. Behav. 52, 460–492 (2005).
Dickinson, A. & Balleine, B. Motivational control of goal-directed action. Anim. Learn. Behav. 22, 1–22 (1994).
Berkman, E.T. & Lieberman, M.D. The neuroscience of goal pursuit. in The Psychology of Goals (eds. Moskowitz, G. & Grant, H.) 98–126 (Guilford, New York, 2009).
Tanji, J. Sequential organization of multiple movements: involvement of cortical motor areas. Annu. Rev. Neurosci. 24, 631–651 (2001).
Andersen, R.A. & Cui, H. Intention, action planning, and decision making in parietal-frontal circuits. Neuron 63, 568–583 (2009).
Passingham, R.E. & Wise, S.P. The Neurobiology of the Prefrontal Cortex (Oxford Univ. Press, 2012).
Mushiake, H., Saito, N., Sakamoto, K., Itoyama, Y. & Tanji, J. Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron 50, 631–641 (2006).
Saito, N., Mushiake, H., Sakamoto, K., Itoyama, Y. & Tanji, J. Representation of immediate and final behavioral goals in the monkey prefrontal cortex during an instructed delay period. Cereb. Cortex 15, 1535–1546 (2005).
Shima, K., Isoda, M., Mushiake, H. & Tanji, J. Categorization of behavioural sequences in the prefrontal cortex. Nature 445, 315–318 (2007).
Shima, K., Mushiake, H., Saito, N. & Tanji, J. Role for cells in the presupplementary motor area in updating motor plans. Proc. Natl. Acad. Sci. USA 93, 8694–8698 (1996).
Lee, I.H., Seitz, A.R. & Assad, J.A. Activity of tonically active neurons in the monkey putamen during initiation and withholding of movement. J. Neurophysiol. 95, 2391–2403 (2006).
Okano, K. & Tanji, J. Neuronal activities in the primate motor fields of the agranular frontal cortex preceding visually triggered and self-paced movement. Exp. Brain Res. 66, 155–166 (1987).
Romo, R. & Schultz, W. Neuronal activity preceding self-initiated or externally timed arm movements in area 6 of monkey cortex. Exp. Brain Res. 67, 656–662 (1987).
Kurata, K. & Wise, S.P. Premotor and supplementary motor cortex in rhesus monkeys: neuronal activity during externally- and internally-instructed motor tasks. Exp. Brain Res. 72, 237–248 (1988).
Rolls, E.T. Emotion and Decision-Making Explained (Oxford Univ. Press, 2014).
Baxter, M.G. & Murray, E.A. The amygdala and reward. Nat. Rev. Neurosci. 3, 563–573 (2002).
Cardinal, R.N., Parkinson, J.A., Hall, J. & Everitt, B.J. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci. Biobehav. Rev. 26, 321–352 (2002).
Seymour, B. & Dolan, R. Emotion, decision making, and the amygdala. Neuron 58, 662–671 (2008).
Murray, E.A. & Rudebeck, P.H. The drive to strive: goal generation based on current needs. Front. Neurosci. 7, 112 (2013).
Schultz, W. Multiple reward signals in the brain. Nat. Rev. Neurosci. 1, 199–207 (2000).
Paton, J.J., Belova, M.A., Morrison, S.E. & Salzman, C.D. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439, 865–870 (2006).
Schoenbaum, G., Chiba, A.A. & Gallagher, M. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nat. Neurosci. 1, 155–159 (1998).
Price, J.L. Comparative aspects of amygdala connectivity. Ann. NY Acad. Sci. 985, 50–58 (2003).
Belova, M.A., Paton, J.J., Morrison, S.E. & Salzman, C.D. Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron 55, 970–984 (2007).
Bermudez, M.A., Gobel, C. & Schultz, W. Sensitivity to temporal reward structure in amygdala neurons. Curr. Biol. 22, 1839–1844 (2012).
Wellman, L.L., Gale, K. & Malkova, L. GABAA-mediated inhibition of basolateral amygdala blocks reward devaluation in macaques. J. Neurosci. 25, 4577–4586 (2005).
Rudebeck, P.H. & Murray, E.A. Amygdala and orbitofrontal cortex lesions differentially influence choices during object reversal learning. J. Neurosci. 28, 8338–8343 (2008).
Grabenhorst, F., Rolls, E.T., Parris, B.A. & D'Souza, A. How the brain represents the reward value of fat in the mouth. Cereb. Cortex 20, 1082–1091 (2010).
Grabenhorst, F., Schulte, F.P., Maderwald, S. & Brand, M. Food labels promote healthy choices by a decision bias in the amygdala. Neuroimage 74, 152–163 (2013).
De Martino, B., Kumaran, D., Seymour, B. & Dolan, R.J. Frames, biases, and rational decision-making in the human brain. Science 313, 684–687 (2006).
Bechara, A., Damasio, H., Damasio, A.R. & Lee, G.P. Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making. J. Neurosci. 19, 5473–5481 (1999).
Brand, M., Grabenhorst, F., Starcke, K., Vandekerckhove, M.M. & Markowitsch, H.J. Role of the amygdala in decisions under ambiguity and decisions under risk: evidence from patients with Urbach-Wiethe disease. Neuropsychologia 45, 1305–1317 (2007).
Sugase-Miyamoto, Y. & Richmond, B.J. Neuronal signals in the monkey basolateral amygdala during reward schedules. J. Neurosci. 25, 11071–11083 (2005).
Grabenhorst, F., Hernadi, I. & Schultz, W. Prediction of economic choice by primate amygdala neurons. Proc. Natl. Acad. Sci. USA 109, 18950–18955 (2012).
Padoa-Schioppa, C. Neurobiology of economic choice: a good-based model. Annu. Rev. Neurosci. 34, 333–359 (2011).
Lee, D., Seo, H. & Jung, M.W. Neural basis of reinforcement learning and decision making. Annu. Rev. Neurosci. 35, 287–308 (2012).
Kennerley, S.W., Behrens, T.E. & Wallis, J.D. Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nat. Neurosci. 14, 1581–1589 (2011).
Hayden, B.Y., Pearson, J.M. & Platt, M.L. Neuronal basis of sequential foraging decisions in a patchy environment. Nat. Neurosci. 14, 933–939 (2011).
Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).
Belova, M.A., Paton, J.J. & Salzman, C.D. Moment-to-moment tracking of state value in the amygdala. J. Neurosci. 28, 10023–10030 (2008).
Lak, A., Stauffer, W.R. & Schultz, W. Dopamine prediction error responses integrate subjective value from different reward dimensions. Proc. Natl. Acad. Sci. USA 111, 2343–2348 (2014).
Kable, J.W. & Glimcher, P.W. The neurobiology of decision: consensus and controversy. Neuron 63, 733–745 (2009).
Shima, K. & Tanji, J. Role for cingulate motor area cells in voluntary movement selection based on reward. Science 282, 1335–1338 (1998).
Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P. & Dolan, R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).
Gersch, T.M., Foley, N.C., Eisenberg, I. & Gottlieb, J. Neural correlates of temporal credit assignment in the parietal lobe. PLoS ONE 9, e88725 (2014).
Dragoi, G. & Tonegawa, S. Preplay of future place cell sequences by hippocampal cellular assemblies. Nature 469, 397–401 (2011).
Pfeiffer, B.E. & Foster, D.J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79 (2013).
Rudebeck, P.H., Mitz, A.R., Chacko, R.V. & Murray, E.A. Effects of amygdala lesions on reward-value coding in orbital and medial prefrontal cortex. Neuron 80, 1519–1531 (2013).
Pessoa, L. & Adolphs, R. Emotion processing and the amygdala: from a 'low road' to 'many roads' of evaluating biological significance. Nat. Rev. Neurosci. 11, 773–783 (2010).
Bechara, A. Decision making, impulse control and loss of willpower to resist drugs: a neurocognitive perspective. Nat. Neurosci. 8, 1458–1463 (2005).
Bermudez, M.A. & Schultz, W. Reward magnitude coding in primate amygdala neurons. J. Neurophysiol. 104, 3424–3432 (2010).
Aggleton, J.P. & Passingham, R.E. Stereotaxic surgery under X-ray guidance in the rhesus monkey, with special reference to the amygdala. Exp. Brain Res. 44, 271–276 (1981).
Watanabe, M. et al. Behavioral reactions reflecting differential reward expectations in monkeys. Exp. Brain Res. 140, 511–518 (2001).
O'Neill, M. & Schultz, W. Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value. Neuron 68, 789–800 (2010).
Draper, N.R. & Smith, H. Applied Regression Analysis (Wiley-Interscience, 1998).
Kutner, M.H., Nachtsheim, C.J., Neter, J. & Li, W. Applied Linear Statistical Models (McGraw-Hill Irwin, New York, 2004).
Smith, D.V. & Travers, J.B. A metric for the breadth of tuning of gustatory neurons. Chem. Senses 4, 215–219 (1979).
Acknowledgements
We thank K.-I. Tsutsui for help with task design; A. Dickinson, J. Assad, R. Báez-Mendoza, A. Lak, W. Stauffer and M. O'Neill for discussions; and M. Arroyo for histology. We also thank the Wellcome Trust, the European Research Council, the Cambridge Behavioural and Clinical Neuroscience Institute (BCNI) and the US National Institutes of Health Caltech Conte Center for financial support.
Author information
Authors and Affiliations
Contributions
I.H., F.G. and W.S. designed the research. I.H. performed the experiments. F.G. analyzed the data. F.G. and W.S. wrote the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Integrated supplementary information
Supplementary Figure 1 Adaptive saving behavior and control experiments.
(a) Reward-saving behavior for different interest rates in both animals (n = 11,533 trials). The monkeys saved more with higher interest rates. (b) Behavioral adaptation to changes in interest rate. We defined a criterion for speed of behavioral adaptation to current interest rate at the start of a testing session. Adaptation criterion was the number of saving sequences before the animal produced the preferred sequence (i.e. the one with the highest value given current interest rate). On average, the animals produced one sequence before criterion was reached (median for all three distributions was one sequence). Thus, the animals readily adjusted their behavior to changes in interest. (c,d) Optimality in saving behavior. For the lowest and highest interest rate conditions, the rate of reward return either decreased (low interest) or was a non-monotonic function of sequence length (high interest, see Methods). We tested whether behavior for these conditions approximated optimal behavior (i.e. tracked reward return rate). Rate of reward return was calculated for each saving step as difference in reward magnitude between current and previous trial, normalized by base amount from the first saving trial (Eq. 11):
with R as rate of reward return, RM as reward magnitude, i as trial index, and base as base reward magnitude. (c) Relative choice frequencies, reward magnitudes, and rates of reward return for high and low interest. Choice frequencies tracked rate of reward return (compare black bars and orange curves), suggesting that the animals approximated optimal behavior. (d) Across animals and interest rates there was a significant relationship (r = 0.73, P = 3.3 × 10−7, linear regression, n = 38) between relative choice frequency and normalized rate of reward return. (e) Control test with uncued changes in interest. An example testing session in which interest rate decreased in blocks of about 20 saving sequences. Colored boxes show block durations for different interest rates q. The identical save cue was used throughout the session. At session start, the animal had no knowledge of the current interest rate but quickly learned to generate longer sequences to adapt to current high interest rate (steady rise in the blue curve in first ten sequences). Green curve: block-wise mean saving lengths tracked interest across whole session. (f) Behavioral test for internal tracking of saved juice amounts. On random catch trials, the animals chose between currently saved juice amounts and offered fixed amounts (indicated by pre-trained cues). Intersections between horizontal gray lines and choice curves (exponential fits to spending frequencies) indicate points of subjective indifference between saved and fixed rewards. Both animals were indifferent when saved and fixed rewards had equal magnitudes and consistently chose fixed reward when it exceeded saved magnitude (z28 = 4.56, P = 5.06 × 10−6, Mann-Whitney test, n = 1,122). Thus, the animals’ choices reflected knowledge about accumulated saved rewards.
Supplementary Figure 2 Factors influencing reaction times and licking durations.
(a) Reaction times during saving (across all trials of a saving sequence, n = 11,533) depended on sequence value as shown by significant sequence value regressor. Negative beta for sequence value indicates faster reactions when saving towards higher sequence values. This effect of sequence value on reaction times was not explained by alternative variables, including trial-specific choice (Choice, positive beta indicates longer reactions on save compared to spend trials), trial-specific subjective values (Spend value and Save value), interest rate, consumed juice (Juice/day) or animal identity (Monkey). (A supplementary stepwise regression also selected the key variables choice, spend value, save value, sequence value; all P < 0.002). (b) Reaction times on the first trial of a sequence depended on sequence value. The animals reacted faster on the first trial of each sequence when they were going to save towards higher sequence values (significant negative coefficient for the interaction term Sequence value × first trial indicator, n = 11,533). ** P < 0.005, * P < 0.05, n.s. not significant. The relationship between reaction times and subjective value in (a) and (b) was not explained by increased ‘urgency’ for longer sequences, as subjective value explained a significant proportion of reaction time variance even after sequence length was accounted for (P = 0.0028, partial correlation). The relationship was also not explained by a ‘first-choice’ effect, whereby reaction times would be longer specifically on first trials, as the animal plans the entire sequence in advance: contrary to such a ‘first-choice’ effect, the animals reacted on average faster on first trials (significant First trial indicator coefficient). (c) Regression of anticipatory (pre-cue) licking durations in animal A on sequence value and other variables. (Licking behavior in animal B was less consistent.) This analysis revealed a significant (P = 1.0 × 10−15, n = 6,270) effect of sequence value on licking duration in animal A, with more anticipatory licking for sequences that resulted in higher value. (d) Reaction times in the imperative saving task (n = 3,961). Left: Similar to the free choice task, reaction times depended on sequence length. Reactions on spend trials (black) and save trials (magenta) were faster for longer sequences (linear regressions). Reactions were also faster on spend trials compared to save trials (P < 10−9, Wilcoxon test). Thus, the animals anticipated saving outcomes even when saving was instructed. (This was possible because we matched saving lengths between tasks for animals and interest rates). Right: Multiple regression. Similar to the free choice task, reaction times depended on sequence value, with faster reactions when saving towards higher values. ** P < 0.005, * P < 0.05, n.s. not significant.
Supplementary Figure 3 Control variables for the single neuron shown in Figure 2.
Linear regressions of activity on different trial-by-trial variables illustrate that this neuron’s activity was best explained by sequence value (Fig. 2) rather than by trial-specific variables. Data are shown for illustration purposes; formal selection of neurons was performed using a multiple regression approach as described in the Methods.
Supplementary Figure 4 Population data shown separately for the four types of planning activity illustrated in Figure 3.
(a) Planning activity of 61 responses in 45 neurons encoding sequence value across all saving trials. Right: Average activity of this group of neurons reflected sequence value (r2 = 0.88, P = 0.0015, linear regression, n = 7) rather than sequence length (r2 = 0.23, P > 0.1). (b) Planning activity of 31 responses in 30 neurons encoding sequence value specifically on first saving trials. Right: Average activity of this group of neurons reflected sequence value (r2 = 0.85, P = 0.0038, linear regression, n = 7) rather than sequence length (r2 = 0.28, P > 0.1). (c) Planning activity of 55 responses in 45 neurons encoding sequence length across all saving trials. Right: Average activity of this group of neurons reflected sequence length (r2 = 0.94, P = 0.0015, linear regression, n = 7) rather than sequence value (r2 = 0.41, P > 0.1). (d) Planning activity of 37 responses in 33 neurons encoding sequence length specifically on first saving trials. Right: Average activity of this group of neurons reflected sequence length (r2 = 0.88, P = 0.0018, linear regression, n = 7) rather than sequence value (r2 = 0.12, P > 0.1).
Supplementary Figure 5 Planning activity reflects parametric variables, rather than narrow tuning to specific sequences.
In analogy to sensory neurons, amygdala neurons with planning activity might show tuning to specific sequences. For example, sequence value neurons might respond selectively when the most highly valued sequence is planned, with little response to other, less preferred sequences. We compared regression betas from our main analysis (which used parametric regressors for sequence value and sequence length) with those from a supplementary model which used indicator functions to model the peak sequence in the sequence value/sequence length distribution. For sequence value neurons, peak sequence corresponded to the sequence with the highest value (or lowest value, for neurons with negative coding). For sequence length neurons, peak sequence corresponded to the longest sequence chosen in a session (or the shortest sequence, for neurons with negative coding). (a) Sign-corrected betas from our main analysis (‘Seq. value/length beta’) are plotted against sign-corrected betas from the supplementary analysis (‘Peak sequence beta’). The angled histogram shows the distribution of beta differences between models. Data are shown for all responses encoding sequence value or sequence length across whole sequences (n = 116 responses from 86 neurons). In most cases, the sequence value/sequence length model provided the better fit compared to the peak sequence model. As shown in (b) the two distributions are significantly different (Kolmogorov Smirnov test), with sequence value/sequence length betas shifted towards higher values. In addition, we used a conventional approach to examine tuning of amygdala neurons to different sequences by calculating coefficients for breadth of tuning. The breadth of tuning metric ranges from 0 to 1.0 with 0 indicating total specificity to one sequence and 1.0 indicating equal responses to all sequences. We show in (c) the distribution of this metric for neuronal responses related to sequence value or sequence length across all trials (n = 116). Distribution mean was at 0.95 (± 0.06 s.d.), indicating broad tuning to different sequences. Similar broad tuning in amygdala neurons has previously been reported in response to oral sensory stimuli. These results confirm a parametric representation of the planning variables sequence value and sequence length, whereby a given neuron shows graded, rather than categorical, responses to a set of sequence lengths, with the response magnitude depending on value or length of each sequence.
Supplementary Figure 6 Histological reconstruction of recording sites based on marker lesions.
Photomicrographs of cresyl violet-stained coronal sections of the amygdala in monkey A (upper three panels) and monkey B (lower three panels). Arrows mark electrolytic lesions made after recordings. Lesions were placed to indicate the estimated extent of the amygdala region (upper panels) and typical recording sites (lower panels) in the ventral, intermediate and dorsal parts of the amygdala. The dashed lines indicate the approximate boundaries of the centromedial (dorsal) and basolateral (ventral) areas, respectively. Boundaries were determined based on anatomical landmarks and with the help of a stereotaxic atlas.
Supplementary Figure 7 A single amygdala neuron with planning activity that reflected final reward magnitude.
The fixation-period activity of this neuron showed a linear relationship to the objective magnitude of juice reward that would be obtained at the end of a saving sequence (linear regression, n = 36). The neuron had higher activity for lower final magnitudes. As sequence length and final reward magnitude were often correlated, it was difficult in the present data set to distinguish these two objective quantities. Nevertheless, some observations suggest that planning activity in the present neurons was better described by sequence length than by reward magnitude: First, a stepwise regression with final reward magnitude as covariate identified a larger number of responses related to sequence length than reward magnitude (48 vs. 23 responses). Second, partial correlation based on the identified responses related to sequence length on all trials (n = 55) showed that the relationship between neuronal activity and sequence length remained significant (P = 2.2 × 10−10) even after reward magnitude was accounted for. Third, averaged population activity for responses related to sequence length was better described by a linear function rather than by an exponential increase related to reward magnitude (Fig. 4d, Supplementary Fig. 4c,d, compare magenta and green curves). Thus, sequence length seemed a more appropriate variable to characterize the present population of neurons.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–7 and Supplementary Tables 1–3 (PDF 2280 kb)
Rights and permissions
About this article
Cite this article
Hernádi, I., Grabenhorst, F. & Schultz, W. Planning activity for internally generated reward goals in monkey amygdala neurons. Nat Neurosci 18, 461–469 (2015). https://doi.org/10.1038/nn.3925
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.3925
This article is cited by
-
Movement errors during skilled motor performance engage distinct prediction error mechanisms
Communications Biology (2020)
-
Distinct cortical–amygdala projections drive reward value encoding and retrieval
Nature Neuroscience (2019)
-
Free choice shapes normalized value signals in medial orbitofrontal cortex
Nature Communications (2018)
-
Neural encoding of choice during a delayed response task in primate striatum and orbitofrontal cortex
Experimental Brain Research (2018)
-
Single-cell coding of sensory, spatial and numerical magnitudes in primate prefrontal, premotor and cingulate motor cortices
Experimental Brain Research (2016)