Abstract
Neuromodulators such as dopamine have a central role in cognitive disorders. In the past decade, biological findings on dopamine function have been infused with concepts taken from computational theories of reinforcement learning. These more abstract approaches have now been applied to describe the biological algorithms at play in our brains when we form value judgements and make choices. The application of such quantitative models has opened up new fields, ripe for attack by young synthesizers and theoreticians.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Sutton, R. S. & Barto, A. G. Reinforcement learning (MIT, Cambridge, Massachusetts, 1998).
Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).
Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Friston, K. J., Tononi, G., Reeke, G. N., Sporns, O. & Edelman, G. M. Value-dependent selection in the brain: simulation in a synthetic neural model. Neuroscience 59, 229–243 (1994).
Houk, J. C., Adams, J. L. & Barto, A. G. in Models of Information Processing in the Basal Ganglia (eds Houk, J. C. Davis, J. L. & Beiser, D. G.) Ch. 13, 249–270 (MIT, Cambridge, Massachusetts, 1995).
Skinner, B. F. Behaviorism at fifty. Science 140, 951–958 (1963).
Sutton, R. S. Learning to predict by the methods of temporal difference. Mach. Learn. 3, 9–44 (1988).
Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).
Dayan, P. & Abbott, L. F. Theoretical Neuroscience Ch. 9, 331–358 (MIT, Cambridge, Massachusetts, 2001).
Rescorla, R. A. & Wagner A. R. in Classical Conditioning 2: Current Research and Theory (eds Black, A. H. & Prokasy, W. F.) 64–69 (Appleton Century-Crofts, New York, 1972).
Bertsekas, D. P. & Tsitsiklis, J. N. in Neuro-Dynamic Programming (Athena Scientific, Belmont, Massachusetts, 1996).
Schultz W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).
Hollerman, J. R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci. 1, 304–309 (1998).
Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).
Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).
Bayer, H. M. & Glimcher, P. W. Subjective estimates of objective rewards: using economic discounting to link behavior and brain. Soc. Neurosci. Abstr. 28, 358.6 (2002).
Berridge, K. C. & Robinson, T. E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res. Rev. 28, 309–369 (1998).
Everitt, B. J. et al. Associative processes in addiction and reward: the role of amygdala-ventral striatal subsystems. Ann. NY Acad. Sci. 877, 412–438 (1999).
Ikemoto, S. & Panksepp, J. The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res. Rev. 31, 6–41 (1999).
Di Chiara, G. & Imperato, A. Drugs abused by humans preferentially increase synaptic dopamine concentrations in the mesolimbic system of freely moving rats. Proc. Natl Acad. Sci. USA 85, 5274–5278 (1988).
Berke, J. D. & Hyman, S. E. Addiction, dopamine, and the molecular mechanisms of memory. Neuron 25, 515–532 (2000).
Ikemoto, S. & Panksepp, J. Dissociations between appetitive and consummatory responses by pharmacological manipulations of reward-relevant brain regions. Behav. Neurosci. 110, 331–345 (1996).
Salamone, J. D. & Correa, M. Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav. Brain Res. 137, 3–25 (2002).
Redgrave, P., Prescott, T. J. & Gurney, K. Is the short-latency dopamine response too short to signal reward error? Trends Neurosci. 22, 146–151 (1999).
Egelman, D. M., Person, C., Montague, P. R. A computational role for dopamine delivery in human decision-making. J. Cogn. Neurosci. 10, 623–630 (1998).
McClure, S. M., Daw, N. & Montague, P. R. A computational substrate for incentive salience. Trends Neurosci. 26, 423–428 (2003).
Balleine, B. W. & Dickinson, A. The effect of lesions of the insular cortex on instrumental conditioning: evidence for a role in incentive memory. Neurosci. 20, 8954–8964 (2000).
Berridge, K. C. in The Psychology of Learning and Motivation: Advances in Research and Theory Vol. 40 (ed. Medin, D. L.) 223–278 (Academic, San Diego, 2001).
Dayan, P. & Balleine, B. W. Reward, motivation and reinforcement learning. Neuron 36, 285–298 (2002).
Berns, G. S., McClure, S. M., Pagnoni, G. & Montague, P. R. Predictability modulates human brain response to reward. J. Neurosci. 21, 2793–2798 (2001).
O'Doherty, J. P., Deichmann, R., Critchley, H. D. & Dolan, R. J. Neural responses during anticipation of a primary taste reward. Neuron 33, 815–826 (2002).
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H. & Dolan, R. J. Temporal difference models and reward related learning in the human brain. Neuron 38, 329–337 (2003).
McClure, S. M., Berns, G. S. & Montague, P. R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 38, 339–346 (2003).
O'Doherty, J. P. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).
Aharon, I. et al. Beautiful faces have variable reward value: fMRI and behavioral evidence. Neuron 32, 537–551 (2001).
Breiter, H. C. et al. Acute effects of cocaine on human brain activity and emotion. Neuron 19, 591–611 (1997).
Breiter, H. C., Aharon, I., Kahneman, D., Dale, A. & Shizgal, P. Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 30, 619–639 (2001).
Knutson, B., Westdorp, A., Kaiser, E. & Hommer, D. fMRI visualization of brain activity during a monetary incentive delay task. Neuroimage 12, 20–27 (2000).
Knutson, B., Adams, C. M., Fong, G. W. & Hommer, D. J. Anticipation of increasing monetary reward selectively recruits nucleus accumbens. J. Neurosci. 15, 1–5 (2001).
Thut, G. et al. Activation of the human brain by monetary reward. Neuroreport 8, 1225–1228 (1997).
Delgado, M. R., Nystrom, L. E., Fissel, C., Noll, D. C. & Fiez, J. A. Tracking the hemodynamic responses to reward and punishment in the striatum. J. Neurophysiol. 84, 3072–3077 (2000).
Elliott, R., Friston, K. J. & Dolan, R. J. Dissociable neural responses in human reward systems. J. Neurosci. 20, 6159–6165 (2000).
Montague, P. R. & Berns, G. S. Neural economics and the biological substrates of valuation. Neuron 36, 265–284 (2002).
Pagnoni, G., Zink, C. F., Montague, P. R. & Berns, G. S. Activity in human ventral striatum locked to errors of reward prediction. Nature Neurosci. 5, 97–98 (2002).
Gehring, W. J., Goss, B., Coles, M. G. H., Meyer, D. E. & Donchin, E. A neural system for error detection and compensation. Psychol. Sci. 4, 385–390 (1993).
Falkenstein, M., Hohnsbein, J. & Hoormann, J. in Perspectives of Event-Related Potentials Research (eds Karmos, G. et al.) 287–296 (Elsevier Science, Amsterdam, 1994).
Gehring, W. J. & Willoughby, A. R. The medial frontal cortex and the rapid processing of monetary gains and losses. Science 295, 2279–2282 (2002).
Ullsperger, M. & von Cramon, D. Y. Error monitoring using external feedback: specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging. J. Neurosci. 23, 4308–4314 (2003).
Nieuwenhuis, S., Yeung, N., Holroyd, C. B., Schurger, A. & Cohen, J. D. Sensitivity of electrophysiological activity from medial frontal cortex to utilitarian and performance feedback. Cereb. Cort. 14, 741–747 (2004).
Holroyd, C. B. & Coles, M. G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679–709 (2002).
Holroyd, C. B., Nieuwenhuis, S., Yeung, N. & Cohen, J. D. Errors in reward prediction are reflected in the event-related brain potential. Neuroreport 14, 2481–2484 (2003).
Holroyd, C. B., Larsen, J. T. & Cohen, J. D. Context dependence of the event-related brain potential associated with reward and punishment. Psychophysiol. 41, 245–253 (2004).
Holroyd, C. B. et al. Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nature Neurosci. 7, 497–498 (2004).
Miller, E. K. & Cohen, J. D. An integrative theory of prefrontal cortex function. Annu. Rev. of Neurosci. 24, 167–202 (2001).
'Reilly, R. C., Braver, T. S. & Cohen, J. D. in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control (eds Miyake, A. & Shah, P.) Ch. 11, 375–411 (Cambridge Univ. Press, New York, 1999).
Miller, E. K., Li, L. & Desimone, R. A neural mechanism for working and recognition memory in inferior temporal cortex. Science 254, 1377–1379 (1991).
Miller, E. K., Erickson, C. A. & Desimone, R. Neural mechanisms of visual working memory in prefrontal cortex of the macaque. J. Neurosci. 16, 5154–5167 (1996).
Duncan, J. Disorganization of behavior after frontal lobe damage. Cog. Neuropsychol. 3, 271–290 (1986).
Shallice, T. in From Neuropsychology to Mental Structure (Cambridge Univ. Press, Cambridge, 1988).
Koechlin, E., Ody, C. & Kouneiher, F. The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185 (2003).
Stuss, D. T. & Knight, R. T. Principles of Frontal Lobe Function (Oxford Univ. Press, New York, 2002).
Braver, T. S. & Cohen, J. D. in Attention and Performance XVIII; Control of Cognitive Processes (eds Monsell, S. & Driver, J.) 713–737 (MIT, Cambridge, Massachusetts, 2000).
Daw, N. D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).
O'Reilly, R. C., Noelle, D. C., Braver, T. S. & Cohen, J. D. Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control. Cereb. Cort. 12, 246–257 (2002).
Rougier, N. P. & O'Reilly, R. C. Learning representations in a gated prefrontal cortex model of dynamic task switching. Trends Cogn. Sci. 26, 503–520 (2002).
Wise, R. A. & Bozarth, M. A. A psychomotor stimulant theory of addiction. Psychol. Rev. 94, 469–492 (1987).
Hyman, S. E. & Malenka, R. C. Addiction and the brain: the neurobiology of compulsion and its persistence. Nature Rev. Neurosci. 2, 695–703 (2001).
Potenza, M. N. et al. Gambling urges in pathological gambling: a functional magnetic resonance imaging study. Arch. Gen. Psych. 60, 828–836 (2003).
Cohen, B. Dopamine receptors and antipsychotic drugs. Mclean Hosp. J. 6, 95–115 (1981).
Weinberger, D. R. Implications of normal brain development for the pathogenesis of schizophrenia. Arch. Gen. Psych. 44, 660–669 (1987).
Servan-Schreiber, D., Printz, H. & Cohen, J. D. A network model of catecholamine effects: gain, signal-to-noise ratio and behavior. Science 249, 892–895 (1990).
Montague, P. R. et al. Dynamic gain control of dopamine delivery in freely moving animals. J. Neurosci. 24, 1754–1759 (2004).
Author information
Authors and Affiliations
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Rights and permissions
About this article
Cite this article
Montague, P., Hyman, S. & Cohen, J. Computational roles for dopamine in behavioural control. Nature 431, 760–767 (2004). https://doi.org/10.1038/nature03015
Published:
Issue Date:
DOI: https://doi.org/10.1038/nature03015
This article is cited by
-
Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder
Translational Psychiatry (2024)
-
Electrophysiological signatures of reward learning in the rodent touchscreen-based Probabilistic Reward Task
Neuropsychopharmacology (2023)
-
Art Value Creation and Destruction
Integrative Psychological and Behavioral Science (2023)
-
A neuroeconomic signature of opioid craving: How fluctuations in craving bias drug-related and nondrug-related value
Neuropsychopharmacology (2022)
-
Value order in disorder
International Journal of Dynamics and Control (2022)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.