Abstract
We recently showed that improved perceptual performance on a visual motion direction–discrimination task corresponds to changes in how an unmodified sensory representation in the brain is interpreted to form a decision that guides behavior. Here we found that these changes can be accounted for using a reinforcement-learning rule to shape functional connectivity between the sensory and decision neurons. We modeled performance on the basis of the readout of simulated responses of direction-selective sensory neurons in the middle temporal area (MT) of monkey cortex. A reward prediction error guided changes in connections between these sensory neurons and the decision process, first establishing the association between motion direction and response direction, and then gradually improving perceptual sensitivity by selectively strengthening the connections from the most sensitive neurons in the sensory population. The results suggest a common, feedback-driven mechanism for some forms of associative and perceptual learning.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Gibson, E.J. Perceptual learning. Annu. Rev. Psychol. 14, 29–56 (1963).
Gilbert, C.D., Sigman, M. & Crist, R.E. The neural basis of perceptual learning. Neuron 31, 681–697 (2001).
Mukai, I. et al. Activations in visual and attention-related areas predict and correlate with the degree of perceptual learning. J. Neurosci. 27, 11401–11411 (2007).
Law, C.T. & Gold, J.I. Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area. Nat. Neurosci. 11, 505–513 (2008).
Sutton, R.S. & Barto, A.G. Reinforcement Learning: an Introduction (MIT Press, Cambridge, Massachusetts, 1998).
Schultz, W. Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87–115 (2006).
Schultz, W. Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288 (2007).
Berridge, K.C. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology (Berl.) 191, 391–431 (2007).
Redgrave, P., Gurney, K. & Reynolds, J. What is reinforced by phasic dopamine signals? Brain Res. Rev. 58, 322–339 (2008).
Bao, S., Chan, V.T. & Merzenich, M.M. Cortical remodeling induced by activity of ventral tegmental dopamine neurons. Nature 412, 79–83 (2001).
Seitz, A. & Watanabe, T. A unified model for perceptual learning. Trends Cogn. Sci. 9, 329–334 (2005).
Roitman, J.D. & Shadlen, M.N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci. 22, 9475–9489 (2002).
Hanks, T.D., Ditterich, J. & Shadlen, M.N. Microstimulation of macaque area LIP affects decision-making in a motion discrimination task. Nat. Neurosci. 9, 682–689 (2006).
Platt, M.L. & Glimcher, P.W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).
Andersen, R.A. & Buneo, C.A. Intentional maps in posterior parietal cortex. Annu. Rev. Neurosci. 25, 189–220 (2002).
Colby, C.L. & Goldberg, M.E. Space and attention in parietal cortex. Annu. Rev. Neurosci. 22, 319–349 (1999).
Gold, J.I. & Shadlen, M.N. The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007).
Salzman, C.D., Britten, K.H. & Newsome, W.T. Cortical microstimulation influences perceptual judgments of motion direction. Nature 346, 174–177 (1990).
Britten, K.H., Shadlen, M.N., Newsome, W.T. & Movshon, J.A. The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurosci. 12, 4745–4765 (1992).
Williams, R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992).
Shadlen, M.N., Britten, K.H., Newsome, W.T. & Movshon, J.A. A computational analysis of the relationship between neuronal and behavioral responses to visual motion. J. Neurosci. 16, 1486–1510 (1996).
Dayan, P. & Daw, N.D. Decision theory, reinforcement learning and the brain. Cogn. Affect. Behav. Neurosci. 8, 429–453 (2008).
Zohary, E., Shadlen, M.N. & Newsome, W.T. Correlated neuronal discharge rate and its implications for psychophysical performance. Nature 370, 140–143 (1994).
Bair, W., Zohary, E. & Newsome, W.T. Correlated firing in macaque visual area MT: time scales and relationship to behavior. J. Neurosci. 21, 1676–1697 (2001).
Barto, A.G. & Anadan, P. Pattern recognizing stochastic learning automata. IEEE Trans. Syst. Man Cybern. 28, 360–374 (1985).
Loewenstein, Y. & Seung, H.S. Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc. Natl. Acad. Sci. USA 103, 15224–15229 (2006).
Mazurek, M.E., Roitman, J.D., Ditterich, J. & Shadlen, M.N. A role for neural integrators in perceptual decision making. Cereb. Cortex 13, 1257–1269 (2003).
Kepecs, A., Uchida, N., Zariwala, H.A. & Mainen, Z.F. Neural correlates, computation and behavioral impact of decision confidence. Nature 455, 227–231 (2008).
Gold, J.I. & Shadlen, M.N. Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5, 10–16 (2001).
Kiani, R., Hanks, T.D. & Shadlen, M.N. Bounded integration in parietal cortex underlies decisions even when viewing duration is dictated by the environment. J. Neurosci. 28, 3017–3029 (2008).
Spiegelhalter, D.J. & Lauritzen, S.L. Sequential updating of conditional probabilities on directed graphical structures. Networks 20, 579–605 (1990).
Royer, S. & Pare, D. Conservation of total synaptic weight through balanced synaptic depression and potentiation. Nature 422, 518–522 (2003).
Connolly, P.M., Bennur, S. & Gold, J.I. Correlates of perceptual learning in an oculomotor decision variable. J. Neurosci. 29, 2136–2150 (2009).
Purushothaman, G. & Bradley, D.C. Neural population code for fine perceptual decisions in area MT. Nat. Neurosci. 8, 99–106 (2005).
Britten, K.H., Newsome, W.T., Shadlen, M.N., Celebrini, S. & Movshon, J.A. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Vis. Neurosci. 13, 87–100 (1996).
Sompolinsky, H., Yoon, H., Kang, K. & Shamir, M. Population coding in neuronal systems with correlated noise. Phys. Rev. E 64, 051904 (2001).
Smith, M.A. & Kohn, A. Spatial and temporal scales of neuronal correlation in primary visual cortex. J. Neurosci. 28, 12591–12603 (2008).
Gold, J.I. & Shadlen, M.N. The influence of behavioral context on the representation of a perceptual decision in developing oculomotor commands. J. Neurosci. 23, 632–651 (2003).
Jazayeri, M. & Movshon, J.A. A new perceptual illusion reveals mechanisms of sensory decoding. Nature 446, 912–915 (2007).
Hol, K. & Treue, S. Different populations of neurons contribute to the detection and discrimination of visual motion. Vision Res. 41, 685–689 (2001).
Ball, K. & Sekuler, R. Direction-specific improvement in motion discrimination. Vision Res. 27, 953–965 (1987).
Fahle, M. Perceptual learning: specificity versus generalization. Curr. Opin. Neurobiol. 15, 154–160 (2005).
Ahissar, M. & Hochstein, S. Task difficulty and the specificity of perceptual learning. Nature 387, 401–406 (1997).
Newsome, W.T. & Pare, E.B. A selective impairment of motion perception following lesions of the middle temporal visual area (MT). J. Neurosci. 8, 2201–2211 (1988).
Celebrini, S. & Newsome, W.T. Neuronal and psychophysical sensitivity to motion signals in extrastriate area MST of the macaque monkey. J. Neurosci. 14, 4109–4124 (1994).
Shamir, M. & Sompolinsky, H. Correlation codes in neuronal networks. in Advances in Neural Information Processing Systems (eds. Dietterich, T.G., Becker, S. & Ghahramani, Z.) (MIT Press, Cambridge, Massachusetts, 2002).
Yoon, H. & Sompolinsky, H. The effect of correlations on the fisher information of population codes. in Advances in Neural Information Processing Systems (eds. Kearns, M.S., Solla, S.A. & Cohn, D.A.) 167–173 (MIT Press, Cambridge, Massachusetts, 1998).
van Veen, V. & Carter, C.S. Error detection, correction, and prevention in the brain: a brief review of data and theories. Clin. EEG Neurosci. 37, 330–335 (2006).
Dosher, B.A. & Lu, Z.L. Mechanisms of perceptual learning. Vision Res. 39, 3197–3221 (1999).
Britten, K.H., Shadlen, M.N., Newsome, W.T. & Movshon, J.A. Responses of neurons in macaque MT to stochastic motion signals. Vis. Neurosci. 10, 1157–1169 (1993).
Acknowledgements
We thank L. Ding, M. Nassar, B. Heasley, R. Kalwani, C.-L. Teng, S. Bennur and M. Todd for helpful comments on this manuscript and J. Zweigle for expert technical assistance. This work was supported by the Sloan Foundation, the McKnight Foundation, the Burroughs-Wellcome Fund, and US National Institutes of Health grants R01-EY015260 and T32-EY007035.
Author information
Authors and Affiliations
Contributions
C.-T.L. and J.I.G. planned the study and wrote the manuscript together. C.-T.L. implemented the model.
Corresponding author
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–9, Supplementary Table 1 and Supplementary Methods (PDF 1122 kb)
Rights and permissions
About this article
Cite this article
Law, CT., Gold, J. Reinforcement learning can account for associative and perceptual learning on a visual-decision task. Nat Neurosci 12, 655–663 (2009). https://doi.org/10.1038/nn.2304
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.2304
This article is cited by
-
Distinct basal ganglia contributions to learning from implicit and explicit value signals in perceptual decision-making
Nature Communications (2024)
-
Trial-history biases in evidence accumulation can give rise to apparent lapses in decision-making
Nature Communications (2024)
-
Category representation in primary visual cortex after visual perceptual learning
Cognitive Neurodynamics (2024)
-
Neural ensembles in the murine medial prefrontal cortex process distinct information during visual perceptual learning
BMC Biology (2023)
-
Adaptive control of synaptic plasticity integrates micro- and macroscopic network function
Neuropsychopharmacology (2023)