Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Brief Communication
  • Published:

A large-scale neural network training framework for generalized estimation of single-trial population dynamics

Nature Methods volume 19, pages 1572–1577 (2022)Cite this article

Subjects

Abstract

Achieving state-of-the-art performance with deep neural population dynamics models requires extensive hyperparameter tuning for each dataset. AutoLFADS is a model-tuning framework that automatically produces high-performing autoencoding models on data from a variety of brain areas and tasks, without behavioral or task information. We demonstrate its broad applicability on several rhesus macaque datasets: from motor cortex during free-paced reaching, somatosensory cortex during reaching with perturbations, and dorsomedial frontal cortex during a cognitive timing task.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Combining a regularization technique with a large-scale framework for automated hyperparameter optimization.
Fig. 2: Applying AutoLFADS to four diverse datasets.

Similar content being viewed by others

Data availability

Highly similar maze, random target, area 2 and DMFC datasets have been made publicly available in a standardized format through the NLB13, although those are different recording sessions from the specific sessions used in this paper. We encourage readers to use the NLB sessions to reproduce results and compare to available benchmark numbers. The specific maze and random target sessions used in this paper are already available at https://dandiarchive.org/dandiset/000070/ and https://doi.org/10.5281/zenodo.3854034, respectively. We have not made the specific area 2 and DMFC sessions available because they are not standardized, but we can make them available upon reasonable request.

Code availability

The code and tutorial accompanying this paper can be found at https://snel-repo.github.io/autolfads/ (MIT license). An implementation of the original LFADS model used for random searches and manual tuning can be found at https://github.com/tensorflow/models/tree/master/research/lfads/.

References

  1. Jun, J. J. et al. Fully integrated silicon probes for high-density recording of neural activity. Nature 551, 232–236 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Stevenson, I. H. & Kording, K. P. How advances in neural recording affect data analysis. Nat. Neurosci. 14, 139–142 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Stringer, C., Pachitariu, M., Steinmetz, N., Carandini, M. & Harris, K. D. High-dimensional geometry of population responses in visual cortex. Nature 571, 361–365 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Berger, M., Agha, N. S. & Gail, A. Wireless recording from unrestrained monkeys reveals motor goal encoding beyond immediate reach in frontoparietal cortex. eLife 9, e51322 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Steinmetz, N. A. et al. Neuropixels 2.0: a miniaturized high-density probe for stable, long-term brain recordings. Science 372, eabf4588. (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  6. Hernandez, D. et al. Nonlinear evolution via spatially dependent linear dynamics for electrophysiology and calcium data. NBDT 3, (2020).

  7. Koppe, G., Toutounji, H., Kirsch, P., Lis, S. & Durstewitz, D. Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI. PLoS Comput. Biol. 15, e1007263 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Pandarinath, C. et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nat. Methods 15, 805–15. (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. She, Q. & Wu, A. Neural dynamics discovery via Gaussian process recurrent neural networks. In Proc. of The 35th Uncertainty in Artificial Intelligence Conference, PMLR 115, 454–464 (2020).

    Google Scholar 

  10. Sussillo, D., Jozefowicz, R., Abbott, L. F. & Pandarinath, C. LFADS—latent factor analysis via dynamical systems. Preprint at http://arxiv.org/abs/1608.06315 (2020).

  11. Gao, Y., Archer, E. W., Paninski, L. & Cunningham, J. P. Linear dynamical neural population models through nonlinear embeddings. In Proc. 30th Conference on Neural Information Processing Systems (NIPS 2016).

  12. Keshtkaran, M. R. & Pandarinath, C. Enabling hyperparameter optimization in sequential autoencoders for spiking neural data. In Proc. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).

  13. Pei, F. et al. Neural Latents Benchmark ’21: evaluating latent variable models of neural population activity. In Proc. 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks (2021).

  14. Jaderberg M, et al. Population-based training of neural networks. Preprint at https://doi.org/10.48550/arXiv.1711.09846 (2017).

  15. Jaderberg, M. et al. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science 364, 859–65. (2019).

    Article  CAS  PubMed  Google Scholar 

  16. Kaufman, M.T. et al. The largest response component in the motor cortex reflects movement timing but not movement type. eNeuro https://www.eneuro.org/content/3/4/ENEURO.0085-16.2016 (2016).

  17. Gao, P. et al. A theory of multineuronal dimensionality, dynamics and measurement. Preprint at bioRxiv https://doi.org/10.1101/214262 (2017).

  18. O’Doherty J. E., Cardoso, M. M. B., Makin, J. G. & Sabes, P.N. Nonhuman primate reaching with multichannel sensorimotor cortex electrophysiology. Zenodo https://zenodo.org/record/3854034#.Xz_iqpNKhuU (2020).

  19. Chowdhury, R. H., Glaser, J. I. & Miller, L. E. Area 2 of primary somatosensory cortex encodes kinematics of the whole arm. eLife 9, e48198 (2020).

  20. Kaas, J. H., Nelson, R. J., Sur, M., Lin, C. S. & Merzenich, M. M. Multiple representations of the body within the primary somatosensory cortex of primates. Science 204, 521–523 (1979).

  21. Jennings, V. A., Lamour, Y., Solis, H. & Fromm, C. Somatosensory cortex activity related to position and force. J. Neurophysiol. 49, 1216–1229 (1983).

    Article  CAS  PubMed  Google Scholar 

  22. Nelson, R. J. Activity of monkey primary somatosensory cortical neurons changes prior to active movement. Brain Res. 406, 402–407 (1987).

    Article  CAS  PubMed  Google Scholar 

  23. Padberg, J., Cooke, D. F., Cerkevich, C. M., Kaas, J. H. & Krubitzer, L. Cortical connections of area 2 and posterior parietal area 5 in macaque monkeys. J. Comp. Neurol. 527, 718–737 (2019).

    Article  PubMed  Google Scholar 

  24. Prud’homme, M. J. & Kalaska, J. F. Proprioceptive activity in primate primary somatosensory cortex during active arm reaching movements. J. Neurophysiol. 72, 2280–2301 (1994).

    Article  PubMed  Google Scholar 

  25. London, B. M. & Miller, L. E. Responses of somatosensory area 2 neurons to actively and passively generated limb movements. J. Neurophysiol. 109, 1505–1513 (2013).

    Article  CAS  PubMed  Google Scholar 

  26. Yu, B. M. et al. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. J. Neurophysiol. 102, 614–635 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  27. Sohn, H., Narain, D., Meirhaeghe, N. & Jazayeri, M. Bayesian computation through cortical latent dynamics. Neuron 103, 934–947 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Remington, E. D., Narain, D., Hosseini, E. A. & Jazayeri, M. Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics. Neuron 98, 1005–1019 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Wang, J., Narain, D., Hosseini, E. A. & Jazayeri, M. Flexible timing by temporal scaling of cortical responses. Nat. Neurosci. 21, 102–110. (2018).

    Article  CAS  PubMed  Google Scholar 

  30. Patel, A., Sedler, A., Huang, J., Pandarinath, C. & Gilja, V. Deployment strategies for scaling AutoLFADS to model neural population dynamics. Zenodo https://zenodo.org/record/6786931 (2022).

  31. Zhu, F. et al. Deep inference of latent dynamics with spatio-temporal super-resolution using selective backpropagation through time. In Proc. 35th Conference on Neural Information Processing Systems (NeurIPS 2021).

  32. Zhu F, et al. A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with sub-frame temporal resolution. Preprint at bioRxiv https://doi.org/10.1101/2021.11.21.469441 (2002).

  33. Wimalasena, L. N. et al. Estimating muscle activation from EMG using deep learning-based dynamical systems models. J. Neural Eng. 19, 036013 (2022).

    Article  Google Scholar 

  34. Perich, M. G., Gallego, J. A. & Miller, L. E. A neural population mechanism for rapid learning. Neuron 100, 964–976 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Kobak, D. et al. Demixed principal component analysis of neural population data. eLife 5, e10989 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank K. Shenoy, M. Churchland, M. Kaufman and S. Ryu (Stanford) for sharing the Monkey J Maze dataset. We also thank J. O’Doherty, M. Cardoso, J. Makin and P. Sabes (UCSF) for making the random target dataset publicly available. This work was supported by the Emory Neuromodulation and Technology Innovation Center (ENTICe), NSF NCS 1835364, DARPA PA-18-02-04-INI-FP-021, National Institutes of Health (NIH) Eunice Kennedy Shriver NICHD K12HD073945, NIH NINDS/OD DP2 NS127291, NIH BRAIN/NIDA RF1 DA055667, the Sloan Foundation, the Burroughs Wellcome Fund and the Simons Foundation as part of the Simons-Emory International Consortium on Motor Control (to C.P.); NIH NINDS R01 NS053603, R01 NS095251 and NSF NCS 1835345 (to L.E.M.); NSF Graduate Research Fellowships DGE-2039655 (to A.R.S.) and DGE-1324585 (to R.H.C.); the Center for Sensorimotor Neural Engineering and NARSAD Young Investigator grant from the Brain & Behavior Research Foundation (to H.S.); NIH NINDS NS078127, the Sloan Foundation, the Klingenstein Foundation, the Simons Foundation (542993SPI), the McKnight-Endowment Fund for Neuroscience, the National Science Foundation Center for Sensorimotor Neural Engineering (UWSC6200 BPO4405) and the McGovern Institute (to M.J.).

Author information

Author notes

  1. These authors contributed equally: Mohammad Reza Keshtkaran, Andrew R. Sedler.

Authors and Affiliations

  1. Wallace H. Coulter Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA, USA

    Mohammad Reza Keshtkaran, Andrew R. Sedler, Raghav Tandon, Diya Basrai & Chethan Pandarinath

  2. Center for Machine Learning, Georgia Institute of Technology, Atlanta, GA, USA

    Andrew R. Sedler, Raghav Tandon & Chethan Pandarinath

  3. Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA

    Raeed H. Chowdhury & Lee E. Miller

  4. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA

    Raeed H. Chowdhury

  5. Physiology and Neuroscience, University of California, San Diego, La Jolla, CA, USA

    Diya Basrai

  6. College of Computing, Georgia Institute of Technology, Atlanta, GA, USA

    Sarah L. Nguyen

  7. Department of Brain and Cognitive Sciences, McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA

    Hansem Sohn & Mehrdad Jazayeri

  8. Department of Neuroscience, Northwestern University, Chicago, IL, USA

    Lee E. Miller

  9. Department of Physical Medicine and Rehabilitation, Northwestern University, Chicago, IL, USA

    Lee E. Miller

  10. Shirley Ryan AbilityLab, Chicago, IL, USA

    Lee E. Miller

  11. Department of Neurosurgery, Emory University, Atlanta, GA, USA

    Chethan Pandarinath

Authors
  1. Mohammad Reza Keshtkaran
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew R. Sedler
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Raeed H. Chowdhury
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raghav Tandon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Diya Basrai
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sarah L. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hansem Sohn
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Mehrdad Jazayeri
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Lee E. Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Chethan Pandarinath
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization was performed by M.R.K., A.R.S. and C.P. Funding acquisition was performed by L.E.M. and C.P. Investigation was conducted by M.R.K., A.R.S., R.H.C., H.S. and C.P. Resources were provided by R.H.C., H.S., M.J., L.E.M. and C.P. Software was written by M.R.K., A.R.S., R.H.C., R.T., S.L.N. and C.P. Tutorials were created by R.T. and D.B. Writing was completed by M.R.K., A.R.S. and C.P. Revision was done by M.R.K., A.R.S., R.H.C., H.S., M.J., L.E.M. and C.P.

Corresponding author

Correspondence to Chethan Pandarinath.

Ethics declarations

Competing interests

C.P. is a consultant to Synchron and Meta (Reality Labs). These entities did not support this work, have a role in the study or have any competing interests related to this work. The remaining authors declare no competing interests.

Peer review

Peer review information

Nature Methods thanks Anqi Wu and the other, anonymous, reviewers for their contribution to the peer review of this work. Peer reviewer reports are available. Primary Handling editor: Nina Vogt, in collaboration with the Nature Methods team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Training AutoLFADS models with Population-Based Training.

(a) Schematic of the PBT approach to HP optimization. Each colored circle represents an LFADS model with a certain HP configuration and partially filled bars represent model performance (higher is better). In our case, performance is measured by exponentially smoothed validation log-likelihood at the end of each generation. Models are trained for fixed intervals (generations), between which poorly performing models are replaced by copies of better-performing models with perturbed HPs. (b) True rate recovery performance of AutoLFADS vs. best random search model (no CD) for a given number of workers. We did not run AutoLFADS with more than 20 workers. Instead, we extrapolate with a dashed line for comparison. Random searches were simulated by drawing from the pool of runs shown in Fig. 1c. Center line denotes median and shaded regions denote upper and lower quartiles for 100 draws. (c) Hyperparameter progressions for the 20-worker AutoLFADS run shown in the previous panel. Initialization values are shown as gray points and initialization ranges are shown as gray lines.

Extended Data Fig. 2 Single-trial and PSTH recovery in diverse brain areas.

Results are shown for M1 (a, b), area 2 (c, d) and DMFC (e, f). (a) Average reach trajectories (top), PSTHs (second row) and single-trial firing rates (bottom) obtained by smoothing (Gaussian kernel, 30 ms s.d.) or AutoLFADS for a single neuron across 4 reach conditions. Data is modeled at 2 ms bins. Dashed lines indicate movement onset and vertical scale bars denote rates (spikes per second). (b) Performance in replicating the empirical PSTHs computed on all trials using rates inferred from a 184-trial training set using AutoLFADS and LFADS with random HPs (100 models; no CD). (c) PSTHs and single-trial firing rates for a single neuron across 4 passive perturbation directions. Smoothing was performed using a Gaussian kernel with 10 ms s.d. Dashed lines indicate movement onset. (d) Comparison of AutoLFADS vs. random search (no CD) in matching empirical PSTHs. (e) PSTHs and single-trial firing rates for an example neuron during the ’set-go' period of leftward saccade trials across 4 different values of ts (vertical scale bar: spikes per second). Smoothing was performed using a Gaussian kernel with 25 ms s.d. (f) Performance in replicating the empirical PSTHs.

Extended Data Fig. 3 Further characterization of maze modeling.

(a) Brain area and task schematics for the maze dataset, MM. (b) Example PSTHs, aligned to movement onset (dashed line). Colors indicate reach conditions, shaded regions denote standard error, and vertical scale bars denote rates (spikes per second). (c) Hand velocity decoding performance from a 184-trial training subset of the maze dataset in comparison to smoothing, GPFA, and LFADS R.S. (100 models; no CD) baselines.

Extended Data Fig. 4 Further characterization of random target modeling.

(a) Brain area and task schematics for the random target dataset, MR. (b) Schematic of the random target task (top), revealing the unstereotyped trial structure. Continuous neural spiking data was divided into overlapping segments for modeling (bottom). After modeling, the inferred rates were merged using a weighted average of data at overlapping time points. (c) Hand velocity decoding performance for AutoLFADS in the random target task in comparison to smoothing, GPFA, and LFADS R.S. (random search) baselines.

Extended Data Fig. 5 Further characterization of area 2 dataset modeling.

(a) Brain area and task schematics for the area 2 dataset, A. (b) Comparison of spike count predictive performance for AutoLFADS and GLMs. Filled circles correspond to neurons for which AutoLFADS pR2 was significantly higher than GLM pR2 (P < 0.05, bootstrap), and open circles correspond to neurons for which there was no significant difference. Arrows (left) indicate neurons for which GLM pR2 was outside of the plot bounds. (c) PSTHs produced by smoothing spikes (top), AutoLFADS (middle), or GLM predictions (bottom) for 2 example neurons for the area 2 dataset. (d) Subspace representations of hand x-velocity during active and passive movements extracted from smoothed spikes and rates inferred by AutoLFADS for the area 2 dataset. (e) Hand velocity decoding performance for AutoLFADS during active trials of the area 2 dataset in comparison to smoothing, GPFA, and LFADS R.S. (random search) baselines.

Extended Data Fig. 6 Further characterization of DMFC dataset modeling.

(a) Brain area and task schematics for the DMFC dataset, D. (b) PSTHs for three additional example neurons during the ‘set-go’ period of rightward trials for two response modalities and all values of ts. Vertical scale bars denote spikes per second. (c) Example plots showing correlations between neural speed and behavior (that is, production time, tp) for individual trials across two timing intervals (red: 640 ms blue: 1000 ms). Neural speed was obtained based on the firing rates inferred from smoothing, GPFA, the LFADS model (no CD) with best median speed-tp correlation across the 40 different task conditions (Best LFADS), and AutoLFADS. (j) Distributions of correlation coefficients across 40 different task conditions. Horizontal lines denote medians. For LFADS, the distribution includes correlation values for all 96 models with random HPs (40 × 96 values). AutoLFADS performed better than all unsupervised approaches (P = 1.14 × 10−7, 1.78 × 10−8, and 3.53 × 10−7 for smoothing, PCA, and GPFA respectively, one-sided Wilcoxon signed rank test), and was comparable to the supervised approach (Best LFADS; P = 0.379, one-sided Wilcoxon signed rank test).

Supplementary information

Supplementary Information

Supplementary Tables 1–5, Notes 1–4 and Figs. 1–3

Reporting Summary

Peer Review File

Supplementary Video 1

Projection of four-dimensional inputs, from −100 ms to 200 ms around movement onset, into the top three PCs across all conditions, with separate plots for each movement direction. Darker lines indicate active trials, while lighter lines denote passive trials. Large dots indicate average initial input in PC space. Thick lines indicate average input trace for each direction, denoted by color, while thin colored lines show input traces for ten randomly chosen trials. Vertical scale bar shows arbitrary units.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Keshtkaran, M.R., Sedler, A.R., Chowdhury, R.H. et al. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nat Methods 19, 1572–1577 (2022). https://doi.org/10.1038/s41592-022-01675-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41592-022-01675-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing