Abstract
In many areas of the brain, neural populations act as a coordinated network whose state is tied to behavior on a millisecond timescale. Two-photon (2p) calcium imaging is a powerful tool to probe such network-scale phenomena. However, estimating the network state and dynamics from 2p measurements has proven challenging because of noise, inherent nonlinearities and limitations on temporal resolution. Here we describe Recurrent Autoencoder for Discovering Imaged Calcium Latents (RADICaL), a deep learning method to overcome these limitations at the population level. RADICaL extends methods that exploit dynamics in spiking activity for application to deconvolved calcium signals, whose statistics and temporal dynamics are quite distinct from electrophysiologically recorded spikes. It incorporates a new network training strategy that capitalizes on the timing of 2p sampling to recover network dynamics with high temporal precision. In synthetic tests, RADICaL infers the network state more accurately than previous methods, particularly for high-frequency components. In 2p recordings from sensorimotor areas in mice performing a forelimb reach task, RADICaL infers network state with close correspondence to single-trial variations in behavior and maintains high-quality inference even when neuronal populations are substantially reduced.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
Dataset mouse 2/M1 will be made available at the time of publication.
Code availability
RADICaL for Google Cloud Platform can be downloaded from GitHub at https://github.com/snel-repo/autolfads and the tutorial is available at https://snel-repo.github.io/autolfads/. RADICaL for NeuroCAAS64 is available at http://www.neurocaas.org/analysis/17. Source code for RADICaL is available at https://github.com/snel-repo/lfads-cd/tree/radical.
References
Stevenson, I. H. & Kording, K. P. How advances in neural recording affect data analysis. Nat. Neurosci. 14, 139–142 (2011).
Steinmetz, N. A. et al. Neuropixels 2.0: a miniaturized high-density probe for stable, long-term brain recordings. Science 372, eabf4588 (2021).
Demas, J. et al. High-speed, cortex-wide volumetric recording of neuroactivity at cellular resolution using light beads microscopy. Nat. Methods 18, 1103–1111 (2021).
Vyas, S., Golub, M. D., Sussillo, D. & Shenoy, K. V. Computation through neural population dynamics. Annu. Rev. Neurosci. 43, 249–275 (2020).
Wei, Z. et al. A comparison of neuronal population dynamics measured with calcium imaging and electrophysiology. PLoS Comput. Biol. 16, e1008198 (2020).
Siegle, J. H. et al. Reconciling functional differences in populations of neurons recorded with two-photon imaging and electrophysiology. eLife 10, e69068 (2021).
Chen, T.-W. et al. Ultrasensitive fluorescent proteins for imaging neuronal activity. Nature 499, 295–300 (2013).
Pachitariu, M. et al. Suite2p: beyond 10,000 neurons with standard two-photon microscopy. Preprint at bioRxiv https://doi.org/10.1101/061507 (2017).
Peron, S. P., Freeman, J., Iyer, V., Guo, C. & Svoboda, K. A cellular resolution map of barrel cortex activity during tactile behavior. Neuron 86, 783–799 (2015).
Chen, J. L., Carta, S., Soldado-Magraner, J., Schneider, B. L. & Helmchen, F. Behaviour-dependent recruitment of long-range projection neurons in somatosensory cortex. Nature 499, 336–340 (2013).
Chen, S. X., Kim, A. N., Peters, A. J. & Komiyama, T. Subtype-specific plasticity of inhibitory circuits in motor cortex during motor learning. Nat. Neurosci. 18, 1109–1115 (2015).
Pnevmatikakis, E. A. Analysis pipelines for calcium imaging data. Curr. Opin. Neurobiol. 55, 15–21 (2019).
Berens, P. et al. Community-based benchmarking improves spike rate inference from two-photon calcium imaging data. PLoS Comput. Biol. 14, e1006157 (2018).
Pachitariu, M., Stringer, C. & Harris, K. D. Robustness of spike deconvolution for neuronal calcium imaging. J. Neurosci. 38, 7976–7985 (2018).
Sussillo, D., Jozefowicz, R., Abbott, L. & Pandarinath, C. LFADS—latent factor analysis via dynamical systems. Preprint at arXiv https://doi.org/10.48550/arXiv.1608.06315 (2016).
Pandarinath, C. et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nat. Methods 15, 805–815 (2018).
Keshtkaran, M. R. et al. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nat. Methods 19 (2022).
Cunningham, J. P. & Yu, B. M. Dimensionality reduction for large-scale neural recordings. Nat. Neurosci. 17, 1500–1509 (2014).
Pandarinath, C. et al. Latent factors and dynamics in motor cortex and their application to brain–machine interfaces. J. Neurosci. 38, 9390–9401 (2018).
Shenoy, K. V., Sahani, M. & Churchland, M. M. Cortical control of arm movements: a dynamical systems perspective. Annu. Rev. Neurosci. 36, 337–359 (2013).
Keshtkaran, M. R. & Pandarinath, C. Enabling hyperparameter optimization in sequential autoencoders for spiking neural data. Adv. Neural Inf. Proc. Sys. 32 (2018).
Wei, X.-X. et al. A zero-inflated gamma model for post-deconvolved calcium imaging traces. Neurons, Behav. Data Anal. Theory 3 (2020).
Zhu, F. et al. Deep inference of latent dynamics with spatio-temporal super-resolution using selective backpropagation through time. Adv. Neural Inf. Proc. Sys. 34 (2021).
Zhao, Y. & Park, I. M. Variational latent gaussian process for recovering single-trial dynamics from population spike trains. Neural Comput. 29, 1293–1316 (2017).
Friedrich, J., Zhou, P. & Paninski, L. Fast online deconvolution of calcium imaging data. PLoS Comput. Biol. 13, e1005423 (2017).
Giovannucci, A. et al. CaImAn an open source tool for scalable calcium imaging data analysis. eLife 8, e38173 (2019).
Deneux, T. et al. Accurate spike estimation from noisy calcium signals for ultrafast three-dimensional imaging of large neuronal populations in vivo. Nat. Commun. 7, 12190 (2016).
Galiñanes, G. L., Bonardi, C. & Huber, D. Directional reaching for water as a cortex-dependent behavioral framework for mice. Cell Rep. 22, 2767–2783 (2018).
Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281–1289 (2018).
Kaufman, M.T. et al. The largest response component in the motor cortex reflects movement timing but not movement type. eNeuro 3, 0085-16.2016 (2016).
Hatsopoulos, N. G., Xu, Q. & Amit, Y. Encoding of movement fragments in the motor cortex. J. Neurosci. 27, 5105–5114 (2007).
Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A. & Poeppel, D. Neuroscience needs behavior: correcting a reductionist bias. Neuron 93, 480–490 (2017).
Whishaw, I. Q. et al. Organization of the reach and grasp in head-fixed vs freely-moving mice provides support for multiple motor channel theory of neocortical organization. Exp. Brain Res. 235, 1919–1932 (2017).
Wiltschko, A. B. et al. Revealing the structure of pharmacobehavioral space through motion sequencing. Nat. Neurosci. 23, 1433–1443 (2020).
Herzfeld, D. J., Kojima, Y., Soetedjo, R. & Shadmehr, R. Encoding of error and learning to correct that error by the Purkinje cells of the cerebellum. Nat. Neurosci. 21, 736–743 (2018).
Vyas, S., O’Shea, D. J., Ryu, S. I. & Shenoy, K. V. Causal role of motor preparation during error-driven learning. Neuron 106, 329–339 (2020).
Steinmetz, N. A., Zatka-Haas, P., Carandini, M. & Harris, K. D. Distributed coding of choice, action and engagement across the mouse brain. Nature 576, 266–273 (2019).
Stringer, C. et al. Spontaneous behaviors drive multidimensional, brainwide activity. Science 364, 255 (2019).
Musall, S., Kaufman, M. T., Juavinett, A. L., Gluf, S. & Churchland, A. K. Single-trial neural dynamics are dominated by richly varied movements. Nat. Neurosci. 22, 1677–1686 (2019).
Hoang, H. et al. Improved hyperacuity estimation of spike timing from calcium imaging. Sci. Rep. 10, 17844 (2020).
Rupprecht, P. et al. A database and deep learning toolbox for noise-optimized, generalized spike inference from calcium imaging. Nat. Neurosci. 24, 1324–1337 (2021).
Sebastian, J., Sur, M., Murthy, H. A. & Magimai-Doss, M. Signal-to-signal neural networks for improved spike estimation from calcium imaging data. PLoS Comput. Biol. 17, e1007921 (2021).
Dechery, J. B. & MacLean, J. N. Functional triplet motifs underlie accurate predictions of single-trial responses in populations of tuned and untuned V1 neurons. PLoS Comput. Biol. 14, e1006153 (2018).
Kirschbaum, E. et al. LeMoNADe: learned motif and neuronal assembly detection in calcium imaging videos. International Conference on Learning Representations (2019).
Mackevicius, E. L. et al. Unsupervised discovery of temporal sequences in high-dimensional datasets, with applications to neuroscience. eLife 8, e38471 (2019).
Triplett, M. A., Pujic, Z., Sun, B., Avitan, L. & Goodhill, G. J. Model-based decoupling of evoked and spontaneous neural activity in calcium imaging data. PLoS Comput. Biol. 16, e1008330 (2020).
Williams, A. H. et al. Unsupervised discovery of demixed, low-dimensional neural dynamics across multiple timescales through tensor component analysis. Neuron 98, 1099–1115 (2018).
Wu, A. et al. Learning a latent manifold of odor representations from neural responses in piriform cortex. Adv. Neural Inf. Proc. Sys. 31 (2018).
Costa, A. C., Ahamed, T. & Stephens, G. J. Adaptive, locally linear models of complex dynamics. Proc. Natl Acad. Sci. USA 116, 1501–1510 (2019).
Glaser, J., Whiteway, M., Cunningham, J. P., Paninski, L. & Linderman, S. Recurrent Switching Dynamical Systems Models for Multiple Interacting Neural Populations. Adv. Neural Inf. Proc. Sys. 33 (2020).
Picardo, M. A. et al. Population-level representation of a temporal sequence underlying song production in the Zebra finch. Neuron 90, 866–876 (2016).
Mano, O. et al. Using slow frame rate imaging to extract fast receptive fields. Nat. Commun. 10, 4979 (2019).
Prince, L. Y., Bakhtiari, S., Gillon, C. J. & Richards, B. A. Parallel inference of hierarchical latent dynamics in two-photon calcium imaging of neuronal populations. Preprint at bioRxiv https://doi.org/10.1101/2021.03.05.434105 (2021).
Pei, F. et al. Neural Latents Benchmark '21: evaluating latent variable models of neural population activity. In Proceedings of the Neural Information Processing Systems 34, Track on Datasets and Benchmarks 1, (NeurIPS, 2021).
Minderer, M., Brown, K. D. & Harvey, C. D. The spatial structure of neural encoding in mouse posterior cortex during navigation. Neuron 102, 232–248 (2019).
Sofroniew, N. J., Flickinger, D., King, J. & Svoboda, K. A large field of view two-photon mesoscope with subcellular resolution for in vivo imaging. eLife 5, e14472 (2016).
Jaderberg, M. et al. Population based training of neural networks. Preprint at arXiv https://doi.org/10.48550/arXiv.1711.09846 (2017).
Dana, H. et al. High-performance calcium sensors for imaging activity in neuronal populations and microcompartments. Nat. Methods 16, 649–657 (2019).
Art, J. Handbook of Biological Confocal Microscopy (Springer, 2006).
Starck, J.-L., Murtagh, F. D. & Bijaoui, A. Image Processing and Data Analysis: The Multiscale Approach (Cambridge University Press, 1998).
Vogelstein, J. T. et al. Fast nonnegative deconvolution for spike train inference from population calcium imaging. J. Neurophysiol. 104, 3691–3704 (2010).
Heikkila, J. & Silvén, O. A four-step camera calibration procedure with implicit image correction. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition 1106–1112 (IEEE, 1997).
Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1330–1334 (2000).
Abe, T. et al. Neuroscience Cloud Analysis As a Service: An open-source platform for scalable, reproducible data analysis. Neuron 110, 2771–2789 (2022).
Acknowledgements
We thank M. Rivers and R. Vescovi for help with the high-speed camera setup, D. Sabatini for contributions to the behavioral control software, and T. Abe and A. Mosberger for help adapting RADICaL for NeuroCAAS. This work was supported by the Emory Neuromodulation and Technology Innovation Center (ENTICe), NSF NCS 1835364, NIH Eunice Kennedy Shriver NICHD K12HD073945, the Simons Foundation as part of the Simons-Emory International Consortium on Motor Control, NIH NINDS/OD DP2 NS127291, NIH BRAIN/NIDA RF1 DA055667 (C.P.), the Alfred P. Sloan Foundation (C.P. and M.K.), NSF NCS 1835390, The University of Chicago, the Neuroscience Institute at The University of Chicago (M.K.), NIH NINDS R01 NS121535 (M.K.) and a Beckman Young Investigators Award (A.G.). The work was also supported by the following collaborative awards (PI: Ellen Hess, Emory): NIH NINDS R21 NS116311, Imagine, Innovate and Impact (I3) Funds from the Emory School of Medicine and through the Georgia CTSA NIH UL1-TR002378, and a pilot grant from the Emory Udall Center of Excellence for Parkinson’s Research.
Author information
Authors and Affiliations
Contributions
F.Z. and C.P. designed the study, with input from A.G. and M.K. C.P. and M.K. conceptualized the SBTT approach. F.Z. and C.P. performed analyses and wrote the manuscript with input from all other authors. F.Z. and C.P. developed the algorithmic approach. F.Z., C.C. and A.G. developed the simulation pipeline. H.G. and M.K. designed and performed experiments with mice and developed the real data preprocessing pipeline with input from F.Z. and C.P. R.T. contributed to initial simulations and data analysis. F.Z., A.G., M.K. and C.P. edited and revised the manuscript with input from all other authors. F.Z. and A.A. adapted RADICaL for Google Cloud Platform and NeuroCAAS.
Corresponding authors
Ethics declarations
Competing interests
One of the innovations detailed in the manuscript, Selective Backpropagation Through Time (SBTT), is covered in a provisional patent: C.P., M.K., F.Z., A.G., A.S. Selective backpropagation through time. US patent application number 63/262.704, filed as provisional patent. C.P. is a consultant to Synchron and Meta (Reality Labs). These entities did not support this work, have a role in the study or have any financial interests related to this work.
Peer review
Peer review information
Nature Neuroscience thanks the anonymous reviewers for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Simulation of Lorenz system at different speeds.
This figure illustrates the underlying dynamical system used for the simulation experiments. (a) An example Lorenz trajectory in a 3-dimensional state space (far left) and with three dynamic variables plotted as a function of time (middle left) for a system with Z-oscillation peak frequency of 7 Hz (that is, the power spectrum of the Lorenz system’s Z-dimension had a pronounced peak at 7 Hz). Firing rates for the simulated neurons were computed by a linear readout of the Lorenz variables followed by an exponential nonlinearity (middle right). Spikes from the firing rates were then generated by a Poisson process (far right). The example trial shown here is identical to ‘Trial 2’ in Fig. 2a, but with a wider plotting window. (b) Power spectrum of the individual Lorenz variables for the system with a Z-oscillation peak frequency at 7 Hz. Because only the Z variable has a clear peak in the power spectrum, this variable was used exclusively for all further analyses in simulations except Supplementary Fig. 1. (c) Power spectrum of the Z dimension for Lorenz systems simulated with different Z-oscillation peak frequencies.
Extended Data Fig. 2 Simulation pipeline to generate artificial fluorescence traces from the underlying Lorenz system.
(a) This pipeline begins from the Poisson-random spikes generated in the far-right panel of Extended Data Fig. 1. Calcium traces were generated by first corrupting the spikes with amplitude noise, then modeling the dynamics of calcium indicators in response to a spike with an autoregressive process of order 2 transformed by a piecewise-linear non-linearity. Sources of noise corrupting this fluorescence trace were then added. The nonlinearity and noise sources were chosen to approximate the variability observed in real data. (b) Example ground truth and simulated data using a GCaMP6f model. From top to bottom: original ground truth spikes fed into the simulator, perturbed spikes, idealized calcium trace, fluorescence trace with nonlinearity and noise sources added, fluorescence trace after subsampling, deconvolved spikes, and finally original ground truth spikes fed into the simulator (shown again for comparison; same as top). (c) Estimated nonlinearities for GCaMP6f from ref. 58. (d) Example traces generated by the simulator for a train of 10 Hz stimuli, with and without nonlinearity applied.
Extended Data Fig. 3 RADICaL retains high latent recovery performance in a simulation experiment that lacks stereotyped conditions.
This analysis was targeted at determining whether RADICaL simply ‘memorized’ the stereotyped trajectories for a limited number of conditions, or whether it could generalize to cases where each trial was more unique. To answer this question, we designed a ‘zero condition’ simulation experiment, where each trial had its own unique Lorenz initial state and there were no repeated trials with the same underlying latent trajectories. (a) Example true (top left) and estimated Lorenz trajectories by RADICaL (top right), AutoLFADS (bottom left), and smth-dec (bottom right). Each trajectory is an individual trial, colored by the location of the initial state of the true Lorenz trajectory. The initial states of the trials are indicated by the dots in the same colors as the trajectories. (b) Performance in estimating the Lorenz Z dimension as a function of Lorenz oscillation frequency was quantified by variance explained (R2) for all 4 methods.
Extended Data Fig. 4 RADICaL retains high latent recovery performance at slower imaging speeds, but there are limits to deconvolution with slower sampling.
To understand the extent to which the model performance depends on imaging speeds, we simulated data at different sampling rates ranging from 2 Hz to 33.3 Hz. (a) Example ground truth spikes, simulated fluorescence, and deconvolved signals at different sampling rates. Sample times are denoted by gray triangles. Deconvolution performance degraded at slower sampling rates, particularly in regimes when transients could be missed entirely. In our simulation we used a GCaMP6f model with a decay time of 400 ms (see Methods). At an imaging rate of 2 Hz, the majority of transients were missed and the estimate of the decay time constant tau was inaccurate (916.8 +/− 49.4 ms, compared to the ground truth 400 ms). Because deconvolution performs poorly at these sampling rates (that is, <= 2 Hz) with fast indicators, we do not recommend using RADICaL under such circumstances. (b) Performance in estimating the Lorenz Z dimension as a function of sampling rate was quantified by variance explained (R2) for all 3 methods, for Lorenz oscillation frequencies of 10 Hz (top) and 15 Hz (bottom). Squares with solid lines denote experiments with 278 neurons. Triangles with dashed lines denote experiments with 500 neurons. RADICaL retained high performance and outperformed AutoLFADS and smth-dec in recovering the latent states of a 10 Hz Lorenz system at moderately slow sampling rates (8 and16 Hz; top). In real experiments, there may be benefits to slower sampling, for example, one can image more neurons using a larger FOV. Increasing the number of neurons boosted RADICaL’s performance, while AutoLFADS and smth-dec showed negligible improvement (bottom).
Extended Data Fig. 5 Performance of RADICaL and AutoLFADS in capturing the empirical PSTHs on single trials in the mouse water grab experiments.
This figure is related to Fig. 3d, but compares RADICaL with AutoLFADS instead of smth-dec. Correlation coefficient r was computed between the inferred single-trial event rates and empirical PSTHs. Each point represents an individual neuron. These results demonstrate that RADICaL captures the key features of individual neurons’ responses from single-trial activity better than AutoLFADS in nearly every case.
Extended Data Fig. 6 Single-trial neural trajectories for additional mouse water grab experiments.
This figure is related to Fig. 4a, and shows the remaining datasets. Single-trial, log-transformed event rates were projected into a subspace computed by applying PCA to the trial-averaged, log-transformed rates, colored by subgroups. Lift onset times are indicated by the dots in the same colors as the trajectories. Gray dots indicate 200 ms prior to lift onset time. Top row: single-trial neural trajectories derived from RADICaL rates; Bottom row: single-trial neural trajectories derived from smth-dec rates.
Extended Data Fig. 7 Hand trajectories for additional mouse water grab experiments.
This figure is related to Fig. 5a, and shows the remaining datasets. True and decoded hand positions for Mouse1/S1 (left) and Mouse2/M1 (right).
Extended Data Fig. 8 Prediction of single-trial reaction times for additional mouse water grab experiments.
This figure is like Fig. 5d, for the remaining datasets. Each dot represents an individual trial, color-coded by the technique. Correlation coefficient r was computed between the true and predicted reaction times. Data from Mouse2/M1 (left) and Mouse2/S1 (right).
Extended Data Fig. 9 RADICaL retains high decoding performance in an FOV-shrinking experiment.
This is an alternative method for evaluating performance with reduced neuron counts to the method in Fig. 6. (a) The area selected to include was gradually shrunk to the center of the FOV to reduce the number of neurons included in training RADICaL or AutoLFADS. (b) Decoding performance measured using variance explained (R2) as a function of the number of neurons used in each technique (top: Position; bottom: Velocity). Error bar indicates the s.e.m. across 5 folds of test trials. Data from Mouse2/M1.
Supplementary information
Supplementary Information
Supplementary Figs. 1–11.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhu, F., Grier, H.A., Tandon, R. et al. A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution. Nat Neurosci 25, 1724–1734 (2022). https://doi.org/10.1038/s41593-022-01189-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41593-022-01189-0
This article is cited by
-
FIOLA: an accelerated pipeline for fluorescence imaging online analysis
Nature Methods (2023)
-
Dimensionality reduction of calcium-imaged neuronal population activity
Nature Computational Science (2022)