Abstract
Carrying out any everyday task, be it driving in traffic, conversing with friends or playing basketball, requires rapid selection, integration and segregation of stimuli from different sensory modalities. At present, even the most advanced artificial intelligence-based systems are unable to replicate the multisensory processes that the human brain routinely performs, but how neural circuits in the brain carry out these processes is still not well understood. In this Perspective, we discuss recent findings that shed fresh light on the oscillatory neural mechanisms that mediate multisensory integration (MI), including power modulations, phase resetting, phase–amplitude coupling and dynamic functional connectivity. We then consider studies that also suggest multi-timescale dynamics in intrinsic ongoing neural activity and during stimulus-driven bottom–up and cognitive top–down neural network processing in the context of MI. We propose a new concept of MI that emphasizes the critical role of neural dynamics at multiple timescales within and across brain networks, enabling the simultaneous integration, segregation, hierarchical structuring and selection of information in different time windows. To highlight predictions from our multi-timescale concept of MI, real-world scenarios in which multi-timescale processes may coordinate MI in a flexible and adaptive manner are considered.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$189.00 per year
only $15.75 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41583-024-00845-7/MediaObjects/41583_2024_845_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41583-024-00845-7/MediaObjects/41583_2024_845_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41583-024-00845-7/MediaObjects/41583_2024_845_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41583-024-00845-7/MediaObjects/41583_2024_845_Fig4_HTML.png)
Similar content being viewed by others
References
Auvray, M. & Spence, C. The multisensory perception of flavor. Conscious. Cogn. 17, 1016–1031 (2008).
Stein, B. E. & Meredith, M. A. The Merging of the Senses (MIT Press, 1993).
Stein, B. E. & Stanford, T. R. Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9, 255–266 (2008).
Welch, R. B. & Warren, D. H. Immediate perceptual response to intersensory discrepancy. Psychol. Bull. 88, 638–667 (1980).
Talsma, D., Senkowski, D., Soto-Faraco, S. & Woldorff, M. G. The multifaceted interplay between attention and multisensory integration. Trends Cogn. Sci. 14, 400–410 (2010).
Spence, C. & Squire, S. Multisensory integration: maintaining the perception of synchrony. Curr. Biol. 13, R519–R521 (2003).
Panzeri, S., Brunel, N., Logothetis, N. K. & Kayser, C. Sensory neural codes using multiplexed temporal scales. Trends Neurosci. 33, 111–120 (2010).
Buzsáki, G. & Draguhn, A. Neuronal oscillations in cortical networks. Science 304, 1926–1929 (2004).
Singer, W. & Gray, C. M. Visual feature integration and the temporal correlation hypothesis. Annu. Rev. Neurosci. 18, 555–586 (1995).
Singer, W. Neuronal synchrony: a versatile code for the definition of relations? Neuron 24, 49–65 (1999).
Engel, A. K., Fries, P. & Singer, W. Dynamic predictions: oscillations and synchrony in top-down processing. Nat. Rev. Neurosci. 2, 704–716 (2001).
Fries, P., Nikolić, D. & Singer, W. The gamma cycle. Trends Neurosci. 30, 309–316 (2007).
Senkowski, D., Schneider, T. R., Foxe, J. J. & Engel, A. K. Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci. 31, 401–409 (2008).
Engel, A. K. & Fries, P. Beta-band oscillations — signalling the status quo? Curr. Opin. Neurobiol. 20, 156–165 (2010).
Jensen, O., Bonnefond, M. & VanRullen, R. An oscillatory mechanism for prioritizing salient unattended stimuli. Trends Cogn. Sci. 16, 200–206 (2012).
Lisman, J. The theta/gamma discrete phase code occuring during the hippocampal phase precession may be a more general brain coding scheme. Hippocampus 15, 913–922 (2005).
Fries, P. A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends Cogn. Sci. 9, 474–480 (2005).
Jensen, O., Gips, B., Bergmann, T. O. & Bonnefond, M. Temporal coding organized by coupled alpha and gamma oscillations prioritize visual processing. Trends Neurosci. 37, 357–369 (2014).
Hasson, U., Chen, J. & Honey, C. J. Hierarchical process memory: memory as an integral component of information processing. Trends Cogn. Sci. 19, 304–313 (2015).
Wang, X.-J. & Kennedy, H. Brain structure and dynamics across scales: in search of rules. Curr. Opin. Neurobiol. 37, 92–98 (2016).
Cavanagh, S. E., Hunt, L. T. & Kennerley, S. W. A diversity of intrinsic timescales underlie neural computations. Front. Neural Circuits 14, 615626 (2020).
Soltani, A., Murray, J. D., Seo, H. & Lee, D. Timescales of cognition in the brain. Curr. Opin. Behav. Sci. 41, 30–37 (2021). This review discusses how information processing on different neural timescales may contribute to sensory and reward integration in the brain.
Wolff, A. et al. Intrinsic neural timescales: temporal integration and segregation. Trends Cogn. Sci. 26, 159–173 (2022). This paper provides a comprehensive review on the role of intrinsic neural timescales, as measured by the autocorrelation window, for the integration and segregation of sensory input.
Singer, W. et al. Neuronal assemblies: necessity, signature and detectability. Trends Cogn. Sci. 1, 252–261 (1997).
Siegel, M., Donner, T. H. & Engel, A. K. Spectral fingerprints of large-scale neuronal interactions. Nat. Rev. Neurosci. 13, 121–134 (2012).
Engel, A., Gerloff, C., Hilgetag, C. & Nolte, G. Intrinsic coupling modes: multiscale interactions in ongoing brain activity. Neuron 80, 867–886 (2013). This review summarizes evidence for the functional importance of intrinsically generated neural coupling at multiple timescales.
Fries, P. Rhythms for cognition: communication through coherence. Neuron 88, 220–235 (2015).
Bastos, A. M. et al. Visual areas exert feedforward and feedback influences through distinct frequency channels. Neuron 85, 390–401 (2015).
Michalareas, G. et al. Alpha-beta and gamma rhythms subserve feedback and feedforward influences among human visual cortical areas. Neuron 89, 384–397 (2016).
Helfrich, R. F. & Knight, R. T. Oscillatory dynamics of prefrontal cognitive control. Trends Cogn. Sci. 20, 916–930 (2016).
Canolty, R. T. & Knight, R. T. The functional role of cross-frequency coupling. Trends Cogn. Sci. 14, 506–515 (2010).
Hyafil, A., Giraud, A.-L., Fontolan, L. & Gutkin, B. Neural cross-frequency coupling: connecting architectures, mechanisms, and functions. Trends Neurosci. 38, 725–740 (2015).
Yamashita, Y. & Tani, J. in Computational and Robotic Models of the Hierarchical Organization of Behavior (ed. Baldassarre, G. M.) 47–62 (Springer, 2013).
Buschman, T. J. & Kastner, S. From behavior to neural dynamics: an integrated theory of attention. Neuron 88, 127–144 (2015).
Fries, P. Rhythmic attentional scanning. Neuron 111, 954–970 (2023).
Bastos, A. et al. Canonical microcircuits for predictive coding. Neuron 76, 695–711 (2012).
Pomper, U., Keil, J., Foxe, J. J. & Senkowski, D. Intersensory selective attention and temporal orienting operate in parallel and are instantiated in spatially distinct sensory and motor cortices. Hum. Brain Mapp. 36, 3246–3259 (2015).
Daume, J., Graetz, S., Gruber, T., Engel, A. K. & Friese, U. Cognitive control during audiovisual working memory engages frontotemporal theta-band interactions. Sci. Rep. 7, 12585 (2017).
Wang, P., Göschl, F., Friese, U., König, P. & Engel, A. K. Long-range functional coupling predicts performance: oscillatory EEG networks in multisensory processing. Neuroimage 196, 114–125 (2019). This EEG study demonstrates phase coupling in multiple frequency bands, as well as phase–amplitude coupling, during visuotactile pattern matching.
Lakatos, P., Chen, C.-M., O’Connell, M. N., Mills, A. & Schroeder, C. E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292 (2007). This study demonstrates broadband crossmodal tactile-auditory phase resetting in the monkey primary auditory cortex.
Bauer, A. R., Debener, S. & Nobre, A. C. Synchronisation of neural oscillations and cross-modal influences. Trends Cogn. Sci. 24, 481–495 (2020). This review highlights the critical role of crossmodal phase resetting and neural entrainment in multisensory processing.
Galindo-Leon, E. E. et al. Context-specific modulation of intrinsic coupling modes shapes multisensory processing. Sci. Adv. 5, eaar7633 (2019). This study in the ferret cortex shows that phase and amplitude coupling in different frequency bands predicts specific aspects of multisensory processing.
Cooke, J., Poch, C., Gillmeister, H., Costantini, M. & Romei, V. Oscillatory properties of functional connections between sensory areas mediate cross-modal illusory perception. J. Neurosci. 39, 5711–5718 (2019).
Senkowski, D., Talsma, D., Herrmann, C. S. & Woldorff, M. G. Multisensory processing and oscillatory gamma responses: effects of spatial selective attention. Exp. Brain Res. 166, 411–426 (2005).
Senkowski, D., Talsma, D., Grigutsch, M., Herrmann, C. S. & Woldorff, M. G. Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45, 561–571 (2007).
Quinn, B. T. et al. Intracranial cortical responses during visual-tactile integration in humans. J. Neurosci. 34, 171–181 (2014).
Krebber, M., Harwood, J., Spitzer, B., Keil, J. & Senkowski, D. Visuotactile motion congruence enhances gamma-band activity in visual and somatosensory cortices. Neuroimage 117, 160–169 (2015).
Ferraro, S. et al. Stereotactic electroencephalography in humans reveals multisensory signal in early visual and auditory cortices. Cortex 126, 253–264 (2020).
Roa Romero, Y., Keil, J., Balz, J., Gallinat, J. & Senkowski, D. Reduced frontal theta oscillations indicate altered crossmodal prediction error processing in schizophrenia. J. Neurophysiol. 116, 1396–1407 (2016).
Biau, E., Schultz, B. G., Gunter, T. C. & Kotz, S. A. Left motor δ oscillations reflect asynchrony detection in multisensory speech perception. J. Neurosci. 42, 2313–2326 (2022).
Mahjoory, K., Schoffelen, J. M., Keitel, A. & Gross, J. The frequency gradient of human resting-state brain oscillations follows cortical hierarchies. eLife 9, e53715 (2020).
Cao, Y., Summerfield, C., Park, H., Giordano, B. L. & Kayser, C. Causal inference in the multisensory brain. Neuron 102, 1076–1087.e8 (2019).
Rohe, T., Ehlis, A. C. & Noppeney, U. The neural dynamics of hierarchical Bayesian causal inference in multisensory perception. Nat. Commun. 10, 1907 (2019).
Rohe, T. & Noppeney, U. Cortical hierarchies perform Bayesian causal inference in multisensory perception. PLoS Biol. 13, e1002073 (2015).
Rohe, T. & Noppeney, U. Distinct computational principles govern multisensory integration in primary sensory and association cortices. Curr. Biol. 26, 509–514 (2016).
Murray, J. D. et al. A hierarchy of intrinsic timescales across primate cortex. Nat. Neurosci. 17, 1661–1663 (2014).
Balz, J. et al. GABA concentration in superior temporal sulcus predicts gamma power and perception in the sound-induced flash illusion. Neuroimage 125, 724–730 (2016).
Ozker, M., Schepers, I. M., Magnotti, J. F., Yoshor, D. & Beauchamp, M. S. A double dissociation between anterior and posterior superior temporal gyrus for processing audiovisual speech demonstrated by electrocorticography. J. Cogn. Neurosci. 29, 1044–1060 (2017).
Michail, G., Senkowski, D., Holtkamp, M., Wächter, B. & Keil, J. Early beta oscillations in multisensory association areas underlie crossmodal performance enhancement. Neuroimage 257, 119307 (2022).
La Rocca, D., Ciuciu, P., Engemann, D. A. & van Wassenhove, V. Emergence of β and γ networks following multisensory training. Neuroimage 206, 116313 (2020). This MEG study demonstrates changes in stimulus-induced beta-band and gamma-band network oscillations after multisensory training.
Theves, S., Chan, J. S., Naumer, M. J. & Kaiser, J. Improving audio-visual temporal perception through training enhances beta-band activity. Neuroimage 206, 116312 (2020).
Göschl, F., Friese, U., Daume, J., König, P. & Engel, A. K. Oscillatory signatures of crossmodal congruence effects: an EEG investigation employing a visuotactile pattern matching paradigm. Neuroimage 116, 177–186 (2015).
Arnal, L. H., Wyart, V. & Giraud, A.-L. Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nat. Neurosci. 14, 797–801 (2011). This MEG study shows that crossmodal prediction errors in audiovisual speech processing are expressed in multi-timescale neural dynamics.
Keil, J. & Senkowski, D. Neural oscillations orchestrate multisensory processing. Neuroscientist 24, 609–626 (2018). This review provides an overview of the functional role of neural oscillations with different timescales for multisensory processing.
Mercier, M. R. et al. Auditory-driven phase reset in visual cortex: human electrocorticography reveals mechanisms of early multisensory integration. Neuroimage 79, 19–29 (2013).
Kayser, C., Petkov, C. I. & Logothetis, N. K. Visual modulation of neurons in auditory cortex. Cereb. Cortex 18, 1560–1574 (2008).
Thorne, J. D., De Vos, M., Viola, F. C. & Debener, S. Cross-modal phase reset predicts auditory task performance in humans. J. Neurosci. 31, 3853–3861 (2011).
Mercier, M. R. et al. Neuro-oscillatory phase alignment drives speeded multisensory response times: an electro-corticographic investigation. J. Neurosci. 35, 8546–8557 (2015). This human electrocorticography study demonstrates the behavioural relevance of visual-auditory crossmodal phase resetting in auditory cortex.
Daume, J., Wang, P., Maye, A., Zhang, D. & Engel, A. K. Non-rhythmic temporal prediction involves phase resets of low-frequency delta oscillations. Neuroimage 224, 117376 (2021).
Lakatos, P. et al. The leading sense: supramodal control of neurophysiological context by attention. Neuron 64, 419–430 (2009).
Sieben, K., Röder, B. & Hanganu-Opatz, I. L. Oscillatory entrainment of primary somatosensory cortex encodes visual control of tactile processing. J. Neurosci. 33, 5736–5749 (2013). This study in rats shows that cortico-cortical interactions contribute to crossmodal phase resetting on multiple timescales in somatosensory cortex.
Kuroki, S. et al. Excitatory neuronal hubs configure multisensory integration of slow waves in association cortex. Cell Rep. 22, 2873–2885 (2018).
Mégevand, P. et al. Crossmodal phase reset and evoked responses provide complementary mechanisms for the influence of visual speech in auditory cortex. J. Neurosci. 40, 8530–8542 (2020).
Crosse, M. J., Butler, J. S. & Lalor, E. C. Congruent visual speech enhances cortical entrainment to continuous auditory speech in noise-free conditions. J. Neurosci. 35, 14195–14204 (2015).
Wang, D., Clouter, A., Chen, Q., Shapiro, K. L. & Hanslmayr, S. Single-trial phase entrainment of theta oscillations in sensory regions predicts human associative memory performance. J. Neurosci. 38, 6299–6309 (2018).
Bauer, A. R., van Ede, F., Quinn, A. J. & Nobre, A. C. Rhythmic modulation of visual perception by continuous rhythmic auditory stimulation. J. Neurosci. 41, 7065–7075 (2021).
Albouy, P., Martinez-Moreno, Z. E., Hoyer, R. S., Zatorre, R. J. & Baillet, S. Supramodality of neural entrainment: rhythmic visual stimulation causally enhances auditory working memory performance. Sci. Adv. 8, eabj9782 (2022).
Maddox, R. K., Atilgan, H., Bizley, J. K. & Lee, A. K. Auditory selective attention is enhanced by a task-irrelevant temporally coherent visual stimulus in human listeners. eLife 4, e04995 (2015).
Kösem, A. & van Wassenhove, V. Temporal structure in audiovisual sensory selection. PLoS ONE 7, e40936 (2012).
Fu, X. & Riecke, L. Effects of continuous tactile stimulation on auditory-evoked cortical responses depend on the audio-tactile phase. Neuroimage 274, 120140 (2023).
Keil, J., Müller, N., Ihssen, N. & Weisz, N. On the variability of the McGurk effect: audiovisual integration depends on prestimulus brain states. Cereb. Cortex 22, 221–231 (2012).
Ghazanfar, A. A., Chandrasekaran, C. & Logothetis, N. K. Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. J. Neurosci. 28, 4457–4469 (2008).
Maier, J. X., Chandrasekaran, C. & Ghazanfar, A. A. Integration of bimodal looming signals through neuronal coherence in the temporal lobe. Curr. Biol. 18, 963–968 (2008).
Kayser, C. & Logothetis, N. K. Directed interactions between auditory and superior temporal cortices and their role in sensory integration. Front. Integr. Neurosci. 3, 7 (2009).
Bieler, M. et al. Rate and temporal coding convey multisensory information in primary sensory cortices. eNeuro https://doi.org/10.1523/ENEURO.0037-17.2017 (2017).
Olofsson, J. K., Zhou, G., East, B. S., Zelano, C. & Wilson, D. A. Odor identification in rats: behavioral and electrophysiological evidence of learned olfactory-auditory associations. eNeuro https://doi.org/10.1523/ENEURO.0102-19.2019 (2019).
Clouter, A., Shapiro, K. L. & Hanslmayr, S. Theta phase synchronization is the glue that binds human associative memory. Curr. Biol. 27, 3143–3148 (2017).
Grabot, L., Kösem, A., Azizi, L. & van Wassenhove, V. Prestimulus alpha oscillations and the temporal sequencing of audiovisual events. J. Cogn. Neurosci. 29, 1566–1582 (2017).
Ikumi, N., Torralba, M., Ruzzoli, M. & Soto-Faraco, S. The phase of pre-stimulus brain oscillations correlates with cross-modal synchrony perception. Eur. J. Neurosci. 49, 150–164 (2019).
Kaiser, M., Senkowski, D., Busch, N. A., Balz, J. & Keil, J. Single trial prestimulus oscillations predict perception of the sound-induced flash illusion. Sci. Rep. 9, 5983 (2019).
Kaiser, M., Senkowski, D. & Keil, J. Mediofrontal theta-band oscillations reflect top-down influence in the ventriloquist illusion. Hum. Brain Mapp. 42, 452–466 (2021).
Keil, J., Müller, N., Hartmann, T. & Weisz, N. Prestimulus beta power and phase synchrony influence the sound-induced flash illusion. Cereb. Cortex 24, 1278–1288 (2014).
Leonardelli, E. et al. Prestimulus oscillatory alpha power and connectivity patterns predispose perceptual integration of an audio and a tactile stimulus. Hum. Brain Mapp. 36, 3486–3498 (2015).
London, R. E. et al. EEG alpha power predicts the temporal sensitivity of multisensory perception. Eur. J. Neurosci. 55, 3241–3255 (2022).
Buergers, S. & Noppeney, U. The role of alpha oscillations in temporal binding within and across the senses. Nat. Hum. Behav. 6, 732–742 (2022).
Cecere, R., Rees, G. & Romei, V. Individual differences in alpha frequency drive crossmodal illusory perception. Curr. Biol. 25, 231–235 (2015).
Keil, J. & Senkowski, D. Individual alpha frequency relates to the sound-induced flash illusion. Multisens. Res. 30, 565–578 (2017).
VanRullen, R. & Koch, C. Is perception discrete or continuous? Trends Cogn. Sci. 7, 207–213 (2003).
Lange, J., Keil, J., Schnitzler, A., van Dijk, H. & Weisz, N. The role of alpha oscillations for illusory perception. Behav. Brain Res. 271, 294–301 (2014).
Samaha, J. & Romei, V. Alpha-band frequency and temporal windows in perception: a review and living meta-analysis of 27 experiments (and counting). J. Cogn. Neurosci. 36, 640–654 (2024).
Schoffelen, J. M., Pesci, U. G. & Noppeney, U. Alpha oscillations and temporal binding windows in perception — a critical review and best practice guidelines. J. Cogn. Neurosci. 36, 655–690 (2024).
Hipp, J. F., Engel, A. K. & Siegel, M. Oscillatory synchronization in large-scale cortical networks predicts perception. Neuron 69, 387–396 (2011).
Zhou, G. et al. Human olfactory-auditory integration requires phase synchrony between sensory cortices. Nat. Commun. 10, 1168 (2019). Using human intracranial recordings, this study shows that auditory–olfactory matching involves phase–amplitude coupling between auditory and olfactory cortical regions.
Powers, A. R., Hillock, A. R. & Wallace, M. T. Perceptual training narrows the temporal window of multisensory binding. J. Neurosci. 29, 12265–12274 (2009).
Lennert, T., Samiee, S. & Baillet, S. Coupled oscillations enable rapid temporal recalibration to audiovisual asynchrony. Commun. Biol. 4, 559 (2021). This MEG study demonstrates rapid temporal recalibration of alpha–gamma phase–amplitude coupling in response to audiovisual temporal asynchronies.
Bidelman, G. M. Musicians have enhanced audiovisual multisensory binding: experience-dependent effects in the double-flash illusion. Exp. Brain Res. 234, 3037–3047 (2016).
Lee, H. & Noppeney, U. Long-term music training tunes how the brain temporally binds signals from multiple senses. Proc. Natl Acad. Sci. USA 108, E1441–E1450 (2011).
O’Donohue, M., Lacherez, P. & Yamamoto, N. Musical training refines audiovisual integration but does not influence temporal recalibration. Sci. Rep. 12, 15292 (2022).
Klein, C., Liem, F., Hänggi, J., Elmer, S. & Jäncke, L. The “silent” imprint of musical training. Hum. Brain Mapp. 37, 536–546 (2016). This EEG study in musicians shows that sustained training influences multi-timescale network dynamics in the brain.
Cocchi, L., Gollo, L. L., Zalesky, A. & Breakspear, M. Criticality in the brain: a synthesis of neurobiology, models and cognition. Prog. Neurobiol. 158, 132–152 (2017). This review highlights the important role of criticality, which is characterized by activity fluctuations that have no preferred timescale and allow the brain to rapidly adapt to environmental changes.
La Rocca, D., Zilber, N., Abry, P., van Wassenhove, V. & Ciuciu, P. Self-similarity and multifractality in human brain activity: a wavelet-based analysis of scale-free brain dynamics. J. Neurosci. Methods 309, 175–187 (2018).
Palva, J. M. et al. Neuronal long-range temporal correlations and avalanche dynamics are correlated with behavioral scaling laws. Proc. Natl Acad. Sci. USA 110, 3585–3590 (2013).
Palva, S. & Palva, J. M. Roles of brain criticality and multiscale oscillations in temporal predictions for sensorimotor processing. Trends Neurosci. 41, 729–743 (2018). This review highlights the importance of scale-free neural dynamics for the role of neural oscillations and cross-frequency coupling in sensorimotor processing.
Linkenkaer-Hansen, K., Nikouline, V. V., Palva, J. M. & Ilmoniemi, R. J. Long-range temporal correlations and scaling behavior in human brain oscillations. J. Neurosci. 21, 1370–1377 (2001).
Gross, J. et al. Speech rhythms and multiplexed oscillatory sensory coding in the human brain. PLoS Biol. 11, e1001752 (2013).
Schmitt, L. M. et al. Predicting speech from a cortical hierarchy of event-based time scales. Sci. Adv. 7, eabi6070 (2021).
Park, H., Kayser, C., Thut, G. & Gross, J. Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility. eLife 5, e14521 (2016).
Aller, M., Økland, H. S., MacGregor, L. J., Blank, H. & Davis, M. H. Differential auditory and visual phase-locking are observed during audio-visual benefit and silent lip-reading for speech perception. J. Neurosci. 42, 6108–6120 (2022).
Perrodin, C., Kayser, C., Logothetis, N. K. & Petkov, C. I. Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex. Proc. Natl Acad. Sci. USA 112, 273–278 (2015).
Thézé, R., Giraud, A. L. & Mégevand, P. The phase of cortical oscillations determines the perceptual fate of visual cues in naturalistic audiovisual speech. Sci. Adv. 6, eabc6348 (2020).
Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A. & Ghazanfar, A. A. The natural statistics of audiovisual speech. PLoS Comput. Biol. 5, e1000436 (2009). This study in monkeys shows that multi-timescale neural responses, as reflected in different frequency band oscillations, match to the natural features of audiovisual speech.
Chalas, N., Omigie, D., Poeppel, D. & van Wassenhove, V. Hierarchically nested networks optimize the analysis of audiovisual speech. iScience 26, 106257 (2023). This MEG study in humans demonstrates that hierarchically nested oscillatory brain networks track audiovisual speech asynchronies.
Noppeney, U. Perceptual inference, learning, and attention in a multisensory world. Annu. Rev. Neurosci. 44, 449–473 (2021).
Feng, W., Störmer, V. S., Martinez, A., McDonald, J. J. & Hillyard, S. A. Involuntary orienting of attention to a sound desynchronizes the occipital alpha rhythm and improves visual perception. Neuroimage 150, 318–328 (2017).
Friese, U. et al. Oscillatory brain activity during multisensory attention reflects activation, disinhibition, and cognitive control. Sci. Rep. 6, 32775 (2016).
Gomez-Ramirez, M. et al. Oscillatory sensory selection mechanisms during intersensory attention to rhythmic auditory and visual inputs: a human electrocorticographic investigation. J. Neurosci. 31, 18556–18567 (2011). This human intracranial recording study demonstrates a multi-timescale phase–amplitude coupling mechanism that supports multisensory attention.
Keil, J., Pomper, U. & Senkowski, D. Distinct patterns of local oscillatory activity and functional connectivity underlie intersensory attention and temporal prediction. Cortex 74, 277–288 (2016).
Zuo, Y., Huang, Y., Wu, D., Wang, Q. & Wang, Z. Spike phase shift relative to beta oscillations mediates modality selection. Cereb. Cortex 30, 5431–5448 (2020).
Panichello, M. F. & Buschman, T. J. Shared mechanisms underlie the control of working memory and attention. Nature 592, 601–605 (2021).
Quak, M., London, R. E. & Talsma, D. A multisensory perspective of working memory. Front. Hum. Neurosci. 9, 197 (2015).
Fiebelkorn, I. C. & Kastner, S. A rhythmic theory of attention. Trends Cogn. Sci. 23, 87–101 (2019).
Axmacher, N. et al. Cross-frequency coupling supports multi-item working memory in the human hippocampus. Proc. Natl Acad. Sci. USA 107, 3228–3233 (2010).
Xie, Y. et al. Theta oscillations and source connectivity during complex audiovisual object encoding in working memory. Front. Hum. Neurosci. 15, 614950 (2021).
Michail, G., Senkowski, D., Niedeggen, M. & Keil, J. Memory load alters perception-related neural oscillations during multisensory integration. J. Neurosci. 41, 1505–1515 (2021). This EEG study shows that interactions between working memory and multisensory perception are expressed in multi-timescale interactions of neural oscillations.
Spitzer, B. & Blankenburg, F. Supramodal parametric working memory processing in humans. J. Neurosci. 32, 3287–3295 (2012).
Friston, K. The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138 (2010).
Arnal, L. H. & Giraud, A.-L. Cortical oscillations and sensory predictions. Trends Cogn. Sci. 16, 390–398 (2012).
Bastos, A. M., Lundqvist, M., Waite, A. S., Kopell, N. & Miller, E. K. Layer and rhythm specificity for predictive routing. Proc. Natl Acad. Sci. USA 117, 31459–31469 (2020).
Fernandez-Ruiz, A., Sirota, A., Lopes-dos-Santos, V. & Dupret, D. Over and above frequency: gamma oscillations as units of neural circuit operations. Neuron 111, 936–953 (2023).
Arnal, L. H., Doelling, K. B. & Poeppel, D. Delta-beta coupled oscillations underlie temporal prediction accuracy. Cereb. Cortex 25, 3077–3085 (2015).
Luo, H., Liu, Z. & Poeppel, D. Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol. 8, e1000445 (2010).
Lakatos, P. et al. An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. J. Neurophysiol. 94, 1904–1911 (2005).
Schroeder, C. E. & Lakatos, P. Low-frequency neuronal oscillations as instruments of sensory selection. Trends Neurosci. 32, 9–18 (2009).
van Atteveldt, N., Murray, M. M., Thut, G. & Schroeder, C. E. Multisensory integration: flexible use of general operations. Neuron 81, 1240–1253 (2014).
Roopun, A. K. et al. Temporal interactions between cortical rhythms. Front. Neurosci. 2, 145–154 (2008).
Yang, H., Shew, W. L., Roy, R. & Plenz, D. Maximal variability of phase synchrony in cortical networks with neuronal avalanches. J. Neurosci. 32, 1061–1072 (2012).
Chandrasekaran, C. Computational principles and models of multisensory integration. Curr. Opin. Neurobiol. 43, 25–34 (2017).
Shams, L. & Beierholm, U. Bayesian causal inference: a unifying neuroscience theory. Neurosci. Biobehav. Rev. 137, 104619 (2022).
Parise, C. V. & Ernst, M. O. Correlation detection as a general mechanism for multisensory integration. Nat. Commun. 7, 11543 (2016).
Pesnot Lerousseau, J., Parise, C. V., Ernst, M. O. & van Wassenhove, V. Multisensory correlation computations in the human brain identified by a time-resolved encoding model. Nat. Commun. 13, 2489 (2022).
Kramer, M. A. et al. Rhythm generation through period concatenation in rat somatosensory cortex. PLoS Comput. Biol. 4, e1000169 (2008).
Gelastopoulos, A., Whittington, M. A. & Kopell, N. J. Parietal low beta rhythm provides a dynamical substrate for a working memory buffer. Proc. Natl Acad. Sci. USA 116, 16613–16620 (2019).
Schirner, M., McIntosh, A. R., Jirsa, V., Deco, G. & Ritter, P. Inferring multi-scale neural mechanisms with brain network modelling. eLife 7, e28927 (2018). This article presents a computational modelling approach for the analysis of neural activity from humans and animals on multiple spatial and temporal scales.
D’Angelo, E. & Jirsa, V. The quest for multiscale brain modeling. Trends Neurosci. 45, 777–790 (2022).
Einevoll, G. T., Kayser, C., Logothetis, N. K. & Panzeri, S. Modelling and analysis of local field potentials for studying the function of cortical circuits. Nat. Rev. Neurosci. 14, 770–785 (2013).
Nasr, K. et al. Breaking the boundaries of interacting with the human brain using adaptive closed-loop stimulation. Progr. Neurobiol. 216, 102311 (2022).
Haslacher, D. et al. In vivo phase-dependent enhancement and suppression of human brain oscillations by transcranial alternating current stimulation (tACS). Neuroimage 275, 120187 (2023).
Fenno, L., Yizhar, O. & Deisseroth, K. The development and application of optogenetics. Annu. Rev. Neurosci. 34, 389–412 (2011).
Grosenick, L., Marshel, J. H. & Deisseroth, K. Closed-loop and activity-guided optogenetic control. Neuron 86, 106–139 (2015).
Zhou, H.-y et al. Multisensory temporal binding window in autism spectrum disorders and schizophrenia spectrum disorders: a systematic review and meta-analysis. Neurosci. Biobehav. Rev. 86, 66–76 (2018).
Gröhn, C., Norgren, E. & Eriksson, L. A systematic review of the neural correlates of multisensory integration in schizophrenia. Schizophr. Res. Cogn. 27, 100219 (2022).
Feldman, J. I. et al. Audiovisual multisensory integration in individuals with autism spectrum disorder: a systematic review and meta-analysis. Neurosci. Biobehav. Rev. 95, 220–234 (2018).
Wallace, M. T., Woynaroski, T. G. & Stevenson, R. A. Multisensory integration as a window into orderly and disrupted cognition and communication. Annu. Rev. Psychol. 71, 193–219 (2020).
van Laarhoven, T., Stekelenburg, J. J., Eussen, M. L. & Vroomen, J. Atypical visual-auditory predictive coding in autism spectrum disorder: electrophysiological evidence from stimulus omissions. Autism 24, 1849–1859 (2020).
Hirano, S. et al. Phase-amplitude coupling of the electroencephalogram in the auditory cortex in schizophrenia. Biol. Psychiatry Cogn. Neurosci. Neuroimaging 3, 69–76 (2018).
Moran, J. K., Michail, G., Heinz, A., Keil, J. & Senkowski, D. Long-range temporal correlations in resting state beta oscillations are reduced in schizophrenia. Front. Psychiatry 10, 517 (2019).
Cruz, G. et al. Long range temporal correlations (LRTCs) in MEG-data during emerging psychosis: relationship to symptoms, medication-status and clinical trajectory. Neuroimage Clin. 31, 102722 (2021).
Rognini, G. et al. Multisensory bionic limb to achieve prosthesis embodiment and reduce distorted phantom limb perceptions. J. Neurol. Neurosurg. Psychiatry 90, 833–836 (2019).
Aurucci, G. V., Preatoni, G., Damiani, A. & Raspopovic, S. Brain-computer interface to deliver individualized multisensory intervention for neuropathic pain. Neurotherapeutics 20, 1316–1329 (2023).
Lloyd-Esenkaya, T., Lloyd-Esenkaya, V., O’Neill, E. & Proulx, M. J. Multisensory inclusive design with sensory substitution. Cogn. Res. Princ. Implic. 5, 37 (2020).
Mackin, A. & Bull, D. Characterizing the spatiotemporal envelope of the human visual system through the visibility of temporal aliasing artifacts. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 37, 1116–1127 (2020).
Mori, S., Kikuchi, Y., Hirose, N., Lepage, H. & Wong, W. Auditory gap detection: psychometric functions and insights into the underlying neural activity. Biol. Cybern. 112, 575–584 (2018).
Victor, J. D. How the brain uses time to represent and process visual information. Brain Res. 886, 33–46 (2000).
Uchida, N., Poo, C. & Haddad, R. Coding and transformations in the olfactory system. Annu. Rev. Neurosci. 37, 363–385 (2014).
Kayser, C., Montemurro, M. A., Logothetis, N. K. & Panzeri, S. Spike-phase coding boosts and stabilizes information carried by spatial and temporal spike patterns. Neuron 61, 597–608 (2009).
Chaudhuri, R., Knoblauch, K., Gariel, M.-A., Kennedy, H. & Wang, X.-J. A large-scale circuit mechanism for hierarchical dynamical processing in the primate cortex. Neuron 88, 419–431 (2015).
Villalonga, M. B. & Sekuler, R. Keep your finger on the pulse: better rate perception and gap detection with vibrotactile compared to visual stimuli. Atten. Percept. Psychophys. 85, 2004–2017 (2023).
Manger, P. R., Engler, G., Moll, C. K. E. & Engel, A. K. The anterior ectosylvian visual area of the ferret: a homologue for an enigmatic visual cortical area of the cat? Eur. J. Neurosci. 22, 706–714 (2005).
Treisman, A. M. & Gelade, G. A feature-integration theory of attention. Cogn. Psychol. 12, 97–136 (1980).
Biederman, I. Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94, 115–147 (1987).
Desimone, R. & Duncan, J. Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193–222 (1995).
Johansen-Berg, H. & Lloyd, D. M. The physiology and psychology of selective attention to touch. Front. Biosci. Landmark 5, 894–904 (2000).
Fritz, J. B., Elhilali, M., David, S. V. & Shamma, S. A. Auditory attention — focusing the searchlight on sound. Curr. Opin. Neurobiol. 17, 437–455 (2007).
Freeman, W. J. Simulation of chaotic EEG patterns with a dynamic model of the olfactory system. Biol. Cybern. 56, 139–150 (1987).
Gray, C. M. & Singer, W. Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc. Natl Acad. Sci. USA 86, 1698–1702 (1989).
Gray, C. M., König, P., Engel, A. K. & Singer, W. Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338, 334–337 (1989).
Fries, P., Reynolds, J. H., Rorie, A. E. & Desimone, R. Modulation of oscillatory neuronal synchronization by selective visual attention. Science 291, 1560–1563 (2001).
Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285 (2006).
Voloh, B. & Womelsdorf, T. A role of phase-resetting in coordinating large scale neural networks during attention and goal-directed behavior. Front. Syst. Neurosci. 10, 18 (2016).
Atilgan, H. et al. Integration of visual information in auditory cortex promotes auditory scene analysis through multisensory binding. Neuron 97, 640–655 (2018).
Gingras, G., Rowland, B. A. & Stein, B. E. The differing impact of multisensory and unisensory integration on behavior. J. Neurosci. 29, 4897–4902 (2009).
Pluta, S. R., Rowland, B. A., Stanford, T. R. & Stein, B. E. Alterations to multisensory and unisensory integration by stimulus competition. J. Neurophysiol. 106, 3091–3101 (2011).
Acknowledgements
D.S. discloses support for this work from the Deutsche Forschungsgemeinschaft (DFG) (SE1859/10-1). A.K.E. acknowledges support for this work from the DFG (SFB936-178316478-A2/A3; TRR169-261402652-B1/B4/Z2) and from the European Union (project cICMs, ERC-2022-AdG-101097402). Views and opinions expressed in this article are those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
Author information
Authors and Affiliations
Contributions
The authors contributed equally to all aspects of the article.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Reviews Neuroscience thanks Nancy Kopell, Christoph Kayser and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Glossary
- Alpha
-
The alpha band comprises frequencies between 8 and 12 Hz; rhythms in this frequency band are also called alpha oscillations; these relatively slow oscillations constitute processing time windows of about 100 ms.
- Amplitude modulation
-
Change of the amplitude of a neural oscillation in response to an external input.
- Autocorrelation
-
Correlation between values of a signal at different time points, as a function of the lag between these time points.
- Beta
-
The beta band comprises frequencies between 13 and 30 Hz; rhythmic activity in this frequency band is also known as beta oscillations; the processing time window provided by these fast oscillations is in the range of 40–70 ms.
- Criticality
-
State of a system close to a phase transition, in which spatiotemporal correlations are highly susceptible to perturbations.
- Cross-frequency coupling
-
Coupling of the power and/or phase of neural oscillations across different frequency bands.
- Crossmodal
-
An interaction between neural systems in which input from one sensory modality influences the processing in another sensory modality.
- Delta
-
The delta band comprises frequencies below 3 Hz; rhythmic activity in this frequency band is also addressed as delta oscillations; these slow oscillations provide long processing time windows with durations of several hundreds of milliseconds.
- Double-flash illusion
-
Illusory perception of two flashes when a single flash is presented together with either two auditory clicks or two tactile stimuli.
- Entrainment
-
Synchronization of neural activity to a rhythmic external input.
- Envelope coupling
-
Dynamic coupling based on the correlation of the amplitude envelope of neural signals.
- Functional connectivity
-
Statistical relationship between signals recorded from different neurons or brain regions.
- Gamma
-
The gamma band comprises oscillations at frequencies above 30 Hz, also called gamma oscillations; often, the gamma band is subdivided into low and high gamma, depending on whether the oscillation frequencies are below or above about 80 Hz; the processing time window provided by these fast oscillations has a duration below 20–30 ms.
- Intrinsic coupling modes
-
Neural coupling patterns that are not imposed by external factors but are generated in the brain.
- Intrinsic neural timescales
-
Processing time windows inferred from the autocorrelation of neural signals in a given brain region.
- Inverse effectiveness
-
Multisensory interactions are strongest when the respective unisensory stimuli, presented alone, elicit weak neural responses.
- Long-range temporal correlations
-
Persistence of correlations in time series data across multiple timescales.
- McGurk effect
-
Fusion of an auditory syllable or word paired with a conflicting visual syllable or word into a combined multisensory percept.
- Neural dynamics
-
Spatiotemporal change of activity patterns in neuronal populations.
- Oscillation cycle
-
The repeatable part of an oscillatory waveform, comprising one trough and one peak; the duration of the cycle defines the length of the functional time window provided by the oscillation.
- Oscillations
-
Rhythmic patterns of electrical activity generated by a cooperating group of neurons.
- Oscillatory frequency bands
-
Range of frequencies that define the characteristic dynamics of a rhythmic activity pattern; neural oscillations are typically classified into five frequency bands: delta (<3 Hz), theta (4–7 Hz), alpha (8–12 Hz), beta (13–30 Hz) and gamma (>30 Hz).
- Phase–amplitude coupling
-
Modulation of the amplitude of an oscillatory signal by the phase of another oscillatory signal.
- Phase coupling
-
Dynamic coupling based on the correlation of the phase of neural signals.
- Phase resetting
-
Shift of the phase of a neural oscillation in response to an external input.
- Power
-
Square of the amplitude of a sinusoidal wave.
- Power-law scaling
-
Relation between two variables in which one variable varies as a power of the other; this yields a linear relation when both variables are displayed in a logarithmic manner.
- Predictive processing
-
Brain process involved in generating and updating predictions about sensory inputs or events.
- Scale-free dynamics
-
Lack of a characteristic timescale in the dynamics of a neural process.
- Supramodal
-
Cognitive process that operates across different sensory modalities.
- Theta
-
The theta band comprises frequencies between 4 and 7 Hz; rhythmic activity in this frequency band is also denoted as theta oscillations; these relatively slow oscillations constitute processing time windows of about 150–250 ms.
- Time windows
-
Temporal epochs in which information about sensory stimuli can be encoded or processed; such windows can vary in duration, giving rise to different processing timescales.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Senkowski, D., Engel, A.K. Multi-timescale neural dynamics for multisensory integration. Nat. Rev. Neurosci. 25, 625–642 (2024). https://doi.org/10.1038/s41583-024-00845-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41583-024-00845-7