skip to main content
10.1145/3641519.3657432acmconferencesArticle/Chapter ViewAbstractPublication PagessiggraphConference Proceedingsconference-collections
research-article
Open access

Accelerating Saccadic Response through Spatial and Temporal Cross-Modal Misalignments

Published: 13 July 2024 Publication History

Abstract

Human senses and perception are our mechanisms to explore the external world. In this context, visual saccades –rapid and coordinated eye movements– serve as a primary tool for awareness of our surroundings. Typically, our perception is not limited to visual stimuli alone but is enriched by cross-modal interactions, such as the combination of sight and hearing. In this work, we investigate the temporal and spatial relationship of these interactions, focusing on how auditory cues that precede visual stimuli influence saccadic latency –the time that it takes for the eyes to react and start moving towards a visual target. Our research, conducted within a virtual reality environment, reveals that auditory cues preceding visual information can significantly accelerate saccadic responses, but this effect plateaus beyond certain temporal thresholds. Additionally, while the spatial positioning of visual stimuli influences the speed of these eye movements, as reported in previous research, we find that the location of auditory cues with respect to their corresponding visual stimulus does not have a comparable effect. To validate our findings, we implement two practical applications: first, a basketball training task set in a more realistic environment with complex audiovisual signals, and second, an interactive farm game that explores previously untested values of our key factors. Lastly, we discuss various potential applications where our model could be beneficial.

Supplemental Material

MP4 File - presentation
presentation
PDF File
Supplementary Material - final version
PDF File
Supplementary Material: Accelerating Saccadic Response through Spatial and Temporal Cross-Modal Misalignments

References

[1]
Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency Requirements for Foveated Rendering in Virtual Reality. ACM Trans. Appl. Percept. 14, 4, Article 25 (sep 2017), 13 pages. https://doi.org/10.1145/3127589
[2]
James Anliker. 1976. Eye Movement: On-Line Measurement, Analysis, and Control. R. A. Monty & J. W. Senders (Eds.), Eye movements and psychological processes (1976), 185–202.
[3]
Elena Arabadzhiyska, Okan Tarhan Tursun, Karol Myszkowski, Hans-Peter Seidel, and Piotr Didyk. 2017. Saccade landing position prediction for gaze-contingent rendering. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–12.
[4]
Derek H Arnold, Alan Johnston, and Shinya Nishida. 2005. Timing sight and sound. Vision Research 45, 10 (2005), 1275–1284.
[5]
A Terry Bahill. 1975. Most naturally occurring human saccades have magnitudes of 15 deg or less. Invest. Ophthalmol 14 (1975), 468–469.
[6]
Peter G.J. Barten. 1999. Contrast sensitivity of the human eye and its effects on image quality. SPIE – The International Society for Optical Engineering. https://doi.org/10.1117/3.353254
[7]
AH Bell, MA Meredith, AJ Van Opstal, and DougP Munoz. 2006. Stimulus intensity modifies saccadic reaction time and visual response latency in the superior colliculus. Experimental Brain Research 174 (2006), 53–59.
[8]
Christopher C Berger, Mar Gonzalez-Franco, Ana Tajadura-Jiménez, Dinei Florencio, and Zhengyou Zhang. 2018. Generic HRTFs may be good enough in virtual reality. Improving source localization through cross-modal plasticity. Frontiers in neuroscience 12 (2018), 21.
[9]
Stefan Berti and Erich Schröger. 2001. A comparison of auditory and visual distraction effects: behavioral and event-related indices. Cognitive brain research 10, 3 (2001), 265–273.
[10]
Benjamin M Bolker, Mollie E Brooks, Connie J Clark, Shane W Geange, John R Poulsen, M Henry H Stevens, and Jada-Simone S White. 2009. Generalized linear mixed models: a practical guide for ecology and evolution. Trends in ecology & evolution 24, 3 (2009), 127–135.
[11]
Kris Boyen, Deniz Baskent, and Pim van Dijk. 2015. The gap detection test: can it be used to diagnose tinnitus?Ear and hearing 36, 4 (2015), e138–e145.
[12]
R.H.S. Carpenter. 2004. Contrast, Probability, and Saccadic Latency: Evidence for Independence of Detection and Decision. Current Biology 14, 17 (2004), 1576–1580.
[13]
Geneviève Charbonneau, Marie Véronneau, Colin Boudrias-Fournier, Franco Lepore, and Olivier Collignon. 2013. The ventriloquist in periphery: impact of eccentricity-related reliability on audio-visual localization. Journal of Vision 13, 12 (2013), 20–20.
[14]
Shaoyu Chen, Budmonde Duinkharjav, Xin Sun, Li-Yi Wei, Stefano Petrangeli, Jose Echevarria, Claudio Silva, and Qi Sun. 2022. Instant reality: Gaze-contingent perceptual optimization for 3d virtual reality streaming. IEEE Transactions on Visualization and Computer Graphics 28, 5 (2022), 2157–2167.
[15]
Yi-Chuan Chen and Charles Spence. 2017. Assessing the role of the ‘unity assumption’on multisensory integration: A review. Frontiers in psychology 8 (2017), 445.
[16]
Hans Colonius and Petra Arndt. 2001. A two-stage model for visual-auditory interaction in saccadic latencies. Perception & psychophysics 63, 1 (2001), 126–147.
[17]
BD Corneil, M Van Wanrooij, DP Munoz, and AJ Van Opstal. 2002. Auditory-visual interactions subserving goal-directed saccades in a complex scene. Journal of Neurophysiology 88, 1 (2002), 438–454.
[18]
Joan M Dafoe, Irene T Armstrong, and Doug P Munoz. 2007. The influence of stimulus direction and eccentricity on pro-and anti-saccades in humans. Experimental Brain Research 179 (2007), 563–570.
[19]
Adele Diederich, Annette Schomburg, and Hans Colonius. 2012. Saccadic reaction times to audiovisual stimuli show effects of oscillatory phase reset. (2012).
[20]
Oliver Doehrmann and Marcus J Naumer. 2008. Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain research 1242 (2008), 136–150.
[21]
Budmonde Duinkharjav, Praneeth Chakravarthula, Rachel Brown, Anjul Patney, and Qi Sun. 2022. Image Features Influence Reaction Time: A Learned Probabilistic Perceptual Model for Saccade Latency. 41, 4, Article 144 (jul 2022), 15 pages.
[22]
Budmonde Duinkharjav, Benjamin Liang, Anjul Patney, Rachel Brown, and Qi Sun. 2023. The Shortest Route is Not Always the Fastest: Probability-Modeled Stereoscopic Eye Movement Completion Time in VR. ACM Trans. Graph. 42, 6, Article 220 (2023), 14 pages.
[23]
David Dunn, Okan Tursun, Hyeonseung Yu, Piotr Didyk, Karol Myszkowski, and Henry Fuchs. 2020. Stimulating the human visual system beyond real world performance in future augmented reality displays. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 90–100.
[24]
JT Enright. 1986. Facilitation of Vergence Changes by Saccades: Influences of Misfocused Images and of Disparity Stimuli in Man.The Journal of physiology 371, 1 (1986), 69–87.
[25]
J. T. Enright. 1984. Changes in Vergence Mediated by Saccades.The Journal of Physiology 350, 1 (1984), 9–31. https://doi.org/10.1113/jphysiol.1984.sp015186
[26]
B Fischer, S Gezeck, and K Hartnegg. 1997. The analysis of saccadic eye movements from gap and overlap paradigms. Brain Research Protocols 2, 1 (1997), 47–52.
[27]
Philippe Fournier and Sylvie Hebert. 2012. Gap detection deficits in humans with tinnitus as assessed with the acoustic startle paradigm: Does tinnitus fill in the gap?Hearing research 295 (06 2012).
[28]
Linus Franke, Laura Fink, Jana Martschinke, Kai Selgrad, and Marc Stamminger. 2021. Time-Warped Foveated Rendering for Virtual Reality Headsets. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 110–123.
[29]
Maarten A Frens, A John Van Opstal, and Robert F Van der Willigen. 1995. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Perception & psychophysics 57 (1995), 802–816.
[30]
Martine Godfroy, Corinne Roumes, and Pierre Dauchy. 2003. Spatial variations of visual—auditory fusion areas. Perception 32, 10 (2003), 1233–1245.
[31]
Souta Hidaka and Masakazu Ide. 2015. Sound can suppress visual perception. Scientific reports 5, 1 (2015), 10483.
[32]
Torin Hopkins, Suibi Che-Chuan Weng, Rishi Vanukuru, Emma Wenzel, Amy Banic, Mark D Gross, and Ellen Yi-Luen Do. 2022. Studying the Effects of Network Latency on Audio-Visual Perception During an AR Musical Task. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 26–34.
[33]
RP Kalesnykas and PE Hallett. 1987. The differentiation of visually guided and anticipatory saccades in gap and overlap paradigms. Experimental Brain Research 68 (1987), 115–121.
[34]
RP Kalesnykas and PE Hallett. 1994. Retinal eccentricity and the latency of eye saccades. Vision research 34, 4 (1994), 517–531.
[35]
Z Kapoula and DA Robinson. 1986. Saccadic undershoot is not inevitable: Saccades can be accurate. Vision research 26, 5 (1986), 735–743.
[36]
Emine Merve Kaya and Mounya Elhilali. 2017. Modelling auditory attention. Philosophical Transactions of the Royal Society B: Biological Sciences 372, 1714 (2017), 20160101.
[37]
Hayeon Kim and In-Kwon Lee. 2022. Studying the effects of congruence of auditory and visual stimuli on virtual reality experiences. IEEE Transactions on Visualization and Computer Graphics 28, 5 (2022), 2080–2090.
[38]
Joohwan Kim, Josef Spjut, Morgan McGuire, Alexander Majercik, Ben Boudaoud, Rachel Albert, and David Luebke. 2019. Esports Arms Race: Latency and Refresh Rate for Competitive Gaming Tasks. Journal of Vision 19, 10 (2019), 2–2.
[39]
Alan Kingstone and Raymond M Klein. 1993. Visual offsets facilitate saccadic latency: does predisengagement of visuospatial attention mediate this gap effect?Journal of Experimental Psychology: Human Perception and Performance 19, 6 (1993), 1251.
[40]
Denis Koposov, Maria Semenova, Andrey Somov, Andrey Lange, Anton Stepanov, and Evgeny Burnaev. 2020. Analysis of the Reaction Time of eSports Players through the Gaze Tracking and Personality Trait. In 2020 IEEE 29th International Symposium on Industrial Electronics (ISIE). 1560–1565.
[41]
Eileen Kowler. 2011. Eye Movements: The Past 25 Years. Vision Research 51, 13 (2011), 1457–1483.
[42]
Matteo Lisi, Joshua A Solomon, and Michael J Morgan. 2019. Gain control of saccadic eye movements is probabilistic. Proceedings of the National Academy of Sciences 116, 32 (2019), 16137–16142.
[43]
Sandra Malpica, Ana Serrano, Julia Guerrero-Viu, Daniel Martin, Edurne Bernal, Diego Gutierrez, and Belen Masia. 2022. Auditory stimuli degrade visual performance in virtual reality. In ACM SIGGRAPH 2022 Posters. 1–2.
[44]
George R Mangun. 1995. Neural mechanisms of visual selective attention. Psychophysiology 32, 1 (1995), 4–18.
[45]
Kumpei Ogawa, Kazuyuki Fujita, Shuichi Sakamoto, Kazuki Takashima, and Yoshifumi Kitamura. 2023. Exploring Visual-Auditory Redirected Walking using Auditory Cues in Reality. IEEE Transactions on Visualization and Computer Graphics (2023).
[46]
Charles Pierrot-Deseilligny, Sophie Rivaud, Bertrand Gaymard, René Müri, and Anne-Isabelle Vermersch. 1995. Cortical control of saccades. Annals of Neurology: Official Journal of the American Neurological Association and the Child Neurology Society 37, 5 (1995), 557–567.
[47]
Patricia A Reuter-Lorenz, Howard C Hughes, and Robert Fendrich. 1991. The reduction of saccadic latency by prior offset of the fixation point: an analysis of the gap effect. Perception & psychophysics 49, 2 (1991), 167–175.
[48]
John Ross, M Concetta Morrone, and David C Burr. 1997. Compression of visual space before saccades. Nature 386, 6625 (1997), 598–601.
[49]
John Ross, M Concetta Morrone, Michael E Goldberg, and David C Burr. 2001. Changes in visual perception at the time of saccades. Trends in neurosciences 24, 2 (2001), 113–121.
[50]
Susan M Ross and Leonard E Ross. 1981. Saccade latency and warning signals: effects of auditory and visual stimulus onset and offset. Perception & Psychophysics 29 (1981), 429–437.
[51]
Olli Rummukainen, Thomas Robotham, Axel Plinge, Frank Wefers, Jürgen Herre, Emanuël Habets, 2019. Listening tests with individual versus generic head-related transfer functions in six-degrees-of-freedom virtual reality. In Audio for Virtual, Augmented and Mixed Realities: Proceedings of ICSA 2019; 5th International Conference on Spatial Audio; September 26th to 28th, 2019, Ilmenau, Germany. 55–62.
[52]
Clifton M Schor, Lori A Lott, David Pope, and Andrew D Graham. 1999. Saccades Reduce Latency and Increase Velocity of Ocular Accommodation. Vision Research 39, 22 (Nov. 1999), 3769–3795. https://doi.org/10.1016/S0042-6989(99)00094-2
[53]
Stefania Serafin, Michele Geronazzo, Cumhur Erkut, Niels C Nilsson, and Rolf Nordahl. 2018. Sonic interactions in virtual reality: State of the art, current challenges, and future directions. IEEE computer graphics and applications 38, 2 (2018), 31–43.
[54]
Ladan Shams, Yukiyasu Kamitani, and Shinsuke Shimojo. 2000. What you see is what you hear. Nature 408, 6814 (2000), 788–788.
[55]
Charles Spence. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73 (2011), 971–995.
[56]
Charles Spence and Jon Driver. 2004. Crossmodal space and crossmodal attention. Oxford University Press.
[57]
John S Stahl. 2001. Eye-head coordination and the variation of eye-movement accuracy with orbital eccentricity. Experimental brain research 136 (2001), 200–210.
[58]
Wolfgang A Teder-Sälejärvi, Francesco Di Russo, John J McDonald, and Steven A Hillyard. 2005. Effects of spatial congruity on audio-visual multimodal integration. Journal of cognitive neuroscience 17, 9 (2005), 1396–1409.
[59]
Donald J Tellinghuisen and Erin J Nowak. 2003. The inability to ignore auditory distractors as a function of visual task perceptual load. Perception & psychophysics 65 (2003), 817–828.
[60]
Marc van Wanrooij, Andrew Bell, Douglas Munoz, and John Opstal. 2009. The effect of spatial–temporal audiovisual disparities on saccades in a complex scene. Experimental Brain Research 198 (09 2009), 425–437.
[61]
Argiro Vatakis and Charles Spence. 2007. Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli. Perception & psychophysics 69 (2007), 744–756.
[62]
Manuel Vidal, Andrea Desantis, and Laurent Madelain. 2020. Irrelevant auditory and tactile signals, but not visual signals, interact with the target onset and modulate saccade latencies. Plos one 15, 2 (2020), e0221192.
[63]
Manuel Vidal and Françoise Vitu. 2022. Multisensory temporal binding induces an illusory gap/overlap that reduces the expected audiovisual interactions on saccades but not manual responses. Plos one 17, 4 (2022), e0266468.
[64]
Jean Vroomen and Mirjam Keetels. 2010. Perception of intersensory synchrony: a tutorial review. Attention, Perception, & Psychophysics 72, 4 (2010), 871–884.
[65]
Chin-An Wang, Gunnar Blohm, Jeff Huang, Susan E Boehnke, and Douglas P Munoz. 2017. Multisensory integration in orienting behavior: Pupil size, microsaccades, and saccades. Biological psychology 129 (2017), 36–44.
[66]
Elizabeth M Wenzel, Marianne Arruda, Doris J Kistler, and Frederic L Wightman. 1993. Localization using nonindividualized head-related transfer functions. The Journal of the Acoustical Society of America 94, 1 (1993), 111–123.
[67]
Shimpei Yamagishi and Shigeto Furukawa. 2020. Factors influencing saccadic reaction time: Effect of task modality, stimulus saliency, spatial congruency of stimuli, and pupil size. Frontiers in Human Neuroscience 14 (2020), 571893.
[68]
Daniela Zambarbieri. 2002. The latency of saccades toward auditory targets in humans. Progress in Brain Research 140 (2002), 51–59.
[69]
Daniela Zambarbieri, Giorgio Beltrami, and Maurizio Versino. 1995. Saccade latency toward auditory targets depends on the relative position of the sound source with respect to the eyes. Vision research 35, 23-24 (1995), 3305–3312.
[70]
Heng Zou, Hermann J Müller, and Zhuanghua Shi. 2012. Non-spatial sounds regulate eye movements and enhance visual search. Journal of Vision 12, 5 (2012), 2–2.

Index Terms

  1. Accelerating Saccadic Response through Spatial and Temporal Cross-Modal Misalignments

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        SIGGRAPH '24: ACM SIGGRAPH 2024 Conference Papers
        July 2024
        1106 pages
        ISBN:9798400705250
        DOI:10.1145/3641519
        This work is licensed under a Creative Commons Attribution International 4.0 License.

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 13 July 2024

        Check for updates

        Author Tags

        1. Audiovisual integration
        2. cross-modal interactions
        3. multisensory perception
        4. saccadic latency
        5. virtual reality

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        SIGGRAPH '24
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • 0
          Total Citations
        • 361
          Total Downloads
        • Downloads (Last 12 months)361
        • Downloads (Last 6 weeks)69
        Reflects downloads up to 13 Feb 2025

        Other Metrics

        Citations

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Login options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media