Jing Yang
ABSTRACT
The auditory sense is an intuitive and immersive channel to experience our surroundings, which motivates us to augment our perception of the real world with digital auditory content. We present a wearable audio augmented reality prototype that tracks the user with six degrees of freedom in a known environment, synthesizes 3D sounds, and plays spatialized audio from arbitrary objects to the user. Our prototype is built using head-mounted visual-inertial odometry, a sound simulation engine on a laptop, and off-the-shelf headphones. We demonstrate an application in a gallery scenario in which visitors can hear objects and scenes drawn in the paintings, feeling audio-visually engaged in the depicted surroundings. In a user study involving 26 participants, we observed that the audio-enhanced exhibition improved people's experience, as well as helped them remember more lively details of the artworks.
- Robert Albrecht, Riitta Väänänen, and Tapio Lokki. 2016. Guided by Music: Pedestrian and Cyclist Navigation with Route and Beacon Guidance. Personal and Ubiquitous Computing 20, 20:1, 20(1), 121--145.Google Scholar
- Amit Barde, Matt Ward, William S Helton, Mark Billinghurst, and Gun Lee. 2016. Attention Redirection using Binaurally Spatialised Cues Delivered over a Bone Conduction Headset. In Human Factors and Ergonomics Society Annual Meeting. SAGE Publications.Google ScholarCross Ref
- Benjamin B. Bederson and Allison Druin. 1995. Computer Augmented Environments: New Places to Learn, Work, and Play. Advances in Human Computer Interaction 5 (1995), 37--66.Google Scholar
- Jens Blauert. 1997. Spatial Hearing: The Psychophysics of Human Sound Localization. MIT press.Google Scholar
- Simon Blessenohl, Cecily Morrison, Antonio Criminisi, and Jamie Shotton. 2015. Improving Indoor mobility of The Visually Impaired with Depth-based Spatial Sound. In IEEE ICCV.Google Scholar
- Michael Bloesch, Sammy Omari, Marco Hutter, and Roland Siegwart. 2015. Robust Visual Inertial Odometry Using A Direct EKF-based Approach. In IEEE/RSJ IROS.Google Scholar
- Marcia de Borba Campos, Jaime Sánchez, Anderson Cardoso Martins, Régis Schneider Santana, and Matías Espinoza. 2014. Mobile Navigation through A Science Museum for Users Who Are Blind. In UAHCI.Google Scholar
- Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. In ECCV.Google Scholar
- Christian Forster, Zichao Zhang, Michael Gassner, Manuel Werlberger, and Davide Scaramuzza. (2017). SVO: Semidirect Visual Odometry for Monocular and Multicamera Systems. IEEE Transactions on Robotics 33, 2, 33(2), 249--265.Google ScholarDigital Library
- Florian Heller and Jan Borchers. 2014. AudioTorch: Using A Smartphone as Directional Microphone in Virtual Audio Spaces. In ACM MobileHCI.Google Scholar
- Florian Heller, Jayan Jevanesan, Pascal Dietrich, and Jan Borchers. 2016. Where Are We?: Evaluating The Current Rendering Fidelity of Mobile Audio Augmented Reality Systems. In ACM MobileHCI.Google Scholar
- Florian Heller and Johannes Schöning. 2018. NavigaTone: Seamlessly Embedding Navigation Cues in Mobile Music Listening. In ACM CHI.Google Scholar
- Yi-Ta Hsieh, Valeria Orso, Salvatore Andolina, Manuela Canaveras, Diogo Cabral, Anna Spagnolli, Luciano Gamberini, and Gamberini Jacucci. 2018. Interweaving Visual and Audio-Haptic Augmented Reality for Urban Exploration. In ACM DIS.Google Scholar
- Kangsoo Kim, Mark Billinghurst, Gerd Bruder, Henry Beenlirn Duh, and Gregory F. Welch. 2018. Revisiting Trends in Augmented Reality Research: A Review of The 2nd Decade of ISMAR (2008--2017). IEEE TVCG 24, 11 (2018), 2947--2962.Google Scholar
- Tiffany CK. Kwok, Peter Kiefer, Victor R. Schinazi, Benjamin Adams, and Martin Raubal. 2019. Gaze-Guided Narratives: Adapting Audio Guide Content to Gaze in Virtual and Real Environments. In ACM CHI.Google Scholar
- Stefan Leutenegger, Simon Lynen, Michael Bosse, Roland Siegwart, and Paul Furgale. (2015). Keyframe-based Visual-Inertial Odometry Using Nonlinear Optimization. The International Journal of Robotics Research 34, 3, 34(4), 314--334.Google ScholarDigital Library
- Jörg Müller, Matthias Geier, Christina Dicke, and Sascha Spors. 2014. The Boom-Room: Mid-air Direct Interaction with Virtual Sound Sources. In ACM CHI.Google Scholar
- Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. (2015). ORB-SLAM: A Versatile And Accurate Monocular SLAM System. IEEE Transactions on Robotics 31, 5.Google ScholarDigital Library
- Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. 2011. DTAM: Dense Tracking And Mapping in Real-Time. In IEEE ICCV.Google Scholar
- Tong Qin, Peiliang Li, and Shaojie Shen. (2018). VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator. IEEE Transactions on Robotics 34, 4, 34(4), 1004--1020.Google ScholarDigital Library
- Spencer Russell, Gershon Dublon, and Joseph A Paradiso. 2016. HearThere: Networked Sensory ProsThetics through Auditory Augmented Reality. In ACM AH.Google Scholar
- Eldon Schoop, James Smith, and Bjoern Hartmann. 2018. HindSight: Enhancing Spatial Awareness by Sonifying Detected Objects in Real-Time 360-Degree Video. In ACM CHI.Google Scholar
- Titus JJ Tang and Wai Ho Li. 2014. An Assistive Eyewear Prototype That Interactively Converts 3D Object Locations into Spatial Audio. In ACM ISWC.Google Scholar
- Yolanda Vazquez-Alvarez, Matthew P Aylett, Stephen A Brewster, Rocio von Jungenfeld, and Antti Virolainen. 2014. Multilevel Auditory Displays for Mobile Eyes-free Location-based Interaction. In ACM CHI.Google Scholar
- Chi Thanh Vi, Damien Ablart, Elia Gatti, Carlos Velasco, and Marianna Obrist. 2017. Not Just Seeing, But Also Feeling Art: Mid-air Haptic Experiences Integrated in a Multisensory Art Exhibition. International Journal of Human-Computer Studies 108 (2017), 1--14.Google ScholarDigital Library
- Ron Wakkary and Marek Hatala. (2007). Situated Play in A Tangible Interface and Adaptive Audio Museum Guide. Personal and Ubiquitous Computing 11, 3, 11(3), 171--191.Google Scholar
Index Terms
- Audio-augmented museum experiences using wearable visual-inertial odometry
Recommendations
Audio augmented reality for human-object interactions
UbiComp/ISWC '19 Adjunct: Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable ComputersIn the past, augmented reality (AR) research focused mostly on visual augmentation, which requires a visual rendering device like head-mounted displays that are usually obtrusive, expensive, and socially unaccepted. In contrast, wearable audio headsets ...
Audio-augmented museum experiences with gaze tracking
MUM '19: Proceedings of the 18th International Conference on Mobile and Ubiquitous MultimediaIn this work, we enrich landscape and genre paintings by spatializing sounds for the drawn objects and scenes, which expands visitors' perception of the paintings and immerses them in the depicted scenarios. Plus, we personalize such spatial audio ...
Interactive Narration within Audio Augmented Realities
ICIDS '09: Proceedings of the 2nd Joint International Conference on Interactive Digital Storytelling: Interactive StorytellingA test installation at IEM Graz realized in the framework of the European Project "Sonic Interaction Design" serves as example to the given issue: transforming a linear novel scene in an interactive sound installation and making it explorable through ...
Comments