skip to main content
research-article

Visar: Projecting Virtual Sound Spots for Acoustic Augmented Reality Using Air Nonlinearity

Published: 09 September 2024 Publication History

Abstract

Augmented reality that integrates virtual content in real-world surroundings has attracted lots of concentration as the growing trend of the metaverse. Acoustic augmented reality (AAR) applications have proliferated due to readily available earphones and speakers. AAR can provide omnidirectional engagement through the all-around sense of spatial information. Most existing AAR approaches offer immersive experiences by playing binaural spatial audios according to head-related transfer functions (HRTF). These involve complex modeling and require the user to wear a headphone. Air nonlinearity that can reproduce audible sounds from ultrasound offers opportunities to achieve device-free and omnidirectional sound source projection in AAR. This paper proposes Visar, a device-free virtual sound spots projection system leveraging air nonlinearity. Visar achieves simultaneous tracking and sound spot generation while suppressing unintended audio leakages caused by grating lobes and nonlinear effects in mixing lobes through optimization. Considering multi-user scenarios, Visar also proposed a multi-spot scheduling scheme to mitigate the mutual interference between the spots. Extensive experiments show the tracking error is 7.83cm and the orientation estimation error is 10.06°, respectively, envisioning the considerable potential of Visar in AAR applications.

References

[1]
2001. BS.1387: Method for objective measurements of perceived audio quality. ITU. https://www.itu.int/rec/R-REC-BS.1387-1-200111-I/en
[2]
Apple. [n.d.]. Airpods Pro (2nd generation). https://www.apple.com/airpods-pro/.
[3]
Keith Attenborough. 2014. Sound propagation in the atmosphere. Springer handbook of acoustics (2014), 117--155.
[4]
HE Bass, H-J Bauer, and LB Evans. 1972. Atmospheric absorption of sound: Analytical expressions. The Journal of the Acoustical Society of America 52, 3B (1972), 821--825.
[5]
Richard W Bohannon and A Williams Andrews. 2011. Normal walking speed: a descriptive meta-analysis. Physiotherapy 97, 3 (2011), 182--189.
[6]
David Boulinguez and André Quinquis. 2002. 3-D underwater object recognition. IEEE journal of oceanic engineering 27, 4 (2002), 814--829.
[7]
Paolo Castellini and Milena Martarelli. 2008. Acoustic beamforming: Analysis of uncertainty and metrological performances. Mechanical systems and signal processing 22, 3 (2008), 672--692.
[8]
Jagmohan Chauhan, Yining Hu, Suranga Seneviratne, Archan Misra, Aruna Seneviratne, and Youngki Lee. 2017. BreathPrint: Breathing acoustics-based user authentication. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. 278--291.
[9]
Xiangru Chen, Dong Li, Yiran Chen, and Jie Xiong. 2022. Boosting the sensing granularity of acoustic signals by exploiting hardware non-linearity. In Proceedings of the 21st ACM Workshop on Hot Topics in Networks. 53--59.
[10]
David G Crighton. 1979. Model equations of nonlinear acoustics. Annual Review of Fluid Mechanics 11, 1 (1979), 11--33.
[11]
Bruce H Deatherage, Lloyd A Jeffress, and Hugh C Blodgett. 1954. A note on the audibility of intense ultrasonic sound. The Journal of the Acoustical Society of America 26, 4 (1954), 582--582.
[12]
Focusonics. [n. d.]. Focusonics Directional Speakers. https://www.focusonics.com/. Accessed on May 1, 2023.
[13]
Yongjian Fu, Shuning Wang, Linghui Zhong, Lili Chen, Ju Ren, and Yaoxue Zhang. 2022. SVoice: Enabling Voice Communication in Silence via Acoustic Sensing on Commodity Devices. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems. 622--636.
[14]
Yongjian Fu, Yongzhao Zhang, Yu Lu, Lili Qiu, Yi-Chao Chen, Yezhou Wang, Mei Wang, Yijie Li, Ju Ren, and Yaoxue Zhang. 2024. Adaptive Metasurface-Based Acoustic Imaging using Joint Optimization. In The 22nd ACM International Conference on Mobile Systems, Applications, and Services.
[15]
Woon-Seng Gan, Jun Yang, and Tomoo Kamakura. 2012. A review of parametric acoustic array in air. Applied Acoustics 73, 12 (2012), 1211--1219.
[16]
Zhihui Gao, Ang Li, Dong Li, Jialin Liu, Jie Xiong, Yu Wang, Bing Li, and Yiran Chen. 2022. Mom: Microphone based 3d orientation measurement. In 2022 21st ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 132--144.
[17]
Olav Rune Godø, Kenneth G Foote, Johnny Dybedal, and Eirik Tenningen. 2009. Observing Atlantic herring by parametric sonar. The Journal of the Acoustical Society of America 125, 4 (2009), 2718--2718.
[18]
Corentin Guezenoc and Renaud Seguier. 2020. HRTF individualization: A survey. arXiv preprint arXiv:2003.06183 (2020).
[19]
Mark F Hamilton. 1986. Fundamentals and applications of nonlinear acoustics. VOLUME HI: BACKGROUND MATERIALS (1986), 82.
[20]
Hao Han, Shanhe Yi, Qun Li, Guobin Shen, Yunxin Liu, and Ed Novak. 2016. AMIL: Localizing neighboring mobile devices through a simple gesture. In IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications. IEEE, 1--9.
[21]
S. Haykin. 1985. Array signal processing.
[22]
HEADREC. [n. d.]. Binal 2 datasheet. https://headrec.com/products/binal-two.
[23]
Jarmo Hietanen, Pentti Mattila, Jyrki Stor-Pellinen, Fabio Tsuzuki, H Vaataja, Ken Sasaki, and Mauri Luukkala. 1993. Factors affecting the sensitivity of electrostatic ultrasonic transducers. Measurement Science and Technology 4, 10 (1993), 1138.
[24]
Sverre Holm et al. 2019. Waves with power-law attenuation. Vol. 714. Springer.
[25]
Holosonics. [n. d.]. Holosonics Audio Spotlight Series. https://www.holosonics.com/. Accessed on May 1, 2023.
[26]
Hongmei Hu, Lin Zhou, Hao Ma, and Zhenyang Wu. 2008. HRTF personalization based on artificial neural network in individual virtual auditory space. Applied Acoustics 69, 2 (2008), 163--172.
[27]
Wenchao Huang, Yan Xiong, Xiang-Yang Li, Hao Lin, Xufei Mao, Panlong Yang, and Yunhao Liu. 2013. Accurate indoor localization using acoustic direction finding via smart phones. arXiv preprint arXiv:1306.1651 (2013).
[28]
Victor F Humphrey, Stephen P Robinson, John D Smith, Michael J Martin, Graham A Beamiss, Gary Hayman, and Nicholas L Carroll. 2008. Acoustic characterization of panel materials under simulated ocean conditions using a parametric array source. The journal of the acoustical society of America 124, 2 (2008), 803--814.
[29]
Ryo Iijima, Shota Minami, Yunao Zhou, Tatsuya Takehisa, Takeshi Takahashi, Yasuhiro Oikawa, and Tatsuya Mori. 2021. Audio Hotspot Attack: An Attack on Voice Assistance Systems Using Directional Sound Beams and its Feasibility. IEEE Transactions on Emerging Topics in Computing 9, 4 (2021), 2004--2018. https://doi.org/10.1109/TETC.2019.2953041
[30]
Texas Instruments. [n. d.]. Data Sheet OPA541. https://www.ti.com/lit/ds/symlink/opa541.pdf.
[31]
Louis Jackowski-Ashley, Gianluca Memoli, Mihai Caleap, Nicolas Slack, Bruce W Drinkwater, and Sriram Subramanian. 2017. Haptics and directional audio using acoustic metasurfaces. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces. 429--433.
[32]
Xue Jiang, Yong Li, Dean Ta, and Weiqi Wang. 2020. Ultrasonic sharp autofocusing with acoustic metasurface. Physical Review B 102, 6 (2020), 064308.
[33]
Soundlazer kickstarter. 2016. https://www.kickstarter.com/projects/richardhaberkern/soundlazer.
[34]
Byung-Chul Kim and I-Tai Lu. 2000. Parameter study of OFDM underwater communications system. In OCEANS 2000 MTS/IEEE Conference and Exhibition. Conference Proceedings (Cat. No. 00CH37158), Vol. 2. IEEE, 1251--1255.
[35]
SE Kim, JH Hwang, TW Kang, SW Kang, and SW Sohn. 2012. Generation of audible sound with ultrasonic signals through the human body. In 2012 IEEE 16th International Symposium on Consumer Electronics. IEEE, 1--3.
[36]
Martin L Lenhardt, Ruth Skellett, Peter Wang, and Alex M Clarke. 1991. Human ultrasonic speech perception. Science 253, 5015 (1991), 82--85.
[37]
Kevin D LePage and Henrik Schmidt. 2002. Bistatic synthetic aperture imaging of proud and buried targets from an AUV. IEEE Journal of Oceanic Engineering 27, 3 (2002), 471--483.
[38]
Yijie Li, Xiatong Tong, Qianfei Ren, Qingyang Li, Lanqing Yang, Yi-Chao Chen, Guangtao Xue, Xiaoyu Ji, and Jiadi Yu. 2023. AUDIOSENSE: Leveraging Current to Acoustic Channel to Detect Appliances at Single-Point. In 2023 20th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 240--248.
[39]
Yijie Li, Juntao Zhou, Dian Ding, Yi-Chao Chen, Lili Qiu, Jiadi Yu, and Guangtao Xue. 2024. MuDiS: An Audio-independent, Wide-angle, and Leak-free Multi-directional Speaker. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 263--278.
[40]
Manni Liu, Linsong Cheng, Kun Qian, Jiliang Wang, Jin Wang, and Yunhao Liu. 2020. Indoor acoustic localization: A survey. Human-centric Computing and Information Sciences 10 (2020), 1--24.
[41]
Robert Malkin, Brian Kappus, Benjamin Long, and Adam Price. 2023. On the non-linear behaviour of ultrasonic air-borne phased arrays. Journal of Sound and Vibration 552 (2023), 117644.
[42]
Wenguang Mao, Jian He, and Lili Qiu. 2016. Cat: high-precision acoustic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. 69--81.
[43]
Microsoft. [n. d.]. SoundScape. https://www.microsoft.com/en-us/research/product/soundscape/.
[44]
Christopher Morse, Adam Chernick, Zeyu Ren, Sabrina Naumovski, and Luke Gehron. 2019. Sound space: Communicating acoustics through interactive visualization. In 2019 IEEE Games, Entertainment, Media Conference (GEM). IEEE, 1--4.
[45]
Philip M Morse and Richard H Bolt. 1944. Sound waves in rooms. Reviews of modern physics 16, 2 (1944), 69.
[46]
Rajalakshmi Nandakumar, Shyamnath Gollakota, and Nathaniel Watson. 2015. Contactless sleep apnea detection on smartphones. In Proceedings of the 13th annual international conference on mobile systems, applications, and services. 45--57.
[47]
Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1515--1525.
[48]
Chunyi Peng, Guobin Shen, Yongguang Zhang, Yanlin Li, and Kun Tan. 2007. Beepbeep: a high accuracy acoustic ranging system using cots mobile devices. In Proceedings of the 5th international conference on Embedded networked sensor systems. 1--14.
[49]
F Joseph Pompei. 2002. Sound from ultrasound: The parametric array as an audible sound source. Ph. D. Dissertation. Massachusetts Institute of Technology.
[50]
ADMP401 POWER. [n.d.]. Data Sheet ADMP401. POWER 8616 ([n.d.]), 00.
[51]
Virginia Puyana-Romero, Lilian Solange Lopez-Segura, Luigi Maffei, Ricardo Hernández-Molina, and Massimiliano Masullo. 2017. Interactive soundscapes: 360-video based immersive virtual reality in a tool for the participatory acoustic environment evaluation of urban areas. Acta acustica united with acustica 103, 4 (2017), 574--588.
[52]
Kun Qian, Chenshu Wu, Fu Xiao, Yue Zheng, Yi Zhang, Zheng Yang, and Yunhao Liu. 2018. Acousticcardiogram: Monitoring heartbeats using acoustic signals on smart devices. In IEEE INFOCOM 2018-IEEE conference on computer communications. IEEE, 1574--1582.
[53]
MRoser, C Appel, and H Ritchie. 2019. Human height. Our world in data. 2019. Availble online at: https://ourworldindata.org/human-height (2019).
[54]
Nirupam Roy, Haitham Hassanieh, and Romit Roy Choudhury. 2017. BackDoor: Making Microphones Hear Inaudible Sounds (MobiSys '17). Association for Computing Machinery, New York, NY, USA, 2--14. https://doi.org/10.1145/3081333.3081366
[55]
Nirupam Roy, Sheng Shen, Haitham Hassanieh, and Romit Roy Choudhury. 2018. Inaudible voice commands: The long-range attack and defense. In 15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18). 547--560.
[56]
Swapnil Sayan Saha, Sandeep Singh Sandha, Siyou Pei, Vivek Jain, Ziqi Wang, Yuchen Li, Ankur Sarker, and Mani Srivastava. 2022. Auritus: An open-source optimization toolkit for training and development of human movement models and filters using earables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1--34.
[57]
Adam Smith, Hari Balakrishnan, Michel Goraczko, and Nissanka Priyantha. 2004. Tracking moving devices with the cricket location system. In Proceedings of the 2nd international conference on Mobile systems, applications, and services. 190--202.
[58]
Greta C Stamper and Tiffany A Johnson. 2015. Auditory function in normal-hearing, noise-exposed human ears. Ear and hearing 36, 2 (2015), 172--184.
[59]
Michael Vorländer, Dirk Schröder, Sönke Pelzer, and Frank Wefers. 2015. Virtual reality for architectural acoustics. Journal of Building Performance Simulation 8, 1 (2015), 15--25.
[60]
Anran Wang and Shyamnath Gollakota. 2019. Millisonic: Pushing the limits of acoustic motion tracking. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--11.
[61]
Han Wang, Jiming Tang, Zhipeng Wu, and Yu Liu. 2022. A Multibeam Steerable Parametric Array Loudspeaker for Distinct Audio Content Directing. IEEE Sensors Journal 22, 13 (2022), 13640--13647.
[62]
Lei Wang, Kang Huang, Ke Sun, Wei Wang, Chen Tian, Lei Xie, and Qing Gu. 2018. Unlock with your heart: Heartbeat-based authentication on commercial mobile phones. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 2, 3 (2018), 1--22.
[63]
Weiguo Wang, Yuan He, Meng Jin, Yimiao Sun, and Xiuzhen Guo. 2023. Meta-Speaker: Acoustic Source Projection by Exploiting Air Nonlinearity. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking. 1--15.
[64]
Yi Weng, Ezra Ip, Zhongqi Pan, and Ting Wang. 2016. Advanced spatial-division multiplexed measurement systems propositions---from telecommunication to sensing applications: a review. Sensors 16, 9 (2016), 1387.
[65]
Peter J Westervelt. 1951. The theory of steady forces caused by sound waves. The Journal of the Acoustical Society of America 23, 3 (1951), 312--315.
[66]
Peter J Westervelt. 1957. Scattering of sound by sound. The Journal of the Acoustical Society of America 29, 2 (1957), 199--203.
[67]
Peter J Westervelt. 1963. Parametric acoustic array. The Journal of the acoustical society of America 35, 4 (1963), 535--537.
[68]
Wei Xu, ZhiWen Yu, Zhu Wang, Bin Guo, and Qi Han. 2019. Acousticid: gait-based human identification using acoustic signal. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1--25.
[69]
Jun Yang, Khim-Sia Tan, Woon-Seng Gan, Meng-Hwa Er, and Yong-Hong Yan. 2005. Beamwidth control in parametric acoustic array. Japanese journal of applied physics 44, 9R (2005), 6817.
[70]
Zhijian Yang and Romit Roy Choudhury. 2021. Personalizing head related transfer functions for earables. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference. 137--150.
[71]
Zhijian Yang, Yu-Lin Wei, Sheng Shen, and Romit Roy Choudhury. 2020. Ear-ar: indoor acoustic augmented reality on earphones. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--14.
[72]
Jingwei Yin, Xiao Zhang, and Yiming Zhou. 2015. Differential pattern time delay shift coding underwater acoustic communication using parametric array. The Journal of the Acoustical Society of America 137, 4 (2015), 2214--2214.
[73]
Masahide Yoneyama, Jun-ichiroh Fujimoto, Yu Kawamo, and Shoichi Sasabe. 1983. The audio spotlight: An application of nonlinear interaction of sound waves to a new type of loudspeaker design. The Journal of the Acoustical Society of America 73, 5 (1983), 1532--1536.
[74]
Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. 15--29.
[75]
Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-grained acoustic-based device-free tracking. In Proceedings of the 15th annual international conference on mobile systems, applications, and services. 15--28.
[76]
Kexin Zeng, Zhendong Li, Zichao Guo, and Zhonggang Wang. 2023. Reconfigurable and Phase-Engineered Acoustic Metasurfaces for Broadband Wavefront Manipulation. Advanced Physics Research (2023), 2300128.
[77]
Guoming Zhang, Chen Yan, Xiaoyu Ji, Tianchen Zhang, Taimin Zhang, and Wenyuan Xu. 2017. DolphinAttack: Inaudible Voice Commands. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS '17). Association for Computing Machinery, New York, NY, USA, 103--117. https://doi.org/10.1145/3133956.3134052
[78]
Hanyun Zhou, SH Huang, and Wei Li. 2020. Parametric acoustic array and its application in underwater acoustic engineering. Sensors 20, 7 (2020), 2148.

Cited By

View all
  • (2024)ExTea: An Evolutionary Algorithm-Based Approach for Enhancing Explainability in Time-Series ModelsMachine Learning and Knowledge Discovery in Databases. Applied Data Science Track10.1007/978-3-031-70381-2_27(429-446)Online publication date: 8-Sep-2024

Index Terms

  1. Visar: Projecting Virtual Sound Spots for Acoustic Augmented Reality Using Air Nonlinearity

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 8, Issue 3
    September 2024
    1782 pages
    EISSN:2474-9567
    DOI:10.1145/3695755
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 09 September 2024
    Published in IMWUT Volume 8, Issue 3

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Acoustic augmented reality
    2. Air nonlinearity
    3. Sound source projection

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • NSFC

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)236
    • Downloads (Last 6 weeks)36
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)ExTea: An Evolutionary Algorithm-Based Approach for Enhancing Explainability in Time-Series ModelsMachine Learning and Knowledge Discovery in Databases. Applied Data Science Track10.1007/978-3-031-70381-2_27(429-446)Online publication date: 8-Sep-2024

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media