Zhijian Yang
ABSTRACT
This paper aims to use modern earphones as a platform for acoustic augmented reality (AAR). We intend to play 3D audio-annotations in the user's ears as she moves and looks at AAR objects in the environment. While companies like Bose and Microsoft are beginning to release such capabilities, they are intended for outdoor environments. Our system aims to explore the challenges indoors, without requiring any infrastructure deployment. Our core idea is two-fold. (1) We jointly use the inertial sensors (IMUs) in earphones and smartphones to estimate a user's indoor location and gazing orientation. (2) We play 3D sounds in the earphones and exploit the human's responses to (re)calibrate errors in location and orientation. We believe this fusion of IMU and acoustics is novel, and could be an important step towards indoor AAR. Our system, Ear-AR, is tested on 7 volunteers invited to an AAR exhibition - like a museum - that we set up in our building's lobby and lab. Across 60 different test sessions, the volunteers browsed different subsets of 24 annotated objects as they walked around. Results show that Ear-AR plays the correct audio-annotations with good accuracy. The user-feedback is encouraging and points to further areas of research and applications.
- 2019. 3Dio: Professional Binaural Microphones. Retrieved Sept 18, 2019 from https://3diosound.com/Google Scholar
- 2019. Beats Solo3 Wireless - Beats by Dre. Retrieved Sep 13, 2019 from https://www.beatsbydre.com/uk/headphones/solo3-wirelessGoogle Scholar
- 2019. Bragi Earbuds. Retrieved Sept 18, 2019 from https://bragi.com/Google Scholar
- 2019. Jabra Wireless Workout Headphones. Retrieved Sept 18, 2019 from https://www.jabra.com/sports-headphones/jabra-sport-coach-wireless#/Google Scholar
- 2019. Listen HRTF Database. Retrieved Sep 19, 2019 from http://recherche.ircam.fr/equipes/salles/listen/Google Scholar
- 2019. Samsung Gear IconX. Retrieved Sept 18, 2019 from https://www.samsung.com/global/galaxy/gear-iconx/Google Scholar
- 2019. Sennheiser's short film shows the power of binaural audio. Retrieved Sept 18, 2019 from https://thenextweb.com/plugged/2018/09/17/sennheisers-short-film-shows-the-power-of-binaural-audio/Google Scholar
- 2020. Airpods Pro. Retrieved Jul 21, 2020 from https://www.apple.com/airpods-pro/Google Scholar
- 2020. Apple spatial audio: what is it? How do you get it? Retrieved Jul 21, 2020 from https://www.whathifi.com/advice/what-is-apple-spatial-audioGoogle Scholar
- Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D tracking via body radio reflections. In 11th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 14). 317--329.Google Scholar
- Muhammad Haris Afzal, Valérie Renaudin, and Gérard Lachapelle. 2010. Assessment of indoor magnetic field anomalies using multiple magnetometers. In ION GNSS, Vol. 10. 21--24.Google Scholar
- Hyun-Sung An, Gregory C Jones, Seoung-Ki Kang, Gregory J Welk, and Jung-Min Lee. 2017. How valid are wearable physical activity trackers for measuring steps? European journal of sport science 17, 3 (2017), 360--368.Google Scholar
- Apple. 2020. Apple Airpods. https://www.apple.com/airpods/ [Online; accessed 20-March-2020].Google Scholar
- C Ascher, C Kessler, M Wankerl, and GF Trommer. 2010. Dual IMU indoor navigation with particle filter based map-matching on a smartphone. In 2010 International Conference on Indoor Positioning and Indoor Navigation. IEEE, 1--5.Google ScholarCross Ref
- Haitao Bao and Wai-Choong Wong. 2013. Improved PCA based step direction estimation for dead-reckoning localization. In 2013 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery. IEEE, 325--331.Google ScholarDigital Library
- Stéphane Beauregard. 2006. A helmet-mounted pedestrian dead reckoning system. In 3rd International Forum on Applied Wearable Computing 2006. VDE, 1--11.Google Scholar
- Durand R Begault and Leonard J Trejo. 2000. 3-D sound for virtual reality and multimedia. (2000).Google Scholar
- Alastair R Beresford and Frank Stajano. 2003. Location privacy in pervasive computing. IEEE Pervasive computing 1 (2003), 46--55.Google Scholar
- Johann Borenstein, Lauro Ojeda, and Surat Kwanmuang. 2009. Heuristic reduction of gyro drift in IMU-based personnel tracking systems. In Optics and Photonics in Global Homeland Security V and Biometric Technology for Human Identification VI, Vol. 7306. International Society for Optics and Photonics, 73061H.Google Scholar
- BOSE. 2020. Bose Developer Portal | NaviGuide Around Your World: Heads Up, Hands Free. https://developer.bose.com/bose-ar/get-inspired/naviguide-around-your-world-heads-hands-free [Online; accessed 16-March-2020].Google Scholar
- BOSE. 2020. Bose Developer Portal | Tap Into Otocast, Your Friendly AR-enabled Tour Guide. https://developer.bose.com/bose-ar/get-inspired/tap-otocast-your-friendly-ar-enabled-tour-guide [Online; accessed 17-March-2020].Google Scholar
- C Phillip Brown and Richard O Duda. 1998. A structural model for binaural sound synthesis. IEEE transactions on speech and audio processing 6, 5 (1998), 476--488.Google Scholar
- Christophe Combettes and Valerie Renaudin. 2017. Walking direction estimation based on statistical modeling of human gait features with handheld MIMU. IEEE/ASME Transactions on Mechatronics 22, 6 (2017), 2502--2511.Google ScholarCross Ref
- Alejandro Correa, Estefania Munoz Diaz, Dina Bousdar Ahmed, Antoni Morell, and Jose Lopez Vicario. 2016. Advanced pedestrian positioning system to smartphones and smartwatches. Sensors 16, 11 (2016), 1903.Google ScholarCross Ref
- Daniel de Godoy, Bashima Islam, Stephen Xia, Md Tamzeed Islam, Rishikanth Chandrasekaran, Yen-Chun Chen, Shahriar Nirjon, Peter R Kinget, and Xiaofan Jiang. 2018. Paws: A wearable acoustic system for pedestrian safety. In 2018 IEEE/ACM Third International Conference on Internet-of-Things Design and Implementation (IoTDI). IEEE, 237--248.Google ScholarCross Ref
- Zhi-An Deng, Guofeng Wang, Ying Hu, and Di Wu. 2015. Heading estimation for indoor pedestrian navigation using a smartphone in the pocket. Sensors 15, 9 (2015), 21518--21536.Google ScholarCross Ref
- Richard O Duda. 1993. Modeling head related transfer functions. In Proceedings of 27th Asilomar Conference on Signals, Systems and Computers. IEEE, 996--1000.Google ScholarCross Ref
- Google. 2020. Pixel Buds. https://store.google.com/product/pixel_buds [Online; accessed 20-March-2020].Google Scholar
- Dorte Hammershøi and Henrik Møller. 2002. Methods for binaural recording and reproduction. Acta Acustica united with Acustica 88, 3 (2002), 303--311.Google Scholar
- Fabian Höflinger, Rui Zhang, and Leonhard M Reindl. 2012. Indoor-localization system using a micro-inertial measurement unit (imu). In 2012 European Frequency and Time Forum. IEEE, 443--447.Google Scholar
- JC Hung, JR Thacher, and HV White. 1989. Calibration of accelerometer triad of an IMU with drifting Z-accelerometer bias. In Proceedings of the IEEE National Aerospace and Electronics Conference. IEEE, 153--158.Google ScholarCross Ref
- Tong-Hun Hwang, Julia Reh, Alfred Effenberg, and Holger Blume. 2016. Real-time gait event detection using a single head-worn inertial measurement unit. In 2016 IEEE 6th International Conference on Consumer Electronics-Berlin (ICCE-Berlin). IEEE, 28--32.Google ScholarCross Ref
- Tong-Hun Hwang, Julia Reh, Alfred O Effenberg, and Holger Blume. 2018. Realtime gait analysis using a single head-worn inertial measurement unit. IEEE Transactions on Consumer Electronics 64, 2 (2018), 240--248.Google ScholarCross Ref
- Ian Dickson. 2020. Bose and HERE - creating the future of AR. https://360.here.com/bose-and-here-creating-the-future-of-ar [Online; accessed 24-March-2020].Google Scholar
- Ian Dickson. 2020. Bose and HERE - creating the future of AR. https://developer.bose.com/content/here-maps-and-bose-ar-integration [Online; accessed 24-March-2020].Google Scholar
- Lloyd A Jeffress. 1948. A place theory of sound localization. Journal of comparative and physiological psychology 41, 1 (1948), 35.Google ScholarCross Ref
- Antonio R Jimenez, Fernando Seco, Carlos Prieto, and Jorge Guevara. 2009. A comparison of pedestrian dead-reckoning algorithms using a low-cost MEMS IMU. In 2009 IEEE International Symposium on Intelligent Signal Processing. IEEE, 37--42.Google ScholarCross Ref
- Antonio Ramón Jiménez, Fernando Seco, José Carlos Prieto, and Jorge Guevara. 2010. Indoor pedestrian navigation using an INS/EKF framework for yaw drift reduction and a foot-mounted IMU. In 2010 7th Workshop on Positioning, Navigation and Communication. IEEE, 135--143.Google ScholarCross Ref
- Haojian Jin, Jingxian Wang, Zhijian Yang, Swarun Kumar, and Jason Hong. 2018. Rf-wear: Towards wearable everyday skeleton tracking using passive rfids. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 369--372.Google ScholarDigital Library
- Haojian Jin, Jingxian Wang, Zhijian Yang, Swarun Kumar, and Jason Hong. 2018. Wish: Towards a wireless shape-aware world using passive rfids. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. 428--441.Google ScholarDigital Library
- Haojian Jin, Zhijian Yang, Swarun Kumar, and Jason I Hong. 2018. Towards wearable everyday body-frame tracking using passive rfids. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--23.Google ScholarDigital Library
- Wonho Kang and Youngnam Han. 2014. SmartPDR: Smartphone-based pedestrian dead reckoning for indoor localization. IEEE Sensors journal 15, 5 (2014), 2906--2916.Google ScholarCross Ref
- Kleomenis Katevas, Hamed Haddadi, and Laurissa Tokarchuk. 2016. Sensingkit: Evaluating the sensor power consumption in ios devices. In 2016 12th International Conference on Intelligent Environments (IE). IEEE, 222--225.Google ScholarCross Ref
- Maan Khedr and Nasser El-Sheimy. 2017. A smartphone step counter using IMU and magnetometer for navigation and health monitoring applications. Sensors 17, 11 (2017), 2573.Google ScholarCross Ref
- Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015. Spotfi: Decimeter level localization using wifi. In ACM SIGCOMM computer communication review, Vol. 45. ACM, 269--282.Google ScholarDigital Library
- Zeqi Lai, Y Charlie Hu, Yong Cui, Linhui Sun, Ningwei Dai, and Hung-Sheng Lee. 2019. Furion: Engineering high-quality immersive virtual reality on today's mobile devices. IEEE Transactions on Mobile Computing (2019).Google Scholar
- Teesid Leelasawassuk, Dima Damen, and Walterio W Mayol-Cuevas. 2015. Estimating visual attention from a head mounted IMU. In Proceedings of the 2015 ACM International Symposium on Wearable Computers. 147--150.Google ScholarDigital Library
- Tobias Lentz, Dirk Schröder, Michael Vorländer, and Ingo Assenmacher. 2007. Virtual reality system with integrated sound field simulation and reproduction. EURASIP journal on advances in signal processing 2007, 1 (2007), 070540.Google Scholar
- Hui Liu, Houshang Darabi, Pat Banerjee, and Jing Liu. 2007. Survey of wireless indoor positioning techniques and systems. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, 6 (2007), 1067--1080.Google ScholarDigital Library
- Yu Liu, Yanping Chen, Lili Shi, Zengshan Tian, Mu Zhou, and Lingxia Li. 2015. Accelerometer based joint step detection and adaptive step length estimation algorithm using handheld devices. Journal of Communications 10, 7 (2015), 520--525.Google Scholar
- Yunhao Liu, Zheng Yang, Xiaoping Wang, and Lirong Jian. 2010. Location, localization, and localizability. Journal of Computer Science and Technology 25, 2 (2010), 274--297.Google ScholarCross Ref
- Kieran Mansley, Alastair R Beresford, and David Scott. 2004. The carrot approach: encouraging use of location systems. In International Conference on Ubiquitous Computing. Springer, 366--383.Google ScholarCross Ref
- Wenguang Mao, Mei Wang, and Lili Qiu. 2018. Aim: acoustic imaging on a mobile. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 468--481.Google ScholarDigital Library
- Wenguang Mao, Zaiwei Zhang, Lili Qiu, Jian He, Yuchen Cui, and Sangki Yun. 2017. Indoor follow me drone. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 345--358.Google ScholarDigital Library
- Microsoft. 2020. Microsoft Soundscape: A Map Delivered in 3D Sound - Microsoft Research. https://www.microsoft.com/en-us/research/video/microsoft-soundscape-map-delivered-3d-sound/ [Online; accessed 17-March-2020].Google Scholar
- Pauli Minnaar, S Krarup Olesen, Flemming Christensen, and Henrik Møller. 2001. Localization with binaural recordings from artificial and human heads. Journal of the Audio Engineering Society 49, 5 (2001), 323--336.Google Scholar
- Henrik Møller. 1992. Fundamentals of binaural technology. Applied acoustics 36, 3--4 (1992), 171--218.Google Scholar
- Rajalakshmi Nandakumar, Vikram Iyer, and Shyamnath Gollakota. 2018. 3D Localization for Sub-Centimeter Sized Devices. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. ACM, 108--119.Google ScholarDigital Library
- Anh Nguyen, Raghda Alqurashi, Zohreh Raghebi, Farnoush Banaei-Kashani, Ann C Halbower, and Tam Vu. 2016. A lightweight and inexpensive in-ear sensing system for automatic whole-night sleep stage monitoring. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM. ACM, 230--244.Google ScholarDigital Library
- David R Perrott and AD Musicant. 1977. Minimum auditory movement angle: Binaural localization of moving sound sources. The Journal of the Acoustical Society of America 62, 6 (1977), 1463--1466.Google ScholarCross Ref
- Swadhin Pradhan, Ghufran Baig, Wenguang Mao, Lili Qiu, Guohai Chen, and Bo Yang. 2018. Smartphone-based Acoustic Indoor Space Mapping. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 75.Google ScholarDigital Library
- Jay Prakash, Zhijian Yang, Yu-Lin Wei, and Romit Roy Choudhury. 2019. STEAR: Robust Step Counting from Earables. In Proceedings of the 1st International Workshop on Earable Computing. 36--41.Google ScholarDigital Library
- Sujatha Rajagopal. 2008. Personal dead reckoning system with shoe mounted inertial sensors. Master's Degree Project, Stockholm, Sweden (2008).Google Scholar
- Eike Rehder, Horst Kloeden, and Christoph Stiller. 2014. Head detection and orientation estimation for pedestrian safety. In 17th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 2292--2297.Google ScholarCross Ref
- Zhi-Hong Ren, Gui-Lin Wang, and David Ho. 2006. Mobile phone with pedometer. US Patent App. 11/136,265.Google Scholar
- Rico Malvar. 2020. A new Soundscape experience with Bose Frames - Microsoft Accessibility Blog. https://blogs.microsoft.com/accessibility/soundscape-boseframes/ [Online; accessed 17-March-2020].Google Scholar
- Nirupam Roy, He Wang, and Romit Roy Choudhury. 2014. I am a smartphone and I can tell my user's walking direction. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 329--342.Google ScholarDigital Library
- Sheng Shen, Daguan Chen, Yu-Lin Wei, Zhijian Yang, and Romit Roy Choudhury. 2020. Voice localization using nearby wall reflections. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--14.Google ScholarDigital Library
- SH Shin, CG Park, JW Kim, HS Hong, and JM Lee. 2007. Adaptive step length estimation algorithm using low-cost MEMS inertial sensors. In 2007 ieee sensors applications symposium. IEEE, 1--5.Google Scholar
- Anran Wang and Shyamnath Gollakota. 2019. MilliSonic: Pushing the Limits of Acoustic Motion Tracking. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 18.Google ScholarDigital Library
- He Wang, Souvik Sen, Ahmed Elgohary, Moustafa Farid, Moustafa Youssef, and Romit Roy Choudhury. 2012. No Need to War-drive: Unsupervised Indoor Localization. In Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services (MobiSys '12). ACM, New York, NY, USA, 197--210. Google ScholarDigital Library
- Harvey Weinberg. 2002. Using the ADXL202 in pedometer and personal navigation applications. Analog Devices AN-602 application note 2, 2 (2002), 1--6.Google Scholar
- Jens Windau and Laurent Itti. 2016. Walking compass with head-mounted IMU sensor. In 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 5542--5547.Google ScholarDigital Library
- Zhuoling Xiao, Hongkai Wen, Andrew Markham, and Niki Trigoni. 2014. Robust pedestrian dead reckoning (R-PDR) for arbitrary mobile device placement. In 2014 International Conference on Indoor Positioning and Indoor Navigation (IPIN). IEEE, 187--196.Google ScholarCross Ref
- Yaxiong Xie, Jie Xiong, Mo Li, and Kyle Jamieson. 2019. mD-Track: Leveraging Multi-Dimensionality for Passive Indoor Wi-Fi Tracking. In The 25th Annual International Conference on Mobile Computing and Networking. ACM, 1--16.Google ScholarDigital Library
- Jie Xiong and Kyle Jamieson. 2013. Arraytrack: A fine-grained indoor location system. In Presented as part of the 10th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 13). 71--84.Google ScholarDigital Library
- Zheng Yang, Chenshu Wu, and Yunhao Liu. 2012. Locating in fingerprint space: wireless indoor localization with little human intervention. In Proceedings of the 18th annual international conference on Mobile computing and networking. ACM, 269--280.Google ScholarDigital Library
- Ali Yassin, Youssef Nasser, Mariette Awad, Ahmed Al-Dubai, Ran Liu, Chau Yuen, Ronald Raulefs, and Elias Aboutanios. 2016. Recent advances in indoor localization: A survey on theoretical approaches and applications. IEEE Communications Surveys & Tutorials 19, 2 (2016), 1327--1346.Google ScholarDigital Library
- Ji Zhang and Sanjiv Singh. 2017. Low-drift and real-time lidar odometry and mapping. Autonomous Robots 41, 2 (2017), 401--416.Google ScholarDigital Library
- Yanzi Zhu, Zhujun Xiao, Yuxin Chen, Zhijing Li, Max Liu, Ben Y. Zhao, and Haitao Zheng. 2018. Et Tu Alexa? When Commodity WiFi Devices Turn into Adversarial Motion Sensors. arXiv e-prints, Article arXiv:1810.10109 (Oct 2018), arXiv:1810.10109 pages. arXiv:cs.CR/1810.10109Google Scholar
Index Terms
- Ear-AR: indoor acoustic augmented reality on earphones
Recommendations
Natural listening over headphones in augmented reality using adaptive filtering techniques
Augmented reality (AR), which composes of virtual and real world environments, is becoming one of the major topics of research interest due to the advent of wearable devices. Today, AR is commonly used as assistive display to enhance the perception of ...
Personalizing head related transfer functions for earables
SIGCOMM '21: Proceedings of the 2021 ACM SIGCOMM 2021 ConferenceHead related transfer functions (HRTF) describe how sound signals bounce, scatter, and diffract when they arrive at the head, and travel towards the ears. HRTFs produce distinct sound patterns that ultimately help the brain infer the spatial properties ...
Local positioning system for human tracking
iCREATe '08: Proceedings of the 2nd International Convention on Rehabilitation Engineering & Assistive TechnologyWe propose a local positioning system that tracks human's movement. The system uses two 3-axis accelerometers mounted at the human's feet to detect steps. An electronic compass is used to obtain an absolute heading. Both step and heading data are ...
Comments