skip to main content
research-article

Unobtrusive Air Leakage Estimation for Earables with In-ear Microphones

Published: 12 January 2024 Publication History

Abstract

Earables (in-ear wearables) are gaining increasing attention for sensing applications and healthcare research thanks to their ergonomy and non-invasive nature. However, air leakages between the device and the user's ear, resulting from daily activities or wearing variabilities, can decrease the performance of applications, interfere with calibrations, and reduce the robustness of the overall system. Existing literature lacks established methods for estimating the degree of air leaks (i.e., seal integrity) to provide information for the earable applications. In this work, we proposed a novel unobtrusive method for estimating the air leakage level of earbuds based on an in-ear microphone. The proposed method aims to estimate the magnitude of distortions, reflections, and external noise in the ear canal while excluding the speaker output by learning the speaker-to-microphone transfer function which allows us to perform the task unobtrusively. Using the obtained residual signal in the ear canal, we extract three features and deploy a machine-learning model for estimating the air leakage level. We investigated our system under various conditions to validate its robustness and resilience against the motion and other artefacts. Our extensive experimental evaluation shows that the proposed method can track air leakage levels under different daily activities.
"The best computer is a quiet, invisible servant."
~Mark Weiser

References

[1]
Tobias Röddiger, Christopher Clarke, Paula Breitling, Tim Schneegans, Haibin Zhao, Hans Gellersen, and Michael Beigl, "Sensing with Earables: A Systematic Literature Review and Taxonomy of Phenomena," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 6, no. 3, pp. 135:1--135:57, Sept. 2022.
[2]
Zi Wang, Sheng Tan, Linghan Zhang, Yili Ren, Zhi Wang, and Jie Yang, "Eardynamic: An ear canal deformation based continuous user authentication using in-ear wearables," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 5, no. 1, mar 2021.
[3]
Zi Wang, Yili Ren, Yingying Chen, and Jie Yang, "Toothsonic: Earable authentication via acoustic toothprint," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 6, no. 2, jul 2022.
[4]
Mark Mirtchouk, Drew Lustig, Alexandra Smith, Ivan Ching, Min Zheng, and Samantha Kleinberg, "Recognizing eating from body-worn sensors: Combining free-living and laboratory data," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 1, no. 3, sep 2017.
[5]
Yang Gao, Wei Wang, Vir V. Phoha, Wei Sun, and Zhanpeng Jin, "Earecho: Using ear canal echo for wearable authentication," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 3, no. 3, sep 2019.
[6]
Yincheng Jin, Yang Gao, Xuhai Xu, Seokmin Choi, Jiyang Li, Feng Liu, Zhengxiong Li, and Zhanpeng Jin, "Earcommand: "hearing" your silent speech commands in ear," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 6, no. 2, jul 2022.
[7]
Abdelkareem Bedri, Richard Li, Malcolm Haynes, Raj Prateek Kosaraju, Ishaan Grover, Temiloluwa Prioleau, Min Yan Beh, Mayank Goel, Thad Starner, and Gregory Abowd, "Earbit: Using wearable sensors to detect eating episodes in unconstrained environments," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 1, no. 3, sep 2017.
[8]
Yang Gao, Yincheng Jin, Jagmohan Chauhan, Seokmin Choi, Jiyang Li, and Zhanpeng Jin, "Voice in ear: Spoofing-resistant and passphrase-independent body sound authentication," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 5, no. 1, mar 2021.
[9]
Dong Ma, Andrea Ferlini, and Cecilia Mascolo, "Oesense: Employing occlusion effect for in-ear human sensing," in Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, New York, NY, USA, 2021, MobiSys '21, p. 175--187, Association for Computing Machinery.
[10]
Fahim Kawsar, Chulhong Min, Akhil Mathur, and Alesandro Montanari, "Earables for personal-scale behavior analytics," IEEE Pervasive Computing, vol. 17, no. 3, pp. 83--89, 2018.
[11]
Fahim Kawsar, Chulhong Min, Akhil Mathur, Alessandro Montanari, Utku Günay Acer, and Marc Van den Broeck, "Esense: Open earable platform for human sensing," in Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, New York, NY, USA, 2018, SenSys '18, p. 371--372, Association for Computing Machinery.
[12]
Tahera Hossain, Md Shafiqul Islam, Md Atiqur Rahman Ahad, and Sozo Inoue, "Human activity recognition using earable device," New York, NY, USA, 2019, UbiComp/ISWC '19 Adjunct, p. 81--84, Association for Computing Machinery.
[13]
Swapnil Sayan Saha, Sandeep Singh Sandha, Siyou Pei, Vivek Jain, Ziqi Wang, Yuchen Li, Ankur Sarker, and Mani Srivastava, "Auritus: An open-source optimization toolkit for training and development of human movement models and filters using earables," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 6, no. 2, jul 2022.
[14]
Andrea Ferlini, Alessandro Montanari, Cecilia Mascolo, and Robert Harle, "Head motion tracking through in-ear wearables," in Proceedings of the 1st International Workshop on Earable Computing, 2019, pp. 8--13.
[15]
Seungchul Lee, Chulhong Min, Alessandro Montanari, Akhil Mathur, Youngjae Chang, Junehwa Song, and Fahim Kawsar, "Automatic smile and frown recognition with kinetic earables," in Proceedings of the 10th Augmented Human International Conference 2019, 2019, pp. 1--4.
[16]
Dhruv Verma, Sejal Bhalla, Dhruv Sahnan, Jainendra Shukla, and Aman Parnami, "Expressear: Sensing fine-grained facial expressions with earables," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 5, no. 3, sep 2021.
[17]
Chulhong Min, Akhil Mathur, and Fahim Kawsar, "Audio-kinetic model for automatic dietary monitoring with earable devices," in Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, New York, NY, USA, 2018, MobiSys '18, p. 517, Association for Computing Machinery.
[18]
Tobias Röddiger, Daniel Wolffram, David Laubenstein, Matthias Budde, and Michael Beigl, "Towards respiration rate monitoring using an in-ear headphone inertial measurement unit," in Proceedings of the 1st International Workshop on Earable Computing, New York, NY, USA, 2019, EarComp'19, p. 48--53, Association for Computing Machinery.
[19]
Paul G. Puskarich, "Earphones with ear presence sensors.," Dec. 2017, US9648409B2.
[20]
Andrea Ferlini, Alessandro Montanari, Chulhong Min, Hongwei Li, Ugo Sassi, and Fahim Kawsar, "In-ear ppg for vital signs," IEEE Pervasive Computing, vol. 21, no. 1, pp. 65--74, 2022.
[21]
Hoang Truong, Alessandro Montanari, and Fahim Kawsar, "Non-invasive blood pressure monitoring with multi-modal in-ear sensing," in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6--10.
[22]
Ananta Narayanan Balaji, Andrea Ferlini, Fahim Kawsar, and Alessandro Montanari, "Stereo-bp: Non-invasive blood pressure sensing with earables," in Proceedings of the 24th International Workshop on Mobile Computing Systems and Applications, 2023, pp. 96--102.
[23]
Dan Li, Xiaoyuan Ma, and Jianming Wei, "Beyond pulse: Can the in-ear photoplethysmogram signal understand what you say?," in Proceedings of the 7th International Workshop on Real-World Embedded Wireless Systems and Networks, New York, NY, USA, 2018, RealWSN'18, p. 25--30, Association for Computing Machinery.
[24]
Seokmin Choi, Yang Gao, Yincheng Jin, Se jun Kim, Jiyang Li, Wenyao Xu, and Zhanpeng Jin, "Ppgface: Like what you are watching? earphones can "feel" your facial expressions," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 6, no. 2, jul 2022.
[25]
Kayla-Jade Butkow, Ting Dang, Andrea Ferlini, Dong Ma, and Cecilia Mascolo, "heart: Motion-resilient heart rate monitoring with in-ear microphones," arXiv preprint arXiv:2108.09393, 2021.
[26]
Chulhong Min, Akhil Mathur, Alessandro Montanari, and Fahim Kawsar, "An early characterisation of wearing variability on motion signals for wearables," in Proceedings of the 2019 ACM International Symposium on Wearable Computers, 2019, pp. 166--168.
[27]
Dale Purves and S. Mark Williams, Eds., Neuroscience, Sinauer Associates, Sunderland, Mass, 2nd ed edition, 2001.
[28]
Ylva Björk and Ebba Wilhelmsson, "Linearisation of micro loudspeakers using adaptive control," 2014.
[29]
Wei-Chen Hsiao, Yung-Ching Chen, and Yi-Wen Liu, "Measuring distortion-product otoacoustic emission with a single loudspeaker in the ear: Stimulus design and signal processing techniques," Frontiers in Digital Health, vol. 3, 2021.
[30]
Daniel Stoller, Sebastian Ewert, and Simon Dixon, "Wave-u-net: A multi-scale neural network for end-to-end audio source separation," arXiv preprint arXiv:1806.03185, 2018.
[31]
Martin Schetzen, "The volterra and wiener theories of nonlinear systems," 1980.
[32]
Diederik P. Kingma and Jimmy Ba, "Adam: A method for stochastic optimization," 2017.
[33]
Sankha Subhra Bhattacharjee and Nithin V. George, "Nearest kronecker product decomposition based linear-in-the-parameters nonlinear filters," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2111--2122, 2021.
[34]
Pratik Shah, Ian Lewis, Steven Grant, and Sylvain Angrignon, "Nonlinear acoustic echo cancellation using voltage and current feedback," IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 23, no. 10, pp. 1589--1599, oct 2015.
[35]
H. Gustav Mueller, Kathryn E. Bright, and Jerry L. Northern, "Studies of the Hearing Aid Occlusion Effect," Seminars in Hearing, vol. 17, no. 1, pp. 21--31, Feb. 1996.
[36]
Hyafil Laurent and Ronald L Rivest, "Constructing optimal binary decision trees is np-complete," Information processing letters, vol. 5, no. 1, pp. 15--17, 1976.
[37]
Douglas C Montgomery, Elizabeth A Peck, and G Geoffrey Vining, Introduction to linear regression analysis, John Wiley & Sons, 2021.
[38]
Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf, "Support vector machines," IEEE Intelligent Systems and their applications, vol. 13, no. 4, pp. 18--28, 1998.
[39]
Carl Edward Rasmussen, Christopher KI Williams, et al., Gaussian processes for machine learning, vol. 1, Springer, 2006.
[40]
Thomas G Dietterich, "Ensemble methods in machine learning," in Multiple Classifier Systems: First International Workshop, MCS 2000 Cagliari, Italy, June 21--23, 2000 Proceedings 1. Springer, 2000, pp. 1--15.
[41]
Tara P. McAlexander, Robyn RM Gershon, and Richard L. Neitzel, "Street-level noise in an urban setting: assessment and contribution to personal exposure," Environmental Health, vol. 14, no. 1, pp. 18, Feb. 2015.
[42]
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey, "Sdr -- half-baked or well done?," ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626--630, 2018.
[43]
Claude Sammut and Geoffrey I. Webb, Eds., Leave-One-Out Cross-Validation, pp. 600--601, Springer US, Boston, MA, 2010.
[44]
Jay Prakash, Zhijian Yang, Yu-Lin Wei, Haitham Hassanieh, and Romit Roy Choudhury, "Earsense: Earphones as a teeth activity sensor," in Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, New York, NY, USA, 2020, MobiCom '20, Association for Computing Machinery.
[45]
Arthur Moss, Hyunjong Lee, Lei Xun, Chulhong Min, Fahim Kawsar, and Alessandro Montanari, "Ultra-low power dnn accelerators for iot: Resource characterization of the max78000," in Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, 2022, pp. 934--940.
[46]
Igor Kofman, "Earbud insertion sensing method with infrared technology," Dec. 2019, US10257602B2.
[47]
Charles L. Greenlee, "Ultrasonic proximity sensors, and related systems and methods," Dec. 2021, US10932062B2.
[48]
Victor Tiscareno, "Electronic device and headset with speaker seal evaluation capabilities," 2013, US8401200B2.
[49]
Aleksey S. Khenkin, "System and method for evaluating an ear seal using normalization," 2021, US20210400410A1.
[50]
Aleksey S. Khenkin, "System and method for evaluating an ear seal using external stimulus," 2021, US20210400408A1.
[51]
Ryan M. Moriyama Hongfeng Wang, Chen Na, "Acoustic in ear detection for a hearable device," Dec. 2019, US10491981B1.
[52]
Marko Orescanin Mehmet Ergezer, Joseph H. Cattell, "On/off head detection of personal acoustic device," Dec. 2018, US9860626B2.
[53]
Brenton STEELE, "Systems and methods for on ear detection of headsets," 2021, US20220223137A1.
[54]
Esge B. AndersenAndre L. Goldstein, "Off-ear detector for personal listening device with active noise control," Dec. 2016, US9486823B2.
[55]
Xiaoran Fan, Longfei Shangguan, Siddharth Rupavatharam, Yanyong Zhang, Jie Xiong, Yunfei Ma, and Richard Howard, "Headfi: Bringing intelligence to all headphones," in Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, New York, NY, USA, 2021, MobiCom '21, p. 147--159, Association for Computing Machinery.
[56]
John Paul Lesso, "Ear proximity detection," Dec. 2021, US20210064729A1.
[57]
"AirPods Pro (2nd generation)," https://www.apple.com/uk/airpods-pro/, Accessed: 2022-09-10.
[58]
E. H. Berger, "Methods of measuring the attenuation of hearing protection devices," The Journal of the Acoustical Society of America, vol. 79, no. 6, pp. 1655--1687, 1986.
[59]
Toshiyuki Ando, Yuki Kubo, Buntarou Shizuki, and Shin Takahashi, "Canalsense: Face-related movement recognition system based on sensing air pressure in ear canals," in Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, New York, NY, USA, 2017, UIST '17, p. 679--689, Association for Computing Machinery.
[60]
M. E. Smith, D.J. Scoffings, and J. R. Tysome, "Imaging of the Eustachian tube and its function: a systematic review," Neuroradiology, vol. 58, no. 6, pp. 543--556, June 2016.
[61]
Jan L. Mayes and Daniel Fink, "Personal audio system use can harm auditory health," The Journal of the Acoustical Society of America, vol. 149, pp. A124--A124, 04 2021.
[62]
Ruilin Xu, Rundi Wu, Yuko Ishiwaka, Carl Vondrick, and Changxi Zheng, "Listening to sounds of silence for speech denoising," in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 9633--9648, Curran Associates, Inc.
[63]
Douglas H. Keefe, Jay C. Bulen, Kathy Hoberg Arehart, and Edward M. Burns, "Ear-canal impedance and reflection coefficient in human infants and adults," The Journal of the Acoustical Society of America, vol. 94, no. 5, pp. 2617--2638, 1993.

Cited By

View all
  • (2024)LR-Auth: Towards Practical Implementation of Implicit User Authentication on EarbudsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997938:4(1-27)Online publication date: 21-Nov-2024
  • (2024)Functional Now, Wearable Later: Examining the Design Practices of Wearable TechnologistsProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676615(71-81)Online publication date: 5-Oct-2024
  • (2024)OptiBreathe: An Earable-based PPG System for Continuous Respiration Rate, Breathing Phase, and Tidal Volume MonitoringProceedings of the 25th International Workshop on Mobile Computing Systems and Applications10.1145/3638550.3641136(99-106)Online publication date: 28-Feb-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 7, Issue 4
December 2023
1613 pages
EISSN:2474-9567
DOI:10.1145/3640795
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 January 2024
Published in IMWUT Volume 7, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. ear canal sealing
  2. transfer function approximation
  3. unobtrusive

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)333
  • Downloads (Last 6 weeks)17
Reflects downloads up to 18 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)LR-Auth: Towards Practical Implementation of Implicit User Authentication on EarbudsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997938:4(1-27)Online publication date: 21-Nov-2024
  • (2024)Functional Now, Wearable Later: Examining the Design Practices of Wearable TechnologistsProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676615(71-81)Online publication date: 5-Oct-2024
  • (2024)OptiBreathe: An Earable-based PPG System for Continuous Respiration Rate, Breathing Phase, and Tidal Volume MonitoringProceedings of the 25th International Workshop on Mobile Computing Systems and Applications10.1145/3638550.3641136(99-106)Online publication date: 28-Feb-2024
  • (2024)Towards Enabling DPOAE Estimation on Single-Speaker EarbudsICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10446436(246-250)Online publication date: 14-Apr-2024

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media