Yurina Mizuho
Takashi Amesaka
ABSTRACT
Earphones are now used for longer hours than before with the advancement in wireless technology and miniaturization. In addition, the application of earphones has become more diverse, and opportunities to access highly confidential information through them have increased. We propose a method comprising a hearable device equipped with a small camera for user authentication from ear images. This method improves the security of the hearable device. Ear images are first captured with the camera. The ear regions in the images are then extracted using a mask region-based convolutional neural network. Finally, the user is identified using histograms of oriented gradient features and a support vector machine (SVM). Our method was able to identify 18 participants with an accuracy of 84.1%. Users are authenticated through unsupervised anomaly detection using an autoencoder with an error rate of 8.36%. This method facilitates hands- and eye-free operations without requiring any explicit authentication action by the user.
Supplemental Material
- Ayman Abaza, Arun Ross, Christina Hebert, Mary Ann F. Harrison, and Mark S. Nixon. 2013. A survey on ear biometrics. ACM Comput. Surv. 45, 2, Article 22 (Mar 2013), 1–35 pages. https://doi.org/10.1145/2431211.2431221Google Scholar
Digital Library
- Kaustubh Adhikari, Guillermo Reales, Andrew J. P. Smith, Esra Konka, Jutta Palmen, Mirsha Quinto-Sanchez, Victor Acuña-Alonzo, Claudia Jaramillo, William Arias, Macarena Fuentes, María Pizarro, Rodrigo Barquera Lozano, Gastón Macín Pérez, Jorge Gómez-Valdés, Hugo Villamil-Ramírez, Tábita Hunemeier, Virginia Ramallo, Caio C. Silva de Cerqueira, Malena Hurtado, Valeria Villegas, Vanessa Granja, Carla Gallo, Giovanni Poletti, Lavinia Schuler-Faccini, Francisco M. Salzano, Maria-Cátira Bortolini, Samuel Canizales-Quinteros, Francisco Rothhammer, Gabriel Bedoya, Rosario Calderón, Javier Rosique, Michael Cheeseman, Mahmood F. Bhutta, Steve E. Humphries, Rolando Gonzalez-José, Denis Headon, David Balding, and Andrés Ruiz-Linares. 2015. A genome-wide association study identifies multiple loci for variation in human ear morphology. Nature Communications 6, 1 (24 Jun 2015), 7500. https://doi.org/10.1038/ncomms8500Google ScholarCross Ref
- Anton.H.M. Akkermans, Tom.A.M. Kevenaar, and Daniël.W.E. Schobben. 2005. Acoustic ear recognition for person identification. In Fourth IEEE Workshop on Automatic Identification Advanced Technologies. IEEE, Buffalo, NY, USA, 219–223. https://doi.org/10.1109/AUTOID.2005.11Google ScholarDigital Library
- Takashi Amesaka, Hiroki Watanabe, Masanori Sugimoto, Yuta Sugiura, and Buntarou Shizuki. 2023. User Authentication Method for Hearables Using Sound Leakage Signals. In Proceedings of the 2023 ACM International Symposium on Wearable Computers (Cancun, Quintana Roo, Mexico) (ISWC ’23). Association for Computing Machinery, New York, NY, USA, 119–123. https://doi.org/10.1145/3594738.3611376Google ScholarDigital Library
- Apple. 2019. AirPods Pro. https://www.apple.com/airpods-pro/. (Accessed on 03/22/2024).Google Scholar
- Saurav Keshari Aryal, Teanna Barrett, and Gloria Washington. 2023. Evaluating Novel Mask-RCNN Architectures for Ear Mask Segmentation. In Proceedings of the 2022 11th International Conference on Bioinformatics and Biomedical Science (Nanning, China) (ICBBS ’22). Association for Computing Machinery, New York, NY, USA, 46––53. https://doi.org/10.1145/3571532.3571538Google ScholarDigital Library
- Matic Bizjak, Peter Peer, and Žiga Emeršič. 2019. Mask R-CNN for Ear Detection. In 2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). 1624–1628. https://doi.org/10.23919/MIPRO.2019.8756760Google ScholarCross Ref
- Mark Burge and Wilhelm Burger. 1996. Ear biometrics. Springer US, Boston, MA, Chapter 13, 273–285. https://doi.org/10.1007/0-306-47044-6_13Google ScholarCross Ref
- Yetong Cao, Cai Chao, Fan Li, Zhe Chen, and Jun Luo. 2023. HeartPrint: Passive Heart Sounds Authentication Exploiting In-Ear Microphones. In Proceedings of the IEEE INFOCOM 2023 - IEEE Conference on Computer Communications. IEEE, New York City, USA.Google ScholarCross Ref
- Tuochao Chen, Benjamin Steeper, Kinan Alsheikh, Songyun Tao, François Guimbretière, and Cheng Zhang. 2020. C-Face: Continuously Reconstructing Facial Expressions by Deep Learning Contours of the Face with Ear-Mounted Miniature Cameras. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 112––125. https://doi.org/10.1145/3379337.3415879Google ScholarDigital Library
- Seokmin Choi, Junghwan Yim, Yincheng Jin, Yang Gao, Jiyang Li, and Zhanpeng Jin. 2023. EarPPG: Securing Your Identity with Your Ears. In Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23). Association for Computing Machinery, New York, NY, USA, 835–849. https://doi.org/10.1145/3581641.3584070Google ScholarDigital Library
- Romit Roy Choudhury. 2021. Earable computing: A new area to think about. In Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications(HotMobile ’21). Association for Computing Machinery, New York, NY, USA, 147–153. https://doi.org/10.1145/3446382.3450216Google ScholarDigital Library
- Naser Damer and Benedikt Führer. 2012. Ear Recognition Using Multi-Scale Histogram of Oriented Gradients. In 2012 Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing. 21–24. https://doi.org/10.1109/IIH-MSP.2012.12Google ScholarDigital Library
- Abhishek Dutta and Andrew Zisserman. 2019. The VIA Annotation Software for Images, Audio and Video. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3343031.3350535Google ScholarDigital Library
- Susan El-Naggar, Ayman Abaza, and Thirimachos Bourlai. 2020. Ear Detection in the Wild Using Faster R-CNN Deep Learning. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (Barcelona, Spain) (ASONAM ’18). IEEE Press, 1124––1130.Google Scholar
- Žiga Emeršič, Vitomir Štruc, and Peter Peer. 2017. Ear recognition: More than a survey. 255 (2017), 26–39. https://doi.org/10.1016/j.neucom.2016.08.139Google ScholarCross Ref
- Xinrui Fang, Chengshuo Xia, and Yuta Sugiura. 2021. FacialPen: Using Facial Detection to Augment Pen-Based Interaction. In Asian CHI Symposium 2021 (Yokohama, Japan) (Asian CHI Symposium 2021). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3429360.3467672Google ScholarDigital Library
- Andrea Ferlini, Dong Ma, Robert Harle, and Cecilia Mascolo. 2021. EarGate: Gait-based user identification with in-ear microphones. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking(MobiCom ’21). Association for Computing Machinery, New York, NY, USA, 337–349. https://doi.org/10.1145/3447993.3483240Google ScholarDigital Library
- Andrea Ferlini, Alessandro Montanari, Chulhong Min, Hongwei Li, Ugo Sassi, and Fahim Kawsar. 2022. In-Ear PPG for Vital Signs. IEEE Pervasive Computing 21, 1 (2022), 65–74. https://doi.org/10.1109/MPRV.2021.3121171Google ScholarCross Ref
- Yang Gao, Yincheng Jin, Jagmohan Chauhan, Seokmin Choi, Jiyang Li, and Zhanpeng Jin. 2021. Voice In Ear: Spoofing-Resistant and Passphrase-Independent Body Sound Authentication. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 1, Article 12 (Mar 2021), 1–25 pages. https://doi.org/10.1145/3448113Google ScholarDigital Library
- Yang Gao, Wei Wang, Vir V. Phoha, Wei Sun, and Zhanpeng Jin. 2019. EarEcho: Using ear canal echo for wearable authentication. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article 81 (Sep 2019), 1–24 pages. https://doi.org/10.1145/3351239Google ScholarDigital Library
- Neil J. Grabham, Matthew A. Swabey, Paul Chambers, Mark E. Lutman, Neil M. White, John E. Chad, and Stephen P. Beeby. 2013. An evaluation of otoacoustic emissions as a biometric. IEEE TIFS 8, 1 (2013), 174–183. https://doi.org/10.1109/TIFS.2012.2228854Google ScholarDigital Library
- David Da He, Eric S. Winokur, and Charles G. Sodini. 2015. An Ear-Worn Vital Signs Monitor. IEEE Transactions on Biomedical Engineering 62, 11 (2015), 2547–2552. https://doi.org/10.1109/TBME.2015.2459061Google ScholarCross Ref
- Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, Venice, Italy, 2980–2988. https://doi.org/10.1109/ICCV.2017.322Google ScholarCross Ref
- Christian Holz, Senaka Buthpitiya, and Marius Knaust. 2015. Bodyprint: Biometric user identification on mobile devices using the capacitive touchscreen to scan body parts. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(CHI ’15). Association for Computing Machinery, New York, NY, USA, 3011–3014. https://doi.org/10.1145/2702123.2702518Google ScholarDigital Library
- Honor. 2022. Earbuds 3 Pro. https://www.hihonor.com/global/audio/honor-earbuds-3-pro/. (Accessed on 03/22/2024).Google Scholar
- Changshuo Hu, Xiao Ma, Dong Ma, and Ting Dang. 2023. Lightweight and Non-Invasive User Authentication on Earables. In Proceedings of the 24th International Workshop on Mobile Computing Systems and Applications (Newport Beach, California) (HotMobile ’23). Association for Computing Machinery, New York, NY, USA, 36–41. https://doi.org/10.1145/3572864.3580332Google ScholarDigital Library
- HUAWEI. 2023. FreeBuds 5. https://consumer.huawei.com/jp/audio/freebuds5/. (Accessed on 03/22/2024).Google Scholar
- D. J. Hurley, B. Arbab-Zavar, and M. S. Nixon. 2007. The ear as a biometric. In 2007 15th European Signal Processing Conference. IEEE, Poznan, Poland, 25–29.Google Scholar
- David J. Hurley, Mark S. Nixon, and John N. Carter. 2002. Force field energy functionals for image feature extraction. Image and Vision Computing 20, 5 (2002), 311–317. https://doi.org/10.1016/S0262-8856(02)00003-3Google ScholarCross Ref
- David J. Hurley, Mark S. Nixon, and John N. Carter. 2005. Ear Biometrics by Force Field Convergence. In Audio- and Video-Based Biometric Person Authentication, Takeo Kanade, Anil Jain, and Nalini K. Ratha (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 386–394.Google Scholar
- A.V. Iannarelli. 1989. Ear identification. Paramount Publishing Company, Fremont, CA. https://books.google.co.jp/books?id=jgPkAAAACAAJGoogle Scholar
- Shunji Itani, Shunsuke Kita, and Yoshinobu Kajikawa. 2022. Multimodal Personal Ear Authentication Using Acoustic Ear Feature for Smartphone Security. IEEE Transactions on Consumer Electronics 68, 1 (2022), 77–84. https://doi.org/10.1109/TCE.2021.3137474Google ScholarCross Ref
- Shunji Itani, Shunsuke Kita, and Yoshinobu Kajikawa. 2022. Multimodal Personal Ear Authentication Using Acoustic Ear Feature for Smartphone Security. IEEE Transactions on Consumer Electronics 68, 1 (2022), 77–84. https://doi.org/10.1109/TCE.2021.3137474Google ScholarCross Ref
- Anil K. Jain, Arun A. Ross, and Karthik Nandakumar. 2011. Introduction to Biometrics. Springer US. https://doi.org/10.1007/978-0-387-77326-1Google ScholarCross Ref
- Erno Jeges and Laszlo Mate. 2006. Model-based human ear identification. In 2006 World Automation Congress. IEEE, Budapest, Hungary, 1–6. https://doi.org/10.1109/WAC.2006.375757Google ScholarCross Ref
- Takashi Kikuchi, Yuta Sugiura, Katsutoshi Masai, Maki Sugimoto, and Bruce H. Thomas. 2017. EarTouch: Turning the Ear into an Input Surface. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) (MobileHCI ’17). Association for Computing Machinery, New York, NY, USA, Article 27, 1–6 pages. https://doi.org/10.1145/3098279.3098538Google ScholarDigital Library
- Ajay Kumar and Tak-Shing T. Chan. 2013. Robust ear identification using sparse representation of local texture descriptors. Pattern Recognition 46, 1 (2013), 73–85. https://doi.org/10.1016/j.patcog.2012.06.020Google ScholarDigital Library
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision – ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 740–755.Google ScholarCross Ref
- Roman Lissermann, Jochen Huber, Aristotelis Hadjakos, and Max Mühlhäuser. 2013. EarPut: Augmenting behind-the-Ear Devices for Ear-Based Interaction. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 1323–1328. https://doi.org/10.1145/2468356.2468592Google ScholarDigital Library
- Lu Lu, Xiaoxun Zhang, Youdong Zhao, and Yunde Jia. 2006. Ear recognition based on statistical shape model. In First International Conference on Innovative Computing, Information and Control - Volume I (ICICIC’06), Vol. 3. IEEE, Beijing, China, 353–356. https://doi.org/10.1109/ICICIC.2006.445Google ScholarDigital Library
- Shivangi Mahto, Takayuki Arakawa, and Takafumi Koshinak. 2018. Ear Acoustic Biometrics Using Inaudible Signals and Its Application to Continuous User Authentication. In 2018 26th European Signal Processing Conference (EUSIPCO). 1407–1411. https://doi.org/10.23919/EUSIPCO.2018.8553015Google ScholarCross Ref
- Fabrice Matulic, Riku Arakawa, Brian Vogel, and Daniel Vogel. 2020. PenSight: Enhanced Interaction with a Pen-Top Camera. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376147Google ScholarDigital Library
- Trade Ministry of Economy and Industry. (Eds.). 1958. JIS Z 9110: General Rules on Lighting Standards. Japanese Industrial Standards.Google Scholar
- Shohei Nagai, Shunichi Kasahara, and Jun Rekimoto. 2015. LiveSphere: Sharing the Surrounding Visual Environment for Immersive Experience in Remote Collaboration. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (Stanford, California, USA) (TEI ’15). Association for Computing Machinery, New York, NY, USA, 113––116. https://doi.org/10.1145/2677199.2680549Google ScholarDigital Library
- Hossein Nejati, Li Zhang, Terence Sim, Elisa Martinez-Marroquin, and Guo Dong. 2012. Wonder ears: Identification of identical twins from ear images. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). 1201–1204.Google Scholar
- Ibrahim Omara, Xiaoming Li, Gang Xiao, Khan Adil, and Wangmeng Zuo. 2018. Discriminative Local Feature Fusion for Ear Recognition Problem. In Proceedings of the 2018 8th International Conference on Bioscience, Biochemistry and Bioinformatics (Tokyo, Japan) (ICBBB ’18). Association for Computing Machinery, New York, NY, USA, 139––145. https://doi.org/10.1145/3180382.3180409Google ScholarDigital Library
- Anika Pflug, Pascal Nicklas Paul, and Christoph Busch. 2014. A comparative study on texture and surface descriptors for ear biometrics. In 2014 International Carnahan Conference on Security Technology (ICCST). 1–6. https://doi.org/10.1109/CCST.2014.6986993Google ScholarCross Ref
- P.J. Phillips, A. Martin, C.L. Wilson, and M. Przybocki. 2000. An introduction evaluating biometric systems. IEEE Computer 33, 2 (2000), 56–63. https://doi.org/10.1109/2.820040Google ScholarDigital Library
- Joseph Plazak and Marta Kersten-Oertel. 2018. A Survey on the Affordances of “Hearables”. Inventions 3, 3 (2018). https://doi.org/10.3390/inventions3030048Google ScholarCross Ref
- Abhinand Poosarala and Jayashree R. 2018. Uniform classifier for biometric ear and retina authentication using smartphone application. In Proceedings of the 2nd International Conference on Vision, Image and Signal Processing(ICVISP 2018). Association for Computing Machinery, New York, NY, USA, Article 58, 1–5 pages. https://doi.org/10.1145/3271553.3271618Google ScholarDigital Library
- Tobias Röddiger, Christopher Clarke, Paula Breitling, Tim Schneegans, Haibin Zhao, Hans Gellersen, and Michael Beigl. 2022. Sensing with earables: A systematic literature review and taxonomy of phenomena. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 3, Article 135 (Sep 2022), 1–57 pages. https://doi.org/10.1145/3550314Google ScholarDigital Library
- Onebae Ryu, Tatsuhiro Tamura, and Katsuyuki Shinohara. 1998. The Individual Identification with Pinna Image: An Examination for Auto Detection of Pinna Area and Individuality. ITE Tech. Report 22, 45 (1998), 49–53.Google Scholar
- Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2009. Brainy Hand: An Ear-Worn Hand Gesture Interaction Device. In CHI ’09 Extended Abstracts on Human Factors in Computing Systems (Boston, MA, USA) (CHI EA ’09). Association for Computing Machinery, New York, NY, USA, 4255––4260. https://doi.org/10.1145/1520340.1520649Google ScholarDigital Library
- Zi Wang, Yili Ren, Yingying Chen, and Jie Yang. 2022. ToothSonic: Earable authentication via acoustic toothprint. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 2, Article 78 (Jul 2022), 1–24 pages. https://doi.org/10.1145/3534606Google ScholarDigital Library
- Zi Wang, Sheng Tan, Linghan Zhang, Yili Ren, Zhi Wang, and Jie Yang. 2021. EarDynamic: An ear canal deformation based continuous user authentication using in-ear wearables. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 1, Article 39 (Mar 2021), 1–27 pages. https://doi.org/10.1145/3448098Google ScholarDigital Library
- Ping Yan and KevinW. Bowyer. 2005. Empirical evaluation of advanced ear biometrics. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) – Workshops. IEEE, San Diego, CA, USA, 41–41. https://doi.org/10.1109/CVPR.2005.450Google ScholarDigital Library
- Yongpan Zou, Haibo Lei, and Kaishun Wu. 2023. Beyond Legitimacy, Also With Identity: Your Smart Earphones Know Who You Are Quietly. IEEE Transactions on Mobile Computing 22, 6 (2023), 3179–3192. https://doi.org/10.1109/TMC.2021.3134654Google ScholarDigital Library
Index Terms
- EarAuthCam: Personal Identification and Authentication Method Using Ear Images Acquired with a Camera-Equipped Hearable Device
Recommendations
User Authentication Method for Hearables Using Sound Leakage Signals
ISWC '23: Proceedings of the 2023 ACM International Symposium on Wearable ComputersWe propose a novel biometric authentication method that leverages sound leakage signals from hearables that are captured by an external microphone. A sweep signal is played from hearables, and sound leakage is recorded using an external microphone. ...
A secure biometric authentication based on PEKS
Biometrics refers to metrics related to human characteristics and traits such as finger prints. How to use biometric to replace an encryption key or an identity certificate efficiently and securely is a hot topic in this big data era. In this field, ...
Inner-ear cochlea testing with earphones
MobiSys '22: Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and ServicesIn this MobiSys demo we show a low-cost earphone based system that can screen for hearing loss with a cost of $10. Our system is designed to detect otoacoustic emissions (OAE) which are sounds generated when the outer hair cells move in a healthy ...
Comments