Abstract
Eye blink detection plays a key role in many real-life applications such as Human-Computer Interaction (HCI), drowsy driving prevention and eye disease detection. Although traditional camera-based techniques are promising, multiple issues hinder their wide adoption including the privacy concern, strict lighting condition and line-of-sight (LoS) requirements. On the other hand, wireless sensing without a need for dedicated sensors gains a tremendous amount of attention in recent years. Among the wireless signals utilized for sensing, acoustic signals show a unique potential for fine-grained sensing owing to their low propagation speed in the air. Another trend favoring acoustic sensing is the wide availability of speakers and microphones in commodity devices. Promising progress has been achieved in fine-grained human motion sensing such as breathing using acoustic signals. However, it is still very challenging to employ acoustic signals for eye blink detection due to the unique characteristics of eye blink (i.e., subtle, sparse and aperiodic) and severe interference (i.e., from the human target himself and surrounding objects). We find that even the very subtle involuntary head movement induced by breathing can severely interfere with eye blink detection. In this work, for the first time, we propose a system called BlinkListener to sense the subtle eye blink motion using acoustic signals in a contact-free manner. We first quantitatively model the relationship between signal variation and the subtle movements caused by eye blink and interference. Then, we propose a novel method that exploits the "harmful" interference to maximize the subtle signal variation induced by eye blinks. We implement BlinkListener on both a research-purpose platform (Bela) and a commodity smartphone (iPhone 5c). Experiment results show that BlinkListener can achieve robust performance with a median detection accuracy of 95%. Our system can achieve high accuracies when the smartphone is held in hand, the target wears glasses/sunglasses and in the presence of strong interference with people moving around.
- 2018. JINS MEME Eyewear. https://jins-meme.com/en/Google Scholar
- 2020. Eye-tracking glasses. https://www.imec-int.com/en/eogGoogle Scholar
- Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D tracking via body radio reflections. In 11th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 14). 317--329.Google Scholar
- Fadel Adib, Hongzi Mao, Zachary Kabelac, Dina Katabi, and Robert C Miller. 2015. Smart homes that monitor breathing and heart rate. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 837--846.Google ScholarDigital Library
- National Highway Traffic Safety Administration et al. 2017. Traffic safety facts crash stats: drowsy driving 2015.Google Scholar
- Raees Ahmad and JN Borole. 2015. Drowsy driver identification using eye blink detection. IJISET-International Journal of Computer Science and Information Technologies 6, 1 (2015), 270--274.Google Scholar
- Doğukan Aksu and M Ali Aydin. 2017. Human computer interaction by eye blinking on real time. In 2017 9th International Conference on Computational Intelligence and Communication Networks (CICN). IEEE, 135--138.Google ScholarCross Ref
- Apple. 2019. iPhone 5c. https://support.apple.com/kb/sp684?locale=en_USGoogle Scholar
- Apple. 2021. iPhone 11 pro. https://support.apple.com/kb/SP805?viewlocale=en_US&locale=en_USGoogle Scholar
- Guha Balakrishnan, Fredo Durand, and John Guttag. 2013. Detecting pulse from head motions in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3430--3437.Google ScholarDigital Library
- Anna Rita Bentivoglio, Susan B Bressman, Emanuele Cassetta, Donatella Carretta, Pietro Tonali, and Alberto Albanese. 1997. Analysis of blink rate patterns in normal subjects. Movement disorders 12, 6 (1997), 1028--1034.Google Scholar
- Andrew Bierman, Mariana G Figueiro, and Mark S Rea. 2011. Measuring and predicting eyelid spectral transmittance. Journal of biomedical optics 16, 6 (2011), 067011.Google ScholarCross Ref
- Andreas Bulling, Daniel Roggen, and Gerhard Tröster. 2008. It's in your eyes: Towards context-awareness and mobile HCI using wearable EOG goggles. In Proceedings of the 10th international conference on Ubiquitous computing. 84--93.Google ScholarDigital Library
- Andreas Bulling, Jamie A Ward, Hans Gellersen, and Gerhard Troster. 2010. Eye movement analysis for activity recognition using electrooculography. IEEE transactions on pattern analysis and machine intelligence 33, 4 (2010), 741--753.Google ScholarDigital Library
- Health Canada. 1991. Guidelines for the safe use of ultrasound: Part II---Industrial and commercial applications safety code 24.Google Scholar
- Fabio Lo Castro. 2008. Class I infrared eye blinking detector. Sensors and actuators A: Physical 148, 2 (2008), 388--394.Google Scholar
- Michael Chau and Margrit Betke. 2005. Real time eye tracking and blink detection with usb cameras. Technical Report. Boston University Computer Science Department.Google Scholar
- Lili Chen, Jie Xiong, Xiaojiang Chen, Sunghoon Ivan Lee, Daqing Zhang, Tao Yan, and Dingyi Fang. 2019. LungTrack: Towards Contactless and Zero Dead-Zone Respiration Monitoring with Commodity RFIDs. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1--22.Google ScholarDigital Library
- Tarik Crnovrsanin, Yang Wang, and Kwan-Liu Ma. 2014. Stimulating a blink: reduction of eye fatigue with visual stimulus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2055--2064.Google ScholarDigital Library
- Taner Danisman, Ian Marius Bilasco, Chabane Djeraba, and Nacim Ihaddadene. 2010. Drowsy driver detection system using eye blink patterns. In 2010 International Conference on Machine and Web Intelligence. IEEE, 230--233.Google ScholarCross Ref
- Artem Dementyev and Christian Holz. 2017. DualBlink: a wearable device to continuously detect, track, and actuate blinking for alleviating dry eyes and computer vision syndrome. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 1 (2017), 1--19.Google ScholarDigital Library
- Matjaz Divjak and Horst Bischof. 2009. Eye Blink Based Fatigue Detection for Prevention of Computer Vision Syndrome.. In MVA. 350--353.Google Scholar
- Allen Earman. 2012. Eye Safety for Proximity Sensing Using Infrared Light-Emitting Diodes.Google Scholar
- Kyosuke Fukuda, John A Stern, Timothy B Brown, and Michael B Russo. 2005. Cognition, blinks, eye-movements, and pupillary movements during performance of a running memory task. Aviation, space, and environmental medicine 76, 7 (2005), C75-C85.Google Scholar
- Daniel Golson. 2019. Every Volvo Will Get In-Car Cameras to Combat Distraction and Drunk Driving. https://www.caranddriver.com/news/a26893035/volvo-interior-cameras-distraction-drunk-driving/Google Scholar
- Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Soundwave: using the doppler effect to sense gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1911--1914.Google ScholarDigital Library
- Seongwon Han, Sungwon Yang, Jihyoung Kim, and Mario Gerla. 2012. EyeGuardian: a framework of eye tracking and blink detection for mobile device users. In Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications. 1--6.Google ScholarDigital Library
- Young-Joo Han, Wooseong Kim, and Joon-Sang Park. 2018. Efficient eye-blinking detection on smartphones: A hybrid approach based on deep learning. Mobile Information Systems 2018 (2018).Google Scholar
- Mark A Hanson. 2010. Health effects of exposure to ultrasound and infrasound: Report of the independent advisory group on non-ionising radiation. Health Protection Agency.Google Scholar
- Yaxiong Huang, Simon C Busbridge, and Deshinder S Gill. 2001. Distortion and directivity in a digital transducer array loudspeaker. Journal of the Audio Engineering Society 49, 5 (2001), 337--352.Google Scholar
- Shoya Ishimaru, Kai Kunze, Koichi Kise, Jens Weppner, Andreas Dengel, Paul Lukowicz, and Andreas Bulling. 2014. In the blink of an eye: combining head motion and eye blink frequency for activity recognition with google glass. In Proceedings of the 5th augmented human international conference. 1--4.Google ScholarDigital Library
- Shoya Ishimaru, Kai Kunze, Yuji Uema, Koichi Kise, Masahiko Inami, and Katsuma Tanaka. 2014. Smarter eyewear: using commercial EOG glasses for activity recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. 239--242.Google ScholarDigital Library
- Takehiro Ito, Shinji Mita, Kazuhiro Kozuka, Tomoaki Nakano, and Shin Yamamoto. 2002. Driver blink measurement by the motion picture processing and its application to drowsiness detection. In Proceedings. The IEEE 5th International Conference on Intelligent Transportation Systems. IEEE, 168--173.Google ScholarCross Ref
- Sheba Jebakani, J Divya, Megha SM, and Santhosh HV. 2020. Eye Blink to Voice for Paralyzed Patients. International Journal of Information Technology (IJIT) 6, 3 (2020).Google Scholar
- Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing: Adjunct publication. 1151--1160.Google ScholarDigital Library
- Nataliya Kosmyna, Caitlin Morris, Thanh Nguyen, Sebastian Zepf, Javier Hernandez, and Pattie Maes. 2019. AttentivU: Designing EEG and EOG compatible glasses for physiological sensing and feedback in the car. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 355--368.Google ScholarDigital Library
- Aleksandra Królak and Pawel Strumiłło. 2012. Eye-blink detection system for human-computer interaction. Universal Access in the Information Society 11, 4 (2012), 409--419.Google ScholarDigital Library
- Dong Li, Jialin Liu, Sunghoon Ivan Lee, and Jie Xiong. 2020. FM-Track: Pushing the Limits of Contactless Multi-target Tracking using Acoustic Signals. In Proceedings of the 18th ACM Conference on Embedded Networked Sensor Systems. 1--14.Google ScholarDigital Library
- Tianxing Li, Chuankai An, Zhao Tian, Andrew T Campbell, and Xia Zhou. 2015. Human sensing using visible light communication. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. 331--344.Google ScholarDigital Library
- Tianxing Li, Xi Xiong, Yifei Xie, George Hito, Xing-Dong Yang, and Xia Zhou. 2017. Reconstructing hand poses using visible light. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--20.Google ScholarDigital Library
- Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--19.Google ScholarDigital Library
- SkyPaw Co. Ltd. 2021. Decibel X: dB Sound Level Meter. https://apps.apple.com/us/app/decibel-x-db-sound-level-meter/id448155923Google Scholar
- Radosław Mantiuk, Michał Kowalik, Adam Nowosielski, and Bartosz Bazyluk. 2012. Do-it-yourself eye tracker: Low-cost pupil-based eye tracker for computer graphics applications. In International Conference on Multimedia Modeling. Springer, 115--125.Google ScholarDigital Library
- Wenguang Mao, Mei Wang, Wei Sun, Lili Qiu, Swadhin Pradhan, and Yi-Chao Chen. 2019. RNN-Based Room Scale Hand Motion Tracking. In The 25th Annual International Conference on Mobile Computing and Networking. ACM, 38.Google Scholar
- Alvaro Marcos-Ramiro, Daniel Pizarro-Perez, Marta Marron-Romera, Daniel Pizarro-Perez, and Daniel Gatica-Perez. 2014. Automatic blinking detection towards stress discovery. In Proceedings of the 16th International Conference on Multimodal Interaction. 307--310.Google ScholarDigital Library
- Denis Metev. 2020. 2020's Voice Search Statistics - Is Voice Search Growing? https://review42.com/voice-search-stats/Google Scholar
- Emiliano Miluzzo, Tianyu Wang, and Andrew T Campbell. 2010. EyePhone: activating mobile phones with your eyes. In Proceedings of the second ACM SIGCOMM workshop on Networking, systems, and applications on mobile handhelds. 15--20.Google ScholarDigital Library
- Danielle Lumi Miura, Rossen Mihaylov Hazarbassanov, Camila Karim Nakase Yamasato, Francisco Bandeira e Silva, Cléber José Godinho, and José Álvaro Pereira Gomes. 2013. Effect of a light-emitting timer device on the blink rate of non-dry eye individuals and dry eye patients. British Journal of Ophthalmology 97, 8 (2013), 965--967.Google ScholarCross Ref
- Assit Prof Aree A Mohammed and Shereen A Anwer. 2014. Efficient eye blink detection method for disabled-helping domain. Eye 10, P1 (2014), P2.Google Scholar
- Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1515--1525.Google ScholarDigital Library
- Md Noman, Talal Bin, Md Ahad, and Atiqur Rahman. 2018. Mobile-based eye-blink detection performance analysis on android platform. Frontiers in ICT 5 (2018), 4.Google ScholarCross Ref
- Tanuja Patgar and Ripal Patel. 2020. Emergency Assistive System for Tetraplegia Patient Using Eye Waver Computer Vision Technique. In Advances in Computational Intelligence Techniques. Springer, 63--79.Google Scholar
- Nhat Pham, Tuan Dinh, Zohreh Raghebi, Taeho Kim, Nam Bui, Phuc Nguyen, Hoang Truong, Farnoush Banaei-Kashani, Ann Halbower, Thang Dinh, et al. 2020. WAKE: a behind-the-ear wearable system for microsleep detection. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services. 404--418.Google ScholarDigital Library
- Vaughan Pratt. 1987. Direct least-squares fitting of algebraic surfaces. ACM SIGGRAPH computer graphics 21, 4 (1987), 145--152.Google Scholar
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking. 27--38.Google ScholarDigital Library
- Kun Qian, Chenshu Wu, Fu Xiao, Yue Zheng, Yi Zhang, Zheng Yang, and Yunhao Liu. 2018. Acousticcardiogram: Monitoring heartbeats using acoustic signals on smart devices. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1574--1582.Google ScholarDigital Library
- Albara Ah Ramli, Rex Liu, Rahul Krishnamoorthy, Xiaoxiao Wang, Ilias Tagkopoulos, Xin Liu, et al. 2020. BWCNN: Blink to Word, a Real-Time Convolutional Neural Network Approach. arXiv preprint arXiv:2006.01232 (2020).Google Scholar
- Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing. 474--485.Google ScholarDigital Library
- Robert Schleicher, Niels Galley, Susanne Briest, and Lars Galley. 2008. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics 51, 7 (2008), 982--1010.Google ScholarCross Ref
- Jürgen Schmidt, Rihab Laarousi, Wolfgang Stolzmann, and Katja Karrer-Gauß. 2018. Eye blink detection for different driver states in conditionally automated driving and manual driving using EOG and a driver camera. Behavior research methods 50, 3 (2018), 1088--1101.Google Scholar
- Tereza Soukupova and Jan Cech. 2016. Eye blink detection using facial landmarks. In 21st computer vision winter workshop, Rimske Toplice, Slovenia.Google Scholar
- Isaac Starr, AJ Rawson, HA Schroeder, and NR Joseph. 1939. Studies on the estimation of cardiac ouptut in man, and of abnormalities in cardiac function, from the heart's recoil and the blood's impacts; the ballistocardiogram. American Journal of Physiology-Legacy Content 127, 1 (1939), 1--28.Google ScholarCross Ref
- Benjamin Tag, Junichi Shimizu, Chi Zhang, Naohisa Ohta, Kai Kunze, and Kazunori Sugiura. 2016. Eye blink as an input modality for a responsive adaptable video system. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. 205--208.Google ScholarDigital Library
- Benjamin Tag, Andrew W Vargo, Aman Gupta, George Chernyshov, Kai Kunze, and Tilman Dingler. 2019. Continuous alertness assessments: Using EOG glasses to unobtrusively monitor fatigue levels In-The-Wild. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--12.Google ScholarDigital Library
- HR Taschenbuch Verlag Schiffman. 2001. Sensation and Perception: An Integrated Approach.Google Scholar
- Bela Team. 2020. Bela Platform. https://bela.ioGoogle Scholar
- Yu-Chih Tung, Duc Bui, and Kang G Shin. 2018. Cross-platform support for rapid development of mobile acoustic sensing applications. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. 455--467.Google ScholarDigital Library
- Anran Wang and Shyamnath Gollakota. 2019. Millisonic: Pushing the limits of acoustic motion tracking. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--11.Google ScholarDigital Library
- Anran Wang, Jacob E Sunshine, and Shyamnath Gollakota. 2019. Contactless infant monitoring using white noise. In The 25th Annual International Conference on Mobile Computing and Networking. 1--16.Google ScholarDigital Library
- Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 851--860.Google ScholarDigital Library
- Tianben Wang, Daqing Zhang, Yuanqing Zheng, Tao Gu, Xingshe Zhou, and Bernadette Dorizzi. 2018. C-FMCW based contactless respiration detection using acoustic signal. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--20.Google ScholarDigital Library
- Wei Wang, Alex X Liu, Muhammad Shahzad, Kang Ling, and Sanglu Lu. 2015. Understanding and modeling of wifi signal based human activity recognition. In Proceedings of the 21st annual international conference on mobile computing and networking. 65--76.Google ScholarDigital Library
- Wei Wang, Alex X Liu, and Ke Sun. 2016. Device-free gesture tracking using acoustic signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. 82--94.Google Scholar
- Binbin Xie, Jie Xiong, Xiaojiang Chen, Eugene Chai, Liyao Li, Zhanyong Tang, and Dingyi Fang. 2019. Tagtag: material sensing with commodity RFID. In Proceedings of the 17th Conference on Embedded Networked Sensor Systems. 338--350.Google ScholarDigital Library
- Yaxiong Xie, Jie Xiong, Mo Li, and Kyle Jamieson. 2019. mD-Track: Leveraging multi-dimensionality for passive indoor Wi-Fi tracking. In The 25th Annual International Conference on Mobile Computing and Networking. 1--16.Google ScholarDigital Library
- Xiangyu Xu, Jiadi Yu, Yingying Chen, Yanmin Zhu, Linghe Kong, and Minglu Li. 2019. Breathlistener: Fine-grained breathing monitoring in driving environments utilizing acoustic signals. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. 54--66.Google ScholarDigital Library
- Michitaka Yoshimura, Momoko Kitazawa, Yasuhiro Maeda, Masaru Mimura, Kazuo Tsubota, and Taishiro Kishimoto. 2017. Smartphone viewing distance and sleep: an experimental study utilizing motion capture technology. Nature and science of sleep 9 (2017), 59.Google Scholar
- Chuang-Wen You, Martha Montes-de Oca, Thomas J Bao, Nicholas D Lane, Hong Lu, Giuseppe Cardone, Lorenzo Torresani, and Andrew T Campbell. 2012. CarSafe: a driver safety app that detects dangerous driving behavior using dual-cameras on smartphones. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. 671--672.Google ScholarDigital Library
- Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-grained acoustic-based device-free tracking. In Proceedings of the 15th annual international conference on mobile systems, applications, and services. 15--28.Google ScholarDigital Library
- Youwei Zeng, Dan Wu, Ruiyang Gao, Tao Gu, and Daqing Zhang. 2018. Fullbreathe: Full human respiration detection exploiting complementarity of csi phase and amplitude of wifi signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1--19.Google ScholarDigital Library
- Youwei Zeng, Dan Wu, Jie Xiong, Enze Yi, Ruiyang Gao, and Daqing Zhang. 2019. FarSense: Pushing the range limit of WiFi-based respiration sensing with CSI ratio of two antennas. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1--26.Google ScholarDigital Library
- Zhiwei Zhu, Kikuo Fujimura, and Qiang Ji. 2002. Real-time eye detection and tracking under various light conditions. In Proceedings of the 2002 symposium on Eye tracking research & applications. 139--144.Google ScholarDigital Library
Index Terms
- BlinkListener: "Listen" to Your Eye Blink Using Your Smartphone
Recommendations
Just blink your eyes: a head-free gaze tracking system
CHI EA '03: CHI '03 Extended Abstracts on Human Factors in Computing SystemsWe propose a head-free, easy-setup gaze tracking system designed for a gaze-based Human-Computer Interaction. Our system enables the user to interact with the computer soon after catching the user's eye blinks. The user can move his/her head freely ...
Eye Blink Detection for Smart Glasses
ISM '13: Proceedings of the 2013 IEEE International Symposium on MultimediaEye blink is a quick action of closing and opening of the eyelids. Eye blink detection has a wide range of applications in human computer interaction and human vision health care research. Existing approaches to eye blink detection often cannot suit ...
Eye Blink Detection Using Facial Features Tracker
ICBRA '17: Proceedings of the 4th International Conference on Bioinformatics Research and ApplicationsA new technique to detect eye blinks is proposed, based on state- of-the-art facial feature detectors to localize the eyes and eyelid contours. This technique shows excellent robustness to varying illumination, facial expressions and head orientation. ...
Comments