skip to main content
10.1145/3615834.3615843acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiwoarConference Proceedingsconference-collections
research-article

TongueMendous: IR-Based Tongue-Gesture Interface with Tiny Machine Learning

Published: 11 October 2023 Publication History

Abstract

This paper presents TongueMendous, an non-intrusive, pervasive tongue-gesture recognition interface for the general population and use cases. It uses an infrared sensor to detect tongue gestures when the tongue sticks in different directions. The collected data is recognized by a tiny machine learning (TinyML) model, allowing TongueMendous to classify tongue gestures on a microcontroller. Evaluations on the initial prototype reported a 91.7% cross-validation accuracy and 89.4% leave-one-person-out accuracy. We also conduct a study to explore the user experience and future design space. These results suggest that the proposed system can be accurate and work well across different users.

Supplemental Material

MP4 File
the demo video

References

[1]
Toshiyuki Ando, Yuki Kubo, Buntarou Shizuki, and Shin Takahashi. 2017. CanalSense: Face-Related Movement Recognition System Based on Sensing Air Pressure in Ear Canals. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 679–689. https://doi.org/10.1145/3126594.3126649
[2]
Takahiro Ando, Ayano Masaki, Qing Liu, Takafumi Ooka, Sho Sakurai, Koichi Hirota, and Takuya Nojima. 2018. Squachu: A Training Game to Improve Oral Function via a Non-Contact Tongue-Mouth-Motion Detection System. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces (Castiglione della Pescaia, Grosseto, Italy) (AVI ’18). Association for Computing Machinery, New York, NY, USA, Article 26, 8 pages. https://doi.org/10.1145/3206505.3206515
[3]
Gunnar Borg. 1998. Borg’s perceived exertion and pain scales.Human kinetics.
[4]
Yetong Cao, Huijie Chen, Fan Li, and Yu Wang. 2021. CanalScan: Tongue-Jaw Movement Recognition via Ear Canal Deformation Sensing. In IEEE INFOCOM 2021 - IEEE Conference on Computer Communications. 1–10. https://doi.org/10.1109/INFOCOM42981.2021.9488852
[5]
Victor Chen, Xuhai Xu, Richard Li, Yuanchun Shi, Shwetak Patel, and Yuntao Wang. 2021. Understanding the Design Space of Mouth Microgestures. In Designing Interactive Systems Conference 2021 (Virtual Event, USA) (DIS ’21). Association for Computing Machinery, New York, NY, USA, 1068–1081. https://doi.org/10.1145/3461778.3462004
[6]
Jingyuan Cheng, Ayano Okoso, Kai Kunze, Niels Henze, Albrecht Schmidt, Paul Lukowicz, and Koichi Kise. 2014. On the Tip of My Tongue: A Non-Invasive Pressure-Based Tongue Interface. In Proceedings of the 5th Augmented Human International Conference (Kobe, Japan) (AH ’14). Association for Computing Machinery, New York, NY, USA, Article 12, 4 pages. https://doi.org/10.1145/2582051.2582063
[7]
Chris S. Crawford, Stephen W. Bailey, Carmen Badea, and Juan E. Gilbert. 2015. Using Cr-Y Components to Detect Tongue Protrusion Gestures. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI EA ’15). Association for Computing Machinery, New York, NY, USA, 1331–1336. https://doi.org/10.1145/2702613.2732916
[8]
Crosshelmet. 2020. CrossHelmet X1– Transform your riding experience. https://www.crosshelmet.com/
[9]
Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Tiezhen Wang, Pete Warden, and Rocky Rhodes. 2021. TensorFlow Lite Micro: Embedded Machine Learning for TinyML Systems. In Proceedings of Machine Learning and Systems, A. Smola, A. Dimakis, and I. Stoica (Eds.). Vol. 3. 800–811. https://proceedings.mlsys.org/paper/2021/file/d2ddea18f00665ce8623e36bd4e3c7c5-Paper.pdf
[10]
Analog Devices. 2018. AD8302: LF–2.7 GHz RF/IF Gain and Phase Detector. https://www.analog.com/en/products/ad8302
[11]
Shigehiro Fujiwara, Masako Fujiu-Kurachi, Kazuhiro Hori, Yoshinobu Maeda, and Takahiro Ono. 2018. Tongue pressure production and submental surface electromyogram activities during tongue-hold swallow with different holding positions and tongue length. Dysphagia 33, 4 (2018), 403–413.
[12]
Pablo Gallego Cascón, Denys J.C. Matthies, Sachith Muthukumarana, and Suranga Nanayakkara. 2019. ChewIt. An Intraoral Interface for Discreet Interactions(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300556
[13]
Maysam Ghovanloo, M Nazmus Sahadat, Zhenxuan Zhang, Fanpeng Kong, and Nordine Sebkhi. 2017. Tapping into tongue motion to substitute or augment upper limbs. In Micro-and Nanotechnology Sensors, Systems, and Applications IX, Vol. 10194. SPIE, 206–217.
[14]
Mayank Goel, Chen Zhao, Ruth Vinisha, and Shwetak N. Patel. 2015. Tongue-in-Cheek: Using Wireless Signals to Enable Non-Intrusive and Flexible Facial Gestures Detection. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 255–258. https://doi.org/10.1145/2702123.2702591
[15]
Takuma Hashimoto, Suzanne Low, Koji Fujita, Risa Usumi, Hiroshi Yanagihara, Chihiro Takahashi, Maki Sugimoto, and Yuta Sugiura. 2018. TongueInput: Input Method by Tongue Gestures Using Optical Sensors Embedded in Mouthpiece. In 2018 57th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE). 1219–1224. https://doi.org/10.23919/SICE.2018.8492690
[16]
Ali Heydarigorji, S. M. Safavi, Ch. T. Lee, and P. H. Chou. 2017. Head-mouse: A simple cursor controller based on optical measurement of head tilt. In 2017 IEEE Signal Processing in Medicine and Biology Symposium (SPMB). 1–5. https://doi.org/10.1109/SPMB.2017.8257058
[17]
Bo Huang, Jinsong Wu, David Zhang, and Naimin Li. 2010. Tongue shape classification by geometric features. Information Sciences 180, 2 (2010), 312–324. https://doi.org/10.1016/j.ins.2009.09.016
[18]
Xueliang Huo, Jia Wang, and Maysam Ghovanloo. 2008. A Magneto-Inductive Sensor Based Wireless Tongue-Computer Interface. IEEE Transactions on Neural Systems and Rehabilitation Engineering 16, 5 (2008), 497–504. https://doi.org/10.1109/TNSRE.2008.2003375
[19]
Broadcom Inc.2017. APDS-9960. https://www.broadcom.com/products/optical-sensors/integrated-ambient-light-and-proximity-sensors/apds-9960
[20]
PixArt Imaging Inc.2020. Products - PAJ7620U2. https://www.pixart.com/products-detail/37/PAJ7620U2
[21]
Maxim Integrated. 2021. MAX25405 IR Gesture Sensor with Lens for Automotive Applications | Maxim Integrated. https://www.maximintegrated.com/en/products/sensors/MAX25405.html
[22]
Basar Koc, Dilip Sarkar, Huseyin Kocak, and Ziya Arnavut. 2015. A study of power consumption on MSP432 family of microcontrollers for lossless data compression. In 2015 12th International Conference on High-capacity Optical Networks and Enabling/Emerging Technologies (HONET). 1–5. https://doi.org/10.1109/HONET.2015.7395418
[23]
Marion Koelle, Swamy Ananthanarayan, and Susanne Boll. 2020. Social Acceptability in HCI: A Survey of Methods, Measures, and Design Strategies. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–19. https://doi.org/10.1145/3313831.3376162
[24]
Hyein Lee, Yoonji Kim, and Andrea Bianchi. 2020. MAScreen: Augmenting Speech with Visual Cues of Lip Motions, Facial Expressions, and Text Using a Wearable Display. In SIGGRAPH Asia 2020 Emerging Technologies (Virtual Event, Republic of Korea) (SA ’20). Association for Computing Machinery, New York, NY, USA, Article 2, 2 pages. https://doi.org/10.1145/3415255.3422886
[25]
Richard Li and Gabriel Reyes. 2018. Buccal: Low-Cost Cheek Sensing for Inferring Continuous Jaw Motion in Mobile Virtual Reality. In Proceedings of the 2018 ACM International Symposium on Wearable Computers (Singapore, Singapore) (ISWC ’18). Association for Computing Machinery, New York, NY, USA, 180–183. https://doi.org/10.1145/3267242.3267265
[26]
Richard Li, Jason Wu, and Thad Starner. 2019. TongueBoard: An Oral Interface for Subtle Input. In Proceedings of the 10th Augmented Human International Conference 2019 (Reims, France) (AH2019). Association for Computing Machinery, New York, NY, USA, Article 1, 9 pages. https://doi.org/10.1145/3311823.3311831
[27]
Zheng Li, Ryan Robucci, Nilanjan Banerjee, and Chintan Patel. 2015. Tongue-n-Cheek: Non-Contact Tongue Gesture Recognition. In Proceedings of the 14th International Conference on Information Processing in Sensor Networks (Seattle, Washington) (IPSN ’15). Association for Computing Machinery, New York, NY, USA, 95–105. https://doi.org/10.1145/2737095.2737109
[28]
Balz Maag, Zimu Zhou, Olga Saukh, and Lothar Thiele. 2017. BARTON: Low Power Tongue Movement Sensing with In-Ear Barometers. In 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS). 9–16. https://doi.org/10.1109/ICPADS.2017.00013
[29]
Masato Miyauchi, Takashi Kimura, and Takuya Nojima. 2012. Development of a Non-Contact Tongue-Motion Acquisition System. In Adjunct Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (Cambridge, Massachusetts, USA) (UIST Adjunct Proceedings ’12). Association for Computing Machinery, New York, NY, USA, 75–76. https://doi.org/10.1145/2380296.2380329
[30]
Mostafa Mohammadi, Hendrik Knoche, Bo Bentsen, Michael Gaihede, and Lotte N. S. Andreasen Struijk. 2020. A pilot study on a novel gesture-based tongue interface for robot and computer control. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE). 906–913. https://doi.org/10.1109/BIBE50027.2020.00154
[31]
Yunjun Nam, Qibin Zhao, Andrzej Cichocki, and Seungjin Choi. 2012. Tongue-Rudder: A Glossokinetic-Potential-Based Tongue–Machine Interface. IEEE Transactions on Biomedical Engineering 59, 1 (2012), 290–299. https://doi.org/10.1109/TBME.2011.2174058
[32]
Phuc Nguyen, Nam Bui, Anh Nguyen, Hoang Truong, Abhijit Suresh, Matt Whitlock, Duy Pham, Thang Dinh, and Tam Vu. 2018. TYTH-Typing On Your Teeth: Tongue-Teeth Localization for Human-Computer Interface. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services (Munich, Germany) (MobiSys ’18). Association for Computing Machinery, New York, NY, USA, 269–282. https://doi.org/10.1145/3210240.3210322
[33]
Shuo Niu, Li Liu, and D. Scott McCrickard. 2019. Tongue-able interfaces: Prototyping and evaluating camera based tongue gesture input system. Smart Health 11 (2019), 16–28. https://doi.org/10.1016/j.smhl.2018.03.001 Emerging Healthcare Technologies.
[34]
Hangue Park, Benoit Gosselin, Mehdi Kiani, Hyung-Min Lee, Jeonghee Kim, Xueliang Huo, and Maysam Ghovanloo. 2012. A wireless magnetoresistive sensing system for an intra-oral tongue-computer interface. In 2012 IEEE International Solid-State Circuits Conference. 124–126. https://doi.org/10.1109/ISSCC.2012.6176947
[35]
Partha Pratim Ray. 2022. A review on TinyML: State-of-the-art and prospects. Journal of King Saud University - Computer and Information Sciences 34, 4 (2022), 1595–1623. https://doi.org/10.1016/j.jksuci.2021.11.019
[36]
T. Scott Saponas, Daniel Kelly, Babak A. Parviz, and Desney S. Tan. 2009. Optically Sensing Tongue Gestures for Computer Input. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (Victoria, BC, Canada) (UIST ’09). Association for Computing Machinery, New York, NY, USA, 177–180. https://doi.org/10.1145/1622176.1622209
[37]
M. Sasaki, K. Onishi, T. Arakawa, A. Nakayama, D. Stefanov, and M. Yamaguchi. 2013. Real-time estimation of tongue movement based on suprahyoid muscle activity. In 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 4605–4608. https://doi.org/10.1109/EMBC.2013.6610573
[38]
Makoto Sasaki, Kohei Onishi, Dimitar Stefanov, Katsuhiro Kamata, Atsushi Nakayama, Masahiro Yoshikawa, and Goro Obinata. 2016. Tongue interface based on surface EMG signals of suprahyoid muscles. Robomech Journal 3, 1 (2016), 1–11.
[39]
Nordine Sebkhi, Nazmus Sahadat, Sinan Hersek, Arpan Bhavsar, Shayan Siahpoushan, Maysam Ghoovanloo, and Omer T. Inan. 2019. A Deep Neural Network-Based Permanent Magnet Localization for Tongue Tracking. IEEE Sensors Journal 19, 20 (2019), 9324–9331. https://doi.org/10.1109/JSEN.2019.2923585
[40]
Bosch MEMS Technology | Bosch Sensortec. 2021. Nicla Sense ME | Bosch Sensortec. https://www.bosch-sensortec.com/software-tools/tools/arduino-nicla-sense-me/
[41]
Steve NH Tsang, John KL Ho, and Alan HS Chan. 2015. Interface Design and Display-Control Compatibility. Measurement and Control 48, 3 (2015), 81–86. https://doi.org/10.1177/0020294015569264 arXiv:https://doi.org/10.1177/0020294015569264
[42]
Radu-Daniel Vatavu and Laura-Bianca Bilius. 2021. GestuRING: A Web-Based Tool for Designing Gesture Input with Rings, Ring-Like, and Ring-Ready Devices. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 710–723. https://doi.org/10.1145/3472749.3474780
[43]
Ruihu Wang. 2012. AdaBoost for Feature Selection, Classification and Its Relation with SVM, A Review. Physics Procedia 25 (2012), 800–807. https://doi.org/10.1016/j.phpro.2012.03.160 International Conference on Solid State Devices and Materials Science, April 1-2, 2012, Macao.
[44]
Pete Warden and Daniel Situnayake. 2019. Tinyml: Machine learning with tensorflow lite on arduino and ultra-low-power microcontrollers. O’Reilly Media.
[45]
Nerys Williams. 2017. The Borg Rating of Perceived Exertion (RPE) scale. Occupational Medicine 67, 5 (07 2017), 404–405. https://doi.org/10.1093/occmed/kqx063 arXiv:https://academic.oup.com/occmed/article-pdf/67/5/404/18688411/kqx063.pdf
[46]
Anusha Withana, Roshan Peiris, Nipuna Samarasekara, and Suranga Nanayakkara. 2015. ZSense: Enabling Shallow Depth Gesture Recognition for Greater Input Expressivity on Smart Wearables. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3661–3670. https://doi.org/10.1145/2702123.2702371
[47]
Yu-Hung Yeh, Davy P. Y. Wong, Cheng-Ting Lee, and Pai H. Chou. 2022. Deep Learning-Based Real-Time Activity Recognition with Multiple Inertial Sensors. In 2022 4th International Conference on Image, Video and Signal Processing (Singapore, Singapore) (IVSP 2022). Association for Computing Machinery, New York, NY, USA, 92–99. https://doi.org/10.1145/3531232.3531245
[48]
Behnaz Yousefi, Xueliang Huo, and Maysam Ghovanloo. 2010. Using Fitts’s law for evaluating Tongue Drive System as a pointing device for computer access. In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. 4403–4406. https://doi.org/10.1109/IEMBS.2010.5627130
[49]
Qian Zhang, Yetong Cao, Huijie Chen, Fan Li, Song Yang, Yu Wang, Zheng Yang, and Yunhao Liu. 2020. airFinger: Micro Finger Gesture Recognition via NIR Light Sensing for Smart Devices. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS). 552–562. https://doi.org/10.1109/ICDCS47774.2020.00073
[50]
Qiao Zhang, Shyamnath Gollakota, Ben Taskar, and Raj P.N. Rao. 2014. Non-Intrusive Tongue Machine Interface(CHI ’14). Association for Computing Machinery, New York, NY, USA, 2555–2558. https://doi.org/10.1145/2556288.2556981

Cited By

View all
  • (2024)lolEpop: A Multisensory Electronic Lollipop for Enhanced Tongue Training and Behavior AnalysisCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677604(152-156)Online publication date: 5-Oct-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
iWOAR '23: Proceedings of the 8th international Workshop on Sensor-Based Activity Recognition and Artificial Intelligence
September 2023
171 pages
ISBN:9798400708169
DOI:10.1145/3615834
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. TinyML
  2. tongue-gesture interface
  3. ubiquitous computing

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

iWOAR 2023

Acceptance Rates

Overall Acceptance Rate 46 of 73 submissions, 63%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)47
  • Downloads (Last 6 weeks)3
Reflects downloads up to 22 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)lolEpop: A Multisensory Electronic Lollipop for Enhanced Tongue Training and Behavior AnalysisCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677604(152-156)Online publication date: 5-Oct-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media