skip to main content
research-article

FinDroidHR: Smartwatch Gesture Input with Optical Heartrate Monitor

Published: 26 March 2018 Publication History

Abstract

We present FinDroidHR, a novel gesture input technique for off-the-shelf smartwatches. Our technique is designed to detect 10 hand gestures on the hand wearing a smartwatch. The technique is enabled by analysing features of the Photoplethysmography (PPG) signal that optical heart-rate sensors capture. In a study with 20 participants, we show that FinDroidHR achieves 90.55% accuracy and 90.73% recall. Our work is the first study to explore the feasibility of using optical sensors on the off-the-shelf wearable devices to recognise gestures. Without requiring bespoke hardware, FinDroidHR can be readily used on existing smartwatches.

References

[1]
Gregory D Abowd. 2012. What next, ubicomp?: celebrating an intellectual disappearing act. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. ACM, 31--40.
[2]
Nasimuddin Ahmed, Rohan Banerjee, Avik Ghose, and Arijit Sinharay. 2015. Feasibility Analysis for Estimation of Blood Pressure and Heart Rate using a Smart Eye Wear. In Proc. WearSys 2015. ACM Press, 9--14.
[3]
Nazneen Akhter, Sumegh Tharewal, Vijay Kale, Ashish Bhalerao, and K.V. Kale. 2016. Heart-Based Biometrics and Possible Use of Heart Rate Variability in Biometric Recognition Systems. Advances in Intelligent Systems and Computing 395 (2016), 15--29.
[4]
Daniel Ashbrook, Kent Lyons, and Thad Starner. 2008. An Investigation into Round Touchscreen Wristwatch Interaction. In Proc. MobileHCI 2008. ACM Press, 311--314.
[5]
IEEE Standards Association. 2003. IEEE Standard for Transitions, Pulses, and Related Waveforms. In IEEE Std. 181-2003. IEEE Press.
[6]
Hamed Azami, Karim Mohammadi, and Behzad Bozorgtabar. 2012. An Improved Signal Segmentation Using Moving Average and Savitzky-Golay Filter. Journal of Signal and Information Processing 3 (2012), 39--44.
[7]
Nick Barnes. 2005. Improved Signal To Noise Ratio And Computational Speed For Gradient-Based Detection Algorithms. In International Conference on Robotics and Automation 2005. IEEE Press, 4661--4667.
[8]
Patrick Baudisch and Gerry Chu. 2009. Back-of-Device Interaction Allows Creating Very Small Touch Devices. In Proc. CHI 2009. ACM Press, 1923--1932.
[9]
Jorge Blasco, Thomas M. Chen, Juan Tapiador, and Pedro Peris-Lopez. 2016. A Survey of Wearable Biometric Recognition Systems. In Proc. CSUR 2016. ACM Press.
[10]
Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proc. UIST 2013. ACM Press, 255--260.
[11]
Jikai Chen, Yanhui Dou, Yang Li, and Jiang Li. 2016. Application of Shannon Wavelet Entropy and Shannon Wavelet Packet Entropy in Analysis of Power System Transient Signals. Entropy 2016 18 (2016), 14.
[12]
Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D Input Using Two Magnetic Sensors. In Proc. UIST 2013. ACM Press, 237--244.
[13]
Sangita Das, Saurabh Pal, and Madhuchhanda Mitra. 2016. Real Time Heart Rate Detection from PPG Signal in Noisy Environment. In Proc. ICICPI 2016. IEEE Press, 70--73.
[14]
Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: Low-Power Gesture Input with Wrist-Worn Pressure Sensors. In Proc. UIST 2014. ACM Press, 161--166.
[15]
Tizen Developers. 2017. Heart Rate Monitor LED Green Sensor. (2017). https://developer.tizen.org/development/guides/native-application/location-and-sensors/device-sensors#hrm_ir
[16]
Florian Grutzmacher, Johann-Peter Wolff, and Christian Haubelt. 2015. Exploiting Thread-Level Parallelism in Template-Based Gesture Recognition with Dynamic Time Warping. In Proc. WOAR 2015. ACM Press, 6.
[17]
Sidhant Gupta, Dan Morris, Shwetak N Patel, and Desney Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. In Proc. CHI 2012. ACM Press, 1911--1914.
[18]
Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions. In Proc. CHI 2016. ACM Press, 1526--1537.
[19]
Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In Proc. NSDI 2014. ACM Press, 303--316.
[20]
Frederic Kerber, Markus Lochtefeld, Antonio Kruger, Jess Mcintosh, Charlie McNeill, and Mike Fraser. 2016. Understanding Same-Side Interactions with Wrist-Worn Devices. In Proc. NordiCHI 2016. ACM Press.
[21]
Jungsoo Kim, Jiasheng He, Kent Lyons, and Thad Starner. 2007. The Gesture Watch: A Wireless Contact-free Gesture based Wrist Interface. In Proc. ISWC 2007. IEEE Press, 1--8.
[22]
Min Sik Kim, Taekhyun Kim, Yong-June Shin, Simon S. Lam, and Edward J. Powers. 2008. A wavelet-based approach to detect shared congestion. In IEEE/ACM Trans. Netw 2008. IEEE Press, 763--776.
[23]
Vassilis Kostakos and Mirco Musolesi. 2017. Avoiding Pitfalls When Using Machine Learning in HCI Studies. INTERACTIONS (2017).
[24]
Sven Kratz and Michael Rohs. MobileHCI. Hoverflow: Exploring Around-Device Interaction with IR Distance Sensors. In Proc. MobileHCI 2009. ACM Press, 4.
[25]
Gierad Laput, Robert Xiao, Xiang Anthony' Chen, Scott E. Hudson, and Chris Harrison. 2014. Skin Buttons: Cheap, Small, Low-Power and Clickable Fixed-Icon Laser Projections. In Proc. UIST 2014. ACM Press, 389--394.
[26]
YANN LECUN, LEON BOTTOU, YOSHUA BENGIO, and PATRICK HAFFNER. 1998. Gradient-Based Learning Applied to Document Recognition. In Proc. IEEE 1998. IEEE Press, 2278--2324.
[27]
Zhiyuan Lu, Xiang Chen, Qiang Li, Xu Zhang, and Ping Zhou. 2014. A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices. In Proc. HUMAN-MACHINE SYSTEMS 2014. IEEE Press, 293--299.
[28]
Yuka Maeda, Masaki Sekine, and Toshiyo Tamura. 2011. Relationship Between Measurement Site and Motion Artifacts in Wearable Reflected Photoplethysmography. Journal of Medical Systems 39, 5(2011), 969--976.
[29]
HENRY MARTIN, PAUL GROVES, and MARK NEWMAN. 2016. The Limits of In-Run Calibration of MEMS Inertial Sensors and Sensor Arrays. NAVIGATION: Journal of The Institute of Navigation 63 (2016).
[30]
Jess Mcintosh, Asier Marzo, Mike Fraser, and Carol Phillips. 2017. EchoFlex: Hand Gesture Recognition using Ultrasound Imaging. In Proc. CHI 2017. ACM Press.
[31]
Kei Nakatsuma, Hiroyuki Shinoda, Yasutoshi Makino, Katsunari Sato, and Takashi Maeno. 2011. Touch Interface on Back of the Hand. In Proc. SIGGRAPH 2011. ACM Press, 1.
[32]
Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proc. CHI 2016. ACM Press, 1515--1525.
[33]
Simon T. Perrault, Eric Lecolinet, James Eagan, and Yves Guiard. 2013. WatchIt: Simple Gestures and Eyes-free Interaction for Wristwatches and Bracelets. In Proc. CHI 2013. ACM Press, 1451--1460.
[34]
Raj Rakshit, V Ramu Reddy, and Parijat Deshpande. 2016. Emotion Detection and Recognition using HRV Features Derived from Photoplethysmogram Signals. In Proc. ERM4CT 2016. ACM Press.
[35]
Marko Robnik-Sikonja and Igor Kononenko. 2003. Theoretical and Empirical Analysis of ReliefF and RReliefF. In Machine Learning, 53, 2003. 23--69.
[36]
T. Scott Saponas, Chris Harrison, and Hrvoje Benko. 2011. PocketTouch: Through-Fabric Capacitive Touch Input. In Proc. UIST 2011. ACM Press, 303--308.
[37]
T. Scott Saponas, Desney S. Tan, Dan Morris, Jim Turner, and James A. Landay. 2010. Making Muscle-Computer Interfaces More Practical. In Proc. CHI 2010. ACM Press, 851--854.
[38]
Mads Soegaard and Rikke Friis Dam. 2013. The Encyclopedia of Human-Computer Interaction, 2nd Ed. (2nd ed.). The Interaction Design Foundation.
[39]
Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. In Proc. Mobicom 2015. ACM Press, 77--89.
[40]
S.K. Deric Tang, Y.Y. Sebastian Goh, M.L. Dennis Wong, and Y.L. Eileen Lew. 2016. PPG signal reconstruction using a combination of discrete wavelet transform and empirical mode decomposition. In Proc. ICIAS 2016. IEEE Press.
[41]
H. Emrah Tasli, Amogh Gudi, and Marten den Uyl. 2014. Integrating Remote PPG in Facial Expression Analysis Framework. In Proc. ICMI 2014. ACM Press, 74--75.
[42]
Ihor Vasyltsov and Seunghwan Lee. 2015. Entropy Extraction from Bio-Signals in Healthcare IoT. In Proc. IoTPTS 2015. ACM Press, 11--17.
[43]
Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jurgen Steimle. 2015. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing. In Proc. CHI 2015. ACM Press, 2991--3000.
[44]
Martin Weigel, Aditya Shekhar, Nittala1 Alex Olwal, and Jurgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In Proc. CHI 2017. ACM Press, 3095--3105.
[45]
Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition using an Off-the-Shelf Smartwatch. In Proc. CHI 2016. ACM Press, 3847--3851.
[46]
Robert Xiao, Gierad Laput, and Chris Harrison. 2014. Expanding the Input Expressivity of Smartwatches with Mechanical Pan, Twist, Tilt and Click. In Proc. CHI 2014. ACM Press, 193--196.
[47]
Liu Xing and Qian Feng. 2016. Measuring and optimizing android smartwatch energy consumption: poster. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 421--423.
[48]
Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2014. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch. In Proc. HotMobile 2015. ACM Press.
[49]
Yalan Ye, Wenwen He, Yunfei Cheng, Wenxia Huang, and Zhilin Zhang. 2017. A Robust Random Forest-Based Approach for Heart Rate Monitoring Using Photoplethysmography Signal Contaminated by Intense Motion Artifacts. MDPI 385 (2017).
[50]
Xiaobo Zhang and Xiangchu Feng. 2014. A New Gradient-based Nonlinear Diffusion Method Applied to Image Denoising. In Proc. PIC 2014. IEEE Press, 321--315.
[51]
Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body as an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proc. CHI 2016. ACM Press, 1491--1503.

Cited By

View all
  • (2024)GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model AgentsProceedings of the ACM on Human-Computer Interaction10.1145/36981458:ISS(462-499)Online publication date: 24-Oct-2024
  • (2024)Efficient and Robust Heart Rate Estimation Approach for Noisy Wearable PPG Sensors Using Ideal Representation LearningProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676606(1-8)Online publication date: 5-Oct-2024
  • (2024)OOBKey: Key Exchange with Implantable Medical Devices Using Out-Of-Band ChannelsProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3670876(1-13)Online publication date: 30-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 2, Issue 1
March 2018
1370 pages
EISSN:2474-9567
DOI:10.1145/3200905
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 March 2018
Accepted: 01 January 2018
Revised: 01 November 2017
Received: 01 August 2017
Published in IMWUT Volume 2, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Gesture Interaction
  2. Machine Learning
  3. Mobile Sensors
  4. Wearable Computing

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

  • The Australian Research Council (ARC)

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)68
  • Downloads (Last 6 weeks)7
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model AgentsProceedings of the ACM on Human-Computer Interaction10.1145/36981458:ISS(462-499)Online publication date: 24-Oct-2024
  • (2024)Efficient and Robust Heart Rate Estimation Approach for Noisy Wearable PPG Sensors Using Ideal Representation LearningProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676606(1-8)Online publication date: 5-Oct-2024
  • (2024)OOBKey: Key Exchange with Implantable Medical Devices Using Out-Of-Band ChannelsProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3670876(1-13)Online publication date: 30-Jul-2024
  • (2024)Understanding Driving Stress in Urban Bangladesh: An Exploratory Study, Wearable Development and ExperimentACM Journal on Computing and Sustainable Societies10.1145/36484342:2(1-28)Online publication date: 13-May-2024
  • (2024)ViObjectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435478:1(1-26)Online publication date: 6-Mar-2024
  • (2024)Enabling WiFi Sensing on New-generation WiFi CardsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36338077:4(1-26)Online publication date: 12-Jan-2024
  • (2024)Hand Gesture Recognition for Blind Users by Tracking 3D Gesture TrajectoryProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642602(1-15)Online publication date: 11-May-2024
  • (2024)Wrist Information Detection Sensor Based on Pinhole ImagingIEEE Photonics Journal10.1109/JPHOT.2024.343583216:6(1-6)Online publication date: Dec-2024
  • (2024)PPGSpotter: Personalized Free Weight Training Monitoring Using Wearable PPG SensorIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621212(2468-2477)Online publication date: 20-May-2024
  • (2023)Non-intrusive Human Vital Sign Detection Using mmWave Sensing Technologies: A ReviewACM Transactions on Sensor Networks10.1145/362716120:1(1-36)Online publication date: 3-Nov-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media