research-article

FinDroidHR: Smartwatch Gesture Input with Optical Heartrate Monitor

Authors:

Vassilis Kostakos,

Aruna SeneviratneAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 2, Issue 1

Article No.: 56, Pages 1 - 42

https://doi.org/10.1145/3191788

Published: 26 March 2018 Publication History

Abstract

We present FinDroidHR, a novel gesture input technique for off-the-shelf smartwatches. Our technique is designed to detect 10 hand gestures on the hand wearing a smartwatch. The technique is enabled by analysing features of the Photoplethysmography (PPG) signal that optical heart-rate sensors capture. In a study with 20 participants, we show that FinDroidHR achieves 90.55% accuracy and 90.73% recall. Our work is the first study to explore the feasibility of using optical sensors on the off-the-shelf wearable devices to recognise gestures. Without requiring bespoke hardware, FinDroidHR can be readily used on existing smartwatches.

References

[1]

Gregory D Abowd. 2012. What next, ubicomp?: celebrating an intellectual disappearing act. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. ACM, 31--40.

Digital Library

[2]

Nasimuddin Ahmed, Rohan Banerjee, Avik Ghose, and Arijit Sinharay. 2015. Feasibility Analysis for Estimation of Blood Pressure and Heart Rate using a Smart Eye Wear. In Proc. WearSys 2015. ACM Press, 9--14.

Digital Library

[3]

Nazneen Akhter, Sumegh Tharewal, Vijay Kale, Ashish Bhalerao, and K.V. Kale. 2016. Heart-Based Biometrics and Possible Use of Heart Rate Variability in Biometric Recognition Systems. Advances in Intelligent Systems and Computing 395 (2016), 15--29.

[4]

Daniel Ashbrook, Kent Lyons, and Thad Starner. 2008. An Investigation into Round Touchscreen Wristwatch Interaction. In Proc. MobileHCI 2008. ACM Press, 311--314.

Digital Library

[5]

IEEE Standards Association. 2003. IEEE Standard for Transitions, Pulses, and Related Waveforms. In IEEE Std. 181-2003. IEEE Press.

[6]

Hamed Azami, Karim Mohammadi, and Behzad Bozorgtabar. 2012. An Improved Signal Segmentation Using Moving Average and Savitzky-Golay Filter. Journal of Signal and Information Processing 3 (2012), 39--44.

[7]

Nick Barnes. 2005. Improved Signal To Noise Ratio And Computational Speed For Gradient-Based Detection Algorithms. In International Conference on Robotics and Automation 2005. IEEE Press, 4661--4667.

[8]

Patrick Baudisch and Gerry Chu. 2009. Back-of-Device Interaction Allows Creating Very Small Touch Devices. In Proc. CHI 2009. ACM Press, 1923--1932.

Digital Library

[9]

Jorge Blasco, Thomas M. Chen, Juan Tapiador, and Pedro Peris-Lopez. 2016. A Survey of Wearable Biometric Recognition Systems. In Proc. CSUR 2016. ACM Press.

Digital Library

[10]

Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proc. UIST 2013. ACM Press, 255--260.

Digital Library

[11]

Jikai Chen, Yanhui Dou, Yang Li, and Jiang Li. 2016. Application of Shannon Wavelet Entropy and Shannon Wavelet Packet Entropy in Analysis of Power System Transient Signals. Entropy 2016 18 (2016), 14.

[12]

Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D Input Using Two Magnetic Sensors. In Proc. UIST 2013. ACM Press, 237--244.

Digital Library

[13]

Sangita Das, Saurabh Pal, and Madhuchhanda Mitra. 2016. Real Time Heart Rate Detection from PPG Signal in Noisy Environment. In Proc. ICICPI 2016. IEEE Press, 70--73.

[14]

Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: Low-Power Gesture Input with Wrist-Worn Pressure Sensors. In Proc. UIST 2014. ACM Press, 161--166.

Digital Library

[15]

Tizen Developers. 2017. Heart Rate Monitor LED Green Sensor. (2017). https://developer.tizen.org/development/guides/native-application/location-and-sensors/device-sensors#hrm_ir

[16]

Florian Grutzmacher, Johann-Peter Wolff, and Christian Haubelt. 2015. Exploiting Thread-Level Parallelism in Template-Based Gesture Recognition with Dynamic Time Warping. In Proc. WOAR 2015. ACM Press, 6.

Digital Library

[17]

Sidhant Gupta, Dan Morris, Shwetak N Patel, and Desney Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. In Proc. CHI 2012. ACM Press, 1911--1914.

Digital Library

[18]

Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions. In Proc. CHI 2016. ACM Press, 1526--1537.

Digital Library

[19]

Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In Proc. NSDI 2014. ACM Press, 303--316.

Digital Library

[20]

Frederic Kerber, Markus Lochtefeld, Antonio Kruger, Jess Mcintosh, Charlie McNeill, and Mike Fraser. 2016. Understanding Same-Side Interactions with Wrist-Worn Devices. In Proc. NordiCHI 2016. ACM Press.

Digital Library

[21]

Jungsoo Kim, Jiasheng He, Kent Lyons, and Thad Starner. 2007. The Gesture Watch: A Wireless Contact-free Gesture based Wrist Interface. In Proc. ISWC 2007. IEEE Press, 1--8.

Digital Library

[22]

Min Sik Kim, Taekhyun Kim, Yong-June Shin, Simon S. Lam, and Edward J. Powers. 2008. A wavelet-based approach to detect shared congestion. In IEEE/ACM Trans. Netw 2008. IEEE Press, 763--776.

Digital Library

[23]

Vassilis Kostakos and Mirco Musolesi. 2017. Avoiding Pitfalls When Using Machine Learning in HCI Studies. INTERACTIONS (2017).

Digital Library

[24]

Sven Kratz and Michael Rohs. MobileHCI. Hoverflow: Exploring Around-Device Interaction with IR Distance Sensors. In Proc. MobileHCI 2009. ACM Press, 4.

Digital Library

[25]

Gierad Laput, Robert Xiao, Xiang Anthony' Chen, Scott E. Hudson, and Chris Harrison. 2014. Skin Buttons: Cheap, Small, Low-Power and Clickable Fixed-Icon Laser Projections. In Proc. UIST 2014. ACM Press, 389--394.

Digital Library

[26]

YANN LECUN, LEON BOTTOU, YOSHUA BENGIO, and PATRICK HAFFNER. 1998. Gradient-Based Learning Applied to Document Recognition. In Proc. IEEE 1998. IEEE Press, 2278--2324.

[27]

Zhiyuan Lu, Xiang Chen, Qiang Li, Xu Zhang, and Ping Zhou. 2014. A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices. In Proc. HUMAN-MACHINE SYSTEMS 2014. IEEE Press, 293--299.

[28]

Yuka Maeda, Masaki Sekine, and Toshiyo Tamura. 2011. Relationship Between Measurement Site and Motion Artifacts in Wearable Reflected Photoplethysmography. Journal of Medical Systems 39, 5(2011), 969--976.

Digital Library

[29]

HENRY MARTIN, PAUL GROVES, and MARK NEWMAN. 2016. The Limits of In-Run Calibration of MEMS Inertial Sensors and Sensor Arrays. NAVIGATION: Journal of The Institute of Navigation 63 (2016).

[30]

Jess Mcintosh, Asier Marzo, Mike Fraser, and Carol Phillips. 2017. EchoFlex: Hand Gesture Recognition using Ultrasound Imaging. In Proc. CHI 2017. ACM Press.

Digital Library

[31]

Kei Nakatsuma, Hiroyuki Shinoda, Yasutoshi Makino, Katsunari Sato, and Takashi Maeno. 2011. Touch Interface on Back of the Hand. In Proc. SIGGRAPH 2011. ACM Press, 1.

Digital Library

[32]

Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proc. CHI 2016. ACM Press, 1515--1525.

Digital Library

[33]

Simon T. Perrault, Eric Lecolinet, James Eagan, and Yves Guiard. 2013. WatchIt: Simple Gestures and Eyes-free Interaction for Wristwatches and Bracelets. In Proc. CHI 2013. ACM Press, 1451--1460.

Digital Library

[34]

Raj Rakshit, V Ramu Reddy, and Parijat Deshpande. 2016. Emotion Detection and Recognition using HRV Features Derived from Photoplethysmogram Signals. In Proc. ERM4CT 2016. ACM Press.

Digital Library

[35]

Marko Robnik-Sikonja and Igor Kononenko. 2003. Theoretical and Empirical Analysis of ReliefF and RReliefF. In Machine Learning, 53, 2003. 23--69.

Digital Library

[36]

T. Scott Saponas, Chris Harrison, and Hrvoje Benko. 2011. PocketTouch: Through-Fabric Capacitive Touch Input. In Proc. UIST 2011. ACM Press, 303--308.

Digital Library

[37]

T. Scott Saponas, Desney S. Tan, Dan Morris, Jim Turner, and James A. Landay. 2010. Making Muscle-Computer Interfaces More Practical. In Proc. CHI 2010. ACM Press, 851--854.

Digital Library

[38]

Mads Soegaard and Rikke Friis Dam. 2013. The Encyclopedia of Human-Computer Interaction, 2nd Ed. (2nd ed.). The Interaction Design Foundation.

[39]

Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. In Proc. Mobicom 2015. ACM Press, 77--89.

Digital Library

[40]

S.K. Deric Tang, Y.Y. Sebastian Goh, M.L. Dennis Wong, and Y.L. Eileen Lew. 2016. PPG signal reconstruction using a combination of discrete wavelet transform and empirical mode decomposition. In Proc. ICIAS 2016. IEEE Press.

[41]

H. Emrah Tasli, Amogh Gudi, and Marten den Uyl. 2014. Integrating Remote PPG in Facial Expression Analysis Framework. In Proc. ICMI 2014. ACM Press, 74--75.

Digital Library

[42]

Ihor Vasyltsov and Seunghwan Lee. 2015. Entropy Extraction from Bio-Signals in Healthcare IoT. In Proc. IoTPTS 2015. ACM Press, 11--17.

Digital Library

[43]

Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jurgen Steimle. 2015. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing. In Proc. CHI 2015. ACM Press, 2991--3000.

Digital Library

[44]

Martin Weigel, Aditya Shekhar, Nittala1 Alex Olwal, and Jurgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In Proc. CHI 2017. ACM Press, 3095--3105.

Digital Library

[45]

Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition using an Off-the-Shelf Smartwatch. In Proc. CHI 2016. ACM Press, 3847--3851.

Digital Library

[46]

Robert Xiao, Gierad Laput, and Chris Harrison. 2014. Expanding the Input Expressivity of Smartwatches with Mechanical Pan, Twist, Tilt and Click. In Proc. CHI 2014. ACM Press, 193--196.

Digital Library

[47]

Liu Xing and Qian Feng. 2016. Measuring and optimizing android smartwatch energy consumption: poster. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 421--423.

Digital Library

[48]

Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2014. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch. In Proc. HotMobile 2015. ACM Press.

Digital Library

[49]

Yalan Ye, Wenwen He, Yunfei Cheng, Wenxia Huang, and Zhilin Zhang. 2017. A Robust Random Forest-Based Approach for Heart Rate Monitoring Using Photoplethysmography Signal Contaminated by Intense Motion Artifacts. MDPI 385 (2017).

[50]

Xiaobo Zhang and Xiangchu Feng. 2014. A New Gradient-based Nonlinear Diffusion Method Applied to Image Denoising. In Proc. PIC 2014. IEEE Press, 321--315.

[51]

Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body as an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proc. CHI 2016. ACM Press, 1491--1503.

Digital Library

Cited By

Zeng XWang XZhang TYu CZhao SChen Y(2024)GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model AgentsProceedings of the ACM on Human-Computer Interaction10.1145/36981458:ISS(462-499)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698145
Niwarthana ASomarathne PQian PYong KWithana AKostakos VKay JHoang T(2024)Efficient and Robust Heart Rate Estimation Approach for Noisy Wearable PPG Sensors Using Ideal Representation LearningProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676606(1-8)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676606
Zhang MMarin ERyan MKostakos VMurray TTag BOswald D(2024)OOBKey: Key Exchange with Implantable Medical Devices Using Out-Of-Band ChannelsProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3670876(1-13)Online publication date: 30-Jul-2024
https://dl.acm.org/doi/10.1145/3664476.3670876
Show More Cited By

Index Terms

FinDroidHR: Smartwatch Gesture Input with Optical Heartrate Monitor

Recommendations

fSense: Unlocking the Dimension of Force for Gestural Interactions using Smartwatch PPG Sensor
AH2019: Proceedings of the 10th Augmented Human International Conference 2019

While most existing gestural interfaces focus on the static posture or the dynamic action of the hand, few have investigated the feasibility of using the forces that are exerted while performing gestures. Using the photoplethysmogram (PPG) sensor of off-...
Hands-free gesture control with a capacitive textile neckband
ISWC '14: Proceedings of the 2014 ACM International Symposium on Wearable Computers

We present a novel sensing modality for hands-free gesture controlled user interfaces, based on active capacitive sensing. Four capacitive electrodes are integrated into a textile neckband, allowing continuous unobtrusive head movement monitoring. We ...
Wearable EOG goggles: eye-based interaction in everyday environments
CHI EA '09: CHI '09 Extended Abstracts on Human Factors in Computing Systems

In this paper, we present an embedded eye tracker for context-awareness and eye-based human-computer interaction - the wearable EOG goggles. In contrast to common systems using video, this unobtrusive device relies on Electrooculography (EOG). It ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 2, Issue 1

March 2018

1370 pages

EISSN:2474-9567

DOI:10.1145/3200905

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 March 2018

Accepted: 01 January 2018

Revised: 01 November 2017

Received: 01 August 2017

Published in IMWUT Volume 2, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

The Australian Research Council (ARC)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

67
Total Citations
View Citations
965
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)7

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zeng XWang XZhang TYu CZhao SChen Y(2024)GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model AgentsProceedings of the ACM on Human-Computer Interaction10.1145/36981458:ISS(462-499)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698145
Niwarthana ASomarathne PQian PYong KWithana AKostakos VKay JHoang T(2024)Efficient and Robust Heart Rate Estimation Approach for Noisy Wearable PPG Sensors Using Ideal Representation LearningProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676606(1-8)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676606
Zhang MMarin ERyan MKostakos VMurray TTag BOswald D(2024)OOBKey: Key Exchange with Implantable Medical Devices Using Out-Of-Band ChannelsProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3670876(1-13)Online publication date: 30-Jul-2024
https://dl.acm.org/doi/10.1145/3664476.3670876
Rony RAhmed MSarcar SAhmed N(2024)Understanding Driving Stress in Urban Bangladesh: An Exploratory Study, Wearable Development and ExperimentACM Journal on Computing and Sustainable Societies10.1145/36484342:2(1-28)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3648434
Chen WLin SPeng ZParizi FHeo SPatel SMatusik WZhao WStankovic J(2024)ViObjectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435478:1(1-26)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643547
Yi EZhang FXiong JNiu KYao ZZhang D(2024)Enabling WiFi Sensing on New-generation WiFi CardsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36338077:4(1-26)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3633807
Khanna PRamakrishnan IJain SBi XBalasubramanian A(2024)Hand Gesture Recognition for Blind Users by Tracking 3D Gesture TrajectoryProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642602(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642602
Qi HXiong JCao MLai WHu Y(2024)Wrist Information Detection Sensor Based on Pinhole ImagingIEEE Photonics Journal10.1109/JPHOT.2024.343583216:6(1-6)Online publication date: Dec-2024
https://doi.org/10.1109/JPHOT.2024.3435832
Liu XLi FCao YZhai SYang SWang Y(2024)PPGSpotter: Personalized Free Weight Training Monitoring Using Wearable PPG SensorIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621212(2468-2477)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621212
Wu YNi HMao CHan JXu W(2023)Non-intrusive Human Vital Sign Detection Using mmWave Sensing Technologies: A ReviewACM Transactions on Sensor Networks10.1145/362716120:1(1-36)Online publication date: 3-Nov-2023
https://dl.acm.org/doi/10.1145/3627161
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents