FacialPen: Using Facial Detection to Augment Pen-Based Interaction
Pages 1 - 8
Abstract
Pen-based interactions have been ubiquitously adopted on mobile and stationary devices, but the usability can be further augmented through the use of advanced techniques. In this work, we propose FacialPen, a prototype that uses facial gestures to trigger commands for pen-based manipulation. In our prototype, a fisheye camera is mounted to the end of a stylus that provides a broad view from which to capture the human face. We facilitated an elicitation study to identify natural and user-defined gestures for interactions with facial expressions. Different gestures can be further discerned via face detection and a classification pipeline. We designed a sketching demonstration application to explore usage scenarios and evaluated the effectiveness of FacialPen through a qualitative study. The user study posits that FacialPen supports efficiency by reducing screen widgets, enabling the continuity of creation work and liberating the user’s stylus holding postures when switching sketch functions.
References
[1]
Anand Agarawala and Ravin Balakrishnan. 2006. Keepin’ It Real: Pushing the Desktop Metaphor with Physics, Piles and the Pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI ’06). Association for Computing Machinery, New York, NY, USA, 1283–1292. https://doi.org/10.1145/1124772.1124965
[2]
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. OpenFace 2.0: Facial Behavior Analysis Toolkit. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) (2018), 59–66.
[3]
Peter Brandl, Clifton Forlines, Daniel Wigdor, Michael Haller, and Chia Shen. 2008. Combining and Measuring the Benefits of Bimanual Pen and Direct-Touch Interaction on Horizontal Interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces (Napoli, Italy) (AVI ’08). Association for Computing Machinery, New York, NY, USA, 154–161. https://doi.org/10.1145/1385569.1385595
[4]
Martin Brinkmann. 2009. Opera Face. Retrieved September 17, 2020 from https://www.ghacks.net/2009/04/02/web-browser-opera-face-gestures/
[5]
Drini Cami, Fabrice Matulic, Richard G. Calland, Brian Vogel, and Daniel Vogel. 2018. Unimanual Pen+Touch Input Using Variations of Precision Grip Postures. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology(Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 825–837. https://doi.org/10.1145/3242587.3242652
[6]
Lenny Engelhardt. 2008. P-36: Native Dual Mode Digitizers: Supporting Pen, Touch and Multi-Touch Inputs in One Device on any LCD. SID Symposium Digest of Technical Papers 39, 1 (2008), 1306. https://doi.org/10.1889/1.3069382
[7]
Nicholas Fellion, Thomas Pietrzak, and Audrey Girouard. 2017. FlexStylus: Leveraging Bend Input for Pen Interaction. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 375–385. https://doi.org/10.1145/3126594.3126597
[8]
Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen + Touch = New Tools. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST ’10). Association for Computing Machinery, New York, NY, USA, 27–36. https://doi.org/10.1145/1866029.1866036
[9]
Sebastian Hueber, Christian Cherek, Philipp Wacker, Jan Borchers, and Simon Voelker. 2020. Headbang: Using head gestures to trigger discrete actions on mobile devices. Conference Proceedings - 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services: Expanding the Horizon of Mobile Interaction, MobileHCI 2020(2020). https://doi.org/10.1145/3379503.3403538
[10]
Mohamed Khamis, Florian Alt, Mariam Hassib, Emanuel von Zezschwitz, Regina Hasholzner, and Andreas Bulling. 2016. GazeTouchPass: Multimodal Authentication Using Gaze and Touch on Mobile Devices. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 2156–2164. https://doi.org/10.1145/2851581.2892314
[11]
Kosuke Kikui, Yuta Itoh, Makoto Yamada, Yuta Sugiura, and Maki Sugimoto. 2018. Intra-/Inter-User Adaptation Framework for Wearable Gesture Sensing Device. In Proceedings of the 2018 ACM International Symposium on Wearable Computers (Singapore, Singapore) (ISWC ’18). Association for Computing Machinery, New York, NY, USA, 21–24. https://doi.org/10.1145/3267242.3267256
[12]
Davis E. King. 2009. Dlib-Ml: A Machine Learning Toolkit. J. Mach. Learn. Res. 10 (Dec. 2009), 1755–1758.
[13]
Michael Lankes, Stefan Riegler, Astrid Weiss, Thomas Mirlacher, Michael Pirker, and Manfred Tscheligi. 2008. Facial Expressions as Game Input with Different Emotional Feedback Conditions. In Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology (Yokohama, Japan) (ACE ’08). Association for Computing Machinery, New York, NY, USA, 253–256. https://doi.org/10.1145/1501750.1501809
[14]
K. Masai, K. Kunze, D. Sakamoto, Y. Sugiura, and M. Sugimoto. 2020. Face Commands - User-Defined Facial Gestures for Smart Glasses. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 374–386. https://doi.org/10.1109/ISMAR50242.2020.00064
[15]
Katsutoshi Masai, Yuta Sugiura, Masa Ogata, Katsuhiro Suzuki, Fumihiko Nakamura, Sho Shimamura, Kai Kunze, Masahiko Inami, and Maki Sugimoto. 2015. AffectiveWear: Toward Recognizing Facial Expression. In ACM SIGGRAPH 2015 Posters (Los Angeles, California) (SIGGRAPH ’15). Association for Computing Machinery, New York, NY, USA, Article 16, 1 pages. https://doi.org/10.1145/2787626.2792632
[16]
Fabrice Matulic, Riku Arakawa, Brian Vogel, and Daniel Vogel. 2020. PenSight: Enhanced Interaction with a Pen-Top Camera. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376147
[17]
Fabrice Matulic, Brian Vogel, Naoki Kimura, and Daniel Vogel. 2019. Eliciting Pen-Holding Postures for General Input with Suitability for EMG Armband Detection. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (Daejeon, Republic of Korea) (ISS ’19). Association for Computing Machinery, New York, NY, USA, 89–100. https://doi.org/10.1145/3343055.3359720
[18]
Daniel McDuff, Abdelrahman Mahmoud, Mohammad Mavadati, May Amr, Jay Turcot, and Rana el Kaliouby. 2016. AFFDEX SDK: A Cross-Platform Real-Time Multi-Face Expression Recognition Toolkit. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 3723–3726. https://doi.org/10.1145/2851581.2890247
[19]
Viral Parekh, Ramanathan Subramanian, and C. V. Jawahar. 2017. Eye Contact Detection via Deep Neural Networks. In HCI International 2017 – Posters’ Extended Abstracts, Constantine Stephanidis(Ed.). Springer International Publishing, Cham, 366–374. https://doi.org/10.1007/978-3-319-58750-9_51
[20]
Ken Pfeuffer, Jason Alexander, Ming Ki Chong, Yanxia Zhang, and Hans Gellersen. 2015. Gaze-Shifting: Direct-Indirect Input with Pen and Touch Modulated by Gaze. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology(Charlotte, NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, USA, 373–383. https://doi.org/10.1145/2807442.2807460
[21]
Ken Pfeuffer, Ken Hinckley, Michel Pahud, and Bill Buxton. 2017. Thumb + Pen Interaction on Tablets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3254–3266. https://doi.org/10.1145/3025453.3025567
[22]
Kimiko Ryokai, Stefan Marti, and Hiroshi Ishii. 2004. I/O Brush: Drawing with Everyday Objects as Ink. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI ’04). Association for Computing Machinery, New York, NY, USA, 303–310. https://doi.org/10.1145/985692.985731
[23]
Ticha Sethapakdi and James McCann. 2019. Painting with CATS: Camera-Aided Texture Synthesis. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–9. https://doi.org/10.1145/3290605.3300287
[24]
Xuepeng Shi, Shiguang Shan, Meina Kan, Shuzhe Wu, and Xilin Chen. 2018. Real-Time Rotation-Invariant Face Detection with Progressive Calibration Networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2295–2303. https://doi.org/10.1109/CVPR.2018.00244 arxiv:1804.06039
[25]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556(2014).
[26]
Brian A. Smith, Qi Yin, Steven K. Feiner, and Shree K. Nayar. 2013. Gaze Locking: Passive Eye Contact Detection for Human-Object Interaction. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 271–280. https://doi.org/10.1145/2501988.2501994
[27]
Hyunyoung Song, Hrvoje Benko, Francois Guimbretiere, Shahram Izadi, Xiang Cao, and Ken Hinckley. 2011. Grips and Gestures on a Multi-Touch Pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 1323–1332. https://doi.org/10.1145/1978942.1979138
[28]
Hemant Bhaskar Surale, Fabrice Matulic, and Daniel Vogel. 2017. Experimental Analysis of Mode Switching Techniques in Touch-Based User Interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3267–3280. https://doi.org/10.1145/3025453.3025865
[29]
Alan Transon, Adrien Verhulst, Jean Marie Normand, Guillaume Moreau, and Maki Sugimoto. 2018. Evaluation of facial expressions as an interaction mechanism and their impact on affect, workload and usability in an AR game. In Proceedings of the 2017 23rd International Conference on Virtual Systems and Multimedia, VSMM 2017 2018-January (2018), 1–8. https://doi.org/10.1109/VSMM.2017.8346263
[30]
Theophanis Tsandilas. 2018. Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation. ACM Trans. Comput.-Hum. Interact. 25, 3, Article 18 (June 2018), 49 pages. https://doi.org/10.1145/3182168
[31]
Akino Umezawa, Yoshinari Takegawa, Katsuhiro Suzuki, Katsutoshi Masai, Yuta Sugiura, Maki Sugimoto, Yutaka Tokuda, Diego Martinez Plasencia, Sriram Subramanian, Masafumi Takahashi, Hiroaki Taka, and Keiji Hirata. 2020. E2-MaskZ: A Mask-Type Display with Facial Expression Identification Using Embedded Photo Reflective Sensors. In Proceedings of the Augmented Humans International Conference (Kaiserslautern, Germany) (AHs ’20). Association for Computing Machinery, New York, NY, USA, Article 36, 3 pages. https://doi.org/10.1145/3384657.3385332
[32]
Daniel Vogel and Géry Casiez. 2011. Conté: Multimodal Input Inspired by an Artist’s Crayon. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST ’11). Association for Computing Machinery, New York, NY, USA, 357–366. https://doi.org/10.1145/2047196.2047242
[33]
Wacom.Com. 2006. Wacom One Creative Pen Display. Retrieved September 17, 2020 from https://www.wacom.com/en-us/products/pen-displays/wacom-one
[34]
Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-Defined Gestures for Surface Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 1083–1092. https://doi.org/10.1145/1518701.1518866
[35]
Haijun Xia, Ken Hinckley, Michel Pahud, Xiao Tu, and Bill Buxton. 2017. WritLarge: Ink Unleashed by Unified Scope, Action, & Zoom. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3227–3240. https://doi.org/10.1145/3025453.3025664
[36]
Yukang Yan, Chun Yu, Wengrui Zheng, Ruining Tang, Xuhai Xu, and Yuanchun Shi. 2020. FrownOnError: Interrupting Responses from Smart Speakers by Facial Expressions. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376810
[37]
Xiaoyi Zhang, Harish Kulkarni, and Meredith Ringel Morris. 2017. Smartphone-Based Gaze Gesture Communication for People with Motor Disabilities. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 2878–2889. https://doi.org/10.1145/3025453.3025790
[38]
Xucong Zhang, Yusuke Sugano, and Andreas Bulling. 2019. Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300646
Recommendations
PenShaft: Enabling Pen Shaft Detection and Interaction for Touchscreens
AH2021: 12th Augmented Human International ConferencePenShaft is a battery-free, easy to implement solution for augmenting the shaft of a capacitive pen with interactive capabilities. By applying conductive materials in a specific pattern on the pen’s shaft, we can detect when it is laid on a capacitive ...
Facial Expression Detection Techniques: Based on Viola and Jones Algorithm and Principal Component Analysis
ACCT '15: Proceedings of the 2015 Fifth International Conference on Advanced Computing & Communication TechnologiesFacial expression is a prominent posture beneath the skin of the face. They are the way of communication in humans which convey many things non-verbally. During the past years face recognition has received significant attention as one of the most ...
Thumb + Pen Interaction on Tablets
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsModern tablets support simultaneous pen and touch input, but it remains unclear how to best leverage this capability for bimanual input when the nonpreferred hand holds the tablet. We explore Thumb + Pen interactions that support simultaneous pen and ...
Comments
Information & Contributors
Information
Published In
May 2021
228 pages
ISBN:9781450382038
DOI:10.1145/3429360
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 September 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- JSPS KAKENHI
Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 347Total Downloads
- Downloads (Last 12 months)32
- Downloads (Last 6 weeks)5
Reflects downloads up to 25 Feb 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format