Fabrice Matulic
Daniel Vogel
ABSTRACT
We propose mounting a downward-facing camera above the top end of a digital tablet pen. This creates a unique and practical viewing angle for capturing the pen-holding hand and the immediate surroundings which can include the other hand. The fabrication of a prototype device is described and the enabled interaction design space is explored, including dominant and non-dominant hand pose recognition, tablet grip detection, hand gestures, capturing physical content in the environment, and detecting users and pens. A deep learning computer vision pipeline is developed for classification, regression, and keypoint detection to enable these interactions. Example applications demonstrate usage scenarios and a qualitative user evaluation confirms the potential of the approach.
Supplemental Material
- 6 Key Ways To Hold A Watercolor Brush. https://watercolorpainting.com/brush-exercise/. Accessed: 2019-09--17.Google Scholar
- Anoto Livescribe. https://www.anoto.com/solutions/livescribe/. Accessed: 2019-09-01.Google Scholar
- Ilhan Aslan, Björn Bittner, Florian Müller, and Elisabeth André. 2018. Exploring the User Experience of Proxemic Hand and Pen Input Above and Aside a Drawing Screen. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia. ACM, 183--192.Google ScholarDigital Library
- Ilhan Aslan, Ida Buchwald, Philipp Koytek, and Elisabeth André. 2016. Pen + Mid-Air: An Exploration of Mid-Air Gestures to Complement Pen Input on Tablets. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (NordiCHI '16). ACM, New York, NY, USA, 1:1--1:10. DOI: http://dx.doi.org/10.1145/2971485.2971511Google ScholarDigital Library
- Ilhan Aslan, Tabea Schmidt, Jens Woehrle, Lukas Vogel, and Elisabeth André. 2018. Pen + Mid-Air Gestures: Eliciting Contextual Gestures. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI '18). ACM, New York, NY, USA, 135--144. DOI: http://dx.doi.org/10.1145/3242969.3242979Google ScholarDigital Library
- Xiaojun Bi, Tomer Moscovich, Gonzalo Ramos, Ravin Balakrishnan, and Ken Hinckley. 2008. An Exploration of Pen Rolling for Pen-based Interaction. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 191--200. DOI: http://dx.doi.org/10.1145/1449715.1449745Google ScholarDigital Library
- Peter Brandl, Clifton Forlines, Daniel Wigdor, Michael Haller, and Chia Shen. 2008. Combining and Measuring the Benefits of Bimanual Pen and Direct-touch Interaction on Horizontal Interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '08). ACM, New York, NY, USA, 154--161. DOI: http://dx.doi.org/10.1145/1385569.1385595Google ScholarDigital Library
- Drini Cami, Fabrice Matulic, Richard G Calland, Brian Vogel, and Daniel Vogel. 2018. Unimanual Pen+Touch Input Using Variations of Precision Grip Postures. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 825--837. DOI: http://dx.doi.org/10.1145/3242587.3242652Google ScholarDigital Library
- Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In arXiv preprint arXiv:1812.08008.Google Scholar
- Liwei Chan, Yi-Ling Chen, Chi-Hao Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2015. CyclopsRing: Enabling Whole-Hand and Context-Aware Interactions Through a Fisheye Ring. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 549--556. DOI: http://dx.doi.org/10.1145/2807442.2807450Google ScholarDigital Library
- Xinghao Chen. Awesome Hand Pose Estimation. https://github.com/xinghaochen/ awesome-hand-pose-estimation. Accessed: 2019-09-01.Google Scholar
- Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. 2014. Air+Touch: Interweaving Touch & In-air Gestures. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 519--525. DOI: http://dx.doi.org/10.1145/2642918.2647392Google ScholarDigital Library
- Artem Dementyev and Joseph A Paradiso. 2014. WristFlex: Low-power Gesture Input with Wrist-worn Pressure Sensors. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 161--166. DOI: http://dx.doi.org/10.1145/2642918.2647396Google ScholarDigital Library
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database.. In 2009 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). IEEE Computer Society, 248--255. DOI: http://dx.doi.org/10.1109/CVPR.2009.5206848Google ScholarCross Ref
- Nicholas Fellion, Thomas Pietrzak, and Audrey Girouard. 2017. FlexStylus: Leveraging Bend Input for Pen Interaction. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 375--385. DOI: http://dx.doi.org/10.1145/3126594.3126597Google ScholarDigital Library
- Tovi Grossman, Ken Hinckley, Patrick Baudisch, Maneesh Agrawala, and Ravin Balakrishnan. 2006. Hover Widgets: Using the Tracking State to Extend the Capabilities of Pen-operated Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 861--870. DOI: http://dx.doi.org/10.1145/1124772.1124898Google ScholarDigital Library
- Yves Guiard. 1987. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of motor behavior 19, 4 (1987), 486--517.Google ScholarCross Ref
- Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. 2016. Synthetic data for text localisation in natural images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2315--2324.Google ScholarCross Ref
- Khalad Hasan, Xing-Dong Yang, Andrea Bunt, and Pourang Irani. 2012. A-coord Input: Coordinating Auxiliary Input Streams for Augmenting Contextual Pen-based Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 805--814. DOI: http://dx.doi.org/10.1145/2207676.2208519Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.Google ScholarCross Ref
- Ken Hinckley, Xiang 'Anthony' Chen, and Hrvoje Benko. 2013. Motion and Context Sensing Techniques for Pen Computing. In Proceedings of Graphics Interface 2013 (GI '13). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 71--78. http://dl.acm.org/citation.cfm?id=2532129.2532143Google ScholarDigital Library
- Ken Hinckley, Michel Pahud, Hrvoje Benko, Pourang Irani, François Guimbretière, Marcel Gavriliu, Xiang 'Anthony' Chen, Fabrice Matulic, William Buxton, and Andrew Wilson. 2014. Sensing Techniques for Tablet+Stylus Interaction. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 605--614. DOI: http://dx.doi.org/10.1145/2642918.2647379Google ScholarDigital Library
- Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen + touch = new tools. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, New York, New York, USA, 27--36. DOI: http://dx.doi.org/10.1145/1866029.1866036Google ScholarDigital Library
- Sungjae Hwang, Andrea Bianchi, Myungwook Ahn, and Kwangyun Wohn. 2013. MagPen: Magnetically Driven Pen Interactions on and Around Conventional Smartphones. In Proceedings of the 15th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '13). ACM, New York, NY, USA, 412--415. DOI: http://dx.doi.org/10.1145/2493190.2493194Google ScholarDigital Library
- Yasha Iravantchi, Yang Zhang, Evi Bernitsas, Mayank Goel, and Chris Harrison. 2019. Interferi: Gesture Sensing Using On-Body Acoustic Interferometry. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 276:1--276:13. DOI: http://dx.doi.org/10.1145/3290605.3300506Google ScholarDigital Library
- David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 167--176. DOI: http://dx.doi.org/10.1145/2380116.2380139Google ScholarDigital Library
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Rui Li, Zhenyu Liu, and Jianrong Tan. 2019. A survey on 3D hand pose estimation: Cameras, methods, and datasets. Pattern Recognition 93 (2019), 251--272.Google ScholarDigital Library
- Michael Linderman, Mikhail A Lebedev, and Joseph S Erlichman. 2009. Recognition of handwriting from electromyography. PLoS One 4, 8 (2009), e6791.Google ScholarCross Ref
- Shenwei Liu and François Guimbretière. 2012. FlexAura: A Flexible Near-surface Range Sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 327--330. DOI: http://dx.doi.org/10.1145/2380116.2380158Google ScholarDigital Library
- Fabrice Matulic and Moira Norrie. 2013. Pen and Touch Gestural Environment for Document Editing on Interactive Tabletops. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces. ACM, St Andrews, UK, 41--50.Google ScholarDigital Library
- Fabrice Matulic and Moira C. Norrie. 2012. Supporting Active Reading on Pen and Touch-operated Tabletops. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '12). ACM, New York, NY, USA, 612--619. DOI: http://dx.doi.org/10.1145/2254556.2254669Google ScholarDigital Library
- Fabrice Matulic, Brian Vogel, Naoki Kimura, and Daniel Vogel. 2019. Eliciting Pen-Holding Postures for General Input with Suitability for EMG Armband Detection. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (ISS '19). ACM, New York, NY, USA, 89--100. DOI: http://dx.doi.org/10.1145/3343055.3359720Google ScholarDigital Library
- Fabrice Matulic, Daniel Vogel, and Raimund Dachselt. 2017. Hand Contact Shape Recognition for Posture-Based Tabletop Widgets and Interaction. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 3--11. DOI: http://dx.doi.org/10.1145/3132272.3134126Google ScholarDigital Library
- Jess McIntosh, Asier Marzo, and Mike Fraser. 2017a. SensIR: Detecting Hand Gestures with a Wearable Bracelet Using Infrared Transmission and Reflection. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 593--597. DOI: http://dx.doi.org/10.1145/3126594.3126604Google ScholarDigital Library
- Jess McIntosh, Asier Marzo, Mike Fraser, and Carol Phillips. 2017b. EchoFlex: Hand Gesture Recognition Using Ultrasound Imaging. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1923--1934. DOI: http://dx.doi.org/10.1145/3025453.3025807Google ScholarDigital Library
- Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2018. Ganerated hands for real-time 3d hand tracking from monocular rgb. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 49--59.Google ScholarCross Ref
- S. K. Nayar. 1997. Catadioptric omnidirectional camera. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 482--488. DOI:http://dx.doi.org/10.1109/CVPR.1997.609369Google ScholarCross Ref
- Ken Pfeuffer, Ken Hinckley, Michel Pahud, and Bill Buxton. 2017. Thumb + Pen Interaction on Tablets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3254--3266. DOI: http://dx.doi.org/10.1145/3025453.3025567Google ScholarDigital Library
- Raf Ramakers, Davy Vanacken, Kris Luyten, Karin Coninx, and Johannes Schöning. 2012. Carpus: A Non-intrusive User Identification Technique for Interactive Surfaces. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 35--44. DOI:http://dx.doi.org/10.1145/2380116.2380123Google ScholarDigital Library
- Jef Raskin. 2000. The Humane Interface: New Directions for Designing Interactive Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA.Google ScholarDigital Library
- Jun Rekimoto. 1997. Pick-and-drop: A Direct Manipulation Technique for Multiple Computer Environments. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology (UIST '97). ACM, New York, NY, USA, 31--39. DOI: http://dx.doi.org/10.1145/263407.263505Google ScholarDigital Library
- T Scott Saponas, Desney S Tan, Dan Morris, and Ravin Balakrishnan. 2008. Demonstrating the Feasibility of Using Forearm Electromyography for Muscle-computer Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 515--524. DOI: http://dx.doi.org/10.1145/1357054.1357138Google ScholarDigital Library
- Dominik Schmidt, Ming Ki Chong, and Hans Gellersen. 2010. HandsDown: Hand-contour-based User Identification for Interactive Surfaces. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI '10). ACM, New York, NY, USA, 432--441. DOI: http://dx.doi.org/10.1145/1868914.1868964Google ScholarDigital Library
- A. Seniuk and D. Blostein. 2009. Pen Acoustic Emissions for Text and Gesture Recognition. In 2009 10th International Conference on Document Analysis and Recognition. 872--876. DOI: http://dx.doi.org/10.1109/ICDAR.2009.251Google ScholarDigital Library
- Hyunyoung Song, Hrvoje Benko, Francois Guimbretiere, Shahram Izadi, Xiang Cao, and Ken Hinckley. 2011. Grips and Gestures on a Multi-touch Pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1323--1332. DOI: http://dx.doi.org/10.1145/1978942.1979138Google ScholarDigital Library
- Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-air Gestures Around Unmodified Mobile Devices. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 319--329. DOI: http://dx.doi.org/10.1145/2642918.2647373Google ScholarDigital Library
- Yu Suzuki, Kazuo Misue, and Jiro Tanaka. 2009. Interaction Technique for a Pen-Based Interface Using Finger Motions. In Human-Computer Interaction. Novel Interaction Methods and Techniques, Julie A. Jacko (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 503--512.Google Scholar
- Marc Teyssier, Gilles Bailly, and Éric Lecolinet. 2017. VersaPen: An Adaptable, Modular and Multimodal I/O Pen. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 2155--2163. DOI: http://dx.doi.org/10.1145/3027063.3053159Google ScholarDigital Library
- Feng Tian, Lishuang Xu, Hongan Wang, Xiaolong Zhang, Yuanyuan Liu, Vidya Setlur, and Guozhong Dai. 2008. Tilt Menu: Using the 3D Orientation Information of Pen Devices to Extend the Selection Capability of Pen-based User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 1371--1380. DOI: http://dx.doi.org/10.1145/1357054.1357269Google ScholarDigital Library
- Andrew M. Webb, Hannah Fowler, Andruid Kerne, Galen Newman, Jun-Hyun Kim, and Wendy E. Mackay. 20Interstices: Sustained Spatial RelationshipsBetween Hands and Surfaces Reveal Anticipated Action. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Article 588, 12 pages. DOI: http://dx.doi.org/10.1145/3290605.3300818Google ScholarDigital Library
- Hongyi Wen, Julian Ramos Rojas, and Anind K Dey. 2016. Serendipity: Finger Gesture Recognition Using an Off-the-Shelf Smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3847--3851. DOI: http://dx.doi.org/10.1145/2858036.2858466Google ScholarDigital Library
- Mike Wu and Ravin Balakrishnan. 2003. Multi-finger and Whole Hand Gestural Interaction Techniques for Multi-user Tabletop Displays. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST '03). ACM, New York, NY, USA, 193--202. DOI: http://dx.doi.org/10.1145/964696.964718Google ScholarDigital Library
- Chao Xu, Parth H Pathak, and Prasant Mohapatra. 2015. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition Using Smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications (HotMobile '15). ACM, New York, NY, USA, 9--14. DOI:http://dx.doi.org/10.1145/2699343.2699350Google ScholarDigital Library
- Xing-Dong Yang, Khalad Hasan, Neil Bruce, and Pourang Irani. 2013. Surround-see: Enabling Peripheral Vision on Smartphones During Active Use. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 291--300. DOI: http://dx.doi.org/10.1145/2501988.2502049Google ScholarDigital Library
- Zhongliang Yang and Yumiao Chen. 2016. Surface EMG-based sketching recognition using two analysis windows and gene expression programming. Frontiers in neuroscience 10 (2016), 445.Google Scholar
- Dongwook Yoon, Ken Hinckley, Hrvoje Benko, François Guimbretière, Pourang Irani, Michel Pahud, and Marcel Gavriliu. 2015. Sensing Tablet Grasp + Micro-mobility for Active Reading. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 477--487. DOI: http://dx.doi.org/10.1145/2807442.2807510Google ScholarDigital Library
- Chun Yu, Xiaoying Wei, Shubh Vachher, Yue Qin, Chen Liang, Yueting Weng, Yizheng Gu, and Yuanchun Shi. 2019. HandSee: Enabling Full Hand Interaction on Smartphone with Front Camera-based Stereo Vision. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Article 705, 13 pages. DOI: http://dx.doi.org/10.1145/3290605.3300935Google ScholarDigital Library
- X Zhang, X Chen, Y Li, V Lantz, K Wang, and J Yang. 2011. A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 41, 6 (nov 2011), 1064--1076. DOI: http://dx.doi.org/10.1109/TSMCA.2011.2116004Google ScholarDigital Library
- Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 167--173. DOI: http://dx.doi.org/10.1145/2807442.2807480Google ScholarDigital Library
- Yang Zhang, Michel Pahud, Christian Holz, Haijun Xia, Gierad Laput, Michael McGuffin, Xiao Tu, Andrew Mittereder, Fei Su, William Buxton, and Ken Hinckley. 20Sensing Posture-Aware Pen+Touch Interaction on Tablets. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 55:1--55:14. DOI: http://dx.doi.org/10.1145/3290605.3300285Google ScholarDigital Library
- Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017. EAST: an efficient and accurate scene text detector. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 5551--5560.Google ScholarCross Ref
Index Terms
- PenSight: Enhanced Interaction with a Pen-Top Camera
Recommendations
An exploration of pen tail gestures for interactions
In this paper, we performed an exploration on the design and evaluation of pen tail gesture, an interaction method that allows the use of pen tail movement to initiate interactions. Based on our interviews with some designers and researchers who ...
Thumb + Pen Interaction on Tablets
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsModern tablets support simultaneous pen and touch input, but it remains unclear how to best leverage this capability for bimanual input when the nonpreferred hand holds the tablet. We explore Thumb + Pen interactions that support simultaneous pen and ...
Unimanual Pen+Touch Input Using Variations of Precision Grip Postures
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyWe introduce a new pen input space by forming postures with the same hand that also grips the pen while writing, drawing, or selecting. The postures contact the multitouch surface around the pen to enable detection without special sensors. A formative ...
Comments