research-article

DeepFisheye: Near-Surface Multi-Finger Tracking Technology Using Fisheye Camera

Authors:

Geehyuk LeeAuthors Info & Claims

UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Pages 1132 - 1146

https://doi.org/10.1145/3379337.3415818

Published: 20 October 2020 Publication History

Abstract

Near-surface multi-finger tracking (NMFT) technology expands the input space of touchscreens by enabling novel interactions such as mid-air and finger-aware interactions. We present DeepFisheye, a practical NMFT solution for mobile devices, that utilizes a fisheye camera attached at the bottom of a touchscreen. DeepFisheye acquires the image of an interacting hand positioned above the touchscreen using the camera and employs deep learning to estimate the 3D position of each fingertip. We created two new hand pose datasets comprising fisheye images, on which our network was trained. We evaluated DeepFisheye's performance for three device sizes. DeepFisheye showed average errors with approximate value of 20 mm for fingertip tracking across the different device sizes. Additionally, we created simple rule-based classifiers that estimate the contact finger and hand posture from DeepFisheye's output. The contact finger and hand posture classifiers showed accuracy of approximately 83 and 90%, respectively, across the device sizes.

Supplementary Material

VTT File (3379337.3415818.vtt)

Download
7.21 KB

ZIP File (ufp1945aux.zip)

The supplementary document contains detail images of the deep learning network and addtional supportive images.

Download
2.93 MB

MP4 File (ufp1945pv.mp4)

Preview video

Download
4.20 MB

MP4 File (ufp1945vf.mp4)

Video figure

Download
58.49 MB

MP4 File (3379337.3415818.mp4)

Presentation Video

Download
20.69 MB

References

[1]

2016. Spatio-Temporal Hough Forest for Efficient Detection-Localisation-Recognition of Fingerwriting in Egocentric Camera. Comput. Vis. Image Underst. 148, C (July 2016), 87--96.

[2]

Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. 2018. Augmented Skeleton Space Transfer for Depth-Based Hand Pose Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]

Seungryul Baek, Kwang In Kim, and Tae-Kyun Kim. 2019. Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]

Gilles Bailly, Jörg Müller, Michael Rohs, Daniel Wigdor, and Sven Kratz. 2012. ShoeSense: A New Perspective on Gestural Interaction and Wearable Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). Association for Computing Machinery, New York, NY, USA, 1239--1248.

Digital Library

[5]

Sven Bambach, Stefan Lee, David J. Crandall, and Chen Yu. 2015. Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions. In The IEEE International Conference on Computer Vision (ICCV).

[6]

Hrvoje Benko, T. Scott Saponas, Dan Morris, and Desney Tan. 2009. Enhancing Input on and above the Interactive Surface with Muscle Sensing. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09). Association for Computing Machinery, New York, NY, USA, 93--100.

Digital Library

[7]

Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. 2018. Weakly-supervised 3D Hand Pose Estimation from Monocular RGB Images. In The European Conference on Computer Vision (ECCV).

[8]

Liwei Chan, Yi-Ling Chen, Chi-Hao Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2015. CyclopsRing: Enabling Whole-Hand and Context-Aware Interactions Through a Fisheye Ring. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). Association for Computing Machinery, New York, NY, USA, 549--556.

Digital Library

[9]

Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013a. UTrack: 3D Input Using Two Magnetic Sensors. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). Association for Computing Machinery, New York, NY, USA, 237--244.

Digital Library

[10]

Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013b. UTrack: 3D Input Using Two Magnetic Sensors. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). Association for Computing Machinery, New York, NY, USA, 237--244.

Digital Library

[11]

Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 1504--1514.

Digital Library

[12]

X. Chen, G. Wang, C. Zhang, T. Kim, and X. Ji. 2018. SHPR-Net: Deep Semantic Hand Pose Regression From Point Clouds. IEEE Access 6 (2018), 43425--43439.

[13]

Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. 2014. Air+touch: Interweaving Touch & in-Air Gestures. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). Association for Computing Machinery, New York, NY, USA, 519--525.

Digital Library

[14]

Ashley Colley and Jonna Häkkilä. 2014. Exploring Finger Specific Touch Screen Interaction for Mobile Phone User Interfaces. In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design (OzCHI '14). Association for Computing Machinery, New York, NY, USA, 539--548.

Digital Library

[15]

Benjamin Coors, Alexandru Paul Condurache, and Andreas Geiger. 2018. SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images. In The European Conference on Computer Vision (ECCV).

[16]

5DT DataGlove. 2020. 5DT Data Glove. Web. (1 May 2020). Retrieved May 1, 2020 from https://5dt.com/5dt-data-glove-ultra/.

[17]

Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 2019. 3D Hand Shape and Pose Estimation From a Single RGB Image. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]

Hyunjae Gil, DoYoung Lee, Seunggyu Im, and Ian Oakley. 2017. TriTap: Identifying Finger Touches on Smartwatches. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, New York, NY, USA, 3879--3890.

Digital Library

[19]

Oliver Glauser, Shihao Wu, Daniele Panozzo, Otmar Hilliges, and Olga Sorkine-Hornung. 2019. Interactive Hand Pose Estimation Using a Stretch-Sensing Soft Glove. ACM Trans. Graph. 38, 4, Article Article 41 (July 2019), 15 pages.

Digital Library

[20]

Alix Goguey, Géry Casiez, Daniel Vogel, Fanny Chevalier, Thomas Pietrzak, and Nicolas Roussel. 2014. A Three-Step Interaction Pattern for Improving Discoverability in Finger Identification Techniques. In Proceedings of the Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST'14 Adjunct). Association for Computing Machinery, New York, NY, USA, 33--34.

Digital Library

[21]

Alix Goguey, Mathieu Nancel, Géry Casiez, and Daniel Vogel. 2016. The Performance and Preference of Different Fingers and Chords for Pointing, Dragging, and Object Transformation. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 4250--4261.

Digital Library

[22]

Aakar Gupta, Muhammed Anwar, and Ravin Balakrishnan. 2016. Porous Interfaces for Small Screen Multitasking Using Finger Identification. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). Association for Computing Machinery, New York, NY, USA, 145--156.

Digital Library

[23]

Aakar Gupta and Ravin Balakrishnan. 2016. DualKey: Miniature Screen Text Entry via Finger Identification. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 59--70.

Digital Library

[24]

Chris Harrison and Scott E. Hudson. 2009. Abracadabra: Wireless, High-Precision, and Unpowered Finger Input for Very Small Mobile Devices. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). Association for Computing Machinery, New York, NY, USA, 121--124.

Digital Library

[25]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]

Ken Hinckley, Seongkook Heo, Michel Pahud, Christian Holz, Hrvoje Benko, Abigail Sellen, Richard Banks, Kenton O'Hara, Gavin Smyth, and William Buxton. 2016. Pre-Touch Sensing for Mobile Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 2869--2881.

Digital Library

[27]

Christian Holz and Patrick Baudisch. 2013. Fiberio: A Touchscreen That Senses Fingerprints. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). Association for Computing Machinery, New York, NY, USA, 41--50.

Digital Library

[28]

Y. Jang, I. Jeon, T. Kim, and W. Woo. 2017. Metaphoric Hand Gestures for Orientation-Aware VR Object Manipulation With an Egocentric Viewpoint. IEEE Transactions on Human-Machine Systems 47, 1 (2017), 113--127.

[29]

Insu Kim, Keunwoo Park, Youngwoo Yoon, and Geehyuk Lee. 2018. Touch180: Finger Identification on Mobile Touchscreen Using Fisheye Camera and Convolutional Neural Network. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings (UIST '18 Adjunct). Association for Computing Machinery, New York, NY, USA, 29--32.

Digital Library

[30]

J. Kim, N. D. Thang, and T. Kim. 2009. 3-D hand motion tracking and gesture recognition using a data glove. In 2009 IEEE International Symposium on Industrial Electronics. 1013--1018.

[31]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[32]

Huy Viet Le, Sven Mayer, and Niels Henze. 2018. InfiniTouch: Finger-Aware Interaction on Fully Touch Sensitive Smartphones. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). Association for Computing Machinery, New York, NY, USA, 779--792.

Digital Library

[33]

Mathieu Le Goc, Stuart Taylor, Shahram Izadi, and Cem Keskin. 2014. A Low-Cost Transparent Electric Field Sensor for 3d Interaction on Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). Association for Computing Machinery, New York, NY, USA, 3167--3170.

Digital Library

[34]

Kuo-Wei Lee, Shih-Hung Liu, Hwann-Tzong Chen, and Koichi Ito. 2019. Silhouette-Net: 3D Hand Pose Estimation from Silhouettes. (2019).

[35]

Y. Lee, J. Jeong, J. Yun, W. Cho, and K. Yoon. 2019. SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360° Images. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9173--9181.

[36]

S. Li. 2006. Monitoring Around a Vehicle by a Spherical Image Sensor. IEEE Transactions on Intelligent Transportation Systems 7, 4 (2006), 541--550.

Digital Library

[37]

Xuan Li, Chun-Ho Chen, and Yi-Pai Huang. 2016. 3D interactive system based on vision computing of direct-flective cameras. Journal of the Society for Information Display 24, 8 (2016), 521--528.

[38]

Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. 35, 4, Article Article 142 (July 2016), 19 pages.

Digital Library

[39]

Hyunchul Lim, Jungmin Chung, Changhoon Oh, SoHyun Park, Joonhwan Lee, and Bongwon Suh. 2018. Touch+Finger: Extending Touch-Based User Interface Capabilities with 'Idle' Finger Gestures in the Air. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). Association for Computing Machinery, New York, NY, USA, 335--346.

Digital Library

[40]

W. Lu, Z. Tong, and J. Chu. 2016. Dynamic Hand Gesture Recognition With Leap Motion Controller. IEEE Signal Processing Letters 23, 9 (2016), 1188--1192.

[41]

J. Malik, A. Elhayek, F. Nunnari, K. Varanasi, K. Tamaddon, A. Heloir, and D. Stricker. 2018. DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth. In 2018 International Conference on 3D Vision (3DV). 110--119.

[42]

ManusVR. 2020. ManusVR Prime One. Web. (1 May 2020). Retrieved May 1, 2020 from https://manus-vr.com/prime-one-gloves/.

[43]

Damien Masson, Alix Goguey, Sylvain Malacria, and Géry Casiez. 2017. WhichFingers: Identifying Fingers on Touch Surfaces and Keyboards Using Vibration Sensors. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). Association for Computing Machinery, New York, NY, USA, 41--48.

Digital Library

[44]

Jess McIntosh, Paul Strohmeier, Jarrod Knibbe, Sebastian Boring, and Kasper Hornbæk. 2019. Magnetips: Combining Fingertip Tracking and Haptic Feedback for Around-Device Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Article Paper 408, 12 pages.

Digital Library

[45]

Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. 2018. V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation From a Single Depth Map. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]

Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2018. GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB. In Proceedings of Computer Vision and Pattern Recognition (CVPR). 11. https: //handtracker.mpi-inf.mpg.de/projects/GANeratedHands/

[47]

Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A. Otaduy, Dan Casas, and Christian Theobalt. 2019. Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera. ACM Transactions on Graphics (TOG) 38, 4 (2019).

Digital Library

[48]

Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2017. Real-Time Hand Tracking Under Occlusion From an Egocentric RGB-D Sensor. In The IEEE International Conference on Computer Vision (ICCV) Workshops.

[49]

Sundar Murugappan, Vinayak, Niklas Elmqvist, and Karthik Ramani. 2012. Extended Multitouch: Recovering Touch Posture and Differentiating Users Using a Depth Camera. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). Association for Computing Machinery, New York, NY, USA, 487--496.

Digital Library

[50]

Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 1515--1525.

Digital Library

[51]

Vassilis C. Nicodemou, Iason Oikonomidis, Georgios Tzimiropoulos, and Antonis Argyros. 2018. Learning to Infer the Depth Map of a Hand from its Color Image. (2018).

[52]

M. Oberweger and V. Lepetit. 2017. DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation. In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). 585--594.

[53]

Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. 2015. Hands Deep in Deep Learning for Hand Pose Estimation. CoRR abs/1502.06807 (2015). http://arxiv.org/abs/1502.06807

[54]

Paschalis Panteleris and Antonis Argyros. 2017. Back to RGB: 3D Tracking of Hands and Hand-Object Interactions Based on Short-Baseline Stereo. In The IEEE International Conference on Computer Vision (ICCV) Workshops.

[55]

P. Panteleris, I. Oikonomidis, and A. Argyros. 2018. Using a Single RGB Frame for Real Time 3D Hand Pose Estimation in the Wild. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). 436--445.

[56]

Keunwoo Park, Daehwa Kim, Seongkook Heo, and Geehyuk Lee. 2020. MagTouch: Robust Finger Identification for a Smartwatch Using a Magnet Ring and a Built-in Magnetometer. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--13.

Digital Library

[57]

Leigh Ellen Potter, Jake Araullo, and Lewis Carter. 2013. The Leap Motion Controller: A View on Sign Language. In Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration (OzCHI '13). Association for Computing Machinery, New York, NY, USA, 175--178.

Digital Library

[58]

Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. 2014. Realtime and Robust Hand Tracking from Depth. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Digital Library

[59]

Zhou Ren, Junsong Yuan, and Zhengyou Zhang. 2011. Robust Hand Gesture Recognition Based on Finger-Earth Mover's Distance with a Commodity Depth Camera. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). Association for Computing Machinery, New York, NY, USA, 1093--1096.

Digital Library

[60]

D. Scaramuzza, A. Martinelli, and R. Siegwart. 2006. A Toolbox for Easily Calibrating Omnidirectional Cameras. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems. 5695--5701.

[61]

Srinath Sridhar, Anders Markussen, Antti Oulasvirta, Christian Theobalt, and Sebastian Boring. 2017. WatchSense: On- and Above-Skin Input Sensing through a Wearable Depth Sensor. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, New York, NY, USA, 3891--3902.

Digital Library

[62]

Yu-Chuan Su and Kristen Grauman. 2019. Kernel Transformer Networks for Compact Spherical Convolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]

Bugra Tekin, Federica Bogo, and Marc Pollefeys. 2019. H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[64]

Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. 2014. Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks. ACM Trans. Graph. 33, 5, Article Article 169 (Sept. 2014), 10 pages.

Digital Library

[65]

James Y Tung, Tea Lulic, Dave A Gonzalez, Johnathan Tran, Clark R Dickerson, and Eric A Roy. 2015. Evaluation of a portable markerless finger position capture device: accuracy of the Leap Motion controller in healthy adults. Physiological Measurement 36, 5 (apr 2015), 1025--1035.

[66]

Pier Paolo Valentini and Eugenio Pezzuti. 2017. Accuracy in fingertip tracking using Leap Motion Controller for interactive virtual applications. International Journal on Interactive Design and Manufacturing (IJIDeM) 11, 3 (2017), 641--650.

[67]

Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. 2019. Self-Supervised 3D Hand Pose Estimation Through Training by Fitting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[68]

Jingtao Wang and John Canny. 2004. FingerSense: Augmenting Expressiveness to Physical Pushing Button by Fingertip Identification. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (CHI EA '04). Association for Computing Machinery, New York, NY, USA, 1267--1270.

Digital Library

[69]

Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). Association for Computing Machinery, New York, NY, USA, 851--860.

Digital Library

[70]

Wenbin Wu, Chenyang Li, Zhuo Cheng, Xin Zhang, and Lianwen Jin. 2017. YOLSE: Egocentric Fingertip Detection From Single RGB Images. In The IEEE International Conference on Computer Vision (ICCV) Workshops.

[71]

W. Xu, A. Chatterjee, M. Zollhöfer, H. Rhodin, P. Fua, H. Seidel, and C. Theobalt. 2019. Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 2093--2101.

[72]

Linlin Yang and Angela Yao. 2019. Disentangling Latent Hands for Image Synthesis and Pose Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]

Xing-Dong Yang, Khalad Hasan, Neil Bruce, and Pourang Irani. 2013. Surround-See: Enabling Peripheral Vision on Smartphones during Active Use. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). Association for Computing Machinery, New York, NY, USA, 291--300.

Digital Library

[74]

Qi Ye and Tae-Kyun Kim. 2018. Occlusion-Aware Hand Pose Estimation Using Hierarchical Mixture Density Network. In Computer Vision -- ECCV 2018, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 817--834.

[75]

Chun Yu, Xiaoying Wei, Shubh Vachher, Yue Qin, Chen Liang, Yueting Weng, Yizheng Gu, and Yuanchun Shi. 2019. HandSee: Enabling Full Hand Interaction on Smartphone with Front Camera-Based Stereo Vision. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Article Paper 705, 13 pages.

Digital Library

[76]

Shanxin Yuan, Qi Ye, Bjorn Stenger, Siddhant Jain, and Tae-Kyun Kim. 2017. BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[77]

Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-Grained Acoustic-Based Device-Free Tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '17). Association for Computing Machinery, New York, NY, USA, 15--28.

Digital Library

[78]

X. Zhang, Q. Li, H. Mo, W. Zhang, and W. Zheng. 2019. End-to-End Hand Mesh Recovery From a Monocular RGB Image. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2354--2364.

[79]

Jingjie Zheng and Daniel Vogel. 2016. Finger-Aware Shortcuts. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 4274--4285.

[80]

Yidan Zhou, Jian Lu, Kuo Du, Xiangbo Lin, Yi Sun, and Xiaohong Ma. 2018. HBE: Hand Branch Ensemble Network for Real-time 3D Hand Pose Estimation. In The European Conference on Computer Vision (ECCV).

[81]

Christian Zimmermann and Thomas Brox. 2017. Learning to Estimate 3D Hand Pose From Single RGB Images. In The IEEE International Conference on Computer Vision (ICCV).

[82]

Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. 2019. FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images. In The IEEE International Conference on Computer Vision (ICCV).

Cited By

Herbert OPérez-Granados DRuiz MCadena Martínez RGutiérrez CAntuñano M(2024)Static and Dynamic Hand Gestures: A Review of Techniques of Virtual Reality ManipulationSensors10.3390/s2412376024:12(3760)Online publication date: 9-Jun-2024
https://doi.org/10.3390/s24123760
Lee LLin Z(2024)Danger, Nuisance, Disregard: Analyzing User-Generated Videos for Augmented Reality Gameplay on Hand-held DevicesProceedings of the ACM on Human-Computer Interaction10.1145/36770638:CHI PLAY(1-33)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677063
Huang ZGao CWang HDeng XLai YMa CQin SLiu YWang H(2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643559
Show More Cited By

Index Terms

DeepFisheye: Near-Surface Multi-Finger Tracking Technology Using Fisheye Camera
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Touch screens

Recommendations

Bare-hand human-computer interaction
PUI '01: Proceedings of the 2001 workshop on Perceptive user interfaces

In this paper, we describe techniques for barehanded interaction between human and computer. Barehanded means that no device and no wires are attached to the user, who controls the computer directly with the movements of his/her hand.Our approach is ...
Investigating the feasibility of finger identification on capacitive touchscreens using deep learning
IUI '19: Proceedings of the 24th International Conference on Intelligent User Interfaces

Touchscreens enable intuitive mobile interaction. However, touch input is limited to 2D touch locations which makes it challenging to provide shortcuts and secondary actions similar to hardware keyboards and mice. Previous work presented a wide range of ...
Finger identification and hand gesture recognition techniques for natural user interface
APCHI '13: Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction

The natural user interface using hand gesture have been popular field in Human-Computer-Interaction(HCI). Many research papers have been proposed in this field. They proposed vision-based, glove-based and depth-based approach for hand gesture ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

October 2020

1297 pages

ISBN:9781450375146

DOI:10.1145/3379337

General Chairs:
Shamsi Iqbal
Microsoft Research, USA
,
Karon MacLean
University of British Columbia, Canada
,
Program Chairs:
Fanny Chevalier
University of Toronto, Canada
,
Stefanie Mueller
MIT CSAIL, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of Science and ICT

Conference

UIST '20

Sponsor:

UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology

October 20 - 23, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
827
Total Downloads

Downloads (Last 12 months)64
Downloads (Last 6 weeks)6

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Herbert OPérez-Granados DRuiz MCadena Martínez RGutiérrez CAntuñano M(2024)Static and Dynamic Hand Gestures: A Review of Techniques of Virtual Reality ManipulationSensors10.3390/s2412376024:12(3760)Online publication date: 9-Jun-2024
https://doi.org/10.3390/s24123760
Lee LLin Z(2024)Danger, Nuisance, Disregard: Analyzing User-Generated Videos for Augmented Reality Gameplay on Hand-held DevicesProceedings of the ACM on Human-Computer Interaction10.1145/36770638:CHI PLAY(1-33)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677063
Huang ZGao CWang HDeng XLai YMa CQin SLiu YWang H(2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643559
Viollet SMartin CJean-Marc I(2023)LinLED: Low latency and accurate contactless gesture interactionCompanion Publication of the 25th International Conference on Multimodal Interaction10.1145/3610661.3617164(61-65)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3610661.3617164
Kato KIkematsu K(2023)Acoustic+Pose: Adding Input Modality to Smartphones with Near-Surface Hand-Pose Recognition using Acoustic SurfaceProceedings of the 2023 International Symposium on Wearable Computers10.1145/3594738.3611362(44-49)Online publication date: 8-Oct-2023
https://doi.org/10.1145/3594738.3611362
Yeo HWu EKim DLee JKim HOh STakagi LWoo WKoike HQuigley A(2023)OmniSense: Exploring Novel Input Sensing and Interaction Techniques on Mobile Device with an Omni-Directional CameraProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580747(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580747
Liu YZhang SGowda M(2023)A Practical System for 3-D Hand Pose Tracking Using EMG Wearables With Applications to Prosthetics and User InterfacesIEEE Internet of Things Journal10.1109/JIOT.2022.322360010:4(3407-3427)Online publication date: 15-Feb-2023
https://doi.org/10.1109/JIOT.2022.3223600
Dobinson RTeyssier MSteimle JFruchard B(2022)MicroPress: Detecting Pressure and Hover Distance in Thumb-to-Finger InteractionsProceedings of the 2022 ACM Symposium on Spatial User Interaction10.1145/3565970.3567698(1-10)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1145/3565970.3567698
Shen VHarrison C(2022)Pull Gestures with Coordinated Graphics on Dual-Screen DevicesProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556620(270-277)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556620
Zhou HLu TLiu YZhang SGowda M(2022)Learning on the RingsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35345876:2(1-31)Online publication date: 7-Jul-2022
https://dl.acm.org/doi/10.1145/3534587
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten