skip to main content
research-article
Public Access

Reconstructing Hand Poses Using Visible Light

Published: 11 September 2017 Publication History

Abstract

Free-hand gestural input is essential for emerging user interactions. We present Aili, a table lamp reconstructing a 3D hand skeleton in real time, requiring neither cameras nor on-body sensing devices. Aili consists of an LED panel in a lampshade and a few low-cost photodiodes embedded in the lamp base. To reconstruct a hand skeleton, Aili combines 2D binary blockage maps from vantage points of different photodiodes, which describe whether a hand blocks light rays from individual LEDs to all photodiodes. Empowering a table lamp with sensing capability, Aili can be seamlessly integrated into the existing environment. Relying on such low-level cues, Aili entails lightweight computation and is inherently privacy-preserving. We build and evaluate an Aili prototype. Results show that Aili’s algorithm reconstructs a hand pose within 7.2 ms on average, with 10.2° mean angular deviation and 2.5-mm mean translation deviation in comparison to Leap Motion. We also conduct user studies to examine the privacy issues of Leap Motion and solicit feedback on Aili’s privacy protection. We conclude by demonstrating various interaction applications Aili enables.

Supplementary Material

li-1 (li-1.zip)
Supplemental movie, appendix, image and software files for, Reconstructing Hand Poses Using Visible Light

References

[1]
All of the Smart TV Gestures, Samsung Smart TV. http://www.samsung.com/ph/smarttv/common/guide_book_3p_si/main.html. (????).
[2]
Nest Protect smoke + CO alarm. https://store.nest.com/product/smoke-co-alarm. (????).
[3]
Nova Lighting 1010604. http://www.totallyfurniture.com/nova-1010604.html. (????).
[4]
2014. Hand Physics Controller. https://www.assetstore.unity3d.com/en/#!/content/21105. (2014).
[5]
2015. Google Soli Project. https://www.google.com/atap/project-soli/. (2015).
[6]
2016. Leap Motion Visualizer. https://www.leapmotion.com/setup. (2016).
[7]
2016a. LIGHTING DESIGN. http://www.bristolite.com/interfaces/media/Footcandle%20Recommendations% 20by%20Guth.pdf. (2016).
[8]
2016b. Recommended Light Levels. https://www.noao.edu/education/QLTkit/ACTIVITY_Documents/Safety/LightLevels_outdoor+indoor.pdf. (2016).
[9]
Vassilis Athitsos and Stan Sclaroff. 2003. Estimating 3D hand pose from a cluttered image. In Proc. of CVPR.
[10]
Luca Ballan, Aparna Taneja, Jürgen Gall, Luc Van Gool, and Marc Pollefeys. 2012. Motion capture of hands in action using discriminative salient points. In Computer Vision--ECCV 2012. Springer, 640--653.
[11]
Jeff C Becker and Nithish V Thakor. 1988. A study of the range of motion of human fingers with application to anthropomorphic designs. Biomedical Engineering, IEEE Transactions on 35, 2 (1988), 110--117.
[12]
Paul Bratley, Bennett L Fox, and Harald Niederreiter. 1992. Implementation and tests of low-discrepancy sequences. ACM Transactions on Modeling and Computer Simulation (TOMACS) 2, 3 (1992), 195--213.
[13]
Matthieu Bray, Esther Koller-Meier, and Luc Van Gool. 2004. Smart particle filtering for 3D hand tracking. In Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE International Conference on. IEEE, 675--680.
[14]
Liwei Chan, Yi-Ling Chen, Chi-Hao Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2015. CyclopsRing: Enabling Whole-Hand and Context-Aware Interactions Through a Fisheye Ring. In Proc. of UIST.
[15]
Martin de La Gorce, David J Fleet, and Nikos Paragios. 2011. Model-based 3d hand pose estimation from monocular video. Pattern Analysis and Machine Intelligence, IEEE Transactions on 33, 9 (2011), 1793--1805.
[16]
Artem Dementyev and Joseph A Paradiso. 2014. WristFlex: Low-power gesture input with wrist-worn pressure sensors. In Proc. of UIST.
[17]
Jeremy Gummeson, Bodhi Priyantha, and Jie Liu. 2014. An Energy Harvesting Wearable Ring Platform for Gesture input on Surfaces. In Proc. of MobiSys.
[18]
Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. In Proc. of CHI.
[19]
Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, and Apu Kapadia. 2014. Privacy behaviors of lifeloggers using wearable cameras. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 571--582.
[20]
Texas Instruments. RM57L843 16/32-Bit RISC Flash Microcontroller. http://www.ti.com/product/RM57L843. (????).
[21]
Stephen Joe and Frances Y Kuo. 2003. Remark on algorithm 659: Implementing Sobol’s quasirandom sequence generator. ACM Transactions on Mathematical Software (TOMS) 29, 1 (2003), 49--57.
[22]
Stephen Joe and Frances Y Kuo. 2008. Constructing Sobol sequences with better two-dimensional projections. SIAM Journal on Scientific Computing 30, 5 (2008), 2635--2654.
[23]
Derek G Kamper, T George Hornby, and William Z Rymer. 2002. Extrinsic flexor muscles generate concurrent flexion of all three finger joints. Journal of biomechanics 35, 12 (2002), 1581--1589.
[24]
Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing Gesture Recognition to All Devices. In Proc. of NSDI.
[25]
Cem Keskin, Furkan Kıraç Yunus Emre Kara, and Lale Akarun. 2012. Hand pose estimation and hand shape classification using multi-layered randomized decision forests. In Computer Vision--ECCV 2012. Springer, 852--863.
[26]
David Kim, Otmar Hilliges, Shahram Izadi, Alex D. Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. In Proc. of UIST.
[27]
David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Timothy Large, Steven Bathiche, Matthias Nießner, and others. 2014. RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces. In Proc. of CHI.
[28]
Ye-Sheng Kuo, Pat Pannuto, Ko-Jen Hsiao, and Prabal Dutta. 2014. Luxapose: Indoor positioning with mobile phones and visible light. In Proc. of MobiCom.
[29]
Hong Li, Wei Yang, Jianxin Wang, Yang Xu, and Liusheng Huang. 2016b. WiFinger: talk to your smart devices with finger-grained gesture. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 250--261.
[30]
Liqun Li, Pan Hu, Chunyi Peng, Guobin Shen, and Feng Zhao. 2014. Epsilon: A Visible Light Based Positioning System. In Proc. of NSDI.
[31]
Tianxing Li, Chuankai An, Zhao Tian, Andrew T Campbell, and Xia Zhou. 2015. Human sensing using visible light communication. In Proc. of MobiCom.
[32]
Tianxing Li, Qiang Liu, and Xia Zhou. 2016a. Practical human sensing in the light. In Proc. of MobiSys.
[33]
Jess McIntosh, Charlie McNeill, Mike Fraser, Frederic Kerber, Markus Löchtefeld, and Antonio Krüger. 2016. EMPress: Practical Hand Gesture Classification with Wrist-Mounted EMG and Pressure Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2332--2342.
[34]
Jon Moeller and Andruid Kerne. 2012. ZeroTouch: An Optical Multi-touch and Free-air Interaction Architecture. In Proc. of CHI.
[35]
William J Morokoff and Russel E Caflisch. 1994. Quasi-random sequences and their discrepancies. SIAM Journal on Scientific Computing 15, 6 (1994), 1251--1279.
[36]
Harald Niederreiter. 1988. Low-discrepancy and low-dispersion sequences. Journal of number theory 30, 1 (1988), 51--70.
[37]
Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros. 2011. Efficient model-based 3D tracking of hand articulations using Kinect. In BmVC, Vol. 1. 3.
[38]
Iason Oikonomidis, Manolis IA Lourakis, and Antonis A Argyros. 2014. Evolutionary quasi-random search for hand articulations tracking. In Proc. of CVPR.
[39]
Santiago Ortega-Avila, Bogdana Rakova, Sajid Sadi, and Pranav Mistry. 2015. Non-invasive optical detection of hand gestures. In Proceedings of the 6th Augmented Human International Conference. ACM, 179--180.
[40]
William H Press, Saul A Teukolsky, William T Vetterling, and Brian P Flannery. 2007. Numerical recipes: the art of scientific computing. (2007).
[41]
Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. 2014. Realtime and robust hand tracking from depth. In Proc. of CVPR.
[42]
Jun Rekimoto. 2001. Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on. IEEE, 21--27.
[43]
Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 474--485.
[44]
Dragos Sbîrlea, Michael G Burke, Salvatore Guarnieri, Marco Pistoia, and Vivek Sarkar. 2013. Automatic detection of inter-application permission leaks in Android applications. IBM Journal of Research and Development 57, 6 (2013), 10--1.
[45]
Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, Daniel Freedman, Pushmeet Kohli, Eyal Krupka, Andrew Fitzgibbon, and Shahram Izadi. 2015. Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. of CHI.
[46]
Il’ya Meerovich Sobol’. 1967. On the distribution of points in a cube and the approximate evaluation of integrals. Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki 7, 4 (1967), 784--802.
[47]
Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. 2013. Interactive markerless articulated hand motion tracking using RGB and depth data. In Proc. of CVPR.
[48]
Bjoern Stenger, Paulo RS Mendonça, and Roberto Cipolla. 2001. Model-based 3D tracking of an articulated hand. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, Vol. 2. IEEE, II--310.
[49]
Paul Strohmeier, Roel Vertegaal, and Audrey Girouard. 2012. With a flick of the wrist: stretch sensors as lightweight input for mobile devices. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction. ACM, 307--308.
[50]
Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. Widraw: Enabling hands-free drawing in the air on commodity wifi devices. In Proc. of MobiCom.
[51]
Andrea Tagliasacchi, Matthias Schröder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly. 2015. Robust Articulated-ICP for Real-Time Hand Tracking. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 101--114.
[52]
Danhang Tang, Hyung Chang, Alykhan Tejani, and Tae-Kyun Kim. 2014. Latent regression forest: Structured estimation of 3d articulated hand posture. In Proc. of CVPR.
[53]
Danhang Tang, Tsz-Ho Yu, and Tae-Kyun Kim. 2013. Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In Proc. of CVPR.
[54]
Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, and others. 2016. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. ACM Transactions on Graphics (TOG) 35, 4 (2016), 143.
[55]
Robert Templeman, Zahid Rahman, David Crandall, and Apu Kapadia. 2012. PlaceRaider: Virtual theft in physical spaces with smartphones. arXiv preprint arXiv:1209.5982 (2012).
[56]
Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. 2014. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics (TOG) 33, 5 (2014), 169.
[57]
Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: Virtual Touch Screen in the Air Using RF Signals. In Proc. of SIGCOMM.
[58]
Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T Campbell. 2014. StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 3--14.
[59]
Robert Wang, Sylvain Paris, and Jovan Popović. 2011. 6D Hands: Markerless Hand-tracking for Computer Aided Design. In Proc. of UIST.
[60]
Saiwen Wang, Jie Song, Jamie Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. In Proc. of UIST.
[61]
Yangang Wang, Jianyuan Min, Jianjie Zhang, Yebin Liu, Feng Xu, Qionghai Dai, and Jinxiang Chai. 2013. Video-based hand manipulation capture through composite motion control. ACM Transactions on Graphics (TOG) 32, 4 (2013), 43.
[62]
Zachary Weinberg, Eric Y Chen, Pavithra Ramesh Jayaraman, and Collin Jackson. 2011. I still know what you visited last summer: Leaking browsing history via user interaction and side channel attacks. In Security and Privacy (SP), 2011 IEEE Symposium on. IEEE, 147--161.
[63]
Mark Weiser. 1999. The Computer for the 21st Century. SIGMOBILE Mob. Comput. Commun. Rev. 3, 3 (July 1999),3--11.
[64]
Ying Wu, John Y Lin, and Thomas S Huang. 2001. Capturing natural hand articulation. In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, Vol. 2. 426--432.
[65]
Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2015. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition Using Smartwatch. In Proc. of HotMobile.
[66]
Nan Xu, Fan Zhang, Yisha Luo, Weijia Jia, Dong Xuan, and Jin Teng. 2009. Stealthy video capturer: a new video-based spyware in 3g smartphones. In Proceedings of the second ACM conference on Wireless network security. ACM, 69--78.
[67]
Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David Harris-Birtill, and Aaron Quigley. 2016. RadarCat : Radar Categorization for input and interaction. In Proc. of UIST.
[68]
Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In Proc. of UIST.
[69]
Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S. Reynolds. 2014. SideSwipe: Detecting In-air Gestures Around Mobile Devices Using Actual GSM Signal. In Proc. of UIST.
[70]
Wenping Zhao, Jinxiang Chai, and Ying-Qing Xu. 2012. Combining marker-based mocap and RGB-D camera for acquiring high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation. Eurographics Association, 33--42.

Cited By

View all
  • (2024)Gesture Recognition Using Visible Light on Mobile DevicesIEEE/ACM Transactions on Networking10.1109/TNET.2024.336999632:4(2920-2935)Online publication date: Aug-2024
  • (2024)A Dynamic Gesture Recognition Algorithm Using Single Halide Perovskite Photovoltaic Cell for Human-Machine Interaction2024 International Conference on Electronics, Information, and Communication (ICEIC)10.1109/ICEIC61013.2024.10457182(1-4)Online publication date: 28-Jan-2024
  • (2024)Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple MaterialsIEEE Access10.1109/ACCESS.2024.336666712(27895-27917)Online publication date: 2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 1, Issue 3
September 2017
2023 pages
EISSN:2474-9567
DOI:10.1145/3139486
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 September 2017
Published in IMWUT Volume 1, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. 3D hand reconstruction
  2. Gestural input
  3. visible light sensing

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)158
  • Downloads (Last 6 weeks)19
Reflects downloads up to 03 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Gesture Recognition Using Visible Light on Mobile DevicesIEEE/ACM Transactions on Networking10.1109/TNET.2024.336999632:4(2920-2935)Online publication date: Aug-2024
  • (2024)A Dynamic Gesture Recognition Algorithm Using Single Halide Perovskite Photovoltaic Cell for Human-Machine Interaction2024 International Conference on Electronics, Information, and Communication (ICEIC)10.1109/ICEIC61013.2024.10457182(1-4)Online publication date: 28-Jan-2024
  • (2024)Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple MaterialsIEEE Access10.1109/ACCESS.2024.336666712(27895-27917)Online publication date: 2024
  • (2023)FORTE: Few Samples for Recognizing Hand Gestures with a Smartphone-attached RadarProceedings of the ACM on Human-Computer Interaction10.1145/35932317:EICS(1-25)Online publication date: 19-Jun-2023
  • (2023)RadarSense: Accurate Recognition of Mid-air Hand Gestures with Radar Sensing and Few Training ExamplesACM Transactions on Interactive Intelligent Systems10.1145/358964513:3(1-45)Online publication date: 11-Sep-2023
  • (2023)Recognizing Hand Gestures Using Solar CellsIEEE Transactions on Mobile Computing10.1109/TMC.2022.314814322:7(4223-4235)Online publication date: 1-Jul-2023
  • (2023)SemiGest: Recognizing Hand Gestures via Visible Light Sensing with Fewer Labels2023 19th International Conference on Mobility, Sensing and Networking (MSN)10.1109/MSN60784.2023.00123(837-842)Online publication date: 14-Dec-2023
  • (2023)Mask2Hand: Learning to Predict the 3D Hand Pose and Shape from Shadow2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)10.1109/APSIPAASC58517.2023.10317159(591-598)Online publication date: 31-Oct-2023
  • (2023)Integrated Sensing, Lighting and Communication Based on Visible Light Communication: A ReviewDigital Signal Processing10.1016/j.dsp.2023.104340(104340)Online publication date: Dec-2023
  • (2022)Hand Gesture Recognition for an Off-the-Shelf Radar by Electromagnetic Modeling and InversionProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511107(506-522)Online publication date: 22-Mar-2022
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media