Xiao Zhang
ABSTRACT
Smart homes, medical devices, and education systems, among other emerging cyber-physical systems, hold immense promise for sensing-based user interfaces, especially for using fingers and hand gestures as system input. However, vision approaches compatible with time-consuming image processing adopt low 60 Hz location sampling rate (frame rate) for real-time hand gesture recognition. Furthermore, they are not suitable for low-light environment and long detection range. In this paper, we propose RoFin, which first exploits 6 temporal-spatial 2D rolling fingertips for real-time 3D reconstructing of 20-joint hand pose. RoFin designs active optical labeling for finger identification and enhances inside-frame 3D location tracking via high rolling shutter rate (5--8 KHz). These features enable great potentials for enhanced multi-user HCI, virtual writing for Parkinson suffers, etc. We implement RoFin prototypes with wearable gloves attached with low-power single-colored LED nodes and commercial cameras. The experiment results show that (1) In flexible sensing distances up to 2.5 m, RoFin achieves an average labeling parsing accuracy of 85%. (2) In comparison to vision-based techniques, RoFin improves the tracking grain with 4× more sampled points each frame. (3) RoFin reconstructs a hand pose in real time with 16 mm mean deviation error compared with Leap Motion under flexible distance.
- 2019. IEEE Standard for Local and metropolitan area networks-Part 15.7: Short-Range Optical Wireless Communications. IEEE Std 802.15.7-2018 (Revision of IEEE Std 802.15.7-2011) (April 2019), 1--407.Google Scholar
- 2022. Micrographia. https://en.wikipedia.org/wiki/Micrographia_%28handwriting%29Google Scholar
- 2023. Leap Motion Tutorial. https://developer-archive.leapmotion.com/documentation/cpp/devguide/Sample_Tutorial.htmlGoogle Scholar
- 2023. XSens. https://www.movella.com/Google Scholar
- Andreas Aristidou, Joan Lasenby, Yiorgos Chrysanthou, and Ariel Shamir. 2018. Inverse kinematics techniques in computer graphics: A survey. In Computer graphics forum, Vol. 37. Wiley Online Library, 35--58.Google Scholar
- Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. 2018. Weakly-supervised 3d hand pose estimation from monocular rgb images. In Proceedings of the European Conference on Computer Vision (ECCV). 666--682.Google ScholarDigital Library
- Weiya Chen, Chenchen Yu, Chenyu Tu, Zehua Lyu, Jing Tang, Shiqi Ou, Yan Fu, and Zhidong Xue. 2020. A survey on hand pose estimation with wearable sensors and computer-vision-based methods. Sensors 20, 4 (2020), 1074.Google ScholarCross Ref
- Minhao Cui, Qing Wang, and Jie Xiong. 2020. Breaking the limitations of visible light communication through its side channel. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 232--244.Google ScholarDigital Library
- Ander Galisteo, Qing Wang, Aniruddha Deshpande, Marco Zuniga, and Domenico Giustiniano. 2017. Follow that light: Leveraging leds for relative two-dimensional localization. In Proceedings of the 13th International Conference on emerging Networking EXperiments and Technologies. 187--198.Google ScholarDigital Library
- Jun Gong, Yang Zhang, Xia Zhou, and Xing-Dong Yang. 2017. Pyro: Thumb-tip gesture recognition using pyroelectric infrared sensing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 553--563.Google ScholarDigital Library
- Jun Sik Kim, Byung Kook Kim, Minsu Jang, Kyumin Kang, Dae Eun Kim, Byeong-Kwon Ju, and Jinseok Kim. 2020. Wearable hand module and real-time tracking algorithms for measuring finger joint angles of different hand sizes with high accuracy using FBG strain sensor. Sensors 20, 7 (2020), 1921.Google ScholarCross Ref
- Ye-Sheng Kuo, Pat Pannuto, Ko-Jen Hsiao, and Prabal Dutta. 2014. Luxapose: Indoor positioning with mobile phones and visible light. In Proceedings of the 20th annual international conference on Mobile computing and networking. 447--458.Google ScholarDigital Library
- Yongjun Lee, Myungsin Kim, Yongseok Lee, Junghan Kwon, Yong-Lae Park, and Dongjun Lee. 2018. Wearable finger tracking and cutaneous haptic interface with soft sensors for multi-fingered virtual manipulation. IEEE/ASME Transactions on Mechatronics 24, 1 (2018), 67--77.Google ScholarCross Ref
- Rui Li, Zhenyu Liu, and Jianrong Tan. 2019. A survey on 3D hand pose estimation: Cameras, methods, and datasets. Pattern Recognition 93 (2019), 251--272.Google ScholarDigital Library
- Tianxing Li, Xi Xiong, Yifei Xie, George Hito, Xing-Dong Yang, and Xia Zhou. 2017. Reconstructing hand poses using visible light. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--20.Google ScholarDigital Library
- Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--19.Google ScholarDigital Library
- Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1515--1525.Google ScholarDigital Library
- Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera. 2021. Sign language recognition: A deep survey. Expert Systems with Applications 164 (2021), 113794.Google ScholarCross Ref
- Sanjib Sur, Ioannis Pefkianakis, Xinyu Zhang, and Kyu-Han Kim. 2018. Towards scalable and ubiquitous millimeter-wave wireless networks. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 257--271.Google ScholarDigital Library
- Andrea Tagliasacchi, Matthias Schröder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly. 2015. Robust articulated-icp for real-time hand tracking. In Computer graphics forum, Vol. 34. Wiley Online Library, 101--114.Google Scholar
- Mary Ann Thenganatt and Elan D Louis. 2012. Distinguishing essential tremor from Parkinson's disease: bedside tests and laboratory evaluations. Expert review of neurotherapeutics 12, 6 (2012), 687--696.Google Scholar
- Robert Wang, Sylvain Paris, and Jovan Popović. 2011. 6D hands: markerless hand-tracking for computer aided design. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 549--558.Google ScholarDigital Library
- Yue Wu, Purui Wang, Kenuo Xu, Lilei Feng, and Chenren Xu. 2020. Turboboosting visible light backscatter communication. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 186--197.Google ScholarDigital Library
- Huichuan Xu, Daisuke Iwai, Shinsaku Hiura, and Kosuke Sato. 2006. User interface by virtual shadow projection. In 2006 SICE-ICASE International Joint Conference. IEEE, 4814--4817.Google ScholarCross Ref
- Y. Yang, J. Hao, and J. Luo. 2017. CeilingTalk: Lightweight Indoor Broadcast Through LED-Camera Communication. IEEE Transactions on Mobile Computing 16, 12 (2017), 3308--3319.Google ScholarCross Ref
- Yanbing Yang and Jun Luo. 2018. Boosting the throughput of LED-camera VLC via composite light emission. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 315--323.Google ScholarDigital Library
- Yanbing Yang and Jun Luo. 2020. Composite Amplitude-Shift Keying for Effective LED-Camera VLC. IEEE Transactions on Mobile Computing 19, 03 (2020), 528--539.Google ScholarCross Ref
- Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. 2020. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214 (2020).Google Scholar
- Xiao Zhang, Hanqing Guo, James Mariani, and Li Xiao. 2022. U-Star: An Underwater Navigation System based on Passive 3D Optical Identification Tags. In The 28th Annual International Conference on Mobile Computing and Networking.Google ScholarDigital Library
- Xiao Zhang, Griffin Klevering, Xinyu Lei, Yiwen Hu, Li Xiao, and Tu Guanhua. 2023. The Security in Optical Wireless Communication: A Survey. ACM Computing Surveys (CSUR) (2023).Google ScholarDigital Library
- Xiao Zhang, Griffin Klevering, Kanishka Wijewardena, and Li Xiao. 2023. Demo: Integrated On-site Localization and Optical Camera Communication for Drones. In 2023 IEEE 24th International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 1--3.Google Scholar
- Xiao Zhang, Griffin Klevering, and Li Xiao. 2022. Exploring Rolling Shutter Effect for Motion Tracking with Objective Identification. In Proceedings of the Twentieth ACM Conference on Embedded Networked Sensor Systems. 816--817.Google ScholarDigital Library
- Xiao Zhang, Griffin Klevering, and Li Xiao. 2023. PoseFly: On-site Pose Parsing of Swarming Drones via 4-in-1 Optical Camera Communication. In 2023 IEEE 24th International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 1--10.Google Scholar
- Xiao Zhang, James Mariani, Li Xiao, and Matt W Mutka. 2022. LiFOD: Lighting Extra Data via Fine-grained OWC Dimming. In 2022 19th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 73--81.Google ScholarDigital Library
- Xiao Zhang and Li Xiao. 2020. Lighting Extra Data via OWC Dimming. In Proceedings of the Student Workshop. 29--30.Google ScholarDigital Library
- Xiao Zhang and Li Xiao. 2020. RainbowRow: Fast Optical Camera Communication. In 2020 IEEE 28th International Conference on Network Protocols (ICNP). IEEE, 1--6.Google Scholar
- Run Zhao, Dong Wang, Qian Zhang, Xueyi Jin, and Ke Liu. 2021. Smartphone-based Handwritten Signature Verification using Acoustic Signals. Proceedings of the ACM on Human-Computer Interaction 5, ISS (2021), 1--26.Google ScholarDigital Library
- Shilin Zhu, Chi Zhang, and Xinyu Zhang. 2017. LiShield: Create a Capture-Resistant Environment Against Photographing. In Proceedings of the 9th ACM Workshop on Wireless of the Students, by the Students, and for the Students. 23--23.Google ScholarDigital Library
Index Terms
- RoFin: 3D Hand Pose Reconstructing via 2D Rolling Fingertips
Recommendations
Designing and Evaluating Hand-to-Hand Gestures with Dual Commodity Wrist-Worn Devices
Hand gestures provide a natural and easy-to-use way to input commands. However, few works have studied the design space of bimanual hand gestures or attempted to infer gestures that involve devices on both hands. We explore the design space of hand-to-...
JINS MEME algorithm for estimation and tracking of concentration of users
UbiComp '17: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable ComputersActivity tracking using a wearable device is an emerging research field. Large-scale studies on activity tracking performed with eyewear-type wearable devices remains a challenging area owing to the negative effect such devices have on users' looks. To ...
A method to recognize eye movements based on uplift movement of skin
UbiComp/ISWC '19 Adjunct: Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable ComputersMovements of human eyes provide useful insights for understanding human's physical and mental condition and ability, which has high attention in various industrial and academic fields. Although many methods for sensing eye movements have been proposed, ...
Comments