Yongsen Ma
Gang Zhou
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We propose SignFi to recognize sign language gestures using WiFi. SignFi uses Channel State Information (CSI) measured by WiFi packets as the input and a Convolutional Neural Network (CNN) as the classification algorithm. Existing WiFi-based sign gesture recognition technologies are tested on no more than 25 gestures that only involve hand and/or finger gestures. SignFi is able to recognize 276 sign gestures, which involve the head, arm, hand, and finger gestures, with high accuracy. SignFi collects CSI measurements to capture wireless signal characteristics of sign gestures. Raw CSI measurements are pre-processed to remove noises and recover CSI changes over sub-carriers and sampling time. Pre-processed CSI measurements are fed to a 9-layer CNN for sign gesture classification. We collect CSI traces and evaluate SignFi in the lab and home environments. There are 8,280 gesture instances, 5,520 from the lab and 2,760 from the home, for 276 sign gestures in total. For 5-fold cross validation using CSI traces of one user, the average recognition accuracy of SignFi is 98.01%, 98.91%, and 94.81% for the lab, home, and lab+home environment, respectively. We also run tests using CSI traces from 5 different users in the lab environment. The average recognition accuracy of SignFi is 86.66% for 7,500 instances of 150 sign gestures performed by 5 different users.
Supplemental Material
Available for Download
zip
ma.zip (1.3 MB)
Supplemental movie, appendix, image and software files for, SignFi: Sign Language Recognition Using WiFi
- H. Abdelnasser, M. Youssef, and K. A. Harras. 2015. WiGest: A ubiquitous WiFi-based gesture recognition system. In 2015 IEEE Conference on Computer Communications (INFOCOM). 1472--1480.Google Scholar
- Naomi K. Caselli, Zed Sevcikova Sehyr, Ariel M. Cohen-Goldberg, and Karen Emmorey. 2017. ASL-LEX: A lexical database of American Sign Language. Behavior Research Methods 49, 2 (April 2017), 784--801.Google ScholarCross Ref
- Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). 255--260. Google ScholarDigital Library
- Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D Input Using Two Magnetic Sensors. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). 237--244. Google ScholarDigital Library
- Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). 1504--1514. Google ScholarDigital Library
- Ching-Hua Chuan, Eric Regina, and Caroline Guardino. 2014. American Sign Language Recognition Using Leap Motion Sensor. In Proceedings of the 2014 13th International Conference on Machine Learning and Applications (ICMLA '14). 541--544. Google ScholarDigital Library
- Biyi Fang, Jillian Co, and Mi Zhang. 2017. DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation. In Proceedings of the 15th ACM Conference on Embedded Networked Sensor Systems (SenSys '17). Google ScholarDigital Library
- Makiko Funasaka, Yu Ishikawa, Masami Takata, and Kazuki Joe. 2015. Sign Language Recognition using Leap Motion Controller. In Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA).Google Scholar
- David Goldberg, Dennis Looney, and Natalia Lusin. 2015. Enrollments in Languages Other Than English in United States Institutions of Higher Education, Fall 2013. In Modern Language Association.Google Scholar
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org. Google ScholarDigital Library
- Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). 1911--1914. Google ScholarDigital Library
- Daniel Halperin, Wenjun Hu, Anmol Sheth, and David Wetherall. 2011. Tool Release: Gathering 802.11n Traces with Channel State Information. SIGCOMM Comput. Commun. Rev. 41, 1 (Jan. 2011), 53--53. Google ScholarDigital Library
- Jie Huang, Wengang Zhou, Houqiang Li, and Weiping Li. 2015. Sign Language Recognition using 3D Convolutional Neural Networks. In 2015 IEEE International Conference on Multimedia andExpo (ICME). 1--6.Google ScholarCross Ref
- Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 (ICML'15). JMLR.org, 448--456. http://dl.acm.org/citation.cfm?id=3045118.3045167 Google ScholarDigital Library
- Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing Gesture Recognition to All Devices. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI'14). 303--316. http://dl.acm.org/citation.cfm?id=2616448.2616477 Google ScholarDigital Library
- Swarun Kumar, Diego Cifuentes, Shyamnath Gollakota, and Dina Katabi. 2013. Bringing Cross-layer MIMO to Today's Wireless LANs. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM (SIGCOMM '13). 387--398. Google ScholarDigital Library
- H.L. Lane, R. Hoffmeister, and B.J. Bahan. 1996. A Journey Into the Deaf-world. DawnSignPress.Google Scholar
- Hong Li, Wei Yang, Jianxin Wang, Yang Xu, and Liusheng Huang. 2016. WiFinger: Talk to Your Smart Devices with Finger-grained Gesture. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). 250--261. Google ScholarDigital Library
- Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. 35, 4, Article 142 (July 2016), 19 pages. Google ScholarDigital Library
- Rajesh B. Mapari and Govind Kharat. 2016. American Static Signs Recognition Using Leap Motion Sensor. In Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies (ICTCS '16). Article 67, 5 pages. Google ScholarDigital Library
- Pedro Melgarejo, Xinyu Zhang, Parameswaran Ramanathan, and David Chu. 2014. Leveraging Directional Antenna Capabilities for Fine-grained Gesture Recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). 541--551. Google ScholarDigital Library
- P. Molchanov, S. Gupta, K. Kim, and K. Pulli. 2015. Multi-sensor system for driver's hand-gesture recognition. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 1. 1--8.Google Scholar
- MotionSavvy. 2017. MotionSavvy UNI. (2017). Retrieved July 23, 2017 from http://www.motionsavvy.com/uni.html.Google Scholar
- D. Naglot and M. Kulkarni. 2016. Real time sign language recognition using the leap motion controller. In 2016 International Conference on Inventive Computation Technologies (ICICT), Vol. 3. 1--5.Google Scholar
- Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). 1515--1525. Google ScholarDigital Library
- University of Washington. 2016. SignAloud Demo. (2016). Retrieved July 23, 2017 from https://youtu.be/4uY-MyoRq4c.Google Scholar
- Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. E-Gesture: A Collaborative Architecture for Energy-efficient Gesture Recognition with Hand-worn Sensor and Mobile Devices. In Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems (SenSys '11). 260--273. Google ScholarDigital Library
- Lionel Pigou, Sander Dieleman, Pieter-Jan Kindermans, and Benjamin Schrauwen. 2015. Sign Language Recognition Using Convolutional Neural Networks. Springer International Publishing, Cham, 572--578.Google Scholar
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home Gesture Recognition Using Wireless Signals. In Proceedings of the 19th Annual International Conference on Mobile Computing and Networking (MobiCom '13). 27--38. Google ScholarDigital Library
- Luis Quesada, Gustavo López, and Luis A. Guerrero. 2015. Sign Language Recognition Using Leap Motion. Springer International Publishing, Cham, 277--288.Google Scholar
- Wenjie Ruan, Quan Z. Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: Enabling Fine-grained Hand Gesture Detection by Decoding Echo Signal. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). 474--485. Google ScholarDigital Library
- C. Savur and F. Sahin. 2015. Real-Time American Sign Language Recognition System Using Surface EMG Signal. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). 497--502.Google Scholar
- Jiacheng Shang and Jie Wu. 2017. A Robust Sign Language Recognition System with Multiple Wi-Fi Devices. In Proceedings of the Workshop on Mobility in the Evolving Internet Architecture (MobiArch '17). 19--24. Google ScholarDigital Library
- SignAll. 2017. SignAll Prototype. (2017). Retrieved July 23, 2017 from http://www.signall.us.Google Scholar
- StartASL. 2017. Basic Words in Sign Language. (2017). Retrieved July 14, 2017 from https://www.startasl.com/basic-words-in-sign-language.html.Google Scholar
- Chao Sun, Tianzhu Zhang, and Changsheng Xu. 2015. Latent Support Vector Machine Modeling for Sign Language Recognition with Kinect. ACM Trans. Intell. Syst. Technol. 6, 2, Article 20 (March 2015), 20 pages. Google ScholarDigital Library
- Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15). 77--89. Google ScholarDigital Library
- Sheng Tan and Jie Yang. 2016. WiFinger: Leveraging Commodity WiFi for Fine-grained Finger Gesture Recognition. In Proceedings of the 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc '16). 201--210. Google ScholarDigital Library
- David Tse and Pramod Viswanath. 2005. Fundamentals of Wireless Communication. Cambridge University Press. Google ScholarDigital Library
- Aditya Virmani and Muhammad Shahzad. 2017. Position and Orientation Agnostic Gesture Recognition Using WiFi. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '17). 252--264. Google ScholarDigital Library
- Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: Virtual Touch Screen in the Air Using RF Signals. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM '14). 235--246. Google ScholarDigital Library
- Wei Wang, Alex X. Liu, and Ke Sun. 2016. Device-free Gesture Tracking Using Acoustic Signals. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking (MobiCom '16). 82--94. Google ScholarDigital Library
- Hikaru Watanabe, Masahiro Mochizuki, Kazuya Murao, and Nobuhiko Nishio. 2016. A Recognition Method for Continuous Gestures with an Accelerometer. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). 813--822. Google ScholarDigital Library
- Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition Using an Off-the-Shelf Smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). 3847--3851. Google ScholarDigital Library
- Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2015. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition Using Smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications (HotMobile '15). 9--14. Google ScholarDigital Library
- Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-Grained Acoustic-based Device-Free Tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '17). 15--28. Google ScholarDigital Library
- Zahoor Zafrulla, Helene Brashear, Thad Starner, Harley Hamilton, and Peter Presti. 2011. American Sign Language Recognition with the Kinect. In Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI '11). 279--286. Google ScholarDigital Library
- Ouyang Zhang and Kannan Srinivasan. 2016. Mudra: User-friendly Fine-grained Gesture Recognition Using WiFi Signals. In Proceedings of the 12th International on Conference on Emerging Networking Experiments and Technologies (CoNEXT '16). 83--96. Google ScholarDigital Library
- Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S. Reynolds. 2014. SideSwipe: Detecting In-air Gestures Around Mobile Devices Using Actual GSM Signal. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). 527--534. Google ScholarDigital Library
Index Terms
- SignFi: Sign Language Recognition Using WiFi
Recommendations
Recent methods and databases in vision-based hand gesture recognition
The paper surveys RGB and RGB-D sensors based hand gesture recognition methods.Dynamic as well as static gesture (posture/pose) recognition methods are reviewed.Qualitative as well as quantitative comparison of algorithms is provided.Twenty-six publicly ...
Isolated sign language recognition using Convolutional Neural Network hand modelling and Hand Energy Image
This paper presents an isolated sign language recognition system that comprises of two main phases: hand tracking and hand representation. In the hand tracking phase, an annotated hand dataset is used to extract the hand patches to pre-train ...
Signing Exact English (SEE): Modeling and recognition
We present effective and robust algorithms to recognize isolated signs in Signing Exact English (SEE). The sign-level recognition scheme comprises classifiers for handshape, hand movement and hand location. The SEE gesture data are acquired using ...
Comments