Skip to main content
SpringerLink
Log in

Deep transfer learning for gesture recognition with WiFi signals

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Different gestures have different action speeds, directions, and trajectories that can cause distinctive effects on the propagation of WiFi signals. In this paper, we present a new approach that uses deep transfer learning techniques to recognize gestures based on the channel state information (CSI) extracted from WiFi signals. Firstly, the CSI streams of gestures are captured and the gesture segments are extracted based on the CSI amplitude changes, and then the WiFi-based gesture recognition problem is innovatively converted to an image classification problem by expressing CSI streams as an image matrix. After that, two deep transfer learning methods are applied to recognize gestures using high-level features extracted by deep convolutional neural network (CNN) and fine-tuned CNN models. We evaluated our method using a collected dataset with 12 gestures in two environments, and the experimental results demonstrated that the proposed method outperformed other state-of-the-art WiFi-based gesture recognition methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Ma J, Hao W, Zhang D, Wang Y, Wang Y (2017) A survey on wi-fi based contactless activity recognition. In: Ubiquitous intelligence & computing, advanced & trusted computing, scalable computing & communications, Cloud & Big Data Computing, Internet of People, & Smart World Congress

  2. Chang W, Wu C, Tsai RY, Lin KC, Tseng Y (2018) Eye on you fusing gesture data from depth camera and inertial sensors for person identification. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp 2021–2026

  3. Chiu P, Kim C, Oda H (2018) Recognizing gestures on projected button widgets with an RGB-D camera using a CNN. In: Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, ISS ’18. ACM, New York, pp 369–374

  4. Zhang Q, Yang M, Zheng Q, Zhang X (2017) Segmentation of hand gesture based on dark channel prior in projector-camera system. In: 2017 IEEE/CIC International Conference on Communications in China (ICCC), pp 1–6

  5. Li F, Fei J (2019) Gesture recognition algorithm based on image information fusion in virtual reality. Personal and Ubiquitous Computing

  6. Wang S, Jie S, Lien J, Poupyrev I, Hilliges O (2016) Interacting with soli: exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In: Symposium on User Interface Software & Technology

  7. Kriara L, Alsup M, Corbellini G, Trotter M, Griffin JD, Mangold S (2013) RFID shakables: pairing radio-frequency identification tags with the help of gesture recognition. In: Acm Conference on Emerging Networking Experiments & Technologies

  8. Zou H, Zhou Y, Yang J, Jiang H, Xie L, Spanos CJ (2018) WiFi-enabled device-free gesture recognition for smart home automation. In: 14Th IEEE International Conference on Control and Automation, ICCA 2018, anchorage, AK, USA, June 12-15, 2018, pp 476–481

  9. Shahzad M, Zhang S (2018) Augmenting user identification with WiFi based gesture recognition. IMWUT 2(3):134:1–134:27

    Google Scholar 

  10. Chen X, Chen L, Chao F, Fang D, Xiong J, Wang Z, Cui Y (2019) Sensing our world using wireless signals. IEEE Internet Comput 23:38–45

    Article  Google Scholar 

  11. Halperin D, Hu W, Sheth A, Wetherall D (2011) Tool release: gathering 802.11n traces with channel state information. Acm Sigcomm Computer Commun Rev 41(1):53–53

    Article  Google Scholar 

  12. Wang Y, Wu K, Ni LM (2017) Wifall: device-free fall detection by wireless networks. IEEE Trans Mob Comput 16(2):581–594

    Article  Google Scholar 

  13. Wang Y, Liu J, Chen Y, Gruteser M, Yang J, Liu H (2014) E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In: Proceedings of the 20th Annual International Conference on Mobile Computing and Networking, MobiCom ’14. ACM, New York, pp 617–628

  14. Wang W, Liu AX, Shahzad M, Ling K, Lu S (2015) Understanding and modeling of wifi signal based human activity recognition. In: Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, MobiCom ’15. ACM, New York, pp 65–76

  15. Virmani A, Shahzad M (2017) Position and orientation agnostic gesture recognition using wifi. In: Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys ’17. ACM, New York, pp 252–264

  16. Zhu W, Guo B, Yu Z, Zhou X (2018) Wi-Fi CSI based behavior recognition: from signals, actions to activities. IEEE Commun Mag 56(5):109–115

    Article  Google Scholar 

  17. Zhang D, Hao W, Dan W (2017) Toward centimeter-scale human activity sensing with Wi-Fi signals. Computer 50(1):48–57

    Article  Google Scholar 

  18. Wang Q, Yin X, Tan J, Xing T, Niu J, Fang D (2019) Dtransfer: extremely low cost localization irrelevant to targets and regions for activity recognition. Pers Ubiquit Comput 23(1):3–16

    Article  Google Scholar 

  19. Chang L, Chen X, Yu W, Fang D, Ju W, Xing T, Tang Z (2017) FitLoc: Fine-grained and low-cost device-free localization for multiple targets over various areas. IEEE/ACM Trans Netw PP(99):1–14

    Google Scholar 

  20. Tong X, Guo B, Zhu W, Li M, Yu Z (2016) Freesense:indoor human identification with WiFi signals. In: IEEE Global Communications Conference

  21. Arshad S, Feng C, Liu Y, Hu Y, Yu R, Zhou S, Li H (2017) Wi-chase: a WiFi based human activity recognition system for sensorless environments. In: 2017 IEEE 18Th International Symposium on a World of Wireless, Mobile and Multimedia Networks (woWMom), pp 1–6

  22. Chang J-Y, Lee K-Y, Wei Y-L, Lin KC-J, Hsu W (2016) Location-independent wifi action recognition via vision-based methods. In: Proceedings of the 24th ACM International Conference on Multimedia, MM ’16. ACM, New York, pp 162–166

  23. Xiao Y (2005) Ieee 802.11n: Enhancements for higher throughput in wireless lans. IEEE Wirel Commun 12 (6):82–91

    Article  Google Scholar 

  24. Zhang D, Wang H, Wang Y, Ma J (2015) Anti-fall: a non-intrusive and real-time fall detector leveraging CSI from commodity WiFi devices. In: Geissbühler A, Demongeot J, Mokhtari M, Abdulrazak B, Aloulou H (eds) Inclusive Smart Cities and e-Health. Springer International Publishing, Cham, pp 181–193

  25. Wang J, Xiong J, Jiang H, Jamieson K, Chen X, Fang D, Wang C (2018) Low human-effort, device-free localization with fine-grained subcarrier information. IEEE Trans Mob Comput 17(11):2550–2563

    Article  Google Scholar 

  26. Wang H, Zhang D, Ma J, Wang Y, Wang Y, Wu D, Gu T, Xie B (2016) Human respiration detection with commodity wifi devices: do user location and body orientation matter?. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’16. ACM, New York, pp 25–36

  27. Abdelnasser H, Youssef M, Harras KA (2015) Wigest: A ubiquitous wifi-based gesture recognition system. In: 2015 IEEE Conference on Computer Communications (INFOCOM), pp 1472–1480

  28. Pu Q, Gupta S, Gollakota S, Patel S (2013) Whole-home gesture recognition using wireless signals. In: Proceedings of thes 19th Annual International Conference on Mobile Computing & Networking, MobiCom ’13. ACM, New York, pp 27–38

  29. Adib F, Kabelac Z, Katabi D, Miller RC (2014) 3d tracking via body radio reflections. In: Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation, NSDI’14. USENIX Association, Berkeley, pp 317–329

  30. Zhang Z, Tian Z, Zhou M (2018) Latern: Dynamic continuous hand gesture recognition using FMCW radar sensor. IEEE Sensors J 18(8):3278–3289

    Article  Google Scholar 

  31. He W, Wu K, Zou Y, Ming Z (2015) Wig: Wifi-based gesture recognition system. In: 2015 24Th International Conference on Computer Communication and Networks (ICCCN), pp 1–7

  32. Al-qaness MAA, Li F (2016) Wiger: Wifi-based gesture recognition system. ISPRS International Journal of Geo-Information, 5(6):92

    Article  Google Scholar 

  33. Bu Q, Yang G, Feng J, Ming X (2018) Wi-fi based gesture recognition using deep transfer learning. In: 2018 IEEE Smartworld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation, Smartworld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI 2018, Guangzhou, China, october 8-12, 2018, pp 590–595

  34. Zeiler MD, Fergus R (2013) Visualizing and understanding convolutional networks CoRR, arXiv:abs/1311.2901

  35. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift CoRR, arXiv:abs/1502.03167

  36. Cho K (2013) Understanding dropout: training multi-layer perceptrons with auxiliary independent stochastic neurons CoRR, arXiv:abs/1306.2801

Download references

Funding

This work was supported by the National Key Research & Development Program of China (2017YFB1002504), the Shaanxi International Science and Technology Cooperation and Exchange Program (2017KW-010), and the ShaanXi Science and Technology Innovation Team Support Project under grant agreement (2018TD-026).

Author information

Authors and Affiliations

  1. Department of Computer Science, Northwest University, Xi’an, China

    Qirong Bu, Gang Yang, Xingxia Ming, Tuo Zhang & Jun Feng

  2. Shaanxi International Joint Research Centre for the Battery-free Internet of things, Xi’an, China

    Qirong Bu

  3. Computer Science Department, Lamar University, Beaumont, TX, 77710, USA

    Jing Zhang

Authors
  1. Qirong Bu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xingxia Ming
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tuo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jun Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qirong Bu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bu, Q., Yang, G., Ming, X. et al. Deep transfer learning for gesture recognition with WiFi signals. Pers Ubiquit Comput 26, 543–554 (2022). https://doi.org/10.1007/s00779-019-01360-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-019-01360-8

Keywords

Search

Navigation