Skip to main content

Advertisement

SpringerLink
Log in

Enabling non-invasive and real-time human-machine interactions based on wireless sensing and fog computing

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

In the era of Industry 4.0, human plays an important role in the design, installation, updating, and maintenance of the intelligent manufacturing system. To facilitate natural and convenient interactions between humans and machines, we need to develop advanced human-machine interaction technologies. In this paper, we propose a novel gesture recognition system by integrating the advantages of Doppler radar-based wireless sensing and fog computing, which is able to facilitate non-invasive and real-time human-machine interactions. We first collect and preprocess the dual channel Doppler information (i.e., I and Q signals), and then adopt a threshold detection method to extract gesture segments. Afterwards, we propose a two-stage classification method to recognize human gestures. We implement the system in real-world environments and recruit volunteers for performance evaluation. Experimental results show that our system can achieve accurate gesture recognition with in less than 1 s. Particularly, the average accuracy for motion detection and gesture recognition is 98.6% and 96.4%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Abdelnasser H, Youssef M, Harras KA (2015) Wigest: a ubiquitous wifi-based gesture recognition system. In: The 34th annual IEEE international conference on computer communications, pp 1472–1480

  2. Adib F, Mao H, Kabelac Z, Katabi D, Miller RC (2015) Smart homes that monitor breathing and heart rate. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI’15. ACM, New York, pp 837–846

  3. Agrawal S, Constandache I, Gaonkar S, Roy Choudhury R, Caves K, DeRuyter F (2011) Using mobile phones to write in air. In: Proceedings of the 9th international conference on mobile systems, applications, and services, MobiSys’11. ACM, New York, pp 15–28

  4. Al-qaness MAA, Li F (2016) Wiger: WiFi-based gesture recognition system. ISPRS International Journal of Geo-Information 5(6):92

    Article  Google Scholar 

  5. Ali K, Liu AX, Wang W, Shahzad M (2015) Keystroke recognition using wifi signals. In: Proceedings of the 21st annual international conference on mobile computing and networking, MobiCom’15. ACM, New York, pp 90–102

  6. Chen C, Jiao S, Zhang S, Liu W, Feng L, Wang Y (2018) Tripimputor: real-time imputing taxi trip purpose leveraging multi-sourced urban data. IEEE Trans Intell Transp Syst, 1–13. https://doi.org/10.1109/TITS.2017.2771231

  7. Chetty K, Chen Q, Ritchie M, Woodbridge K (2017) A low-cost through-the-wall FMCW radar for stand-off operation and activity detection

  8. Dura-Bernal S, Garreau G, Andreou C, Andreou A, Georgiou J, Wennekers T, Denham S (2011) Human action categorization using ultrasound micro-doppler signatures. In: Salah AA, Lepri B (eds) Human behavior understanding. Springer, Berlin, pp 18–28

  9. Fang B, Lane ND, Zhang M, Boran A, Kawsar F (2016) Bodyscan: enabling radio-based sensing on wearable devices for contactless activity and vital sign monitoring. In: Proceedings of the 14th annual international conference on mobile systems, applications, and services, MobiSys’16. ACM, New York, pp 97–110

  10. Fowler JE (2005) The redundant discrete wavelet transform and additive noise. IEEE Signal Process Lett 12 (9):629–632

    Article  Google Scholar 

  11. Gorecky D, Schmitt M, Loskyll M, Zhlke D (2014) Human-machine-interaction in the industry 4.0 era. In: 2014 12th IEEE international conference on industrial informatics (INDIN), pp 289–294

  12. Inc. C (2015) Fog computing and the internet of things: extend the cloud to where the things are Cisco White Paper

  13. Ketabdar H, Moghadam P, Naderi B, Roshandel M (2012) Magnetic signatures in air for mobile devices. In: Proceedings of the 14th international conference on human-computer interaction with mobile devices and services companion, MobileHCI’12. ACM, New York, pp 185–188

  14. Khan MAAH, Kukkapalli R, Waradpande P, Kulandaivel S, Banerjee N, Roy N, Robucci R (2016) Ram: radar-based activity monitor. In: The 35th Annual IEEE international conference on computer communications, pp 1–9

  15. Kim DB, Hong SM (2013) Multiple-target tracking and track management for an FMCW radar network. EURASIP Journal on Advances in Signal Processing 2013(1):159

    Article  Google Scholar 

  16. Kim Y, Ling H (2009) Human activity classification based on micro-Doppler signatures using a support vector machine. IEEE Trans Geosci Remote Sens 47(5):1328–1337

    Article  Google Scholar 

  17. Li H, Yang W, Wang J, Xu Y, Huang L (2016) Wifinger: talk to your smart devices with finger-grained gesture. In: Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing, UbiComp’16. ACM, New York, pp 250–261

  18. Li Z, Robucci R, Banerjee N, Patel C (2015) Tongue-n-cheek: non-contact tongue gesture recognition. In: Proceedings of the 14th international conference on information processing in sensor networks, IPSN’15. ACM, New York, pp 95–105

  19. Molchanov P, Gupta S, Kim K, Pulli K (2015) Multi-sensor system for driver’s hand-gesture recognition. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–8

  20. Molchanov P, Gupta S, Kim K, Pulli K (2015) Short-range FMCW monopulse radar for hand-gesture sensing. In: 2015 IEEE Radar conference, pp 1491–1496

  21. Mouradian C, Naboulsi D, Yangui S, Glitho RH, Morrow MJ, Polakos PA (2018) A comprehensive survey on fog computing: state-of-the-art and research challenges. IEEE Commun Surv Tutorials 20(1):416–464

    Article  Google Scholar 

  22. Ni LM, Liu Y, Lau YC, Patil AP (2004) Landmarc: indoor location sensing using active rfid. Wirel Netw 10(6):701–710

    Article  Google Scholar 

  23. Park T, Lee J, Hwang I, Yoo C, Nachman L, Song J (2011) E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In: Proceedings of the 9th ACM conference on embedded networked sensor systems, SenSys’11. ACM, New York, pp 260– 273

  24. Peng Z, Li C, Mu?oz-ferreras J, Gmez-Garca R (2017) An FMCW radar sensor for human gesture recognition in the presence of multiple targets. In: 2017 First IEEE MTT-s international microwave bio conference (IMBIOC), pp 1–3

  25. Plouffe G, Cretu A (2016) Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans Instrum Meas 65(2):305–316

    Article  Google Scholar 

  26. Pu Q, Gupta S, Gollakota S, Patel S (2013) Whole-home gesture recognition using wireless signals. In: Proceedings of the 19th annual international conference on mobile computing and networking, MobiCom’13. ACM, New York, pp 27–38

  27. Ralston TS, Charvat GL, Peabody JE (2010) Real-time through-wall imaging using an ultrawideband multiple-input multiple-output (mimo) phased array radar system. In: 2010 IEEE international symposium on phased array systems and technology, pp 551–558

  28. Sigg S, Scholz M, Shi S, Ji Y, Beigl M (2014) RF-sensing of activities from non-cooperative subjects in device-free recognition systems using ambient and local signals. IEEE Trans Mob Comput 13(4):907–920

    Article  Google Scholar 

  29. Sigg S, Shi S, Buesching F, Ji Y, Wolf L (2013) Leveraging RF-channel fluctuation for activity recognition: active and passive systems, continuous and RSSI-based signal features. In: Proceedings of international conference on advances in mobile computing and multimedia, MoMM’13. ACM, New York, pp 43–52

  30. Stojmenovic I, Wen S (2014) The fog computing paradigm: scenarios and security issues. In: 2014 federated conference on computer science and information systems, pp 1–8

  31. Wan J, Cai H, Zhou K (2015) Industrie 4.0: enabling technologies. In: Proceedings of 2015 international conference on intelligent computing and internet of things, pp 135–140

  32. Wang G, Zou Y, Zhou Z, Wu K, Ni LM (2014) We can hear you with wi-fi!. In: Proceedings of the 20th annual international conference on mobile computing and networking, MobiCom’14. ACM, New York, pp 593–604

  33. Wang Y, Liu J, Chen Y, Gruteser M, Yang J, Liu H (2014) E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In: Proceedings of the 20th annual international conference on mobile computing and networking, MobiCom’14. ACM, New York, pp 617–628

  34. Wang Y, Wu K, Ni LM (2017) Wifall: device-free fall detection by wireless networks. IEEE Trans Mob Comput 16(2):581–594

    Article  Google Scholar 

  35. Wang Z, Guo B, Yu Z, Zhou X (2018) Wi-Fi CSI-based behavior recognition: from signals and actions to activities. IEEE Commun Mag 56(5):109–115

    Article  Google Scholar 

  36. Wang Z, Zhang D, Zhou X, Yang D, Yu Z, Yu Z (2014) Discovering and profiling overlapping communities in location-based social networks. IEEE Trans Syst Man Cybern Syst Hum 44(4):499–509

    Article  Google Scholar 

  37. Wu D, Zhang D, Xu C, Wang Y, Wang H (2016) Widir: walking direction estimation using wireless signals. In: Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing, UbiComp’16, ACM, pp 351–362

  38. Xin T, Guo B, Wang Z, Wang P, Lam J, Li V, Yu Z (2018) Freesense: a robust approach for indoor human detection using wi-fi signals. Proc ACM Interact Mob Wearable Ubiquitous Technol 2(3):143

    Article  Google Scholar 

  39. Xin T, Guo B, Wang Z, Wang P, Yu Z (2018) Freesense: human-behavior understanding using wi-fi signals. J Ambient Intell Humaniz Comput 9(5):1611–1622

    Article  Google Scholar 

  40. Yang J, Lee J, Choi J (2011) Activity recognition based on RFID object usage for smart mobile devices. J Comput Sci Technol 26(2):239–246

    Article  Google Scholar 

  41. Yang L, Chen Y, Li XY, Xiao C, Li M, Liu Y (2014) Tagoram: real-time tracking of mobile RFID tags to high precision using cots devices. In: Proceedings of the 20th annual international conference on mobile computing and networking, MobiCom’14. ACM, New York, pp 237–248

  42. Yang Z, Zhou Z, Liu Y (2013) From RSSI to CSI: indoor localization via channel response. ACM Comput Surv 46(2):25:1–25:32

    Article  MATH  Google Scholar 

  43. Yu Z, Xu H, Yang Z, Guo B (2016) Personalized travel package with multi-point-of-interest recommendation based on crowdsourced user footprints. IEEE Transactions on Human-Machine Systems 46(1):151–158

    Article  Google Scholar 

  44. Zhang D, Wang H, Wu D (2017) Toward centimeter-scale human activity sensing with wi-fi signals. Computer 50(1):48–57

    Article  Google Scholar 

  45. Zheng X, Wang J, Shangguan L, Zhou Z, Liu Y (2016) Smokey: ubiquitous smoking detection with commercial wifi infrastructures. In: The 35th annual IEEE international conference on computer communications, pp 1–9

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their valuable comments.

Funding

This work was partially supported by the National Key R&D Program of China (No. 2016YFB1001401), the National Natural Science Foundation of China (No. 61332013, 61772428, 61725205), and the Innovative Talents Promotion Program of Shaanxi Province (No. 2018KJXX-011).

Author information

Authors and Affiliations

  1. School of Computer Science, Northwestern Polytechnical University, Xi’an, 710072, China

    Zhu Wang, Xinye Lou, Zhiwen Yu, Bin Guo & Xingshe Zhou

Authors
  1. Zhu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinye Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhiwen Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xingshe Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhu Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Lou, X., Yu, Z. et al. Enabling non-invasive and real-time human-machine interactions based on wireless sensing and fog computing. Pers Ubiquit Comput 23, 29–41 (2019). https://doi.org/10.1007/s00779-018-1185-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-018-1185-7

Keywords

Search

Navigation