Skip to main content
Log in

A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The growing demand for seamless human–computer interaction and accurate health monitoring has driven interest in device-free human behavior recognition. This paper presents a novel approach for simultaneous human activity recognition and pose estimation, utilizing a skeleton-based methodology that leverages WiFi Channel State Information (CSI). The proposed system, integrated with YOLOv8 and Mediapipe frameworks, is meticulously designed to accurately identify human skeletal structures and poses, overcoming limitations in conventional methodologies. Comprehensive experiments in indoor environments demonstrate the system's effectiveness in precisely recognizing and categorizing diverse human activities within WiFi-covered areas. The results highlight the robustness of the proposed system in simultaneous human activity recognition and pose estimation with precision and reliability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Data availability

There are no supplementary materiel, and the data is available upon reasonable request.

References

  1. Liu, J., Liu, H., Chen, Y., Wang, Y., Wang, C.: Wireless sensing for human activity: a survey. IEEE Commun. Surv. Tutor. 22, 1629–1645 (2020). https://doi.org/10.1109/COMST.2019.2934489

    Article  Google Scholar 

  2. Arshad, M.H., Bilal, M., Gani, A.: Human activity recognition: review. Taxonom. Open Chall. Sens. 22, 6463 (2022). https://doi.org/10.3390/s22176463

    Article  Google Scholar 

  3. Gupta, N., Gupta, S.K., Pathak, R.K., Jain, V., Rashidi, P., Suri, J.S.: Human activity recognition in artificial intelligence framework: a narrative review. Artif. Intell. Rev. 55, 4755–4808 (2022). https://doi.org/10.1007/s10462-021-10116-x

    Article  Google Scholar 

  4. Hussain, Z., Sheng, M., Zhang, W.E.: Different approaches for human activity recognition: a survey. J. Netw. Comput. Appl. 167, 102738 (2020). https://doi.org/10.1016/j.jnca.2020.102738

    Article  Google Scholar 

  5. Zhou, H., Gao, Y., Song, X., Liu, W., Dong, W.: LimbMotion: decimeter-level limb tracking for wearable-based human–computer interaction. Proc ACM Interact Mob Wearable Ubiquitous Technol (2020). https://doi.org/10.1145/3369836

    Article  Google Scholar 

  6. Mukhopadhyay, S.C.: Wearable sensors for human activity monitoring: a review. IEEE Sens. J. 15, 1321–1330 (2015). https://doi.org/10.1109/JSEN.2014.2370945

    Article  Google Scholar 

  7. Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recognit. Lett. 119, 3–11 (2019). https://doi.org/10.1016/j.patrec.2018.02.010

    Article  Google Scholar 

  8. Wang, J., Gao, Q., Pan, M., Fang, Y.: Device-free wireless sensing: challenges, opportunities, and applications. IEEE Netw. 32, 132–137 (2018). https://doi.org/10.1109/MNET.2017.1700133

    Article  Google Scholar 

  9. Zhang, R., Jing, X., Wu, S., Jiang, C., Mu, J., Yu, F.R.: Device-free wireless sensing for human detection: the deep learning perspective. IEEE Internet Things J. 8, 2517–2539 (2021). https://doi.org/10.1109/JIOT.2020.3024234

    Article  Google Scholar 

  10. Hussain, Z., Sheng, Q.Z., Zhang, W.E.: A review and categorization of techniques on device-free human activity recognition. J. Netw. Comput. Appl. 167, 102738 (2020). https://doi.org/10.1016/j.jnca.2020.102738

    Article  Google Scholar 

  11. Ahmed, H.F.T., Ahmad, H.C.V.A.: Device free human gesture recognition using Wi-Fi CSI: A survey. Eng. Appl. Artif. Intell. 87, 103281 (2020). https://doi.org/10.1016/j.engappai.2019.103281

    Article  Google Scholar 

  12. Adib, F., Katabi, D.: See through walls with WiFi! ACM SIGCOMM Comput. Commun. Rev. 43, 75–86 (2013). https://doi.org/10.1145/2486001.2486039

    Article  Google Scholar 

  13. Boudlal, H., Serrhini, M., Tahiri, A.: An Intelligent Monitoring Approach Based on WiFi Sensing for Smart Hospital. In: Farhaoui, Y., Rocha, A., Brahmia, Z., Bhushab, B. (eds.) Artificial Intelligence and Smart Environment, pp. 212–223. Springer International Publishing, Cham (2023). https://doi.org/10.1007/978-3-031-26254-8_30

    Chapter  Google Scholar 

  14. Boudlal, H., Serrhini, M., Tahiri, A.: Design and deployment of a practical wireless sensing system for HAR with WiFi CSI in the 5 GHz band. In: 2023 IEEE International Conference on Networking, Sensing and Control (ICNSC). pp. 1–6 (2023). https://doi.org/10.1109/ICNSC58704.2023.10319031.

  15. Boudlal, H., Serrhini, M., Tahiri, A.: Leveraging WiFi channel state information for efficient and reliable device-free human activity recognition. In: 2023 IEEE 6th International Conference on Cloud Computing and Artificial Intelligence: Technologies and Applications (CloudTech). pp. 1–8 (2023). https://doi.org/10.1109/CloudTech58737.2023.10366171.

  16. Zheng, T., Chen, Z., Luo, J., Ke, L., Zhao, C., Yang, Y.: SiWa: see into walls via deep UWB radar. In: Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. pp. 323–336 (2021). https://doi.org/10.1145/3447993.3483258.

  17. Guan, J., Madani, S., Jog, S., Gupta, S., Hassanieh, H.: Through fog high-resolution imaging using millimeter wave radar. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11461–11470. IEEE, Seattle, WA, USA (2020). https://doi.org/10.1109/CVPR42600.2020.01148.

  18. Yang, Z., Zhou, Z., Liu, Y.: From RSSI to CSI: indoor localization via channel response. ACM Comput. Surv. 46, 1–32 (2013). https://doi.org/10.1145/2543581.2543592

    Article  Google Scholar 

  19. Boudlal, H., Serrhini, M., Tahiri, A.: A comprehensive review of WiFi sensing technologies: tools. Chall. Future Res. Directions (2023). https://doi.org/10.1063/5.0148509

    Article  Google Scholar 

  20. Boudlal, H., Serrhini, M., Tahiri, A.: Human activity monitoring system with commodity WiFi infrastructure using channel state information. Indones. J. Electr. Eng. Comput. Sci. 31, 763 (2023). https://doi.org/10.11591/ijeecs.v31.i2.pp763-776

    Article  Google Scholar 

  21. Boudlal, H., Serrhini, M., Tahiri, A.: A monitoring system for elderly people using WiFi sensing with channel state information. Int. J. Interact. Mob. Technol. 17, 112–131 (2023). https://doi.org/10.3991/ijim.v17i12.36613

    Article  Google Scholar 

  22. Halperin, D., Hu, W., Sheth, A., Wetherall, D.: Tool release: gathering 802.11n traces with channel state information. ACM SIGCOMM Comput. Commun. Rev. 41, 53–53 (2011). https://doi.org/10.1145/1925861.1925870

    Article  Google Scholar 

  23. Ong, E.H., Kneckt, J., Alanen, O., Chang, Z., Huovinen, T., Nihtilä, T.: IEEE 802.11ac: Enhancements for very high throughput WLANs. In: 2011 IEEE 22nd International Symposium on Personal, Indoor and Mobile Radio Communications. pp. 849–853 (2011). https://doi.org/10.1109/PIMRC.2011.6140087.

  24. CSI-Tool-Camera-Shooting/csi_fun.h at master qiyinghua/CSI-Tool-Camera-Shooting, https://github.com/qiyinghua/CSI-Tool-Camera-Shooting, Last Accessed 2023/12/03.

  25. MATLAB-MathWorks-MATLAB & Simulink, https://www.mathworks.com/products/matlab.html, Last Accessed 2023/12/03.

  26. Guo, L., Lu, Z., Zhou, S., Wen, X., He, Z.: When Healthcare Meets Off-the-Shelf WiFi: A Non-Wearable and Low-Costs Approach for In-Home Monitoring, http://arxiv.org/abs/2009.09715, (2020).

  27. Schäfer, J., Barrsiwal, B.R., Kokhkharova, M., Adil, H., Liebehenschel, J.: Human activity recognition using CSI information with nexmon. Appl. Sci. 11, 8860 (2021). https://doi.org/10.3390/app11198860

    Article  Google Scholar 

  28. Xu, Y., Weaver, J.B., Healy, D.M., Lu, J.: Wavelet transform domain filters: a spatially selective noise filtration technique. IEEE Trans. Image Process. 3, 747–758 (1994). https://doi.org/10.1109/83.336245

    Article  Google Scholar 

  29. Li, W., Bocus, M.J., Tang, C., Vishwakarma, S., Piechocki, R.J., Woodbridge, K., Chetty, K.: A Taxonomy of WiFi Sensing: CSI vs Passive WiFi Radar. In: 2020 IEEE Globecom Workshops (GC Wkshps. pp. 1–6. IEEE, Taipei, Taiwan (2020). https://doi.org/10.1109/GCWkshps50303.2020.9367546.

  30. Ma, Y., Zhou, G., Wang, S.: WiFi sensing with channel state information: a survey. ACM Comput. Surv. 52, 1–36 (2020). https://doi.org/10.1145/3310194

    Article  Google Scholar 

  31. Terven, J., Cordova-Esparza, D.: A Comprehensive Review of YOLO: From YOLOv1 and Beyond, http://arxiv.org/abs/2304.00501, (2023).

  32. Hussain, M.: YOLO-v1 to YOLO-v8, the rise of YOLO and its complementary nature toward digital manufacturing and industrial defect detection. Machines. 11, 677 (2023). https://doi.org/10.3390/machines11070677

    Article  Google Scholar 

  33. MediaPipe | Google for Developers, https://developers.google.com/mediapipe., Last Accessed 2023/12/04.

  34. Kim, J.-W., Choi, J.-Y., Ha, E.-J., Choi, J.-H.: Human pose estimation using mediapipe pose and optimization method based on a humanoid model. Appl. Sci. 13, 2700 (2023). https://doi.org/10.3390/app13042700

    Article  Google Scholar 

  35. Latreche, A., Kelaiaia, R., Chemori, A., Kerboua, A.: Reliability and validity analysis of MediaPipe-based measurement system for some human rehabilitation motions. Measurement 214, 112826 (2023). https://doi.org/10.1016/j.measurement.2023.112826

    Article  Google Scholar 

  36. Jocher, G., Chaurasia, A., Qiu, J.: YOLO by Ultralytics, https://github.com/ultralytics/ultralytics, (2023).

  37. Roboflow: Give your software the power to see objects in images and video, https://roboflow.com/, Last Accessed 2023/12/09.

  38. Garg, S., Saxena, A., Gupta, R.: Yoga pose classification: a CNN and MediaPipe inspired deep learning approach for real-world application. J. Ambient. Intell. Humaniz. Comput. (2022). https://doi.org/10.1007/s12652-022-03910-0

    Article  Google Scholar 

  39. Mekruksavanich, S., Phaphan, W., Hnoohom, N., Jitpattanakul, A.: Attention-based hybrid deep learning network for human activity recognition using WiFi channel state information. Appl. Sci. 13, 8884 (2023). https://doi.org/10.3390/app13158884

    Article  Google Scholar 

  40. Jannat, M.K.A., Islam, Md.S., Yang, S.-H., Liu, H.: Efficient Wi-Fi-based human activity recognition using adaptive antenna elimination. IEEE Access. 11, 105440–105454 (2023). https://doi.org/10.1109/ACCESS.2023.3320069

    Article  Google Scholar 

  41. Islam, M.S., Jannat, M.K.A., Hossain, M.N., Kim, W.-S., Lee, S.-W., Yang, S.-H.: STC-NLSTMNet: an improved human activity recognition method using convolutional neural network with NLSTM from WiFi CSI. Sensors 23, 356 (2022). https://doi.org/10.3390/s23010356

    Article  Google Scholar 

  42. Mliki, H., Zaafouri, R., Hammami, M.: Human action recognition based on discriminant body regions selection. Signal Image Video Process. 12, 845–852 (2018). https://doi.org/10.1007/s11760-017-1227-z

    Article  Google Scholar 

  43. Bulugu, I.: Gesture recognition system based on cross-domain CSI extracted from Wi-Fi devices combined with the 3D CNN. Signal Image Video Process. 17, 3201–3209 (2023). https://doi.org/10.1007/s11760-023-02545-8

    Article  Google Scholar 

Download references

Funding

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Author information

Authors and Affiliations

Authors

Contributions

HB: Writing—review & editing, Writing—original draft, Methodology, Formal analysis, Data curation, Conceptualization. MS: Supervision, Writing—review & editing. AT: Writing—review & editing, Supervision.

Corresponding author

Correspondence to Hicham Boudlal.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Boudlal, H., Serrhini, M. & Tahiri, A. A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks. SIViP 18, 3673–3689 (2024). https://doi.org/10.1007/s11760-024-03031-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-024-03031-5

Keywords

Navigation