Abstract
The growing demand for seamless human–computer interaction and accurate health monitoring has driven interest in device-free human behavior recognition. This paper presents a novel approach for simultaneous human activity recognition and pose estimation, utilizing a skeleton-based methodology that leverages WiFi Channel State Information (CSI). The proposed system, integrated with YOLOv8 and Mediapipe frameworks, is meticulously designed to accurately identify human skeletal structures and poses, overcoming limitations in conventional methodologies. Comprehensive experiments in indoor environments demonstrate the system's effectiveness in precisely recognizing and categorizing diverse human activities within WiFi-covered areas. The results highlight the robustness of the proposed system in simultaneous human activity recognition and pose estimation with precision and reliability.
Similar content being viewed by others
Data availability
There are no supplementary materiel, and the data is available upon reasonable request.
References
Liu, J., Liu, H., Chen, Y., Wang, Y., Wang, C.: Wireless sensing for human activity: a survey. IEEE Commun. Surv. Tutor. 22, 1629–1645 (2020). https://doi.org/10.1109/COMST.2019.2934489
Arshad, M.H., Bilal, M., Gani, A.: Human activity recognition: review. Taxonom. Open Chall. Sens. 22, 6463 (2022). https://doi.org/10.3390/s22176463
Gupta, N., Gupta, S.K., Pathak, R.K., Jain, V., Rashidi, P., Suri, J.S.: Human activity recognition in artificial intelligence framework: a narrative review. Artif. Intell. Rev. 55, 4755–4808 (2022). https://doi.org/10.1007/s10462-021-10116-x
Hussain, Z., Sheng, M., Zhang, W.E.: Different approaches for human activity recognition: a survey. J. Netw. Comput. Appl. 167, 102738 (2020). https://doi.org/10.1016/j.jnca.2020.102738
Zhou, H., Gao, Y., Song, X., Liu, W., Dong, W.: LimbMotion: decimeter-level limb tracking for wearable-based human–computer interaction. Proc ACM Interact Mob Wearable Ubiquitous Technol (2020). https://doi.org/10.1145/3369836
Mukhopadhyay, S.C.: Wearable sensors for human activity monitoring: a review. IEEE Sens. J. 15, 1321–1330 (2015). https://doi.org/10.1109/JSEN.2014.2370945
Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recognit. Lett. 119, 3–11 (2019). https://doi.org/10.1016/j.patrec.2018.02.010
Wang, J., Gao, Q., Pan, M., Fang, Y.: Device-free wireless sensing: challenges, opportunities, and applications. IEEE Netw. 32, 132–137 (2018). https://doi.org/10.1109/MNET.2017.1700133
Zhang, R., Jing, X., Wu, S., Jiang, C., Mu, J., Yu, F.R.: Device-free wireless sensing for human detection: the deep learning perspective. IEEE Internet Things J. 8, 2517–2539 (2021). https://doi.org/10.1109/JIOT.2020.3024234
Hussain, Z., Sheng, Q.Z., Zhang, W.E.: A review and categorization of techniques on device-free human activity recognition. J. Netw. Comput. Appl. 167, 102738 (2020). https://doi.org/10.1016/j.jnca.2020.102738
Ahmed, H.F.T., Ahmad, H.C.V.A.: Device free human gesture recognition using Wi-Fi CSI: A survey. Eng. Appl. Artif. Intell. 87, 103281 (2020). https://doi.org/10.1016/j.engappai.2019.103281
Adib, F., Katabi, D.: See through walls with WiFi! ACM SIGCOMM Comput. Commun. Rev. 43, 75–86 (2013). https://doi.org/10.1145/2486001.2486039
Boudlal, H., Serrhini, M., Tahiri, A.: An Intelligent Monitoring Approach Based on WiFi Sensing for Smart Hospital. In: Farhaoui, Y., Rocha, A., Brahmia, Z., Bhushab, B. (eds.) Artificial Intelligence and Smart Environment, pp. 212–223. Springer International Publishing, Cham (2023). https://doi.org/10.1007/978-3-031-26254-8_30
Boudlal, H., Serrhini, M., Tahiri, A.: Design and deployment of a practical wireless sensing system for HAR with WiFi CSI in the 5 GHz band. In: 2023 IEEE International Conference on Networking, Sensing and Control (ICNSC). pp. 1–6 (2023). https://doi.org/10.1109/ICNSC58704.2023.10319031.
Boudlal, H., Serrhini, M., Tahiri, A.: Leveraging WiFi channel state information for efficient and reliable device-free human activity recognition. In: 2023 IEEE 6th International Conference on Cloud Computing and Artificial Intelligence: Technologies and Applications (CloudTech). pp. 1–8 (2023). https://doi.org/10.1109/CloudTech58737.2023.10366171.
Zheng, T., Chen, Z., Luo, J., Ke, L., Zhao, C., Yang, Y.: SiWa: see into walls via deep UWB radar. In: Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. pp. 323–336 (2021). https://doi.org/10.1145/3447993.3483258.
Guan, J., Madani, S., Jog, S., Gupta, S., Hassanieh, H.: Through fog high-resolution imaging using millimeter wave radar. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11461–11470. IEEE, Seattle, WA, USA (2020). https://doi.org/10.1109/CVPR42600.2020.01148.
Yang, Z., Zhou, Z., Liu, Y.: From RSSI to CSI: indoor localization via channel response. ACM Comput. Surv. 46, 1–32 (2013). https://doi.org/10.1145/2543581.2543592
Boudlal, H., Serrhini, M., Tahiri, A.: A comprehensive review of WiFi sensing technologies: tools. Chall. Future Res. Directions (2023). https://doi.org/10.1063/5.0148509
Boudlal, H., Serrhini, M., Tahiri, A.: Human activity monitoring system with commodity WiFi infrastructure using channel state information. Indones. J. Electr. Eng. Comput. Sci. 31, 763 (2023). https://doi.org/10.11591/ijeecs.v31.i2.pp763-776
Boudlal, H., Serrhini, M., Tahiri, A.: A monitoring system for elderly people using WiFi sensing with channel state information. Int. J. Interact. Mob. Technol. 17, 112–131 (2023). https://doi.org/10.3991/ijim.v17i12.36613
Halperin, D., Hu, W., Sheth, A., Wetherall, D.: Tool release: gathering 802.11n traces with channel state information. ACM SIGCOMM Comput. Commun. Rev. 41, 53–53 (2011). https://doi.org/10.1145/1925861.1925870
Ong, E.H., Kneckt, J., Alanen, O., Chang, Z., Huovinen, T., Nihtilä, T.: IEEE 802.11ac: Enhancements for very high throughput WLANs. In: 2011 IEEE 22nd International Symposium on Personal, Indoor and Mobile Radio Communications. pp. 849–853 (2011). https://doi.org/10.1109/PIMRC.2011.6140087.
CSI-Tool-Camera-Shooting/csi_fun.h at master qiyinghua/CSI-Tool-Camera-Shooting, https://github.com/qiyinghua/CSI-Tool-Camera-Shooting, Last Accessed 2023/12/03.
MATLAB-MathWorks-MATLAB & Simulink, https://www.mathworks.com/products/matlab.html, Last Accessed 2023/12/03.
Guo, L., Lu, Z., Zhou, S., Wen, X., He, Z.: When Healthcare Meets Off-the-Shelf WiFi: A Non-Wearable and Low-Costs Approach for In-Home Monitoring, http://arxiv.org/abs/2009.09715, (2020).
Schäfer, J., Barrsiwal, B.R., Kokhkharova, M., Adil, H., Liebehenschel, J.: Human activity recognition using CSI information with nexmon. Appl. Sci. 11, 8860 (2021). https://doi.org/10.3390/app11198860
Xu, Y., Weaver, J.B., Healy, D.M., Lu, J.: Wavelet transform domain filters: a spatially selective noise filtration technique. IEEE Trans. Image Process. 3, 747–758 (1994). https://doi.org/10.1109/83.336245
Li, W., Bocus, M.J., Tang, C., Vishwakarma, S., Piechocki, R.J., Woodbridge, K., Chetty, K.: A Taxonomy of WiFi Sensing: CSI vs Passive WiFi Radar. In: 2020 IEEE Globecom Workshops (GC Wkshps. pp. 1–6. IEEE, Taipei, Taiwan (2020). https://doi.org/10.1109/GCWkshps50303.2020.9367546.
Ma, Y., Zhou, G., Wang, S.: WiFi sensing with channel state information: a survey. ACM Comput. Surv. 52, 1–36 (2020). https://doi.org/10.1145/3310194
Terven, J., Cordova-Esparza, D.: A Comprehensive Review of YOLO: From YOLOv1 and Beyond, http://arxiv.org/abs/2304.00501, (2023).
Hussain, M.: YOLO-v1 to YOLO-v8, the rise of YOLO and its complementary nature toward digital manufacturing and industrial defect detection. Machines. 11, 677 (2023). https://doi.org/10.3390/machines11070677
MediaPipe | Google for Developers, https://developers.google.com/mediapipe., Last Accessed 2023/12/04.
Kim, J.-W., Choi, J.-Y., Ha, E.-J., Choi, J.-H.: Human pose estimation using mediapipe pose and optimization method based on a humanoid model. Appl. Sci. 13, 2700 (2023). https://doi.org/10.3390/app13042700
Latreche, A., Kelaiaia, R., Chemori, A., Kerboua, A.: Reliability and validity analysis of MediaPipe-based measurement system for some human rehabilitation motions. Measurement 214, 112826 (2023). https://doi.org/10.1016/j.measurement.2023.112826
Jocher, G., Chaurasia, A., Qiu, J.: YOLO by Ultralytics, https://github.com/ultralytics/ultralytics, (2023).
Roboflow: Give your software the power to see objects in images and video, https://roboflow.com/, Last Accessed 2023/12/09.
Garg, S., Saxena, A., Gupta, R.: Yoga pose classification: a CNN and MediaPipe inspired deep learning approach for real-world application. J. Ambient. Intell. Humaniz. Comput. (2022). https://doi.org/10.1007/s12652-022-03910-0
Mekruksavanich, S., Phaphan, W., Hnoohom, N., Jitpattanakul, A.: Attention-based hybrid deep learning network for human activity recognition using WiFi channel state information. Appl. Sci. 13, 8884 (2023). https://doi.org/10.3390/app13158884
Jannat, M.K.A., Islam, Md.S., Yang, S.-H., Liu, H.: Efficient Wi-Fi-based human activity recognition using adaptive antenna elimination. IEEE Access. 11, 105440–105454 (2023). https://doi.org/10.1109/ACCESS.2023.3320069
Islam, M.S., Jannat, M.K.A., Hossain, M.N., Kim, W.-S., Lee, S.-W., Yang, S.-H.: STC-NLSTMNet: an improved human activity recognition method using convolutional neural network with NLSTM from WiFi CSI. Sensors 23, 356 (2022). https://doi.org/10.3390/s23010356
Mliki, H., Zaafouri, R., Hammami, M.: Human action recognition based on discriminant body regions selection. Signal Image Video Process. 12, 845–852 (2018). https://doi.org/10.1007/s11760-017-1227-z
Bulugu, I.: Gesture recognition system based on cross-domain CSI extracted from Wi-Fi devices combined with the 3D CNN. Signal Image Video Process. 17, 3201–3209 (2023). https://doi.org/10.1007/s11760-023-02545-8
Funding
The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.
Author information
Authors and Affiliations
Contributions
HB: Writing—review & editing, Writing—original draft, Methodology, Formal analysis, Data curation, Conceptualization. MS: Supervision, Writing—review & editing. AT: Writing—review & editing, Supervision.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Boudlal, H., Serrhini, M. & Tahiri, A. A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks. SIViP 18, 3673–3689 (2024). https://doi.org/10.1007/s11760-024-03031-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-024-03031-5