Skip to main content

Advertisement

SpringerLink
Log in

A real-time fall detection model based on BlazePose and improved ST-GCN

  • Research
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Providing timely rescue when a fall occurs can greatly reduce fall mortality for older people. With the growing number of single-resided elders, real-time smart fall incident detection has become a new research hotspot. Accuracy, computational complexity and real-time response are key issues to be solved in this topic. A fall detection model that combines an improved Spatial–Temporal Graph Convolutional Network (ST-GCN) with the BlazePose algorithm is proposed in this paper. The computational speed is improved by removing four redundant layers from the ST-GCN network. Meanwhile, an attention mechanism focussing on the key joints involved in the falling action and their correlation is applied in the model, which introduces an Effective SE Block (ESE Block) to the residual structure of ST-GCN. It is achieved by fusing the original features with channel attention weights obtained by global average pooling and fully connected operations for the joint features. The BlazePose algorithm of the mediapipe framework is used to recognise human targets and locate the spatial coordinates of specific joints. Then the spatiotemporal graph features of the human body are extracted by the improved ST-GCN from the temporal and spatial displacements of 30 consecutive frames. Furthermore, fall behaviour is judged by the action type defined by the spatiotemporal graph. The accuracy of the proposed model for public datasets, such as Le2i Fall, Multicam Fall and UR Fall, is 99.29%, 99.22% and 98.64% respectively, which are higher than the Alphapose + ST-GCN model by 9.04%, 20% and 25.2%. Such accuracy is even better than the existing best algorithms by 0.89%, 0.92% and 1.04%. When running on the i5-10200H CPU and the Jetson Nano edge computing device, the Alphapose + ST-GCN model achieves frame rates of 11.42fps and 1.5fps, whilst the frame rates of this paper are up to 24.5fps and 9.37fps. The experimental results fully show that based on BlazePose with the improved ST-GCN makes the fall detection model higher accuracy, faster speed, real-time performance and high compatibility with the Jetson Nano edge computing device.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig.1
Fig.2
Fig.3
Fig.4
Fig.5
Fig.6
Fig.7
Fig. 8
Fig.9
Fig.10
Fig. 11
Fig.12
Fig.13

Similar content being viewed by others

Data availability

The data used to support the findings of this study are available from the corresponding author upon request.

References

  1. Daher, M., Diab, A., El Najjar, M.E., Khalil, M.A., Charpillet, F.: Elder tracking and fall detection system using smart tiles. IEEE Sensors J. 17, 469–479 (2017). https://doi.org/10.1109/JSEN.2016.2625099

    Article  Google Scholar 

  2. Zigel, Y., Litvak, D., Gannot, I.: A method for automatic fall detection of elderly people using floor vibrations and sound—proof of concept on human mimicking doll falls. IEEE Trans. Biomed. Eng. 56, 2858–2867 (2009). https://doi.org/10.1109/TBME.2009.2030171

    Article  Google Scholar 

  3. Salman Khan, M., Yu, M., Feng, P., Wang, L., Chambers, J.: An unsupervised acoustic fall detection system using source separation for sound interference suppression. Signal Process. 110, 199–210 (2015). https://doi.org/10.1016/j.sigpro.2014.08.021

    Article  Google Scholar 

  4. Lee, J.-S., Tseng, H.-H.: Development of an enhanced threshold-based fall detection system using smartphones with built-in accelerometers. IEEE Sensors J. 19, 8293–8302 (2019). https://doi.org/10.1109/JSEN.2019.2918690

    Article  Google Scholar 

  5. Xi, X., Jiang, W., Lü, Z., Miran, S.M., Luo, Z.-Z.: Daily activity monitoring and fall detection based on surface electromyography and plantar pressure. Complexity 2020, 1–12 (2020). https://doi.org/10.1155/2020/9532067

    Article  Google Scholar 

  6. Wang, X., Ellul, J., Azzopardi, G.: Elderly fall detection systems: a literature survey. Front Robot AI. 7, 71 (2020). https://doi.org/10.3389/frobt.2020.00071

    Article  Google Scholar 

  7. Serpa YR, Nogueira MB, Neto PP, Rodrigues MA (2020) Evaluating Pose Estimation as a Solution to the Fall Detection Problem. In: 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH). pp. 1–7. IEEE, Vancouver, BC, Canada

  8. Cao, J., Zhu, Y., Zhang, Y., Lyu, J., Yang, H.: Fall detection algorithm based on joint point features. J. Comput. Appl. 42, 622–630 (2022)

    Google Scholar 

  9. Yadav, S.K., Tiwari, K., Pandey, H.M., Akbar, S.A.: Skeleton-based human activity recognition using ConvLSTM and guided feature learning. Soft. Comput. 26, 877–890 (2022). https://doi.org/10.1007/s00500-021-06238-7

    Article  Google Scholar 

  10. Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y.: Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1302–1310. IEEE, Honolulu, HI (2017)

  11. Fang HS, Xie S, Tai YW, Lu C. RMPE: Regional Multi-person Pose Estimation. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 2353–2362. IEEE, Venice (2017)

  12. Bazarevsky V, Grishchenko I, Raveendran K, Zhu T, Zhang F, Grundmann M BlazePose: On-device Real-time Body Pose tracking, http://arxiv.org/abs/2006.10204, (2020)

  13. Hu J, Shen L, Sun G. Squeeze-and-Excitation Networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7132–7141. IEEE, Salt Lake City, UT (2018)

  14. Lee Y, Park J CenterMask: Real-Time Anchor-Free Instance Segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13903–13912. IEEE, Seattle, WA, USA (2020)

  15. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: Convolutional block attention module. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018, pp. 3–19. Springer International Publishing, Cham (2018)

    Chapter  Google Scholar 

  16. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. AAAI (2018). https://doi.org/10.1609/aaai.v32i1.12328

    Article  Google Scholar 

  17. Shi L, Zhang Y, Cheng J, Lu H Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12018–12027. IEEE, Long Beach, CA, USA (2019)

  18. Duan H, Wang J, Chen K, Lin D PYSKL: Towards Good Practices for Skeleton Action Recognition, http://arxiv.org/abs/2205.09443, (2022)

  19. Charfi, I., Miteran, J., Dubois, J., Atri, M., Tourki, R.: Optimized spatio-temporal descriptors for real-time fall detection: comparison of support vector machine and Adaboost-based classification. J. Electron. Imaging 22, 041106 (2013). https://doi.org/10.1117/1.JEI.22.4.041106

    Article  Google Scholar 

  20. Kwolek, B., Kepski, M.: Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput. Methods Programs Biomed. 117, 489–501 (2014). https://doi.org/10.1016/j.cmpb.2014.09.005

    Article  Google Scholar 

  21. Auvinet E, Rougier C, Meunier J, St-Arnaud A, Rousseau J: Multiple cameras fall data set.

  22. Wang Q, Wu B, Zhu P, Li, P, Zuo W, Hu Q ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks, http://arxiv.org/abs/1910.03151, (2020)

  23. Wang, B.-H., Yu, J., Wang, K., Bao, X.-Y., Mao, K.-M.: Fall detection based on dual-channel feature integration. IEEE Access. 8, 103443–103453 (2020). https://doi.org/10.1109/ACCESS.2020.2999503

    Article  Google Scholar 

  24. Alaoui, A.Y., El Fkihi, S., Thami, R.O.H.: Fall detection for elderly people using the variation of key points of human skeleton. IEEE Access. 7, 154786–154795 (2019). https://doi.org/10.1109/ACCESS.2019.2946522

    Article  Google Scholar 

  25. Chang, W.-J., Hsu, C.-H., Chen, L.-B.: A pose estimation-based fall detection methodology using artificial intelligence edge computing. IEEE Access 9, 12 (2021)

    Article  Google Scholar 

  26. Zheng, H., Liu, Y.: Lightweight fall detection algorithm based on alphapose optimization model and ST-GCN. Math. Probl. Eng. 2022, 1–15 (2022). https://doi.org/10.1155/2022/9962666

    Article  Google Scholar 

  27. Tian, Z., Zhang, L., Wang, G., Wang, X.: An RGB camera-based fall detection algorithm in complex home environments. Interdiscip. Nurs. Res. 1, 14–26 (2022). https://doi.org/10.1097/NR9.0000000000000007

    Article  Google Scholar 

  28. Zhang H, Cui W, Shi T, Tao Y, Zhang J ATMLP: Attention and Time Series MLP for Fall Detection. 53, (2023)

  29. Yuan J, Liu C, Liu C, Wang L, Chen Q Real-Time Human Falling Recognition via Spatial and Temporal Self-Attention Augmented Graph Convolutional Network. In: 2022 IEEE International Conference on Real-time Computing and Robotics (RCAR). pp. 438–443. IEEE, Guiyang, China (2022)

  30. Harrou, F., Zerrouki, N., Sun, Y., Houacine, A.: An integrated vision-based approach for efficient human fall detection in a home environment. IEEE Access. 7, 114966–114974 (2019). https://doi.org/10.1109/ACCESS.2019.2936320

    Article  Google Scholar 

  31. Vishnu, C., Datla, R., Roy, D., Babu, S., Mohan, C.K.: Human fall detection in surveillance videos using fall motion vector modeling. IEEE Sensors J. 21, 17162–17170 (2021). https://doi.org/10.1109/JSEN.2021.3082180

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by Guangdong Province Enterprise Science and Technology Commissioner Project (GDKTP20210557700).

Author information

Authors and Affiliations

  1. College of Electronic Engineering (College of Artificial Intelligence), South China Agricultural University, Guangzhou, China

    Yu Zhang, Junsi Gan, Zewei Zhao, Junliang Chen, Xiaofeng Chen & Yinliang Diao

  2. College of Mathematics and Informatics (College of Software Engineering), South China Agricultural University, Guangzhou, China

    Shuqin Tu

Authors
  1. Yu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junsi Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zewei Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junliang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaofeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yinliang Diao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shuqin Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors have equal contribution in this manuscript.

Corresponding author

Correspondence to Yinliang Diao.

Ethics declarations

Conflict of interest

The authors of this manuscript declare that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Gan, J., Zhao, Z. et al. A real-time fall detection model based on BlazePose and improved ST-GCN. J Real-Time Image Proc 20, 121 (2023). https://doi.org/10.1007/s11554-023-01377-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11554-023-01377-6

Keywords

Search

Navigation